[SOLVED] RHEL6: NTP not synchronising

[suaswe]

Hi all,

I'm currently struggling with an odd NTP synchronisation issue on a web server I maintain.

This is what I've got in /etc/ntp.conf:

tinker panic 0
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 10.0.0.0 mask 255.0.0.0 nomodify notrap
restrict 195.x.x.0 mask 255.255.255.0 nomodify notrap
restrict 81.x.x.0 mask 255.255.255.0 nomodify notrap
restrict 213.x.x.0 mask 255.255.254.0 nomodify notrap
restrict 127.0.0.1
restrict -6 ::1
server ntp.localdomain.com
restrict ntp.localdomain.com kod nomodify notrap
driftfile /var/lib/ntp/drift

Running ntpstat shows that NTP is not synchronised:

[root(a)server ~]# ntpstat
unsynchronised
   polling server every 16 s

Having a look in /var/log/messages flags up a lot of 'time reset' messages:

Jan 28 12:39:19 server ntpd[18875]: synchronized to 10.x.x.x, stratum 3
Jan 28 12:40:58 server ntpd[18875]: time reset +34.186686 s
Jan 28 12:56:13 server ntpd[18875]: synchronized to 10.x.x.x, stratum 3
Jan 28 12:57:53 server ntpd[18875]: time reset +34.225229 s
Jan 28 13:11:44 server ntpd[18875]: synchronized to 10.x.x.x, stratum 3
Jan 28 13:14:24 server ntpd[18875]: time reset +34.259120 s
Jan 28 13:32:24 server ntpd[18875]: synchronized to 10.x.x.x, stratum 3
Jan 28 13:34:03 server ntpd[18875]: time reset +34.311748 s
Jan 28 13:47:57 server ntpd[18875]: synchronized to 10.x.x.x, stratum 3
Jan 28 13:50:37 server ntpd[18875]: time reset +34.343890 s
Jan 28 14:07:08 server ntpd[18875]: synchronized to 10.x.x.x, stratum 3
Jan 28 14:08:47 server ntpd[18875]: time reset +34.378650 s
Jan 28 14:24:36 server ntpd[18875]: synchronized to 10.x.x.x, stratum 3
Jan 28 14:26:15 server ntpd[18875]: time reset +34.414954 s
Jan 28 14:48:07 server ntpd[18875]: synchronized to 10.x.x.x, stratum 3
Jan 28 14:49:47 server ntpd[18875]: time reset +34.466802 s
Jan 28 15:04:03 server ntpd[18875]: synchronized to 10.x.x.x, stratum 3
Jan 28 15:05:44 server ntpd[18875]: time reset +34.498446 s
Jan 28 15:19:15 server ntpd[18875]: synchronized to 10.x.x.x, stratum 3
Jan 28 15:22:00 server ntpd[18875]: time reset +34.536837 s

On advice of tips found via Google and suchlike I have manually set the time to match that of the NTP server using 'date -s <time>' then restarted ntpd, but this has had no effect.

The following is what happens when running ntpdate - note that the offset is quite high on the first ntpdate attempt, then drops down to a normal level on the second attempt:

[root(a)server ~]# cat /etc/ntp.conf | grep ^server
server ntp.localdomain.com

[root(a)server ~]# service ntpd stop
Shutting down ntpd:                                        [  OK  ]
[root(a)server ~]# ntpdate ntp.localdomain.com
28 Jan 15:35:38 ntpdate[17276]: step time server 10.x.x.x offset 34.551693 sec
[root(a)server ~]# ntpdate ntp.localdomain.com
28 Jan 15:35:39 ntpdate[17280]: adjust time server 10.x.x.x offset 0.000320 sec
[root(a)server ~]# ntpdate ntp.localdomain.com
28 Jan 15:35:40 ntpdate[17284]: adjust time server 10.x.x.x offset 0.000255 sec
[root(a)server ~]# service ntpd start
Starting ntpd:                                             [  OK  ]
[root(a)server ~]#

This is what I've got uncommented in ntp.conf:

[root(a)server ~]# grep -v ^# /etc/ntp.conf | grep -v ^$
tinker panic 0
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 10.0.0.0 mask 255.0.0.0 nomodify notrap
restrict 195.x.x.0 mask 255.255.255.0 nomodify notrap
restrict 81.x.x.0 mask 255.255.255.0 nomodify notrap
restrict 213.x.x.0 mask 255.255.254.0 nomodify notrap
restrict 127.0.0.1
restrict -6 ::1
server ntp.localdomain.com
restrict ntp.localdomain.com kod nomodify notrap
driftfile /var/lib/ntp/drift

Please note that the given NTP server is in use by many other servers on the same network, so I know that's not the issue.

Any tips on this would be fantastic, as I've been scratching my head over it for a while.

Cheers!

SuaSwe

[Rubberman]

We have a server in one of our data centers that has a similar problem - it is configured identically to all other 500 servers in the DC, yet drifts about 4-5 minutes per day, ntp notwithstanding. When our network operations team figures it out, I will repost here. One possibility is that the hardware clock has a problem and that the drift is too much for ntp to deal with unless you loosen the limits somewhat.

[Miven]

I'm not sure if this has anything to do with it, but, on my system, kernel keeps time from boot that is independent of ntp and the local clock. The discrepancy is about 4 minutes per week of uptime. I have not been able to resolve this issue, but my kernel is 2.6.6 and it may have been fixed in a newer version.

It easily becomes noticeable in the ps axu listings.

Also, my ntpd works fine, but all I have in /etc/ntpd.conf is

server  127.127.1.0
fudge   127.127.1.0 stratum 10
server ntp.cpsc.ucalgary.ca
driftfile /etc/ntp/drift
multicastclient 
broadcastdelay  0.008
authenticate no

ntpd version 4.2.0

[suaswe]

I would also have suspected the hardware clock but this server is a virtual machine running on VMWare, which excludes that possibility entirely as far as I know.

Miven, how is the problem obvious in the ps axu listings?

[suaswe]

I've just cracked it! The VMWare bit was the key: went to the vSphere GUI and had a look at the settings, and found an option called "Synchronise guest time with host", which was enabled for this server for some reason. Checked the ESX host on which the guest OS resides and found that ntpd was not started. Eureka! So, looks like the ESX node has been continuously resetting the time on the guest OS causing it to lose synch wtih the NTP server.. It's all making sense now. I've disabled the "Synchronise guest time with host" option as it is not required on this network, and NTP is now working as it should - happy days!