mjhjmtu

Our Nagios server NTP has isn't working right. First here is the issues:

root@ccsd-lx-noc03 /var/log> tail -n 10000 messages | grep "NTP"
Jul 5 16:19:36 ccsd-lx-noc03 nagios: SERVICE ALERT: localhost;NTP-localhost;WARNING;HARD;4;NTP WARNING: Offset 53.03026778 secs
Jul 5 16:20:49 ccsd-lx-noc03 nagios: SERVICE ALERT: localhost;NTP-localhost;CRITICAL;HARD;4;NTP CRITICAL: Offset -84.96075022 secs
Jul 5 16:20:50 ccsd-lx-noc03 nagios: SERVICE ALERT: localhost;NTP2-localhost;CRITICAL;HARD;4;NTP CRITICAL: Offset -84.95908391 secs
Jul 5 16:22:49 ccsd-lx-noc03 nagios: SERVICE ALERT: localhost;NTP2-localhost;CRITICAL;HARD;4;NTP CRITICAL: Offset -84.96072233 secs
Jul 5 16:23:40 ccsd-lx-noc03 nagios: SERVICE ALERT: localhost;NTP-localhost;CRITICAL;HARD;4;NTP CRITICAL: Offset -84.96058169 secs
Jul 5 16:24:20 ccsd-lx-noc03 nagios: SERVICE ALERT: localhost;NTP2-localhost;WARNING;HARD;4;NTP WARNING: Offset 53.01928848 secs
Jul 5 16:24:44 ccsd-lx-noc03 nagios: SERVICE ALERT: localhost;NTP-localhost;CRITICAL;HARD;4;NTP CRITICAL: Offset -84.961512 secs
Jul 5 16:25:14 ccsd-lx-noc03 nagios: SERVICE ALERT: localhost;NTP2-localhost;CRITICAL;HARD;4;NTP CRITICAL: Offset -84.9693791 secs
Jul 5 16:26:01 ccsd-lx-noc03 nagios: SERVICE ALERT: localhost;NTP-localhost;CRITICAL;HARD;4;NTP CRITICAL: Offset -84.96211889 secs
Jul 5 16:26:18 ccsd-lx-noc03 nagios: SERVICE ALERT: localhost;NTP2-localhost;CRITICAL;HARD;4;NTP CRITICAL: Offset -71.26003572 secs
Jul 5 16:27:10 ccsd-lx-noc03 nagios: SERVICE ALERT: localhost;NTP2-localhost;CRITICAL;HARD;4;NTP CRITICAL: Offset -71.26059958 secs
Jul 5 16:27:20 ccsd-lx-noc03 nagios: SERVICE ALERT: localhost;NTP-localhost;WARNING;HARD;4;NTP WARNING: Offset 53.03374252 secs
Jul 5 16:27:32 ccsd-lx-noc03 nagios: SERVICE ALERT: localhost;NTP-localhost;CRITICAL;HARD;4;NTP CRITICAL: Offset -71.26115555 secs
Jul 5 16:28:00 ccsd-lx-noc03 nagios: SERVICE ALERT: localhost;NTP2-localhost;CRITICAL;HARD;4;NTP CRITICAL: Offset -84.96324414 secs
Jul 5 16:28:19 ccsd-lx-noc03 nagios: SERVICE ALERT: localhost;NTP-localhost;WARNING;HARD;4;NTP WARNING: Offset 53.03296909 secs
Jul 5 16:28:25 ccsd-lx-noc03 nagios: SERVICE ALERT: localhost;NTP-localhost;CRITICAL;HARD;4;NTP CRITICAL: Offset -84.96396494 secs
Jul 5 16:29:09 ccsd-lx-noc03 nagios: SERVICE ALERT: localhost;NTP2-localhost;CRITICAL;HARD;4;NTP CRITICAL: Offset -71.26274931 secs

Next, I am no the admin, he departed a few weeks ago, trying to keep things in order. On the crontab file I see this:

root@ccsd-lx-noc03 /data/nagios/var> crontab -l
59 * * * * /usr/sbin/ntpd -q > /dev/null 2>&1

How do I go about fixing this?

ntpdate -d time.ccsd.net
 5 Jul 17:58:48 ntpdate[5098]: ntpdate 4.2.6p5@1.2349-o Wed Jun 18 21:20:36 UTC 2014 (1)
Looking for host time.ccsd.net and service ntp
host found : ns1.ccsd.net
transmit(206.194.10.13)
receive(206.194.10.13)
transmit(206.194.10.13)
receive(206.194.10.13)
transmit(206.194.10.13)
receive(206.194.10.13)
transmit(206.194.10.13)
receive(206.194.10.13)
server 206.194.10.13, port 123
stratum 2, precision -23, leap 00, trust 000
refid [206.194.10.13], delay 0.02682, dispersion 0.00015
transmitted 4, in filter 4
reference time: d9444d0c.ef1c4dc0 Sun, Jul 5 2015 17:40:44.934
originate timestamp: d94450f9.d4061577 Sun, Jul 5 2015 17:57:29.828
transmit timestamp: d944514e.d51c3225 Sun, Jul 5 2015 17:58:54.832
filter delay: 0.02911 0.02684 0.02730 0.02682
 0.00000 0.00000 0.00000 0.00000
filter offset: -85.0055 -85.0048 -85.0046 -85.0048
 0.000000 0.000000 0.000000 0.000000
delay 0.02682, dispersion 0.00015
offset -85.004863
 5 Jul 17:58:54 ntpdate[5098]: step time server 206.194.10.13 offset -85.004863 sec

So I'm not sure why ntpdate seems to be running if you have NTP installed. You don't mention what OS/distribution this is on, so I'll have to keep this general, but:

1. Remove any crontab calls to ntpdate and remove the call to ntpd -q.


2. Remove ntpd from your system package manager and reinstall it.


3. View /etc/ntp.conf and make sure the upstream timeservers are specified correctly. The default `.pool.*" NTP servers will be fine in most cases.


4. After reinstalling, ntpd should already be running. If not, start it (usually service ntp start).


5. After giving it a minute or two to get in sync, run the command ntpq -p. You should see a * at the beginning of the line for one of the servers. This indicates that NTP is working and synchronizing.

The daemon should stay running forever--it's stable and simple shouldn't crash. Just make sure nothing else is interfering with it.

Reffering to the "pre-poster": don't keep ntpdate on your server if you use ntpd!

Considering for any reasons whatsoever why your system is drifting in time, whatcha gonna do 'bout it?

When you face time issues on Linux, firstly check any virtualization system - if you use VMware/Hyper-V/Xen/KVM/Whatever - think about what really is your "hardware clock" here!?! Sometimes your "BIOS" clock is misbehaving but it is as virtual as your network card(!) so if you use VMs talk to your colleagues in charge.

Considerring the differenciation of ntpd and ntpd put in a nutshell:
ntpd is "cheating" your system by slighty adding fractions of time to system time (date) until local system time is synced with whatever your time source says, whereas ntpdate sets your local time to "network time" promptly.

System time is what the command date will tell you on Linux. But there is also hwclock (needs sudo) which is indeed your hardware a.k.a BIOS clock.

For once - literally speaking - you may need to do

sudo hwclock ---systohc

to set your BIOS/hardware/virtual clock to system date - what ntpd says. If you need to do this more often, there is a problem.

I know for sure, this applies to Ubuntu, so I assume it also applies to Debian. Reason is that the system may call ntpdate in time (cron.weekly?) but when you have ntpd running, it would ususally complain! You simply can't run ntpdate when ntpd is running. Test it yourself on a fresh installation. So I assume - with your cronjob in mind - your ntpd is actually not running as a daemon. Why else would you start it every hour at minute 59 in silent (-q) mode?

Strange thing about your cronjob is, that it starts ntpd in "--quiet mode" - why so? It's a daemon, right? They do run in the background all the time, don't they?

Usually, you install ntpd because it owns a drift file that enables ntpd to adjust your time smoothly but "over time" - it may add here a fraction of a second and there but not forcing to reset the time to let's say 30 seconds in the future - or even worse the past - because you can assume e.g. most databases do not like time travels at all!

And that's what ntpd is for. ntpdate - in opposite - will promptly correct any time drifts what may cause time critical application issues - ntpd on the other hand will add some milliseconds to every past second so things will get the right time "in time", not surprising any application that is deeply relying on timestaps with BIG jumps in time for- or backwards. Serious Databases do rely on timestamps (e.g. mysql/postgres/mariadb).

A daemon should always run in the background, so I must assume your ex-admin was mixing ntpdate with ntpd - because usually - without ntpd - you would call ntpdate once in a while to adjust time promptly when drifted. ntpd instead should run and run and run and you would normally forget about it "in time" ...

Or, you have a very old system indeed, so you might consider upgrading depending on the crucialness of your Linux system.

I suggest, you should have your system checked by some Linux Admin because IMHO your system seems improperly configured.

You have been warned offically ;-)