NTP performance, events, and notes on using NTP (or alternative presentation)

	PC Pixie 1.66GHz Intel Atom, 1GB FreeBSD 8.2, NTP Server ntpd 4.2.7 GPS 18 LVC GPS/PPS reference
	PC Feenix Intel Pentium 4 1.9GHz, 1GB Windows XP SP3, Main EUMETCast RX ntpd 4.2.7 Sure GPS evaluation board reference
	PC Stamsund 2.8GHz Pentium 4 HT, 3GB Windows-7 Ultimate 32-bit, EUMETCast backup ntpd 4.2.7 GPS 18x LVC GPS/PPS reference
	PC Alta Intel i5-760 2.8GHz quad-core 8GB Windows-7 64-bit, Interactive ntpd 4.2.7 Sure GPS evaluation board reference
	PC Bacchus Intel Pentium III 550MHz, 512MB Windows 2000 Server SP4, APT RX ntpd 4.2.7 Synced from PC Pixie/Feenix/Stamsund (sometimes from GPS 18 LVC/PPS)
	PC Hydra AMD 64 3200+, 2GB Windows-7 Ultimate 64-bit, EUMETCast tests ntpd 4.2.7 Synced from PC Pixie/Feenix/Stamsund
	PC Molde Intel Atom D330 dual-core HT, 4GB Windows-7 64-bit, Compact Entertainment PC ntpd 4.2.7 WiFi synced from PC Pixie/Feenix/Stamsund
	PC Narvik Intel E6600 dual-core 2.4GHz, 3GB Windows XP Pro SP3, Interactive ntpd 4.2.7 Synced from PC Pixie/Feenix/Stamsund
	PC Puffin Intel E8300 dual-core, 3GB Windows Vista SP2, Interactive use ntpd 4.2.7 WiFi synced from PC Pixie/Feenix/Stamsund
	Portable PC Bergen Intel T5450 dual-core 1.66GHz, 2GB Windows-8 32-bit, Interactive use ntpd 4.2.7 WiFi synced from PC Pixie/Feenix/Stamsund
	Portable PC Mercury Intel Celeron 1.6GHz, 1.25GB Windows-7 Home Premium 32-bit, Interactive ntpd 4.2.7 WiFi synced from PC Pixie/Feenix/Stamsund
	Portable PC Torvik Intel T3000 dual-core 1.8GHz, 2GB Windows-8 32-bit, Interactive ntpd 4.2.7 WiFi synced from PC Pixie/Feenix/Stamsund
	Portable PC Ystad Intel N455 Atom, 2GB Windows-7 Starter 32-bit, Plane Plotter use ntpd 4.2.7 WiFi synced from PC Pixie/Feenix/Stamsund
Click on a graph for weekly, monthly and yearly data

Please note, because I happen to use MRTG to gather and plot this data, and negative values aren't allowed, I needed to add a bias to the actual offset to derive the graphs above (100ms for Gemini, 3ms for most PCs, and 500µs for the much more accurate PCs Feenix and Stamsund).  An ideal timekeeper would therefore display a straight line at the mid-scale level.  The value plotted is the offset from the server clock that the NTP servers on each of the PCs reports when interrogated, every five minutes.  There appears to be an oddity with MRTG in that is looses small figures in the year data, so you may see a gradual drift upwards in some of the data, and that is probably just a recording artefact.
How I obtain this data.

Hardware and OS configuration

From April 2010, my primary NTP server is a FreeBSD server Pixie.  There are also two stratum-1 NTP servers, sometimes peered together, each fed from a separate GPS receiver.  One of the GPS receivers had its RS-232 output parallel-connected so that a temporary test could be made, and this was most recently used with a serial-to-USB converter box for testing the feasibility of using a USB connection where the PC has no serial port.  The parallel connection is also infrequently used for checking test versions of ntpd.exe on PC Bacchus, so Bacchus may well be a stratum-1 server at any particular time.  From April 2010, the FreeBSD server Pixie normally uses that parallel feed. The performance graph should be a good indicator!  Internet backup servers are configured for all PCs, with a much longer poll interval.  From February 2011 a Sure GPS module was added to the configuration - just as an experiment - and that is now feeding PC Feenix, with the paralleled RS-232 from the GPS-18 puck on the roof now feeding both the FreeBSD box Pixie and the Windows 2000 box, Bacchus.  In March 2011, a second Sure GPS board was acquired for testing.

Active NTP Versions

Configuration changes and other events

Please note that some transients are caused by system reboots, e.g. after a security update, and these events are not usually individually recorded.

• 09:00 UTC, Sunday, 2012-Dec-01, week 48.  Updated all PCs to use locally compiled ntp4.2.7p329, including Linux and FreeBSD systems.
• 12:50 UTC, Thursday, 2012-Nov-29, week 48.  Updated all PCs to use locally compiled ntp4.2.7p327.
• 15:20 UTC, Wednesday, 2012-Nov-21, week 47.  Updated all PCs to use locally compiled ntp4.2.7p326.  PC Bacchus excluded as VS 2010 compiled programs don't run on Windows 2000.
• 14:45 UTC, Monday, 2012-Nov-19, week 47.  Updated most PCs to use locally compiled ntp4.2.7p324 including Stamsund.  Stopped when I saw that the GPS serial was being reported with -0.5 seconds offset, so the new version has removed Dave Hart's tweak to mark the serial data as being at DCD time.  Have re-opened bug 2306 to see whether the general view is to leave this change or restore the Windows-specific tweak.  Was decided to restore the tweak.
• 08:45 UTC, Thursday, 2012-Nov-15, week 46.  Altered most PCs to use locally-compiled ntp4.2.7p321.  This version compiled with SSL 1.0.0c which produces .EXE files compatible with both the old and the new DLLs.
• 16:15 UTC, Sunday, 2012-Nov-11, week 45.  Altered most PCs to use locally-compiled ntp4.2.7p319.
• 13:15 UTC, Tuesday, 2012-Oct-30, week 44.  Altered most PCs to use locally-compiled ntp4.2.7p316.  Exceptions: Windows-2000 (my VS 2010 too new to produce Win 2K executables), Windows-8 (my SSL too old as the Meinberg new install has a more recent SSL library).
• 16:00 UTC, Monday, 2012-Otc-29, week 44.  Managed to get a Raspberry Pi working with a USB GPS and kernel-mode PPS.
• 05:31 UTC, Wednesday, 2012-Oct-10, week 41.  Altered PC Molde to use the POOL command, and remove tinker huffpuff.  This should provide it with more potential servers and, I understand, drop those which stop responding.  No specific need for this, just an experiment, but as it works correctly, intend moving all PCs this way in the future.
• 15:37 UTC, Wednesday, 2012-Oct-03, week 40.  Altered PC Ystad to use the POOL command as an experiment.  Also removed the tinker huffpuff entry.
• 00:00 UTC, Sunday, 2012-Jul-01, week 26.  Leap second event.  FreeBSD and Windows LAN/WAN-synced OK.  Windows stratum-1 servers had some upset likely because the GPS devices took 17-24 minutes to output the correct timestamp.
• 15:10 UTC, Friday, 2012-Jun-23, week 25.  Updated all PCs to ntpd 4.2.7p285.  No significant changes noted.
• 05:55 UTC, Thursday, 2012-Mar-22, week 12.  Updated all PCs to ntpd 4.2.7p265.
• 16:20 UTC, Tuesday, 2012-Mar-13, week 11.  Updated all PCs to ntpd 4.2.7p263.  No problems.
• 07:30-09:30 UTC, Friday, 2012-Feb-24, week 08.  Updated all PCs from ntpd 4.2.7p258 to ntpd 4.2.7p259.  No adverse changes noted.  Created semi-automated update procedure.
• 19:30 UTC Tuesday, 2012-Feb-21, week 08.  Updated PCs to ntpd 4.2.7p258.  Continued until Wednesday at 07:15 UTC.  No adverse changes noted, but this did correct the large number of badformat reports on the Sure/PSS stratum-1 servers, using the command: ntpq "-c cv &2" <node>
• 07:00 UTC, Tuesday, 2012-Feb-14, week 07.  Updated stratum-1 PCs Feenix and Stamsund to have rationalised ntp.conf parameters: local server poll at 64s, prefer pointing to PC Pixie, iburst on all remote servers.
• 05:50 UTC, Friday 2012-Feb-10, week 06.  Discovered that I could use portmaster net/ntp-devel  to update NTP to the most recent version on PC Pixie, ntpd 4.2.7p255.
• 13:00 UTC, Thursday, 2012-Feb-09, week 06.  Updated PC Pixie from FreeBSD 8.0 to 8.2.  This (a) corrupted the ssh configuration file, so had to find a keyword and display to correct that problem, and (b) lost the kernel-PPS support so had to recompile the Kernel and reinstall it to correct that issue.  19:00 UTC, installed csup and portmaster so that I could update NTP to a later version.  Using "portmaster ntp" got me up to ntp4.2.6p5, which correctly reports a "o" tally code in the ntpq -p display.
• Satrurday, 2011-Dec-19, week 50.  Was asked to test ntpd 4.2.7p241 as changes may have made the Windows port much better, possibly restoring the excellent performance seen with 4.2.4.  Installed as a test, and then on all PCs as it is so much better than 4.2.7p238.  So good that it has its own Web page!
• Saturday, 2011-Dec-12, week 49.  Updated PCs to 4.2.7p238. 05:52 PC Ystad, 07:11-07:52 PCs Alta, Bacchus, Feenix, Gemini, Hydra, Narvik, Puffin & Stamsund.
• 07:30 UTC, Thursday, 2011 Mar 17, week 11.  On PC Alta, set the system to allow test-mode signed drivers, and added Dave Hart's 64-bit serialpps.sys driver (and DLL, and environment variable pointing to the DLL).  Results were a great reduction on offset, frequency and jitter graphs, but jitter was still at the 0.977 ms level.  At 15:25, I had realised that the environment variable NTPD_USE_INTERP_DANGEROUS=1 was set on Stamsund, but not on Alta, further NTPD_USE_SYSTEM_CLOCK=Yes was set on Alta.  On removing system_clock and setting interp_dangerous, performance was much improved.
• 09:30 UTC, Tuesday, 2011 Mar 15, week 11.  On PC Alta, use a second Sure GPS/PPS board for various tests.  Using the user-mode PPS got the jitter down from ~1.7ms to 1.55ms, which was disappointing compared to tens of microseconds on PC Stamsund.
• 11:00 UTC, Sunday, 2011 Feb 27, week 8.  Updated PC Feenix to use the new Sure GPS module instead of the parallel RS-232 feed from the GPS-18 on the roof. Unfortunately, the baud rate was different so I did need to restart NTP at ~11:10, which caused a timing glitch. Before the change, the offset was 0.003 and the jitter 0.002, but the graphs tomorrow will show a better picture. 11:44 - restored the now spare serial feed from GPS 18 (roof) to PC Bacchus. Restart because of the baud rate change (9600 back to 4800).  It seems that the NMEA part is seeing too long sentences (there's no direct control of this on the Sure module), so set the module to 115,200 baud, and restarted NTP with mode 82 (80 for the 115,200 baud, and 2 for $GPGGA as the sentence to look for, which is the first sentence sent by the Sure module.  Later discovered that with the mode 2 bit set, there was no need for the 115,200 baud, which was fortunate as the Sure board seems to lose some settings after a few hours powered down!
• Saturday, 2011 Feb 28, week 8.  Sure GPS module will have been disconnected for a while as I put it into a rather smart-looking Eddystone diecast box - that aluminium is a rather pleasant material with which to work.
• Saturday, 2011 Feb 21, week 7.  Connected new Sure GPS module as a timekeeper to PC Bacchus.
• 14:49 UTC, Monday, 2011 Jan 10, week 2.  Serial connector fell out from PC Stamsund, as it hadn't been screwed in.  Reconnected the next morning at 05:40.
• 09:32 UTC, Sunday, 2011 Jan 09, week 1.  Having tested the GPS 18x LVC puck on PC Stamsund with a noselect on the NMEA driver, it appears that the offset as seen by NTP is almost exactly -1.000 seconds.  Therefore I have added a time2 1.000 fudge to the NMEA driver and re-enabled it, to see whether the erratic timekeeping we have seen before is now resolved.  See here.
• 07:25 UTC, Monday, 2010 Dec 27, week 52.  On PC Alta, reset poll of PCs Feenix and Stamsund to 32s, but decreased the poll to the primary server Pixie to 8s (minpoll=maxpoll=3).  This has resulted in smaller offsets (and jitter) being reported.
• 14:07 UTC, Sunday, 2010 Dec 26, week 51.  As an experiment on Alta, decreased the poll interval for Pixie from 5 to 4 (32 sec => 16 sec), to decrease the jitter at the expense of slightly increased network traffic and loading on Pixie.  17:48, this looks good, so will reducing the poll interval for Stamsund and Feenix make things even better?  No, it did not.
• 07:12 UTC, Sunday, 2010 Dec 19, week 50.  Updated PC Narvik from ntp-4.2.7p58 to ntp-4.2.7p96, and not expecting to see any change.  07:30 updated Gemini from ntpd 4.2.6p2-RC5-o to ntp-4.2.7p96, also expecting no change.  08:25 updated PC Stamsund from ntpd 4.2.5p181-o to ntpd 4.2.7p96-o.
• Thursday, 2010 Dec 16, week 50.  Problems with new PC Alta (Intel i5-760 quad core, Windows-7/64-bit).  Noticed that the timekeeping was not as good as on PC Molde (same OS), and found that NTPD was choosing, incorrectly, to use the interpolation code (which does not work well under Windows Vista/Win-7).  Discussed with Dave Hart, and he suggested filing a bug report so that an environment variable could be used to force interpolation to "off".  This should be in ntp-4.2.7p97, so awaiting a version to test.  Sunday 10:49, updated to ntpd 4.2.7p97-o and it works correctly with the environment variable NTPD_USE_SYSTEM_CLOCK set (it could be any value, I set it to "Yes").
• 17:40 UTC, Monday, 2010 Dec 13, week 50.  Once again Stamsund's timekeeping had become erratic.  Is the really a problematic GPS18x LVC?  This may have been when I used the USB port next to the eToken.  Will try a power-down reboot after the monthly security upgrades this morning, Wednesday 15th.  No difference.  Try and NTP restart at 10:36.  No difference - the GPS/PPS was showing a 1-second offset compared to the LAN sources.  Discovered that there was a new 3.50 firmware for the puck, so programmed it in.  At first glance, the 1-second offset is gone.  Continue to monitor.  A little while later, the GPS is showing -1.000s compared to the other sources, so at 19:00 on the 15th I changed the configuration to remove the GPS/NMEA ref-clock, and leave just the Atom ref-clock, following Dave Hart's advice.  He commented on possible delays in the NMEA data versus the PPS signal with that particular puck.  He also suggested making one of the other peers "prefer", so did that with PC Pixie - my most reliable, I hope.  Still no idea why the previously reliable system has become unreliable.  See Possible GPS 18x LVC issue.
• 06:05 UTC, Monday, 2010 Nov 22, week 47.  Noticed that Stamsund's timekeeping had degenerated again since mid last Thursday, so moved the GPS-18X puck for Stamsund from the loudspeaker to on top of Stamsund itself (raised on a box) to see whether the possible drop-outs improve.  This seems to have started half-way through week 44, approximately Thursday, 2010 Nov 04, and become noticeably worse on Thursday November 19.  Last NTP restart was Friday Nov 12.  The move didn't seem to help, so tried another location on the wall near a south-facing window from 06:00 Tuesday November 23.  Still as bad.  Wonder about either reverting to the (273) version which works well on other Vista/Win-7 systems, or just a reboot?  08:45 Tuesday Nov 23, replaced the NTPD (ntpd 4.2.6p2-RC5-o Jun 03 7:08:48.05 (UTC-00:00) 2010 (1)) with the (273) version, having checked that only one sentence ($GPRMC) was being sent a minute.  The offset shown by the GPS_NMEA(1) was sometimes 1000ms and sometime 500ms, so I wonder whether that GPS-18 is either faulty or needs another firmware upgrade?  Unit was already on the current firmware (3.30), but the file on the Garmin site has changed, so flashed the unit once again.  Made the PPS length 200ms instead of 100ms.  Correct locking showing immediately after update, but will it last?  No, it didn't.  So moved the GPS/PPS feed onto Narvik via its USB-COM4 setup, which we documented previously.  This didn't seem happy either, but not full investigated, so reset 100ms pulse and tried 19200 baud (shorter sentences).  Wed 24 at 07:00, rebooted to see whether there would be any change.  Noted that the PPS timestamp was user-mode and not kernel-mode, even though the kernel-mode driver and DLL were in place (but this is correct with the current ntpd).  Changed ntpd.exe back to ntpd 4.2.5p181-o Jun 06 15:21:08.65 (UTC-00:00) 2009 (1) as this has been recorded as working, but under Windows-7 it still seems to be the user-mode timestamps.  Now appears to be working OK with 100ms pulse and 19200 baud COM port.  See Possible GPS 18x LVC issue.
• 05:40 UTC, Monday, 2010 Oct 25, week 43.  Updated ntp.conf on PC Puffin to include the main server, Pixie.  Not sure why it was missing  -perhaps it wasn't available when PC Puffin was installed?  Ensured that Pixie was "prefer" on Gemini, Hydra and Narvik.
• 14:00 UTC, Thursday, 2010 Oct 21, week 42.  Updated Narvik from: ntpd 4.2.6p2-RC5-o Jun 03 7:08:48.05 (UTC-00:00) 2010 (1), to: To: ntpd 4.2.7p58-o Sep 30 5:38:55.35 (UTC-00:00) 2010 (1).  Supposedly, this version has a faster initial convergence time, so a reboot may be called for to test.  Could not see any difference in the initial convergence time, but otherwise this update seemed to do no harm, so I left it.
  14:18 Updated Hydra from: ntpd 4.2.4p6@DLH-QPC-o May 30 3:58:32.88 (UTC) 2009 (273)  to: ntpd 4.2.7p58-o Sep 30 5:38:55.35 (UTC-00:00) 2010 (1)  to see whether it behaves in the "good" 4.2.4 or the "bad" 4.2.5 fashion.  It behaves badly (~5ms jitter versus ~1ms), so replaced the 4.2.4p6(273) version at 07:45 UTC the next day.
  Plots here: 2010-10-20-21-22-NTP-offset.png
• 07:25 UTC, Friday, 2010 Oct 15, week 41.  Noted that after yesterday's power down, PC Pixie didn't auto-start, so re-configured its BIOS to start on power-on, rather than waiting for the button to be pressed.
• 09:30 - 10:30 UTC, Thursday, 2010 Oct 14, week 41.  Due to the electricity supply meter being changed, there was an enforced power-off during this period, and hence start-up glitches from the various PCs.
• 10:30 UTC, Saturday, 2010 Aug 21, week 33.  Switched PC Torvik to a fresh 64-bit installation, and installed Meinberg's  "Lennon" ntp - version: ntpd 4.2.4p8@lennon-o Dec 09 14:38:09 (UTC+01:00) 2009 (1).   19:10 UTC, switched PC Torvik to ntpd 4.2.4p6@DLH-QPC-o May 30 3:58:32.88 (UTC) 2009 (273).
• 06:34 UTC, Thursday, 2010 Aug 19, week 33.  Swapped out the "Copenhagen" NTP on portable PC Torvik and replaced with the "ntpd 4.2.4p6@DLH-QPC-o May 30 3:58:32.88 (UTC) 2009 (273)" version, as the jitter on Puffin appeared less than that on Torvik.  Seems to be very much better, but is that just the incorrect OS detection and hence the wrong interpolation parameters?  Will it survive a reboot?  Rebooted at 15:40 UTC, and it looks as if the hot restart (service stop-start) and the reboot have similar event-log messages, so I am expecting similar performance.  However, it seems to be as bad as before the hot restart.  Puzzling.  04:55 Friday 20th, stopped and restarted NTP to try and reproduce yesterday's restart and smoother performance, but it didn't work like that and the performance was unchanged.
• 05:45 UTC, Monday, 2010 Jun 07, week 23.  I had noticed a slight transient yesterday when PC Feenix was rebooted, so altered ntp.conf on Bacchus so that Pixie was the preferred server from the local stratum-1 servers.  This means that it has two prefers, one for the serial GPS/PPS (although that's now been disconnected), and one to prefer one out of three of its local stratum-1 servers. 
• 16:00 UTC, Sunday, 2010 Jun 06, week 22.  PC Feenix was rebooted for a security update - EUMETCast had a short outage of which I took advantage.
• 09:00-09:20 UTC, Thursday, 2010 Jun 03, week 22.  Updated PCs Narvik, Bacchus, Gemini & Stamsund to "ntpd 4.2.6p2-RC5-o Jun 03 7:08:48.05 (UTC-00:00) 2010 (1)". 
• 08:00-09:00 UTC, Sunday, 2010 May 30, week 21.  Changed Backup PC Stamsund from using a SkyStar 2.3 card to using a DVBWorld USB box.  May adversely affect timekeeping precision.  Screenshot.  Another restart at 14:45 UTC after removing the SkyStar 2.3 DVB PCI card.
• 06:15-06:20 UTC, Sunday, 2010 May 30, week 21.  PC Hydra also showed a positive ~3ms glitch.  It changed sync source away from Pixie, and then bounced between Feenix and Stamsund.
• 06:00 UTC, Sunday, 2010 May 30, week 21.  Unexplained ~500µs negative spike on PC Pixie.  Again at ~15:10 UTC.  This PC has the outside GPS connection.  Noticed that the locking screws on the serial connector we're screwed in, so did that.  At about 14:40 there seems to have been a slight increase in network input traffic, followed by a gradually rising positive offset - this before the 15:10 negative glitch.  Looking again, at 06:05 there may also have been a network traffic input increase.  The increase in network output is due to a portable PC being served.  Screenshots.
• 05:45 UTC, Sunday, 2010 May 30, week 21.  Unexplained ~500µs positive spike on PC Feenix.  Screenshots.  Event log shows just the usual "clock would have gone backwards" messages, including one at 05:49 "clock would have gone backwards 2 times, max 81.5 usec".
• ~18:35 UTC, Wednesday, 2010 May 26, week 21.  At 17:18 UTC NTP reported "Clock would have gone backwards 1 times, max 324usec" on GPS-synced PC Stamsund, and from ~18:35 UTC there appeared to a slow, several hour recovery back to its normal state.  Screenshot.  At 18:38 UTC, there was an event log entry: "the Windows Module Installer service entered the running state", just like an NTP restart.  On the same say, I noticed that the GPS signal on a hand-held GPS 12 XL receiver in the same region (internal, top floor room) seemed to have gone down quite a bit.  I wonder whether something locally, or even next door, is causing interference, or whether there is a blockage on the roof.
• 22:00 UTC, Saturday, 2010 May 22, week 20.  Very hot in the shack (30°C or more) this evening.  Opened windows on return from an event, and the temperature immediately dropped 3-4°C, resulting in a timekeeping glitch from all PCs.  About 10µs negative and 3µs positive overshoot on PC Pixie (FreeBSD), 300+µs negative on PC Feenix (no overshoot), about 0.5ms on the LAN-synched Bacchus, and about 1ms negative on the LAN-synced PC Narvik.  No noticeable glitch on the Windows-7 stratum-1 server Stamsund, nor on the LAN-synched PCs Gemini & Hydra.  Screenshot.  So as we already know, NTP is affected by the rate of change of temperature, and the rapid change here gave a large step.
• 18:38 UTC, Wednesday, 2010 May 05, week 18.  On PC Hydra, noted lots of clock-hopping events in the Event Log, so added prefer to server Pixie. This has stopped the events, but resulted in visibly worse jitter!
• 17:00 UTC, Thursday, 2010 April 08, week 14.  Add new Intel Atom-based Pixie-II stratum-1 server in using a GPS/PPS source with a FreeBSD 8.0 operating system.  I kept the same IP address as the now-retired Pixie 133MHz Pentium server, and externally the name remains as Pixie.  Note that periods of high CPU or high network I/O may be removed from performance plots for this PC as they hide the more important details.  Thus there may be odd periods of apparently poor timekeeping, which may happen more frequently in the early days.  Expect a ~10-microsecond/1-hour bump when the heating starts....
• 08:45 UTC, Tuesday, 2010 Mar 30, week 13.  Updated SkyStar driver on Stamsund from 4.5.0a beta to 4.5.1.  This seems to have resulted in a much greater number of timing glitches from very occasional to several per day.  Will run a latency check.  Yes, V4.5.1 drivers under Windows-7 shows a typical value of 79µs versus 56µs with V4.5.0a drivers under Windows XP (798µs versus 436µs maximum latency).  But no obvious massive peaks or anything.
• 23:00 UTC, Monday, 2010 Mar 29, week 13.  PSU seems to have failed on PC Pixie.  May not replace.
• Wednesday, 2010 Mar 24, week 12.  PC Hydra updated to 64-bit Windows-7, earlier statistics lost.  Performance seems much poorer than with Win-7 32-bit, so I suspect that I haven't got the configuration quite right, sigh!  13:30 UTC reverted down to 4.2.4p8 Lennon from Meinberg.  This seems to be no better, so tried 4.2.4p6 from Dave Hart (files from: ntp-4.2.4p6-DLH-QPC-20090310-bin.zip).  Still nothing like as good as I expected, so at 16:52 tried Dave Hart's ntpd 4.2.4p6@DLH-QPC-o May 30 3:58:32.88 (UTC) 2009 (273) version.
• 11:10 UTC, Monday, 2010 Mar 22, week 12.  Edited Pixie configuration to avoid polling the Internet servers every 64s, and to decrease the GPS/PPS poll from 64s to 16s in the hope of an improved performance.
• 06:15 UTC, Friday, 2010 Mar 12, week 10.  Altered Bacchus to have the same 32s poll of the local stratum-1 servers as the other PCs have.  Accidentally restarted NTP on Narvik as well as Bacchus. 09:55 gave Narvik the option of three stratum-1 servers.  14:15-14:20 updated PCs Gemini and Hydra to have the stratum-1 Pixie as one of three stratum-1 servers.  This does seem to have reduced the jitter on Hydra - contacting three servers rather than just two every 32 seconds.
• 14:00 UTC, Thursday, 2010 Mar 11, week 10.  Powered-up the old FreeBSD stratum-1 server Pixie and took the parallel RS-232 feed from Bacchus to use for Pixie.  Will record offset on the same 1000µsec scale as used for the other stratum-1 PCs.
• 09:27 UTC, Thursday, 2010 Feb25, week 8.  Updated wireless PC Puffin from ntpd 4.2.4p6@DLH-QPC-o May 30 3:58:32.88 (UTC) 2009 (273) to ntpd 4.2.6-o Dec 09 11:48:30.27 (UTC-00:00) 2009 (1) as an experiment.  I hope that this will make little or no difference to the timekeeping, but knowing the issues with Vista I am not sure.  Noted that I tried this on Dec 07 last year, but with no results noted.  The idea later  is to make the polling interval less for the local stratum-1 server (reduce 64s to 32s), but to increase the Internet server poll right up to 1024s - only the later 4.2.6 allows this.  At the moment, the Internet servers are being polled every minute, and I would like to stop this.  14:24 - looks like the jitter is worse, but I have now set the stratum-1 local poll down to 32s (maxpoll 5) and upped the Internet poll to 1024s (minpoll 10).
• 06:45 UTC, Tuesday, 2009 Dec 22, week 52.  Noted that PC Hydra was getting its event log full with events where NTP was bouncing between the two stratum one servers, so I restored the "prefer" keyword for server Feenix.  This PC is running the older 4.2.4p6 (273).  The PCs running 4.2.6 do not show this clock-hopping.
• 08:35 UTC, Saturday, 2009 Dec 12, week 50.  After reboot of Feenix, noted that some of its client PCs followed its transient rather than changing to the still-accurate Stamsund.  I therefore removed the "prefer" from the Feenix entry on the three main client PCs (Gemini, Hydra and Narvik) and restarted NTP on those systems.  Now need to keep an eye open for clock-hopping!  Noted that on Hydra, at least, the averaged jitter seemed to be quite a lot lower, dropping from a variable 1.12-1.18ms to a steady 0.98us.  The instantaneous jitter was much steadier.  Was this simply the NTP restart, or the removal of the "prefer" keyword?  Narvik showed no average jitter change - 30-65µs.  Gemini showed no average jitter change - 5-13ms.
• 14:10 UTC, Wednesday, 2009 Dec 9, week 50.  Updated PCs to ntp 4.2.6 from Dave Hart - 14:10 Narvik, 14:15 Stamsund, 14:20 Gemini, 14:25 Bacchus, 14:30 Feenix.  This leaves one Windows Vista PC (wireless LAN sync) and one Winodws-7 PC (direct connection LAN sync) still running 4.2.4 because of unresolved performance problems.
• 10:15 UTC, Sunday, 2009 Dec 06, week 49.  Updated Windows XP GPS/PSS-synced PC Feenix from ntpd 4.2.5p231-RC to ntpd 4.2.5p250-RC.
• 07:30 UTC, Sunday, 2009 Dec 06, week 49.  Started monitoring NTP on PC Puffin with ntpd 4.2.4p6@DLH-QPC (230).  At 15:48 UTC changed to ntpd 4.2.4p6@DLH-QPC (273), with the thought of trying 4.2.5 later.  20:00 UTC installed ntpd 4.2.5p250-RC.  06:00 UTC Monday, re-installed ntpd 4.2.4p6@DLH-QPC (273), but noted that the 4.2.5p250 ntpq -p running on Puffin returned "no such file or directory" when run against "puffin", but worked against other PCs, and a 4.2.5p250 running on another PC correctly showed the data from PC Puffin.  Wonder if it's because IPv6 may be enabled by default on Puffin?  Yes, running the local ntpq against the IPv4 address works correctly.
• 11:28 UTC, Saturday, 2009 Dec 05, week 49.  Updated Windows Vista, LAN-synced PC Gemini from ntpd 4.2.5p246-RC to ntpd 4.2.5p250-RC.  Appears OK.  11:38 - updated Windows 2000 Server, GPS/PSS ref-clock synced PC Bacchus from ntpd 4.2.5p231-RC to ntpd 4.2.5p250-RC.  Also appears OK.
• ~08:59 UTC, Friday, 2009 Dec 04, week 49.  PCs Feenix and Stamsund lost PPS signal from the roof-mounted GPS puck.  Powered the puck down and back up, and the PPS signal restarted OK.  Those PCs NTP picked up the returned PPS signal right away.
• 07:05 UTC, Tuesday, 2009 Dec 01, week 49.  Updated NTP on Narvik from ntpd 4.2.5p231-RC to ntpd 4.2.5p250-RC.  At 07:20, updated Stamsund from ntpd 4.2.5p246 to ntpd 4.2.5p250-RC, so that version 250 gets a good checkout.  This introduced a 11ms step, from which it will take a little while to recover.
• 06:26 UTC, Saturday, 2009 November 28, week 48.  Reduced LAN poll rate on Narvik from 64s to 32s.  Has a noticeable affect on reducing the offset.
• 06:30 UTC, Thursday, 2009 November 26, week 48.  Reduced LAN poll rate on Hydra from 64s to 32s for an experiment.  If there is no ill effect from these faster polling rates for Gemini and Hydra I'm likely to leave them.  This has reduced the offsets seen on Hydra.
• 07:45 UTC, Wednesday, 2009 November 25, week 48.  As an experiment on Gemini, left 4.2.5p246 running, but increased the polling rate from maxpoll=6 to maxpoll=5 (i.e. from 64 seconds down to 32 seconds maximum interval).  The idea is to see what reduction in visible offset jitter might be achieved.  I also want to see whether there is an visible effect on the network traffic to and from Gemini.  No noticeable effect on network traffic.  No noticeable effect on the network traffic.  Results don't seem to be a lot different, but perhaps I should to back to 4.2.4p6 (273)?  Tried this, but 4.2.4 won't accept maxpoll=5.  Another odd thought - should I kill the unused IPv6 connectivity I have?  As NTP doesn't seem to report using servers on IPv6, I suppose the connectivity doesn't matter?
• 10:15 UTC, Tuesday, 2009 November 24, week 48.  Updated Vista PC Gemini to 4.2.5p246 following a request for testing from Dave Hart.  Updated Windows-7 PC Stamsund at 10:25.  Both appear to work correctly, although Stamsund had a 50ms step which will spoil the graphs!
• 07:09 UTC, Monday, 2009 October 26, week 44.  Step of NTP frequency on Stamsund from 3.4 to 6.4ppm from 07:09 to 07:15 UTC, followed by a recovery over the next five hours or so.  Nothing in the event logs suggesting why - it could have been a bad set of lost interrupts. Instead of the typical 0.45ms steps in jitter there was a 5.9ms step, offset showed a +16.6ms step.
• 12:40 UTC, Monday, 2009 Oct 19, week 43.  Broken GPS 18x LVC was replaced by Garmin (no comment as to fault), so that unit was used on PC Stamsund (no restart required) and the parallel feed from the external GPS was fed to PC Bacchus (restart of NTP required after editing the ntp.conf, Windows 2000).  It seems that there may be a known issue with these units, when the firmware is earlier than 3.10.
• 06:37 UTC, Saturday, 2009 Oct 17, week 42.  Noted that since the Vista PC Gemini's reboot for security updates on Wednesday 14, NTP seemed much less stable.  The jitter figures confirmed this.  Tried restarting NTP in case it had not picked up something at the reboot (although it did say "using native clock directly").  NTP seemed to have been rather stable between about October 3/4 and Oct 14.  Noted that after the reboot, the average CPU for the tc-recv.exe process had increased from 1.5% to 2.0%, and that the working set for FlightDisplay had increased by a couple of MB.
• 14:00 UTC, Saturday, 2009 Oct 10, week 41.  Updated Narvik (Win XP Pro) from ntpd 4.2.5p212 to ntp-4.2.5p231-RC.  14:09 updated Stamsund (Win-7 Ultimate).  14:17 updated Feenix (Win-XP Home) from ntpd 4.2.5p186 to ntp-4.2.5p231-RC.  15:23 updated Bacchus (Windows 2000 Server) from ntpd 4.2.5p181 to ntp-4.2.5p231-RC.  This version appears to work OK at first glance.
• Early on Wednesday, Oct 07, week 41, the new GPS 18x LVC appeared to fail, so the parallel feed from the main roof-mounted GPS 18 LVC was removed from Bacchus and connected to Stamsund.  Note that NTP on Bacchus failed when the serial source was disconnected, requiring an edit of the ntp.conf to comment out the GPS serial port and ATOM driver (PPS) lines.
• Sometime on either Tuesday, 2009 Sep 22 or Wednesday, 2009 Sep 23, week 39, NTP became very unstable on the Vista PC Gemini.  As I was away at a conference at the time, it wasn't anything I did!  There was a Windows Defender automated update and run at 04:00 UTC on the Wednesday morning, so perhaps that was the cause.  Offset on the weekly graph changed from small peaks of about 10ms to variations peaking at 100ms.  Averaged jitter increased from around 3ms to around 6.5ms, but the offset    The frequency offset increased from a stable 5ppm to a rather instable 55ppm, and there were a number of "no servers available" message in the event log, suggesting that NTP couldn't contact the servers, or there was some sort of network error.  No other applications seemed to complain about a network problem.  Restarted NTP at 07:25 UTC on Tuesday, 2009 Sep 29.  Decided to reboot, as both the TelliCast had restarted at 08:37 UTC, and I got a disk full error message from Plane Plotter when drive F: had plenty of space.
• 07:20 UTC, Thursday, 2009 Sep 17, week 38.  Updated Stamsund NTP from ntpd 4.2.5p186 to ntpd-4.2.5p212, to check a current version with the serial GPS/ATOM drivers.
• 05:10 UTC, Wednesday, 2009 Sep 16, week 38.  Updated Narvik NTP from ntpd 4.2.5p186 to ntpd-4.2.5p212, 05:15 updated Gemini NTP from ntpd 4.2.4p6 to ntpd-4.2.5p212, 05:20 updated Hydra NTP from ntpd 4.2.4p6 to ntpd-4.2.5p212.  The behaviour of 4.2.5p212 seems little or no different to other 4.2.5 versions on Vista and Windows-7.  It seems no worse on Windows XP, so I'll likely leave it running there, but I've reverted top 4.2.4p6 in the Vista and Windows-7 system from 05:05 UTC on Sep 17.
• 00:50 UTC, Monday, 2009 Sep 07, week 37.  Changed the ntp.conf on Gemini (Vista) to have "tick 1", and changed from ntpd 4.2.4p6(273) to ntpd 4.2.5p186, following a request from Dave Hart.  PC Vista is badly behaved for timekeeping in any case.  Offset looks noisier and jitter is higher, but it's workable.  00:55 changed Windows-7 PC Hydra as well, but discovered that it had found the wrong value for "Windows clock precision", so restarted again just after 05:00 UTC.  18:30 UTC tried changing tick to 0.3 on Gemini - not much difference.  05:00 UTC Sep 08, tried tick as 0.1 on Gemini.  On the Windows-7 PC Hydra, this just doesn't seem to have worked at all, but in the spirit of experimentation I tried changing tick to 10 at 05:23 on Sep 08, but it made no obvious difference.  This seems to have made little or no change to the jitter, so Hydra was returned to the older 4.2.4p6(273) NTP at 07:54 UTC, and as the offset variation and jitter on Gemini was also made worse by the change, Gemini was reverted at 08:00 UTC.
• 09:38 UTC, 2009 Aug 24, week 35.  Updated Stamsund to use the slightly later 4.2.5p186 replacing 4.2.5p181, which had been the version is use when this PC had been running Windows XP SP3.
• 2009 Aug 23, week 34.  Rebooted Stamsund around 08:00 UTC, and got good timekeeping restored.  Talked with Dave Hart who suggested setting the environment variable NTPD_USE_INTERP_DANGEROUS=1 to force NTP to use the interpolation code in any case.  This also worked nicely.  Why Windows-7 behaves like this on this particular PC is unknown as yet.
• 2009 Aug 22, week 34.  Installed different network card on Stamsund at about 11:10 UTC, and NTP went well until I restarted it at about 13:30 rather than rebooting.  It dropped the interpolation code and the timekeeping went wild.
• 2009 Aug 18, week 34.  Installed Windows-7 RTM on Stamsund.  NTP seemed surprisingly good running as a stratum-1 server, with an increased jitter compared to Windows XP, but a much reduced offset.  Used version [ntpd 4.2.5p181-o Jun 06 15:21:08.65 (UTC-00:00) 2009 (1)] by accident.
• 0730-0800 UTC, Thursday, 2009 Jul 08, week 29.  Updated Narvik, Stamsund and Feenix to "ntpd 4.2.5p186-o Jul 09 3:58:51.88 (UTC-00:00) 2009 (2)ntpd 4.2.5p186-o Jul 09 3:58:51.88 (UTC-00:00) 2009 (2)" - the latest from Dave Hart.  Tried to update Bacchus but it gave an unknown software exception 0xC0000417 at location 0x00444D99, so I reported it to Dave.  Left the Vista and WIndows-7 PCs alone.
• 0525 UTC, Monday, 2009 Jul 06, week 28.  Updated NTPD on Gemini to 4.2.4 (273) as it is the most recent 4.2.4 that I have [ntpd 4.2.4p6@DLH-QPC-o May 30 3:58:32.88 (UTC) 2009 (273)].  It will then be the same as on the Windows-7 PC.
• 1415 UTC, Thursday, 2009 Jul 02, week 27.  Some disturbance which caused NTP on Hydra to loose some network connectivity.  Restarted NTP, and as it was still version 230, it discovered the wrong system clock precision and performed poorly.  Neither a soft reboot or a power-down reboot helped, until I recalled that I still had version (230).  Updated to version (273) and normal operation was restored.
• 0940 UTC, Sunday, 2009 Jun 14, week 25. Updated Hydra to ntpd-QPC-20090614-0900.zip (ntpd 4.2.4p6@DLH-QPC-o May 30 3:58:32.88 (UTC) 2009 (273)) to see whether the initial detection worked any better.  It may do.  Replaced version (230) to see what happens at the next reboot.  Yes, it failed again.
• 1230 UTC, Saturday, 2009 Jun 13, week 25.  After a reboot of Hydra, noted that NTP was performing much worse than expected.  Hadn't settled after several hours, so stopped and restarted the NTP service, and all was well.  Seems that NTP may not have recognised the need to switch off its interpolation code when running under Windows-7.
• 1053 UTC, Tuesday, 2009 Jun 09, week 25.  As version 4.2.4p6(230) works well on the Windows-7 PC Hydra, tried in on the Vista PC Gemini just to see how well it faired.  Nicely, thanks very much.
• 0841 UTC, Monday, 2009 Jun 08, week 24.  PC Hydra still giving poor performance with ntp 4.2.5.  Noticed that it had the -U 3 command-line parameter which would cause it to scan the network interfaces every three seconds, which seems excessive on that PC compared to the default of every 5 minutes, so removed that parameter and value to see whether it made any changes.  1848 UTC - reverted to 4.2.4p6 (230) on Hydra - couldn't stand the poor performance any longer!
• 1530 UTC, Sunday, 2009 Jun 07, week 23.  Updated PCs in sequence to "ntpd 4.2.5p181-o Jun 06 15:21:08.65 (UTC-00:00) 2009 (1)".  Narvik (1528), Stamsund (1533), Gemini (1554), Hydra (1605), Bacchus (1615), Feenix (1621).  Hydra (Windows-7 RC) is not as good as it was with "ntpd 4.2.4p6@DLH-QPC-o (230)" - see note from Sunday 2009 May 17.  However, I did set the NTP account to have the Increase Scheduling Priority right, in case that helps, and restarted at 1730 UTC.
• 0605 UTC, Sunday, 2009 Jun 07, week 23.  Updated ntp on Stamsund to the (13) version to see if the jitter was reduced by using the kernel-mode timestamping in the new DLL-based provider.  Needed to update to the serialpps.sys driver as well, of course, which needed a reboot.
• 0953 UTC, Saturday, 2009 Jun 06, week 23.  Updated "ntpd 4.2.5p175@1.1858-o May 12 15:20:06.38 (UTC-00:00) 2009 (1)" on Narvik to "ntpd 4.2.5p180@1.1877-o Jun 05 10:01:52.09 (UTC-00:00) 2009 (13)" to test the higher numbered serial ports.  Ports 4, 15, 101, and 255 work OK.  Port 256 does not, but that is outside the range allowed.
• 0510 UTC, Saturday, 2009 Jun 06, week 23.  On PC Feenix, updated "ntpd 4.2.5p175@1.1858-o May 12 15:20:06.38 (UTC-00:00) 2009 (1)" to "ntpd 4.2.5p180@1.1877-o Jun 05 10:01:52.09 (UTC-00:00) 2009 (13)" to confirm correct operation.  Appears OK, so at 0935 UTC enabled the new DLL-based atom/pps provider.  On the average jitter plots, you could clearly see where the kernel-mode PPS support was enabled, and the average jitter was reduced from about 3.5 µs to under 2.5 µs.
• 1300 UTC, Friday, 2009 Jun 05, week 23.  Restored duplicate serial-GPS feed to PC Bacchus and confirmed operation.  Updated to "ntpd 4.2.5p180@1.1877-o Jun 05 10:01:52.09 (UTC-00:00) 2009 (13)" and confirmed correct operation.  Later, tried adding the serialpps support DLL, but needed to alter the server order to put the Atom driver before the NMEA driver to get it to show in the ntpq -p display.  Kernel-mode does not appear to be working yet, at least on this PC.  Later that evening, updated the DLL to one which produced fewer error messages.  The interpolation is no so good on this 100Hz system, and you could not clearly see where the kernel-mode PPS support was enabled..
• 0537 UTC, Saturday, 2009 May 30, week 22.  Updated NTP on Bacchus from: ntpd 4.2.5p161@1.1825-o Apr 02 11:57:05.06 (UTC-00:00) 2009 (9) to: ntpd 4.2.5p180@1.1877-o May 29 16:02:39.34 (UTC-00:00) 2009 (8).
• 0500 UTC, Saturday, 2009 May 23, week 21.  Noted that PC Narvik was synced to Stamsund, rather than the lower-offset Feenix, so made Feenix "prefer" and restarted NTP.  Interesting to see whether there is any difference - at least clock hopping should stop.
• 1655 UTC, Friday, 2009 May 22, week 21.  Removed parallel RS232 feed from from roof-mounted GPS 18LVC to PC Stamsund, and replaced it with an indoor GPS 18x LVC (should be more sensitive) and USB-powered +5V.  2110 UTC - made Feenix and Stamsund a peered pair.  Initially they rejected each other ("x" in ntpq -p), but later changed to candidate ("+").
• 1620 UTC, Thursday, 2009 May 21, week 21.  Removed the GPS-serial-USB source from PC Narvik, as testing was complete.
• 1520 UTC, Sunday, 2009 May 17, week 20.  Switched PC Bacchus back to 4.2.5p161, once I had worked with Dave Hart to resolve the problem in 4.2.5p175 (which would also have been a problem in the stable branch).  Without the improved interpolation, the performance on Bacchus was noticeable poorer (10+ms vs. 3ms offset, 0.6ms average jitter vs. 0.1ms).  1526 UTC switched PC Hydra back to (230) with the better support for Vista/Windows-7, by /not/ using the 1ms interpolation when the system clock is already running at 1ms intervals.  (20ms "noisy" offset vs. 2ms "cleaner", 8-13ms average jitter vs. 1.1-19ms).
• 0805 UTC, Friday, 2009 May 15, week 20.  Updated PC Stamsund to 4.2.5p175@1.858-o.  0813 updated PC Gemini.  Didn't work on PC Bacchus (seemed to corrupt the Event Log, so re-installed ntpd 4.2.5p161@1.1825-o.  08:43 updated PC Hydra, running Windows-7 RC (7100).  0905 updated PC Feenix, but here we seem to have lost access to the ATOM driver (was version ntpd 4.2.4p6@DLH-QPC-o Mar 15 21:30:20.47 (UTC) 2009 (239) before).
• 1545 UTC, Saturday, 2009 May 09, week 19.  Added serial-over-USB ref-clock to PC Narvik.  May 12 updated to 4.2.5p175@1.858-o to check that the new COM4 support worked - it does.
• 0747 UTC, Saturday, 2009 May 09, week 19.  Noted that NTP wasn't running at real-time priority on Stamsund, so used secpol.msc to allow this, and restarted NTP.  I don't expect it will make a lot of difference - it will be interesting to see whether it makes any difference either to the NTP jitter, or to the TelliCast performance.  No significant change to NTP performance.
• 1330 UTC, Wednesday, 2009 May 06, week 19.  Changed PC Feenix to use an indoor GPS 18x LVC rather than the outdoor unit.  The 18x is more sensitive, but indoor coverage will be worse.  So this is just a test to see how good or how bad the performance might be.  My estimate is that the basic timekeeping accuracy should be unchanged.  There may be a transient at the changeover time.  Let's see!
• 0615 UTC, Tuesday, 2009 May 05, week 19.  Changed PC Hydra (Windows 7) from Meinberg VegasV2 to Dave Hart's v.230, so that it is the same version as Gemini, and we may be able to compare Vista with Windows 7, albeit on two different PCs.  They have the same 64s poll of local ref-clocks and 1024s poll on Internet backup servers.  I will be looking to see if the spikes seen on PC Gemini are also seen on PC Hydra.
• Sunday, 2009 May 03, week 18.  Moved PC Hydra onto Windows 7, initially using Meinberg VegasV2.
• 0815 UTC, Friday, May 01, week 18.  Had noticed increased EUMETCast packet loss on Gemini since April 21/23 so, while I don't think this is due to NTP changes, reset Gemini to use Dave Hart's v.230 (March 10).  Seemed to be the most recent DLH version (apart from p163).  I don't seem to have a DLH v.239 to hand on that PC.
• 1100 UTC, Wednesday, 2009 Apr 29, week 18.  Edited Perl MRTG monitoring script to limit just below 6000µs so that limiting values on Bacchus, Hydra and Narvik can be more accurately represented.
• 1536 UTC, Wednesday, 2009 Apr 22, week 17.  Tried setting the MMCSS service on the Vista PC Gemini to disabled.  It might affect the errors on the DVB system (by not prioritising the multi-media network traffic), or improve the NTP performance by not delaying its network I/O.  Or it may have no effect!  Seemed to have no effect, so restored the service at about 10:30 UTC on Thursday 23rd.
• 0800 UTC, Tuesday, 2009 Apr 14, week 16.  It seems I have been comparing the wrong versions to check bug 1030, I should compare p162 or later versus e.g. Meinberg VegasV2.  So replaced p163 with the VegasV2 on Gemini to see what happens....
• 0645 UTC, Sunday, 2009 Apr 12, week 15.  Updated PC Gemini from Dave Hart's (239) of March 15 to test build "4.2.5p161@1.1828-o Apr 09 9:56:38.95" to see whether the the jitter stays normally low and asymmetrical (as in recent -stable builds), or whether it reverts to much higher levels but symmetrical as in a recent -dev test version (as noted here).  Suspect this should have been build 163 rather than 161, but it's OK, it will establish how 161 performed.  14:20 UTC, updated to true test build 163.  "ntpd 4.2.5p163@1.1831-o Apr 12 12:22:16.13 (UTC-00:00) 2009 (2)"
• 0630 UTC, Saturday, 2009 Apr 11, week 15.  Started new graphs for Bacchus, Hydra and Narvik with a +/-3ms range rather than a +/- 100ms range.  The old data is still being collected and can be seen here.
• 1300 UTC, Thursday, 2009 Apr 02, week 14.  Updated PCs Bacchus, Narvik and Stamsund to Dave Hart's latest -dev release 4.2.5p161@1.1825 (9) for testing.  (About 1100 had updated PCs Bacchus, Narvik and Stamsund to have maxpoll 6 on the local ref-clock servers, and minpoll 10 on the Internet servers.  Had been running Hydra that way for some time.
• 1445 UTC, Tuesday, 2009 Mar 31, week 14.  Disappointed with the very poor offset and five times worse jitter of the -dev branch of ntpd on Gemini, reverted to the (239) build of Dave Hart's 4.2.4 -stable version.
• 0900-0910 UTC, Saturday, 2009 Mar 28, week 13.  Changed NTPD to Dave Hart's 4.2.5p158@1.1823-o (5) version on PCs Narvik, Stamsund (ref-clock) and Gemini (Vista).  This is the "dev" rather than the "stable" version, and does include the leap-second majority vote code.  0940 UTC, updated Windows 2000 PC Bacchus as well, since Dave confirmed this is the 1-receive-per-socket coding.
  1025 UTC - changed Hydra to have minpoll=10 for the Internet servers, maxpoll=6 for the local GPS/PPS servers.
• 0900 UTC, Wednesday, 2009 Mar 25, week 13.  Restarted NTPD as a regular service on Bacchus with Dave Hart's 1-receive socket (254).  When tested after 24 hours earlier this morning, this version did not crash, so testing for a longer spell as a service is next.
• 0600 UTC, Tuesday, 2009 Mar 24, week 13.  Retired PC Pixie as the prime GPS reference PC, to be replaced by PC Feenix.  Added PC Stamsund as a second NMEA/PPS system, as a backup.
• 1820 UTC, Friday, 2009 Mar 20, week 12.  Changed Bacchus to Dave Hart (239) running in debug mode to see why I couldn't stop NTPD cleanly earlier.  1835 updated Feenix to (239) for its better leap-second handling.  1842 updated Gemini.  1847 Updated Hydra.  1855 updated Stamsund.  1940 Bacchus NTP shut down cleanly with Ctrl-C from the command line, so restarted it for a longer run.
• 0645 UTC, Monday, 2009 Mar 16, week 12.  Changed PC Bacchus back to Vegas.V2 to compare the jitter on this 100Hz system.  Changed Narvik to Dave Hart's (239) to check for the leap-second majority code.  At his suggestion, updated the other programs as well (e.g. ntpq).
• ~1900 UTC, Saturday, 2009 Mar 14, week 11.  Reset all PCs (except Feenix) to use the (230) release of Dave Hart's version.  PC Bacchus was reluctant to let the .exe go after stopping the service, so needed a reset-reboot.
• 0705 UTC, Friday, 2009 Mar 13, week 11.  Reboot Feenix, twice, to install the serialpps.sys device driver, and see how NTP performed with the kernel-mode rather than user-mode time-stamping.  This didn't seem to alter the mode, so I added the ATOM driver as well, which worked.  The lack of a stratum-1 server did upset Bacchus which was syncing with a fixed 64s maxpoll.  0951 restarted NTP on Gemini to pick up second stratum-1 server and Internet servers (emergency use only).  1145 restarted NTP on Bacchus and Stamsund to pick up the new configuration files (back to 1024s polling).
• 0830 UTC, Thursday, 2009 Mar 12, week 11.  Restarted Bacchus without the NTPD_PCC_FREQ = 551240000, to force it to use QueryPerformanceCounter rather than RDTSC.  22:46 restarted Bacchus with maxpoll 6 (64s) rather than maxpoll 4 (16s), as this is closer to the intended final configuration.  Internet servers still at minpoll 10 (1024s).
• 1726-1745 UTC, Tuesday, 2009 Mar 10, week 11.  Tried to restart Feenix with the NTPD_PCC_FREQ set, but this not sync even after 15 minutes, so reverted to new version but without NTPD_PCC_FREQ.  This seems to have slightly less jitter.
• 0707 UTC, Monday, 2009 Mar 09, week 11.  Finally restarted Gemini's NTP with just three local servers (one stratum 1) and a maxpoll of 6 (64 seconds).
• 2115-2135 UTC, Sunday, 2009 Mar 08, week 10.  Restarted NTP on Vista PC Gemini to minpoll 4 maxpoll 4, single server (pixie) for a final comparison.  
• 0615 UTC, Sunday, 2009 Mar 08, week 10.  Restarted NTP on Vista PC Gemini back to maxpoll 10, for a comparison.  0620, restarted NTP on PC Feenix to change the priority from high to real-time.  I would have just used the Task Manager, but I didn't have the priority, so I chose to add NTP to the Administrators local group and restart.  Restarted Feenix NTP at 0834 and 0906 to pick up updates from Dave Hart.
• 0925 UTC, Saturday, 2009 Mar 07, week 10.  Switched the Vista PC Gemini down to a maxpoll=6 (64s) just to show how flat the graph can be - or not.
• 1930 UTC, Friday, 2009 Mar 06, week 10.  Downloaded Dave Hart's v.204 and installed same on all Windows PCs.  No configuration changes.
• 1015 UTC, Friday, 2009 Mar 06, week 10.  Rôle reversal for Bacchus and Feenix.  Transferred serial port from Bacchus to Feenix, to see whether the 64Hz clock and much higher specification processor would perform better.  If this works, will most likely keep Feenix as the reference server for me, so that PC Pixie and perhaps PC Bacchus can be retired.
• 1025 UTC, Thursday, 2009 Mar 05, week 10.  Continued experiments on Bacchus by adding the atom ref-clock back in, as it will eventually be required when serialpps support is removed from Dave's ntpd.exe.  1225 UTC, moved Narvik, Stamsund and Gemini onto V.177, and added Bacchus as a server, making each of these PCs have two PPS devices.  Thinking towards when Pixie may be switched off.  1400 UTC, also installed on Hydra, but it failed because the NTP account wasn't a member of the Administrators group.  Fixed with:
    net localgroup administrators ntp /add
• 1000 UTC Wednesday, 2009 Mar 04, week 10.  Updated Bacchus to Dave Hart's V.177.  May have reduced jitter after 45 minutes operating.  1400-1430 - tests with INT=26 to INT=32.  Left running with INT=27 which seemed best.
• 0600 UTC, Monday, 2009 Mar 02, week 10.  Continuing experiments with Gemini, I restored the GPS-server Bacchus, and changed to maxpoll to 9.  This is one combination we haven't yet tried.  Changed to maxpoll from 4 to 6 on Bacchus (16s -> 64s).  This triggered another 20msec step, positive this time.  0756 - Dave Hart noted that Bacchus is a 100Hz system, where his interpolation code requires a slightly different sample interval to the 43msec default..  So I set environment variable NTPD_INT_INT=26 and restarted Bacchus.  There was a -120ms transient at the restart, which was taking a long time to settle down, so at 16:44 I changed the minpoll to 4, and restarted ntpd on Bacchus.  Tuesday 0312, restarted on Bacchus  with with NTPD_INT_INT=28 - this looks to work a little better, and has reduced the amplitude of jitter in the offset while increasing its frequency.  0610 added ATOM driver: server 127.127.22.1 minpoll 4.
• 1130 UTC, Sunday, 2009 Mar 01, week 10.  Cut the GPS/PPS feed to Pixie for few minutes while I soldered in a parallel RS-232 feed for tests on using a Windows PC with PPS (Bacchus).  This cut resulted in a transient, but that had recovered by about 12:30 UTC.  The graphs for Bacchus (with a serial GPS/PPS ref-clock) starting at about 17:40 on Sunday don't look too good.  An unexplained 20msec negative step at around 19:15.
• 2135 UTC, Friday, 2009 Feb 27, week 09.  As an experiment, changed the Vista PC Gemini to sync purely from local sources (the stratum 1 GPS/PPS and a couple of others as backup), removed the Internet servers, and set maxpoll to 6, thus checking every 64 seconds.  This looks good, so let's change it to maxpoll 8 from the next boot (due at 06:55 UTC, Saturday, 2009 Feb 28, week 09).  This may, or may not, still prevent NTP from going into that zone where "the frequency graph matches the offset graph", which I feel is not a correct way to operate.  Surprisingly, NTP on Gemini seems to have gone crazy.  It's running at real-time priority (presumably as the account is now a function of the admins group), so at 11:30 UTC I tried changing the affinity to a single processor.  No difference, so I just stopped and restarted NTP to see what happens.  Ah, my own fault.  I discovered that the -M had been omitted from the command-line for the NTP service, and that's why the jitter had been awful.  With Dave Hart's version the -M is essential (at least on Vista).  Found that with maxpoll=8 the typical offset was -7.5ms with transient offset peaks up to +0..+2.5ms with a couple up to +17..18ms.  0600 Sunday, just as another experiment, removed the local GPS/PPS server (Pixie) and left just four Internet servers, and will allow the max poll to reach 10 (1024s) - transients after 24 hours were in excess of 50ms, so reverted to using the local GPS/PPS server.  With maxpoll=9, the typical offset was 0, with most of the jitter being positive excursions of 20-30ms.
• 0840 UTC, Thursday, 2009 Feb 26, week 09.  As an experiment, changed Narvik so that just the local GPS/PPS server was polled at 128s intervals (maxpoll 7), in the hope that the Internet servers would eventually be polled at the standard 1024s intervals.  Change undone at 0900 UTC, Friday, 2009 Feb 27, as all servers were polled at 128s intervals, which is not normally acceptable for outside servers.
• 0725 UTC, Thursday, 2009 Feb 26, week 09.  Installed Dave Hart's latest update (157) on the Windows XP SP3 PC Stamsund, and at 07:45 on the Vista SP1 PC Gemini.  On Gemini it stayed stable for several hours, but then reverted to oscillating.  The short-term jitter on Stamsund was reduced, but it's too early to judge whether the longer-term performance has been improved.
• 1125 UTC, Monday, 2009 Feb 16, week 08.  As an experiment, I restarted ntp on PC Gemini, having altered the config file to exclude all the Internet servers and just leave the local LAN GPS-based server Pixie.  See how that does (although I suspect I've tried this before).  Seems to have reduced both the amplitude of the oscillations, and the tendency to limit.  So at 21:20 I added the Internet servers back in, a restarted NTP on PC Gemini.  Having the Internet servers may have made things a little worse, but not back to the extreme swings.  It's almost as if the act of stopping and restarting the NTP service improves its stability.  I checked, and the ntpd.drift file does exist, is being updated, and does contain a sensible-looking value.
• 0930 UTC, Monday, 2009 Feb 16, week 08.  Changed the priority of ntpd.exe on PC Gemini from "high" to "realtime" at Dave Hart's suggestion, but this seemed to make little difference.
• 0655 UTC, Sunday, 2009 Feb 15, week 07.  Rebooted PC Gemini after a security update.  NTP performance seems worse - back to the old ways with extreme oscillations.  Thought that perhaps I had the old version back, but "ntpq -rcv gemini" still shows version (69).
• 0755 UTC, Monday, 2009 Feb 09, week 07.  Updated XP PC Stamsund to Dave Hart's version (69).
• 0615 UTC, Monday, 2009 Feb 09, week 07.  Updated Vista PC Gemini to Dave Hart's version (69).
• 1623 UTC, Sunday, 2009 Feb 08, week 06.  Updated Vista PC Gemini to experimental ntpd from Dave Hart - version "ntpd 4.2.4p6@DLH-QPC-o Feb 08 12:16:40.85 (UTC-00:00) 2009 (68)".
• 2130 UTC, Saturday, 2009 Feb 07, week 06.  Updated Vista PC Gemini to experimental ntpd from Dave Hart.  Left the -M parameter present for the moment.  Although stable to start with, this version eventually had no better stability than the "vegas" release.
• 1000 UTC, Tuesday, 2009 Jan 13, week 02.  Updated all Windows PCs to Meinberg's "vegas-v2" release (ntp 4.2.4p6).
• 1150 UTC, Sunday, 2009 Jan 11, week 02.  Updated Narvik and Gemini to Meinberg's "vegas" release (ntp 4.2.4p6).  Security update only.
• 1000 UTC, Friday, 2008 Nov 14, week 46.  PC Feenix replaced PC Hermes as the main EUMETCast receiver.  For continuity's sake, I carried over the old MRTG data from Hermes.
• 0820 UTC, Sunday, 2008 Sep 28, week 40.  Just for interest, tried setting the priority of NTPD.exe to "real-time" on the Vista PC, Gemini.  Whilst I don't expect it to make any difference, it will be interesting to observe any changes or side-effects (e.g. on the satellite data capture software)!
• 0950 UTC, Thursday, 2008 Sep 25, week 39.  At the suggestion of Mike Hughes, I tried setting CPU affinity for the NTPD process on the Vista PC, Gemini, to just a single CPU.  I happened to choose CPU 1.  It seems to have helped Mike get better timekeeping, so let's see if it helps me.  Mike may force CPU on the ntpd.exe with the ImgeCfg.exe program, if the improvement continues.
  Didn't seem to help, so tried stopping ntp service, marking the image uni-processor only, and CPU-affinity 0x1, and restarted.
  See: http://www16.brinkster.com/salvage/thief/darkengine.htm
• 1230 UTC, Tuesday, 2008 Sep 02, week 36.  Updated Hermes, Narvik and Stamsund to Meinberg build "ntpd 4.2.4p5@beijing-o Sep 01".
• 0940 UTC, Tuesday, 2008 Sep 02, week 36.  Turned of the -M flag for NTP on the Vista system (Gemini).
• 1430 UTC, Monday, 2008 Sep 01, week 36.  Updated NTP on Bacchus and Hydra to Meinberg build "ntpd 4.2.4p5@beijing-o Sep 01".  1525 updated Vista PC Gemini to "beijing" build..
• 2110 UTC, Thursday, 2008 Aug 07, week 32.  EUMETCast uplink glitch caused timing glitch on Stamsund.
• 0255 UTC, Monday, 2008 Aug 04, week 32.  EUMETCast uplink glitch caused timing glitch on Stamsund, and about an hour later on Hermes.
• 1750 UTC, Saturday, 2008 Jul 19, week 29.  The uplink of the EUMETCast service suffered a brief rain outage, and whilst this didn't trigger the automatic restart scripts, it did trigger a timing glitch on two out of three PCs (Hermes and Stamsund).
• 0545 UTC, Thursday, 2008 Jun 19, week 25.  The Vista system downloaded service pack 1 yesterday, and invited me to install it this morning.  So far, NTP seems as good now on Vista as it was on XP, but that's only a three hour spell.  It didn't last, though, and NTP is now as bad as ever on Vista.
• 0756 UTC, Wednesday, 2008 Jun 18, week 25.  On Vista, set system environment variable RANDFILE=C:\Tools\NTP\etc\.rnd, so that NTP might be able to access the file.  Used this directory as it already exists on my system, and it's where I write the NTP statistics.  This is in the hope that NTP time keeping on Vista might improve, but I am not optimistic about this!  No significant change noted.
• 0530 UTC, Tuesday, 2008 May 27, week 22.  Discovered that the W32Time service was still running on Gemini, despite me asking the Meinberg installer to disable it.  Probably needed more privilege. Disabled manually, and lets see how it goes.....
• 1330 UTC, Thursday, 2008 Apr 03, week 14.  Changed the LED in the Pixie GPS interface, and it will take a little more current (around 2mA), but be a lot brighter.  Seems to have caused some oscillation in the timekeeping, but settled down OK after a few hours.
• 1730 UTC, Tuesday, 2008 Feb 26, week 9.  EUMETCast signal loss and NTP restarts will cause transients on PCs Hermes, Hydra and Stamsund.
• Thursday, 2007 Dec 20, week 51.  Only discovered on Sunday Dec 23 that the PSU fan on Bacchus had stopped, so PC replaced with a spare one on Monday, Dec 24.
• 2200 UTC, Tuesday, 2007 Nov 13, week 46.  Restart Gemini with Windows Vista.  Some issues with Vista, currently under investigation.
• 1725-1737 and 2012-2034 UTC, Tuesday, 2007 Nov 13, week 46. power outage.
• 2345 UTC, Monday, 2007 Oct 22, week 43.  Break in EUMETCast from 2345-2359, and 0029-0042 (Tuesday 23).  On Hermes and Hydra, the timekeeping suffered a glitch.  Stamsund OK.
• 0020 UTC, Monday, 2007 Oct 01, week 40.  Seems to have been some sort of transient causing Bacchus to go wild, leaving a large positive value (494.739) in the drift file.  Restarted NTP OK at 0525 with a drift value which was approximately correct (-72.717).  Gemini and Hermes also showed a small glitch around that time, and Hydra nearer 0100.  Unsure what this was.  Spurious leap-second flag on Bacchus?  Yes, it inserted a leap-second at 0100 clock.  None of the currently selected servers show a +1 second offset.
• 0520 UTC, Tuesday, 2007 Aug 21, week 34.  Rebooted Bacchus, which seemed to cause quite a transient!
• 1845 UTC, Sunday, 2007 Aug 19, week 33.  EUMETCast drop-out triggered NTP transients on Hermes and Hydra.  PC Stamsund, with its older V2.3 card, had a TelliCast restart and therefore an NTP restart.
• 0600 UTC, Sunday, 2007 Aug 12, week 32.  Discovered that NTP had fallen over on Bacchus with an access violation.  Message logged at 02:37 on Aug 09 "no servers reachable".  Restarted OK but with a visible transient.
• 1510 UTC, Tuesday Jul 31, week 31.  Updated NTP on the Windows 2000 system to "ntpd 4.2.4p3@1.1502-foehr-o".  A new installer has resolved the LIBEAY32.dll error, and there were other problems due to having an earlier installation living in the \WinNT\ directory.  Resolved by not doing a files-only upgrade, and editing the ntp.conf appropriately.
• 0820 UTC, Friday Jul 27, week 30.  Updated NTP on the XP machines to "ntpd 4.2.4p3@1.1502-foehr-o".  The Windows 2000 system failed with "The ordinal 3837 could not be located in the dynamic link library LIBEAY32.dll".  No upgrade attempted on the one Windows NT4 system.
• 1146-1149 UTC, Friday, Jul 20, week 29,  EUMETCast drop-out triggered NTP transients on Hermes and Hydra.
• 0400 - 0430 UTC, Friday Jun 29, week 26.  Upgraded PCs Hydra, Narvik & Stamsund to Meinberg NTP V1.1499 (V4.2.4p3-RC1).  This does not appear to work correctly on PC Bacchus which runs Windows NT4 SP6, and hence the glitch while reverting to the older Terje Matheson version.
• 0546 UTC, Thursday Jun 28, week 26.  Upgraded PC Gemini to Meinberg NTP V1.1499 (V4.2.4p3-RC1) as a test. Seems to be OK.
• 0317-0325 UTC and 1141 - 1146 UTC, Thursday, Jun 21, week 25.  EUMETCast outages caused timekeeping glitches on Hermes and Hydra.
• 1700-1730 UTC, Sunday Jun 10, week 23, EUMETCast outages caused timekeeping glitches on Hermes and Hydra.
• 1330 UTC, Saturday Jun 09, week 23, EUMETCast outage caused timekeeping glitches on Hermes and Hydra.
• 2100 UTC, Friday Jun 01, week 22.  Experiments on Gemini setting the time zone between UTC (Casablanca) and GMT/BST (Edinburgh) seem to have upset NTP, so restarted it from scratch at about 06:10UTC on Saturday, June 02.  Still not settled by 07:00 UTC, and the drift value showed 481, so created a small reset file to stop, set the correct drift value, and restart NTP.  Ran at 07:03 UTC.
• 0015 and 1926 UTC, Saturday May 26, week 21.  EUMETCast drop-out triggered auto-restart and hence timekeeping glitches.
• 2145 UTC, Monday, 2007 May 07, week 19.  EUMETCast drop-out triggered auto-restart and hence a timekeeping glitch on Hermes, Hydra but not Stamsund.
• 0530 UTC, Saturday, 2007 Mar 31, week 13.  Discovered that NTP had fallen over on Bacchus with an access violation.  Restarted OK but caused a transient lasting to ??.
• 1930 UTC, Sunday, 2007 Mar 18, week 11.  Discovered that NTP had fallen over on Bacchus with an access violation.  Restarted OK but caused a transient lasting to 0300 Monday.
• 0807 UTC, Monday, 2007 Mar 05, week 10.  Discovered that NTP had fallen over on Bacchus with an access violation.  Only logged event was Mar 02 at 1825 "no servers reachable".  Restarted OK but caused a transient.  Seems same as Jan 05 event?
• 0820 UTC, Thursday, 2007 Feb 01, week 5.  There was another glitch which caused a timekeeping step on the Test PC, and required an NTP restart on the main PC.
• 07:50 UTC, Thursday, 2007 Feb 01, week 5.  Test PC (Hydra) lost EUMETCast connection for more than 90 seconds, the automatic restart causing a slight glitch.
• 0424 UTC, Friday, 2007 Jan 19, week 3.  EUMETCast drop-out triggered restart event on Hermes.  Hydra timekeeping affected.
• 1750 - 1820 UTC, Thursday, 2007 Jan 18, week 3.  EUMETCast drop-out on all three PCs affected timekeeping.
• 0945 UTC, Thursday, 2007 Jan 11, week 2.  Rebooted Bacchus after monitor change caused the system to hang (may have disturbed mouse connection at the same time).  Bacchus seems to have ignored the ntp.drift file, or somehow entered a wrong value (-20) there.  Reset to near-correct value of -72 and restarted at 1415 UTC.
• 0923 UTC, Friday, 2007 Jan 05, week 1.  Discovered that NTP had fallen over on Bacchus with an access violation.  Only logged event was Jan 02 at 1733 "no servers reachable".  Restarted OK but caused a transient.
• ~2200 UTC, Friday, 2006 Dec 22, week 51.  Added PC Narvik as replacement for Gemini.
• 0700 UTC, Sunday, 2006 Dec 17, week 50.  PC Gemini rebooted after monthly security update and Zone Alarm update.
• 0200 UTC, Friday, 2006 Dec 15, week 50.  EUMETCast seemed to drop out on Hermes and Hydra, but OK on Stamsund (older V2.3 card).  NTP consequently affected.  Two more drops at 2047 and 2153, handled by automatic restart on Hydra.
• 0200 UTC, Wednesday, 2006 Dec 13, week 50.  EUMETCast seemed to drop out on Hermes and Hydra, but OK on Stamsund (older V2.3 card).  Hydra also showed a drop from 1000 to 1100 (from about 9dB to 7.5dB).  NTP also affected.
• ~2230 UTC, 2006 Nov 29, Wednesday, week 48.  Glitch on EUMETCast caused PC Hermes and Hydra to stop collecting, so I restarted NTP as a precaution.
• 0155 UTC,2006 Nov 27, Monday, week 48.  Glitch on EUMETCast stopped Hydra and caused timekeeping transient.  NTP restarted at 1850.
• 0130 UTC, 2006 Nov19, Sunday, week 46.  Timekeeping glitch on Hydra suggests EUMETCast transient.
• 1440 UTC, 2006 Nov 17, Friday, week 46.  Stamsund rebooted after security updates.
• 1625 UTC, 2006 Nov 15, Wednesday, week 46.  Noted that Bacchus timekeeping had been a straight line so looked rather good.  Further checks showed that NTP had crashed with a memory access violation, with "no servers reachable" in the event log at 20:04:55 on Sunday Nov 12.  Restarted.
• 0140 UTC, 2006 Nov 02, Thursday, week 44.  Seemed to be a glitch on both Hydra and Hermes, so I suspect a EUMETCast glitch.  Another smaller glitch at 0605 UTC.
• 1500, 2006 Oct 18, Wednesday, week 42.  Rebooted Gemini after hardware hang.
• 1500, 2006 Oct 12, Thursday, week 41.  Large tree at the house front cut down, which should have the incidental effect of improving the GPS coverage slightly.
• 0910, 2006 Oct 10, Tuesday, week 41.  Stamsund spontaneously reboot during a period of high network activity, and I took the opportunity to update the Zone Alarm.  Hence the glitch.
• 0843, 2006 Oct 04, Wednesday, week 40.  Rebooted Bacchus, because the CMOS clock will likely have been well of when the system restarted, so making the 16-bit sub-system time incorrect..
• 0500 - 0630, 2006 Oct 04, Wednesday, week 40.  Power failure will have upset timekeeping.
• 1030, 2006 Sep 22, Friday, week 38.  Repositioned GPS antenna for Pixie onto the roof.  This should give it a good vertical view and remove the 180 degrees restriction due to the house, but may restrict the view near to the horizon.
• 1140, 2006 Sep 21, Thursday, week 38.  Due to building maintenance, temporary loss of EUMETCast signal caused Hermes NTP to go wild.  I wonder if the TCP/IP stack is getting overloaded by repeated "where-is-the-signal" activity and delaying NTP requests?  Restarted NTP on Hermes around 1200.
• 0715, 2006 Aug 26, Saturday, week 34.  Rebooted Hydra to correct EUMETCast signal strength reporting.  0730 rebooted Stamsund for the same reason (this made no difference).
• 0900, 2006 Aug 25, Friday, week 34. Loss of EUMETCast signal caused Hermes and Stamsund to loose proper timekeeping (they were increasingly slow while the red satellite icon was displayed), so restarted NTP.  Both Hydra and Stamsund were rebooted after security updates were applied.  NTP again failed to pick up the UK pool servers on the rebooted PCs, so added one other external stratum 2 server into the configuration.
• 0245, 2006 Aug 18, Friday, week 33.  Loss of EUMETCast signal (or high error rate on the Usingen upload), caused NTP on Hermes and Hydra to go wild.  Restarted with a near-correct drift file.  Stamsund appears to have survived OK.
• 1130, 2006 Aug 14, Monday, week 33.  Restarted NTP on Hydra to enable statistics collection.
• 0030 - 0300, 2006 Aug 13, Sunday, week 32.  Local power cut, so all PCs restarted manually (except the older Pixie and Bacchus which automatically restarted).  Bacchus seems to have -10 instead of -75 for drift, so copied the more correct drift file and restarted NTP.
• 1600, 2006 Aug 08, Tuesday, week 32.  GPS source lost sync altogether and had to be rebooted (switched off and on again).  No obvious reason, unless it's too few satellites.  Only noticed this at 0820 on Wednesday 9th!
• 1000 - 1900, 2006 Aug 07, Monday, week 32.  PC Stamsund rebuilt wit Windows XP Pro.
• 1855 - 1940, 2006 Aug 03, Thursday, week 31.  PC Gemini froze and had to be manually rebooted.
• 1456 - 1522, 2006 Aug 03, Thursday, week 31.  PC Stamsund was down for maintenance.  Memory upgraded from 1 to 3GB, and thoroughly cleaned!
• 1100, 2006 Aug 03, Thursday, week 31.  Discovered that Hydra's NTP had stopped, because Hydra was not connected to the network when booted.  Restarted NTP.  It would be helpful if NTP attempted to repair connections itself rather than just giving up. 
• ~0900, 2006 July 17, Monday, week 29.  Gemini rebooted after a hang, and then security upgrades.
• 1903, 2006 July 15, Saturday, week 28.  Discovered that NTP had stopped on Bacchus, perhaps at 0748 on Friday, with a "No servers reachable" event log message.  Restarted, and the initial offset was +74msec.
• 2000, 2006 July 10, Monday, week 28.  GPS source lost sync, and the PPS signal did not re-appear.  Power cycled at 0500 Tuesday.  What is happening here?
• 1830, 2006 Jul 08, Saturday, week 27.  Pixie seemed to loose sync with its GPS/PPS source.  Could have been 2030.  Sunday 0710 discovered this.  No flashing PPS pulses from the GPS, so powered down GPS and restarted it.  PPS pulses OK within less than a minute, and Pixie regained sync.
• 1640, 2006 Jul 05, Wednesday, week 27.  A thunderstorm with lightning (helped by the rain?) may have stopped the GPS 18 LVC working.  Power off and on restored operation, although it took a few minutes after power on before the PPS pulses appeared.  EUMETCast loss of signal at the same time.  This seems to have started as a lack of satellite signals (?) around 1415 UTC, although the EUMETCast outage was between about 1500 and 1530 UTC.  Perhaps these events were not related?  The EUMETCast outage caused NTP to go wild on Hermes and Hydra, but I detected this (by ear) and manually restarted NTP on those systems.
• 0715, 2006 Jul 01, Saturday week 26.  Timekeeping on both Bacchus and Stamsund had gone wild since about midnight UTC, with incorrect very large drift values (over 400) being recorded in ntp.drift.  Both these PCs are 100% reliable.  Reset drift file to correct content and restarted ntp.  This is just like what happened at the new year, with the leap second.  Could it be that some of the Internet servers these PCs were seeing were still sending the leap second flag?  See: ntp-events.
• 1900, 2006 Jun 25, Sunday, week 25.  Restarted NTP on Hermes as the DVB reception system restarted (due to satellite signal loss - bad weather over Germany), and the automatic restart caused a transient.
• 1300, 2006 Jun 18, Sunday, week 24.  Restarted NTP on Hermes with -94.557 as the drift value.
• ~1100, 2006 Jun 17, Saturday, week 23.  Transient on Hermes due to signal loss and re-acquisition on DVB system.  Although signal was back OK, the NTP software seemed to "go wild".
• 2006 Jun 11, Sunday, week 23.  Some sort of transient or issue on Pixie seemed to stop its precision lock for a period, but it recovered.
• 1146, 2006 Jun 07, Wednesday, week 23.  Checked that the GPS 18 LVC really was in 2D, and it was.  No improvement seen, so probably the glitches are not loss of satellites.  Ah, well!
• 0955, 2006 Jun 07, Wednesday, week 23.  Reboot Gemini to get network cards back to standard IDs for MRTG.
• 0810, 2006 Jun 06, Tuesday, week 23.  Attempted to set GPS 18 LVC to 2D mode, so that fewer satellites are required to get a time "fix", but I'm not sure if this worked.
• 0750, 2006 May 31, Wednesday, week 22.  Gemini rebooted after a security update.
• 2006 May 19 - 22, week 20-21.  Gemini hang followed by a reboot late Monday 22.
• 0700, 2006 May 12, Friday, week 19.  Stamsund rebooted after security update.
• 1645, 2006 May 11, Thursday, week 19.  Hermes rebooted after update of TelliCast client software.
• 1530, 2006 May 09, Tuesday, week 19.  Found that the ntp.drift file on Gemini contained 9 0x00 bytes, and that ntp.drift.TEMP had the correct value.  Could not delete ntp.drift, and the permissions were correct.  Stopped NTP, and copied ntp.drift.TEMP to ntp.drift.  Restarted NTP OK.
• 0000 (clock) 2006 May 06, Saturday, week 18.  Hermes had about a +80ms time step, and gradually recovered back to zero.  No obvious cause.  There was a EUMETCast issue at almost exactly the same time - a co-incidence?  Had been up for about 80 days 1h 45m, and is still running without a reboot.
• 1120, 2006 Apr 09, Sunday, week 14.  Noted that Gemini USB WiFi hadn't restarted after the reboot, so unplugged and re-plugged the adaptor.  This seemed to kill NTP, and it was manually restarted at 1435 when it showed a 30ms offset.
• 1530, 2006 Apr 05, Wednesday, week 14, took Gemini down for about an hour to add a fan to the hard disk bays.  Will be interesting to see any effect or not on the diurnal offset.  Disk temperatures about 17°C lower.
• 1600, 2006 Feb 10, Saturday, week 6, removed Odin after 4 years service, it will be replaced by Gemini.
• 1110, 2006 Feb 01, Wednesday, week 5.  Rebooted Pixie following a request for a test.
• 1400, 2006 Jan 30, Monday, week 5.  Made all four PCs sync to Pixie as a preferred server.  Slight glitch resulted.
• 1100, 2006 Jan 30, Monday, week 5.  Finally got kernel recompiled on FreeBSD PC (pixie - it took 5 hours!) and enabled the PPS stuff.  Will keep an eye on this before changing the other servers.
• 0730, 2006 Jan 30, Monday, week 5.  Some sort of glitch on Hermes seeming to stop DVB reception.  DVB restored at 0745, but there was a timekeeping transient just after 0800 with a step of approximately 75ms.  Why?  There is nothing in the event log.
• 1700, 2006 Jan 29, Sunday, week 4.  Add new FreeBSD system with GPS NMEA - Pixie.  Discovered that you need to re-compile the FreeBSD Kernel to enable PPS support, so this is in progress.
• 1600, 2006 Jan 28, Saturday, week 4.  Add GPS NMEA-only reference clock to Odin for testing.  Looks like the 4800 baud serial output is about 200msec earlier than the (unused) PPS signal, but this can be fudged.  Jitter is bad (up to 10ms), so it will be interesting to see how this shows on the plots.  Lots of errors "clock GPS_NMEA(1) event 'clk_fault' (0x03)" so I've abandoned this at 1715.
• 0800, 2006 Jan 14, Saturday, week 2.  Daily drift on Stamsund and Odin is more than I like, so reset the maxpoll to the local servers to 9 (512 seconds).
• 1100 - 1500, 2006 Jan 13, Friday, week 2.  Odin down and rebooted following peculiar hang.
• 2030, 2006 Jan 11, Wednesday, week 2.  Removed all MaxPoll lines from Odin and Stamsund to see how they get on in a "default" configuration.  This should make all servers MaxPoll 1024 seconds.
• 1000, 2006 Jan 11, Wednesday, week 2.  Removed the "prefer hermes" line from Odin and Stamsund in case yesterday's events should happen again.  At least with Bacchus remaining stable, perhaps these clients would not have behaved as badly.
• 2100, 2006 Jan 10, Tuesday, week 2.  For some reason, Hermes NTP went wild.  About 1 hour later, Odin started going wild, and nearly three hours later Stamsund joined the wild bunch!  PC Bacchus showed no problems, perhaps showing the Internet connection stayed OK (the traffic looked normal).  PCs Odin and Stamsund only showed changes of selected server. At 2119 Hermes changed to server 130.88.200.6 (utserv.mcc.ac.uk) and suffered an immediate time reset of +0.194392s.  It continued seeing time resets until 06:43:15 on Jan 11, and seems to have been able to recover without further resets since then.  There's nothing in the event log to suggest any other problems around the time when NTP on Hermes went wild, however, at 2105 there was a burst of wind which caused momentary loss of a satellite signal.  Perhaps a mains glitch?
• 1350, 2006 Jan 09, Monday, week 2.  Rebooted Hermes after addition of RAMdisk and Windows security updates.
• 0910, 2006 Jan 06, Friday, week 1.  Rebooted Odin after official WMF security update.
• 1000, 2006 Jan 04, Wednesday, week 1.  Rebooted Odin and Stamsund following WMF security patch update.
• 1500, 2006 Jan 02, Monday, week 1.  Decided to replace older NTP software on Hermes and Bacchus by the more leap-second complaint version.  Perhaps should have done this before the leap-second rather than after!
• 0740, 2006 Jan 01, Sunday, week 1.  Hermes and Odin had not settled down after the leap second, and the drift files values were very big (over 400), so stopped ntpd on those systems, deleted the drift files, and restarted.  Ntpd seems to have some instability judging by the offset graphs. Bacchus (drift -55.3) and Stamsund (drift -11.5) did seem to be sorting themselves out, albeit slowly.  See: ntp-events.
• 2010, 2005 Dec 26, Monday, week 52.  Advised that having a large difference between MaxPoll and MinPoll was not a good idea.  "You have forced the poll interval and time constant to 64 s.  Should that source fail and the host switch to another source with poll interval clamped to 1024 s, the discipline loop is seriously undersampled and could well become unstable.", so reset Odin and Stamsund to MaxPoll=8 as a compromise.
• 0735, 2005 Dec 24, Saturday, week 51.  Updated Odin and Stamsund to have MaxPoll 6 on internal LAN servers and MinPoll 10 for their Internet servers.  This provides a fast response while minimising the load on the backup servers.
• 1530, 2005 Dec 22, Thursday, week 51.  Updated Odin and Stamsund Meinberg ntp-4.2.0b@1.1436mbg-xmas version.  This accepts the -M parameter to enable the MM timer while ntpd is running, so I could now disable this frig in my own program on Odin.  SSL library updated on Odin.  [Didn't use the Meinberg installer].
• 1125, 2005 Dec 20, Tuesday, week 51.  Changed Odin and Stamsund from 128s poll to default 1024s poll.  Since the steps are gone, there is no need for the shorter poll interval.  Noted that Bacchus seems to have rather more drift than Hermes, even though both are nominally synced the same.
• 0639, 2005 Dec 13, Tuesday, week 50.  Restarted NTP on Stamsund to collect loop statistics.  Noted that this caused an initial 20ms offset.
• 1855, 2005 Dec 12, Monday, week 50.  Installed experimental NTP version on Stamsund.  Version: ntpd 4.2.0b@1.1432-o Dec 12 17:03:03 (UTC+01:00) 2005 (43) from Meinberg.  Needed to edit the Image path in the registry to include a "-M" parameter to force the MM timer to 1ms resolution.  Having this parameter present caused previous NTP versions to fail.  Needed to update the SSL libraries as well.
• 0910, 2005 Dec 09, Friday, week 49.  Tried the same "force MM timer to 1ms" experiment on Stamsund, as a test.
• 1700, 2005 Dec 07, Wednesday, week 49.  Started running QuickTime player permanently on Odin to force the MM timer to run with a 1ms period.  Seems to have stopped the violent swings.  Looks as if the MM timer was permanently running on Odin until Friday, Dec 05.
• 1030, 2005 Dec 06, Tuesday, week 49, updated all systems to V4.2.0b from Terje Matheson ("ntpd 4.2.0b@1.1431-o Dec 06 10:23:21 (UTC+01:00) 2005 (1)")
• 0941, 2005 Dec 05, Monday, week 49, replaced older 2003 V4.2.0 ntpd on Odin by V4.2.0a from Meinberg 1370 download.  The jitter still seems high.  Version numbers were:  old: ntpd 4.2.0 fr Oct 17 8:49:33 2003 (1), new: ntpd 4.2.0a@1.1370-o Apr 12 14:10:45 (UTC+02:00) 2005 (12).  Also, at 1310 replaced Stamsund's ntpd.exe with the Meinberg 1370, as a test to see if the extreme swings were any different.
• 1500, 2005 Dec 02, Friday, week 48, something has upset the otherwise excellent timekeeping on Odin.  Tried to get rid of this on Saturday by reboots at 1100, 1800 and 2100.  Can't discover what the problem is.  See: ntp-events.
• 1830, 2005 Oct 27, Thursday, week 43, reverted to maxpoll 10 on Bacchus as no permanent change noted.
• 0800, 2005 Oct 04, Tuesday, week 40, changed maxpoll interval on Bacchus from 10 down to 9 to reduce diurnal variation.
• 0820, 2005 Sep 09, Friday, week 36, found that local ISP head-end was no longer offering NTP services so switched to a different external server.
• 2230, 2005 Sep 06, Tuesday, week 36, power glitch takes down systems from about 1600 to 2230
• 0830, 2005 Aug 16, Tuesday, week 33, was moved by ISP onto a different upstream connection to the cable modem service.  Watch and see if there is any improvement.  Yes (two days later), it's much better.  Thanks for an understanding ISP!
• 1510, 2005 Aug 11, Thursday, week 32, noted more noise over the last couple of weeks or so, so added another two NTP pool servers to Hermes as a test.
• 1510, 2005 May 28. Saturday, week 21, remove NPL servers from Bacchus - too noisy because too many servers?
• 1830, 2005 May 27, Friday, week 21, remove NPL servers from Hermes - too noisy?
• 2205, 2005 May 25, Wednesday, week 21, added NPL servers to Hermes & Bacchus.
• 0000, 2005 Mar 25, Friday, week 12, Blueyonder network maintenance causes glitch.
• 0930, 2005 Feb 04, Friday, week 05, moved Bacchus and Hermes onto NTP Pool Internet servers.
• 2230, 2005 Feb 03, Thursday, week 05, a Blueyonder network problem in Scotland causes another glitch.
• 1200, 2005 Jan24, Monday, week 04, reboot of Odin following internal problem.
• 2300, 2005 Jan 15, Saturday week 02, a Blueyonder network problem in Scotland causes another glitch.
• 1900, 2005 Jan 03, Tuesday week 1, some sort of disruption of level 3 links at Blueyonder cause transient glitch in timekeeping.
• 1945, 2004 Nov 16, Tuesday week 46, add 'prefer' option to Hermes entry in Odin & Stamsund's configuration
• 1200, 2004 Nov 16, Tuesday week 46, major configuration change, Bacchus & Hermes sync to the Internet, Odin and Stamsund sync to Bacchus and Hermes.  This change because Hermes is really an always-on and lightly loaded server.
• 0930, 2004 Nov 10, Wednesday week 45, made Stamsund not have Tinker Step 0.050 to see if the extreme swings were handled any better.  Not sure what is causing the swings.
• 2004 Nov 06, Saturday week 44, multiple reboots of Stamsund to update DVB card drivers
• 2004 Nov 05, Friday week 44, multiple reboots of Hermes to update DVB card drivers
• 1830, 2004 Sep 27, Sun week 39, power glitch required reboots of all PCs
• 2004 Sep 15, Wed week 37, lost Internet connection for 4.5 hours
• 1400, 2004 Mar 08, Mon week 9, tested one PC as an Internet server.  As the bandwidth throttling appears not to work, it eats all the upstream bandwidth and therefore clobbers timekeeping.
• 0800, 2004 Mar 05, noted that on Odin & Hermes there were some 10ms steps, and that the Internet servers were 1024s maxpoll (10).  Altered the configuration to make them maxpoll 7 and restarted.
• 2030, 2004 Mar 04, increased maxpoll to 7 (from 6) on Odin & Hermes.  Hermes & Odin not seeing the external servers, although they had 6 hours and 42? hours ago!
• 1200, 2004 Feb 26, changed external server, rebooted Odin & Stamsund.  Hermes & Stamsund not seeing the external server.  Why not?
• 1705, 2004 Feb 24, removed Stamsund as peer for Hermes and Odin.  Hermes still not seeing the external server.
• 0037, 2004 Feb 21, Sat, changed the external network server.  Hermes still not seeing the external server, Odin got a 30ms step during service restart sequence.
• 1638, 2004 Feb 14, Sat, as an experiment, added one more external network server to Hermes, Odin and Stamsund.  Note that Hermes is in the state of not talking to external servers.
• 1230, 2004 Feb 03, Tue, a power cut at 1150 resulted in all PCs being rebooted at 1235.
• 1145, 2003 Dec 05, Fri, a reboot of Odin again gave a large step and slow recovery cf. mid-November.
• 1205, 2003 Nov 18, Tue, added tinker step 0.050 to try and avoid these large steps on reboot (Hermes, Odin, Stamsund).
• 0915, 2003 Nov 16, Sun, Bacchus performance not so good, nor the rest of the network, so reverted to no "prefer".
• 1740, 2003 Nov 15, Sat, gave Bacchus a "prefer" server as it was clock-hopping.
• 1415, 2003 Nov 09, Sun, restored Hermes/Odin/Stamsund peering, in view of the number of reboots that Hermes is requiring.
• 1900, 2003 Nov 08, Sat, replaced Hermes with a different PC (actually old Wotan).
• 1000, 2003 Nov 07, Fri,  Hermes rebooted at 1000, DVB card added to Stamsund.
• 2003 Nov 04, for operational reasons, Hermes required a number of reboots, and seems to have lost accurate drift information.....
• 1745, 2003 Nov 01, Sat, noticed that Stamsund is noisy (doing large panoramas!), so remove it from peering with Hermes and Odin.
• 1000, 2003 Oct 31, Fri, added one more server to Bacchus.
• 2115, 2003 Oct 30, Thu, noticed that Bacchus was more noisy, so tried a "burst" option on one server.
• 1834, 2003 Oct 28, Tue, PC Wotan replaced by PC Stamsund.
• 0815, 2003 Oct 26, Sun, reset Bacchus to 1024s Internet poll, Hermes, Odin and Wotan to 64s poll.  So Bacchus may get a bigger offset, but should be lower jitter...
• 1700, 2003 Oct 25, Sat, left just Bacchus connected to the Internet servers, with Hermes, Odin and Wotan peering with each other and being served from Bacchus with one Internet as backup.
• 1050, 2003 Oct 25, Sat, made least-used interactive PCs the main peered Internet connected ones (Hermes and Bacchus) and the most heavily used PCs (Odin and Wotan) the LAN served peers.  Resulted in more drift than I liked on Hermes.
• 0915, 2003 Oct 25, Sat, changed LAN PCs Odin and Hermes to 128 second poll
• 1000, 2003 Oct 18, Sat, removed ntp.blueyonder.co.uk from Odin and Hermes as it seemed to pull Hermes during the evening and night, and Odin wouldn't talk to it in any case (permanently in INIT).
• 1315, 2003 Oct 17, Fri, updated to NTP release V4.2.0, added ntp.blueyonder.co.uk to Odin and Hermes as it has now had the large offset removed
• 0830, 2003 Oct 04, Sat, set Internet facing servers to 512s poll to reduce Bacchus offset
• 1530, 2003 Sep 05, Fri, restored Odin as peer for Hermes - graphs showed no change.
• 2100, 2003 Aug 21, Thu, removed Odin as peer for Hermes - likely to be unreliable due to animations.
• 1010, 2003 Jul 23, Wed, updated to V4.1.80-rc1 from Terje Mathisen.
• 1330, 2003 Apr 24, added Odin/Hermes as peer pair, and Bacchus/Wotan likewise.
• 1700, 2003 Mar 20, Thu, installed DVB card in Hermes, timekeeping goes to pot!
• 1000, 2003 Mar 18, Tue, updated to V4.1.74 with the CMOS code present.
• 1550, 2003 Feb 18, Tue, updated all to V4.1.74.
• 1615, 2003 Feb 07, Fri, revert to 1024s poll on Internet facing servers for a test.
• 0840, 2003 Jan 29, Wed, V4.1.73 (dev) installed on PCs Odin and Hermes, 0945 installed on Bacchus and Wotan.
• 0845, 2002 Dec 19, Thu, Bacchus and Wotan changed down to 512s maximum polling (to reduce Bacchus' drift).
• 1000, 20002 Dec 17, Tue, Bacchus and Wotan synced to Internet servers, PC Hermes  and Odin synced to Bacchus/Wotan with 64s maximum polling.
• 2340, 2002 Dec 12, power cut, all PCs rebooted.  Power back at 0020 Dec 13
• 1345, 2002 Nov 08, Fri, all PCs synced to Internet servers.
• 1100, 2002 Nov 07, Thu, Bacchus and Wotan synced to Internet servers, and PC Hermes and Odin synced to Bacchus/Wotan instead of Bacchus alone.
• 1445, 2002 Nov 06, Wed, only Bacchus was left synced to Internet servers, and PC Hermes, Odin and Wotan were synched to Bacchus instead of Internet NTP servers.

NTP Notes

For my own system

• Building a precise time source using FreeBSD and a GPS receiver
• Using Windows with a GPS/PPS source
• Using NTP on Windows Vista
• Using NTP in a cable-modem environment
• NTP 4.2.4 compared with NTP 4.2.5
• Mechanising NTP performance recording with MRTG
• My Windows multi-PC NTP monitoring program
• My Windows loopstats plotter
• My daily network traffic analysis, an example of using MRTG
• Some events in more detail
  • Leap-second December 2008
  • Using NTP on Windows Vista, 2007-2009
  • False leap-second July 2006
  • Leap second at the end of 2005
  • MMtimer issues uncovered December 2005
• Loopstats data - only updated from time to time
  • PC Bacchus
  • PC Feenix
  • PC Gemini
  • PC Hydra
  • PC Narvik
  • PC Stamsund

External links

• Official NTP documentation
• Where to find NTP software
• Meinberg Windows NTP package download (includes Windows installer which can be automated for hands-off install)
• Meinberg free NTP Time Server Monitor (und auf Deutsch)
• The NTP Windows port from Terje Mathisen which I have used in the past

Hardware suppliers - in alphabetical order - vendor-supplied descriptions

• Galleon Systems - Providers of NTP Servers for Time Synchronisation of computer networks
• Meinberg - NTP Time Servers and innovative solutions for Time and Frequency Synchronization using GPS, DCF77 and IRIG
• Spectracom - NetClock GPS time servers for vital communications networks
• TimeTools - TimeTools provides accurate MSF DCF-77 and GPS NTP Time Servers for accurate network timing.
• World Time Solutions - Manufacturer of NTP Servers and GPS Time Receiver Systems