2021-04-14 Year 2038 is coming!
Interesting kernel message in Linux today:[ 3906.977410] ext2 filesystem being mounted at /media/koos/disk supports timestamps until 2038 (0x7fffffff)So that filesystem (and lots of others) will give issues in 2038. Things need work before that date!
2021-04-08 Stopping with NTP servers at work
For almost 20 years I was involved with the running of NTP time servers at work. But the hardware aged and my job is no longer in systems administration and not in the department actually housing the timeservers. So, time to stop doing it. The pool ntp server has been retracted, DNS names removed and soon I will make one final trip to shut down hardware one last time and remove it from racks. The end for ntp.cs.uu.nl and others. I still run an NTP server at home which is available in the IPv6 NTP pool. That server also compared itself to one of the servers at work so it has been reconfigured. I added a few upstream servers and made sure all of them are reachable via IPv6. The log of NTP service at cs.uu.nl was kept, here is the final version:
Date Event 8 Apr 2021 DNS names for ntp service at cs.uu.nl removed 2 Apr 2021 Announcement posted to system administration mailing list that ntp service at cs.uu.nl will stop 24 Sep 2014 A second stratum-1 ntp appliance is brought on-line, galileo.cs.uu.nl 28 Nov 2011 Fixed the networking for stardate, the full time lab is up and running. 23 Nov 2011 The antenna cable connectors are soldered on which results in a working setup after a few tries. Stardate is better at reporting the state of the power to the GPS antenna, but has no working network. Huygens has working network and serves time to metronoom. 22 Nov 2011 The server ntp.cs.uu.nl is active at its new IP. Our own GPS reference doesn't work yet: we still need to solder the right connectors on the antenna cable. The server is added to the ntp pool and traffic starts to flow a few hours later. 15 Nov 2011 The ntp servers are moved to their new location 14 Nov 2011 The ntp servers are switched off 13 Nov 2011 We retract ntp.cs.uu.nl at its current address from the pool because the serverroom will move physically, the ntp equipment will move to a different location and the IP will change to deal with the traffic better 18 Sep 2011 Stats for doei.cs.uu.nl, five years after withdrawing it from the ntp pool 19 Sep 2010 Stats for doei.cs.uu.nl, four years after withdrawing it from the ntp pool 4 Mar 2010 The turkish adsl provider ttnet falls off the Internet for a few hours, traffic falls from 2000 packets/second to 100 packets/second in that time 22 Jan 2010 We volunteer ntp.cs.uu.nl for the turkish part of the ntp pool. Traffic explodes, peaks over 5000 packets/second 18 Sep 2009 Stats for doei.cs.uu.nl, three years after withdrawing it from the ntp pool 28 Jul 2009 ntp.cs.uu.nl back at full speed in the ntp pool, firewall configuration fixed 15 Jul 2009 rear doors of racks closed again 2 Jul 2009 10:00 serverroom airco has problems with high temperatures (28-30 C), we open rear doors of racks which makes the temperature go down a bit in the racks but the airco still has hard work Mar 2009 ntp.cs.uu.nl tuned down in the ntp pool to avoid firewall issue 18 Sep 2008 Stats for doei.cs.uu.nl, two years after withdrawing it from the ntp pool 17 Jan 2008 huygens.cs.uu.nl has a GPS reception failure, fixed with a software update 18 Sep 2007 Stats for doei.cs.uu.nl, a year after withdrawing it from the ntp pool 11 Mar 2007 airco failure serverroom 5 Mar 2007 all ntp servers moved to one rack close together for temperature stability 20 Jan 2007 airco failure serverroom 9 Jan 2007 huygens.cs.uu.nl added as stratum-1 23 Dec 2006 airco failure serverroom 29 Nov 2006 powerfailure in our building 1 Nov 2006 metronoom.dmz.cs.uu.nl takes over as ntp.cs.uu.nl and joins pool.ntp.org ~ 24 Oct 2006 antenna cable to stardate.cs.uu.nl reconnected ~ 6 Oct 2006 ntpd on stardate disabled: free running clock starts to differ too much from correct time ~ 25 Aug 2006 antenna cable from stardate.cs.uu.nl disconnected because of building and recabling activities 1 Aug 2006 doei.cs.uu.nl leaves pool.ntp.org 19 Aug 2003 doei.cs.uu.nl joins pool.ntp.org 10 Jan 1999 stardate.cs.uu.nl set up as stratum-1 with GPS time reference
2021-04-07 The NTP ham clock is ticking
Recently the parts for the NTP ham clock I saw in the Electron magazine arrived: an ESP32 module and a TFT display. It took a bit before I had time to actually do something with them but recently I put the modules on breadboard and started making the needed connections. There are not a lot of those, only 8 wires need to be connected between the ESP32 microcontroller and the TFT display. After some fiddling it worked and I managed to program it all with the settings I like, such as the right timezone rules for the Netherlands, 24 hour display on both clocks and it fetches the NTP time from the NTP server in the shed so it doesn't rely on outside connectivity. Now to find a case for it and wire it neatly.
Recently the parts for the 2021-02-27 Ordered parts for an NTP ham clock
Today the Electron magazine of the Veron amateur radio club came in, the March 2021 Veron Electron (Dutch). As I was browsing the magazine and reading articles I came across an article about building an NTP ham clock, consisting of an ESP32 module and a TFT LCD display, and the rest is all in software. I directly wanted to build this, as this combines two of my interests: amateur radio and NTP time synchronization. It displays both the local time and the UTC time on the TFT display, just like PyHamClock does on my screen. The article is based on the same project at W8BH projects which gives me a good descriptive pdf. So I ordered an ESP32 module and ILI9341 TFT LCD display from an aliexpress seller and now I wait, because this will take about a month.
2021-01-13 Fiber to the shed
There is no fiber to our home in the near future but I am working on laying another fiber route: from the switch in the cupboard downstairs to the shed. This is because the NTP server in the shed still has intermittent connectivity issues when using 2.4 GHz wifi due to the 2.4 GHz wifi channels being very crowded. The wifi dongle has no 5 GHz support and I don't think I would get it very reliable. But other options are also not ideal. As a radio amateur I can't go back to using powerline (network over power cables) and I wouldn't feel safe with a network cable running that far should a lightning strike ever occur. I should write "occur again" since I have had a network switch with probable lightning damage before. The only option left is what you guessed from the title of this post: fiberoptic cable. No interference to my radio reception and a lot less chance of lightning blowing up parts of the network and connected computers. But a whole new world of fiber types, fiber lengths, wavelengths, connector types and interface types opened up to me. The switch in the cupboard downstairs has SFP ports, but how to get beyond that. The raspberry Pi 3B+ that I use is 100 mbit only and I wasn't sure how to handle that. So I asked someone who is very good with fiber networks to explain to me what options are available and that person dug up some lengths of fiber that are no longer used and some 100 mbit fiberoptic transverters that were also a wrong purchase. So I already have the connectivity hardware available. Now all I need is a physical route between the shed and the rest of the network. There is an old plastic pipe from the shed to the crawlspace of our house that was once used for heating will probably do the trick once I figure out how to remove the old heating pipes from it. I guess there is some real dirty work below the floor of our house and in the shed in my near future. I will also need to buy plastic tubing to safely guide the vulnerable fiber. And some hooks to hang this tube and other cables from the floor instead of having them lie in the sand in the crawlspace. Since there is also an old gas pipe in the plastic pipe I will make really really sure first that one isn't connected somewhere. This was all triggered by adding the ntp server in the shed to the NTP pool and having the pool monitoring system gripe about the server becoming unreachable as soon as I have wifi problems. The things I will do for serving the right time!
2020-10-04 Moved the new Raspberry Pi ntp server to the shed and did the last bits of configuration
I moved the new ntp server to the shed today. I found a nice case for it: an actual wooden box. I climbed on the roof of the shed to find a place for the GPS antenna (with magnetic base). Parts of the enclosures around our solar panels are from ferrous metals, so I found a place with an ok view of the sky to place the antenna and led the cable to a ventilation shaft to get it inside the shed. I made sure the cable was going up in the ventilation shaft first to avoid having a drip loop on one of our bicycles.Read the rest of Moved the new Raspberry Pi ntp server to the shed and did the last bits of configurationAlthough I did most work on the w1retap configuration before I couldn't get it running at first. I kept seeing the error message:
koos@henkp:~ $ LD_LIBRARY_PATH=/usr/local/lib/w1retap w1find DS2490-1 Error 119: Failed to set libusb configurationIt took some serious searching to find a hint: that is caused by the usb device file access rights. Solution is to install the 45-w1retap.rules that comes with w1retap into /etc/udev/rules.d.At the moment weather data is being fetched on the Raspberry but the wifi between shed and house is so bad that the data stays there. I'm not sure how that can be fixed. It turns out the external wi-fi dongle I bought was listed as having 5 GHz support, but the reviews of the chipset used say it doesn't. The congestion in the 2.4 GHz band makes it very difficult to reach the pi. Doing a ping test over longer time gives me 91% packet loss.
I dug up a different 2.4 GHz antenna from the junkbox and suddenly the connection is stable with a lot less packet loss. This antenna is directional and now pointing right at my access point.
Now the weather data is collected and forwarded to the server for Weather station Utrecht Overvecht.
NTP didn't seem to work on the first try, I'm not seeing any data for the GPS_NMEA server. This works again after a powerdown/up.
2020-07-04 Again with systemd in the new GPS Pi
Again and again systemd annoys me. This time in the GPS Pi configured for timing. Since I want it to work perfectly at start I added the systemd rules as suggested by A Raspberry Pi Stratum 1 NTP Server - Phil's Occasional Blog with /etc/systemd/system/ublox-init.service containing:[Unit] Description=u-blox initialisation Before=gpsd.service Before=ntp.service [Service] Type=oneshot ExecStart=/usr/local/bin/gpsctl -q -a -B 115200 --configure_for_timing [Install] WantedBy=multi-user.targetAfter reboot ntp was running, but no data at all from the gps unit, and gpsctl was unable to revive it. The solution was to disable the above unit and ntpd, powerdown and restart the whole system and try again. After that doing the changes by hand and starting ntpd worked fine. It's probably some sort of race condition, but any time I try to make a system with systemd do something reliably I run into things like this.
2020-07-03 Switched the GPS configuration to one optimized for timing
Based on A Raspberry Pi Stratum 1 NTP Server - Phil's Occasional Blog I switched the gps to a configuration optimized for timing. The default settings are optimized for location services, but I want an NTP server. I used gpsctl to configure the ublox chip in the GPS/RTC Hat:$ gpsctl -a -B 115200 --configure_for_timing -vv Serial port ("/dev/ttyAMA0") open... Serial port open and configured... Automatically determining baud rate... Trying 230400 baud... Trying 115200 baud... Trying 57600 baud... Trying 38400 baud... Trying 19200 baud... Trying 9600 baud... Synchronized on 9600 baud... Changing baud rate to 115200... Successfully changed baud rate to 115200...After that I got location data at a high speed. I changed the /etc/ntp.conf parameters to use the GPS_NMEA and PPS drivers, with:# PPS reference server 127.127.22.0 minpoll 4 maxpoll 4 fudge 127.127.22.0 refid PPS # GPS NMEA driver server 127.127.20.0 mode 89 minpoll 4 maxpoll 4 iburst prefer fudge 127.127.20.0 flag1 0 flag2 0 flag3 0 time2 0.043 refid GPSAnd now I get much better numbers:$ ntpq -p remote refid st t when poll reach delay offset jitter ============================================================================== oPPS(0) .PPS. 0 l 14 16 377 0.000 -1.656 0.134 *GPS_NMEA(0) .GPS. 2 l 13 16 377 0.000 -11.730 0.517 +ntpritchie.idef 131.211.8.244 3 u 44 64 377 4.263 1.436 62.373 +metronoom.dmz.c 192.87.106.3 2 u 44 64 377 12.141 -2.250 49.247 koos@henkp:~ $ ntpdc -c kern pll offset: -0.00142676 s pll frequency: 7.468 ppm maximum error: 4.934e-06 s estimated error: 3.372e-06 s status: 2001 pll nano pll time constant: 4 precision: 1e-09 s frequency tolerance: 500 ppmThe time offset factors still need work, but I'm getting close!
2020-07-03 The GPS ticks!
I remembered the junkbox contains an active GPS antenna which I bought together
with the gpskit gps unit in 2003(!). And some other bits and pieces included a
SMA to BNC adapter so I put the little GPS antenna outside and connected it
to the GPS/RTC Hat.
Before I was back behind a computer it was showing a location and within a
few minutes it had a PPS pulse. I was used to cold start taking at least 15
minutes with the gpskit!
So I tested with ntpd talking to gpsd via shared memory. This gave an
interesting offset between local gps time and a nearby ntp server.
$ ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
*SHM(2) .PPS. 0 l 6 64 377 0.000 -0.149 1.672
xSHM(0) .GPS. 0 l 5 64 377 0.000 -104.51 1.943
+ntpritchie.idef 131.211.8.244 3 u 101 64 376 2.774 0.950 13.948
+metronoom.dmz.c 131.211.8.252 2 u 99 64 376 10.482 -0.844 10.638
$ ntpdc -c kern
pll offset: -0.000136461 s
pll frequency: -11.054 ppm
maximum error: 1.3748e-05 s
estimated error: 1.7071e-05 s
status: 2001 pll nano
pll time constant: 6
precision: 1e-09 s
frequency tolerance: 500 ppm
I'm not too happy about the fact that the GPS NMEA messages are seen as wrong,
so I'm going to stop using gpsd and go for a setup optimized for timing.
2020-07-02 Setting up the Raspberry Pi to talk to the GPS/RTC board
Items with tag time before 2020-07-02With most of the hardware in, it is time to configure the Raspberry Pi to allow the GPS/RTC board to be installed. One tip was to do this before installing the board to avoid serial conflicts. First steps based on Building a GPS Time Server with the Raspberry Pi 3 which uses a different GPS board. Disabling tty service on the UART:# systemctl stop serial-getty@ttyAMA0.service # systemctl disable serial-getty@ttyAMA0.serviceAnd make changes to /boot/cmdline.txt to disable serial console, removing the console=serial0,115200 part. Also needed is to disable the use of the hardware uart for bluetooth. This device does not need to do bluetooth at all, so I disable the software.sudo systemctl disable hciuartAnd add the lines to disable the bluetooth uart to /boot/config.txt:dtoverlay=pi3-disable-btAnd with that the UART is completely free to use for GPS and PPS messages. I made all these changes and only added the GPS/RTC hat to the Pi after these changes were done. Next steps were to add the i2c settings according to the GPS/RTC manual. For this I addeddtoverlay=i2c-rtc,rv3028 dtoverlay=pps-gpioAnd indeed the i2c bus appears as the manual says:# apt-get install python-smbus i2c-tools [..] # i2cdetect -y 1 0 1 2 3 4 5 6 7 8 9 a b c d e f 00: -- -- -- -- -- -- -- -- -- -- -- -- -- 10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 40: -- -- 42 -- -- -- -- -- -- -- -- -- -- -- -- -- 50: -- -- UU -- -- -- -- -- -- -- -- -- -- -- -- -- 60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 70: -- -- -- -- -- -- -- --I removed the fake-hwclock package and tested operation. On the commandline it works, but in a reboot I still see weird times in the log. After that I did the changes to /lib/udev/hwclock-set, now it looks like:dev=$1 #if [ -e /run/systemd/system ] ; then # exit 0 #fi if [ -e /run/udev/hwclock-set ]; then exit 0 fi if [ -f /etc/default/rcS ] ; then . /etc/default/rcS fi # These defaults are user-overridable in /etc/default/hwclock BADYEAR=no HWCLOCKACCESS=yes HWCLOCKPARS= HCTOSYS_DEVICE=rtc0 if [ -f /etc/default/hwclock ] ; then . /etc/default/hwclock fi if [ yes = "$BADYEAR" ] ; then # /sbin/hwclock --rtc=$dev --systz --badyear /sbin/hwclock --rtc=$dev --hctosys --badyear else # /sbin/hwclock --rtc=$dev --systz /sbin/hwclock --rtc=$dev --hctosys fi # Note 'touch' may not be available in initramfs > /run/udev/hwclock-setThe rtc has to be configured correctly, I used information from A Raspberry Pi Stratum 1 NTP Server - Phil's Occasional Blog to configure the rv3028 chip. Get the gpsctl tool and use configure-rv3208.sh to set up the chip. Now the rtc is correct and used at boot time. I'm seeing NMEA messages when I run gpsd or ask the serial port for data. The NMEA messages are very limited because there is no GPS antenna connected yet.