News items for tag time - Koos van den Hout

It seems the Turkish provider ttnet.tr fell off the Internet for a few hours today. Since we volunteered ntp.cs.uu.nl for tr.pool.ntp.org the drop in traffic was very, very noticeable.

First peak at 5000 packets/second ntp traffic seen on ntp.cs.uu.nl. Still going strong under this load.

We volunteered ntp.cs.uu.nl for extra capacity for the Turkish ntp pool, and the results are quite visible in the ntp.cs.uu.nl statistics. Suddenly peaks are near 5000 packets per second. But ntpd (and the freebsd kernel) deal with it without problems.

I upgraded ntpd on ntp.cs.uu.nl from 4.2.4 to 4.2.6 and suddenly I notice in the output that this has changed the stratum from 2 to 1.

$ ntpq -c rv ntp.cs.uu.nl
status=011d leap_none, sync_atomic, 1 event, event_13,
version="ntpd 4.2.6@1.2089-o Fri Jan 15 14:31:14 UTC 2010 (1)",
processor="i386", system="FreeBSD/5.4-RELEASE-p13", leap=00, stratum=1,
precision=-19, rootdelay=0.000, rootdisp=1.456, refid=PPS,
reftime=cefb066f.cbe638ff  Fri, Jan 15 2010 16:21:19.796,
clock=cefb0693.889dd5ee  Fri, Jan 15 2010 16:21:55.533, peer=7047, tc=6,
mintc=3, offset=-0.001, frequency=15.448, sys_jitter=0.002,
clk_jitter=0.001, clk_wander=0.002

Which matches the peer list where the PPS stratum is now 0:

$ ntpq -c peer ntp.cs.uu.nl
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*huygens.cs.uu.n .PPS.            1 u   23   64  377    0.197    0.009   0.258
+stardate.cs.uu. .PPS.            1 u   13   64  377    0.998   -0.058   0.033
+tijger.phys.uu. metronoom.dmz.c  2 u   15   64  376    0.599    0.004   0.185
 LOCAL(0)        .LOCL.          10 l  627   64    0    0.000    0.000   0.002
oPPS(0)          .PPS.            0 l   49   64  377    0.000   -0.002   0.002
 NTP.MCAST.NET   .MCST.          16 u    -   64    0    0.000    0.000   0.002

I guess some definition of PPS input has changed. Now I wonder how much more ntp traffic this will cause.

Work on home server greenblatt: time for less disks with more storage. So I bought two sata disks, one huge one to store the camera archive and scratch files, and one for the system and home directories. The choice for two disks is so the one with the camera archive and the scratch files can fall asleep when not in use, to save a bit of power. Installing both new disks at once wasn't going to happen due to space and cabling considerations so I started with the big one. When that one is done I can remove three pata disks from the system. I also updated the system bios to the latest version which made the system clock a lot more stable, ntpd now runs without having to use tickadj. Bios updates are easy these days: this bios can update itself from a USB stick. I chose logical volume management (lvm2) again for managing the big disks so it will be easy to expand storage when needed without getting a big tree of filesystem mounts.

I noticed requests for port 37/udp in our firewall to our ntp server. That is the 'daytime' protocol which is absolutely ancient in an Internet timescale. I opened the port and started the service as an experiment and started tcpdump on it. The results are interesting:

09:50:09.749723 IP xx.xx.178.51.37 > 131.211.84.189.123: NTPv4 client, strat 2, poll 7, prec -20
09:50:09.749782 IP 131.211.84.189.123 > xx.xx.178.51.37: NTPv4 server, strat 2, poll 7, prec -19
09:52:19.808243 IP xx.xx.178.51.37 > 131.211.84.189.123: NTPv4 client, strat 3, poll 7, prec -20
09:52:19.808301 IP 131.211.84.189.123 > xx.xx.178.51.37: NTPv4 server, strat 2, poll 7, prec -19
09:53:08.511939 IP xx.xxx.183.183.34505 > 131.211.84.189.37: UDP, length: 0
09:53:08.513364 IP 131.211.84.189.37 > xx.xxx.183.183.34505: UDP, length: 4

Most traffic seen by 'tcpdump port 37' is from source port 37. Which is an artifact of certain NAT devices translating privileged ports (< 1024) to other privileged ports. Certain versions ntpd seem to ignore these requests. But there are real clients using the 'daytime' protocol.

The firewall at work got adjusted to not keep NAT state (cisco-term: xlates) for non-NAT sessions. So now I can run our ntp pool server at maximum speed again without the firewall overflowing its state tables. If you ever run into this problem, the right command is xlate-bypass.

Just seen in the ntp stats: A peak of 3271 packets/second of ntp traffic on ntp.cs.uu.nl.

New record in ntp.cs.uu.nl traffic: 2429 packets/second of ntp traffic. At that level the server does seem to miss some traffic: according to the pool.ntp.org: Stats for 131.211.84.189 it missed one request from the monitoring system completely, which drops the score several points at once.

You can check out of the NTP Pool anytime, but you can never leave...
Two years after we removed a server from the ntp pool we still see regular traffic to that IP. Documented in the NTP events log, we still have over a hundred regular clients.
With regards to the Eagles.

More ntp fun: I now have multicast ntp time working and documented. With the listed ntp key clients can use the multicasted network time from ntp.cs.uu.nl. The outgoing timestamps look like this in tcpdump:

14:45:06.821419 IP (tos 0x0, ttl  30, id 6195, offset 0, flags [none], proto 17, length: 96) 131.211.84.189.ntp > 224.0.1.1.ntp: NTPv4, length 68
        Broadcast, Leap indicator:  (0), Stratum 2, poll 6s, precision -19
        Root Delay: 0.000000, Root dispersion: 0.001724, Reference-ID: 127.127.22.0
          Reference Timestamp:  3424077852.817587474 (2008/07/03 14:44:12)
          Originator Timestamp: 0.000000000
          Receive Timestamp:    0.000000000
          Transmit Timestamp:   3424077906.819248263 (2008/07/03 14:45:06)
            Originator - Receive Timestamp:  0.000000000
            Originator - Transmit Timestamp: 3424077906.819248263 (2008/07/03 14:45:06)

I also see multicasted time from 192.36.143.151 (time2.stupi.se) but I can't find a key for it.

A new peak in ntpd traffic on ntp.cs.uu.nl : 1986 packets/second. From a very limited peek at the IP numbers it seems the 'friends' at turkish telecom were quite interested in the correct time.

I found a really neat description of a project to build an accurate NTP stratum-0 server with the garmin gps 18 lvc.

At work we now graph several temperatures in the serverroom (results are not public). We joked (or not..) last Friday that we could add a lot of sensors inside and outside the serverroom (that is where my thinking about 1-wire systems came in again) and have someone research this micro-climate and correlate the micro-climate with the ntp statistics. We did see the influence of the cold wind from the east on the pll stats of several ntp servers.

About one and a half hour later, ntp.cs.uu.nl peaked at 1000.60 packets/second.

I'm a timegeek, and part of that is making our timeserver at work perform great in the NTP Pool project. With the recently updated pool dns system, servers that have more upstream bandwidth get more clients. We have been ogling our ntp stats for ntp.cs.uu.nl a lot seeing how the client count is through the roof (the internal data structures of ntp can't count beyond 3500 clients without serious hacking) and traffic is rising seriously lately. Still waiting for the first time we get over 1000 packets/second ntp traffic. Our ntp server has no problem at all dealing with this.

Timegeeks who can make the serious commitment towards being a long-time ntp pool member but don't have the hardware available to pick that last nanosecond of precision from the sky should peek at this announcement: Meinberg is donating some really great timekeeping equipment to the ntp pool project and it will be available to a pool member.

I played with Asterisk scripting last evening. Started out with an empty extensions.conf and went from there to create a speaking clock. I was trying to copy the old Dutch ptt telecom 002 experience, with a Dutch voice I found at VoIP wiki nl: Asterisk en taal afhankelijke sound files. I run this ofcourse on a ntp-synchronized server, a talking clock has to be correct! I tried to make it run on intervals of 10 seconds, but that doesn't work completely yet.

Sometimes you run into a very oooh, shiny! subject when looking for something else. I visited David Taylor's pages looking for his writeup of FreeBSD NTP server with an external GPS mainly to 'compare notes' with my own FreeBSD ntpd PPS setup (PPS slave) writeup. No surprises there, but one click led to the other, when I found out that David Taylor is also interested in receiving weather information and dove into that. He has a nice overview of his experiments in weather satellite receiving and decoding and I found out about EUMETCast data where you can receive weather satellite images via a satellite dish. Which uses a 'data' PID on a transponder, which is implemented as multicast data streams. Several personal interests for me and work-related stuff comes together in this. It's good to know this exists, but I won't be running out to buy a satellite weather map receiver now.

I've been thinking about building a weather-station / ntp server with solar power for a while, and decided to start documenting the design, the ideas and the (lack of) progress. Named project sundial because it uses the sun to tell time, just in a somewhat convoluted way.