News items for tag work - Koos van den Hout

Met alle aankomende wijzigingen op het werk hebben we besloten om de spamfiltering uit te besteden aan de surfnet mailfilter dienst. Die worden er voor betaald om de filtering dagelijks bij te houden en wij hebben er straks minder tijd voor. Totnogtoe was het natuurlijk altijd onze 'eigen' mailsetup en konden we zelf de spamstats bijhouden, en dat verliezen we.
We hebben eerst students.cs.uu.nl omgezet en vanmorgen cs.uu.nl. In de logs van de studentenmailserver viel me opeens op dat de smtpd ratelimiting (anvil) van postfix aansloeg op de surfnet mailfilters dus ik heb de surfnet mailfilter adressen toegevoegd aan de smtpd_client_event_limit_exceptions setting in postfix. Bij cs.uu.nl gebruiken we postfix, al van toen het nog vmailer heette. In sendmail zou ik voor die IP blokken andere ratelimits kunnen zetten maar postfix heeft blijkbaar alleen de opties default en geen ratelimits.

It seems the Turkish provider ttnet.tr fell off the Internet for a few hours today. Since we volunteered ntp.cs.uu.nl for tr.pool.ntp.org the drop in traffic was very, very noticeable.

First peak at 5000 packets/second ntp traffic seen on ntp.cs.uu.nl. Still going strong under this load.

Lots of phishing attempts for webmail accounts flying by, at the moment it seems popular to use webform hosters to ask for account credentials. I seem to miss a part of these. Probably my spamfilters being too good or something. But at work there are some people who know I am interested in new and recurring strains of Internet abuse so I still get interesting stuff forwarded to investigate. The latest catch advertised a dot.tk domain which inlined a webform from a tripod hosted site which was a copy of an emailmeform.com form and used emailmeform.com to process it and redirected to a generic thankyou form by a new zealand printer supplies company. It takes a bit of tracing and trying to solve such a puzzle and notify all parties about their role in the abuse.

No license to rdesktop for me: I recently got a really weird error from rdesktop:

koos@leek:~$ rdesktop -M -g 1200x900 -d something terminalserver
Autoselected keyboard map en-us
disconnect: No valid license available.

Some searching found me: License to rdesktop. Indeed, setting a different hostname from my own hostname helps:

koos@leek:~$ rdesktop -M -g 1200x900 -d something -n leeks terminalserver
Autoselected keyboard map en-us
/users/koos/.rdesktop/licence.leeks.new: Permission denied
WARNING: Remote desktop does not support colour depth 24; falling back to 16

The license file error has to do with another workaround. But maybe the running out of licenses for 'leek' is because I never give licenses back. Why is all this software very busy with making sure money is made for its maker and not busy with helping the user.

We volunteered ntp.cs.uu.nl for extra capacity for the Turkish ntp pool, and the results are quite visible in the ntp.cs.uu.nl statistics. Suddenly peaks are near 5000 packets per second. But ntpd (and the freebsd kernel) deal with it without problems.

I upgraded ntpd on ntp.cs.uu.nl from 4.2.4 to 4.2.6 and suddenly I notice in the output that this has changed the stratum from 2 to 1.

$ ntpq -c rv ntp.cs.uu.nl
status=011d leap_none, sync_atomic, 1 event, event_13,
version="ntpd 4.2.6@1.2089-o Fri Jan 15 14:31:14 UTC 2010 (1)",
processor="i386", system="FreeBSD/5.4-RELEASE-p13", leap=00, stratum=1,
precision=-19, rootdelay=0.000, rootdisp=1.456, refid=PPS,
reftime=cefb066f.cbe638ff  Fri, Jan 15 2010 16:21:19.796,
clock=cefb0693.889dd5ee  Fri, Jan 15 2010 16:21:55.533, peer=7047, tc=6,
mintc=3, offset=-0.001, frequency=15.448, sys_jitter=0.002,
clk_jitter=0.001, clk_wander=0.002

Which matches the peer list where the PPS stratum is now 0:

$ ntpq -c peer ntp.cs.uu.nl
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*huygens.cs.uu.n .PPS.            1 u   23   64  377    0.197    0.009   0.258
+stardate.cs.uu. .PPS.            1 u   13   64  377    0.998   -0.058   0.033
+tijger.phys.uu. metronoom.dmz.c  2 u   15   64  376    0.599    0.004   0.185
 LOCAL(0)        .LOCL.          10 l  627   64    0    0.000    0.000   0.002
oPPS(0)          .PPS.            0 l   49   64  377    0.000   -0.002   0.002
 NTP.MCAST.NET   .MCST.          16 u    -   64    0    0.000    0.000   0.002

I guess some definition of PPS input has changed. Now I wonder how much more ntp traffic this will cause.

I tried to use the --filter option in rsync but I was a bit baffled by the syntax and the manpage is nice but I couldn't understand. I wanted certain directories completely, other directories default excluded and certain files in one directory but not all. After some trail and error and talking to the teddybear:

rsync -rvv --progress /home/koos/rsyncsource/ /home/koos/rsyncdest --filter='merge /home/koos/rsyncfilter'

And in the filter file name things to include and exclude:

+ /wel/
- /niet/
+ /random/file
- /random/*

And the result is what I want:

$ ~/bin/testrsync 
building file list ... 
[sender] showing directory wel because of pattern /wel/
[sender] hiding directory niet because of pattern /niet/
[sender] hiding file random/niet because of pattern /random/*
[sender] showing file random/file because of pattern /random/file
7 files to consider
delta-transmission disabled for local transfer or --whole-file
random/
random/file
           0 100%    0.00kB/s    0:00:00 (xfer#1, to-check=4/7)
wel/
wel/file1
           0 100%    0.00kB/s    0:00:00 (xfer#2, to-check=2/7)
wel/file2
           0 100%    0.00kB/s    0:00:00 (xfer#3, to-check=1/7)
wel/file4
           0 100%    0.00kB/s    0:00:00 (xfer#4, to-check=0/7)
total: matches=0  hash_hits=0  false_alarms=0 data=0

sent 319 bytes  received 126 bytes  890.00 bytes/sec
total size is 0  speedup is 0.00

Now to do this on a filesystem with 151000 files.

Y2.01K problem: SpamAssassin had a rule since 2006 that e-mail with a date in the 'far future' was likely spam. The 'far future' was defined as 2010-2099. So today that rule started firing, leading to missed e-mail. Documentation for SpamAssassin Rule: FH_DATE_PAST_20XX. Time for an update there...

It's that xsnow time of year. I wanted to compile it for our students and staff to use and found a major Makefile and a real Imakefile (remember those?):

$ wc Makefile  Imakefile 
  957  2413 26799 Makefile
    7    21   172 Imakefile

Trying to find the 'real' problem I managed to reduce all that to:

xsnow: xsnow.o toon_root.o
        gcc -o xsnow xsnow.o toon_root.o -lm -lXpm -L/usr/X11R6/lib

imake gave us somewhat overkill Makefiles...

I was replacing ssl certificates on a lot of servers and got it working everywhere except on our ldap server. The SSL certificate chain wasn't given out so there was no link between a trusted root and the certificate on the server. I had it configured:

TLSCACertificateFile /etc/openldap/ssl/cacert.pem
TLSCertificateFile /etc/openldap/ssl/servercrt.pem
TLSCertificateKeyFile /etc/openldap/ssl/serverkey.pem

With the certificate in servercrt.pem and the intermediate certificates in cacert.pem. But that was a config from an older server which uses OpenSSL, including openssl libraries (libssl). The newer ldap server uses the gnu tls libraries (libgnutls) which really need:

TLSCertificateFile /etc/openldap/ssl/servercrt.pem
TLSCertificateKeyFile /etc/openldap/ssl/serverkey.pem

With the server certificate and the entire chain together in servercrt.pem. Something to keep in mind, so I documented it on our internal wiki.

Power failure this morning at work.. which left us not in the dark (enough emergency lighting) but with a completely silent serverroom. When the power came back we had some hours of work to get everything up and running again. Worst problem was with a number of Xen based virtualhosts, some centos upgrade had suddenly created a network device virbr0 which uses NAT and a local dhcp pool and enslaved all xen domU network interfaces under that bridge with no access to the 'real' network because NAT was not set up so their NFS root mount failed. The details on virbr0:

virbr0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet addr:192.168.122.1  Bcast:192.168.122.255

A bit hard to disable, but at the end ifconfig virbr0 down ; brctl delbr virbr0 helps to get rid of the weird bridge, and all domUs will start after that.

Going old-school today: I wrote a sed script to massage grub.conf to add a windows partition on a second disk. Searching google for has this been done before yields loads of page with handholding on how to add windows by hand to a grub.conf generated by anaconda but no simple 'automated' solution. I am always in favor of letting the computer do the boring work. But a bit of thinking and testing and now sed does the job:

if [ -b /dev/sdb1 ]; then

        cp /boot/grub/grub.conf /boot/grub/grub.conf.pre

        sed -e 's/timeout=5/timeout=30/' -e '/hiddenmenu/a\
title Windows XP (Service Pack 3)\
        rootnoverify (hd1,0)\
        map (hd0) (hd1)\
        map (hd1) (hd0)\
        makeactive\
        chainloader (hd1,0)+1
' -e '/hiddenmenu/d' < /boot/grub/grub.conf.pre > /boot/grub/grub.conf
fi

Everybody knows sed -e 's/../../' but I had to look up 'insert', 'append' and 'delete'.
Update 2009-11-12: Changed insert to append because the previous version inserted windows multiple times with multiple linux kernels. Once is enough. Also moved it from the post-install instructions to the post-reboot script so linux is fully configured before windows gets booted.

Mijn werkgever, het departement informatica van de Universiteit Utrecht, biedt ook een opleiding tot leraar informatica aan. Totnogtoe hebben ze daar nog niet zo veel reclame voor gemaakt, maar daar komt nu verandering in.

I noticed requests for port 37/udp in our firewall to our ntp server. That is the 'daytime' protocol which is absolutely ancient in an Internet timescale. I opened the port and started the service as an experiment and started tcpdump on it. The results are interesting:

09:50:09.749723 IP xx.xx.178.51.37 > 131.211.84.189.123: NTPv4 client, strat 2, poll 7, prec -20
09:50:09.749782 IP 131.211.84.189.123 > xx.xx.178.51.37: NTPv4 server, strat 2, poll 7, prec -19
09:52:19.808243 IP xx.xx.178.51.37 > 131.211.84.189.123: NTPv4 client, strat 3, poll 7, prec -20
09:52:19.808301 IP 131.211.84.189.123 > xx.xx.178.51.37: NTPv4 server, strat 2, poll 7, prec -19
09:53:08.511939 IP xx.xxx.183.183.34505 > 131.211.84.189.37: UDP, length: 0
09:53:08.513364 IP 131.211.84.189.37 > xx.xxx.183.183.34505: UDP, length: 4

Most traffic seen by 'tcpdump port 37' is from source port 37. Which is an artifact of certain NAT devices translating privileged ports (< 1024) to other privileged ports. Certain versions ntpd seem to ignore these requests. But there are real clients using the 'daytime' protocol.

In een discussie over 'durf je nu echt AAAA records te publiceren' vroeg ik me af of er een goeie, klantvriendelijke ipv6 test is voor websites met behulp van javascript. Natuurlijk is die er: http://ipv6test.max.nl/. Die heb ik dus snel geimplementeerd op 2 websites op het werk die nog geen ipv6 verbinding hebben maar waar we dat wel snel hopen: www.cs.uu.nl en helpdesk.cs.uu.nl. Hier komen 'onze' gebruikers langs dus is het erg interresant om te weten of in deze gebruikersgroep er een aandeel is wat problemen gaat krijgen als we AAAA records publiceren.

I brought some more USB sticks to test with and tested the filler script with 4 sticks. Interesting new problem: some USB sticks are partitioned like a harddisk and some aren't, now to find what to mount. Trying to mount everything gives a lot of kernel error messages. Using vol_id was the way to find the valid filesystems. The writing speed is still at maximum when I write 4 in parallel and no USB errors happen.

Ok, discovering 'all USB storage' is not that hard:

ALLSTICKS=`/bin/ls /dev/disk/by-path/*usb*part1 2>/dev/null`

Now for the choice whether to fill them in parallel or serially. With two sticks (the amount I have available at the moment for testing) running two rsync processes in parallel makes the whole script (discover, mount, fill with rsync, unmount) take 27 seconds, waiting for the first rsync to finish before starting the second one takes 35 seconds. Interesting will be how these numbers look when I add more USB sticks.

An interesting project at work: copying a given set of data to as big a number of USB storage devices as possible. So we buy 4 USB hubs, which got delivered today. Connecting them to the 4 different external USB ports on my laptop shows an interesting result:

 lsusb -t
Bus#  7
`-Dev#   1 Vendor 0x0000 Product 0x0000
  |-Dev#  35 Vendor 0x2001 Product 0xf103
  | `-Dev#  36 Vendor 0x0718 Product 0x0075
  |-Dev#  34 Vendor 0x2001 Product 0xf103
  |-Dev#  33 Vendor 0x2001 Product 0xf103
  `-Dev#  32 Vendor 0x2001 Product 0xf103
Bus#  6
`-Dev#   1 Vendor 0x0000 Product 0x0000
Bus#  5
`-Dev#   1 Vendor 0x0000 Product 0x0000
  `-Dev#  44 Vendor 0x0b97 Product 0x7761
    `-Dev#  45 Vendor 0x0b97 Product 0x7772
Bus#  4
`-Dev#   1 Vendor 0x0000 Product 0x0000
Bus#  3
`-Dev#   1 Vendor 0x0000 Product 0x0000
Bus#  2
`-Dev#   1 Vendor 0x0000 Product 0x0000
Bus#  1
`-Dev#   1 Vendor 0x0000 Product 0x0000
  `-Dev#  24 Vendor 0x413c Product 0x8140

Notice it? No? All the high-speed USB hubs (Vendor 0x2001 Product 0xf103) are behind the same root USB hub. Interesting USB congestion problems ahead probably.

My next step will be to discover all attached usb storage (probably thanking udev a lot in the process) and filling that storage with the wanted set of data.

Actual progress in windows printing: windows 2008 with office 2003 will give a warning message about trying to print an A4 document on a printer with Letter as default format. Fixed it by selecting the right papersize for the printer, no letter paper in this building. No PC LOAD LETTER on the printer display!