News archive September 2021 - Koos van den Hout

The 'moment of truth' for LetsEncrypt: the end of the validity of the root certificate that was used to kickstart LetsEncrypt before they got their own root certificate in (most) certificate stores.

I notice openssl is still showing the old chain (but not the expired intermediate):

---
Certificate chain
 0 s:CN = koos.idefix.net
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---

Which is interesting as the ISRG Root X1 is also in the root store. But it's also cross-signed to the DST Root CA.

Checking the verification steps (and not the chain as given out by the server) gives the new path already:

depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = koos.idefix.net
verify return:1

This is a subtle but important difference.

Only hours left until the DST Root expires:

$ openssl x509 -in DST_Root_CA_X3.crt -noout -enddate
notAfter=Sep 30 14:01:15 2021 GMT

If services break after 14:01:15 GMT (UTC) today you're not working according to best practices (replacing the certificate chain with every certificate replacement) or you have old clients.

Slight update: I requested a new LetsEncrypt certificate for a service after 14:01:15 GMT (UTC) and it still has the certificate chain with cross-certification to DST Root CA X3:

---
Certificate chain
 0 s:CN = koos.idefix.net
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---

The verification steps are as above.

Het nieuws over de woningmarkt in nederland is blijkbaar ook opgepikt door de fraudeurs die graag mensen geld afhankelijk maken.

Onderstaande uit de inkomende berichten. Keurig alle kenmerken van een poging fraude: urgentie 'binnen 2 weken' en vervelende consequenties als de gebruiker niets doet 'Uw opgebouwde inschrijfduur en eventuele reacties vervallen dan'.

Allemaal vooral tekenen van een poging tot oplichting. Trap hier niet in. Meer informatie bij Diverse onderwepen Klik op “Lees meer” - fraudehelpdesk.nl.

Geachte klant,

Uw inschrijving bij WoningNet

Over 2 weken verloopt uw inschrijving. Uw inschrijving verlengen wij met een
jaar als u de verlenginskosten van

WoningNet-verlengingskosten                                                   

Factuurnummer: SRA-00717113

Wilt u betalen via een doorlopende machtiging? dit kan, nadat u dit jaar nog
via iDEAL betaald heeft. U kunt na de betaling bij overzicht van mijn gegevens
uw betaalwijze veranderen.

Volgend jaar zullen wij dan de verlengingskosten van uw rekening incasseren.
Ook hiervan krijgt u via e-mail een bericht. Meer informatie over betalen van
de verlengingskosten vindt u op onze website.

Uitschrijven

Als u niet binnen 2 weken betaalt, schrijven wij u uit. Uw opgebouwde
inschrijfduur en eventuele reacties vervallen dan.

De url voor de 'WoningNet-verlengingskosten' :

hxxps://t.co/1nuIzIn4KV?amp=1

t.co is de url-verkorter van twitter.

Gaat door naar:

hxxps://0x1.co/WE8FY

Een andere url-verkorter.

Gaat door naar:

hxxps://0mgeving-online.ga/

;; ANSWER SECTION:
0mgeving-online.ga.     3511    IN      A       195.133.40.102

En die reageert op dit moment niet. Opvallend: Er is ook (nog) geen certificaat voor bekend in de CT-logs.

Update: Aparte vermelding nu bij fraudehelpdesk.nl met voorbeeld: Lidmaatschap behouden - Fraudehelpdesk.

Today I ran into an issue related to systemd and I decided to try to fix it without too much cursing. The result was a number of google searches ending up on unix.stackexchange.com but eventually I fixed the problem.

At work we use splunk for security monitoring and one of the indexers failed to start the splunk processes after a reboot. On browsing the systemd boot log with journalctl -b -l I discovered that the main issue was that creating files in /opt/splunk failed. This was due to an interesting race condition: splunk may start as soon as target network.target has been reached, but mounting /opt over iscsi also needs network.target to start. So the unit file has been updated to:

[Unit]
Description=Systemd service file for Splunk, generated by 'splunk enable boot-start'
After=network.target opt.mount

The next problem was the systemctl start Splunkd.service failing in some intricate way. I had a look at the logging and saw that it was actually trying to restart the service and failed at killing one of the old processes. It turned out the /opt/splunk/var/run/splunk/splunkd.pid file had old contents and one of the PIDs in that file was now in use by a kernel thread. Those you can't kill, the restart failed and therefore the service did not start at all. Solution: remove the .pid file.

Past weekend was the 2021 version of the CQ World Wide RTTY DX Contest and I participated. I had time to participate in a few intervals during the weekend and it helped that it was a 48-hour contest so I could get some more contacts in the log Sunday evening.

In the end I made 151 contacts, a really nice number.

Update 2022-09-28: I saw I never updated this post with the results: 151 valid contacts, 2 states, 17 zones, 52 countries. One of the rankings: #57 The Netherlands.

In the phishing mail today:

   -------- Original Message --------

   Subject: Mail delivery failed: returning message to sender

   From: Mail Delivery System
   Date: 9/23/2021 7:09:49 p.m.

   To: koos@[..]



   This message was created automatically by mail delivery software.



   Some recent messages that you sent could not be delivered to one or
   more of its recipients.
   This is a temporary error tha can be corrected.

   Reporting-MTA: dns;[1]click to retry delivery
   Action: failed
   Status: Queued on server

The 'click to retry' was a link to a phishing site. Nicely copied from a standard mailer error message. Too bad the phishing site doesn't work!

I haven't written about new countries I had contacts with via amateur radio for a while, but today I finally got New Zealand in the log, which has been on my wishlist for a while. Indeed all countries that I don't have in the log yet are on that wishlist, but ranked according to Clublog New Zealand should be the 'easiest' for me.

This morning I had a contact using the FT8 mode with ZL4AS on the 40 meter amateur band. This means a contact over a distance of 18726 kilometers. And by the time I write this the contact is already confirmed via Logbook of The World.

The previous new country was Taiwan, last Wednesday using FT8 on the 20 meter band. That contact was with amateur station BU2FF who also confirmed very fast.

And a while ago I had a contact with a station in Trinidad & Tobago but I haven't seen confirmation yet.

So I am still going strong with contacts on amateur radio, sometimes finding that rare opportunity for a new country.

I always noticed that I had to plug in the USB cable for the remote radio with the radio switched off, otherwise the Kenwood TS480 would switch into transmit mode and stay there until I powered the radio off.

Annoying, and I thought it was something in the serial initialization. Recently I was thinking about this and remembered something about query sequences on serial devices triggering weird behaviour in other devices. From what I read about the Kenwood serial protocol the chance of a few stray characters changing something in the radio is quite possible.

So I considered what Linux software could do a query as soon as a serial port is added to the system. Well, modemmanager was the ideal candidate for this:

Package: modemmanager
[..]
Description-en: D-Bus service for managing modems
 ModemManager is a DBus-activated daemon which controls mobile broadband
 (2G/3G/4G) devices and connections. Whether built-in devices, USB dongles,
 Bluetooth-paired telephones or professional RS232/USB devices with external
 power supplies, ModemManager is able to prepare and configure the modems and
 setup connections with them.

And indeed, simply removing modemmanager made the problem go away. I can now plug in the USB cable when the radio is on and nothing happens.

After starting the collection of a lot of the system data I wanted with telegraf/influxdb/grafana one small part was missing: the system temperature sensors. I like these, so I had a look and found the inputs.temp plugin in telegraf which is normally disabled.

Enabling it on hosts that have actual hardware to measure worked ok. On the Raspberry Pi systems it gives one temperature:

> SHOW TAG VALUES ON "telegraf" WITH key="sensor" WHERE host='joy'
name: temp
key    value
---    -----
sensor cpu_thermal_input

On the home server conway it gives quite a lot of temperatures:

> SHOW TAG VALUES ON "telegraf" WITH key="sensor" WHERE host='conway'
name: temp
key    value
---    -----
sensor coretemp_core0_crit
sensor coretemp_core0_critalarm
sensor coretemp_core0_input
sensor coretemp_core0_max
sensor coretemp_core1_crit
sensor coretemp_core1_critalarm
sensor coretemp_core1_input
sensor coretemp_core1_max
sensor coretemp_core2_crit
sensor coretemp_core2_critalarm
sensor coretemp_core2_input
sensor coretemp_core2_max
sensor coretemp_core3_crit
sensor coretemp_core3_critalarm
sensor coretemp_core3_input
sensor coretemp_core3_max
sensor coretemp_core4_crit
sensor coretemp_core4_critalarm
sensor coretemp_core4_input
sensor coretemp_core4_max
sensor coretemp_core5_crit
sensor coretemp_core5_critalarm
sensor coretemp_core5_input
sensor coretemp_core5_max
sensor coretemp_physicalid0_crit
sensor coretemp_physicalid0_critalarm
sensor coretemp_physicalid0_input
sensor coretemp_physicalid0_max

For the dashboard showing all relevant temperatures for a system this is a bit overkill and makes the dashboard hard to read. Solution: go for all the temperature sensors that end in 'input', with the variable in the dashboard defined as 'ending in input':

> SHOW TAG VALUES ON "telegraf"  WITH key="sensor" WHERE host='conway' AND sensor=~/input$/
name: temp
key    value
---    -----
sensor coretemp_core0_input
sensor coretemp_core1_input
sensor coretemp_core2_input
sensor coretemp_core3_input
sensor coretemp_core4_input
sensor coretemp_core5_input
sensor coretemp_physicalid0_input

So far this works with all physical systems.

Grafana host dashboard with telegraf data including entropy. The dip in entropy is caused by the dnssec-signzone process

I have been collecting certain system data for ages with rrdtool, but now I see what is possible with Telegraf collecting agent and after some initial attempts I'm all in favour and data is flowing.

All the data I collected is already standard in telegraf, including entropy! Other data is also collected that is good to keep an eye on for performance.

I made some tweaks to the standard telegraf configuration: collect every 5 minutes, not exactly on the clock since I read The mystery of load average spikes which reminded me of my own experience Be very careful of what you measure. I also avoid gathering data on nfs filesystems (which come and go thanks to autofs).

I rolled out telegraf over all systems at home, and now there is a nice 'System info' dashboard in Grafana.

I continued my slow migration of statistics to Influxdb/Grafana and added the network traffic. I've been gathering this for ages in rrdtool, my earlier view was that I've been using rrdtool for network and other statistics since October 2002 so it is a bit of a change.

I updated the perl scripts that fetch network traffic statistics over SNMP to also add the data to influxdb. And it was simple to create a dashboard with that data. The overview pages with data for all interfaces for one measured host also link to detail pages per interface which also show the number of errors.

I'm personally not a huge fan of wildcard TLS certificates (risks with reuse of the private key) so I didn't try those yet, but based on my experiences with certificates with multiple names with zerossl I got a response: Stephen Harris on Twitter: Do they support wildcards and I just had to try. And it works! I requested a certificate:

        Requested Extensions:
            X509v3 Subject Alternative Name:
                DNS:gosper.idefix.net, DNS:*.gosper.idefix.net

And indeed it worked:

        Issuer: C = AT, O = ZeroSSL, CN = ZeroSSL ECC Domain Secure Site CA
        Validity
            Not Before: Sep  1 00:00:00 2021 GMT
            Not After : Nov 30 23:59:59 2021 GMT
        Subject: CN = gosper.idefix.net
[..]
            X509v3 Subject Alternative Name: 
                DNS:gosper.idefix.net, DNS:*.gosper.idefix.net

So that works too! The choice for gosper.idefix.net is because I already had dns records setup for dns-01 based verification of that name.