News archive June 2023 - Koos van den Hout

While working with the other NFC tags I had a crazy idea: what if I can 'recycle' used one-time public transport tickets as NDEF tags. The one-time public transport tickets are mifare ultralight tags just like the touchatag tag.

In March 2015 we went on a ski trip to Geilo ski area in Norway. And I kept two ski cards. Time to analyze them with the proxmark3.

[usb] pm3 --> hf search 
 🕖  Searching for ISO15693 tag...            
[+]  UID: E0 16 24 66 07 50 CE 09
[+] TYPE: EM-Marin SA (Skidata); EM4233 [IC id = 09] 23,5pF CustomerID-102

[+] Valid ISO 15693 tag found

[usb] pm3 --> hf 15 info                      

[+]  UID: E0 16 24 66 07 50 CE 09
[+] TYPE: EM-Marin SA (Skidata); EM4233 [IC id = 09] 23,5pF CustomerID-102
[+] Using UID... E0 16 24 66 07 50 CE 09

[=] --- Tag Information ---------------------------
[=] -------------------------------------------------------------
[+]       TYPE: EM-Marin SA (Skidata); EM4233 [IC id = 09] 23,5pF CustomerID-102
[+]        UID: E0 16 24 66 07 50 CE 09
[+]    SYSINFO: 00 0F 09 CE 50 07 66 24 16 E0 02 00 33 03 02 
[+]      - DSFID supported        [0x02]
[+]      - AFI   supported        [0x00]
[+]      - IC reference supported [0x02]
[+]      - Tag provides info on memory layout (vendor dependent)
[+]            4 (or 3) bytes/blocks x 52 blocks

Error messages like this make me fix things fast:

Jun 24 13:42:59 conway kernel: [6925745.388604] sd 0:0:0:0: [sda] tag#6 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
Jun 24 13:42:59 conway kernel: [6925745.389388] sd 0:0:0:0: [sda] tag#6 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00
Jun 24 13:42:59 conway kernel: [6925745.390157] print_req_error: I/O error, dev sda, sector 616464
Jun 24 13:42:59 conway kernel: [6925745.390923] md: super_written gets error=10
Jun 24 13:42:59 conway kernel: [6925745.391705] md/raid1:md127: Disk failure on sda3, disabling device.
Jun 24 13:42:59 conway kernel: [6925745.391705] md/raid1:md127: Operation continuing on 1 devices.
Jun 24 13:42:59 conway mdadm[2559]: Fail event detected on md device /dev/md127, component device /dev/sda3

The part that makes me go 'hmmm' is that this was another Kingston A400 SSD, just like the one that failed in December 2021 for which I ordered a replacement from a different brand. Since that disk failed under warranty it was replaced with another Kingston A400 which I still had available in packaging. So that is now in use and the failed SSD is removed. I wonder how long that replacement disk will work fine.

I did all the bits to replace the disk and recreate the software raid mirror. This worked fine, and all my work to make sure the system can boot from either disk of the mirror worked.

I've been interested in RFID and RFID security for years, the first post on my website is from 2010: I found out this week that the rfid card my employer uses to give out coffee is also a mifare classic card. Since that moment I collected all kinds of contactless cards with the idea to check into their security. Mostly from our wintersport holidays since ski passes use rfid technology to make reading them on wintersport easy.

Now the time has come to check my collection with the proxmark3. The simple approach is to scan for tags with lf search or hf search.

Touchatag tags

In 2010 these were a great idea to put tags on products. These are Mifare Ultralight MF0ICU1:

[usb] pm3 --> hf mfu info 

[=] --- Tag Information --------------------------
[=] -------------------------------------------------------------
[+]       TYPE: MIFARE Ultralight (MF0ICU1)  
[+]        UID: 04 C8 54 19 3E 25 80 
[+]     UID[0]: 04, NXP Semiconductors Germany
[+]       BCC0: 10 ( ok )
[+]       BCC1: 82 ( ok )
[+]   Internal: 48 ( default )
[+]       Lock: FF 7F  - 1111111101111111
[+] OneTimePad: E1 10 06 00  - 11100001000100000000011000000000

Somewhere on irc the term "don't host your website on a wet newspaper" is sometimes used when an url getting a bit of serious traffic makes it really respond slow or give errors.

So I looked at my own webservers at home and what would happen if one of the sites got hit with the Slashdot Effect. As I don't like guessing I played with ab - Apache HTTP server benchmarking tool to get some idea of what happens under load and/or highly concurrent access.

Especially highly concurrent access turns out to be an issue because there are only so much database connections available for the webservers. The load average does go up, but the main problem is clients getting a database connection error.

I started looking at caching options to allow the dynamic pages to be cached for short periods. This would make high amounts of traffic have the advantages of having a cached version without losing the advantages of dynamic pages.

By now this has cost me more time and energy than the advantage of ever surviving a high amount of valid traffic. And to be honest the chances of a DDoS attack on my site because someone didn't like something I wrote is higher than the chances of a lot of people suddenly liking something I wrote.

This was all tested with the test and development servers, so actual production traffic was never affected by the tests.

Apache built-in memory cache with memcached

I first tried the Apache module socache_module with socache_memcache_module as backend. This did not cache the dynamic pages, just .css and other static files which originate from diskcache or ssd storage anyway. All kinds of fiddling with the caching headers did not make this work. With debugging enabled all I could see was that the dynamic pages coming from cgid or modperl were not a candidate for caching.

I could have used memcached from the web applications directly, but that would mean I would have to rewrite every script to handle caching. I was hoping to add the caching in a layer between the outside world and the web applications, so I can just detour the traffic via a caching proxy when needed.

Haproxy cache

Between the outside world and the webservers is a haproxy installation anyway, so I looked at that option. But the haproxy cache will not cache pages that have a Vary: header, but even after removing that header in Apache the next problem is that the Content-Length: http header has to be set in the answer from the webserver. With my current setup that header is missing in dynamic pages.

Varnish cache

Using varnish cache means I really have to 'detour' web traffic through another application before it goes on to the final webserver. This turned out to be the working combination. But this caused confusion as Varnish adds to the X-Forwarded-For header and I had an entire setup based on this header being added by haproxy listing the correct external IP address from the view of haproxy. It took a few tries and some reading to find the right incantation to specifically mangle back the X-Forwarded-For header to the right state in the outgoing request to the backend server. The varnish cache runs on the same virtual machine as the test haproxy, so the rule was to delete , ::1 from the header.

Tuning haproxy to avoid overloading a backend

In looking at things and testing I also found out haproxy has a maxconn parameter for backend servers, listing the maximum number of open connections to the backend. By changing this number to something lower than the maximum amount of database connections the site starts to respond slow under a high number of concurrent requests, but it keeps working and doesn't give database errors.

Jopen is a Dutch brewery from Haarlem and they offer a number of beers.

This time I ran into an IPA from Jopen and decided to give it a try. It's light in hop taste for an IPA, or my tastebuds are getting used to hops. Amber colour, good taste.

The beer details

Company	Jopen
Beer name	Mooie Nel North Sea IPA
Beer style	IPA - India Pale Ale
Alcohol by volume	6.5 %

Recently in Not Always Right: Atomic Clocks: Not Great, Not Terrible which reminded me a lot about my work with precision timekeeping at the computer science department of Utrecht University.

At least once a year a student would complain that the work was really turned in before the deadline and the clocks on our servers were probably wrong. The answer was always a complete explanation why the clocks were within microseconds correct compared to international standard time so the student was actually too late.

Oh and talking about 'atomic' clocks made me remember First Atomic Clock Wristwatch.

I noticed grafana hadn't updated in a while. Normally cron-apt does the prefetching of updates and notifies me when new updates are available, so I can make sure updating doesn't break running stuff or I can resolve it quick.

But cron-apt held an error message from apt update away from me which I saw by hand:

root@gosper:~# apt update
Get:1 https://packages.grafana.com/oss/deb stable InRelease [5,983 B]
Err:1 https://packages.grafana.com/oss/deb stable InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 9E439B102CF3C0C6
Get:2 http://deb.devuan.org/merged beowulf InRelease [33.5 kB]
Get:3 http://deb.devuan.org/merged beowulf-security InRelease [26.1 kB]
Get:4 http://deb.devuan.org/merged beowulf-updates InRelease [26.1 kB]
Fetched 85.7 kB in 3s (28.4 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 9E439B102CF3C0C6
W: Failed to fetch https://packages.grafana.com/oss/deb/dists/stable/InRelease  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 9E439B102CF3C0C6
W: Some index files failed to download. They have been ignored, or old ones used instead.

I partly followed the instructions in Problem with debian repository key - Grafana / Installation - Grafana Labs Community Forums to get things going again. I used /etc/apt/trusted.gpg.d because this is the standard directory, is already available and the remark about ubuntu means this is the only supported directory for gpg keys.

root@gosper:~# cd /etc/apt/trusted.gpg.d/
root@gosper:/etc/apt/trusted.gpg.d# wget -q -O - https://apt.grafana.com/gpg.key  | gpg --dearmor > grafana.gpg

By putting the grafana.gpg in this directory it gets detected and used automatically. No need for a pointer in /etc/apt/sources.list.d/grafana.list.

Now apt update doesn't complain, so I will be notified of new grafana versions available.

It's been a while since I played with rfid technology but recently some news around LF cards has made me interested again. The proxmark3 is the best device for going deep with rfid technology so I considered buying one.

Reading various sources about the availability of proxmark3 hardware taught me the latest and greatest version (currently Proxmark3 RDV4) has not a lot of advantages over the previous version (RDV3) which is available at seriously lower prices from several webshops. So I ordered one using aliexpress and the wait started.

Today the proxmark3 came in. I built the software for Linux using the guide at proxmark3 Linux Installation Instructions where I noticed I had to add packages libbz2-dev and gcc-arm-none-eabi by hand to get things to compile/build correctly.

After doing the firmware upgrade dance I started testing and looking around. The proxmark3 detects 125 kHz (LF) and 13.56 MHz (HF) cards fine. With the order came a blank card which is both a 13.56 MHz Mifare 1K with changeable UID and a 125 kHz T5577. There were also two small keyring tags, a mifare 1K and a mifare 4K.

First attempts

The proxmark3 shows information for all the cards I tried. To my surprise the skipass from our last ski trip to Austria was an HF only card, I thought ski passes used 125 kHz technology so they could be read through jackets or other layers more easily. It's an ISO 15693 tag and I can access all data easily.

[usb] pm3 --> hf search 
 🕗  Searching for ISO15693 tag...            
[+]  UID: E0 16 24 66 09 99 B3 70
[+] TYPE: EM-Marin SA (Skidata); EM4233 [IC id = 09] 23,5pF CustomerID-102

[+] Valid ISO 15693 tag found
[usb] pm3 --> hf 15 info

[+]  UID: E0 16 24 66 09 99 B3 70
[+] TYPE: EM-Marin SA (Skidata); EM4233 [IC id = 09] 23,5pF CustomerID-102
[+] Using UID... E0 16 24 66 09 99 B3 70

[=] --- Tag Information ---------------------------
[=] -------------------------------------------------------------
[+]       TYPE: EM-Marin SA (Skidata); EM4233 [IC id = 09] 23,5pF CustomerID-102
[+]        UID: E0 16 24 66 09 99 B3 70
[+]    SYSINFO: 00 0F 70 B3 99 09 66 24 16 E0 02 00 33 03 02 
[+]      - DSFID supported        [0x02]
[+]      - AFI   supported        [0x00]
[+]      - IC reference supported [0x02]
[+]      - Tag provides info on memory layout (vendor dependent)
[+]            4 (or 3) bytes/blocks x 52 blocks

As all the tag readers in that ski area are on-line anyway, I guess the card is just a big serial number and all the checking whether the user isn't trying to do something that wasn't paid for is done in central computers.

First error

While trying to clone an LF card into the T5577 I managed to make the T5577 card end up in a weird state: it now only returns 0x0000 or 0xFFFF patterns on read depending on the communication configuration.