2021-07-26 MicroSD failure in a Raspberry Pi 1 month ago
The Raspberry Pi in the attic running mainly dump1090 and some other software wasn't showing up in the system monitoring. On checking it turned out the MicroSD card was failing. This is a known issue in the Raspberry Pi which uses MicroSD as root filesystem. For as far as I can tell this card has been running continuously since February 2016, so over five years. I do have a different MicroSD card which is the old card from the Raspberry Pi in the utility closet which became available after I used a different card to reinstall the Raspberry Pi for smart meter monitoring. But that card has seen some wear since it has been running since installing the smart meter and starting energy monitoring on it in August 2016 so maybe it's not a good idea to rescue a system with it, it's also five years old. Time to order some new MicroSD cards! In the mean time I noticed I could get a bit of access to the broken card, but things stopped on mounting the linux root filesystem. It turned out that mount tries to write to the card to update the ext4 journal and the card stops completely on a write. When I mount it really readonly with# mount -o ro,noload /dev/sdb2 /mnt/scratch/almost all files are readable, so I recovered the dump1090 software and other configuration items. Yes, I need to add the Raspberry Pi systems to the backups. It would be really nice if I could monitor the health of the MicroSD card like I monitor other disks (including SSD) with smartmontools. Update: Three MicroSD cards ordered so I can replace this one and have a few spares ready. The size of those cards does mean I now have to make small bags labeled 'spare' or 'old card from system X' so I can see what they are without trying to mount them.