Another panic in a virtual machine / 2021-07-08

At the end of this morning I noticed the root filesystem of the shell server on the homeserver had turned itself read-only. Another DRIVER_TIMEOUT error in the kernel messages. And I didn't want to get to a situation with half of the filesystem in lost+found like the previous time.

This time I decided to use a different approach in the hopes of getting back to a working system faster. And they worked this time.

1. echo s > /proc/sysrq-trigger to force a sync
2. echo u > /proc/sysrq-trigger to force an unmount of all filesystems
3. I killed the virtual machine with virsh destroy (the virtualization equivalent of pulling the plug)
4. I created a snapshot of the virtual machine disk to make have a state of file system to return to in case of problems in the next steps
5. I booted the virtual machine and it had indeed filesystem issues
6. So reboot in maintainance mode and did a filesystem check
7. After that it booted fine and the filesystem was fine, nothing in lost+found

After things ran ok for a while I removed the snapshot. I also changed the configuration to use virtio disks and not ide emulation. Ide emulation disks have a timeout (DRIVER_TIMEOUT) after which things are given up. The fact that (emulated) I/O hangs for 30 seconds is bad, but maybe related to the rcu_sched messages. Maybe time for some more updates.