2021-07-08 Another panic in a virtual machine 2 weeks ago
At the end of this morning I noticed the root filesystem of the shell server on the homeserver had turned itself read-only. Another DRIVER_TIMEOUT error in the kernel messages. And I didn't want to get to a situation with half of the filesystem in lost+found like the previous time. This time I decided to use a different approach in the hopes of getting back to a working system faster. And they worked this time.
After things ran ok for a while I removed the snapshot. I also changed the configuration to use virtio disks and not ide emulation. Ide emulation disks have a timeout (DRIVER_TIMEOUT) after which things are given up. The fact that (emulated) I/O hangs for 30 seconds is bad, but maybe related to the rcu_sched messages. Maybe time for some more updates.
- echo s > /proc/sysrq-trigger to force a sync
- echo u > /proc/sysrq-trigger to force an unmount of all filesystems
- I killed the virtual machine with virsh destroy (the virtualization equivalent of pulling the plug)
- I created a snapshot of the virtual machine disk to make have a state of file system to return to in case of problems in the next steps
- I booted the virtual machine and it had indeed filesystem issues
- So reboot in maintainance mode and did a filesystem check
- After that it booted fine and the filesystem was fine, nothing in lost+found