Power failure this morning at work.. whi ... / 2009-11-16

Power failure this morning at work.. which left us not in the dark (enough emergency lighting) but with a completely silent serverroom. When the power came back we had some hours of work to get everything up and running again. Worst problem was with a number of Xen based virtualhosts, some centos upgrade had suddenly created a network device virbr0 which uses NAT and a local dhcp pool and enslaved all xen domU network interfaces under that bridge with no access to the 'real' network because NAT was not set up so their NFS root mount failed. The details on virbr0:

virbr0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet addr:192.168.122.1  Bcast:192.168.122.255

A bit hard to disable, but at the end ifconfig virbr0 down ; brctl delbr virbr0 helps to get rid of the weird bridge, and all domUs will start after that.