Weblog entry #284 for simonw
#284
Bad day for the server and the weather
Posted by simonw on Thu 30 Oct 2008 at 01:47
Replaced a disk in DELL PERC array just before lunch, and when I tried to bring the disk back into the array the server immediately started reporting file system corruption.
6 hours to rebuild the array, by 9 hours(!) most of this boxes services were restored. There is clearly one crucial element still wrong as a key in-house application is returning corrupt HTTP responses some of the time, and the remains of this applications data are still being restored 14 hours after the problem occurred.
Software RAID never looked better.
The drive home was illuminated by a spectacular thunderstorm to put it into perspective. Many people in east Devon are going to have a lot of clearing up to do in the morning. I've never seen the village this flooded before, but only one detour needed to get home (just!).
6 hours to rebuild the array, by 9 hours(!) most of this boxes services were restored. There is clearly one crucial element still wrong as a key in-house application is returning corrupt HTTP responses some of the time, and the remains of this applications data are still being restored 14 hours after the problem occurred.
Software RAID never looked better.
The drive home was illuminated by a spectacular thunderstorm to put it into perspective. Many people in east Devon are going to have a lot of clearing up to do in the morning. I've never seen the village this flooded before, but only one detour needed to get home (just!).
Comments on this Entry
I have some Dell 2850s with (known) bad firmware, wich makes the RAID controller crash when trying to rebuild your RAID5 array. And you're not supposed to upgrade the firmware when the RAID array is not complete. Yay.
[ Parent | Reply to this comment ]