Hate computers. Hate them lots.
Around 5:30 am yesterday, Gehenna (the machine that contains this weblog, as well as a half dozen other services) fell over dead. Unfortunately, yesterday morning was a dentist morning, so by the time I woke up it was too late to do anything about it until I finished with the dentist's appointment (at 11:30) and could go into the co-lo to see what happened. When I got into the co-lo, gehenna was well and thoroughly dead, to the point where all I saw on the console was a little bit of "hi! I'm a dead vga!" scramble at the top of the screen.
I hit the Big Red Switch (on this machine, the Big Red Switch is the power cord) and gehenna chirped happily back to life, with no indication that anything had gone wrong except for, um, having to have me reboot it.
I spent part of yesterday afternoon researching replacement hardware and pricing PC deadmans switches, because if the machine was going to fall over and leave nothing but dead vga scramble, that meant that something was seriously wrong with the hardware. The deadmans switches were extra paranoia; as the hardware in Pell and Gehenna gets older, there are more and more (at convenient 100 day intervals, so the people at the co-lo won't completely forget who I am) mysterious failures that are "fixed" by a therapeutic whack at the reset switch, and I'd rather have this whack happen without having to go downtown and back.
Last night, at approximately midnight, Gehenna fell over dead again. I was preparing to go to sleep, so there was no way I'd throw on my clothes and scuttle down to the co-lo. So I left it until this morning, when I packed up a new motherboard (scavenged from the Glass PC) and a new power supply and shlepped off to the co-lo before going into work.
My plan was to just replace the motherboard and scuttle back to work. Shouldn't take more than 10 minutes, right?
Well, no. It took about 10 minutes to take the old parts out and put the new parts in, but then it took another 40 minutes of fighting with the kvm switch before I realized that it was dead (but dead in that sort of charming way that kvm switches die; it was relaying vga signals to the monitor, but other than that it was merely a quietly humming doorstop) and that I'd need to remove it before I could actually talk to Gehenna and get it to boot properly.
So it's dimly possible that Gehenna was "fine", but was just being killed by debris from a disintegrating kvm switch (it's happened before; I've got some systems at my day job which will not boot if their keyboards are connected to a kvm switch) and all I needed to do was to pull the horrible thing out and everything would work perfectly. But by the point I'd realized this I'd already swapped out everything except the disks, and I didn't want to unswap anything for fear it would die AGAIN and I'd have to do another run for the co-lo.
And the ME-6000 motherboard from the Glass PC has a bios option to restart after a power failure, so I can ask the people at the co-lo if they could pull the power cord before running down the next time (oh yes, there will be a next time. Even if I put in a peer to peer failover mechanism, there will still come a time when that code will fail and the bottom will fall out) to replace hardware or kernel.
<obsigh>And did I mention that today is the day I'm switching network providers to Qwest, and they started the day by, um, disconnecting the network without bothering to mail me any instructions about how to connect to the Qwest routers? *sigh* </obsigh>