bleep bloop

Fah. Mondays suck. Particularly when they are after a week long vacation. Egad is it hard to wake up at 0630 after shifting to some European timezone. It didn’t help much that the day has been crap overall. I had 1.2 billion emails waiting for me, and two meetings. Sigh. Did I mention I hate meetings? I think I did. If not, let me mention that I hate meetings.

And then Zidane fell over and hardlocked around lunchtime. He’s been doing this about once a month since we got him, and Brian and I are pretty much stumped at this point. I’m thinking memory, he’s thinking hard disk, but we’re both guessing. What’s most frustrating is that there’s nothing in the logs to help diagnose the problem. We even started logging the output of ‘ps aux’ to a file every minute to see if there was some sort of runaway process or something that was breaking things. Nope. It just STOPS.

Sigh. Smart checks on the disk report no problems. Things that normally go to shit with bad memory, like kernel compiles, work perfectly. The only other clue we have is ocassionally tar will report a corrupt tarball that isn’t really. One ‘tar xjf’ will fail, and then a second will succeed on the same exact file. That makes me think memory, but who knows. I suppose we should ask Rackshack to run the ram through a checker. If that works then we’ll have to bite the bullet and try a new disk, which means migrating everything again.

At least we have most of it scripted now after the Rocket to Zidane weekend marathon move. And of course our monitor that notifies us when Zidane’s being cranky had silently failed a few days back and so I didn’t find out about it until I tried to check my mail. We had about 40 minutes of downtime (from 1154 to 1238 EST) instead of the usual 5 minutes. So I spent most of my lunch trying to figure that out.

Goddamn thing was blowing up on a config file that was exactly the same as the goddamn example in the goddamn documentation. I decided to use the brute force method of software repair and commented the stanza out. Yay for violence. I hate computers.