• BlogNav:
• Archive of all posts

Insane/Brilliant Idea of the Day

Posted: Fri, 24 April 2009 | permalink | No comments

I've been talking serial consoles with a couple of the other guys at work: how nice they are to have for machines in the datacenter, how annoying it is that vPro serial-over-LAN doesn't seem to be robust (yet?), and how serial access concentrators are lung-and-kidney expensive (especially when you've got 50-some racks to outfit).

This discussion, combined with my ongoing embedded-hardware-fascination lust for a SheevaPlug appears to have spurred my brain into coming up with a Brilliant Idea: tie a SheevaPlug to a pile of USB to serial adapters and use that as your per-rack serial concentrator. Imagine: faffenheimer, a dedicated server you manage for a customer, and located in rack 27 of your DC, has just crashed, and you'd like to know WTF has happened rather than just blindly reboot, but you're in the office 15 minutes away from the DC floor, and the customer's going to want that machine back up and running pretty quickly.

workstation:~/porn$ ssh rack27.serial
rack27:~$ sconsole faffenheimer
[screen session attached]
[minicom running, shows the horror of a kernel crash dump]
[oh look at that, kernel bug]
^A ESC
[pgup pgup]
[enter]
[pgdn pgdn]
[enter]
^A >/tmp/faffenheimer-crash-dump
^A d
rack27:~$ exit
workstation:~/porn$ scp rack27.serial:/tmp/faffenheimer-crash-dump ~
workstation:~/porn$ powercycle rack27 faffenheimer

Shiny! We got a crash dump in a minute or so (rather than having to take phonecam photos of KVM screens in the DC), never had to leave our comfy seat, and the machine's on it's way back up. We're now free to pursue diagnostic activities on that crash dump at our leisure.

10 minutes later, the downtime for faffenheimer that was automatically set when we ran powercycle runs out and Nagios sends us threatening messages. Hmm, something's gone wrong here. Back into the console...

workstation:~/porn$ ssh rack27.serial
rack27:~$ sconsole faffenheimer
[screen session attached]
[Boot is hung waiting for root password after initrd has bombed]
[Type root password]
[Oh look, the root MD appears to have come asunder]
[clickety-click... fixee fixee]
[reboot]

The more I think about this, the more I reckon I'm onto a bona fide winner. The sheevaplug is a powerful ARM-based system with USB/ethernet/SD ports that is packaged literally in it's own power supply wall wart -- it's a plastic box with power plug prongs poking out the side. That's all there is to it. The USB to serial adapter things are likely to be a bit more of a pain, but I've played with enough of them by now to not be too scared. So, you plug the Sheeva into a power socket, plug an Ethernet cable and USB hub into the Sheeva, configure things a bit so that the system knows which serial adapter maps to which machine, and you're away. Oh, and the best bit: the Sheeva apparently draws as little as 2W when idle. A whole datacentre's worth of serial goodness for about a server's worth of power. The cost per rack should be somewhere below AU$250, especially in bulk.

Let's see if I can convince work to spring for a Sheeva, a USB hub, and a half dozen or so USB to serial adapters to test this whole thing out. Given that the whole thing looks like it'd cost less than AU$250 (plus my R&D time), I can't imagine it'll be too hard a sell to at least give it a go. Watch this space...