Introduction to Cereal - the serial line management tool
Posted by drgraefy on Mon 15 Oct 2007 at 13:01
Many people view serial ports as antiquated, out-dated connectors taking up space on their computers. However, serial ports still offer one of the best ways to communicate reliably and simply with a machine. For example, a serial port can be configured to act as a full-featured system console. This article describes how you can use cereal to monitor, log, and control access to serial lines connected to the consoles of other computers.
A computer today has several subsystems that should be run over the serial port to make it a full-featured system console:
- most modern operating systems (including the GNU/Linux utilities favored by Debian), are able to provide login terminals on serial ports,
- most modern kernels (including Linux) are capable of directing their console I/O to a serial line,
- most bootloaders (including stalwarts like GRUB, the SYSLINUX family, and LILO) can communicate via serial ports,
- and many BIOSs, including most modern server BIOSs, can be redirected over the serial port.
Using a null-modem cable, you can connect the primary serial port on one computer to a remote computer's spare serial port. The remote computer can have full access to the first machine's console using only a terminal emulator like GNU screen or minicom. Not only do you not need yet another cumbersome monitor and keyboard, but you get a digital text stream to manipulate directly. You can log, timestamp, copy, paste, diff, etc the entire data stream flowing over the console, including boot-time spew, kernel crash output, etc.
Using serial ports as system consoles is especially useful in
high-density computing environments, such as racks in data centers.
Clutter like monitors, vga cables, video cards, keyboards, keyboard
cables, and kvm switches can all be thrown out in favor of serial
cables and a single computer that we will refer to as a serial
console server. A serial console server is a machine with many serial ports,
each connected to the serial console of another machine. An
administrator accesses the serial console server over the network
(e.g. via ssh) and from there can connect to the console of any of the
connected machines. In all examples below, scs
is the
hostname of a serial console server.
This is where cereal comes in. cereal is a management framework designed to monitor, log, and control access to serial lines. It is installed on the serial console server, and makes the task of maintaining such a machine simpler. It can also be used in simpler configurations, such as two machines which each monitor the other's console. This article describes how to setup and use cereal. We assume that the computers you are connecting to are already configured to send their consoles over the serial line. If that's not the case yet, read The Remote Serial Console HOWTO and get 'em configured first!
The cereal utility is not yet available in Debian proper, but it is packaged and available from the CMRG APT repository. Once you've added this repository to your sources.list (you may want to update your apt keys if you care to about verifying packages from the repo), you can pull it with apt-get or aptitude:
scs:0:~# echo deb http://cmrg.fifthhorseman.net/debian unstable cereal >> /etc/apt/sources.list.d/cereal.list scs:0:~# aptitude update scs:0:~# aptitude install cereal
Administering cereal sessions: /usr/sbin/cereal-admin
Serial line management with cereal is broken out into sessions. A cereal session is a single runit service controlling a special screen session attached to a single serial line. Creation and control of these sessions is done with the cereal-admin utility:
scs:0:~# cereal-admin help Usage: cereal-admin[options] [args] Cereal session management program. subcommands: create (c) SESSION TTY BAUD USER LOGGROUP create cereal session start (s) [options] SESSION [SESSION]... start cereal session(s) -a (--all) start all sessions stop (k) [options] SESSION [SESSION]... stop cereal session(s) -a (--all) stop all sessions restart (r) [options] SESSION [SESSION]... restart cereal session(s) -a (--all) restart all sessions -r (--running) restart any currently running sessions destroy (d) [options] SESSION [SESSION]... destroy cereal session(s) -a (--all) destroy all sessions list (l) [SESSION]... list session(s) help (h,?) this help scs:0:~#
Creating a cereal session
Let's assume the first computer whose serial console you want to
monitor with cereal is called lemur
. Let's assume
lemur's serial console is configured with baud rate 115200,
and is connected to the serial console server on /dev/ttyS1
(/dev/ttyS0 on scs is of course reserved for the
serial console server's own serial console!).
You can give access to cereal sessions, and the ability to read the session logs, to existing users. However, it is often a good idea to create new users and groups for these purposes. We begin by creating a new group that will be allowed to view ('follow') lemur's serial console logs, followed by creating a new user that will be able to control ('attach' to) lemur's cereal session.
scs:0:~# addgroup lemurviewers Adding group `lemurviewers' (GID 1001) ... Done. scs:0:~# adduser --ingroup lemurviewers --gecos 'lemur cereal session admin,,,' lemuradmin Adding user `lemuradmin' ... Adding new user `lemuradmin' (1001) with group `lemurviewers' ... Creating home directory `/home/lemuradmin' ... Copying files from `/etc/skel' ... scs:0:~#
You are now ready to create lemur's cereal session with cereal-admin:
scs:0:~# cereal-admin create lemur /dev/ttyS1 115200 lemuradmin lemurviewers Created session 'lemur': --f lemur /dev/ttyS1 115200 lemuradmin lemurviewers scs:0:~#
The first three characters of the session status line are the session flags that indicate the state of the cereal session. In order, they are:
- state
- The first flag indicates whether the session is running (+) or not (-), or in some unusual state (!).
- attach
- The second flag indicates whether the user can attach to the session (a) or not (-).
- follow
- The third flag indicates whether the user can follow the session (f) or not (-).
Once a cereal session is created, you can begin monitoring
of it with the cereal-admin start
command:
scs:0:~# cereal-admin list lemur --f lemur /dev/ttyS1 115200 lemuradmin lemurviewers scs:0:~# cereal-admin start lemur Started session 'lemur'. scs:0:~# cereal-admin list lemur +-f lemur /dev/ttyS1 115200 lemuradmin lemurviewers scs:0:~#
Accessing cereal sessions: /usr/bin/cereal
Now that you have a cereal session up and running, you can test the connection. Log into scs as the non-privileged lemuradmin user and use the cereal tool to attach to or follow the session:
lemuradmin@scs:~$ cereal help Usage: cereal[options] [args] Cereal client program. subcommands: attach (a) SESSION attach to session follow (f) [options] SESSION follow session log -c (--cat) cat log instead of follow -p (--path) output just log file path list (l) [SESSION]... list session(s) help (h,?) this help lemuradmin@scs:~$ cereal list lemur +af lemur /dev/ttyS1 115200 lemuradmin lemurviewers lemuradmin@scs:~$
Attaching to a cereal session
Using the attach function will put the the user into a cereal console, provided by a command-restricted screen session. To detach from an attached session, use "C-\ d". For help on other available commands, use "C-\ ?".
lemuradmin@scs:~$ cereal attach lemur
Debian GNU/Linux 4.0 lemur ttyS0 lemur login: root Password: Last login: Sat Oct 6 10:16:26 2007 on ttyS0 Linux lemur 2.6.18-5-amd64 #1 SMP Tue Oct 2 20:37:02 UTC 2007 x86_64 The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. lemur:0:~# exit cereal console: lemur 2007-10-06 10:17 "C-\ d" to detach
lemuradmin@scs:~$ cereal attach lemur [detached] lemuradmin@scs:~$
Following a cereal session
You can use the follow command to easily view the logs of a cereal session.
lemuradmin@scs:~$ cereal follow --cat lemur | cat -A ############################################################ ### cereal log: lemur ### +af lemur /dev/ttyS1 115200 lemuradmin lemurviewers ### log file: /var/lib/cereal/sessions/lemur/log/main/current ############################################################ 2007-10-06_14:39:44.56895 2007-10-06_14:39:44.56900 cereal: session 'lemur' started. 2007-10-06_14:39:44.56900 starting screen session... 2007-10-06_14:39:58.86707 2007-10-06_14:39:58.86708 cereal: user 'lemuradmin' attaching to session... 2007-10-06_14:40:02.57346 2007-10-06_14:40:02.57348 Debian GNU/Linux 4.0 lemur ttyS0 2007-10-06_14:40:02.57348 2007-10-06_14:40:02.57349 lemur login: root 2007-10-06_14:40:13.58131 Password: 2007-10-06_14:40:14.58129 Last login: Sat Oct 6 10:16:48 2007 on ttyS0 2007-10-06_14:40:14.58129 Linux lemur 2.6.18-5-amd64 #1 SMP Tue Oct 2 20:37:02 UTC 2007 x86_64 2007-10-06_14:40:14.58130 2007-10-06_14:40:14.58131 The programs included with the Debian GNU/Linux system are free software; 2007-10-06_14:40:14.58131 the exact distribution terms for each program are described in the 2007-10-06_14:40:14.58132 individual files in /usr/share/doc/*/copyright. 2007-10-06_14:40:14.58132 2007-10-06_14:40:14.58133 Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent 2007-10-06_14:40:14.58134 permitted by applicable law. 2007-10-06_14:40:14.58135 Last was Sat 06 Oct 2007 10:33:15 AM EDT on ttyS0. 2007-10-06_14:40:14.58135 ESC[31;1mlemur:0:~#ESC[0m exit 2007-10-06_14:40:17.58525 logout 2007-10-06_14:40:17.58525 ESC[HESC[J 2007-10-06_14:40:18.58523 Debian GNU/Linux 4.0 lemur ttyS0 2007-10-06_14:40:18.58523 2007-10-06_14:40:18.58524 lemur login: 2007-10-06_14:40:19.62101 cereal: user 'lemuradmin' detached from session. lemuradmin@scs:~$
Conclusion
Now that you've created your first cereal session, you're ready to create serial sessions for all the rest of the attached serial consoles:
scs:0:~# cereal-admin list +-f lemur /dev/ttyS1 115200 lemuradmin lemurviewers +-f chimp /dev/ttyS2 115200 chimpadmin chimpviewers +-f douc /dev/ttyS3 115200 doucadmin doucviewers +-f galago /dev/ttyS4 115200 galagoadmin galagoviewers +-f baboon /dev/ttyS5 115200 baboonadmin baboonviewers ... scs:0:~#
Cereal is developed and maintained by CMRG. Hopefully the package will be included in Debian soon, at which point bug tracking will be handled in the Debian BTS. Until then, please sent all bug reports, questions, and suggestions to cereal AT fifthhorseman.net.
Serial links are also still used often in HA situations for heart beats.
Laptops with serial ports are also handy when at the DC
[ Parent | Reply to this comment ]
If your aim is to provide "emergency" serial port access to e.g. customers' co-located machines, a web interface might be a nice feature (especially if you already have a web interface for your power switch, or remote KVM or whatever). In that case, you could install Anyterm, which works well with serial ports. Disclaimer: I wrote it. http://anyterm.org/
[ Parent | Reply to this comment ]
As far as anyterm goes, I think it's an interesting concept, but I would be extremely wary of putting a system console on the web. The web is far too fragile in my opinion for mission critical stuff like this. I would much rather have ssh access to anywhere I'm going to be dealing with a system console, so I can be much more sure of the security of the connection (both in terms of encryption and auth{n,z} to access the console).
[ Parent | Reply to this comment ]
> is no way that I know of to permanently map a particular
> USB-serial device to a given serial port (/dev/ttyUSB?)
USB-serial devices generally don't have unique IDs in them, which makes this difficult but not impossible. In my case, my serial cables were all bought at different times and are of different brands, so they can be distinguished. If your cables are all identical then you need to associate with the hub port numbers, and then remember to always plug the cables into the same hub sockets. A few udev rules can do this; I'm sure google would be able to find an example but post again if you can't find anything.
> I would be extremely wary of putting a system console on the
> web. The web is far too fragile in my opinion for mission critical
> stuff like this. I would much rather have ssh access
Well you can have both of course. I wrote Anyterm to cope with the problem of access from behind an HTTP[S]-only firewall where ssh was impossible.
[ Parent | Reply to this comment ]
[ Send Message | View dkg's Scratchpad | View Weblogs ]
An alternative to multi-port serial cards like the Lava Octopus that's linked to in the article is to use USB-serial cables. These are cheap and can be combined with a USB hub or hubs.I've tried this, but was stymied by the lack of identifiers on the devices that drgraefy mentioned. Also, i'm not sure that they're much cheaper for mid-to-large size installs. The Octopus costs ~$120 these days, which is $15/port. Many single-port USB-to-serial adapters cost about the same amount, and if you need to add USB hubs to chain them out, then you need to factor in the cost of the hubs as well. Also, the USB hub will often require its own power supply, which means more physical space and more outlets -- in a racked environment, these are often extra hurdles.
I'd be curious to hear of other devices, that might offer similar tradeoffs, though.
[ Parent | Reply to this comment ]