Introduction to Cereal - the serial line management tool

Posted by drgraefy on Mon 15 Oct 2007 at 13:01

Many people view serial ports as antiquated, out-dated connectors taking up space on their computers. However, serial ports still offer one of the best ways to communicate reliably and simply with a machine. For example, a serial port can be configured to act as a full-featured system console. This article describes how you can use cereal to monitor, log, and control access to serial lines connected to the consoles of other computers.

A computer today has several subsystems that should be run over the serial port to make it a full-featured system console:

  • most modern operating systems (including the GNU/Linux utilities favored by Debian), are able to provide login terminals on serial ports,
  • most modern kernels (including Linux) are capable of directing their console I/O to a serial line,
  • most bootloaders (including stalwarts like GRUB, the SYSLINUX family, and LILO) can communicate via serial ports,
  • and many BIOSs, including most modern server BIOSs, can be redirected over the serial port.

Using a null-modem cable, you can connect the primary serial port on one computer to a remote computer's spare serial port. The remote computer can have full access to the first machine's console using only a terminal emulator like GNU screen or minicom. Not only do you not need yet another cumbersome monitor and keyboard, but you get a digital text stream to manipulate directly. You can log, timestamp, copy, paste, diff, etc the entire data stream flowing over the console, including boot-time spew, kernel crash output, etc.

Using serial ports as system consoles is especially useful in high-density computing environments, such as racks in data centers. Clutter like monitors, vga cables, video cards, keyboards, keyboard cables, and kvm switches can all be thrown out in favor of serial cables and a single computer that we will refer to as a serial console server. A serial console server is a machine with many serial ports, each connected to the serial console of another machine. An administrator accesses the serial console server over the network (e.g. via ssh) and from there can connect to the console of any of the connected machines. In all examples below, scs is the hostname of a serial console server.

This is where cereal comes in. cereal is a management framework designed to monitor, log, and control access to serial lines. It is installed on the serial console server, and makes the task of maintaining such a machine simpler. It can also be used in simpler configurations, such as two machines which each monitor the other's console. This article describes how to setup and use cereal. We assume that the computers you are connecting to are already configured to send their consoles over the serial line. If that's not the case yet, read The Remote Serial Console HOWTO and get 'em configured first!

The cereal utility is not yet available in Debian proper, but it is packaged and available from the CMRG APT repository. Once you've added this repository to your sources.list (you may want to update your apt keys if you care to about verifying packages from the repo), you can pull it with apt-get or aptitude:

scs:0:~# echo deb http://cmrg.fifthhorseman.net/debian unstable cereal >> /etc/apt/sources.list.d/cereal.list
scs:0:~# aptitude update
scs:0:~# aptitude install cereal

Administering cereal sessions: /usr/sbin/cereal-admin

Serial line management with cereal is broken out into sessions. A cereal session is a single runit service controlling a special screen session attached to a single serial line. Creation and control of these sessions is done with the cereal-admin utility:

scs:0:~# cereal-admin help
Usage: cereal-admin  [options] [args]
Cereal session management program.

subcommands:
  create (c) SESSION TTY BAUD USER LOGGROUP    create cereal session
  start (s) [options] SESSION [SESSION]...     start cereal session(s)
    -a (--all)                                   start all sessions
  stop (k) [options] SESSION [SESSION]...      stop cereal session(s)
    -a (--all)                                   stop all sessions
  restart (r) [options] SESSION [SESSION]...   restart cereal session(s)
    -a (--all)                                   restart all sessions
    -r (--running)                               restart any currently running
                                                  sessions 
  destroy (d) [options] SESSION [SESSION]...   destroy cereal session(s)
    -a (--all)                                   destroy all sessions
  list (l) [SESSION]...                        list session(s)
  help (h,?)                                   this help

scs:0:~#

Creating a cereal session

Let's assume the first computer whose serial console you want to monitor with cereal is called lemur. Let's assume lemur's serial console is configured with baud rate 115200, and is connected to the serial console server on /dev/ttyS1 (/dev/ttyS0 on scs is of course reserved for the serial console server's own serial console!).

You can give access to cereal sessions, and the ability to read the session logs, to existing users. However, it is often a good idea to create new users and groups for these purposes. We begin by creating a new group that will be allowed to view ('follow') lemur's serial console logs, followed by creating a new user that will be able to control ('attach' to) lemur's cereal session.

scs:0:~# addgroup lemurviewers
Adding group `lemurviewers' (GID 1001) ...
Done.

scs:0:~# adduser --ingroup lemurviewers --gecos 'lemur cereal session admin,,,' lemuradmin
Adding user `lemuradmin' ...
Adding new user `lemuradmin' (1001) with group `lemurviewers' ...
Creating home directory `/home/lemuradmin' ...
Copying files from `/etc/skel' ...

scs:0:~#

You are now ready to create lemur's cereal session with cereal-admin:

scs:0:~# cereal-admin create lemur /dev/ttyS1 115200 lemuradmin lemurviewers
Created session 'lemur':
--f lemur /dev/ttyS1 115200 lemuradmin lemurviewers
scs:0:~#

The first three characters of the session status line are the session flags that indicate the state of the cereal session. In order, they are:

state
The first flag indicates whether the session is running (+) or not (-), or in some unusual state (!).
attach
The second flag indicates whether the user can attach to the session (a) or not (-).
follow
The third flag indicates whether the user can follow the session (f) or not (-).

Once a cereal session is created, you can begin monitoring of it with the cereal-admin start command:

scs:0:~# cereal-admin list lemur
--f lemur /dev/ttyS1 115200 lemuradmin lemurviewers
scs:0:~# cereal-admin start lemur
Started session 'lemur'.
scs:0:~# cereal-admin list lemur
+-f lemur /dev/ttyS1 115200 lemuradmin lemurviewers
scs:0:~#

Accessing cereal sessions: /usr/bin/cereal

Now that you have a cereal session up and running, you can test the connection. Log into scs as the non-privileged lemuradmin user and use the cereal tool to attach to or follow the session:

lemuradmin@scs:~$ cereal help
Usage: cereal  [options] [args]
Cereal client program.

subcommands:
  attach (a) SESSION                      attach to session
  follow (f) [options] SESSION            follow session log
    -c (--cat)                              cat log instead of follow
    -p (--path)                             output just log file path
  list (l) [SESSION]...                   list session(s)
  help (h,?)                              this help

lemuradmin@scs:~$ cereal list lemur
+af lemur /dev/ttyS1 115200 lemuradmin lemurviewers
lemuradmin@scs:~$

Attaching to a cereal session

Using the attach function will put the the user into a cereal console, provided by a command-restricted screen session. To detach from an attached session, use "C-\ d". For help on other available commands, use "C-\ ?".

lemuradmin@scs:~$ cereal attach lemur
Debian GNU/Linux 4.0 lemur ttyS0

lemur login: root
Password: 
Last login: Sat Oct  6 10:16:26 2007 on ttyS0
Linux lemur 2.6.18-5-amd64 #1 SMP Tue Oct 2 20:37:02 UTC 2007 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
lemur:0:~# exit




cereal console: lemur              2007-10-06 10:17              "C-\ d" to detach
lemuradmin@scs:~$ cereal attach lemur
[detached]
lemuradmin@scs:~$

Following a cereal session

You can use the follow command to easily view the logs of a cereal session.

lemuradmin@scs:~$ cereal follow --cat lemur | cat -A
############################################################
### cereal log: lemur
### +af lemur /dev/ttyS1 115200 lemuradmin lemurviewers
### log file: /var/lib/cereal/sessions/lemur/log/main/current
############################################################
2007-10-06_14:39:44.56895 
2007-10-06_14:39:44.56900 cereal: session 'lemur' started.
2007-10-06_14:39:44.56900 starting screen session...
2007-10-06_14:39:58.86707 
2007-10-06_14:39:58.86708 cereal: user 'lemuradmin' attaching to session...
2007-10-06_14:40:02.57346 
2007-10-06_14:40:02.57348 Debian GNU/Linux 4.0 lemur ttyS0
2007-10-06_14:40:02.57348 
2007-10-06_14:40:02.57349 lemur login: root
2007-10-06_14:40:13.58131 Password: 
2007-10-06_14:40:14.58129 Last login: Sat Oct  6 10:16:48 2007 on ttyS0
2007-10-06_14:40:14.58129 Linux lemur 2.6.18-5-amd64 #1 SMP Tue Oct 2 20:37:02 UTC 2007 x86_64
2007-10-06_14:40:14.58130 
2007-10-06_14:40:14.58131 The programs included with the Debian GNU/Linux system are free software;
2007-10-06_14:40:14.58131 the exact distribution terms for each program are described in the
2007-10-06_14:40:14.58132 individual files in /usr/share/doc/*/copyright.
2007-10-06_14:40:14.58132 
2007-10-06_14:40:14.58133 Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
2007-10-06_14:40:14.58134 permitted by applicable law.
2007-10-06_14:40:14.58135 Last was Sat 06 Oct 2007 10:33:15 AM EDT on ttyS0.
2007-10-06_14:40:14.58135 ESC[31;1mlemur:0:~#ESC[0m exit
2007-10-06_14:40:17.58525 logout
2007-10-06_14:40:17.58525 ESC[HESC[J
2007-10-06_14:40:18.58523 Debian GNU/Linux 4.0 lemur ttyS0
2007-10-06_14:40:18.58523 
2007-10-06_14:40:18.58524 lemur login: 
2007-10-06_14:40:19.62101 cereal: user 'lemuradmin' detached from session.
lemuradmin@scs:~$

Conclusion

Now that you've created your first cereal session, you're ready to create serial sessions for all the rest of the attached serial consoles:

scs:0:~# cereal-admin list
+-f lemur /dev/ttyS1 115200 lemuradmin lemurviewers
+-f chimp /dev/ttyS2 115200 chimpadmin chimpviewers
+-f douc /dev/ttyS3 115200 doucadmin doucviewers
+-f galago /dev/ttyS4 115200 galagoadmin galagoviewers
+-f baboon /dev/ttyS5 115200 baboonadmin baboonviewers
...
scs:0:~#

Cereal is developed and maintained by CMRG. Hopefully the package will be included in Debian soon, at which point bug tracking will be handled in the Debian BTS. Until then, please sent all bug reports, questions, and suggestions to cereal AT fifthhorseman.net.

 

 


Posted by Anonymous (202.7.xx.xx) on Mon 15 Oct 2007 at 13:55
Nice article.

Serial links are also still used often in HA situations for heart beats.

Laptops with serial ports are also handy when at the DC

[ Parent | Reply to this comment ]

Posted by endecotp (86.6.xx.xx) on Wed 17 Oct 2007 at 22:48
[ View Weblogs ]
An alternative to multi-port serial cards like the Lava Octopus that's linked to in the article is to use USB-serial cables. These are cheap and can be combined with a USB hub or hubs. I have also seen (but not used) some USB-to-multiple-serial cables, which have a USB plug, small dongle and perhaps 4 serial plugs.

If your aim is to provide "emergency" serial port access to e.g. customers' co-located machines, a web interface might be a nice feature (especially if you already have a web interface for your power switch, or remote KVM or whatever). In that case, you could install Anyterm, which works well with serial ports. Disclaimer: I wrote it. http://anyterm.org/

[ Parent | Reply to this comment ]

Posted by drgraefy (128.59.xx.xx) on Wed 17 Oct 2007 at 23:09
[ View Weblogs ]
USB-serial adapters are great devices, especially for providing serial ports on laptops. I use them for that purpose all the time. I have also used them for serial console monitoring (with cereal). However, the problem with using them for this purpose is that there is no way that I know of to permanently map a particular USB-serial device to a given serial port (/dev/ttyUSB?). So you might occasionally get remappings of USB-serial devices to serial ports on system reboots and such. This can be problematic if you're trying keep good consistent monitoring. If someone knows how to make permanent mappings I would absolutely love to hear about it (please post here!).

As far as anyterm goes, I think it's an interesting concept, but I would be extremely wary of putting a system console on the web. The web is far too fragile in my opinion for mission critical stuff like this. I would much rather have ssh access to anywhere I'm going to be dealing with a system console, so I can be much more sure of the security of the connection (both in terms of encryption and auth{n,z} to access the console).

[ Parent | Reply to this comment ]

Posted by endecotp (86.6.xx.xx) on Wed 17 Oct 2007 at 23:23
[ View Weblogs ]
> the problem with using them for this purpose is that there
> is no way that I know of to permanently map a particular
> USB-serial device to a given serial port (/dev/ttyUSB?)

USB-serial devices generally don't have unique IDs in them, which makes this difficult but not impossible. In my case, my serial cables were all bought at different times and are of different brands, so they can be distinguished. If your cables are all identical then you need to associate with the hub port numbers, and then remember to always plug the cables into the same hub sockets. A few udev rules can do this; I'm sure google would be able to find an example but post again if you can't find anything.

> I would be extremely wary of putting a system console on the
> web. The web is far too fragile in my opinion for mission critical
> stuff like this. I would much rather have ssh access

Well you can have both of course. I wrote Anyterm to cope with the problem of access from behind an HTTP[S]-only firewall where ssh was impossible.

[ Parent | Reply to this comment ]

Posted by dkg (216.254.xx.xx) on Wed 24 Oct 2007 at 17:43
[ View dkg's Scratchpad | View Weblogs ]
An alternative to multi-port serial cards like the Lava Octopus that's linked to in the article is to use USB-serial cables. These are cheap and can be combined with a USB hub or hubs.
I've tried this, but was stymied by the lack of identifiers on the devices that drgraefy mentioned. Also, i'm not sure that they're much cheaper for mid-to-large size installs. The Octopus costs ~$120 these days, which is $15/port. Many single-port USB-to-serial adapters cost about the same amount, and if you need to add USB hubs to chain them out, then you need to factor in the cost of the hubs as well. Also, the USB hub will often require its own power supply, which means more physical space and more outlets -- in a racked environment, these are often extra hurdles.

I'd be curious to hear of other devices, that might offer similar tradeoffs, though.

[ Parent | Reply to this comment ]

Posted by Anonymous (217.226.xx.xx) on Fri 20 Apr 2012 at 23:49
Hello,
my timestamp in the serial session log has is wrong.
The difference i 2h11m.
How can i edit this?

[ Parent | Reply to this comment ]

Sign In

Username:

Password:

[Register|Advanced]

 

Flattr

 

Current Poll

What do you use for configuration management?








( 76 votes ~ 0 comments )