Building a simple dashboard with redis and the node.js server.

Posted by Steve on Fri 18 Jan 2013 at 08:49

Recently we introduced redis, and the built-in data-types it has. Taking that introduction a step-further we're going to present a simple "dashboard" written using a combination of Redis and node.js.

The basic goal of a dashboard is to show you interesting activity in real-time. That interesting activity might be simple text strings describing events, load graphs, or even sales-counts. If you're running a cluster of some sort a dashboard is very useful, because it allows you to see events that have happened globally - rather than logging on to each machine in turn to pull out metrics and ensure all is well.

Once you realize that you want to combine events from multiple machines it becomes obvious that you need to listen on the network, somehow, to accept the submission of "interesting events". In the past I've used various things to run simple servers, from Apache + CGI scripts, to sinatra, and most recently node.js.

If you've not yet heard of node.js it can be explained as event-driven server-side javascript. It is designed to allow you to write highly-scalable software, because everything is based on events and callbacks.

Unfortunately the biggest downside to using node.js is that it is still a very young project, with semi-regular updates. There is a Debian GNU/Linux package for node, but it is available only in the unstable distribution - and as of January 2013 even that is out of date (#692312).

For my own personal systems I track recent releases and install directly into /opt. Every time a new release is issued I update the system, and I've not yet found this too painful a prospect.

The most recent release right now is v0.8.17, although that might change if you're reading this piece in the future, and it may be installed by downloading the source and building like so:

# cd /opt
# wget http://nodejs.org/dist/v0.8.17/node-v0.8.17.tar.gz
# tar zxf node-v0.8.17.tar.gz
# cd node-v0.8.17
# ./configure --prefix=/opt/node-v0.8.17 && make install

The end result of this build will be a binary installation located in /opt/node-v0.8.17, but to avoid having to deal with version numbers it makes sense to create a symlink:

# cd /opt
# ln -s node-v0.8.17 node

Once you've created this symlink you may then refer to the node.js binary, regardless of the version, with the path /opt/node/bin/node - and this means you never need to update your PATH, or your scripts, more than once. (This approach also allows you to roll back to previous versions if you ever need to.)

We're going to be using node.js to listen upon the network, and receive the "interesting event" messages. The obvious question is then "How do we store these messages?" To that the answer is redis. The reason for choosing redis is that it is fast, both for storing the messages and retrieving them.

When you consider what you're going to do with a status page there are two obvious cases you need to handle:

As we've seen with our Redis introduction adding items to a list is trivial, and so are both receiving the list, and truncating the list. We're going to need to a library which allows us to interface with Redis from node.js, and happily interfacing with Redis is possible using this library.

The next decision is how to receive the messages? I chose to use UDP because although delivery isn't guaranteed it is "good enough" in practice, especially within a private network. There is low overhead to sending UDP messages, and such things can be done from many many languages and environments - this means that our event-emitters don't also need to be written in node.js, and can be perl, ruby, or anything else that is capable of sending a UDP packet.

The core of the service is simple:

Writing such a service in node.js is almost trivial, and the resulting code is easily understandable if you're familiar with javascript.

To run the code you'll need the script referenced above, as well as the client library for talking to Redis. I've bundled both together into a repository you can checkout from github:

# git clone https://github.com/skx/dashboard.git
Cloning into dashboard...
remote: Counting objects: 70, done.
remote: Compressing objects: 100% (67/67), done.
remote: Total 70 (delta 4), reused 69 (delta 3)
Unpacking objects: 100% (70/70), done.
#

Once you've checked out the code launching it is as simple as:

# /opt/node/bin/node dashboard.js
server listening 0.0.0.0:4433

The banner will show you that the server is listening upon :4433, and it will insert any events it receives into the Redis database running upon the localhost. Submitting a new event is a good test of the code. So save this as "submit.pl":

#!/usr/bin/perl

use strict;
use warnings;

use IO::Socket;

my $msg = shift( @ARGV );
exit(0) unless( defined( $msg ) && length($msg) );

my $sock = IO::Socket::INET->new( Proto    => 'udp',
                                  PeerPort => 4433,
                                  PeerAddr => "localhost",
                                ) ;

$sock->send( $msg );
exit(0);

With this code you can now submit some sample events:

# perl submit.pl "This is a test"
# perl submit.pl "So is this"

The end result is that you'll have two entries submitted to the Redis list "dashboard", which you can verify with the redis-cli tool:

#redis-cli lrange dashboard 0 -1
1. 127.0.0.1#Thu Jan 17 2013 18:46:39 GMT+0000 (GMT)#So is this
2. 127.0.0.1#Thu Jan 17 2013 18:46:35 GMT+0000 (GMT)#This is a test

As you can see the messages have been inserted along with the source-IP, and the date/time-stamp. Both of these have been deliminated with the hash ("#") character.

Now that the core of the service has been written we're in good shape:

From here we're nearly done, all we need is a web-page that will show the most recent 100 events, or so. This can be carried out via a simple Perl CGI-script with ease. Something like this will suffice:

#!/usr/bin/perl -w

use strict;
use warnings;
use Redis;

print <<EOH;
Content-type: text/html


<html>
 <head>
  <title>Recent Events</title>
 </head>
 <body>
EOH
my $redis  = new Redis();
my @values = $redis->lrange( "dashboard", 0, 100);

#
#  Process each event.
#
foreach my $event (@values )
{
    my $ip;
    my $time;
    my $msg;

    if ( $entry =~ /^([^#]+)#([^#]+)#(.*)$/ )
    {
        $ip = $1;
        $time = $2;
        $msg = $3;
    }

    print "$ip - $time - $msg\n";
}
print <<EOF;
  </pre>
 </body>
</html>
EOF
;

This CGI script talks to redis directly to receive the most recent 100 events, and then dumps them out in a simple fashion. A mort artistic person than myself might use a pretty table, and nice CSS to make it display well. For bonus points you could invoke the CGI script via AJAX to give yourself a constantly updating status page.


This article can be found online at the Debian Administration website at the following bookmarkable URL (along with associated comments):

This article is copyright 2013 Steve - please ask for permission to republish or translate.