Using runit for maintaining services
Posted by Steve on Sat 4 Jan 2014 at 10:38
There are several programs, and systems, out there for supervising the execution of child-processes. Perhaps one of the most well-known is runit, which was modelled upon the daemontools software by djb.
Once installed the runit package allows you to easily deploy and control services:
- Finding them.
- Starting them.
- Stopping them.
One of the biggest reasons for using runit is that services restart automatically when they crash, or otherwise exit. For that purpose I use it to launch the vast majority of my node.js applications.
Although we've been talking about the control of services I should make it clear that we're not talking about system servers - although runit could be used in that way if you prefer, we're explicitly not talking about replacing init, we're talking about controlling user-installed services.
To get started install the package:
root@shelob:~# apt-get install runit
Once you've done that you'll find a new daemon has been launched, via an addition to your /etc/inittab file. There are a few processes involved but we mostly don't need to care about them. The important thing about runit is that it launches processes by scanning a particular directory - /etc/service - looking for subdirectories which contain executable scripts named run.
If you create a new directory, and inside that create an executable file called run it will be launched, and relaunched if it exits.
For example if you were to create the directory /etc/service/test, then save the following contents to the file /etc/service/test/run you'd have created a new service:
#!/bin/sh touch /tmp/blah
This service would run, executing the run script, and then it would exit. Then it would start again, over and over again until you stopped it.
Stopping the service is a matter of using the most useful command installed via runit: sv. The sv command can be used to check on the statutus of a service:
root@shelob:~# sv status /etc/service/test down: /etc/service/test: 0s, normally up, want up root@shelob:~# sv status /etc/service/test down: /etc/service/test: 1s, normally up, want up
It can also be used to start a service, or stop it:
root@shelob:~# sv stop /etc/service/test/ ok: down: /etc/service/test/: 0s, normally up, want up root@shelob:~# sv status /etc/service/test/ down: /etc/service/test/: 17s, normally up
This trivial service isn't terribly interesting, continuously touching a file. But for simple deamons such as node.js-based servers it can be very useful.
Simply create a directory, and inside that place an executable script named run with contents such as this:
#!/bin/sh
#
# Run our BlogSpam.net API - v2 - restarting on failure.
#
#
# Steve deploys node beneath /opt/node
#
if [ -d /opt/node ]; then
export PATH=/opt/node/bin:$PATH
fi
#
# Ensure we're current.
#
git pull --update --quiet
#
# Install any missing modules, via npm.
#
if [ -e "package.json" ]; then
npm install
fi
#
# Finally launch the service, with suitable logging.
#
exec node server.js 2>&1 >> /var/log/node.js.log
That's all it takes to :
- Refresh the service from a remote git repository.
- Launch it.
- Being confident that when the service exits it will be automatically restarted.
If ever I want to update my service I can either run sv stop ..", followed by "sv start ..", or I can run:
root@shelob:~# sv restart /etc/service/blogspam.js/ ok: run: /etc/service/blogspam.js/: (pid 14712) 0s
It is possible to run many scripts like this, for example you can launch SSH under runit by removing the standard init-script, and creating a file /etc/service/sshd/run:
#!/bin/sh exec /usr/sbin/sshd -D
You'd not see much gain, or difference, but if your system ran out of RAM and the SSH service died it would be restarted if you went down this route. Similarly you can deploy a service that a user can have permission to restart, simply execute the process under their user-ID, and they have permission to "kill -9" it, and it will then restart:
#!/bin/sh su - user -c "exec /home/user/daemon.sh"
I've used that technique before to ensure a particular webserver deamons are running, and still allowing the users to restart them if they wished:
#!/bin/sh exec thttpd -D -C /srv/steve.org.uk/thttpd.conf -M 3600
In this case the thttpd webserver changes to a particular user-id after it launches:
root@www ~# ps -ef | grep s-steve s-steve 7792 2752 0 04:25 ? 00:00:00 thttpd -D -C /srv/steve.org.uk/thttpd.conf -M 3600
Now that user can kill it, to trigger a restart:
s-steve@www:~$ kill 7792
As expected the service is alive again:
root@www~# sv status /etc/service/steve.org.uk/ run: /etc/service/steve.org.uk/: (pid 16948) 32s
The runit website contains a small collection of run-scripts which can be useful to examine; they've very handy if you have a deamon, or two, which you need to launch and which doesn't come with a real initscript.