Running programs when filesystem events occur

Posted by Steve on Wed 20 Feb 2008 at 10:00

There are many little jobs which people tend to schedule, via cron, which do nothing unless particular files have appeared. These busy-wait style scripts may easily be replaced if you have the ability to execute commands when files are created, or filesystem events happen. Read on to see how to do that.

I have several cronjobs which execute every minute or two looking for new files to process. I want the script to immediately start jobs when it finds new files, but I'm not terribly keen on running a cron-job every minute, which will immediately exit as there is no work most of the time.

The solution to this is to attack the problem the other way round. Rather than running a job every minute to see if there is a new file in a spool directory to process it makes more sense to begin the execution of a script whenever a file has just been created - and ideally without having to code a single-use daemon.

Thankfully all of the recent Debian kernels have support for something called inotify. inotify is the name of a kernel interface which allows you to efficiently watch parts of a directory tree and see when events occur.

There is a package called incron which turns this kernel interface into something that you can directly use.

Installing The Package On Etch

If you're running Debian's Etch release incron isn't available as a package in the Debian repository, only as an etch backports.

To add the backports site to your system run these commands:
rt:~# echo 'deb http://www.backports.org/debian etch-backports main contrib non-free' >> /etc/apt/sources.list
rt:~# apt-get update
rt:~# apt-get install debian-backports-keyring
Once you've done that you may install the package as follows:
rt:~# apt-get install -t etch-backports incron
Reading package lists... Done
Building dependency tree... Done
The following NEW packages will be installed
  incron
0 upgraded, 1 newly installed, 0 to remove and 18 not upgraded.
Need to get 124kB of archives.

Installing The Package On Lenny / Sid

Because the package is available directly for these releases of Debian you should be able to install it as usual with this:
rt:~# apt-get install incron

Using incron

incron is very similar in concept and usage to using cron, as the interface is a clone of it.

Each user who is allowed to use incron may use the incrontab command to view, or edit, their rule list. These rules are processed via the daemon, and when a match occurs the relevant command is executed.

To list the current rules you've got defined run "incrontab -l", and to edit them use "incrontab -e". If you do that just now you'll receive the following error message:
rt:~# incrontab  -l
user 'root' is not allowed to use incron
This error may be fixed in one of two ways:

Allow the root user to make use of incron:

By editing /etc/incron.allow, adding 'root' to it.

Allowing all local users the ability to use incron:

By removing the file /etc/incron.allow.

Once you've allowed root, or your user, to run the incrontab command we can have a lookg at an example.

Example Usage

Assume for the moment that you have a script which needs to run when a new file is created in /tmp/spool. That script is /usr/local/bin/run-spool.

To do that you could run "incrontab -e" and add this line:
/tmp/spool IN_CLOSE_WRITE /usr/local/bin/run-spool $@/$#
This says "Watch /tmp/spool, and when an IN_CLOSE_WRITE event occurs run /usr/local/bin/run-spool with the name of the file that was created".

There are two sets of magic flags here - the first is the IN_CLOSE_WRITE, and the second is $@/$#. The second set of flags is simpler to explain as there are fewer of them:

$$ - a dollar sign

$@ - the watched filesystem path (ie. the path you're watching)

$# - the event-related file name

$% - the event flags (textually)

$& - the event flags (numerically)

The actual event flags include IN_CLOSE_WRITE, which means that a file was closed for writing. The full list of supported flags include:

IN_ACCESS File was accessed (read)

IN_ATTRIB Metadata changed (permissions, timestamps, extended attributes, etc.)

IN_CLOSE_WRITE File opened for writing was closed

IN_CLOSE_NOWRITE File not opened for writing was closed

IN_CREATE File/directory created in watched directory

IN_DELETE File/directory deleted from watched directory

IN_DELETE_SELF Watched file/directory was itself deleted

IN_MODIFY File was modified

IN_MOVE_SELF Watched file/directory was itself moved

IN_MOVED_FROM File moved out of watched directory

IN_MOVED_TO File moved into watched directory

IN_OPEN File was opened

Multiple flags may be separated via commas.

Summary

Provided you have kernel support for inotify - which you can check by running "zgrep CONFIG_INOTIFY /proc/config.gz" - then this is a very lightweight way of running programs on filesystem events.

It scales nicely, providing you don't need to add lots of recursive definitions. (For example /home/steve/Maildir/*/new isn't a workable path to monitor unfortunately.)

This is definitely my most interesting, and useful, Debian discovery this year. Why not write about yours?

This article can be found online at the Debian Administration website at the following bookmarkable URL (along with associated comments):

http://www.debian-administration.org/article/Running_programs_when_filesystem_events_occur