Running programs when filesystem events occur

Posted by Steve on Wed 20 Feb 2008 at 10:00

There are many little jobs which people tend to schedule, via cron, which do nothing unless particular files have appeared. These busy-wait style scripts may easily be replaced if you have the ability to execute commands when files are created, or filesystem events happen. Read on to see how to do that.

I have several cronjobs which execute every minute or two looking for new files to process. I want the script to immediately start jobs when it finds new files, but I'm not terribly keen on running a cron-job every minute, which will immediately exit as there is no work most of the time.

The solution to this is to attack the problem the other way round. Rather than running a job every minute to see if there is a new file in a spool directory to process it makes more sense to begin the execution of a script whenever a file has just been created - and ideally without having to code a single-use daemon.

Thankfully all of the recent Debian kernels have support for something called inotify. inotify is the name of a kernel interface which allows you to efficiently watch parts of a directory tree and see when events occur.

There is a package called incron which turns this kernel interface into something that you can directly use.

Installing The Package On Etch

If you're running Debian's Etch release incron isn't available as a package in the Debian repository, only as an etch backports.

To add the backports site to your system run these commands:

rt:~# echo 'deb http://www.backports.org/debian etch-backports main contrib non-free' >> /etc/apt/sources.list
rt:~# apt-get update
rt:~# apt-get install debian-backports-keyring

Once you've done that you may install the package as follows:

rt:~# apt-get install -t etch-backports incron
Reading package lists... Done
Building dependency tree... Done
The following NEW packages will be installed
  incron
0 upgraded, 1 newly installed, 0 to remove and 18 not upgraded.
Need to get 124kB of archives.

Installing The Package On Lenny / Sid

Because the package is available directly for these releases of Debian you should be able to install it as usual with this:

rt:~# apt-get install incron

Using incron

incron is very similar in concept and usage to using cron, as the interface is a clone of it.

Each user who is allowed to use incron may use the incrontab command to view, or edit, their rule list. These rules are processed via the daemon, and when a match occurs the relevant command is executed.

To list the current rules you've got defined run "incrontab -l", and to edit them use "incrontab -e". If you do that just now you'll receive the following error message:

rt:~# incrontab  -l
user 'root' is not allowed to use incron

This error may be fixed in one of two ways:

  • Allow the root user to make use of incron:
    • By editing /etc/incron.allow, adding 'root' to it.
  • Allowing all local users the ability to use incron:
    • By removing the file /etc/incron.allow.

Once you've allowed root, or your user, to run the incrontab command we can have a lookg at an example.

Example Usage

Assume for the moment that you have a script which needs to run when a new file is created in /tmp/spool. That script is /usr/local/bin/run-spool.

To do that you could run "incrontab -e" and add this line:

/tmp/spool IN_CLOSE_WRITE /usr/local/bin/run-spool $@/$#

This says "Watch /tmp/spool, and when an IN_CLOSE_WRITE event occurs run /usr/local/bin/run-spool with the name of the file that was created".

There are two sets of magic flags here - the first is the IN_CLOSE_WRITE, and the second is $@/$#. The second set of flags is simpler to explain as there are fewer of them:

  • $$ - a dollar sign
  • $@ - the watched filesystem path (ie. the path you're watching)
  • $# - the event-related file name
  • $% - the event flags (textually)
  • $& - the event flags (numerically)

The actual event flags include IN_CLOSE_WRITE, which means that a file was closed for writing. The full list of supported flags include:

  • IN_ACCESS File was accessed (read)
  • IN_ATTRIB Metadata changed (permissions, timestamps, extended attributes, etc.)
  • IN_CLOSE_WRITE File opened for writing was closed
  • IN_CLOSE_NOWRITE File not opened for writing was closed
  • IN_CREATE File/directory created in watched directory
  • IN_DELETE File/directory deleted from watched directory
  • IN_DELETE_SELF Watched file/directory was itself deleted
  • IN_MODIFY File was modified
  • IN_MOVE_SELF Watched file/directory was itself moved
  • IN_MOVED_FROM File moved out of watched directory
  • IN_MOVED_TO File moved into watched directory
  • IN_OPEN File was opened

Multiple flags may be separated via commas.

Summary

Provided you have kernel support for inotify - which you can check by running "zgrep CONFIG_INOTIFY /proc/config.gz" - then this is a very lightweight way of running programs on filesystem events.

It scales nicely, providing you don't need to add lots of recursive definitions. (For example /home/steve/Maildir/*/new isn't a workable path to monitor unfortunately.)

This is definitely my most interesting, and useful, Debian discovery this year. Why not write about yours?

 

 


Posted by Anonymous (202.180.xx.xx) on Wed 20 Feb 2008 at 10:37
I found this very useful. Thanks!

[ Parent | Reply to this comment ]

Posted by ineiti (84.226.xx.xx) on Wed 20 Feb 2008 at 15:30
[ Send Message ]
What's the difference between

* Allow the root user to make use of incron:
o By executing "rm /etc/incron.allow".
* Allowing all local users the ability to use incron:
o By removing /etc/incron.allow.
??
Perhaps the first recipee should read
echo root >> /etc/incron.allow

Ineiti

[ Parent | Reply to this comment ]

Posted by Steve (80.68.xx.xx) on Wed 20 Feb 2008 at 15:32
[ Send Message | View Steve's Scratchpad | View Weblogs ]

Yes, that was a bad bit of text, I've updated it now.

The choices were supposed to be "edit the file" or "remove the file".

Steve

[ Parent | Reply to this comment ]

Posted by Anonymous (2001:0xx:0xx:0xxx:0xxx:0xxx:xx) on Wed 20 Feb 2008 at 17:19
Very nice article about, for me, a new tool. Thanks.

To add a repository to apt, I usally find it easy to just make a new file in /etc/apt/sources.list.d instead of changing /etc/apt/sources.list

Just have a file name ending with '.list', and aptitude will use it like it was added to /etc/apt/sources.list

So I would do:

cat <<EOF >/etc/apt/sources.list.d/etch-backports.list
# Debian backports packages
# See: http://www.backports.org/
# Please use a mirror
deb http://www.backports.org/debian etch-backports main contrib non-free
# eof
EOF


Then it is easy to just remove that file if it is something wrong. And you can comment it to get a history when/why you added it.

[ Parent | Reply to this comment ]

Posted by Anonymous (76.160.xx.xx) on Wed 20 Feb 2008 at 18:03
Recent Debian kernels don't have /proc/config.gz; instead, do this:
$ grep CONFIG_INOTIFY /boot/config-`uname -r`
CONFIG_INOTIFY=y
CONFIG_INOTIFY_USER=y

[ Parent | Reply to this comment ]

Posted by Anonymous (216.165.xx.xx) on Thu 21 Feb 2008 at 00:38
How does this compare with the much older and more widely known FAM (File Alteration Monitor) from SGI? FAM is also portable (inotify is not); there's even a SGI::FAM Perl Module.

[ Parent | Reply to this comment ]

Posted by hildeb (193.175.xx.xx) on Thu 21 Feb 2008 at 09:16
[ Send Message ]
FAM looks fairly dead, at least judging from their website.

[ Parent | Reply to this comment ]

Posted by Anonymous (83.15.xx.xx) on Wed 27 Feb 2008 at 21:03
Maybe dnotify:
Description: Execute a command when the contents of a directory change
dnotify is a simple program based on Linux kernel 2.4.19+'s dnotify
API. dnotify can execute a specified command each time the content
of a specific directory changes. It is run from the command line and
takes two arguments: one or more directories to monitor and a command
to execute whenever a directory has changed. Options control what
events to trigger on: when a file was read in the directory, when one
was created, deleted and so on.
Tag: interface::commandline, use::monitor, works-with::file

regards,
LiNiO

[ Parent | Reply to this comment ]

Posted by docelic (78.134.xx.xx) on Sat 15 Mar 2008 at 18:45
[ Send Message ]
The idea of inotify was, among other things, to replace FAM. Linux port of FAM lacked significant amounts of documented functionality.

[ Parent | Reply to this comment ]

Posted by Anonymous (62.172.xx.xx) on Thu 21 Feb 2008 at 10:38
In Ubuntu, you can do similar things with the new Upstart system that replaces init.

Create a 'job' description file in /etc/event.d that specifies what to run and when.
More details at http://upstart.ubuntu.com/getting-started.html.

[ Parent | Reply to this comment ]

Posted by Steve (80.68.xx.xx) on Thu 21 Feb 2008 at 10:40
[ Send Message | View Steve's Scratchpad | View Weblogs ]

That's not the same thing at all, as far as I can see..

Steve

[ Parent | Reply to this comment ]

Posted by jwm (203.97.xx.xx) on Thu 21 Feb 2008 at 14:10
[ Send Message ]
Upstarts aiming to become a complete process supervisor, and one of the immediate goals is to start programs and scripts on time based events to replace cron. No reason not to pull inotify events from the kernel to the same ends.

But, that's a way off, yet.

[ Parent | Reply to this comment ]

Posted by rubasov (84.3.xx.xx) on Thu 28 Feb 2008 at 19:17
[ Send Message ]
I had some problems lately, some of our daemons (asterisk, bind, ulogd) died unexpectedly because of segfaults. I don't have the neccessary programming skills to interpret the core dumps and this way file a proper bug report on these issues. But this article gave me an idea how can I work around these service outages by notifying myself on segfaults.

I did the following in order to receive e-mail notification when some daemon segfaults. (On Debian Etch with syslog-ng.)

1) Edit /etc/syslog-ng/syslog-ng.conf and add the following statements to the proper sections:

destination df_segfault { file("/var/log/segfault.log"); };
filter f_segfault { match("segfault at"); };
log {
        source(s_all);
        filter(f_segfault);
        destination(df_segfault);
};
Note that the order of log statements _does_ matter. Beware not to match the log messages of incrond and your own scripts to avoid loops.

2) Make syslog-ng reread its config file:

killall -HUP syslog-ng
Right now you have a log file named /var/log/segfault.log. (Though it is not created while the first log message does not arrive.) syslog-ng writes the log messages of the linux kernel containing 'segfault at' to this file.

A typical segfault message:
Feb 28 13:32:45 pbx kernel: ulogd[2125]: segfault at ffffffffffffffff rip 00002b7890314b5e rsp 00007fff1acee0f0 error 6

3) Install incron as mentioned in the article above:

vi /etc/apt/sources.list
apt-get update
apt-get install debian-backports-keyring
apt-get install incron

4) Configure incron tables: Create /etc/incron.d/segfault with the following contents:

/var/log/segfault.log IN_MODIFY /usr/local/bin/notify.log.change.sh $@

5) Create the script referenced in incron table (/usr/local/bin/notify.log.change.sh):
(Don't forget to change the e-mail address hardcoded in the script.)

#! /bin/bash
LOG="$1"
HOST="$(/bin/hostname -f)"
{ echo "last lines of '$LOG' at '$HOST': "; echo; tail "$LOG" 2>/dev/null || echo "'$LOG' does not exist" ; } | /usr/bin/mail -s "segfault on $HOST" your@email-address.tld
Note: this little script sends you an e-mail with the subject "segfault on $HOST" and the last 10 lines of the watched config file.

Add x bit:
chmod 755 /usr/local/bin/notify.log.change.sh

6) And finally execute this command for a setting of the mail program (not to prompt for cc: addresses)

echo 'unset askcc' >>/root/.mailrc
Done. Now every log message containing 'segfault at' triggers an e-mail. Not recommended for developer machines. ;-)

[ Parent | Reply to this comment ]

Posted by impact24 (58.71.xx.xx) on Thu 6 Mar 2008 at 12:55
[ Send Message ]
Wow, that's really useful stuff. Great artcile and great comment just above me (which is like another useful article altogether). I'll probably do the same thing to the experimental servers at work which happen to spew a segfault from time to time.

[ Parent | Reply to this comment ]

Posted by Anonymous (89.102.xx.xx) on Sun 9 Mar 2008 at 16:50
You could also try monit (http://www.tildeslash.com/monit/). It monitors you processes and restarts them in case they fail or stop responding for some reason.

[ Parent | Reply to this comment ]

Posted by Anonymous (77.192.xx.xx) on Thu 20 Mar 2008 at 14:46
hi,

I was looking for a way to synchronise two filesystems in a kind of real time (or closed to) design.

I did look at inotify but didn't find howto use it simply (without lot of development). I just see this article and discovered incron.

Do you think i can use incron with rsync/scp/unison/home_mode_script for synchronize two hosts when any action on the filesystem occurred ?

Regards,
Mathieu

[ Parent | Reply to this comment ]

Posted by Steve (80.68.xx.xx) on Thu 20 Mar 2008 at 15:09
[ Send Message | View Steve's Scratchpad | View Weblogs ]

Possibly.

More likely using rsync for a single-direction, or unison for bi-way syncing is you want.

Steve

[ Parent | Reply to this comment ]

Posted by Anonymous (80.248.xx.xx) on Tue 29 Apr 2008 at 23:08
Yep, that's easy. First you need to set up rsync and ssh to authentificate with keys. Beware of security issues!

Here is my quick n dirty incrontab config:

incrontab -l :

/var/www/download IN_DELETE,IN_CLOSE_WRITE /user/sync/rsync_xyz.sh

rsync_xyz.sh:
#!/bin/bash

SOURCEPATH="/var/www/download/"
DESTPATH="/var/www/download"
DESTHOST=10.xx.xx.xx
DESTUSER=xyz
LOGFILE="rsync_xyz.log"

echo $.\n\n. >> $LOGFILE
rsync -av --delete --rsh=ssh $SOURCEPATH $DESTUSER@$DESTHOST:$DESTPATH 2>&1 >> $LOGFILE
echo "Completed at: `/bin/date`" >> $LOGFILE

[ Parent | Reply to this comment ]

Posted by Anonymous (88.96.xx.xx) on Thu 1 May 2008 at 14:48
Summery should be spelt Summary

[ Parent | Reply to this comment ]

Posted by Steve (82.41.xx.xx) on Thu 1 May 2008 at 15:08
[ Send Message | View Steve's Scratchpad | View Weblogs ]

Thanks. Fixed now.

Steve

[ Parent | Reply to this comment ]

Sign In

Username:

Password:

[Register|Advanced]

 

Flattr

 

Current Poll

Which init system are you using in Debian?






( 1626 votes ~ 7 comments )