Weblog entry #104 for simonw

apache2-mpm-worker and logrotate
Posted by simonw on Mon 6 Nov 2006 at 14:46
Tags: none.
Whilst investigating why Apache2 stopped when it rotated its log files on Sunday morning, I got this error on a Debian Sarge box.

/etc/init.d/apache2 restart
Forcing reload of web server: Apache2(98)Address already in use: make_sock: could not bind to address 0.0.0.0:8001
no listening sockets available, shutting down
Unable to open logs

Looks like there is a race condition in the "/etc/init.d/apache2 restart". It is intermittent (very), and I suspect dependent on the state of the mod_perl2 modules inside Apache, since the logging suggests it is having trouble killing off the old tasks in a timely manner. However because logrotate directs the output of the command to /dev/null I assume this information is normally thrown away when a problem occurs.

I keenly joined the Debian apache mailing list, but time is too short, and in the end I installed apache-mpm-prefork, as we've never had this issue on any other Apache server at work, which all have to use prefork due to the need to run PHP.

 

Comments on this Entry

Posted by bma (82.46.xx.xx) on Tue 7 Nov 2006 at 00:15
[ Send Message ]
We've been getting this error with apache2-mpm-prefork. Ugly hack solution - cronjob to make sure apache is running...

[ Parent | Reply to this comment ]

Posted by dkg (216.254.xx.xx) on Tue 7 Nov 2006 at 00:53
[ Send Message | View dkg's Scratchpad | View Weblogs ]
A cleaner solution to handle a sloppy apache restart is to stop using sysvinit scripts to supervise the apache daemons (a job which they do poorly at any rate), and use a tool that does proper process supervision like runit (or the non-free daemontools).

The runit-services package even has service configuration scripts for apache (maybe not apache2), but you can find apache2 and many other supervision scripts here.

As a bonus, you get clean process state and several other benefits over the old sysv scheme.

[ Parent | Reply to this comment ]

Posted by Steve (80.68.xx.xx) on Tue 7 Nov 2006 at 09:43
[ Send Message | View Steve's Scratchpad | View Weblogs ]

I've found people switching to runit having a lot of problems come package-upgrade time.

e.g. The last time I upgraded MySQL, the preinst script ran "mysql stop", and this didn't work. So the upgrade failed.

I had to manually stop the service, run the upgrade, then start it up again, since the Debian package knew nothing about using runsvc.

If the init scripts were supported then this would be much less pain.

Steve

[ Parent | Reply to this comment ]

Posted by dkg (64.80.xx.xx) on Tue 7 Nov 2006 at 17:06
[ Send Message | View dkg's Scratchpad | View Weblogs ]
Good point, Steve. Switching anything away from the defaults usually incurs a maintenance cost. This is an extra burden on the folks who want to make something non-default, but that's probably as it should be.

If you plan on installing something like runit, it's probably wise to replace your /etc/init.d scripts with scripts which invoke /usr/bin/sv to control the relevant service directory, so that tools which want to use the invoke-rc.d interface can do so without knowing that they're controlling a runit-supervised service underneath.

[ Parent | Reply to this comment ]

Posted by simonw (212.24.xx.xx) on Tue 7 Nov 2006 at 10:06
[ Send Message | View Weblogs ]
A neater solution might be to replace the restart settings in logrotate.d/apache2 with "apache2ctl restart" or "apache2ctl graceful". Which takes advantage of the applications functionality to avoid stopping if the config file has been corrupted (we really want the output if this happens, no piping stuff to /dev/null).

Since whilst Apache is running, it effectively keeps itself going, I don't think it calls for init style restarting of the service (that is built-in), the service will run as long as root doesn't tell it to stop, or some other catastrophic system event occurs.

The more crucial question is what is going wrong, since it seems that the "/proc/nnnn" directory is removed before the port Apache is bound to is freed. I suspect a well placed "sleep 1" would also resolve it from a practical perspective. However I hadn't realised that the Debian init script authors were writing scripts quite so intimately connected to the (current) kernel in use. Also these scripts are duplicating code rather than using shared functions, since several try and check for process death in this fashion.

Perhaps I'm missing something here (Debian usually has a good reason for these sort of things), but this all seems to be storing up trouble for the future, and the init scripts seem to be overly arcane in nature.

[ Parent | Reply to this comment ]

User Login

Username:

Password:

[ Advanced Login ]

Register Account

Quick Site Search