Weblog entry #8 for lykwydchykyn
See all weblog entries made by lykwydchykyn.
It's worked ok for quite some time, but recently apache2 has been slowing down and locking up on us, such that we have to restart the process. The log files only indicate that we need to raise the "maxclients" value, which I did (from whatever the default is (20?) to 30, then later to 150 because the problems persisted).
Are sarge's default settings too conservative? Should I have to tweak these out for a relatively low-traffic site like this?
Comments on this Entry
[ Send Message | View dkg's Scratchpad | View Weblogs ]
At a time when the server seems hung, what does a process listing show? are there really many hung apache processes? I would try:
ps -eFH | grep apacheAre there a ton of apache processes listed there? If so, what can you tell about what they're doing?
You might also be interested to look at the existing network connections on the box:
lsof -iAre there a ton of connections being held open by some particular IP address? What do you see?
If the steps above aren't illuminating, you might try attaching strace to one of the apache processes to see what it's doing while it's hung.
Please report back with what you find!
[ Parent | Reply to this comment ]
There's no telling if or when it will hang next, the last time there were about 13 days between problems; and unfortunately other coworkers know just enough to restart apache, so I will have to be the one to catch it if I want to diagnose anything!
I did a webalizer analysis of the logs and discovered that we're actually getting around 500-600 unique visitors a day, IIRC (that's the only stat I can remember at this moment). Is that excessive for the server settings, or just right?
Now for the embarassing question... how do I tell which apache server I'm using? I just installed apache2 from the repos, so whatever it runs by default is what I'm using.
[ Parent | Reply to this comment ]
[ Send Message | View dkg's Scratchpad | View Weblogs ]
[0 dkg@grunt dkg]$ apt-cache show apache2 | grep ^Depends: Depends: apache2-mpm-worker (= 2.0.54-5sarge1) | apache2-mpm-prefork (= 2.0.54-5sarge1) | apache2-mpm-perchild (= 2.0.54-5sarge1) [0 dkg@grunt dkg]$You can tell which one you've got installed with:
dpkg -l | grep apache2if your terminal is wider than 80 chars, and yer running sarge, and you want to see more data in the listing, you can use the COLUMNS environment variable to make the output wider:
COLUMNS=120 dpkg -l | grep apache2(that's for bash and similar shells if yer using tcsh or something else wacky like that, you'll need to set the environment variable in a different way).
Even if the web server isn't hung right now, you might want to try the commands i mentioned above to get a baseline for what things look like now. With 500-600 unique visitors a day (and let's say 5 page views per visitor? i'm making that up), we've got ~3000 page views, which is about one every half-minute. Even if all the requests came in over the course of a single hour (lunchtime, say), it's still less than 1/sec. For serving smallish static pages, there is absolutely no reason to need >20 MaxClients for this kind of load.
Anonymous below mentioned DNS timeouts: that's certainly a possibility. DNS timeouts can take a long time; if each apache2 log entry tries to do a reverse DNS lookup (to write down the name of the requesting site instead of the IP address), and DNS is misconfigured on your server, each request will hang for quite a long time. iirc, though, debian ships with HostnameLookups Off in /etc/apache2.conf, so unless you've turned that on for some reason, it's probably not the issue.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
1. We are using mpm-prefork
2. Page hits have been averaging 10k/day, though the average went down to 7.7k/day last month when the problems started. Data going out totalled about 1.4GB last month, but had been at 2.0 GB in prior months.
3. HostNameLookups is off
4. "ps -eFH | grep apache" shows me 10 healthy apache processes, at the moment.
5. I didn't get much from lsof -i. But I'm not checking at peak hours, maybe I should check this stuff later in the afternoon when things pick up.
Thanks for your help; heaven knows we have enough other issues on our network that the box itself could be innocent (we've had some hacker from China trying to DOS our main router off and on), I mostly just wanted to know what sort of load the default settings should be able handle, given an average piece of hardware.
[ Parent | Reply to this comment ]
Even if you hit "maxclients" it shouldn't cause those issues except the need to increase the setting and a few people who don't get in. Whole point of maxclients would be to avoid excessive resource consumptions, and thus slow downs and hangs, (at least I always assumed that is what it is for).
I think I did increase maxclients slightly, it is now 35 with pre-fork (not threaded - it is a PHP thing - don't ask).
I agree it smells, check the error log, and check the running processes, and check the access log for "weird stuff". Run chkrootkit, and use tcpdump to check the traffic is what you expect, for good measure.
[ Parent | Reply to this comment ]