New User? Register here - Existing Users: Username: Password: [Advanced Login]

 

 

Current Poll

Your preferred Interactive shell?









( 1216 votes ~ 14 comments )

 

Weblog entry #32 for Steve

See all weblog entries made by Steve.

XML logo

#32
System monitoring with monit
Posted by Steve on Mon 10 Oct 2005 at 12:30
Tags: none.

After fixing up and posting hardik's article on Monit I've setup my own system monitoring.

I'm monitoring:

  • Apache2
  • Bind
  • ClamAV
  • Exim4
  • MySQL
  • OpenSSH

Here's the /etc/monit/monitrc file I used:

check process apache with pidfile /var/run/apache2.pid
   start program = "/etc/init.d/apache2 start"
   stop  program = "/etc/init.d/apache2 stop"
   if failed host 127.0.0.1 port 80 
        protocol http then restart
   if 5 restarts within 5 cycles then timeout

check process mysql with pidfile /var/run/mysqld/mysqld.pid
   group database
   start program = "/etc/init.d/mysql start"
   stop program = "/etc/init.d/mysql stop"
   if failed host 127.0.0.1 port 3306 then restart
   if 5 restarts within 5 cycles then timeout

check process sshd with pidfile /var/run/sshd.pid
   start program  "/etc/init.d/ssh start"
   stop program  "/etc/init.d/ssh stop"
   if failed port 22 protocol ssh then restart
   if 5 restarts within 5 cycles then timeout

check process named with pidfile /var/run/named.pid
   start program = "/etc/init.d/bind start"
   stop program = "/etc/init.d/bind stop"
   if failed host 127.0.0.1 port 53 type tcp then alert
   if failed host 127.0.0.1 port 53 type udp then alert
   if 5 restarts within 5 cycles then timeout

check process exim4 with pidfile /var/run/exim4/exim.pid
   start program = "/etc/init.d/exim4 start"
   stop program = "/etc/init.d/exim4 stop"
   if failed host 127.0.0.1 port 25 protocol smtp then alert
   if 5 restarts within 5 cycles then timeout

check process clamavd with pidfile /var/run/clamav/clamd.pid
   start program = "/etc/init.d/clamav-daemon start"
   stop  program = "/etc/init.d/clamav-daemon stop"
   if failed unixsocket /var/run/clamav/clamd.ctl then restart
   if 5 restarts within 5 cycles then timeout

I've been meaning to setup this for a while, but it is something I've never gotten around to.

[ 14 Comments | Add Comment | XML logo ]
<<< Previous Next >>>

 

Comments on this Entry

#1
Re: System monitoring with monit
Posted by hardik (61.95.xx.xx) on Mon 10 Oct 2005 at 13:41
[ Send Message ]
Thanks a lot Steve for fixing up my article and giving viewers clear picture. I know it was not too much neat and clean for posting. But i have tried........ Again thanks.


With Cheers,
Hardik Dalwadi.

[ Parent | Reply to this comment ]

#2
Re: System monitoring with monit
Posted by Steve (82.41.xx.xx) on Mon 10 Oct 2005 at 14:22
[ Send Message | View Steve's Scratchpad | View Weblogs ]

You're most welcome. Thanks to you for inspiring me to setup my own monitoring!

Steve
--

[ Parent | Reply to this comment ]

#3
Re: System monitoring with monit
Posted by Anonymous (213.164.xx.xx) on Mon 10 Oct 2005 at 16:39
> if failed host 127.0.0.1 port 25 protocol smtp then alert

:!

[ Parent | Reply to this comment ]

#4
Re: System monitoring with monit
Posted by Steve (82.41.xx.xx) on Mon 10 Oct 2005 at 23:46
[ Send Message | View Steve's Scratchpad | View Weblogs ]

Yeah I guess that's a little strange.

If SMTP is broken on port 25 then mail will not be delivered via the first mailserver - but monit can be configured to use more than one SMTP server, so all is good:

 set mailserver mail.tildeslash.com, mail.foo.bar port 10025, 
     localhost with timeout 15 seconds

Here monit will first try to connect to the server ``mail.tildeslash.com'', if this server is down monit will try ``mail.foo.bar'' on port 10025 and finally ``localhost''.

(This is taken straight from the online manual.)

Steve
--

[ Parent | Reply to this comment ]

#5
Re: System monitoring with monit
Posted by Anonymous (213.164.xx.xx) on Tue 11 Oct 2005 at 12:45
It would be nice if links to DA.org articles had the title of the article in the hover text.

[ Parent | Reply to this comment ]

#6
Re: System monitoring with monit
Posted by Steve (82.41.xx.xx) on Tue 11 Oct 2005 at 12:47
[ Send Message | View Steve's Scratchpad | View Weblogs ]

I could certainly do that on the front page for the 'Read More / Post Comments' link (along with the random links on the sidebar, and the 'next' + 'previous' articles) - is that what you mean?

Updating all the links contained in weblog engries, or comments would be a massive hand-editting job though, and wouldn't happen.

Steve
--

[ Parent | Reply to this comment ]

#7
Re: System monitoring with monit
Posted by Anonymous (213.164.xx.xx) on Tue 11 Oct 2005 at 16:02
I assumed it would be automatic :/
No worries.

[ Parent | Reply to this comment ]

#8
Re: System monitoring with monit
Posted by cyt (140.123.xx.xx) on Tue 29 Nov 2005 at 10:08
[ Send Message ]
I got a "Connection failed" status of my sshd process. But I'm sure my ssh connection is working. Any suggestions?

[ Parent | Reply to this comment ]

#9
Re: System monitoring with monit
Posted by cyt (140.123.xx.xx) on Tue 29 Nov 2005 at 11:52
[ Send Message ]
It seems because I use hosts.allow & hosts.deny to restrict ssh connection. After I disable them, it works.

[ Parent | Reply to this comment ]

#10
Re: System monitoring with monit
Posted by Steve (82.41.xx.xx) on Tue 29 Nov 2005 at 13:22
[ Send Message | View Steve's Scratchpad | View Weblogs ]

Just allow 127.0.0.1 and that will allow loopback connections - which should be sufficient to allow monitoring and keeping the SSH deamon secure.

Steve

[ Parent | Reply to this comment ]

#11
Re: System monitoring with monit
Posted by Anonymous (60.234.xx.xx) on Sun 15 Apr 2007 at 23:51
check process exim4 with pidfile /var/run/exim4/exim.pid
start program = "/etc/init.d/exim4 start"
stop program = "/etc/init.d/exim4 stop"
if failed host 127.0.0.1 port 25 protocol smtp then alert
if 5 restarts within 5 cycles then timeout

will this actually attempt to restart exim? or just alert?

I have this setup on my webserver but I currently have two checks for mail where one restarts and one alerts, but if above does both I can trim down the connects on port 25 to one.

Thanks,

Derek

[ Parent | Reply to this comment ]

#12
Re: System monitoring with monit
Posted by Steve (62.30.xx.xx) on Mon 16 Apr 2007 at 06:22
[ Send Message | View Steve's Scratchpad | View Weblogs ]

It will do both.

Steve

[ Parent | Reply to this comment ]

#13
Re: System monitoring with monit
Posted by Anonymous (66.93.xx.xx) on Sat 26 Jan 2008 at 04:07
Thanks Steve, I was getting around to running monit, when Lighttpd went down on me last week, then it became more urgent. So here's my monitrc for Lighttpd fronted by Varnish, acting in the reverse proxy/http accel role. Varn is listening on 80, then, if needed, it forwards things on to Lighttpd listening on 82. Lighty also listens on the standard 443 for HTTPS requests, so we check that as well.


check process varnish with pidfile /var/run/varnishd.pid
start program = "/etc/init.d/varnish start"
stop program = "/etc/init.d/varnish stop"
if cpu > 60% for 2 cycles then alert
if cpu > 80% for 5 cycles then restart
if totalmem > 200.0 MB for 5 cycles then restart
if children > 250 then restart
if loadavg(5min) greater than 10 for 8 cycles then stop
if failed host 127.0.0.1 port 80 protocol http
then restart
if 3 restarts within 5 cycles then timeout


check process lighttpd with pidfile /var/run/lighttpd.pid
start program = "/etc/init.d/lighttpd start"
stop program = "/etc/init.d/lighttpd stop"
if cpu > 60% for 2 cycles then alert
if cpu > 80% for 5 cycles then restart
if totalmem > 200.0 MB for 5 cycles then restart
if children > 250 then restart
if loadavg(5min) greater than 10 for 8 cycles then stop
if failed host 127.0.0.1 port 82 protocol http
then restart
if failed host 127.0.0.1 port 443 type tcpssl protocol http
with timeout 15 seconds
then restart
if 3 restarts within 5 cycles then timeout


Thanks again, you helped a lot, hopefully these notes will help others.

fak3r
http://fak3r.com

[ Parent | Reply to this comment ]

#14
Re: System monitoring with monit
Posted by tvl (83.163.xx.xx) on Tue 1 Jun 2010 at 13:12
[ Send Message ]
Does "if 5 restarts within 5 cycles then timeout" mean it stops trying forever? Or just til the "set deamon" time expires and it tries again with same restriction?

I would not want my server to stop trying to start ssh, its so mutch easyer to get to your shell with ssh then to drive to the serverlocation in the middle of the night with a keyboard and monitor in your backpack.

[ Parent | Reply to this comment ]

 

 

Flattr