Monitoring systems with munin
Posted by chris on Fri 26 Aug 2005 at 15:41
From the package description: Munin is a highly flexible and powerful solution used to create graphs of virtually everything imaginable throughout your network, while still maintaining a rattling ease of installation and configuration.
Getting munin
Munin is in the Debian archive in two parts - munin - the part that creates the monitoring graphs (you can think of this as the munin server), and munin-node - the munin client program.
So - if monitoring one machine installation should be a simple:
apt-get install munin munin-node
Configuration
Configuration is conducted via configuration files inside the directory /etc/munin:
- munin.conf
The configuration of the grapher/collector.
The default configuration is set up to monitor the local machine. Things that can be set here (defaults shown below)
# The data storage area dbdir /var/lib/munin # The place to put the generated web pages htmldir /var/www/munin # The log files logdir /var/log/munin # Temporary run files (pid file etc) rundir /var/run/munin # Where to look for the HTML templates tmpldir /etc/munin/templates
You can also setup hosts to check here - the default is for localhost:
[localhost.localdomain] address 127.0.0.1 use_node_name yesExamples are given in this file for adding other hosts.
- munin-node.conf
Setup of the node which server addresses can connect. For monitoring just your local machine the default is fine.
- plugin-conf.d/munin-node
Configuration of the plugins for this node. The syntax is simple:
[pluginname] param val param val
You can set user, group, etc for each plugin.
Allowed parameters are described at the top of the file.
- plugins/
A directory in which each file is a symlink to a real plugin in /usr/share/munin/plugins.
Any plugin linked in here will be checked for and displayed in the resulting web pages.
Add the plugins you want (e.g. if running exim4 then I'd add exim_mailqueue and exim_mailstats).
You'll need to set user/group rights in the munin-node conf file.
Most plugins can be run from the command line with the autoconf param to check if they can run - e.g.
./exim_mailstats autoconf yes
Running munin
Munin sets up a cron job via the file /etc/cron.d/munin which will run /usr/bin/munin-cron.
Running this file will poll each of the nodes - and then will create the graphs in /var/www/html which you can then browse under http://hostname.example.com/munin.
Examples can be found here: http://www.linpro.no/projects/munin/example/.
Keeping an eye on these graphs will help you to keep your server running healthily - and can give advance warning of problems to come.
[ Parent | Reply to this comment ]
I've done two things - obtained a spamstats plugin for monitoring spamassassin (seems to be available in the standard munin package but not in debian) and modified the exim_mailstats plugin to not mark a greylist rejection as a rejection.
Keep thinking of adding one to monitor white/grey/black listing.
I think the daftest plugin is the one that monitors uptime - you just get a rising graph :)
[ Parent | Reply to this comment ]
I'd definitely recommend it.
--
Sam Bashton
Bashton Ltd - Linux Consultancy
[ Parent | Reply to this comment ]
I think that for my use (two small PCs that serve websites for small groups of people) that munin will probably do. If I was going to monitor larger networks with SNMP enabled hardware etc - then cacti would probably be better.
[ Parent | Reply to this comment ]
I had created a Data Template for the Munin CPU graph and used that in some Data Sources.
The Cacti version is 0.8.4-2, a backport for Woody I built then.
Many people are using Munin to monitor extensive Networks. You can create ustoms views to collate e.g. all CPU graphs. (I never tried that myself.)
Munin has quite a few SNMP-based plugins, so people with that requirement should have a look at those. They can used to monitor Windows boxes.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
Cacti can read the RRDtool files created by Munin. I have set this up quite some time ago, so I don't remember the details, but the config is still there.Could you give some hints as to how you approached this? Cheers :)
[ Parent | Reply to this comment ]
1) Create a data template. One data source item per Munin RRD file. E.g. for a CPU graph, define cpu_idle, cpu_nice, cpu_system and cpu_user. Enter the values as used by Munin, e.g. step is 300.
2) Create a host template and associate the data template.
3) Create a graph template. For each value (idle, nice, system, user) define an AREA and three GRPINTs, LAST, AVERAGE, and MAX.
4) Define data sources. Enter the correct Data Source Pathes to the Munin RRD files. Again, do this for each value, idle, nice, system, user.
5) Define a graph using the data sources and the graph template.
I wrote this while inspecting the unused but still existing Cacti test installation. So this probably won't work. Please post your recipe when you got thi working.
Lupe
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
I'd guess - if your bug is on the BTS then wait for it to be fixed (or send a patch :)). If adding the patch from the BTS still doesn't work (and you think you have a new bug) then send it in with reportbug.
[ Parent | Reply to this comment ]
Luckily some SNMP can be mixed in. Not that that's exactly a picnic to run and feel safe with either.
[ Parent | Reply to this comment ]
Stripping a Unix-system of Perl is quite seldom these days and I honestly can't see the advantage. If you feel that running som snmpd is more secure in some way - I really can't see how.
[ Parent | Reply to this comment ]
SVK on the other hand, yikes.
[ Parent | Reply to this comment ]
solution: apt-get install libwww-perl
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
I am new to munin package and i need some one help.I have installed munin and default local host is working fine now i need to add other hosts for that i have added following details
i am try to add new node in munin.conf file like this
[director1]
address 212.13.5.133
use_node_name yes
Now in munin-node.conf i have added new node ipaddress
allow ^212\.13\.5\.133$
If i want to add which plugins need to add for this host where i can define.
and grapha automatically appears i need to modify any html code.
thanks for your help
[ Parent | Reply to this comment ]
However, it seems you need to set up munin-client on your client machines (the hosts you'll want to monitor), so that they have a way of reporting their status to the server.
[ Parent | Reply to this comment ]
thanks for your help
[ Parent | Reply to this comment ]
The server uses munin.conf
The client uses munin-node.conf
Quite often the server is also a client - in which case both files are used.
The client (munin-node.conf) needs both it's own IP address and the server's IP address.
The server needs all client IP addresses in the munin.conf file as you have described.
[ Parent | Reply to this comment ]
is there any configuration i need to do in server side to see my client machine graphs what configuration need to be done to see graphs.
thanks for your help
[ Parent | Reply to this comment ]
OK - here's a sample 2 machine config (it's an anonymized copy of what I have running):
On the server /etc/munin.conf contains
dbdir /var/lib/munin
htmldir /var/www/munin
logdir /var/log/munin
rundir /var/run/munin
tmpldir /etc/munin/templates
[host1.domain.tld]
address xxx.xxx.xxx.xxx
[host2.domain.tld]
address yyy.yyy.yyy.yyyAnd the munin-node.conf contains
log_level 4 log_file /var/log/munin/munin-node.log port 4949 pid_file /var/run/munin/munin-node.pid background 1 setseid 1 host * user root group root setsid yes ignore_file ~$ ignore_file \.bak$ ignore_file %$ ignore_file \.dpkg-(tmp|new|old|dist)$ ignore_file \.rpm(save|new)$ host_name host1.domain.tld allow ^127\.0\.0\.1$ allow ^xxx\.xxx\.xxx\.xxx$
On the second machine (client only) munin-node.conf contains
log_level 4 log_file /var/log/munin/munin-node.log port 4949 pid_file /var/run/munin/munin-node.pid background 1 setseid 1 host * user root group root setsid yes ignore_file ~$ ignore_file \.bak$ ignore_file %$ ignore_file \.dpkg-(tmp|new|old|dist)$ ignore_file \.rpm(save|new)$ host_name host2.domain.tld allow ^127\.0\.0\.1$ allow ^xxx\.xxx\.xxx\.xxx$ allow ^yyy\.yyy\.yyy\.yyy$
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
#telnet localhost 4949
Trying 127.0.0.1...
Connected to localhost.localdomain.
Escape character is '^]'.
# munin node at test.domain.net
Connection closed by foreign host.
thanks for your help
[ Parent | Reply to this comment ]
On a client you should see munin-node.log under /var/log/munin
On a server you should see munin-graph.log, munin-html.log, munin-limits.log and munin-update.log
I guess the important ones are:
munin-node.log should show the connections that are occuring.
munin-graph.log should show info on the services being graphed.
munin-html.log should show info on the html being generated.
On the server there should be a cron job /etc/cron.d/munin
On clients /etc/cron.d/munin-node
Some kind of cron daemon must be running
[ Parent | Reply to this comment ]
i have check the log files
munin-node.log - client
006/01/25-10:42:26 CONNECT TCP Peer: "127.0.0.1:1063" Local: "127.0.0.1:4949"
2006/01/25-10:51:29 Server closing!
Process Backgrounded
2006/01/25-10:51:33 MyPackage (type Net::Server::Fork) starting! pid(9193)
Binding to TCP port 4949 on host *
2006/01/25-10:52:43 CONNECT TCP Peer: "172.29.5.132:1715" Local: "172.14.5.23:4949"
munin-graph.log - server
Jan 25 11:45:13 - Processing domain: test1
Jan 25 11:45:13 - Graphed service : open_inodes (0.11 sec * 4)
Jan 25 11:45:15 - Graphed service : iostat (0.12 sec * 4)
Jan 25 11:45:15 - Graphed service : forks (0.10 sec * 4)
Jan 25 11:45:16 - Graphed service : memory (0.22 sec * 4)
Jan 25 11:45:16 - Graphed service : vmstat (0.11 sec * 4)
Jan 25 11:45:16 - Processed node: test1 (3.12 sec)
Jan 25 11:45:16 - Processed domain: test1 (3.12 sec)
munin-html.log - server
Jan 25 11:50:22 - processing domain: test1
Jan 25 11:50:22 - processing node: test1
Jan 25 11:50:22 - processing service: iostat
Jan 25 11:50:22 - processing service: entropy
Jan 25 11:50:22 - processing service: forks
Jan 25 11:50:22 - processing service: iostat
Jan 25 11:50:22 - processing service: memory
Jan 25 11:50:23 - processing service: vmstat
From these logs it is trying to graph and trying to create html file but i am not able to see any graphs as i menctioned before when i try to run telnet to localmachine i am getting connection closed by forein host error is there anything to do with this and my cron files are running fine
[ Parent | Reply to this comment ]
Do you have any png/html files under /var/www/munin/DOMAINNAME ?
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
chown -R munin.munin /var/lib/munin/
[ Parent | Reply to this comment ]
I then proceeded to hook it up with hddtemp so that i could monitor the drive temperatures of my boxen (as per this great d-a article).
This required me to do three things:
- apt-get install smartmontool
which is apparently needed by munin's hddtemp_smartctl plugin ; - and ln -s /usr/share/munin/plugins/hddtemp_smartctl /etc/munin/plugins/hddtemp_smartctl
.. which 'activates' the plugin ; - and /etc/init.d/munin-node restart to apply these changes.
Now that this is done, I have temperature graphs for all my sensor-enabled hard drives, which is pretty great. But ultimately staring at a graph all day isn't all that efficient, so what I'd like to do now is have munin send me an e-mail when the temperature of any drive hits, say, 42ºC.
Unfortunately the official documentation pages are kinda incomplete at the moment, so I'm not sure what to do.. there seems to be a page specifically about sending alerts, but it's not too clear..
Am I supposed to just dive in and start modifying the actual /etc/munin/plugins/hddtemp_smartctl plugin file (it's Perl) to stick in the warning trigger values?
Or is there a cleaner, more global way of doing this somewhere?
Can anyone here provide an example?
Thanks.
[ Parent | Reply to this comment ]
Take a look at http://www.debian-administration.org/articles/299
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
contact.email.command mail -s "Munin-notification for ${var:group} :: ${var:host}" your@email.address.here
The problem is, I'm still not getting any e-mail alerts even though two of my hard drives have reached 'critical' capacity levels, and my Munin-generated pages recognize this fact by marking said values with an orange background.
Does anyone have a clue as to what it is that I am missing?
(and while I do appreciate all advice from d-a users, I'd like to stress that Nagios is not an answer in my opinion -- given the fact that Munin should already be able to send a simple mail by itself, I think installing Nagios only for this purpose would be like installing Windows 98 for its screensavers..)
[ Parent | Reply to this comment ]
i have tried this, but i only get a temp-graph for my hda.
But i have 5 sata drives, too. (sda, sdb...)
Anybody any ideas?
Thank you very much,
Rolleff
[ Parent | Reply to this comment ]
I have configured munin on my network, which has one server and 40 nodes and I've made an easy-to-use manual, a quick reference with all important things with munin. It's been written in Spanish, but I hope it would be userful. It's here:
http://www.n1mh.org/weblog/index.php/2006/01/30/monitorizando-red es-con-munin/
Thanks for be my first reference!
Cheers,
diego martínez
[ Parent | Reply to this comment ]
munin installs good
but when I try to install munin-mod , after "Initializing plugins.." it stops for a long time , the only way to get rid of it is Ctrl-C
I tried to find out what the problem is? and removed all of the plugins from the directory , installation completed , then I copied one plugin back to the directory , still doing well , but only with that plugin , but each of other plugins makes it hang.
Any Ideas?
Regards
[ Parent | Reply to this comment ]
I ran 'strace apt-get install munin-node' and notices that right after it printed "Initializing Plugins...", it did a fork&wait! A little snooping showed the file it was executing: munin-node-configure.
If you want to see progress as it runs through the configuration, edit the file
/var/lib/dpkg/info/munin-node.postinst
(or where ever it is on your system)
And change the line:
munin-node-configure --shell |sh
to
munin-node-configure --shell --debug
...or just wait a LONG time.
Enjoy.
[ Parent | Reply to this comment ]
step by step guide for munin configuration in debian
[ Parent | Reply to this comment ]
Two things that I find useful
As root
munin-node-configure --suggest
which will show you what you could be running and if munin reckons it would work
and
munin-node-configure --suggest --shell
which will generate a list of ln commands that can be run in a shell to install the services that munin said "suggest: yes" to in the above output
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
rm /etc/munin/plugins/<plugin-name>
sudo /etc/init.d/munin-node restart
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
Need help in configuring munin on debian. Installed munin server and node on debian env and munin-node alone on RedHat Linux. Add the necessary comments to the config file.
munin server config file contents:
# Example configuration file for Munin, generated by 'make build'
# The next three variables specifies where the location of the RRD
# databases, the HTML output, and the logs, severally. They all
# must be writable by the user running munin-cron.
dbdir /var/lib/munin
htmldir /var/www/munin
logdir /var/log/munin
rundir /var/run/munin
# Where to look for the HTML templates
tmpldir /etc/munin/templates
# Make graphs show values per minute instead of per second
#graph_period minute
# Drop somejuser@fnord.comm and anotheruser@blibb.comm an email everytime
# something changes (OK -> WARNING, CRITICAL -> OK, etc)
#contact.someuser.command mail -s "Munin notification" somejuser@fnord.comm
#contact.anotheruser.command mail -s "Munin notification" anotheruser@blibb.comm
#
# For those with Nagios, the following might come in handy. In addition,
# the services must be defined in the Nagios server as well.
#contact.nagios.command /usr/sbin/send_nsca -H nagios.host.com -c /etc/send_nsca.cfg
# a simple host tree
[localhost.localdomain]
address 127.0.0.1
use_node_name yes
[cu088.cubit.maa.collab.net]
address 208.75.198.98
use_node_name yes
munin-node server config file
#
# Example config-file for munin-node
#
log_level 4
log_file /var/log/munin/munin-node.log
port 4949
pid_file /var/run/munin/munin-node.pid
background 1
setseid 1
# Which port to bind to;
host *
host 127.0.0.1
user root
group root
setsid yes
# Regexps for files to ignore
ignore_file ~$
ignore_file \.bak$
ignore_file %$
ignore_file \.dpkg-(tmp|new|old|dist)$
ignore_file \.rpm(save|new)$
# Set this if the client doesn't report the correct hostname when
# telnetting to localhost, port 4949
#
#host_name localhost.localdomain
host_name cu088.cubit.maa.collab.net
# A list of addresses that are allowed to connect. This must be a
# regular expression, due to brain damage in Net::Server, which
# doesn't understand CIDR-style network notation. You may repeat
# the allow line as many times as you'd like
allow ^127\.0\.0\.1$
allow ^208\.75\.198\.98$
munin-node client config file contents
client hostname : cu088.cubit.maa.collab.net
#
# Example config-file for munin-node
#
log_level 4
log_file /var/log/munin/munin-node.log
port 4949
pid_file /var/run/munin/munin-node.pid
background 1
setseid 1
# Which port to bind to;
host *
user root
group root
setsid yes
# Regexps for files to ignore
ignore_file ~$
ignore_file \.bak$
ignore_file %$
ignore_file \.dpkg-(tmp|new|old|dist)$
ignore_file \.rpm(save|new)$
# Set this if the client doesn't report the correct hostname when
# telnetting to localhost, port 4949
#
#host_name lynx.linpro.no
host_name cu088.cubit.maa.collab.net
# A list of addresses that are allowed to connect. This must be a
# regular expression, due to brain damage in Net::Server, which
# doesn't understand CIDR-style network notation. You may repeat
# the allow line as many times as you'd like
allow ^127\.0\.0\.1$
allow ^10\.2\.10\.140$
~
Contents from the log files:
munin-html.log :
Oct 24 07:40:02 - Starting munin-html, checking lock
Oct 24 07:45:02 - Starting munin-html, checking lock
munin-node.log
2007/10/24-21:02:27 CONNECT TCP Peer: "127.0.0.1:42871" Local: "127.0.0.1:4949"
2007/10/24-21:06:10 CONNECT TCP Peer: "127.0.0.1:44686" Local: "127.0.0.1:4949"
2007/10/24-22:25:16 CONNECT TCP Peer: "127.0.0.1:55134" Local: "127.0.0.1:4949"
2007/10/24-22:25:19 Plugin "exim_mailstats" exited with status 1024. --received.value 2410
completed.value 2412
rejected.value 0
--
2007/10/24-22:30:09 CONNECT TCP Peer: "208.75.198.98:54654" Local: "10.2.10.140:4949"
2007/10/24-22:30:19 Connection timed out.
timeout at /usr/sbin/munin-node line 588.
2007/10/24-22:30:42 CONNECT TCP Peer: "10.2.10.139:32786" Local: "10.2.10.140:4949"
2007/10/24-22:35:05 CONNECT TCP Peer: "10.2.1.105:36225" Local: "10.2.10.140:4949"
2007/10/24-22:54:56 CONNECT TCP Peer: "127.0.0.1:49041" Local: "127.0.0.1:4949"
2007/10/24-22:54:58 Plugin "exim_mailstats" exited with status 1024. --received.value 2425
completed.value 2427
rejected.value 0
munin-graph.log
Oct 24 21:06:15 - Starting munin-graph
Oct 24 21:06:15 - Processing domain: cubit.maa.collab.net
Oct 24 21:06:15 - Graphed service : open_inodes (0.56 sec * 4)
Oct 24 21:06:16 - Graphed service : sendmail_mailtraffic (0.34 sec * 4)
Oct 24 21:06:16 - Graphed service : if_err_eth0 (0.32 sec * 4)
Oct 24 21:06:16 - Graphed service : entropy (0.25 sec * 4)
Oct 24 21:06:17 - Graphed service : irqstats (1.01 sec * 4)
Oct 24 21:06:18 - Graphed service : sendmail_mailstats (0.50 sec * 4)
Oct 24 21:06:18 - Graphed service : processes (0.29 sec * 4)
Oct 24 21:06:18 - Graphed service : if_eth0 (0.38 sec * 4)
Oct 24 21:06:19 - Graphed service : df (0.49 sec * 4)
Oct 24 21:06:20 - Graphed service : netstat (0.68 sec * 4)
Oct 24 21:06:20 - Graphed service : interrupts (0.43 sec * 4)
Oct 24 21:06:20 - Graphed service : swap (0.38 sec * 4)
Oct 24 21:06:21 - Graphed service : load (0.30 sec * 4)
Oct 24 21:06:21 - Graphed service : sendmail_mailqueue (0.32 sec * 4)
Oct 24 21:06:22 - Graphed service : ntp_mgr_maa_corp_collab_net (0.53 sec * 4)
Oct 24 21:06:22 - Graphed service : hddtemp_smartctl (0.32 sec * 4)
Oct 24 21:06:23 - Graphed service : cpu (0.88 sec * 4)
Oct 24 21:06:23 - Graphed service : df_inode (0.50 sec * 4)
Oct 24 21:06:24 - Graphed service : if_eth1 (0.41 sec * 4)
Oct 24 21:06:24 - Graphed service : if_err_eth1 (0.40 sec * 4)
Oct 24 21:06:25 - Graphed service : open_files (0.42 sec * 4)
Oct 24 21:06:25 - Graphed service : forks (0.32 sec * 4)
Oct 24 21:06:25 - Graphed service : iostat (0.40 sec * 4)
Oct 24 21:06:27 - Graphed service : memory (1.68 sec * 4)
Oct 24 21:06:27 - Graphed service : vmstat (0.38 sec * 4)
Oct 24 21:06:30 - Graphed service : nfs_client (2.32 sec * 4)
Oct 24 21:06:30 - Processed node: cu088.cubit.maa.collab.net (14.83 sec)
Oct 24 21:06:30 - Processed domain: cubit.maa.collab.net (14.83 sec)
Oct 24 21:06:30 - Processing domain: localdomain
Oct 24 21:06:30 - Graphed service : open_inodes (0.50 sec * 4)
Oct 24 21:06:31 - Graphed service : if_err_eth0 (0.48 sec * 4)
Oct 24 21:06:31 - Graphed service : entropy (0.42 sec * 4)
Oct 24 21:06:33 - Graphed service : irqstats (1.72 sec * 4)
Oct 24 21:06:33 - Graphed service : processes (0.19 sec * 4)
Oct 24 21:06:33 - Graphed service : if_eth0 (0.25 sec * 4)
Oct 24 21:06:34 - Graphed service : df (0.42 sec * 4)
Oct 24 21:06:34 - Graphed service : interrupts (0.26 sec * 4)
Oct 24 21:06:34 - Graphed service : netstat (0.44 sec * 4)
Oct 24 21:06:35 - Graphed service : swap (0.23 sec * 4)
Oct 24 21:06:35 - Graphed service : load (0.18 sec * 4)
Oct 24 21:06:36 - Graphed service : nfsd (1.45 sec * 4)
"munin-graph.log" 56 lines, 3380 characters
munin-node file under plugin-conf.d
# This file is used to configure how the plugins are invoked.
#
# user <user> # Set the user to run the plugin as.
# group <group> # Set the group to run the plugin as.
# command <command> # Run <command> instead of the plugin. %c expands to
# This file is used to configure how the plugins are invoked.
#
# user <user> # Set the user to run the plugin as.
# group <group> # Set the group to run the plugin as.
# command <command> # Run <command> instead of the plugin. %c expands to
# what would normally be run.
# env.<variable> # Sets <variable> in the plugin's environment, see the
# individual plugins to find out which variables they
# care about.
[apt]
user root
[courier_mta_mailqueue]
group daemon
[courier_mta_mailstats]
group adm
[courier_mta_mailvolume]
group adm
[cps*]
user root
[exim_mailqueue]
group mail, (Debian-exim)
[exim_mailstats]
group mail, adm
[fw_conntrack]
user root
[hddtemp_smartctl]
user root
Tried the telnet but the command stating the
telnet 208.75.198.98 4949
Trying 208.75.198.98...
Connected to 208.75.198.98.
Escape character is '^]'.
# munin node at cu088.cubit.maa.collab.net
Connection closed by foreign host.
Please do the needful. Thanks
[ Parent | Reply to this comment ]
ERROR: Could not rename file: No such file or directory
in /var/log/munin/munin-update.log
Any hints? Thanks !!
[ Parent | Reply to this comment ]
You need to build a munin-plugins-contrib package from the source. Instructions are in the NEWS.Debian file.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]