The Penguinlinux.com Blog

After so much work and research I was able to get MRTG working for penguincares by using the following urls

http://www.linux-sottises.net/en_mrtg.php

and

http://www.linuxhomenetworking.com


Introduction

This page gives some basic information about installation and configuration of MRTG under Linux. The reference page is MRTG.

Necessary tools and libraries

You need:

a C compiler (gcc or ecgs…)
a not too old perl version (at least 5.004_4)
snmp running as a daemon (Simple Network Management Protocol)
All these tools are usually available in most distributions. For snmp, I use ucd-snmp whichi is available in some distributions as a package. You can have a look at: http://net-snmp.sourceforge.net/.

Jean-Louis Noel proposed a script netstat for measuring interfaces input/output which does not require snmp. I think this is safer and easier to use this script if you do not need advanced function required by snmp. See further how to use it.

You also need gd from Thomas Boutell (www.boutell.com). I have here gd 1.8.4 .

To install gd, you need libpng, zlib and jpeg 6b at least. These libraries are generally available in most distributions. You can have a look at: Sinon, voir ftp://ftp.freesoftware.com/pub/infozip/zlib/ for zlib, ftp://ftp.freesoftware.com/pub/png/src/ for libpng, and http://www.ijg.org/ for jpeg.

Finally you need MRTG sources. I have here MRTG 2.9.17.

Be careful!

Check that you have an empty (if you trust you LAN end snmp port is closed from outside) config file snmp.conf in /etc or /usr/share/snmp (or elsewhere…), or containing:
com2sec readonly localhost public
Some distribution provides a default config file which does not allow MRTG to work as:
com2sec paranoid default public

gd installation

So, you already installed zlib, libpng and jpeg (sorry, I do not give explanations for these libraries, their installation is standart).

Uncompress (tar -zxf gd-1.8.4.tar.gz) gd source in (for example) /usr/local/src/. Go to gd directory and edit the Makefile. This is the part to edit:

COMPILER=gcc
AR=ar
CFLAGS=-O -DHAVE_LIBPNG -DHAVE_LIBJPEG -DHAVE_LIBXPM -DHAVE_LIBTTF
LIBS=-lgd -lpng -lz -ljpeg -lm -lttf
INCLUDEDIRS=-I. -I/usr/include/freetype -I/usr/include/X11 \
-I/usr/X11R6/include/X11 -I/usr/local/include
LIBDIRS=-L. -L/usr/local/lib -L/usr/lib/X11 -L/usr/X11R6/lib
INSTALL_LIB=/usr/local/lib
INSTALL_INCLUDE=/usr/local/include
INSTALL_BIN=/usr/local/bin

Modify the lines COMPILER, CFLAGS, LIBS and INCLUDEDIR according to your configuration (read the comments in the Makefile and the readme.txt of gd, for freetype for instance). Setup INSTALL_LIB, INSTALL_INCLUDE et INSTALL_BIN if you want an install in other directories.

Run make and if there are no errors, run make install.

MRTG Installation

Uncompress (tar -zxf mrtg-2.9.17.tar.gz). From the MRTG directory run ./configure. If all is OK, you get a Makefile…after ordering MRTG CD (:-D). Now run make. Si le./configure a produit des messages d’erreur, lisez les, c’est qu’il vous manque des librairies ou outils de d鶥loppement.
make install will install mrtg in /usr/local/mrtg-2

All the necessary pogrammes are in /usr/local/mrtg-2/bin.

MRTG configuration

In the directory /usr/local/mrtg-2/bin, there is a tool cfgmaker which builds a “beginning” of the configuration file. Go to a temp directory and run:

/usr/local/mrtg-2/bin/cfgmaker public@machine_name >test

where machine_name is the hostname of your machine. You get the list of your interfaces in the file test and the “beginning of a configuration file”. You can see here the file test on my machine.

As you can see, there are 5 interfaces: loopback, dummy0, 2 ethernet and PPP0. For each interface, a name is proposed. In your final configuration file, you can (or should) change these names, you can also decide to not mention all these interfaces.

cfgmaker can also configure interfaces by their ip (option -ifref=ip). Run ./cfgmaker –help from the mrtg bin directory for a list of all options and have a look at all available documentation on MRTG website.

Chose a working directory (in your website root directory if you want ro give access to yout mrtg pages). Make a mrtg.cfg in this directory. Start with a line like WorkDir: /web/mrtg (or use a different directory fo logs, images and html with Logdir, Imagedir and Htmldir options, have a look at my config file) . Then copy the “pieces” of test which correspond to the interfaces you want to measure. Though in my file foo ppp0 was marked “unrealistic”, it is the one I use!!

For each interface, you have to chose a name for each “target” order. Mrtg will generate a “name.html” file. Do not forget to copy the *.png Mrtg images from the Mrtg directory to to you Workdir or you Imagedir.

To know all the available option, have a look at the doc directrory in Mrtg install directory.

Run “mrtg mrtg.cfg” and look what happens… if all is OK, Mrtg created html files in Workdir or Htmldir. The two first run of Mrtg generate warning messages, they disappear at the third run. Now, you can run mrtg from a cron job in /etc/crontab. The default time intervall is 5 minutes. If you want to change this, you have to use Intervall keyword and change you cron job accordingly.

indexmaker (available in /usr/local/mrtg-2/bin) allows to make a page with access to your Mrtg pages.

You should read the doc about [_] which defines default options and [^] which defines “prefix” for targets and [$] whiche defines “suffix” for targets.
An example is better than long explanations, have a look at my config file on linux-sottises.

MRTG for pings and other tricks

Mrtg permits to make charts for “anything”! using some scripts. Do not forget to make those script “x” by chmod u+x.
You can see all the results of these script on the page stats_mrtg.
Remark: mrtg stats has been stopped on April 17 2002 as linux-sottises uses MRTNK which is based on RRDTOOL, you can have a look at the graphics generated by MRTNK.
The current stable version of MRTNK is available on the software page.
The development version (which is always the one used at linux-sottises) is available here.
For the moment, only the french documentation is avaliable for the development version in the doc directory.

PING.
It is target [ping_nerim] of my config file.
You can download ping-nerim script which gives min et max over 3 pings.

APACHE
Mrtg can also count the hits number on you Apache server (server-status and ExtendStatus must be enabled in your apache config file)
It is target [apache] in my config file.
The perl script is webstat.pl. This script can also measure apache traffic, look at the section [apache_byte of my config file.

To measure separatly hits and and traffic/documents of virtual hosts, I use mod_watch www.snert.com/Software/mod_watch/
You must hace compiled apache yourself.
There are 2 options of compilation of mod_watch, as a dynamic module if apache has been compiled with dynamic modules support, or statically. I only use mod_watch as a dynamic module and only have explanations for this compilation.
After downloading and compiling it, edit the Makefile. Modify to path to you apache sources in APACHEDIR, the path to the scripts used by mod_watch in SCRIPTDIR, the path of virtual servers logs in SPOOLDIR.
If apache apxs is not in your path, you have to modify APXS a little further in the Makefile. For example, for me it is:

APXS=/usr/local/apache/bin/apxs
Then run make install-dynamic
Check that mod_watch has been added up in your apache config file and that the module is effectly in libexec directory of you apache install directory.
Then add in you apache config file:

SetHandler watch-info

Stop and start apache.
goto /watch-info on you server from your browser to check that mod_watch is active. Go to www.snert.com/Software/mod_watch/ for more details.
Have a look at my MRTG config file to have a look at the 4 mod_watch sections and stats_mrtg to see all the results.

UPTIME
For uptime I use perl script uptime.pl and the corresponding target is [uptime] in my config file.

PROCESS
To count the number of processes, I use the perl script stat.pl, target [procs] in my config file.

MEMORY
For memory, I use mem, target [mem] in my config file.

DNS
For DNS queries (bind V8), it is the same script as process number stat.pl, target [dns] in my config file.

CPU
For CPU, it is a bit more complex. The script is cpustat, target [cpu] in my config file. But, you need to install sysstat-4.0.3. Have a look at http://perso.wanadoo.fr/sebastien.godard/ systat author page. systat must be installed and regularly launched by cron.

NETWORK INTERFACES WITHOUT SNMP To measure interfaces without snmp, you can use netstat. Check cut, grep and uptime location. Usage is:
Target[target_ppp0]: `path_to_netstat target_ppp0 ppp0`
first parameter is the name of the target, the second one is the interface name.

David Du Serre-Telmon proposed another script netstat-ip which permits to call an interface by its IP.
Usage is straightforward: netstat target_name interface or netstat target_name ip ip_value.

OPEN FILES
A script proposed by 0_be_one counts the number of open files (you need lsof). The perl script is files.pl. Look at the section [files] of my config file.

DISK SPACE
A script from AIREY Romuald give disk space.
It is based on df command, example:

Filesystem 1k-blocks Used Available Use% Mounted on
/dev/sda3 8625436 3168832 5018432 39% /
/dev/sda1 23300 5193 16904 24% /boot
/dev/hda1 59084932 44493936 14590996 76% /mnt/idea
/dev/hdb1 38448276 9756360 26738816 27% /mnt/ideb
192.168.0.6:/ 3984704 934288 2848000 25% /pluto
192.168.0.7:/ 1921188 507476 1316120 28% /vador

option -m for Mo is used (for better readness and respect the max integer of MRTG, it is also advised to eliminate nfs file systems by -x nfs (if nfs servers are down, you will get problems with the script!).
df -m -x nfs :

Filesystem 1M-blocks Used Available Use% Mounted on
/dev/sda3 8423 3089 4907 39% /
/dev/sda1 23 6 16 24% /boot
/dev/hda1 57700 43452 14249 76% /mnt/idea
/dev/hdb1 37547 9528 26112 27% /mnt/ideb

The script checkdisk returns free and used disk space. Check df, cut and grep location.
checkdisk is used followod by the partition name, have a look at sections hda1, hdb1 and sda3 of my config file.
Have a look at the use of kMG which is required to avoid kMo in place of Go!!

DATA EXCHANGE AMOUNT ON AN INTERFACE
volume gives the amount of data exchanged on an interface.
As this tiny script is based on ifconfig, an interface down-up reset the statistics to 0,…, what I do not feel disturbing. The data appearing through ifconfig is also reset when some maximum is reached (I did no try to evaluate this maximum).
it is very easy to use: volume interface_name (have a look at volume_ppp0 of my config file and the results on stats_mrtg.

ACTIVE TCP CONNECTIONS WITH NETFILTER
This was contributed by David Du Serre-Telmon. The script is tcpcount.
use is:
tcpcount => total tcp connections through the gateway
tcpcount 21 => FTP connections
tcpcount 80 dst www.linux-sottises.net => web connections to www.linux-sottises.net
tcpcount 1214 src 192.168.0.2 => Kazaa connections fo user 192.168.0.2

THEN I USED ALSO THIS IMPORTANT SETUP PAGE FROM HOMELINUXNETWORKING.

Monitoring your system?s web performance can be done quite easily with a number of graphical tools available for Linux. These include MRTG for raw network traffic which is based on SNMP and Webalizer that monitors web site hits.

SNMP
What is SNMP?
Most routers and firewalls keep their operational statistics in Management Information Blocks (MIBs). Each statistic has an Object Identifier (OID) and can be remotely retrieved from the MIB via the Simple Network Management Protocol (SNMP). However, as a security measure, you need to know the SNMP password or “community string” to do so. There are a number of types of community strings, the most commonly used ones are the “Read Only” community string that only provides access for viewing statistics and system parameters. In many cases the “Read Only” community string or password is set to “public”. There is also a “Read Write” community string for not only viewing statistics and system parameters but also for updating the parameters too.

SNMP on a Linux Server
By default, RedHat Linux has the NetSNMP package installed to provide SNMP services. NetSNMP uses a configuration file /etc/snmp/snmpd.conf in which the community strings may be set. The version of the configuration file that comes with Net-SNMP is quite complicated. I suggest archiving it and using a much simpler version with only a single line containing the keyword “rocommunity” followed by the community string. Here is an example of how to do that.

? Save the old configuration file

[root@bigboy snmp]# cd /etc/snmp/
[root@bigboy snmp]# mv snmpd.conf snmpd.conf.old
[root@bigboy snmp]# vi snmpd.conf

? Enter the following line in the new configuration file to set the Read Only community string to “craz33guy”

rocommunity craz33guy

? Configure Linux to start SNMP services on each reboot with the chkconfig command:

[root@bigboy root]# chkconfig –level 345 snmpd on
[root@bigboy root]#

? You can then start SNMP to load the current configuration file.

[root@bigboy root]# /etc/init.d/snmpd start
Starting snmpd: [ OK ]
[root@bigboy root]#

? Test whether SNMP can read the “system” and “interface” information MIB

[root@bigboy snmp]# snmpwalk -v 1 -c craz33guy localhost system
SNMPv2-MIB::sysDescr.0 = STRING: Linux bigboy 2.4.18-14 #1 Wed Sep 4 11:57:57 EDT 2002 i586
SNMPv2-MIB::sysObjectID.0 = OID: NET-SNMP-MIB::netSnmpAgentOIDs.10
SNMPv2-MIB::sysUpTime.0 = Timeticks: (425) 0:00:04.25
SNMPv2-MIB::sysContact.0 = STRING: root@localhost
SNMPv2-MIB::sysName.0 = STRING: bigboy
…

…

…

[root@bigboy snmp]# snmpwalk -v 1 -c craz33guy localhost interface
IF-MIB::ifNumber.0 = INTEGER: 3
IF-MIB::ifIndex.1 = INTEGER: 1
IF-MIB::ifIndex.2 = INTEGER: 2
IF-MIB::ifIndex.3 = INTEGER: 3
IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: wlan0
IF-MIB::ifDescr.3 = STRING: eth0

…

…

…

[root@bigboy snmp]#

Note: In this case we were polling localhost. You can poll any SNMP aware network device with SNMP enabled. All you need is the IP address and SNMP read only string and you?ll be able to get similar results.

Now that we know SNMP is working correctly on your Linux server, we can configure a SNMP statistics gathering software package such as MRTG to create online graphs of your traffic flows.

MRTG
What is MRTG?
MRTG (Multi Router Traffic Grapher) is a public domain package for producing graphs of various types of router statistics via a web page. You can easily create graphs of traffic flow statistics through your home network’s firewall / router or even your Linux box’s NIC cards using MRTG. The product is available from the MRTG website and also on your distribution CDs.

Download and Install The MRTG Packages
Most RedHat Linux software products are available in the RPM format. Downloading and installing RPMs isn?t hard. If you need a refresher, the chapter on RPMs covers how to do this in detail. The latest version of the RPM for RedHat 8.0 is:

mrtg-2.9.17-8.i386.rpm

o You can install the package like this:

[root@bigboy tmp]# rpm -Uvh mrtg-2.9.17-8.i386.rpm

o You will also need to have a webserver package installed for MRTG to work. The RedHat RPM version seems to work with Apache 1.X. The most current version as of this writing was apache 1.3.23 - 11. This is available from the RedHat website or your installation CDs. Install apache using the following command.

[root@bigboy tmp]# rpm -Uvh apache-1.3.23-14.i386.rpm

o MRTG runs automatically upon startup, but you?ll need to configure Apache to start at boot using the chkconfig command:

[root@bigboy tmp]# chkconfig –level 35 httpd on

o Here?s how to start/stop/restart Apache after booting:

[root@bigboy tmp]# /etc/init.d/httpd start
[root@bigboy tmp]# /etc/init.d/httpd stop
[root@bigboy tmp]# /etc/init.d/httpd restart

By default Apache expects the HTML files for your website to be located in /var/www/html. MRTG will place its HTML files in /var/www/html/mrtg.

Configuring MRTG
By default, MRTG will map the inbound and outbound data throughput rates on the device it is polling. There are ways to specify other OIDs such as CPU and memory usage, but this is beyond the scope of this book. We?ll be discussing the default configuration.

When the MRTG RPM is installed it creates a directory called /etc/mrtg in which all future configuration files are stored. Here are the steps you need to go through to create new configuration files.

o In this example we’ll use MRTG?s cfgmaker command to create a configuration file named localhost.cfg for the server “bigboy” using a read only community string of craz33guy. All data files will be placed in the directory /var/www/html/mrtg/stats.

[root@bigboy tmp]# cfgmaker –output=/etc/mrtg/localhost.cfg \

-ifref=ip –global “workdir: /var/www/html/mrtg/stats” \

craz33guy@localhost

–base: Get Device Info on craz33guy@localhost:
–base: Vendor Id:
–base: Populating confcache
–snpo: confcache craz33guy@localhost: Descr lo –> 1
–snpo: confcache craz33guy@localhost: Descr wlan0 –> 2
–snpo: confcache craz33guy@localhost: Descr eth0 –> 3
–snpo: confcache craz33guy@localhost: Ip 0.0.0.0 –> 3
–snpo: confcache craz33guy@localhost: Ip 127.0.0.1 –> 1
–snpo: confcache craz33guy@localhost: Ip 192.168.1.100 –> 2
–snpo: confcache craz33guy@localhost: Type 24 –> 1
–snpo: confcache craz33guy@localhost: Type 6 –> 2
–snpo: confcache craz33guy@localhost: Type 6 –> 3 (duplicate)
–snpo: confcache craz33guy@localhost: Eth –> 1
–snpo: confcache craz33guy@localhost: Eth 00-06-25-09-6a-b5 –> 2
–snpo: confcache craz33guy@localhost: Eth 00-08-c7-10-74-a8 –> 3
–base: Get Interface Info
–base: Walking ifIndex
–base: Walking ifType
–base: Walking ifSpeed
–base: Walking ifAdminStatus
–base: Walking ifOperStatus
–base: Writing /etc/mrtg/localhost.cfg
[root@bigboy tmp]#

o Next create the /var/www/html/mrtg/stats directory and copy all of MRTG?s standard ?.png? image files into it.

[root@bigboy mrtg]# mkdir /var/www/html/mrtg/stats

[root@bigboy mrtg]# cp /var/www/html/mrtg/*.png /var/www/html/mrtg/stats
[root@bigboy mrtg]#

o Edit /etc/mrtg/localhost.cfg and remove the sections related to interfaces you don’t need to monitor. This would most likely include the loopback interface L0: with the IP address of 127.0.0.1

When the MRTG RPM is installed it places an entry in the /etc/crontab file to make MRTG run every 5 minutes using the default /etc/mrtg/mrtg.cfg configuration file. Add a new line referring to /etc/mrtg/localhost.cfg and comment out the one pointing to mrtg.cfg.

# 0-59/5 * * * * root /usr/bin/mrtg /etc/mrtg/mrtg.cfg

0-59/5 * * * * root /usr/bin/mrtg /etc/mrtg/localhost.cfg

o Run MRTG using /etc/mrtg/localhost.cfg as your argument three times. You’ll get an error the two times as MRTG tries to rename old data files, and naturally, the first time it is run, MRTG has no data files to move.

[root@bigboy mrtg]# mrtg /etc/mrtg/localhost.cfg
Rateup WARNING: /usr/bin/rateup could not read the primary log file for localhost_192.168.1.100
Rateup WARNING: /usr/bin/rateup The backup log file for localhost_192.168.1.100 was invalid as well
Rateup WARNING: /usr/bin/rateup Can’t remove localhost_192.168.1.100.old updating log file
Rateup WARNING: /usr/bin/rateup Can’t rename localhost_192.168.1.100.log to localhost_192.168.1.100.old updating log file
[root@bigboy mrtg]# mrtg /etc/mrtg/localhost.cfg
Rateup WARNING: /usr/bin/rateup Can’t remove localhost_192.168.1.100.old updating log file
[root@bigboy mrtg]# mrtg /etc/mrtg/localhost.cfg
[root@bigboy mrtg]#

o You’ll then want to use MRTG?s indexmaker command to create a combined index page to see all the graphs defined in all the various ?.cfg? files in your /etc/mrtg directory. Once this is done, you can point your browser to http://ip-address/mrtg/ to get a graphical listing of all the monitored interfaces.

Note: The indexmaker command creates a very generic index page which is very similar to the MRTG home page, don?t be fooled, you will find your devices at the very bottom. The format of the command is:

indexmaker –output=filename device1.cfg device2.cfg etc

RedHat Version 8.0 and Indexmaker
RedHat version 8 gives an error like this when running indexmaker.

[root@bigboy mrtg]# indexmaker –output=index.html /etc/mrtg/localhost.cfg
Can’t locate package $VERSION for @MRTG_lib::ISA at /usr/bin/indexmaker line 49
main::BEGIN() called at /usr/bin/../lib/mrtg2/MRTG_lib.pm line 49
eval {…} called at /usr/bin/../lib/mrtg2/MRTG_lib.pm line 49
[root@bigboy mrtg]#

You have a couple choices here:

? Run a version of indexmaker from an older version of RedHat

? Create your own custom index page to replace the default one in /var/www/html/mrtg. You can then add links to all the html files in the /var/www/html/mrtg/stats directory.

Using MRTG To Monitor Other Subsystems
MRTG will generate HTML pages with daily, weekly, monthly and yearly statistics for your interfaces. By default MRTG provides only network interface statistics. The MRTG website www.mrtg.org has links to other sites that show you how to monitor other sub-systems on a variety of devices and operating systems.

Webalizer
What Is Webalizer?
Webalizer is a web server log file analysis tool that comes installed by default on RedHat Linux. Each night, Webalizer reads your Apache log files and creates a set of web pages that allow you to view websurfer statistics for your site. The information provided includes a list of your web site?s most popular pages sorted by ?hits? along with traffic graphs showing the times of day when your site is most popular.

How To View Your Webalizer Statistics
By default webalizer places its index page in the directory /var/www/html/usage, so if you have a default Apache installation you?ll be able to view your data by visiting http://www.my-site.com/usage

The Webalizer Configuration File
Webalizer stores its configuration in the file /etc/webalizer.conf. The default settings should be sufficient for your web server, but you may want to adjust the directory in which Webalizer places your graphic statistics. This can be adjusted with the OutputDir directive in the file.

Make Webalizer run in Quiet Mode
Webalizer has a tendency to create this message in your logs which according to the Webalizer site?s documentation is non-critical.

Error: Unable to open DNS cache file /var/lib/webalizer/dns_cache.db

You can make the software run in quite mode by editing the /etc/cron.daily/00webalizer script file and adding the ?Q (Quiet) switch to the webalizer command like this:

#! /bin/bash

# update access statistics for the web site

if [ -s /var/log/httpd/access_log ] ; then

/usr/bin/webalizer -Q

fi

exit 0

Once you?ve done this, Webalizer will function with few annoyances, however be aware that running in quiet mode could hide deeper problems that could occur in future.