Graph Network Performance with NRG - Part 2

Essential Open Source Network Administration Tools


1. Introduction
1.1. Document conventions
2. Identify SNMP OID's to graph
2.1. Use snmpwalk
2.2. Read a MIB file
2.3. Test OID's with snmpget
3. Make a Manual .conf File
3.1. Use an example .conf file
3.2. Superimposed graphs
3.3. Stacked graphs
4. Install a Manual .conf File
5. Put Graph Links in Tables
6. Graph Non-SNMP Devices
6.1. Perl collector scripts
6.2. Shell collector scripts
6.3. Graph with collector scripts
7. Support Information

1. Introduction

In Part One, we installed NRG and graphed SNMP devices that NRG can auto-discover, such as most switches, hubs, and routers. For non-standard devices or devices that fail to auto-discover properly, you need to make .conf files that define graphs manually. So in Part 2 we'll cover how to identify SNMP Object Identifiers (OID) to graph, making .conf files manually, and also graphing non-SNMP devices.

1.1. Document conventions

Here are the conventions used to distinguish Unix terminal window input and output.

%% Commands to be typed in a terminal window.
Command output to a terminal window.
File text.

2. Identify SNMP OID's to graph

For the purposes of SNMP graphing, the main thing we want to find are a device's SNMP object identifiers (OID’s) that name individual data points we want to graph.

2.1. Use snmpwalk

You may use snmpwalk to get a report on a device's SNMP OID's.

%% snmpwalk –v 2c –c <community-string> <ip-address>

Here is the partial output of a Cisco switch responding to the snmpwalk command shown above.

SNMPv2-MIB::sysDescr.0 = STRING: Cisco IOS Software, C3550 Software
Copyright (c) 1986-2005 by Cisco Systems, Inc.
Compiled Wed 08-Jun-05 03:37 by yenanh
SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.9.1.368
SNMPv2-MIB::sysUpTime.0 = Timeticks: (170506673) 19 days, 17:37:46.73
SNMPv2-MIB::sysContact.0 = STRING: 
SNMPv2-MIB::sysName.0 = STRING: 3550.biola.edu
SNMPv2-MIB::sysLocation.0 = STRING: 
SNMPv2-MIB::sysServices.0 = INTEGER: 6
SNMPv2-MIB::sysORLastChange.0 = Timeticks: (0) 0:00:00.00

                    [ ... ]

RFC1213-MIB::ifDescr.1 = STRING: "GigabitEthernet0/1" (1)
RFC1213-MIB::ifDescr.2 = STRING: "GigabitEthernet0/2"
RFC1213-MIB::ifDescr.3 = STRING: "GigabitEthernet0/3"

                    [ ... ]

RFC1213-MIB::ifInOctets.1 = Counter32: 3275453054 (2)
RFC1213-MIB::ifInOctets.2 = Counter32: 0
RFC1213-MIB::ifInOctets.3 = Counter32: 0

                    [ ... ]

RFC1213-MIB::ifOutOctets.1 = Counter32: 311794524
RFC1213-MIB::ifOutOctets.2 = Counter32: 0
RFC1213-MIB::ifOutOctets.3 = Counter32: 0 (3)

There are SNMP OID’s that return strings, and ones that return numeric values of type ”counter” or “gauge”. There are usually string OID's whose index numbers correspond to index numbers of numeric OID's and tells what the numeric OID's represent.

(1)

ifDescr.1 OID whose index number represents interface Gig0/1.

(2)

ifInOctets.1 OID whose index tells us it represents ifInOctets for interface Gig0/1.

(3)

ifOutOctets.3 OID whose index tells us it represents ifOutOctets for interface Gig0/3.

See “Using snmpget to test OID's” below and make sure that the numbers returned are plausible for a given OID to verify you've selected the OID you wanted.

2.2. Read a MIB file

The OID’s for a device are listed in its Management Information Base (MIB) file. MIB files are provided by a device's manufacturer and are needed if you cannot determine what OID’s you want to graph by using snmpwalk. Locate the object you want to graph by its name and description. The MIB information is a hierarchical tree that you must “walk” backwards from the bottom of the file to the top to construct the OID. Here's how find the OID for the object "ssStatCpuUtil" that appears in the hypothetical MIB file below.

sysStatCpuUtil OBJECT-TYPE
      SYNTAX  Integer32
      MAX-ACCESS  read-only

      STATUS  current
      DESCRIPTION

           "CPU utilization in percentage."
      ::= { systemStats 1 }

Since the identifier “sysStatCpuUtil” is associated with the identifier “systemStats,” search for the latter in the MIB file.

systemStats        OBJECT IDENTIFIER ::=  { statistics 2 }

The identifier “systemStats” is associated with the identifier “statistics,” so search for the latter in the MIB file.

statistics         OBJECT IDENTIFIER ::= { rightServer 13 }

The identifier “statistics” is associated with identifier “rightServer,” so search for the latter in the MIB file.

rightServer         OBJECT IDENTIFIER ::= { rightSocket 1 }

And so on until you reach the item at the top of the MIB file that references and item that begins with “enterprises”.

rightSocket MODULE-IDENTITY

     LAST-UPDATED "200303100000Z"
     ORGANIZATION "Rightsocket Inc."
     CONTACT-INFO

     "         Rightsocket, Inc.
               Dipcidoo, WI 66666
               davey@rightsocket.com

     "
     DESCRIPTION
          "This MIB is DRAFT-ONLY and is subject to change.
           Management Information Base for Rightserver."

    ::= { enterprises 9967 }

Then your search is over and you can assemble the complete OID starting in the reverse order from which you found them (from the bottom of the MIB file to the top) and “build” the OID for “sysStatCpuUtil” as shown in the following diagram.

2.3. Test OID's with snmpget

Once you’ve determined the OID, you should test it using snmpget. Snmpget is what SNMP applications use to get values from monitored devices. However, many OID's require something called an instance sub-identifier, which means an OID often needs an extra .0 on the end. If this is the case, the full numeric OID for object “sysStatCpuUtil” would be enterprises.9967.1.13.2.1.0.

The OID can also be referred to by the name enterprises.9967.rightSocket.rightServer.statistics.systemStats.sysStatCpuUtil. You may test OID's with the snmpget command as shown.

%% snmpget –v 2c –c <community-string> <oid>

Here is an example snmpget response.

enterprises.9967.1.14.5.2.0 = 223

3. Make a Manual .conf File

It is best to learn how to make .conf files by example and experimentation, so I’ll give you some pointers and examples to get you started.

3.1. Use an example .conf file

It is best not to start from scratch. You may copy the .conf file of a device that successfully auto-discovered and use it as a template. Also, frequently the .conf file that NRG generates automatically, even if incomplete, can be used as a template. So if NRG does not successfully graph a device using auto-discovery, use the device’s automatically generated .conf file as a starting point for a manual .conf file with these steps.

  1. Move the device’s .conf file from /opt/local/var/nrg/autoconfs to any directory other than /opt/local/var/nrg.

  2. Remove the device from the Site.mconf file and run “make rediscover” and “make notify” to remove the device from NRG's configuration.

  3. Modify the .conf file following the examples below.

3.2. Superimposed graphs

Here is a .conf file that superimposes graphs of two OID's of type GAUGE, a rising and falling value. First, a few observations:

  • Each line of a .conf file is called a configuration string. A .conf file is composed of configuration strings that identify one or more graphs and their characteristics.

  • The first item in brackets in a configuration string is called the [target]. It identifies the graph to which each configuration string belongs, therefore each configuration string must have a target.

##

System[smtp-active]:            public@mail.mycompany.com  (1)
RRD[smtp-active]:               /net/cgpro/smtp-active.rrd (2)
GraphWebPage[smtp-active]:      /net/cgpro/smtp-active.cgi (3)

Variable[smtp-active][input]:   .1.3.6.1.4.1.5678.2.1.1.1.1.0 GAUGE (4)
Label[smtp-active][input]:      SMTP Input (5)
Graph[smtp-active][input]:      green AREA  (6)

Variable[smtp-active][output]:  .1.3.6.1.4.1.5678.2.1.1.2.1.0 GAUGE
Label[smtp-active][output]:     SMTP Output
Graph[smtp-active][output]:     blue LINE2

YLabel[smtp-active]:            connections (7)
Units[smtp-active]:             %s connections (8) 

PageTitle[smtp-active]: SMTP active input/output connections
PageTop[smtp-active]: <H2>Number of active SMTP input/output connections</H2>
PageBottom[smtp-active]: <H3>Number of active SMTP input/output connections</H3>

##

In addition to [target], the configuration strings in our sample .conf file specify:

(1)

System - community string and hostname of the graphed device.

(2)

RRD - path to the .rrd file.

(3)

GraphWebPage - path to the .cgi file.

(4)

Variable - [data source] name for RRDtool, the SNMP OID to graph, and value type.

(5)

Label - label for a [variable] to appear inside the RRDtool graph.

(6)

Graph - color and type of the [data source] graph.

(7)

YLabel - a label for the graph's Y-axis.

(8)

Units - a label to appear after ave, max, or last. %s is used to autoscale (b/s, Kb/s, Mb/s.)

Here is the graph produced by this .conf file.

3.3. Stacked graphs

Here is a .conf file that stacks graphs of two OID's of type COUNTER, a continually increasing value.

##

System[dev5]:              public@10.6.6.6
RRD[dev5]:                 /net/device5/d5.rrd
GraphWebPage[dev5]:        /net/device5/d5.cgi

Variable[dev5][regular]:   .1.3.6.1.4.1.5678.2.1.1.1  GAUGE
Label[dev5][regular]:      Regular Sessions
Graph[dev5][regular]:      blue AREA

Variable[dev5][remote]:    .1.3.6.1.4.1.5678.2.1.1.2  GAUGE
Label[dev5][remote]:       Remote Sessions
Graph[dev5][remote]:       green STACK

YLabel[dev5]:              Sessions
Units[dev5]:               %sSessions

PageTitle[dev5]: Device5 Sessions
PageTop[dev5]: <H2>Device5 Sessions</H2>
PageBottom[dev5]: <H3>Device5 Sessions</H3>

##

The first graph defined in a graph sets the graph type, an areas graph in this case. Additional graph statements may use the STACK operator to stack subsequent graph elements as shown.

4. Install a Manual .conf File

Copy manual .conf files to /opt/local/var/nrg. NRG adds both auto-discovered items in Site.mconf and manually created .conf files into its configuration seamlessly using the same NRG configuration commands as always.

%% make rediscover
%% make notify

NRG finds your custom .conf files, adds them to the list of monitored devices, and includes them on the NRG web page. For example, here is the NRG home page after adding a manually created Communigate Pro .conf file to the NRG configuration as it was at the end of Part 1 with the auto-discovered Ethernet switch21.

5. Put Graph Links in Tables

You will notice when you create manual.conf files that have multiple graphs, the individual graph links all appear on the main NRG page, which clutters it as in the example above. This is unlike the auto-discovered graph links which all appear within a table. The NRG page would be much more neat and attractive if the links to our manual .conf files were grouped within tables as well. This goal is easily accomplished by creating and using a template as shown.

  1. Copy the table template "table-ifaces.cgi.tm" to create the new table template "table-byPageTitle.cgi.tm"

    %% cd /opt/local/var/templates
    %% copy table-ifaces.cgi.tm table-byPageTitle.cgi.tm
  2. Replace the table section of the new template with the table section below.

    <TABLE>
    <TR ALIGN=LEFT>
      <TH ALIGN=LEFT><FONT SIZE=+1>SNMP Statistic</FONT></TH>
      <TH ALIGN=RIGHT>&nbsp;&nbsp;<FONT SIZE=+1>Last Update</FONT></TH>
    </TR>
    </FONT>
    ___BEGIN_ROWS___
    <TR ALIGN=LEFT>
      <TD ALIGH=LEFT><A HREF="___PAGE_URL___">___PAGE_TITLE___</A></TD>
      <TD ALIGN=RIGHT>&nbsp;&nbsp;<A HREF="___PAGE_URL___">___LAST_UPDATE_%b %d %H:%M___</A></TD>
    </TR>
    ___END_ROWS___
    </TABLE>
  3. Now add these lines to /opt/local/var/nrg/Default.conf to activate the new template.

    TablePageTemplate[.*-table-byPageTitle]: \
      /opt/local/var/nrg/templates/table-byPageTitle.cgi.tm
  4. Now any time you wish to put custom .conf file graphs in a table, use three Table .conf statements and prefix the [target] name with a unique string for the device and append ‘-table-byPageTitle’ as shown.

    TableTitle[CGPro-table-byPageTitle]:      CGPro SNMP Statistics
    TableWebPage[CGPro-table-byPageTitle]:    /net/cgpro/index.cgi
    TableRegExp[CGPro-table-byPageTitle]:     /net/cgpro/.*rrd
    

If you add the table-byPageTitle template statements, then rerun "make rediscover" and "make notify", you'll see all your manual .conf file graph links in a table as shown.

6. Graph Non-SNMP Devices

NRG can also graph devices that do not support SNMP as long as you can write a “collector script” to run on the NRG workstation that can return the data to be graphed in the proper format. A collector script can be a Perl or shell script, but it must print three integers, each on a separate line.

  1. The time in Unix timestamp format (seconds since Jan 1, 1970.)

  2. The uptime value of the monitored host –or “INF” (infinite.)

  3. The value to be graphed, a descriptive label, and GAUGE / COUNTER (rising & falling / incrementing.)

For example, here is screen output from a sample script.

1163201755 Fri Nov 10 15:35:55 2006
INF uptime
115 volts GAUGE

6.1. Perl collector scripts

For a collector script that graphs data of type GAUGE, the uptime is not relevant and you may use “INF” to specify infinite uptime. Therefore elements one and two of a perl NRG collector script will likely be the same as the sample script below. The third element will print out the variable to be graphed, a descriptive label, and either GAUGE or COUNTER depending on whether the value rises and falls or increments respectively.

#!/opt/local/bin/perl

# This script retrieves the voltage of a sensor
# and places it in variable ${graphdata}

     [ ... ]

## Perl NRG collector script output section ##

# Element 1 - print time
    print time(), " ", scalar localtime(time()), "\n";

# Element 2 - print “infinite” uptime
    print "INF uptime\n";

# Element 3 - print variable to be graphed
    print “$graphdata volts GAUGE\n”;
    exit;

6.2. Shell collector scripts

Here is an NRG collector script that will produce identical output as the one above written as a shell script.

#!/bin/bash

# This script retrieves the voltage of a sensor
# and places it in variable ${graphdata}

     [ ... ]

## NRG collector shell script output section #

# Element 1 - Time in Unix timestamp format

set -- `env LC_ALL=C LC_TIME=C LANG=C date -u '+%Y %j %H %M %S'`
DAYS=`expr 365 \* \( $1 - 1970 \) + \( $1 - 1969 \) / 4 + $2 - 1`
# This will begin to fail in 2100 A.D. unless the following line
# that adjusts for 100 and 400 years periods is uncommented.
#[ $1 -gt 2100 ] && DAYS=`expr $DAYS - \( $1 - 2000 \) / 100 \
#¬?+ \( $1 - 2000 \) / 400
TIME=`expr $5 + 60 \* \( $4 + 60 \* \( $3 + 24 \* $DAYS \) \)`

    echo -n $TIME
    echo -n " "
    date

# Element 2 - Print “infinite” uptime
    echo INF uptime

# Element 3 - echo the variable to be graphed
    echo -n $graphdata
    echo " volts GAUGE"

6.3. Graph with collector scripts

Using collector scripts in .conf files is very simple. The only difference with a standard .conf file is that you substitute the SNMP OID entries with the collector script(s). For example, say you have an environmental sensor with an embedded web server, and you have a perl script that captures the temperature in variable $temperature, and another one that captures the humidity in variable $humidity.

Output of the temp.pl script:

1163201755 Fri Nov 10 15:35:55 2006
INF uptime
92 temperature GAUGE

Output of the humid.pl script:

1163204634 Fri Nov 10 16:23:54 2006
INF uptime
50 humidity GAUGE

Here is an example of a manual .conf file that creates a superimposed two element graph of the temperature and humidity variables.

##

System[sensor-1]:           community@ignored-for-collector-scripts
RRD[sensor-1]:             /net/sensor1/sensor-1.rrd
GraphWebPage[sensor-1]:    /net/sensor1/sensor-1.cgi

Variable[sensor-1][temp]:  `/opt/local/var/nrg/temp.pl -H 10.3.3.7` GAUGE
Label[sensor-1][temp]:     Temperature
Graph[sensor-1][temp]:     red LINE2

Variable[sensor-1][humid]: `/opt/local/var/nrg/humid.pl -H 10.3.3.7` GAUGE
Label[sensor-1][humid]:    Humidity
Graph[sensor-1][humid]:    blue LINE2

YLabel[sensor-1]:          Temp / Humidity

PageTitle[sensor-1]: Sensor CPU Room
PageTop[sensor-1]: <H2>Sensor CPU Room</H2>
PageBottom[sensor-1]: <H3>Sensor CPU Room</H3>

##

Now let's see the graph this .conf file generates.

7. Support Information

If you need further assistance with NRG you should carefully check the documentation to see if it answers your questions and/or join the NRG mailing list.