Handling SNMP Traps with Hobbit

Hobbit is a fast replacement for the Big Brother monitoring/alert system. There were some ways to handle SNMP traps with Big Brother, but none that I could find that worked with Hobbit. (I hope I haven't reinvented the wheel....) I'm running all of this on the same computer.

I got the idea after reading an article on integrating SNMP traps with Nagios in Sys Admin. I just adapted it to Hobbit.

Download (it contains all of the configuration files, scripts, and a simple installation script)

Screenshots:

What it does not do:

  1. Does not: Forward SNMP traps to an external Network Management System
  2. Does not: Poll devices using SNMP

Here's what I used:

  1. Net-SNMP - SNMP tools (snmptrapd,...)
  2. SNMPTT - SNMP trap translator
  3. SEC - Simple Event Correlator
  4. Various MIB files

How it works


I won't go into detail how to configure and install each package; so, definitely read the documentation for each one. You'll probably find other uses for them. Be sure to adjust the path names as needed.

Net SNMP

Just compile and install this package. You might have it depending on your distribution.

RC script: rc.snmptrapd

Logrotate configuration:

/var/log/snmp/snmptrapd.log {
    missingok
    postrotate
        /etc/rc.d/rc.snmptrapd restart
    endscript
}

/etc/snmp/snmptrapd.conf :
authCommunity   log,execute,net PUBLIC
traphandle default /usr/local/sbin/snmptthandler 

Get your MIB files

This will depend on what you have, but here's some:

SNMPTT

After installing the package, you'll need to convert MIB files to snmptt.conf files. The SNMPTT manual mentions using the snmptranslate command from the Net-SNMP package to verify that a given MIB file can be parsed correctly.

Examples:

The --net_snmp_perl option creates long description sections with detail about all of the possibile variables for a given OID trap. Using it is optional.

American Power Conversion:

$ export MIBDIRS=/usr/local/share/snmp/mibs
$ /usr/local/sbin/snmpttconvertmib --format_desc=6 --net_snmp_perl --in=powernet368.mib --out=snmptt.conf.apc 2>>errors

Cisco:

$ mkdir v1
$ tar -C v1 -zxvf v1.tar.gz
$ export MIBDIRS=/usr/local/share/snmp/mibs:$PWD/v1
$ for i in v1/*my
> do
> /usr/local/sbin/snmpttconvertmib --format_desc=6 --net_snmp_perl --in=$i --out=snmptt.conf.cisco 2>>errors
> done

Don't be supprised to get warning messages about "can't translate/find OID" in the "errors" file. It's telling you that some of the MIB files require other MIB files that you don't have. You can ignore the warnings, track down all of the required MIBs (by reading the problem MIB file), or only translate the MIBs that pertain to your equipment.

I've noticed that the APC and Dell MIB files have a SEVERITY definition in them. SNMPTT uses that to establish the severity for each trap (Normal|INFORMATIONAL|SEVERE|WARNING|...). However, I've noticed that Cisco and Canoga Perkins don't have those definitions; so, every trap event is considered Normal. You'll need to change the severity for the various traps as desired in the snmptt.conf file.

Copy your various snmptt.conf.* files to /etc/snmp.

RC script: rc.snmptt

Logrotate configuration:

/var/log/snmp/snmptt.log /var/log/snmp/snmpttunknown.log {
    missingok
    postrotate
        /etc/rc.d/rc.snmptt restart
    endscript
}


The default settings for the /etc/snmp/snmptt.ini file are fine. If you run snmptt as non-root, make sure it has permission to read/write log files where indicated. I've listed what I use: /etc/snmp/snmptt.ini. I used the SNMPTT MySQL setup instructions to create the MySQL databse.

SEC - Simple Event Correlator

RC script: rc.sec

Logrotate configuration:

/var/log/sec.log {
    missingok
    postrotate
        /etc/rc.d/rc.sec restart
    endscript
}

/etc/sec.conf :

# sec.conf file
#
# notes:
#  http://simple-evcorr.sourceforge.net/
#  http://sixshooter.v6.thrupoint.net/SEC-examples/article.html
#
# trap.sh time_stamp OID severity hostname message

# Snmptrap event translated by snmptraptt and logged in /var/log/messages
# $1 - Time stamp
# $2 - OID
# $3 - Severity Status
# $4 - Hostname (or IP)
# $5 - Message
type=SingleWithSuppress
ptype=RegExp
pattern=(\w+\s+\d+\s\d+:\d+:\d+)\s[\w\.\-]+ snmptt\[\d+\]: ([\d\.]*) (Normal|INFORMATIONAL|MINOR|WARNING|SEVERE|MAJOR|CRITICAL)\s\"Status Events\"\s([\w\.\-]*)\s\-\s(.*)
desc=snmptrap received from $4
action=shellcmd /home/hobbit/server/ext/trap.sh "$1" $2 $3 $4 "$5"
window=2


# Unknown Snmptrap event logged in /var/log/snmp/snmpttunknown.log
# $1 - Time stamp
# $2 - OID
# $3 - Hostname (or IP)
type=SingleWithSuppress
ptype=RegExp
pattern=(\w+\s+\w+\s+\d+\s\d+:\d+:\d+\s+\d+):\sUnknown trap \(([\d\.]*)\) received from ([\w\.\-]*) at:
desc=unknown snmptrap received from $3
action=shellcmd /home/hobbit/server/ext/trap.sh "$1" $2 Unknown $3 "Unknown trap"
window=2

Trap script

I made the trap.pl script so it can be used by both SEC (which processes the trap) as well as Hobbit (which runs it to keep the trap column from going purple).

You'll need to rename the file to trap.pl since the file is called trap-pl.txt so your web browser can view it.

Copy trap.pl to /home/hobbit/server/ext and make it executable. Make sure the path to perl is correct in trap.pl.

You need to install Text::Autoformat from CPAN for trap.pl.

SEC Client

The wrapper script, trap.sh, is called by SEC when it processes a trap. It uses Hobbit's bbcmd in order to setup the proper environment variables.

trap.sh:

#!/bin/sh
# $1 - Time stamp
# $2 - OID
# $3 - Severity Status
# $4 - Hostname (or IP)
# $5 - Message

TRAPCLIENT="/home/hobbit/server/ext/trap.pl"
HOBBITCFG="/home/hobbit/server/etc/hobbitserver.cfg"

/home/hobbit/server/bin/bbcmd --env=$HOBBITCFG $TRAPCLIENT -C -s $3 -t $4 -d "$1" -m "$5 ($2)"

Hobbit External program

The trap.pl script is run from Hobbit every 5 minutes from hobbitlaunch.cfg.

[trap]
	ENVFILE /home/hobbit/server/etc/hobbitserver.cfg
	NEEDS hobbitd
	CMD $BBHOME/ext/trap.pl -S
	LOGFILE $BBSERVERLOGS/trap.log
	INTERVAL 5m

bb-hosts

Add "trap" for a host in the bb-host file:
0.0.0.0    somehost.domain.com     # trap

Trap History

If you setup SNMPTT to log to MySQL, you can use the trap.php script to query the SNMPTT log database on your MySQL server.
Copy the trap.php script to the hobbit/www directory. You will also need to enable USE_SNMPTTSQL in trap.pl.

Example Cisco Switch Configuration

snmp-server enable traps config
snmp-server enable traps copy-config
snmp-server enable traps syslog
snmp-server enable traps entity
snmp-server enable traps flash insertion removal
snmp-server enable traps bridge
snmp-server enable traps stpx
snmp-server enable traps rtr
snmp-server enable traps c2900
snmp-server enable traps vtp
snmp-server enable traps vlancreate
snmp-server enable traps vlandelete
snmp-server enable traps port-security
snmp-server enable traps MAC-Notification
snmp-server enable traps envmon fan shutdown supply temperature status
snmp-server enable traps hsrp
snmp-server enable traps cluster
snmp-server enable traps vlan-membership
snmp-server host 10.1.1.1 PUBLIC

Don't forget, the Cisco MIBs don't have a SEVERITY set for any traps. You'll need to update the snmptt.conf file for your Cisco equipment to change the SEVERITY status for the OIDs you find interesting.

Troubleshooting

Q: I'm not seeing traps appear in the trap column in Hobbit. What's wrong?

A: Trap.pl matches the hostname as it appears in both the /var/log/messages log file and in bb-hosts. Do the hostnames for the traps in the /var/log/messages file match the hostnames as they are listed in the bb-hosts file? Fixes are either to update DNS or /etc/hosts.

If the snmp log entries list IP addresses, trap.pl is not smart enough to look for IP addresses in bb-hosts.

If you have just hostnames in the bb-hosts file (not FQDN) and the hostnames in the log file are FQDN, they won't match up.

If that's the case (like you want just the hostname to display in Hobbit and not the entire FQDN, we have a long domain name and it's annoying to be displayed), you can modify the trap.pl script to chop off the domain name to the hostname that trap.pl receives from the log file (via SEC) will match the host name as it appears in bb-hosts:

Find subroutine SendTrap and add the following before the CASE statement:

Sub SendTrap {

   # remove domain names from hosts to matchup with entries in bb-hosts
   $trappedhost =~ s/\.foobar\.com//g;
   $trappedhost =~ s/\./,/g;


Last updated: 8/27/2008
andy.farrior (@) victoriacollege.edu