Hobbit is a fast replacement for the Big Brother monitoring/alert system. There were some ways to handle SNMP traps with Big Brother, but none that I could find that worked with Hobbit. (I hope I haven't reinvented the wheel....) I'm running all of this on the same computer.
I got the idea after reading an article on integrating SNMP traps with Nagios in Sys Admin. I just adapted it to Hobbit.
Download (it contains all of the configuration files, scripts, and a simple installation script)
Screenshots:What it does not do:
Here's what I used:
I won't go into detail how to configure and install each package; so, definitely read the documentation for each one. You'll probably find other uses for them. Be sure to adjust the path names as needed.
RC script: rc.snmptrapd
Logrotate configuration:
/var/log/snmp/snmptrapd.log {
missingok
postrotate
/etc/rc.d/rc.snmptrapd restart
endscript
}
/etc/snmp/snmptrapd.conf :
authCommunity log,execute,net PUBLIC traphandle default /usr/local/sbin/snmptthandler
This will depend on what you have, but here's some:
After installing the package, you'll need to convert MIB files to snmptt.conf files. The SNMPTT manual mentions using the snmptranslate command from the Net-SNMP package to verify that a given MIB file can be parsed correctly.
The --net_snmp_perl option creates long description sections with detail about all of the possibile variables for a given OID trap. Using it is optional.
American Power Conversion:
$ export MIBDIRS=/usr/local/share/snmp/mibs $ /usr/local/sbin/snmpttconvertmib --format_desc=6 --net_snmp_perl --in=powernet368.mib --out=snmptt.conf.apc 2>>errors
Cisco:
$ mkdir v1 $ tar -C v1 -zxvf v1.tar.gz $ export MIBDIRS=/usr/local/share/snmp/mibs:$PWD/v1 $ for i in v1/*my > do > /usr/local/sbin/snmpttconvertmib --format_desc=6 --net_snmp_perl --in=$i --out=snmptt.conf.cisco 2>>errors > done
Don't be supprised to get warning messages about "can't translate/find OID" in the "errors" file. It's telling you that some of the MIB files require other MIB files that you don't have. You can ignore the warnings, track down all of the required MIBs (by reading the problem MIB file), or only translate the MIBs that pertain to your equipment.
I've noticed that the APC and Dell MIB files have a SEVERITY definition in them. SNMPTT uses that to establish the severity for each trap (Normal|INFORMATIONAL|SEVERE|WARNING|...). However, I've noticed that Cisco and Canoga Perkins don't have those definitions; so, every trap event is considered Normal. You'll need to change the severity for the various traps as desired in the snmptt.conf file.
Copy your various snmptt.conf.* files to /etc/snmp.
RC script: rc.snmptt
Logrotate configuration:
/var/log/snmp/snmptt.log /var/log/snmp/snmpttunknown.log {
missingok
postrotate
/etc/rc.d/rc.snmptt restart
endscript
}
The default settings for the /etc/snmp/snmptt.ini file are fine. If you run snmptt as non-root, make sure it has permission to read/write log files where indicated. I've listed what I use: /etc/snmp/snmptt.ini. I used the SNMPTT MySQL setup instructions to create the MySQL databse.
RC script: rc.sec
Logrotate configuration:
/var/log/sec.log {
missingok
postrotate
/etc/rc.d/rc.sec restart
endscript
}
/etc/sec.conf :
# sec.conf file # # notes: # http://simple-evcorr.sourceforge.net/ # http://sixshooter.v6.thrupoint.net/SEC-examples/article.html # # trap.sh time_stamp OID severity hostname message # Snmptrap event translated by snmptraptt and logged in /var/log/messages # $1 - Time stamp # $2 - OID # $3 - Severity Status # $4 - Hostname (or IP) # $5 - Message type=SingleWithSuppress ptype=RegExp pattern=(\w+\s+\d+\s\d+:\d+:\d+)\s[\w\.\-]+ snmptt\[\d+\]: ([\d\.]*) (Normal|INFORMATIONAL|MINOR|WARNING|SEVERE|MAJOR|CRITICAL)\s\"Status Events\"\s([\w\.\-]*)\s\-\s(.*) desc=snmptrap received from $4 action=shellcmd /home/hobbit/server/ext/trap.sh "$1" $2 $3 $4 "$5" window=2 # Unknown Snmptrap event logged in /var/log/snmp/snmpttunknown.log # $1 - Time stamp # $2 - OID # $3 - Hostname (or IP) type=SingleWithSuppress ptype=RegExp pattern=(\w+\s+\w+\s+\d+\s\d+:\d+:\d+\s+\d+):\sUnknown trap \(([\d\.]*)\) received from ([\w\.\-]*) at: desc=unknown snmptrap received from $3 action=shellcmd /home/hobbit/server/ext/trap.sh "$1" $2 Unknown $3 "Unknown trap" window=2
I made the trap.pl script so it can be used by both SEC (which processes the trap) as well as Hobbit (which runs it to keep the trap column from going purple).
You'll need to rename the file to trap.pl since the file is called trap-pl.txt so your web browser can view it.
Copy trap.pl to /home/hobbit/server/ext and make it executable. Make sure the path to perl is correct in trap.pl.
You need to install Text::Autoformat from CPAN for trap.pl.
The wrapper script, trap.sh, is called by SEC when it processes a trap. It uses Hobbit's bbcmd in order to setup the proper environment variables.
trap.sh:
#!/bin/sh # $1 - Time stamp # $2 - OID # $3 - Severity Status # $4 - Hostname (or IP) # $5 - Message XYMONDIR="/usr/lib/xymon/server" TRAPCLIENT="$XYMONDIR/ext/trap.pl" XYMONCFG="$XYMONDIR/etc/xymonserver.cfg" $XYMONDIR/bin/xymoncmd --env=$XYMONCFG $TRAPCLIENT -C -s $3 -t $4 -d "$1" -m "$5 ($2)"
The trap.pl script is run from Hobbit every 5 minutes from hobbitlaunch.cfg.
[trap] ENVFILE /home/hobbit/server/etc/hobbitserver.cfg NEEDS hobbitd CMD $BBHOME/ext/trap.pl -S LOGFILE $BBSERVERLOGS/trap.log INTERVAL 5m
0.0.0.0 somehost.domain.com # trap
snmp-server enable traps config snmp-server enable traps copy-config snmp-server enable traps syslog snmp-server enable traps entity snmp-server enable traps flash insertion removal snmp-server enable traps bridge snmp-server enable traps stpx snmp-server enable traps rtr snmp-server enable traps c2900 snmp-server enable traps vtp snmp-server enable traps vlancreate snmp-server enable traps vlandelete snmp-server enable traps port-security snmp-server enable traps MAC-Notification snmp-server enable traps envmon fan shutdown supply temperature status snmp-server enable traps hsrp snmp-server enable traps cluster snmp-server enable traps vlan-membership snmp-server host 10.1.1.1 PUBLICDon't forget, the Cisco MIBs don't have a SEVERITY set for any traps. You'll need to update the snmptt.conf file for your Cisco equipment to change the SEVERITY status for the OIDs you find interesting.
Q: I'm not seeing traps appear in the trap column in Hobbit. What's wrong?
A: Trap.pl matches the hostname as it appears in both the /var/log/messages log file and in bb-hosts. Do the hostnames for the traps in the /var/log/messages file match the hostnames as they are listed in the bb-hosts file? Fixes are either to update DNS or /etc/hosts.
If the snmp log entries list IP addresses, trap.pl is not smart enough to look for IP addresses in bb-hosts.
If you have just hostnames in the bb-hosts file (not FQDN) and the hostnames in the log file are FQDN, they won't match up.
If that's the case (like you want just the hostname to display in Hobbit and not the entire FQDN, we have a long domain name and it's annoying to be displayed), you can modify the trap.pl script to chop off the domain name to the hostname that trap.pl receives from the log file (via SEC) will match the host name as it appears in bb-hosts:
Find subroutine SendTrap and add the following before the CASE statement:
Sub SendTrap {
# remove domain names from hosts to matchup with entries in bb-hosts
$trappedhost =~ s/\.foobar\.com//g;
$trappedhost =~ s/\./,/g;