CRAWDAD tools/process/pads/snmp_parser (v. 2006-09-21)

A PADS-based C library for processing snmp traces.

snmp_parser provides a C library for processing snmp traces, and several tool implementations. Using this library, users can develop their own snmp tools. The PADS system (http://www.padsproj.org) needs to be installed to build and use the library and tools.

Lastmodified :

2006-11-14

Dataname :

tools/process/pads/snmp_parser

File :

snmp_parser.tar.gz, README

Releasedate :

2006-09-21

Change :

the initial version

References :

PADS: A System for Processing Ad-hoc Data
README

Website :

http://www.crawdad.org/tools/process/pads/snmp_parser

Keyword :

SNMP
802.11

License :

/*
* snmp.p: pads description of snmp trace
*
*      Author: Jihwang Yeo
*      version: v. 2006-09-21
*      Copyright (c) 2006 Dartmouth College
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License Version 2 as published by
* the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
* FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
* more details.
*
* You should have received a copy of the GNU General Public License along with
* this program; if not, write to the Free Software Foundation, Inc., 51
* Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/

Support :

You can find more information about PADS at the wiki (see the link above) or
PADS web page (http://www.padsproj.org).
Please let us know any bugfixes and suggestions at crawdad@crawdad.org

Build :

Step 1. Download and install PADS from http://www.padsproj.org/
Step 2. Uncompress and extract snmp-parser.tar.gz
Step 3. `cd snmp-parser` then type `make`
- GNUmakefile will generate pads library source files in gen directory.
- GNUmakefile will create a system-dependent $AST_ARCH directory
(e.g., darwin.ppc on MAC) and generate the executables in the directory

Output :

See "usage" for details about the output of each tool.

Parameters :

See "usage" for details about the parameters needed for each tool.

Usage :

1. snmp-accum

>snmp-accum [snmp trace file (data/snmp by default)]

(See "note" for the supported snmp formats)

The accumulator reports basic statistics (e.g., min, max, avg) and value distribution
of the input file with respect to each field defined in snmp.p.
We implemented snmp-accum by slightly modifying 'wsl-accum.c' in pads/demo directory.

Usage :

2. snmp-fmt
snmp-fmt [snmp trace file (data/snmp by default)]

(See "note" for the supported snmp formats)

The formatter converts a given trace format into a delimited format.
We implemented snmp-fmt by slightly modifying 'wsl-fmt.c' in pads/demo directory.

Example :

1 snmp-accum

>$AST_ARCH/snmp-accum data/sample.snmp.v31.IOS

(...)
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
top.meas.clients.some_clients_0_t.clsv31ios.cls.allArrayElts.cDot11ClientSignalStrength : int32
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
good vals:       1009    bad vals:          0    pcnt-bad:    0.000
Characterizing int32 values:  min -102 max 0 avg -91.801
=> distribution of top 10 values out of 51 distinct values:
val:       -100 count:        130  pcnt-of-good-vals:   12.884
val:        -99 count:        110  pcnt-of-good-vals:   10.902
val:        -94 count:         85  pcnt-of-good-vals:    8.424
val:        -97 count:         82  pcnt-of-good-vals:    8.127
val:        -95 count:         72  pcnt-of-good-vals:    7.136
val:        -98 count:         68  pcnt-of-good-vals:    6.739
val:        -96 count:         57  pcnt-of-good-vals:    5.649
val:        -93 count:         53  pcnt-of-good-vals:    5.253
val:       -101 count:         49  pcnt-of-good-vals:    4.856
val:        -92 count:         43  pcnt-of-good-vals:    4.262
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
SUMMING         count:        749  pcnt-of-good-vals:   74.232
(...)

The above example shows the accumulator result of cDot11ClientSignalStrength
(client signal strength) field. From the output we can see that the trace contains
1009 values as signal strength and there is no bad values in terms of data type
(int32 in this case). The output then shows basic statistics (min, max, and avg)
over the values, followed by the report of the top ten most frequent values.

2 snmp-fmt

>$AST_ARCH/snmp-fmt data/sample.snmp.v31.IOS

(...)
1078030565|ResBldg47AP1|392597232,IOS - 350 - 12.2(13)JA1
|1078030565|ResBldg47AP1|1|Dot11Radio0|71|11000000|127774391|13913074|0|80|124657587|9219659|401805|331275|51672|51672|43521|3991|51665|51665|6|0|0
|1078030565|ResBldg47AP1|003065a1fafb|-1|1|236|3|129.170.161.38|2296|-91|278452|105860|653|1128|2|1762|0
|1078030565|ResBldg47AP1|00904b5088af|-1|1|168|3|129.170.161.222|4148|-94|41730|4923|341|22|0|1935|0
|1078030565|ResBldg47AP1|00904b267708|-1|1|5|3|129.170.160.239|1084|-98|17490|5660|90|25|0|146|0
|1078030565|ResBldg47AP1|00904b86c8ba|-1|1|171|3|129.170.161.208|4076|-95|29785|8545|173|40|0|90|0
gorge:~/pads/snmp-parser jyeo$ vi data/sample.snmp.v31.IOS
(...)

The above example shows the formatter result on sample.snmp.v31.IOS trace.
The formatter converts the original trace format (see below) into tab-delimited format.


sys,1078030565,ResBldg47AP1,392597232,IOS - 350 - 12.2(13)JA1
if,1078030565,ResBldg47AP1,1,Dot11Radio0,71,11000000,127774391,13913074,0,80,124657587,9219659,401805,331275,51672,51672,43521,3991,51665,51665,6,0,0
cl,1078030565,ResBldg47AP1,003065a1fafb,-1,1,236,3,129.170.161.38,2296,-91,,278452,105860,653,1128,2,1762,0
cl,1078030565,ResBldg47AP1,00904b5088af,-1,1,168,3,129.170.161.222,4148,-94,,41730,4923,341,22,0,1935,0
cl,1078030565,ResBldg47AP1,00904b267708,-1,1,5,3,129.170.160.239,1084,-98,,17490,5660,90,25,0,146,0
cl,1078030565,ResBldg47AP1,00904b86c8ba,-1,1,171,3,129.170.161.208,4076,-95,,29785,8545,173,40,0,90,0

Algorithm :

snmp-parser consists of PADS description of SNMP traces (snmp.p) and source code of
accumulator and formatter application (snmp-accum.c and snmp-fmt.c).
To build the applications, the users are required to install PADS system beforehand and
compile the snmp-parser source files (*.p and *.c) using PADS compiler.
PADS compiler generates a C library for manipulating snmp traces, which is used for
building snmp-parser applications (snmp-accum and snmp-fmt).
For more detail about the PADS, go to the wiki (see the link above) or pads web page
(http://www.padsproj.org).

Note :

SNMP data format
--------------------
1. V2.1 snmp format (data/sample.snmp.v21)

This trace format is used for collecting dartmouth/campus/snmp/fall01 traceset.

The first five lines are comments.  The first gives basic information:
#V2.1: file format version 2.1, timestamp of file creation, AP name, and date code YYMMDD
All timestamps are standard Unix timestamps (seconds since 1970). The other four comment
lines describe the format of lines that occur later in the file.  Other than the timestamp
and AP name, the rest of these fields are MIB variable names.
After the five comment lines comes a series of polls.  Each poll consists of one ''sys'' line,
one ''if'' line describing stats of the the wireless interface, and zero or more pairs of
''c1'' and ''c2'' lines, each pair describing a currently connected client.  The c1 and c2
lines are a collection of MIB variables from the AWC MIB (Aironet Wireless Communications is
the name of the company that developed our access points; Aironet was bought by Cisco
who then branded and sold the APs under their name).

#V2.1,1018929767,AdmBldg27AP2,020416
#sys,timestamp,AP,sysUpTime
#if,timestamp,AP,ifIndex,ifType,ifSpeed,ifInOctets,ifInUcastPkts,ifInErrors,ifInDiscards,ifOutOctets,ifOutUcastPkts,ifOutErrors,ifOutDiscards
#c1,timestamp,AP,awcDot11TpFdbAddress,awcDot11TpFdbClientState,awcDot11TpFdbLatestRxSignalStrength,awcDot11TpFdbLatestRxSignalQuality
#c2,timestamp,AP,awcTpFdbAddress,awcTpFdbClassID,awcTpFdbSrcOctetsImmed,awcTpFdbDestOctetsImmed,awcTpFdbIPv4Addr

2. V3.0 trace format (data/sample.snmp.v30)

This trace format is used for collecing a subset of dartmouth/campus/snmp/fall0304 traceset.
V3.0 has the similar format to V2.1 as follws:

#V3.0,1067662868,AcadBldg10AP10,031101
#sys,timestamp,AP,sysUpTime
#if,timestamp,AP,ifIndex,ifDescr,ifType,ifSpeed,ifInOctets,ifInUcastPkts,ifInErrors,ifInDiscards,ifOutOctets,ifOutUcastPkts,ifOutErrors,ifOutDiscards,awcDot11AssociatedStationCount,awcDot11ReassociatedStationCount,awcDot11RoamedStationCount,awcDot11DeauthenticateCount,awcDot11DisassociateCount,awcFtClientSTASelf,awcFtBridgeSelf,awcFtRepeaterSelf
#c1,timestamp,AP,awcDot11TpFdbAddress,awcDot11TpFdbAID,awcDot11TpFdbClientState,awcDot11TpFdbLatestRxSignalStrength,awcDot11TpFdbLatestRxSignalQuality,awcDot11TpFdbCurrentBSS,awcDot11TpFdbSSID
#c2,timestamp,AP,awcTpFdbAddress,awcTpFdbClassID,awcTpFdbSrcOctetsImmed,awcTpFdbDestOctetsImmed,awcTpFdbIPv4Addr,awcTpFdbSrcPktsImmed,awcTpFdbDestPktsImmed,awcTpFdbSrcErrorPktsImmed,awcTpFdbDestErrorPktsImmed

3. V3.1 trace format (data/sample.snmp.v31.IOS and data/sample.snmp.v31.VxWorks)

This trace format is used for collecting a subset of dartmouth/campus/snmp/fall0304 traceset.
At the time of this data collection, Dartmouth mainly used Cisco 340 and 350 APs.
These used to run the VxWorks operating system. During December 2003 to May 2004,
our 350 APs migrated from running VxWorks  to the Cisco IOS (the APs didn't originally run IOS
as they were made by Aironet, a company that was later bought by Cisco). IOS uses completely
different SNMP MIBs to VxWorks, and so the variable names and their order are slightly different.
When the upgrades started taking place, we incremented the log version number to "V3.1"
(the first line of each log) to indicate the new variables being queried. We also folded both
the "c1" and "c2" client-specific lines into one "cl" line (this made the parser code easier to maintain).

To identify whether a given SNMP log is IOS or VxWorks, look at the ''sys'' line in a V3.1 log. The fifth field of this line is a formatted ''sysDescr'' indicating the OS version of the AP.

An example VxWorks file
#V3.1,1073710881,ResBldg48AP1,040110
#sys,timestamp,AP,sysUpTime,sysDescr
#if,timestamp,AP,ifIndex,ifDescr,ifType,ifSpeed,ifInOctets,ifInUcastPkts,ifInErrors,ifInDiscards,ifOutOctets,ifOutUcastPkts,ifOutErrors,ifOutDiscards,awcDot11AssociatedStationCount,awcDot11AuthenticatedStationCount,awcDot11ReassociatedStationCount,awcDot11RoamedStationCount,awcDot11DeauthenticateCount,awcDot11DisassociateCount,awcFtClientSTASelf,awcFtBridgeSelf,awcFtRepeaterSelf
#cl,timestamp,AP,awcDot11TpFdbAddress,awcDot11TpFdbAID,awcDot11TpFdbClientState,awcDot11TpFdbLatestRxSignalStrength,awcDot11TpFdbLatestRxSignalQuality,awcTpFdbClassID,awcTpFdbSrcOctetsImmed,awcTpFdbDestOctetsImmed,awcTpFdbIPv4Addr,awcTpFdbSrcPktsImmed,awcTpFdbDestPktsImmed,awcTpFdbSrcErrorPktsImmed,awcTpFdbDestErrorPktsImmed

An example IOS file:
#V3.1,1075784465,ResBldg47AP1,040203
#sys,timestamp,AP,sysUpTime,sysDescr
#if,timestamp,AP,ifIndex,ifDescr,ifType,ifSpeed,ifInOctets,ifInUcastPkts,ifInErrors,ifInDiscards,ifOutOctets,ifOutUcastPkts,ifOutErrors,ifOutDiscards,cDot11AssStatsAssociated,cDot11AssStatsAuthenticated,cDot11AssStatsRoamedIn,cDot11AssStatsRoamedAway,cDot11AssStatsDeauthenticated,cDot11AssStatsDisassociated,cDot11ActiveWirelessClients,cDot11ActiveBridges,cDot11ActiveRepeaters
#cl,timestamp,AP,cDot11ClientAddress,cDot11ClientRoleClassType,cDot11ClientPowerSaveMode,cDot11ClientAid,cDot11ClientAssociationState,cDot11ClientIpAddress,cDot11ClientUpTime,cDot11ClientSignalStrength,cDot11ClientSigQuality,cDot11ClientBytesSent,cDot11ClientBytesReceived,cDot11ClientPacketsSent,cDot11ClientPacketsReceived,cDot11ClientDuplicates,cDot11ClientMsduRetries,cDot11ClientMsduFails