CRAWDAD tools/process/pcap/WiPal (v. 2010-01-13)

IEEE 802.11 traces manipulation software

WiPal is a piece of software dedicated to IEEE 802.11 traces manipulation. It comes as a set of programs and a C++ library. A distinctive feature of WiPal is its merging tool, which enables merging multiple wireless traces into a unique global trace. This tool works offline on PCAP traces that do not need to be synchronized. WiPal also provides statistics extraction and anonymization tools, and its authors plan to extend it. WiPal’s key features are flexibility, ease of use, and efficiency.

Lastmodified :

2010-01-18

Dataname :

tools/process/pcap/WiPal

File :

wipal-5.0.tar.bz2, wipal-5.0.tar.gz

Releasedate :

2010-01-13

Equiversion :

v5.0

Change :

- Port WiPal to GCC 4.4.

- wipal-list-frame: Enable using -f and -F to provide hexadecimal
dumps of frames.

- wipal-stats: The "STA counters" sections now include "total number
of frames" and "average frames per microsecond" columns.  See
WiPal manual.

- Introduce wipal-plot-onoff to plot the "ON/OFF events" section
reported by wipal-stats.  This requires a valid Python
installation and Pycairo (http://cairographics.org/pycairo/).

- WiPal library: every compound is now located inside the top-level
"wpl" namespace.

- Revamp the library's pcap input/output system.  This induces
faster pcap operations (both for WiPal's binaries and the
library).  But this is also a huge API change for those using the
library.

The most important changes are:
* the pcapxx namespace becomes wpl::pcap,
* pcapxx::descriptor<> becomes wpl::pcap::file<> (in
wipal/pcap/stream.hh),
* pcapxx::dumper becomes wpl::pcap::ostream (in
wipal/pcap/ostream.hh),
* introduction of a wpl::pkt namespace for file-format independent
compounds.  To this regard, pcapxx::frame_descriptor<> becomes
wpl::pkt::packet<> (in wipal/pcap/pkt.hh).  Its pcap_header()
method becomes meta().
* pcap files with indexes are now a separate class:
wpl::pcap::indexed_file<>

- Library: perform a lot of file and compound renamings.  Among the
most important ones:
* move everything from wipal/wifi/frame/filter to wipal/filter,
* move everything else from wipal/wifi/frame to wipal/wifi.  The
wifi::frame namespace does not exist anymore either.
* filter::microseconds_stamper becomes filter::timetracker,
* wifi::mactime_tracker becomes phy::timetracker.

- Fix a few bugs, as usual. ;-)

References :

The WiPal Software Website

Website :

http://wipal.lip6.fr

Keyword :

802.11
802.11 frames
RFMON
packet trace
tcpdump

License :

Copyright © 2007, 2008, 2009 Université Pierre et Marie Curie — Paris 6

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation ; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose. See the GNU General Public License for more details.

Support :

Its maintainer is Thomas Claveirole (thomas.claveirole@lip6.fr).
It is developped with the help of Marcelo Dias de Amorim (Marcelo.Amorim@lip6.fr).

Should you have any question, look at the Frequently Asked Questions web page
(http://wipal.lip6.fr/faq.html).

Build :

To build WiPal the following softwares are needed:

GCC.
The Boost C++ libraries. More specifically:
any,
array,
conversion/lexical_cast,
date_time,
filesystem,
foreach,
format,
multi_array,
optional,
preprocessor
smart_ptr,
string_algo,
tuple.
The GNU MP Bignum Library,
OpenSSL.

WiPal’s packaging follows the GNU conventions. An installation documentation is
provided in the INSTALL file in the package’s root directory. However, with
a standard system, the following commands should do the trick:

mkdir _build
cd _build
../configure
make
make install-strip
make check

On some systems, you might have to customize the configure script’s invocation. e.g.:

mkdir _build
cd _build
../configure CPPFLAGS=-I/foo/bar/libgmp
make
make install-strip
make check

Usage :

WiPal's programs all use the same invocation scheme:

wipal-<command> [options] [inputs] [outputs]

The command line may include no options and, depending on the program,
there may be no inputs or no outputs. Most programs expect at least
one input however. See the specific documentation
(http://wipal.lip6.fr/documentation.html) for each program in order to
know how many inputs and outputs each program expects.

Inputs, outputs, and options may be mixed on the command line, e.g.:

wipal-simple-merge -n -P input1.pcap input2.pcap output.pcap
wipal-simple-merge input1.pcap input2.pcap output.pcap -P -n
wipal-simple-merge input1.pcap -n input2.pcap -P output.pcap
...

are all equivalent.

WiPal's programs use getopt(3) to parse options, so they only have short options
(no long options) composed of a dash followed by a letter (e.g. -a, -t, etc.)
Option letters always have the same meaning whatever the program. All options are
not available for all programs though (some options do not make sens with some programs).
For instance, -P always means the invoked program should consider frames with
non-zero Prism fields as invalid. In order to know which options a program accept,
use the -h option.

Finally, some options expect an extra argument right after they are provided:

wipal-test-uniqueness -a hsh_80211 input.pcap
^^^^^^^^^
This is not an input


- Basic usage

You may provide the name of a PCAP trace file as input.

wipal-cat input.pcap output.pcap


- Input concatenation

You may provide the name of several PCAP traces separated with columns
(do not include any space). This tells the program to consider the concatenation
of each trace as a single input.

wipal-cat input1.pcap:input2.pcap:input3.pcap output.pcap

will put into output.pcap the content of input1.pcap, followed by the content of
input2.pcap and then input3.pcap.

Every program understands this syntax. Note that specifying multiple traces
with columns makes no sense for outputs:

wipal-cat input1.pcap:input2.pcap output1.pcap:output2.pcap

will concatenate input1.pcap and input2.pcap into a single file named output1.pcap:output2.pcap!


- Address specification

Some programs (e.g. wipal-merge with attributes hsh_en2) might need the IPv4
address of the machine that generated a trace to work properly. Attach such an address
to a trace as follows:

wipal-merge -a hsh_en2 foo.pcap=192.168.1.1 bar1.pcap:bar2.pcap=192.168.1.2

The rationale is that, in some cases, timestamps of emitted frames are not as precise
as timestamps of received frames, and thus emitted frames should be ignored during
synchronization.


- Special characters

When your traces' filenames contain the special characters : or = they need to be
escaped with a backslash (\):

wipal-cat weird\=file\:name.pcap out.pcap
wipal-merge -a hsh_en2 weird\=1:weird\:2=192.168.1.1 foo.pcap=192.168.1.2

Algorithm :

1. Concatenation (and Prism noise filtering)

One may concatenate traces using the wipal-cat command. It takes exactly one input
and one output. It may be useful to recombine a trace that was split, or filter out
frames with Prism noise (using the -P option).

wipal-cat in.pcap out.pcap
wipal-cat foo.pcap.0:foo.pcap.1 foo.pcap
wipal-cat -P in.pcap out.pcap
wipal-cat -P bar.pcap.0:bar.pcap.1:bar.pcap.2 bar.pcap

The first example just copies in.pcap into out.pcap. Note that the two files might be
different at the byte level, e.g. if in.pcap is big endian and the program is run
on a little endian machine.

The second example concatenate foo.pcap.0 and foo.pcap.1 and put the result into foo.pcap.

The third example copies in.pcap into out.pcap but removes frames that have a non-zero
noise field in their Prism headers.

The fourth example both concatenates traces while filtering noisy frames out.



2. Comparisons

One may test two PCAP traces for equivalence using the wipal-cmp command.
The default is to compare every bit of information (PCAP headers plus packet bytes)
but you may change this behavior using the -8, -b, -o, or -t options. Note that
this is different however to using diff or cmp since traces with different endianness
may contain the same packets.

By default wipal-cmp produces a report on the standard output indicating either
that traces are equal, either which packet is the first to mismatch.
Use -q if you are only interested in the program's exit status and do not want
to produce any output. Use -o if you are interested in counting the number of
bytes that differ between two traces.

e.g.:

wipal-cmp foo.pcap bar.pcap
wipal-cmp -q foo.pcap bar.pcap
wipal-cmp -q -8 in1.pcap.0:in1.pcap.1 in2.pcap
...


3. Sub-traces

One may extract sub-traces of PCAP traces using wipal-extract-subtrace,
wipal-extract-transmitter, or wipal-extract-bssid.

wipal-extract-subtrace takes two dates and a PCAP trace as inputs, and
produces one output. Unfortunately, it does not support any option currently.

wipal-extract-transmitter takes a MAC address and a PCAP trace as input, and
produces one output. Its output contains the frames from its input that were
transmitted by the given address. Note that the command looks at transmitters,
not originators, e.g. the transmitter of a data frame that crossed the distribution
system is the output access point, not the original sender. Also note that some
frames do not contain information regarding their transmitters (e.g. MAC acknowledgements)
and therefore cannot appear in the output, even if they were effectively sent
by the given address.

wipal-extract-bssid works as wipal-extract-transmitter, but the MAC address
represents a BSSID and the command extracts frames that belong to the corresponding
BSS. Again, note that some frames do not contain information regarding their BSS.
These frames therefore cannot appear in the output, even if they were effectively
belonging to the given BSS.

e.g.:
wipal-extract-subtrace 2007-01-01 2008-01-01 in.pcap.0:in.pcap.1 out.pcap

wipal-extract-subtrace "2004-Aug-30 16:59:39.789221" "2004-Aug-30 16:59:39.929872" kalahari-ath2 subtrace.pcap

wipal-extract-transmitter 71:19:9f:6f:71:33 in.pcap out.pcap
wipal-extract-bssid       9b:d2:d7:7f:aa:63 in.pcap out.pcap



4. Merging

One may merge two IEEE 802.11 traces into one using the wipal-simple-merge command.

Use the -h option to have a description of the command's syntax.
It takes two inputs and produce one output. When ran, the merging process starts
by synchronizing precisely both inputs. Then both traces are merged and special care
is given not to re-order packets or account duplicate packets twice in the output
(that is, packets that are present in both traces appear only once in the output).

This command expects PCAP traces with either Prism headers, AVS headers,
Radiotap headers, raw IEEE 802.11 frames, or pseudo-Ethernet II frames
as link type. The -p and -P options only work with Prism headers.

The following timestamps are used, unless -y is provided:

IEEE 802.11 frames
PCAP timestamps,
Ethernet II frames
PCAP timestamps,
Radiotap headers
Radiotap headers' tsft fields. The command will fail with Radiotap headers that do not contain such fields,
AVS headers
AVS headers' mactime fields,
Prism headers
Prism headers' mactime fields.


e.g.:

wipal-simple-merge a.pcap b.pcap output.pcap
wipal-simple-merge -P -n foo-ath2.0:foo-ath2.1 bar-ath2 foo-bar-ath2
...


5. Synchronization

In order to merge two IEEE 802.11 traces WiPal needs to synchronize them precisely.
In order to do so, it first identifies some frames that appear in both inputs.
These are reference frames. It uses these frames to model clock desynchronization
among the traces. It then update the first trace's timestamps so they are
synchronized with the second trace.

One may use the wipal-synchronize command to synchronize two traces. It takes
two inputs and produce one output. The output contains the same packets
as the first input, but with synchronized timestamps.

To extract reference frames WiPal extract some specific frames called unique frames
from both input traces and then intersect the two obtained sets. One may use
the wipal-intersect-unique-frames command to get the result of this operation
(i.e. the list of reference frames used for synchronization of two traces).

WiPal's synchronization process synchronizes reference frames before it
synchronizes other frames. One may get the result of this operation using
the wipal-synchronize-unique-frames command.

e.g.:

wipal-intersect-unique-frames -n -P foo.0:foo.1:foo.2 bar.0:bar.1
wipal-synchronize-unique-frames -n -P foo.0:foo.1:foo.2 bar.0:bar.1
wipal-synchronize -n -P foo.0:foo.1:foo.2 bar.0:bar.1 foo-sync



6. Unique frames

A frame is said to be unique when it appears on the air once and only once
for the whole duration of a trace. WiPal's unique frame extraction process is
an important stage of its trace synchronization process. WiPal programs' default
policy is to consider all beacon frames and all non-retransmitted probe
responses as unique frames.

One may use the wipal-extract-unique-frames command to get a list of the unique
frames that compose a trace. Run wipal-extract-unique-frames -h to get its invocation syntax.

In practice, WiPal does not extract and load full unique frames into memory.
This would slow the process down and require an excessive amount of memory.
The default is to work on MD5 frame hashes when WiPal was compiled using OpenSSL.
When compiled without OpenSSL, WiPal only extracts a subset of frame fields.
We call the pieces of information WiPal extracts to identify unique frames
“frame attributes”, or sometimes “frame identifiers”.

You may specify frame attributes to use with the -a option. In practice,
the difference in speed and memory consumption between attributes is negligible.
There is an important difference between attributes, though. With some attributes,
different unique frames may yield identical attributes (collisions). This is
of course an undesirable behavior.

One may check that a given trace's unique frames are really unique w.r.t.
unique frame attributes using the wipal-test-uniqueness command.
This command finds collisions inside its input traces.
You might specify different frame attributes using the -a option.

e.g.:

wipal-test-uniqueness -P -a tmp foo.pcap.1:foo.pcap.2
wipal-extract-unique-frames -P foo.pcap.1:foo.pcap.2 > foo-unique.txt


7. Duplicate data frames

One may use the wipal-find-data-dups command to search some invalid data frames.
It looks into traces on a per-sender basis for successive duplicate data frames
(it only considers non-retransmitted frames). Such cases should not occur in theory
-- as it ignores retransmissions, successive data frames from the same sender should
at least show variations in their sequence numbers. Surprisingly, some traces contain
such anomalies: identical data frames that are not retransmissions and are only
spaced by a few milliseconds. We have no explanations why some datasets exhibit
those phenomena.

e.g.:

wipal-find-data-dups foo.pcap.0:foo.pcap.1:foo.pcap.2



8. Statistics

wipal-stats computes several figures concerning its given input PCAP traces.
It displays these figures as plain text on the standard output. You might either
interpret them directly or post-process them with some tools, e.g. to generate plots.

(You can see the manual at http://wipal.lip6.fr/doc/wipal.html/ for more details.)


9. Anonymization

wipal-anonymize is a program to anonymize IEEE 802.11 traces. It takes one input
and one output: the filename of the trace to anonymize, and the filename of
the anonymous trace to produce. The output contains the same frames as the input
with the following modifications:

- NIC specific parts of MAC addresses are anonymized.

- ESSIDs are anonymized with a prefix-preserving scheme. For instance, a valid anonymization
could map ‘operator-4251’, ‘operator-DODO’, and ‘foobar’ to ‘abcdefgh*x0yz’, ‘abcdefgh*9876’,
and ‘zxycba’. The anonymization scheme also preserves character classes, i.e. alpha-numerical
characters are anonymized to other alpha-numerical characters, printable characters stay
printable, and ASCII extended characters (128 to 256) stay extended.

- Data frames are truncated so the output only contains MAC headers.

wipal-anonymize stores valid-to-anonymous MAC and ESSID mappings into files so these mappings
can be re-used latter. wipal-anonymize also reads these file at start-up when they exist.
This enables the creation of distinct anonymous traces with consistent MAC addresses and ESSIDs.
By default these mapping files' names are MAC.map and ESSID.map. Use the -m and -s options
to change this.