CRAWDAD tools/process/pcap/WiPal (v. 2009-04-22)
- Citation Author(s):
- Submitted by:
- CRAWDAD Team
- Last updated:
- Fri, 08/07/2009 - 08:00
- Data Format:
IEEE 802.11 traces manipulation software
WiPal is a piece of software dedicated to IEEE 802.11 traces manipulation. It comes as a set of programs and a C++ library. A distinctive feature of WiPal is its merging tool, which enables merging multiple wireless traces into a unique global trace. This tool works offline on PCAP traces that do not need to be synchronized. WiPal also provides statistics extraction and anonymization tools, and its authors plan to extend it. WiPal’s key features are flexibility, ease of use, and efficiency.
the initial version.
The WiPal Software Website
802.11 802.11 frames RFMON packet trace tcpdump
Copyright © 2007, 2008, 2009 Université Pierre et Marie Curie — Paris 6 This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation ; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose. See the GNU General Public License for more details.
Its maintainer is Thomas Claveirole (email@example.com). It is developped with the help of Marcelo Dias de Amorim (Marcelo.Amorim@lip6.fr). Should you have any question, look at the Frequently Asked Questions web page (http://wipal.lip6.fr/faq.html).
To build WiPal the following softwares are needed: GCC. The Boost C++ libraries. More specifically: any, array, conversion/lexical_cast, date_time, filesystem, foreach, format, multi_array, optional, preprocessor smart_ptr, string_algo, tuple. The GNU MP Bignum Library, OpenSSL. WiPal’s packaging follows the GNU conventions. An installation documentation is provided in the INSTALL file in the package’s root directory. However, with a standard system, the following commands should do the trick: mkdir _build cd _build ../configure make make install-strip make check On some systems, you might have to customize the configure script’s invocation. e.g.: mkdir _build cd _build ../configure CPPFLAGS=-I/foo/bar/libgmp make make install-strip make check
WiPal's programs all use the same invocation scheme: wipal-<command> [options] [inputs] [outputs] The command line may include no options and, depending on the program, there may be no inputs or no outputs. Most programs expect at least one input however. See the specific documentation (http://wipal.lip6.fr/documentation.html) for each program in order to know how many inputs and outputs each program expects. Inputs, outputs, and options may be mixed on the command line, e.g.: wipal-simple-merge -n -P input1.pcap input2.pcap output.pcap wipal-simple-merge input1.pcap input2.pcap output.pcap -P -n wipal-simple-merge input1.pcap -n input2.pcap -P output.pcap ... are all equivalent. WiPal's programs use getopt(3) to parse options, so they only have short options (no long options) composed of a dash followed by a letter (e.g. -a, -t, etc.) Option letters always have the same meaning whatever the program. All options are not available for all programs though (some options do not make sens with some programs). For instance, -P always means the invoked program should consider frames with non-zero Prism fields as invalid. In order to know which options a program accept, use the -h option. Finally, some options expect an extra argument right after they are provided: wipal-test-uniqueness -a hsh_80211 input.pcap ^^^^^^^^^ This is not an input - Basic usage You may provide the name of a PCAP trace file as input. wipal-cat input.pcap output.pcap - Input concatenation You may provide the name of several PCAP traces separated with columns (do not include any space). This tells the program to consider the concatenation of each trace as a single input. wipal-cat input1.pcap:input2.pcap:input3.pcap output.pcap will put into output.pcap the content of input1.pcap, followed by the content of input2.pcap and then input3.pcap. Every program understands this syntax. Note that specifying multiple traces with columns makes no sense for outputs: wipal-cat input1.pcap:input2.pcap output1.pcap:output2.pcap will concatenate input1.pcap and input2.pcap into a single file named output1.pcap:output2.pcap! - Address specification Some programs (e.g. wipal-merge with attributes hsh_en2) might need the IPv4 address of the machine that generated a trace to work properly. Attach such an address to a trace as follows: wipal-merge -a hsh_en2 foo.pcap=192.168.1.1 bar1.pcap:bar2.pcap=192.168.1.2 The rationale is that, in some cases, timestamps of emitted frames are not as precise as timestamps of received frames, and thus emitted frames should be ignored during synchronization. - Special characters When your traces' filenames contain the special characters : or = they need to be escaped with a backslash (\): wipal-cat weird\=file\:name.pcap out.pcap wipal-merge -a hsh_en2 weird\=1:weird\:2=192.168.1.1 foo.pcap=192.168.1.2
1. Concatenation (and Prism noise filtering) One may concatenate traces using the wipal-cat command. It takes exactly one input and one output. It may be useful to recombine a trace that was split, or filter out frames with Prism noise (using the -P option). wipal-cat in.pcap out.pcap wipal-cat foo.pcap.0:foo.pcap.1 foo.pcap wipal-cat -P in.pcap out.pcap wipal-cat -P bar.pcap.0:bar.pcap.1:bar.pcap.2 bar.pcap The first example just copies in.pcap into out.pcap. Note that the two files might be different at the byte level, e.g. if in.pcap is big endian and the program is run on a little endian machine. The second example concatenate foo.pcap.0 and foo.pcap.1 and put the result into foo.pcap. The third example copies in.pcap into out.pcap but removes frames that have a non-zero noise field in their Prism headers. The fourth example both concatenates traces while filtering noisy frames out. 2. Comparisons One may test two PCAP traces for equivalence using the wipal-cmp command. The default is to compare every bit of information (PCAP headers plus packet bytes) but you may change this behavior using the -8, -b, -o, or -t options. Note that this is different however to using diff or cmp since traces with different endianness may contain the same packets. By default wipal-cmp produces a report on the standard output indicating either that traces are equal, either which packet is the first to mismatch. Use -q if you are only interested in the program's exit status and do not want to produce any output. Use -o if you are interested in counting the number of bytes that differ between two traces. e.g.: wipal-cmp foo.pcap bar.pcap wipal-cmp -q foo.pcap bar.pcap wipal-cmp -q -8 in1.pcap.0:in1.pcap.1 in2.pcap ... 3. Sub-traces One may extract sub-traces of PCAP traces using wipal-extract-subtrace, wipal-extract-transmitter, or wipal-extract-bssid. wipal-extract-subtrace takes two dates and a PCAP trace as inputs, and produces one output. Unfortunately, it does not support any option currently. wipal-extract-transmitter takes a MAC address and a PCAP trace as input, and produces one output. Its output contains the frames from its input that were transmitted by the given address. Note that the command looks at transmitters, not originators, e.g. the transmitter of a data frame that crossed the distribution system is the output access point, not the original sender. Also note that some frames do not contain information regarding their transmitters (e.g. MAC acknowledgements) and therefore cannot appear in the output, even if they were effectively sent by the given address. wipal-extract-bssid works as wipal-extract-transmitter, but the MAC address represents a BSSID and the command extracts frames that belong to the corresponding BSS. Again, note that some frames do not contain information regarding their BSS. These frames therefore cannot appear in the output, even if they were effectively belonging to the given BSS. e.g.: wipal-extract-subtrace 2007-01-01 2008-01-01 in.pcap.0:in.pcap.1 out.pcap wipal-extract-subtrace "2004-Aug-30 16:59:39.789221" "2004-Aug-30 16:59:39.929872" kalahari-ath2 subtrace.pcap wipal-extract-transmitter 71:19:9f:6f:71:33 in.pcap out.pcap wipal-extract-bssid 9b:d2:d7:7f:aa:63 in.pcap out.pcap 4. Merging One may merge two IEEE 802.11 traces into one using the wipal-simple-merge command. Use the -h option to have a description of the command's syntax. It takes two inputs and produce one output. When ran, the merging process starts by synchronizing precisely both inputs. Then both traces are merged and special care is given not to re-order packets or account duplicate packets twice in the output (that is, packets that are present in both traces appear only once in the output). This command expects PCAP traces with either Prism headers, AVS headers, Radiotap headers, raw IEEE 802.11 frames, or pseudo-Ethernet II frames as link type. The -p and -P options only work with Prism headers. The following timestamps are used, unless -y is provided: IEEE 802.11 frames PCAP timestamps, Ethernet II frames PCAP timestamps, Radiotap headers Radiotap headers' tsft fields. The command will fail with Radiotap headers that do not contain such fields, AVS headers AVS headers' mactime fields, Prism headers Prism headers' mactime fields. e.g.: wipal-simple-merge a.pcap b.pcap output.pcap wipal-simple-merge -P -n foo-ath2.0:foo-ath2.1 bar-ath2 foo-bar-ath2 ... 5. Synchronization In order to merge two IEEE 802.11 traces WiPal needs to synchronize them precisely. In order to do so, it first identifies some frames that appear in both inputs. These are reference frames. It uses these frames to model clock desynchronization among the traces. It then update the first trace's timestamps so they are synchronized with the second trace. One may use the wipal-synchronize command to synchronize two traces. It takes two inputs and produce one output. The output contains the same packets as the first input, but with synchronized timestamps. To extract reference frames WiPal extract some specific frames called unique frames from both input traces and then intersect the two obtained sets. One may use the wipal-intersect-unique-frames command to get the result of this operation (i.e. the list of reference frames used for synchronization of two traces). WiPal's synchronization process synchronizes reference frames before it synchronizes other frames. One may get the result of this operation using the wipal-synchronize-unique-frames command. e.g.: wipal-intersect-unique-frames -n -P foo.0:foo.1:foo.2 bar.0:bar.1 wipal-synchronize-unique-frames -n -P foo.0:foo.1:foo.2 bar.0:bar.1 wipal-synchronize -n -P foo.0:foo.1:foo.2 bar.0:bar.1 foo-sync 6. Unique frames A frame is said to be unique when it appears on the air once and only once for the whole duration of a trace. WiPal's unique frame extraction process is an important stage of its trace synchronization process. WiPal programs' default policy is to consider all beacon frames and all non-retransmitted probe responses as unique frames. One may use the wipal-extract-unique-frames command to get a list of the unique frames that compose a trace. Run wipal-extract-unique-frames -h to get its invocation syntax. In practice, WiPal does not extract and load full unique frames into memory. This would slow the process down and require an excessive amount of memory. The default is to work on MD5 frame hashes when WiPal was compiled using OpenSSL. When compiled without OpenSSL, WiPal only extracts a subset of frame fields. We call the pieces of information WiPal extracts to identify unique frames “frame attributes”, or sometimes “frame identifiers”. You may specify frame attributes to use with the -a option. In practice, the difference in speed and memory consumption between attributes is negligible. There is an important difference between attributes, though. With some attributes, different unique frames may yield identical attributes (collisions). This is of course an undesirable behavior. One may check that a given trace's unique frames are really unique w.r.t. unique frame attributes using the wipal-test-uniqueness command. This command finds collisions inside its input traces. You might specify different frame attributes using the -a option. e.g.: wipal-test-uniqueness -P -a tmp foo.pcap.1:foo.pcap.2 wipal-extract-unique-frames -P foo.pcap.1:foo.pcap.2 > foo-unique.txt 7. Duplicate data frames One may use the wipal-find-data-dups command to search some invalid data frames. It looks into traces on a per-sender basis for successive duplicate data frames (it only considers non-retransmitted frames). Such cases should not occur in theory -- as it ignores retransmissions, successive data frames from the same sender should at least show variations in their sequence numbers. Surprisingly, some traces contain such anomalies: identical data frames that are not retransmissions and are only spaced by a few milliseconds. We have no explanations why some datasets exhibit those phenomena. e.g.: wipal-find-data-dups foo.pcap.0:foo.pcap.1:foo.pcap.2 8. Statistics wipal-stats computes several figures concerning its given input PCAP traces. It displays these figures as plain text on the standard output. You might either interpret them directly or post-process them with some tools, e.g. to generate plots. (You can see the manual at http://wipal.lip6.fr/doc/wipal.html/ for more details.) 9. Anonymization wipal-anonymize is a program to anonymize IEEE 802.11 traces. It takes one input and one output: the filename of the trace to anonymize, and the filename of the anonymous trace to produce. The output contains the same frames as the input with the following modifications: - NIC specific parts of MAC addresses are anonymized. - ESSIDs are anonymized with a prefix-preserving scheme. For instance, a valid anonymization could map ‘operator-4251’, ‘operator-DODO’, and ‘foobar’ to ‘abcdefgh*x0yz’, ‘abcdefgh*9876’, and ‘zxycba’. The anonymization scheme also preserves character classes, i.e. alpha-numerical characters are anonymized to other alpha-numerical characters, printable characters stay printable, and ASCII extended characters (128 to 256) stay extended. - Data frames are truncated so the output only contains MAC headers. wipal-anonymize stores valid-to-anonymous MAC and ESSID mappings into files so these mappings can be re-used latter. wipal-anonymize also reads these file at start-up when they exist. This enables the creation of distinct anonymous traces with consistent MAC addresses and ESSIDs. By default these mapping files' names are MAC.map and ESSID.map. Use the -m and -s options to change this.
The files in this directory are a CRAWDAD toolset hosted by IEEE DataPort.
About CRAWDAD: the Community Resource for Archiving Wireless Data At Dartmouth is a data resource for the research community interested in wireless networks and mobile computing.
CRAWDAD was founded at Dartmouth College in 2004, led by Tristan Henderson, David Kotz, and Chris McDonald. CRAWDAD toolsets are hosted by IEEE DataPort as of November 2022.
Note: Please use the tools in an ethical and responsible way with the aim of doing no harm to any person or entity for the benefit of society at large. Please respect the privacy of any human subjects whose wireless-network activity is captured by the tools and comply with all applicable laws, including without limitation such applicable laws pertaining to the protection of personal information, security of data, and data breaches. Please do not apply, adapt or develop algorithms for the extraction of the true identity of users and other information of a personal nature, which might constitute personally identifiable information or protected health information under any such applicable laws. Do not publish or otherwise disclose to any other person or entity any information that constitutes personally identifiable information or protected health information under any such applicable laws derived from the tools through manual or automated techniques.
Please acknowledge the source of the tools in any publications or presentations reporting use of this tools.
Thomas Claveirole, Marcelo Dias de Amorim, CRAWDAD toolset tools/process/pcap/WiPal (v. 2009‑04‑22), https://doi.org/10.15783/C7P017, Apr 2009.
- wipal-4.0.tar.gz (1.47 MB)