PCAP Compression
PCAP compression is essential for any serious packet capture work as it dramatically reduces the amount of storage the system uses. While compressing during capture is in the development pipeline, we now support realtime GZIP on download. Which is great for reducing download bandwidth, disk space and Wireshark has built in support for .pcap.gz too.
Bandwidth Reduction
Compression is great to reduce disk space but also helps when transferring files off the capture device. Particularly over WAN connections and to the local desktop machine that almost always has "last-mile" 1G copper connectivity. As such you can now download fully GZIP compressed PCAP`s from the Web GUI as shown below.
It works by compressing AFTER splitting and filtering as a post processing operation. This means you can split and filter in any way and still have full GZIP support. For example using our Python Helper Scripts generating a compressed rsync 1 minute PCAP on a remote machine is as follows.
$ ./capture_rsync.py --output /tmp/ --split split_1min --compress remote_basic_1453950384697651968_20160128_1206
RSync Capture Named [remote_basic_1453950384697651968_20160128_1206]
[/tmp/remote_basic_1453950384697651968_20160128_1206_split_1min/remote_basic_1453950384697651968_20160128_1206__20160128_12:06:40.164.294.912.pcap.gz] Downloading...
.
.
.
.
Compression Performance
While the compression performance will never run at 10Gbps, it does run realtime at around ~1Gbps with compression rates roughly equivalent to gzip --fast. For packet captures that are not media heavy (e.g. not already compressed) we see a x2-x3 compression ratio. This means downloading over a 1G connection will be x2-x3 faster AND use half or less disk space. Best of all its completely free if your constrained by a 1G connection.
... there is plenty more compression work to be done. On the roadmap is compression on write back to HDD`s as this effectively doubles or tripples the local storage size.
