Script Integration for Packet Analysis

Posted by fmadio | 100G Ethernet

Good cost effective full line rate packet sniffers are great, but by them self aren`t all that helpful. The power of packet capture is when its coupled with analyzer software, such as opensource tools like Snort or Suricata, our opensource tools or your customized analysis software. We understand this and our device has exceptional integration that's easy, simple, and enables you to go from 0 to 100 within your existing infrastructure in no time flat.


  • packet data analyzer

Our 10G packet sniffer appliance has been integrated into many systems, and its always used in a slightly different way. It ranges from drop in replacements to deep dive network trouble shooting devices. In all cases one of the most important features is script integration and how you can utilize our packet sniffer as easily as possible with existing systems.

List all streams on the device

The best way to access the device is with wget or CURL. We prefer CURL but that's just a personal preference. The following commands will list all streams on the device in JSON format.

NOTE: have added a some line returns for easier reading.

aaron@remote:/tmp$ curl -u username:password http://192.168.1.1:/stream/list { "Path":"/capture/", "List":[ { "Path":"remote_writeback_1432983349926713088_20150530_1955", "PCAP":"/pcap/single?StreamName=remote_writeback_1432983349926713088_20150530_1955&", "Filter":"/en.filter.html?StreamName=remote_writeback_1432983349926713088_20150530_1955&", "Link":"/en.files.html?Fn=view&StreamName=remote_writeback_1432983349926713088_20150530_1955&", "Date":"Sat May 30 19:55:54 2015", "Size":8650752, "PacketCnt":100000, "Type":"", "Desc":"" }, { "Path":"remote_writeback_1432983318759834880_20150530_1955", "PCAP":"/pcap/single?StreamName=remote_writeback_1432983318759834880_20150530_1955&", "Filter":"/en.filter.html?StreamName=remote_writeback_1432983318759834880_20150530_1955&", "Link":"/en.files.html?Fn=view&StreamName=remote_writeback_1432983318759834880_20150530_1955&", "Date":"Sat May 30 19:55:23 2015", "Size":8650752, "PacketCnt":100000, "Type":"", "Desc":"" }, { "Path":"remote_cache_wrap_1432977797640_B_20150530_1952", "PCAP":"/pcap/single?StreamName=remote_cache_wrap_1432977797640_B_20150530_1952&", . . . . }

... but sometimes full JSON formatting can be a pain in the ass, so there`s also a simpler comma delimited, one line per file text output - we like simple things at fmadio. Makes parsing easy in any scripting language.

aaron@remote:/tmp$ curl -u username:password http://192.168.1.1:/plain/list Filename , Size Bytes , Packet Count , Date ,Single PCAP Link,File Link, remote_writeback_1432983349926713088_20150530_1955 , 8650752, 100000, Sat May 30 19:55:54 2015,/pcap/single?StreamName=remote_writeback_143298334992671308 remote_writeback_1432983318759834880_20150530_1955 , 8650752, 100000, Sat May 30 19:55:23 2015,/pcap/single?StreamName=remote_writeback_143298331875983488 remote_cache_wrap_1432977797640_B_20150530_1952 , 8400142336, 100000000, Sat May 30 19:52:46 2015,/pcap/single?StreamName=remote_cache_wrap_1432977797640_B_2 remote_cache_wrap_1432977797640_A_20150530_1823 , 21000093696, 250000000, Sat May 30 18:23:22 2015,/pcap/single?StreamName=remote_cache_wrap_1432977797640_A_2 remote_reboot1432977720397351936_20150530_1822 , 8400142336, 100000000, Sat May 30 18:22:05 2015,/pcap/single?StreamName=remote_reboot1432977720397351936_20 remote_reboot1432977526113184000_20150530_1818 , 8400142336, 100000000, Sat May 30 18:18:50 2015,/pcap/single?StreamName=remote_reboot1432977526113184000_20 remote_reboot1432977331317114112_20150530_1815 , 8400142336, 100000000, Sat May 30 18:15:36 2015,/pcap/single?StreamName=remote_reboot1432977331317114112_20 remote_reboot1432977136725125120_20150530_1812 , 8400142336, 100000000, Sat May 30 18:12:21 2015,/pcap/single?StreamName=remote_reboot1432977136725125120_20 remote_reboot1432976942348261888_20150530_1809 , 8400142336, 100000000, Sat May 30 18:09:07 2015,/pcap/single?StreamName=remote_reboot1432976942348261888_20 remote_reboot1432976747717306880_20150530_1805 , 8400142336, 100000000, Sat May 30 18:05:52 2015,/pcap/single?StreamName=remote_reboot1432976747717306880_20 remote_reboot1432976553132131072_20150530_1802 , 8400142336, 100000000, Sat May 30 18:02:37 2015,/pcap/single?StreamName=remote_reboot1432976553132131072_20 remote_reboot1432976358992337920_20150530_1759 , 8400142336, 100000000, Sat May 30 17:59:23 2015,/pcap/single?StreamName=remote_reboot1432976358992337920_20 remote_reboot1432976164484417024_20150530_1756 , 8400142336, 100000000, Sat May 30 17:56:09 2015,/pcap/single?StreamName=remote_reboot1432976164484417024_20 remote_reboot1432975970060336896_20150530_1752 , 8400142336, 100000000, Sat May 30 17:52:55 2015,/pcap/single?StreamName=remote_reboot1432975970060336896_20 remote_basic_1432975532469615104_20150530_1745 , 17582260224, 98500000, Sat May 30 17:45:36 2015,/pcap/single?StreamName=remote_basic_1432975532469615104_20 remote_download_1432975370809_20150530_1742 , 84000112640, 1000000000, Sat May 30 17:42:55 2015,/pcap/single?StreamName=remote_download_1432975370809_20150 remote_download_1432975194214_20150530_1739 , 84000112640, 1000000000, Sat May 30 17:39:58 2015,/pcap/single?StreamName=remote_download_1432975194214_20150 remote_download_1432975019216_20150530_1737 , 84000112640, 1000000000, Sat May 30 17:37:03 2015,/pcap/single?StreamName=remote_download_1432975019216_20150 remote_download_1432974843210_20150530_1734 , 84000112640, 1000000000, Sat May 30 17:34:07 2015,/pcap/single?StreamName=remote_download_1432974843210_20150 remote_download_1432974666210_20150530_1731 , 84000112640, 1000000000, Sat May 30 17:31:10 2015,/pcap/single?StreamName=remote_download_1432974666210_20150 remote_download_1432974489235_20150530_1728 , 84000112640, 1000000000, Sat May 30 17:28:13 2015,/pcap/single?StreamName=remote_download_1432974489235_20150 remote_download_1432974313296_20150530_1725 , 84000112640, 1000000000, Sat May 30 17:25:17 2015,/pcap/single?StreamName=remote_download_1432974313296_20150 remote_download_1432974188843_20150530_1723 , 84000112640, 1000000000, Sat May 30 17:23:13 2015,/pcap/single?StreamName=remote_download_1432974188843_20150 remote_basic_1432974081464509952_20150530_1721 , 8400142336, 100000000, Sat May 30 17:21:26 2015,/pcap/single?StreamName=remote_basic_1432974081464509952_20 remote_basic_1432973997062732032_20150530_1720 , 8400142336, 100000000, Sat May 30 17:20:01 2015,/pcap/single?StreamName=remote_basic_1432973997062732032_20 remote_basic_1432973912520381952_20150530_1718 , 8400142336, 100000000, Sat May 30 17:18:37 2015,/pcap/single?StreamName=remote_basic_1432973912520381952_20 remote_basic_1432973828123203072_20150530_1717 , 8400142336, 100000000, Sat May 30 17:17:12 2015,/pcap/single?StreamName=remote_basic_1432973828123203072_20

PCAP Download

Downloading is also very simple and best performed with wget/CURL + an appropriately formatted URL. Simply use the following URL and replace "InsertName" with the real stream filename.

curl -u user:pass http://192.168.1.1/pcap/single?StreamName=InsertName > /capture/todays.pcap

For example if we want to download the file remote_writeback_1432983349926713088_20150530_1955 from the list above, the command is:

aaron@remote:/tmp$ curl -u user:pass http://192.168.1.1/pcap/single?StreamName=remote_writeback_1432983349926713088_20150530_1955 > /capture/todays.pcap % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 60 78.2G 60 47.3G 0 0 1010M 0 0:01:19 0:00:47 0:00:32 1024M  

.. quick side note, check out the outstanding download speed, 1GByte / second! yes, that`s a capital B for GigaByte! Our PCAP downloads run at a sustained 8GBit+ when the management port is at 10G, tho its likely the processing / disk at the other end will be the bottleneck.


There`s also some shortcuts and helpers, for example to download todays PCAP, use the stream name "today" or "last".

aaron@remote:/tmp$ curl -u user:pass http://192.168.1.1/pcap/single?StreamName=today > /capture/todays.pcap OR aaron@remote:/tmp$ curl -u user:pass http://192.168.1.1/pcap/single?StreamName=last > /capture/todays.pcap

This will set the PCAP to be downloaded as the last capture made (or currently capturing). Its a small thing but keeps your scripts even cleaner, as simple URL`s can be hardcoded into the script. You can also scan backwards a few captures using last-1 (last minus 1), last-2 (last minus 2), last-3, etc etc.

System Status

Alot of the time its a good idea to write scripts that constantly monitor each part of your infrastructure. Typically such scripts will send automated status email`s to the relevant parties. For example if you`ve scheduled the packet capture device to start at 7:00 AM you have a cron based script running on some other machine that kicks off every hour to confirm it really is capturing, how much it captured and other basic stats.

This is easily achieved by fetching the system status from the following CURL command

aaron@remote:/tmp$ curl -u user:pass http://192.168.1.1/sysmaster/status uptime, 0D 0H 59M packets_received, 1563455488 packets_dropped, 0 packets_errors, 0 packets_captured, 1563455360 bytes_captured, 106314964992 bytes_pending, 122241679360 bytes_cache, 0 bytes_disk, 9135718400 capture_link, up capture_link_uptime, 0D 0H 5M capture_link_speed, 10000 capture_bytes, 106314967948 capture_packets, 1563455410 capture_bps, 7726886912 capture_pps, 14203840 capture_name, remote_download_1434109471701 capture_active, true

.. which is pretty easy to parse and determine if the system is operating as expected.

Summary

Hopefully you can start to see the power of our 10G packet sniffer. There`s some cool things you can do with just the basic system, for example running tcpdump remotely over the wire can be done like this.

aaron@remote:/tmp$ curl -s -u user:pass http://192.168.1.1/pcap/single?StreamName=today | tcpdump -r - -nn | head reading from file -, link-type EN10MB (Ethernet) 20:45:02.717599 66:77:88:99:aa:bb > 00:33:33:33:33:33 Null Information, send seq 0, rcv seq 0, Flags [Command], length 54 20:45:02.717599 66:77:88:99:aa:bb > 00:33:33:33:33:33 Null Information, send seq 6, rcv seq 0, Flags [Command], length 54 20:45:02.717599 66:77:88:99:aa:bb > 00:33:33:33:33:33 Null Information, send seq 12, rcv seq 0, Flags [Command], length 54 20:45:02.717599 66:77:88:99:aa:bb > 00:33:33:33:33:33 Null Information, send seq 18, rcv seq 0, Flags [Command], length 54 20:45:02.717599 66:77:88:99:aa:bb > 00:33:33:33:33:33 Null Information, send seq 24, rcv seq 0, Flags [Command], length 54 20:45:02.717599 66:77:88:99:aa:bb > 00:33:33:33:33:33 Null Information, send seq 30, rcv seq 0, Flags [Command], length 54 20:45:02.717599 66:77:88:99:aa:bb > 00:33:33:33:33:33 Null Information, send seq 36, rcv seq 0, Flags [Command], length 54 20:45:02.717599 66:77:88:99:aa:bb > 00:33:33:33:33:33 Null Information, send seq 42, rcv seq 0, Flags [Command], length 54 20:45:02.717599 66:77:88:99:aa:bb > 00:33:33:33:33:33 Null Information, send seq 48, rcv seq 0, Flags [Command], length 54 20:45:02.717600 66:77:88:99:aa:bb > 00:33:33:33:33:33 Null Information, send seq 54, rcv seq 0, Flags [Command], length 54 . .

Next up we will walk though how to setup a simple packet analyzer and protocol decoder onto the system and show you why this is the ultimate packet capture device, and you should purchase immediately! :)