AWS Cloud Packet Capture
Packet Capture in the Cloud, e.g. AWS/EC2/VPC is a some what difficult subject, as the network topologies in the cloud vs traditional infastructure are extremely different. The primary difference is the cloud operates at Layer 3 (IP endpoints) where as traditional inhouse networks operate at layer 1 & 2 (Physical cables + MAC endpoints). This means the traditional layer 2 SPAN/TAP approach no longer works in virtualized network environment, but we have a solution, FMADIO AWS Cloud Packet Capture!
These days the reason to setup and deploy your own hardware in the datacenter get smaller and smaller each year. For us, it means the market for physical packet capture devices becomes narrower and more focused as everyone migrates their infra to the cloud.
Why the cloud? Because the cloud is cheaper than running your own servers? not really. IMHO its because the entire AWS eco system is designed for running web apps, you get gigantican sized scalability, and the brilliance of trading OpEx for CapEx using a pay as you go model. The cloud has changed the game on so many levels, packet capture included too!
Is Hardware Dead? Hardware packet capture is absolutely NOT dead. On the contary our network data consumption rate is growing at insane rates, and somewhere on that datapath there`s a packet capture device running. But it does mean the market for high quality packet capture becomes much narrower, specifically for onsite monitoring and security and a few specality markets such as HFT.
Virtualized Network Fabrics
Amazon Web Services (AWS) Network fabric is an entirely Layer 3 IP peer-to-peer routing network, which judging by latencies is some sort of customized SDN stack. This is completely different to a traditional server/rack setup Layer 1/2/3 network where you have control over all 1/2/3 layers. For AWS the only access points are at Layer 3 (IP networks). Amazon could have included support for a SDN based network SPAN ports however at present (2016/1) they do not. The result is we need to use inline packet capture, specifically Bump-In-The-Wire Layer 3 style packet capture to acheive a (almost) passive packet capture.
Inline Packet Capture?
Inline packet capture is not the best tool in the toolbox, however its currently the only tool at this point in time. It works by forwarding an IP Packet to an alternate IP Address on the network and taking a copy along the way.
The above system block diagram shows the basic layout of our system. Packets arrive on one virtualized NIC port, and are forwarded to the other port with a capture taken and stored in the process. Note that, the IP address will also change in the process as it gets forwarded.
The forwarding setup is probably unusual but straight forward (no pun intended), consider it a front end or virtualized (forwarding) NIC running infront of some service running on an EC2 network interface.
Macine 2 Machine Bump In the Wire
An example EC2 instance to EC2 instance example config and routing table is shown below. In this case we`re setup as a bump in the wire between two IP endpoints, e.g. Machine A -> FMADIO.AWS -> Machine B.
The important points are:
1) atleast one interface on the capture machine instance is dedicated to capture (hilighted in green). This enables the user-space offload stack (Intel DPDK) to fully take over the hardware stack and provide optimial thoughput with minimum jitter.
2) IP Forwarding Table (hilighted in blue). This is 1-1 mapping for each IP forwarding rule the system receives. It is fully run-time editable with 3 different packet capture opations, Igress, Egress, Inline.
Ingress Egress Inline
As we`re capturing Bump In The Wire style, there`s 3 options for capturing.
Ingress
This captures the exact ingress packet. Problem is the IP Destination address will be the capture port`s IP.
Egress
This captures the exact egress packet. Problem is the IP Source address will be the capture port`s IP.
Inline
This modifies the packet, such that the IP Source is from Machine A, and the IP Destination is Machine B. This is the best option to make analysis simpler, but the PCAP data is not 100% exactly what was on the wire. If your paranoid and need the exact wire data, its best to capture both Ingress and Egress.
More Info?
If your looking to dig deeper and give our system a quick try, head over the the documentation page at: FMADIO.AWS Documentation page or AWS Detail page Here and spin up an instance now.
Summary
We`re not sure how you will use our cloud packet capture system, there`s a small EC2 instance avaliable now for testing with support for the large 48TB local storage machine type (hs1.8xlarge) avaliable shortly.
As always send us your feedback, questions and feature requests and have fun with full packet capture in the cloud.
