The Packet Sniffer Blog

Often in networking technology circles we hear the term "line rate" such as "10Gbit line rate" which roughly translates to network traffic at the maximum capacity and in this case at 10Gbit/second. While it sounds clear, the actual data transfer rate can be a bit non-intuitive.

The first thing to know about 10Gbit line rate is, its not exactly 10,000,000,000 bits / second but slightly higher. How much higher? 10.3125Gbit`s to be precise which we discussed in a previous post HERE. The summary of that post is, to achieve 10.0000Gbps of actual data you need an extra 312.5Mbps worth of bandwidth for control data. The post goes into gory detail about how data is actually encoded onto the wire, which might surprise you.

The key point is, that 64B ICMP ping data you see in tcpdump "on the wire" is only part of what is actually going down the wire. We discussed how such a packet gets translated into XGMII in a previous post All about XGMII which goes into the details of the first 10G ethernet layer of abstraction. The summary is each data packet is surrounded by an ethernet preamble, frame check sequence, ethernet epilogue and the minimum inter-frame-gap, as shown below.

In this example each section is as follows:

When a vendor (such as us) boasts about "line rate" performance, it means the device can process the maximum amount of data payload that can be serialized using the protocol shown above. So lets makes this more concrete and calculate what "10Gbit line rate 64B packets" actually means, as its the gold standard as far as network devices are concerned.

10Gbit Line Rate 64Byte

Starting with the 64B packets part, which means 64Bytes between Ethernet Preamble and Ethernet Epilogue and translates to 60B of data and 4B of FCS. This is not as typically assumed 64B worth of data payload!

Which makes a 60 byte data packet, bloat out to 84B in total on the wire. Next we calculate total number of these packets that can be pushed at exactly 10.00Gbps. 10.00e9 bits / (8 bits * 84 bytes) = 14.88e6 packets And we see that familiar 14.88M packets per second number that everyone likes to write. That means on a 10 Gbit link the maximum number of packets with 60 bytes of data (plus 4B FCS) is 14.88M packets. But what is the actual data transfer rate ? 14.88e6 packets * 60 bytes * 8 bits = 7.14Gbps So while we`re sending 60 byte packets as fast as possible, consuming the full 10Gbps worth of bandwidth we`re only sending 7.14Gbps worth of data!

10Gbit Line Rate 1500 Byte

From the above is pretty clear using 64B packets to acheive maximum bandwidth aint such a good idea, so lets see what the numbers are for a 1500 byte packets.

.. and crunching the above to get the number of packets per second @ 10.0Gbps

which results in the following amount of data bandwidth

Meaning sending 1500B packets at line rate down a 10Gbps link results in 9.85 Gbps worth of actual payload data. Certainly alot better than 7 Gbps but even with 1500B packets you can`t hit the full 10.0Gbps worth of data.

TCP 10Gbit Line Rate 1500 Byte Packets

... but above calculation of 9.85Gbps is kind of miss leading, as its only the raw ethernet frame bandwidth. In the real world packets use higher level protocols such as FTP, SSH, HTTP, HTTPS, etc etc and 99% of the time over a TCP connection. Thus a new question: using a 1500 byte ethernet frame, what is the real payload bandwidth of a TCP packet?

As you can see from the table above, its 1426 B of data payload wraped in 1524B worth of 10G bandwidth. If we do the math on this

With the resulting effective TCP payload bandwidth

Its not bad but we`ve effectively lost close to 1Gbps of bandwidth from the ethernet + IP + TCP encoding. So if your getting high 8.xx Gbps numbers in a TCP bandwidth test, your doing pretty dam well.

Summary

Its nice to think your fancy new 10Gbps link can push 10Gbps worth of market data, porn and lol catz (in that priority) but it never can and never does. Even the above is an oversimplification as technically its an "average inter frame gap of 12 bytes". As almost all hardware has alignment restrictions on when a packet can start/stop within a XGMII cycle. For example our own FPGA based traffic generator outputs 8B IFG, then 16B IFG, then 8B IFC.. etc making the average IFG (8 + 16 + 8 + 16 ...) / Packet Count = 12B.

Hope that clears up any confusion over what 10Gbit Line rate is and 10G ethernet != 10Gbps of data.