Accuracy of PTPv2 clock synchronization
High resolution time stamps are one thing, accurate world time is a completely different topic. Our systems have 3.2nsec resolution hardware timestamp, however this is the timer resolution not the accuracy of keeping world time. For precise global time we support stock PTPv2 and PTPv2 augmented with a PPS (pulse-per-second) signal. As we will see below PTPv2 gives ~100nsec synchronization levels and the accuracy of PTPv2+PPS will is discussed separately.
PTPv2 is a protocol for synchronizing clocks over ethernet at sub microsecond levels. Its a master/slave system and requires ethernet packet hardware timestamping to get decent accuracy. One of the cool things about it is, there is little difference between 10G fiber vs 100Mbit CAT5 copper cables connecting the two systems. It works perfectly well over 100Mbit copper. The algorithm is a simple feedback loop that incrementally increases or reduces the estimated latency between two NIC`s based on RTT (4 timestamps). In theory this latency should be fixed, but reality is messy and there is always a slight(or large) drift due to clock stability. The stability of the clock is a function of how much the clocks/crystals cost from 10c to $10K to north of $1MM for those doing particle physics. You can read the details on PTP here and the full IEEE 1588-2008 spec here.
Top-Of-Rack capture
The above diagram shows the external ports of our systems. Where "Management Ports" are x2 dual mode SFP/SFP+ ports running at 1G or 10G either copper or fiber. PPS in/out is an SMA coax connector used for pulse-per-second synchronization. And finally the capture ports who`s timestamp is sourced from the internal clock.
A PTPv2 slave runs on the management ports which synchronizes either to a local boundary clock (usually a switch) or a GPS sourced PTP grandmaster. .. and woha.. that's alot of 3 latter acronyms on one page.. definitions are at the bottom of the page to help! An example setup synced to a boundary clock switch is shown below.
The above is a typical setup when monitoring many ports across multiple racks, as running PPS/GPS coax cabling everywhere is a royal pain in the ass. This usage model has a top of rack switch + top of rack capture device per rack where the TOR(Top of Rack) switch sources global time from the PTP grandmaster somewhere (without PPS) and the capture devices supplies the entire rack with the PPS signal. Its a blend of performance and maintenance as there is no inter rack coax required but intra-rack is synchronized to within single digit nanoseconds via PTP + PPS OUT signal for the entire rack. Rack shown in the picture below.
Local grand master
Of course if the capture device global time requires single digit nanosecond accuracy, a local GPS sourced grand master can also be placed TOR. This requires the datacenter ops guys to run coax from the GPS antenna on the roof and burns your budget too! Visualized below.
In this setup the GPS grandmaster supplies the PPS signal for the entire rack with our capture devices consuming the PPS signal and sourcing PTP either directly from the GPS grandmaster or via the TOR switch.
PTP accuracy
So the N x $20K question here is do you really need a GPS grandmaster in every rack for accurate global time ? Or phrased differently how accurate can PTPv2 synchronize two clocks. The answer is, pretty dam good but not perfect. You can achieve sub 100ns synchronization assuming a direct cable between two NIC`s and assuming they support HW time stamps for PTP packets (most cards do now).
Above chart is a time series of clock adjustments when fmadio40 first connects to a PTP master over 1G copper. The peek offset is around 7.0e4 nsec which is around 70usec (it was previously synced) with the adjustments tapering off nicely. What kind of accuracy does it eventually settle to? plot below is near the end of the timeseries.
That's nanoseconds in the vertical axis, where the time deltas applied are nicely in the 0-150ns range, making a stock PTPv2 synchronizing within +/- 100nsec. At this point (sub 100ns) we need better tools as the first major problem is the source clock is of low quality. In a later post we will dig deeper and look at single digit nanosecond deviations from a GPS sourced PPS synchronized clock and fun tricks like this.
PTP - Precision Time Protocol
PPS - Pulse per second
GPS - Global Position System
TOR - Top of Rack
SMA - SubMinature version A. Coax cable connector
CAT5 - Category 5 cable, copper 1G cabling
RTT - Round Trip Time
