Highly accurate PPS clock synchronization
Pulse per Second (PPS), Stratum 3 oscillators, nanoseconds, picoseconds, femtosecond? sounds like a particle physics experiment? Not quite particle physics but not that far off as 40Gbit and 100Gbit data rates are getting seriously insane and clients appetite for extreme time accuracy continues. The primary problem is simple, how do you keep highly accurate(Stratum 3) and high resolution(1ns) time to stamp on each and every packet?
We discussed PTPv2 in a previous article with the result of about +/- 100ns in its ability to synchronize to a master clock. Sub 100ns sounds pretty good but we can do better. A Stratum 3 oscillator with 0.37PPM drift per day aka a very stable clock is a start but runs at a very low frequency. Making it kinda of useless by itself as a low granularity time stamp is not very helpful. The perfect solution would be a 1Ghz 0.37PPM (or less) oscillator to drive the timestamp logic but... thats extremely expensive and requires conversations with god leaving us mortals with high accuracy, low frequency clocks or high frequency, low accuracy clocks - you cant have both frequency and accuracy together. Our fmadio 10G packet sniffer system makes a trade off by using a high(ish) frequency clock to drive the time stamps thus giving an time granularity of +/- 4ns that is synchronized to a highly accurate 1Hz stratum 3 clock.
Nanosecond Counters
The above diagram shows how our system is separated, packet hw time stamp engine is fed a 64b nanosecond counter representing the current world time. With the generation of that counter completely separated. In cheaper (and more expensive) packet capture systems the usual approach is to increment a counter based on the period of the clock. So if the time stamp engine is running at 250MHz, it means a 4ns clock period, thus its time += 4 for every cycle @ 250MHz. It works perfectly, if your actually running at 250,000,000.0 cycles per second but its never running at exactly that frequency thus the time drifts. The actual frequency is largely based on the temperature of the oscillator and generally speaking the hotter the device the lower the frequency.
Our solution is logically the same as above, time += clock_period except we model the clock_period to an extreme degree. In fact our world time counter is 96bits, 64bits for nanoseconds, and 32bits for fraction of nanos. This enables us to tweak the clock period past the picoseconds mark, and well into the femtosecond range enabling a clock period of say 3.99999998451 nanoseconds. So what kind of accuracy do you expect from a regular oscillator running at 250MHz ?
250MHz clock drift vs PPS signal
A little explanation is required here. The above plot shows the number of 250MHz cycles counted between the positive edge/assertion of an accurate PPS signal (y axis). This value is then normalized by subtracting 250,000,000 from the actual count thus only the residuals are shown. On the X axis each bar is a single PPS signal sample, and all up about 2min worth of samples. For example if the clock was running at exactly 250.00MHz all the time, every cycle, the residuals would be near constant 0 on the y axis. What we see in the above plot is the number of cycles drifts wave like between 250,001,180 to 250,001,165 cycles with a bunch of outliers due to lost/error-nous PPS edges.
Thus clearly if we do time = time + 4ns the time will be totally out of wack fairly soon, as the clock is certainly not running at exactly 250.00MHz. Our solution is with the magic of a little software we can calculate a very accurate clock period (down to femto second) and most importantly dynamically change this period based on the PPS sampling process. When done correctly it results in a highly accurate global time and high precision/granualtity resolution.
Internal time vs PPS signal
The above plot shows the number of nanoseconds we calculated between PPS signals, minus 1,000,000,000 (y axis). Horizontal axis shows 1 bar for each PPS sample and is about 2 minutes worth of data. As you can see its good, with an error of about +/- 15 nanoseconds. Our goal is to get this to sub nanosecond range but for now its pretty dam good given our pricing and we hope you agree.
That's about it for PPS and timing, all our devices contain Stratum 3 rated clocks at 0.37PPM drift per day. Making the hardware timestamps on your packets exceptionally accurate and always on time!
