The Packet Sniffer Blog

In any environment it is important to monitor the performance of systems, whether they be servers, infrastructure or monitoring and security tools. 10G 40G 100G packet capture systems are no different. Systems monitoring in large organizations, where a significant number of tools from different vendors are used, can be piecemeal unless an industry standard method is used. For decades the standard of choice was SNMP, and this still has its place, but in pursuit of simplicity and reliability, customers have pushed for other methods of monitoring.

FMADIO are proud to once again be a sponsor of this year’s Sharkfest conference, and although this year the conference is being held online, we’ve no doubt that attendees will learn a lot.

Many times there is disbelief about our sustained 100Gbps write to disk performance, but its real. We really do full sustained 100Gbps capture to disk. To prove this we used the gold standard, a Spirent 100Gbps tester to benchmark the capture performance of our system. Here are the results.

インフラストラクチャーを動き回るトラフィック・フローの良好な可視性と理解は、全てのネットワーク・オパレーション・センター（NOC）及びセキュリティ・オパレーション・センター（SOC）にとって非常に重要であり、色々な取り組み方があります。簡単な方法、難しい方法、高価な方法、無料の方法などがあります。

Network Flow Monitoring

Cyber Security Manga!

我社のオープンソース・モニター・ツール- pcap_bpfcounter がリリースされました！

Our Opensource network monitoring tool pcap_bpfcounter is released!

Oh no, the network is down! Help!

One of the most often asked questions is how much storage do I need for N minutes of capture @ X Gbps? Or the inverse of how many Minutes of X Gbps capture does can the capture system hold? We have condensed this into some easy to read charts.

Increasing Packet Capture Port Density with FMAD 40Gbps Packet Capture Systems and a regular switch. Using our 2x40G Packet Capture System and 2 separate switches you can achieve excellent cost per port capture numbers.

We`re back again at Interop Tokyo 2017, setup and running on SHOWNET!

当社の100Gパケットキャプチャシステムについて、数多くのご意見・ご感想を頂いてまいりましたが、100Gbpsのキャプチャデータを当社が持続的に保存できるとは、ほとんどのお客様が考えておられない様です。最高の書き込み性能のピークを常に持続する為には多くの時間とリソースがかかるので、半信半疑になられるのも理解できます。　　当社のSSDについて、これから順を追ってご説明したいと思います。

We`ve received alot of feedback on our 100G packet capture system and most customers don't believe we can do sustained 100Gbps capture to storage! We understand the skepticism as its taken alot of time and resources to achieve sustained peek write performance. We explain a little on our SSD journey.

100Gイーサネットは現在増加中していますが、100Gイーサネットとは実際のところ何を意味しているのでしょう? CFP、CXP、QSFP28、10x10、4x25、SR4、LR4、FEC、非FEC...といった色々な名称が出てきます。 私たちはこの記事で掘り下げて、その組み合わせとこの分野で実際に展開されているものについて説明します。

100G Ethernet is here and on the rise, yet what does 100G ethernet really mean? There`s an alphabetical soup of CFP, CXP, QSFP28, 10x10, 4x25, SR4, LR4, FEC, no FEC we go deep in this post and explain the combinations and whats actually being deployed in the field.

Our FMADIO 100G packet capture system won Best of Show Grand Prize at Interop Tokyo 2016! A very big thank you to our strategic partner Glean Corporation and all our suppliers!

Our 100Gbps Packet capture system is a Interop Tokyo 2016 Finalist in the Monitoring / Measurment / Test section!

思いのほか時間が掛かってしまいましたが、この度、遂に100Gパケットキャプチャシステムが完成したことをお知らせします! FPGA、SSD、スタイリッシュな2U 48ベイシャシーを組み合わせた画期的なシステムで、価格は1千万円以下とお求めやすくなっています。

Its certainly taken longer than expected but we can finally announce our full 100G Packet capture system! Its an incredible combination of FPGA`s, SSD`s and slick 2U 48 bay chassis all for a modest ~ $100K USD.

New features on our 20GbE packet capture system has been released, automatic Multicast join`s. Usually packet capture runs using a SPAN/Mirror port Layer 1 optical taps with the capture port being strictly Rx only. Yet sometimes this isn't enough and the capture port needs to manually join multiple sessions.

PCAP 圧縮はシステムが使うストレージ量を大幅に削減するので、あらゆる真剣なパケットキャプチャ作業に必須です。キャプチャ中の圧縮は開発中ですが、我々はダウンロードの際のリアルタイム GZIP をサポートするようになりました。これはダウンロード帯域幅の削減に効果的で、Wireshark にもビルトインで .pcap.gz のサポートがあります。

PCAP compression is essential for any serious packet capture work as it dramatically reduces the amount of storage the system uses. While compressing during capture is in the development pipeline, we now support realtime GZIP on download. Which is great for reducing download bandwidth, disk space and Wireshark has built in support for .pcap.gz too.

Packet Capture in the Cloud, e.g. AWS/EC2/VPC is a some what difficult subject, as the network topologies in the cloud vs traditional infastructure are extremely different. The primary difference is the cloud operates at Layer 3 (IP endpoints) where as traditional inhouse networks operate at layer 1 & 2 (Physical cables + MAC endpoints). This means the traditional layer 2 SPAN/TAP approach no longer works in virtualized network environment, but we have a solution, FMADIO AWS Cloud Packet Capture!

Scripting is an essential part of the FMADIO packet capture systems as packet capture is only the first step in the process of understanding your network traffic. We have released a small (but growing) library of commands to help integrate into your environment.

あなたの使っているスイッチのレイテンシを知っていますか?これはSDN(software defined networking: ソフトウェア定義ネットワーク)の特徴を述べる時によく聞かれる質問であり、HFT(high frequency trading:高頻度取引)ではどこでも非常に重要なことです。簡単な質問のように見えますし、私達のマイクロバースト・アナライザーを使えば答えを得るのは非常に簡単です。

Ever wondered what latency your switch is running at? Its an often asked question when characterizing SDN devices and also extremely important for everything HFT. It sounds like a simple question, and with our Microburst Analyzer its extremely simple to get answers.

When you have 10`s or 100`s of real (or virtual) servers it becomes quite a task to monitor and manage what their all doing. FMADIO 10G, 20G, 40G and 100G packet capture systems now support SNMP using the standard net-snmp package so you can easily track and monitor your systems.

パケットキャプチャシステムの興味深く不明確な問題のひとつはすべてのデータを保管するファイルシステムにあるエンジニアリングです。標準のEXT4 / XFS / JFS /の中から好きなファイルシステムを選べば良いと思うかもしれませんが、パケットロスをひき起こす書き込みパフォーマンスが持続する保証はないため、これは大変重大な赤信号です。

One of the interesting and non-obvious problems of a packet capture system is engineering the underlying filesystem to store all data. You might think writing to a standard EXT4 / XFS / JFS / pick your favorite file system is enough. However this is a huge huge red flag, as there is no guarantee on the sustained write performance, which can result in packet loss.

The magic and ubiquity of image sensors over the last decade has been incredible, and its not only Megapixels of your camera but also sensing the non-visible light spectrum. Have been wanting to check the thermal footprint of our 20GbE and 10Gbe packet capture devices for a while now. After all our competition uses large and bulky 3U, 4U, 5U or more devices, our packet capture systems from 10G to 100G are all a compact 1U form factor. This compact and lightweight form factor makes thermal management and optimization absolutely critical.

10GbE、20GbE、40GbE、100GbEパケットキャプチャの素晴らしい点のひとつはネットワーク破損を含めてすべてをキャプチャする事です！エラーは全体のパケットのペイロード全体で計算された32ビットのCRCであるイーサネットフレームチェックシーケンス（FCS）の形態をとります。この投稿ではWANリンクに良く見られるネットワークをどのようにキャプチャし、エラーをデバッグするツールをご紹介します。

One of the great things about 10GbE, 40GbE and 100GbE packet capture is it captures everything, included network corruption! Errors can take the form of Ethernet Frame Check Sequence (FCS) which is a 32bit CRC calculated across the entire packets payload. This post discuss how we capture and some of the tools to help debug errors on the network typically seen on WAN links.

弊社によく質問が来ます。「なぜあなた達の10Gbpsパケットキャプチャはそんなにお買い得ですか？」ほとんどのお客様が弊社の値段を信じられないようです。弊社の値段とそのようなお買い得なパケットキャプチャシステムを提供できるのには秘密があります。それは魔法ではなく高品質エンジニアリングです。この記事でその方法と理由をご説明します。

We get asked alot, "why is your 10Gbps Packet capture price so good?". Almost all our clients have some disbelief about our prices. There`s a secret to our pricing and how we achieve such affordable packet capture systems. Its`s not magic just high quality engineering, we discuss the how`s and whys in this post.

ついに弊社の20Gbpsデュアルポート 2x10Gキャプチャシステムが利用可能になりました！ 30M パケット/秒処理は簡単ではないので想定よりも多くの作業が発生しましたが、ついに完成し注文可能となりました。

Finally our 20Gbps dual port 2x10G capture system is available! Its been quite a bit more work than expected as 30M Packets/Sec processing isn`t easy, but we`ve done it and open for orders.

パケットキャプチャはとても便利ですが、あなたは具体的にネットワークのパケットをどのようにキャプチャしたいですか？レイヤ1光学タップからスマートSPANプロトコルやその中間といった様々なアプローチがあります。私達はそれぞれのアプローチに対する長所・短所を分析してみました。

Packet capture is good and all, but how exactly do you capture packets on a network ? There`s a range of approaches from Layer1 optical TAP`s to Smart SPAN protocols and everything in between. We dig into the various Pro`s and Con`s of each approach.

All our production inventory is purchased via our trusted partners but for a R&D project we received not 1 but 2 fake CPU`s from Amazon.com. Still can`t beleive it, espeically since its not some random 3rd party seller but the seller was Amazon itself!

Our FMADIO10 and FMADIO20 products are unique by operating in a hybrid SSD / HDD mode. However the lower HDD sustained throughput reduces the sustained capture rate when used in continuous high bandwidth 24H / 7 capture mode. We dig deeper and understand whats happening under the hood for long run capture and what the system limitations are.

Network microburst analysis is important for all network topologies to trouble shoot problems, understand network flow and optimization throughput/latency. FMADIO products have a powerful builtin microburst analyzer to assist and answer any microburst questions you can ask.

ネットワークマイクロバースト解析は問題のトラブルシューティング、ネットワークフローの理解、スループット/待ち時間の最適化という全てのネットワークトポロジーにとって重要です。FMADIO製品は強力な内臓マイクロバースト解析機を含み、あなたのマイクロバーストに対する疑問に答えるお手伝いをします。

Its an often used cliche "only in Japan" do you get XYZ, usually some sort of outrageous TV show, strange food or something worse. I`ve been here in Tokyo for long enough that nothing really surprises me these days but once in a while you find something that's pretty cool.

Often in networking technology circles we hear the term "line rate" such as "10Gbit line rate" which roughly translates to network traffic at the maximum capacity and in this case at 10Gbit/second. While it sounds clear, the actual data transfer rate can be a bit non-intuitive.

ネットワーキングテクノロジーサークルでは良く「10Gbpsラインレート」のような「ラインレート」という用語を耳にします。これは簡単に言うとネットワークトラフィックの最大容量で、この場合10Gbit/秒になります。簡単に聞こえますが実際のデータ転送レートについてはちょっと分かりづらいです。

Good cost effective full line rate packet sniffers are great, but by them self aren`t all that helpful. The power of packet capture is when its coupled with analyzer software, such as opensource tools like Snort or Suricata, our opensource tools or your customized analysis software. We understand this and our device has exceptional integration that's easy, simple, and enables you to go from 0 to 100 within your existing infrastructure in no time flat.

The accuracy of 1PPS otherwise known as 1 Pulse per second is quite an interesting topic. We discussed how our 40G packet sniffer system calculated extreme nano accuracy time before, whats missing is how accurate our 10G packet sniffer can synchronize UTC world time using the onboard GPS PPS signal thats included with every system.

Packet sniffers are great (especially ours :) but a 100TB of packet captures is pretty useless without any tools for analysis. We understand this and are building up a small suite of packet analyzers that anyone can use. Better yet its fully open source on GitHub and starts with a tools to analyze packet latency.

One of the great thing about server motherboards is their remote KVM functionality, IPMI interfaces and general ability to run head-less. We use Supermicro boards which have a full featured graphical Java based KVM client you can run from a browser, which is cool. But sometimes in restrictive network topologies its would be simpler and easier to SSH into the serial port.

Ever spent hours waiting for a capture to download when all you want is a few MB worth of packets? Or spent hours waiting for Wireshark to load and process a PCAP file ? We certainly have, and it totally sucks ass. Your either running "needle in a haystack" network troubleshooting or deep dive packet analytics. In both cases its more efficient to download only parts of the data for analysis as working with 100GB+ PCAP files can stretch and break many tools.

Hacking together a 10G packet capture system from parts is pretty easy these days, yet making that system absolutely rock solid is a completely different ball game. We`ve spent so much time fixing the 1 in 1000 type of errors which is time consuming and painful but absolutely critical to creating a rock solid packet capture system.

Sadly most network packet capture systems like to talk about how shiny fast their capture rate is or the huge amount of CPU/RAM/IO/X the system has. But how often do they discuss reliability of stored captured data? after all whats the point of capturing all this data only to have it bit-rot away?

Pulse per Second (PPS), Stratum 3 oscillators, nanoseconds, picoseconds, femtosecond? sounds like a particle physics experiment? Not quite particle physics but not that far off as 40Gbit and 100Gbit data rates are getting seriously insane and clients appetite for extreme time accuracy continues. The primary problem is simple, how do you keep highly accurate(Stratum 3) and high resolution(1ns) time to stamp on each and every packet?

High resolution time stamps are one thing, accurate world time is a completely different topic. Our systems have 3.2nsec resolution hardware timestamp, however this is the timer resolution not the accuracy of keeping world time. For precise global time we support stock PTPv2 and PTPv2 augmented with a PPS (pulse-per-second) signal. As we will see below PTPv2 gives ~100nsec synchronization levels and the accuracy of PTPv2+PPS will is discussed separately.

After months of slaving away, problem after problem finally, yes finally! fmadio40 the 1x40G or 4x10G line rate packet capture system gets its first demo in RL (real life). There`s alot of info on the blog about the system but its completely different when your there talking with a real live person about whats great, whats ok and what sucks.

We`ve spent a fair amount of time talking about our capture hardware so lets switch gears and dig into the software interface. Interaction with our 10G, 40G and 100G ethernet capture systems is over HTTP or HTTPS. You can use a web browser on port 80 or CURL and HTTPS on port 1337, its completely configurable and only uses a single TCP port. Alternatively there is also a WebDAV interface if your deep in the colo and without a browser.

SSD`s are the life blood of our fmadio 10G, 40G, 100G ethernet capture systems. They provide backing storage that well exceeds that of conventional RAM, are super fast and light weight. Thus its prudent to keep the SSD farms in top nick, optimal health, and occasionally perform emergency room brain surgery on them. Unfortunately today is one of those days.

Finally after weeks of searching, designing and plenty of cursing, Chassis Number One has finally arrived at the office! Its been surprisingly frustrating as the market for low volume customized chassis is hmmm... small. Luckily we did find a company in the US of A (of all places) which specializes in exactly that, low volume high quality custom chassis.

Bit encoding is an art unto itself, where there`s plenty of different encoding schemes to choose from. There`s 8b/10b encoding of PCIexpress Version 2, the 128b/130b encoding of PCIExpress Version 3, 64b/67b of interlaken and of course stock 64b/66b encoding used in all 10G, 40G and 100G ethernet protocols.

what is a bit, is in the eye of the beholder

We previously discussed how packets are framed in hardware, specifically how do you know when a packet starts and ends using application level signaling. Now we enter the world of the XGMII protocol, this is a (relatively) sane protocol where you can decode it by eyeballing the data e.g. 0x1234 on the wire usually means the packet contains data 0x1234. XGMII stands for X(roman 10)-G-Media-Independant-Interface which is IEEE 802.3 Clause 46 ratified specification enabling a variety of PHY and MAC chips from different vendors to talk the exact same protocol. In the FPGA world where the MAC and PHY are implemented in the same chip technically this interface layer is not required, as the MAC and PHY will be grouped into a single IP Library with a single public interface. However in our own 10G, 40G, 100G ethernet capture system we did separate these layers because its a clear and obvious way to decompose the complexity of the problem.

In the previous post we went over the components of a 10G Layer 1 ethernet system, MAC+PCS+PMA+MDI yet for those without a software background there`s some more fundamental problems that need address before drilling deep into the guts of the Layer 1 dungeon. Hardware systems are just different, its not orthogonal to software, its more a parallel universe connected via a wormhole. You need to think differently, de-construct problems differently and use completely different data structures and algorithms, so lets starts with a seemly simple problem.

In hardware how do you know when a packet starts and when it ends?

あなたはネットワーキングイーサネットの天才？tcpdumpカンフーマスター？それとも朝ごはんにジャンボパケットを食べますか？それであれば「レイヤ1」迷宮へようこそ。あなたの知識を全て忘れてください。私たちは、トランシーバ、クロック・リカバリ、DCバランス、エンコーディングシンボルセット、スクランブル、レーンスキューなどについて話します。10G、40G、100Gネットワークでは、あなたが配線を通ると思っているデータが実際はケーブルないでは完全に異なる1や0であることに驚くでしょう。

まずは10Gイーサネットシステムの基本的なビルディング・ブロックについて話しましょう。上図は主要コンポーネントを英語で記述しています。（802.3条項49ではありません）

Think your`re an networking ethernet guru ? tcpdump kung fu master? eat jumbo packets for breakfast? Then welcome to the "Layer 1" dungeon, forget everything you know. We are talking about transceiver, clock recovery, DC balance, encoding symbol sets, scrambling, lane skew and more. For 10G, 40G and 100G networks you will be surprised that, what data you think is going down the wire, is actually a completely different set of 1`s and 0`s in the cable.

Lets start with the basic building blocks of a 10G ethernet system. The diagram above maps out the major components in english (not 802.3 clause 49 legalize).

Ladies and gentelmen... Start your engines!

We have completed our prototype 100% network capture system and now in the later stages of transforming this into a world class production system. I think you will be impressed with the specs and even more impressed how much bang for buck you get. Stay tuned for updates.