Programmable Per-Packet Network Telemetry: From Wire to Kafka at Scale

Efficient and secure management of networks requires collecting and analyzing fine-grained telemetry data, preferably in real-time. Existing monitoring and analysis frameworks (e.g., Netflow, SNMP counters) do not provide fine-grained, per-packet information, are hard or not possible to customize, and do not provide an expressive programming interface to extract information. We present ESnet High Touch Services, a programmable, scalable, and expressive hardware and software solution that produces and analyzes per-packet telemetry information with nanosecond-accurate timing. We highlight our architecture, the most critical performance considerations that allow the processing of 10.4 million telemetry packets per second with only 5 CPU cores, which is more than enough to handle 127 Gbit/s of original traffic with 1512B MTU. We also present applications of the system that use real-time stream processing with elegant filtering, aggregation, and windowing functionalities. Our use-cases show that High Touch Services can support a variety of advanced performance monitoring, troubleshooting, and security tasks.