MoonGen A Scriptable High-Speed Packet Generator Paul Emmerich - PowerPoint PPT Presentation

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen MoonGen A Scriptable High-Speed Packet Generator Paul Emmerich March 4th, 2015 Chair for Network Architectures and Services Department of Informatics Technische Universit¨ at M¨ unchen Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 1/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen Agenda State of the Art in Packet Generation High-Speed Multi-Core Packet Processing with DPDK and LuaJIT Hardware Timestamping on Commodity NICs Precise Rate Control Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 2/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen State of the Art ◮ Hardware Packet Generators ◮ (+) Precise & accurate (timestamps, rate control) ◮ (+) Fast ◮ (–) Inflexible ◮ (–) Expensive Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 3/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen State of the Art ◮ Hardware Packet Generators ◮ (+) Precise & accurate (timestamps, rate control) ◮ (+) Fast ◮ (–) Inflexible ◮ (–) Expensive ◮ Software Packet Generators ◮ (+) Run on cheap commodity hardware ◮ (+) Some are fast ◮ ( ) Some are flexible to a certain degree (e.g. Pktgen-DPDK, Ostinato) ◮ (–) Imprecise (timestamps, rate control) ◮ (–) Inaccurate (timestamps) Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 3/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen Design Goals Design Goal of MoonGen Combine the advantages of both approaches while avoiding their disadvantages. Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 4/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen Design Goals Design Goal of MoonGen Combine the advantages of both approaches while avoiding their disadvantages. ◮ Fast: DPDK for packet I/O, explicit multi-core support ◮ Flexible: Craft all packets in user-controller Lua scripts ◮ Timestamping: Utilize hardware features found on modern commodity NICs ◮ Rate control: Hardware features and a novel software approach Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 4/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen Lua VM Architecture Lua VM Lua VM Userscript Userscript Userscript Userscript Userscript master slave spawn config API data API MoonGen MoonGen Core config API data API DPDK Q 0 ... Q n HW NIC NIC Port Port Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 5/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen Hardware Timestamping ◮ NICs support PTP for precise clock synchronization ◮ PTP support requires hardware timestamping capabilities ◮ These can be (mis-)used for delay measurements ◮ Typical precision ◮ ± 6.4 ns (Intel 10 GbE chips) ◮ ± 32 ns (Intel GbE chips) ◮ Some restrictions ◮ Packets must be UDP or PTP L2 protocol ◮ Minimum UDP packet size is 84 bytes Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 6/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen Hardware Timestamping Precision and Accuracy ◮ Measure latencies of cables of various length ◮ Calculate coding time k and propagation speed v p NIC t 2 m [ns] t 8.5 m [ns] t 10 m [ns] t 20 m [ns] k [ns] v p 82599 (fiber) 320 352 - 403.2 310.7 ± 3.9 0.72 c ± 0.056 c X540 (copper) - - 2252.8 2310.4 2195.2 ± 9.6 0.59 c ± 0.065 c Timestamping Precision and Accuracy The linear behaviour and reasonable results for coding time and propagation speed show that MoonGen can measure latency with sub-microsecond precision and accuracy. Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 7/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen Software Rate Control in Existing Packet Generators DuT Loadgen p 5 NIC NIC p 5 p 4 p 3 p 2 p 1 p 0 Q memory Q NIC Wire ◮ Try to push single packets to the NIC ◮ Queues cannot be used, no batch processing ◮ NICs work with an asynchronous push-pull model ◮ Can lead to micro-bursts ◮ Unreliable, imprecise, and bad performance Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 8/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen Hardware Rate Control HW rate control DuT Loadgen enabled p 9 NIC NIC p 9 p 8 p 7 p 6 p 5 p 4 p 3 p 2 p 1 p 0 Q memory Q NIC Wire ◮ Modern NICs support rate control in hardware ◮ Limited to constant bit rate and bursty traffic ◮ Precision controlled by the hardware ◮ High performance as queues can be used Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 9/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen Evaluation: 500 kpps MoonGen 40 Probability [%] 0 Pktgen-DPDK 20 0 zsend 20 0 0.5 1 1.5 2 2.5 3 3.5 4 Inter-Arrival Time [ µ s] Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 10/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen Evaluation: 1,000 kpps MoonGen 20 Probability [%] 0 Pktgen-DPDK 20 0 zsend 40 0 0.5 1 1.5 2 Inter-Arrival Time [ µ s] Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 11/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen Software Rate Control Based on Invalid Packets HW rate control DuT Loadgen disabled p 6 NIC NIC p i p i p i p i p i p 6 p 5 p 5 p 4 p 3 p 2 p 1 p 0 4 3 2 1 0 Q memory Q NIC Wire ◮ Fill gaps with invalid packets p i (e.g. bad CRC) ◮ NIC in the DuT drops invalid packets without side-effects ◮ Combines advantages of both approaches ◮ Precision limited by byte rate (0.8 ns) and minimum packet size (50 Byte) Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 12/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen Q & A Try MoonGen yourself! https://github.com/emmericp/MoonGen Questions? Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 13/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen [Backup Slide] Performance I: Lua Can be Faster Than C ◮ UDP packets from varying source IP addresses Packet Rate [Mpps] 15 10 MoonGen 5 Pktgen-DPDK 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2 CPU Frequency [GHz] ◮ Pktgen-DPDK needs a complicated main loop that covers all possibilites ◮ MoonGen can use a tight inner loop Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 14/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen [Backup Slide] Performance II: Heavy Workload and Multi-Core Scaling ◮ Generate random UDP packets on 2 10 GBit NICs ◮ 8 calls to Lua’s standard math.random per packet ◮ CPUs artificially clocked down to 1.2 GHz Packet Rate [Mpps] 30 20 10 0 1 2 3 4 5 6 7 8 Number of 1.2 GHz CPU Cores Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 15/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen [Backup Slide] HW/SW Rate Control Details Rate Software Bursts ± 64 ns ± 128 ns ± 256 ns ± 512 ns MoonGen 0.02% 49.9% 74.9% 99.8% 99.8% 500 kpps Pktgen-DPDK 0.01% 37.7% 72.3% 92% 94.5% zsend 28.6% 3.9% 5.4% 6.4% 13.8% MoonGen 1.2% 50.5% 52% 97% 100% 1000 kpps Pktgen-DPDK 14.2% 36.7% 58% 70.6% 95.9% zsend 52% 4.6% 7.9% 24.2% 88.1% Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 16/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen [Backup Slide] Effects of Bad Rate Control ◮ Interrupt rate of an Open vSwitch packet forwarder · 10 5 Interrupt Rate [Hz] 1.5 Load generated with MoonGen Load generated with zsend 1 0.5 0 0 0.5 1 1.5 2 Offered Load [Mpps] ◮ Micro-bursts confuse dynamic interrupt throttling ◮ This affects latency (cannot be measured with zsend) Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 17/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen [Backup Slide] Effects of Invalid Packets ◮ Median latency of an Open vSwitch packet forwarder ◮ Packet rate controlled by hardware vs. invalid frames 4 Deviation [%] 2 0 − 2 − 4 0 0.5 1 1.5 2 Offered Load [Mpps] ◮ Minor modifications to the DuT (e.g. an active SSH session) result in a deviation of up to 15% with the same rate control mechanism Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 18/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen [Backup Slide] Poisson Traffic CBR (Median) CBR (25th/75th Percentile) 150 Poisson (Median) Poisson (25th/75th Percentile) Latency [ µ s] 100 50 0 0 0.5 1 1.5 2 Offered Load [Mpps] Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 19/13