MoonGen A Scriptable High-Speed Packet Generator Paul Emmerich - - PowerPoint PPT Presentation

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen

MoonGen

A Scriptable High-Speed Packet Generator Paul Emmerich

March 4th, 2015 Chair for Network Architectures and Services Department of Informatics Technische Universit¨ at M¨ unchen

Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 1/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen

Agenda

State of the Art in Packet Generation High-Speed Multi-Core Packet Processing with DPDK and LuaJIT Hardware Timestamping on Commodity NICs Precise Rate Control

Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 2/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen

State of the Art

◮ Hardware Packet Generators

◮ (+) Precise & accurate (timestamps, rate control) ◮ (+) Fast ◮ (–) Inflexible ◮ (–) Expensive Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 3/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen

State of the Art

◮ Hardware Packet Generators

◮ (+) Precise & accurate (timestamps, rate control) ◮ (+) Fast ◮ (–) Inflexible ◮ (–) Expensive

◮ Software Packet Generators

◮ (+) Run on cheap commodity hardware ◮ (+) Some are fast ◮ ( ) Some are flexible to a certain degree

(e.g. Pktgen-DPDK, Ostinato)

◮ (–) Imprecise (timestamps, rate control) ◮ (–) Inaccurate (timestamps) Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 3/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen

Design Goals Design Goal of MoonGen Combine the advantages of both approaches while avoiding their disadvantages.

Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 4/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen

Design Goals Design Goal of MoonGen Combine the advantages of both approaches while avoiding their disadvantages.

◮ Fast: DPDK for packet I/O, explicit multi-core support ◮ Flexible: Craft all packets in user-controller Lua scripts ◮ Timestamping: Utilize hardware features found on modern

commodity NICs

◮ Rate control: Hardware features and a novel software

approach

Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 4/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen

Architecture

MoonGen Core DPDK Userscript MoonGen HW NIC NIC Port Q0 ... Qn Port Userscript

Lua VM

Userscript spawn Userscript

slave

Lua VM

Userscript

master

Lua VM

config API data API config API data API

Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 5/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen

Hardware Timestamping

◮ NICs support PTP for precise clock synchronization ◮ PTP support requires hardware timestamping capabilities ◮ These can be (mis-)used for delay measurements ◮ Typical precision

◮ ±6.4 ns (Intel 10 GbE chips) ◮ ±32 ns (Intel GbE chips)

◮ Some restrictions

◮ Packets must be UDP or PTP L2 protocol ◮ Minimum UDP packet size is 84 bytes Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 6/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen

Hardware Timestamping Precision and Accuracy

◮ Measure latencies of cables of various length ◮ Calculate coding time k and propagation speed vp

NIC t2m [ns] t8.5m [ns] t10m [ns] t20m [ns] k [ns] vp 82599 (fiber) 320 352

• 403.2

310.7 ± 3.9 0.72c ± 0.056c X540 (copper)

• 2252.8

2310.4 2195.2 ± 9.6 0.59c ± 0.065c

Timestamping Precision and Accuracy The linear behaviour and reasonable results for coding time and propagation speed show that MoonGen can measure latency with sub-microsecond precision and accuracy.

Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 7/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen

Software Rate Control in Existing Packet Generators

Loadgen NIC DuT NIC p5 p5 p4 p3 p2 p1 p0 Qmemory QNIC Wire

◮ Try to push single packets to the NIC ◮ Queues cannot be used, no batch processing ◮ NICs work with an asynchronous push-pull model ◮ Can lead to micro-bursts ◮ Unreliable, imprecise, and bad performance

Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 8/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen

Hardware Rate Control

Loadgen DuT NIC NIC p9 p9 p8 p7 p6 p5 p4 p3 p2 p1 p0 HW rate control enabled Qmemory QNIC Wire

◮ Modern NICs support rate control in hardware ◮ Limited to constant bit rate and bursty traffic ◮ Precision controlled by the hardware ◮ High performance as queues can be used

Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 9/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen

Evaluation: 500 kpps 40 MoonGen 20 Probability [%] Pktgen-DPDK 0.5 1 1.5 2 2.5 3 3.5 4 20 Inter-Arrival Time [µs] zsend

Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 10/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen

Evaluation: 1,000 kpps 20 MoonGen 20 Probability [%] Pktgen-DPDK 0.5 1 1.5 2 40 Inter-Arrival Time [µs] zsend

Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 11/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen

Software Rate Control Based on Invalid Packets

Loadgen DuT NIC NIC p6 p6 pi

3

p5 pi

4

pi p2 pi

1

p3 pi

2

p4 p1 p0 HW rate control disabled p5 Qmemory QNIC Wire

◮ Fill gaps with invalid packets pi (e.g. bad CRC) ◮ NIC in the DuT drops invalid packets without side-effects ◮ Combines advantages of both approaches ◮ Precision limited by byte rate (0.8 ns) and minimum packet

size (50 Byte)

Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 12/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen

Q & A

Try MoonGen yourself!

https://github.com/emmericp/MoonGen

Questions?

Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 13/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen

[Backup Slide] Performance I: Lua Can be Faster Than C

◮ UDP packets from varying source IP addresses

1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2 5 10 15 CPU Frequency [GHz] Packet Rate [Mpps] MoonGen Pktgen-DPDK

◮ Pktgen-DPDK needs a complicated main loop that covers

all possibilites

◮ MoonGen can use a tight inner loop

Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 14/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen

[Backup Slide] Performance II: Heavy Workload and Multi-Core Scaling

◮ Generate random UDP packets on 2 10 GBit NICs ◮ 8 calls to Lua’s standard math.random per packet ◮ CPUs artificially clocked down to 1.2 GHz

1 2 3 4 5 6 7 8 10 20 30 Number of 1.2 GHz CPU Cores Packet Rate [Mpps]

Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 15/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen

[Backup Slide] HW/SW Rate Control Details

Rate Software Bursts ±64 ns ±128 ns ±256 ns ±512 ns MoonGen 0.02% 49.9% 74.9% 99.8% 99.8% 500 kpps Pktgen-DPDK 0.01% 37.7% 72.3% 92% 94.5% zsend 28.6% 3.9% 5.4% 6.4% 13.8% MoonGen 1.2% 50.5% 52% 97% 100% 1000 kpps Pktgen-DPDK 14.2% 36.7% 58% 70.6% 95.9% zsend 52% 4.6% 7.9% 24.2% 88.1%

Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 16/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen

[Backup Slide] Effects of Bad Rate Control

◮ Interrupt rate of an Open vSwitch packet forwarder

0.5 1 1.5 2 0.5 1 1.5 ·105 Offered Load [Mpps] Interrupt Rate [Hz] Load generated with MoonGen Load generated with zsend

◮ Micro-bursts confuse dynamic interrupt throttling ◮ This affects latency (cannot be measured with zsend)

Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 17/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen

[Backup Slide] Effects of Invalid Packets

◮ Median latency of an Open vSwitch packet forwarder ◮ Packet rate controlled by hardware vs. invalid frames

0.5 1 1.5 2 −4 −2 2 4 Offered Load [Mpps] Deviation [%]

◮ Minor modifications to the DuT (e.g. an active SSH

session) result in a deviation of up to 15% with the same rate control mechanism

Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 18/13

Chair for Network Architectures and Services Technische Universit¨ at M¨ unchen

[Backup Slide] Poisson Traffic 0.5 1 1.5 2 50 100 150 Offered Load [Mpps] Latency [µs] CBR (Median) CBR (25th/75th Percentile) Poisson (Median) Poisson (25th/75th Percentile)

Paul Emmerich MoonGen – A Scriptable High-Speed Packet Generator 19/13