MoonGen A Scriptable High-Speed Packet Generator Paul Emmerich - - PowerPoint PPT Presentation

▶

Nov 06, 2022 880 likes •1.2k views

Chair for Network Architectures and Services Technical University of Munich (TUM) MoonGen A Scriptable High-Speed Packet Generator Paul Emmerich January 31st, 2016 FOSDEM 2016 Chair for Network Architectures and Services Department of

SLIDE 1

Chair for Network Architectures and Services Technical University of Munich (TUM)

MoonGen

A Scriptable High-Speed Packet Generator Paul Emmerich

January 31st, 2016 FOSDEM 2016 Chair for Network Architectures and Services Department of Informatics Technical University of Munich (TUM)

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 1

SLIDE 2

Chair for Network Architectures and Services Technical University of Munich (TUM)

Outline

Hardware vs. Software Packet Generators Architecture of MoonGen Hardware Timestamping on Commodity NICs Precise Rate Control Example Measurements

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 2

SLIDE 3

Chair for Network Architectures and Services Technical University of Munich (TUM)

Source: www.spirent.com

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 3

SLIDE 4

Chair for Network Architectures and Services Technical University of Munich (TUM)

Challenges for software packet generators

◮ Hardware packet generators are

◮ Precise ◮ Accurate ◮ Fast Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 4

SLIDE 5

Chair for Network Architectures and Services Technical University of Munich (TUM)

Challenges for software packet generators

◮ Hardware packet generators are

◮ Precise ◮ Accurate ◮ Fast

◮ Software packet generators

◮ Run on cheap commodity hardware ◮ Flexible Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 4

SLIDE 6

Chair for Network Architectures and Services Technical University of Munich (TUM)

Challenges for software packet generators

◮ Hardware packet generators are

◮ Precise ◮ Accurate ◮ Fast

◮ Software packet generators

◮ Run on cheap commodity hardware ◮ Flexible

◮ Key challenges for software packet generators

◮ Rate control ◮ Timestamping Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 4

SLIDE 7

Chair for Network Architectures and Services Technical University of Munich (TUM)

Design goals Design goal of MoonGen Combine the advantages of both approaches while avoiding their disadvantages.

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 5

SLIDE 8

Chair for Network Architectures and Services Technical University of Munich (TUM)

Design goals Design goal of MoonGen Combine the advantages of both approaches while avoiding their disadvantages.

◮ Fast: DPDK for packet I/O, explicit multi-core support ◮ Flexible: Craft all packets in user-controlled Lua scripts ◮ Timestamping: Utilize hardware features found on modern

commodity NICs

◮ Rate control: Hardware features and a novel software

approach

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 5

SLIDE 9

Chair for Network Architectures and Services Technical University of Munich (TUM)

Architecture

MoonGen Core DPDK Userscript MoonGen HW NIC NIC Port Q0 ... Qn Port Userscript

Lua VM

Userscript spawn Userscript

slave

Lua VM

Userscript

master

Lua VM

config API data API config API data API

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 6

SLIDE 10

Chair for Network Architectures and Services Technical University of Munich (TUM)

Hardware timestamping

◮ NICs support PTP for precise clock synchronization ◮ PTP support requires hardware timestamping capabilities ◮ These can be (mis-)used for delay measurements ◮ Typical precision

◮ ±6.4 ns (Intel 10 GbE chips) ◮ ±32 ns (Intel GbE chips)

◮ Some restrictions

◮ Packets must be UDP or PTP L2 protocol ◮ Minimum UDP packet size is 84 bytes Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 7

SLIDE 11

Chair for Network Architectures and Services Technical University of Munich (TUM)

Software rate control in existing packet generators

Loadgen NIC DuT NIC p5 p5 p4 p3 p2 p1 p0 Qmemory QNIC Wire

◮ Software tries to push single packets to the NIC ◮ Queues cannot be used, no batch processing ◮ NICs work with an asynchronous push-pull model ◮ This can lead to micro-bursts

Unreliable, imprecise, and bad performance

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 8

SLIDE 12

Chair for Network Architectures and Services Technical University of Munich (TUM)

Hardware rate control

Loadgen DuT NIC NIC p9 p9 p8 p7 p6 p5 p4 p3 p2 p1 p0 HW rate control enabled Qmemory QNIC Wire

◮ Modern NICs support rate control in hardware ◮ Limited to constant bit rate and bursty traffic ◮ Precision controlled by the hardware

High performance as queues can be used, but inflexible

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 9

SLIDE 13

Chair for Network Architectures and Services Technical University of Munich (TUM)

Software rate control based on invalid packets

Loadgen DuT NIC NIC p6 p6 pi

p5 pi

pi p2 pi

p3 pi

p4 p1 p0 HW rate control disabled p5 Qmemory QNIC Wire

◮ Fill gaps with invalid packets pi (e.g. bad CRC) ◮ NIC in the DuT drops invalid packets without side-effects ◮ Combines advantages of both approaches ◮ Precision limited by byte rate (0.8 ns per byte) and minimum

packet size (33 byte)

High performance & high precision

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 10

SLIDE 14

Chair for Network Architectures and Services Technical University of Munich (TUM)

Does it work?

◮ Test setup: forward packets with Open vSwitch ◮ Measure the latency of the device under test

0.5 1 1.5 2 20 40 60 80 100 120 140 160

Offered load [Mpps] Latency [µs]

CBR Hardware (Median) CBR Hardware (25/75th perc.)

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 11

SLIDE 15

Chair for Network Architectures and Services Technical University of Munich (TUM)

Does it work?

◮ Compare both rate control approaches ◮ Maximum deviation: 2%

0.5 1 1.5 2 20 40 60 80 100 120 140 160

Offered load [Mpps] Latency [µs]

CBR Hardware (Median) CBR Hardware (25/75th perc.) CBR MoonGen (Median) CBR MoonGen(25/75th perc.)

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 12

SLIDE 16

Chair for Network Architectures and Services Technical University of Munich (TUM)

Does it matter?

◮ Compare CBR with Poisson traffic ◮ Different response from the device under test

0.5 1 1.5 2 20 40 60 80 100 120 140 160

Offered load [Mpps] Latency [µs]

CBR MoonGen (Median) CBR MoonGen (25/75th perc.) Poisson MoonGen (Median) Poisson MoonGen (25/75th perc.)

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 13

SLIDE 17

Chair for Network Architectures and Services Technical University of Munich (TUM)

Example: Linux NAPI

◮ Open vSwitch on Linux ◮ Uniform distribution caused by interrupt throttling

10 20 30 40 50 60 70 0.2 0.4 0.6 0.8 Latency [µs] Probability [%]

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 14

SLIDE 18

Chair for Network Architectures and Services Technical University of Munich (TUM)

Example: Linux Virtualization (VirtIO)

◮ Open vSwitch forwarding through a VM ◮ Long tail distribution, typical for VMs

100 150 200 250 300 350 0.1 0.2 0.3 Latency [µs] Probability [%]

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 15

SLIDE 19

Chair for Network Architectures and Services Technical University of Munich (TUM)

Example: Hardware Switch

◮ AS5712-54X 10/40 GbE OpenFlow switch ◮ Bimodal distribution caused by more input than output ports

◮ Some packets are forwarded directly (cut-through switch) ◮ Some packets are blocked by another flow and buffered

1 1.5 2 2.5 3 3.5 4 2 4 6 Latency [µs] Relative Probability [%]

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 16

SLIDE 20

Chair for Network Architectures and Services Technical University of Munich (TUM)

Summary

◮ Speeds of 10 Gbit/s per CPU core (64 byte packets) ◮ Sub-microsecond precision and accuracy ◮ Execute user-defined script code for each packet ◮ Easy to use

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 17

SLIDE 21

Chair for Network Architectures and Services Technical University of Munich (TUM)

Q & A

Try MoonGen yourself!

https://github.com/emmericp/MoonGen

Questions?

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 18

SLIDE 22

Chair for Network Architectures and Services Technical University of Munich (TUM)

[Backup slide] Performance I: Lua can be faster than C

◮ UDP packets from varying source IP addresses

1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2 5 10 15 CPU frequency [GHz] Packet rate [Mpps] MoonGen Pktgen-DPDK

◮ Pktgen-DPDK needs a complicated main loop that covers all

possibilites

◮ MoonGen can use a tight inner loop

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 19

SLIDE 23

Chair for Network Architectures and Services Technical University of Munich (TUM)

[Backup slide] Performance II: heavy workload and multi-core scaling

◮ Generate random UDP packets on 2 10 Gbit NICs ◮ 8 calls to Lua’s standard math.random per packet ◮ CPUs artificially clocked down to 1.2 GHz

1 2 3 4 5 6 7 8 10 20 30 Number of 1.2 GHz CPU cores Packet rate [Mpps]

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 20

SLIDE 24

Chair for Network Architectures and Services Technical University of Munich (TUM)

[Backup slide] Performance III: 40 GbE

◮ Generate random UDP packets on 2 10 Gbit NICs ◮ 8 calls to Lua’s standard math.random per packet ◮ CPUs artificially clocked down to 1.2 GHz

64 96 128 160 192 224 256 10 20 30 40 50 Packet size [byte] Rate [Gbit/s] 1 core 2 cores 3 cores

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 21

SLIDE 25

Chair for Network Architectures and Services Technical University of Munich (TUM)

[Backup Slide] Rate control: 500 kpps 40 MoonGen (hardware) 20 Probability [%] Pktgen-DPDK 0.5 1 1.5 2 2.5 3 3.5 4 20 Inter-arrival time [µs] zsend

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 22

SLIDE 26

Chair for Network Architectures and Services Technical University of Munich (TUM)

[Backup Slide] Rate control: 1,000 kpps 20 MoonGen (hardware) 20 Probability [%] Pktgen-DPDK 0.5 1 1.5 2 40 Inter-arrival time [µs] zsend

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 23

SLIDE 27

Chair for Network Architectures and Services Technical University of Munich (TUM)

[Backup Slide] HW/SW rate control details

Rate Software Bursts ±64 ns ±128 ns ±256 ns ±512 ns MoonGen 0.02% 49.9% 74.9% 99.8% 99.8% 500 kpps Pktgen-DPDK 0.01% 37.7% 72.3% 92% 94.5% zsend 28.6% 3.9% 5.4% 6.4% 13.8% MoonGen 1.2% 50.5% 52% 97% 100% 1000 kpps Pktgen-DPDK 14.2% 36.7% 58% 70.6% 95.9% zsend 52% 4.6% 7.9% 24.2% 88.1%

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 24

SLIDE 28

Chair for Network Architectures and Services Technical University of Munich (TUM)

[Backup slide] Effects of bad rate control

◮ Interrupt rate of an Open vSwitch packet forwarder

0.5 1 1.5 2 0.5 1 1.5 ·105 Offered load [Mpps] Interrupt rate [Hz] Load generated with MoonGen Load generated with zsend

◮ Micro-bursts confuse dynamic interrupt throttling ◮ This affects latency (cannot be measured with zsend)

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 25

SLIDE 29

Chair for Network Architectures and Services Technical University of Munich (TUM)

[Backup slide] Hardware timestamping precision and accuracy

◮ Measure latencies of cables of various length ◮ Calculate encoding time k and propagation speed vp

2 4 6 8 10 12 14 16 18 20 22 100 200 300 400 Cable length [m] Time [ns] Latency

◮ Result for fiber cable: k ≈ 311ns, vp = 0.72c ± 0.056c

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 26

SLIDE 30

Chair for Network Architectures and Services Technical University of Munich (TUM)

[Backup slide] Effects of invalid packets

◮ Median latency of an Open vSwitch packet forwarder ◮ Packet rate controlled by hardware vs. invalid frames

0.5 1 1.5 2 −4 −2 2 4 Offered load [Mpps] Deviation [%]

◮ Minor modifications to the DuT (e.g. an active SSH session)

result in a deviation of up to 15% with the same rate control mechanism

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 27