¨ Munchen Technische Universitat ¨

Chair for Network Architectures and Services

MoonGen A Scriptable High-Speed Packet Generator Paul Emmerich March 4th, 2015 Chair for Network Architectures and Services Department of Informatics ¨ Munchen Technische Universitat ¨

Paul Emmerich

MoonGen – A Scriptable High-Speed Packet Generator

1/13

Chair for Network Architectures and Services

¨ Munchen Technische Universitat ¨

Agenda

State of the Art in Packet Generation

High-Speed Multi-Core Packet Processing with DPDK and LuaJIT

Hardware Timestamping on Commodity NICs

Precise Rate Control

Paul Emmerich

MoonGen – A Scriptable High-Speed Packet Generator

2/13

Chair for Network Architectures and Services

¨ Munchen Technische Universitat ¨

State of the Art

I

Hardware Packet Generators I I I I

Paul Emmerich

(+) Precise & accurate (timestamps, rate control) (+) Fast (–) Inflexible (–) Expensive

MoonGen – A Scriptable High-Speed Packet Generator

3/13

Chair for Network Architectures and Services

¨ Munchen Technische Universitat ¨

State of the Art

I

Hardware Packet Generators I I I I

I

Software Packet Generators I I I

I I

Paul Emmerich

(+) Precise & accurate (timestamps, rate control) (+) Fast (–) Inflexible (–) Expensive (+) Run on cheap commodity hardware (+) Some are fast ( ) Some are flexible to a certain degree (e.g. Pktgen-DPDK, Ostinato) (–) Imprecise (timestamps, rate control) (–) Inaccurate (timestamps)

MoonGen – A Scriptable High-Speed Packet Generator

3/13

Chair for Network Architectures and Services

¨ Munchen Technische Universitat ¨

Design Goals

Design Goal of MoonGen Combine the advantages of both approaches while avoiding their disadvantages.

Paul Emmerich

MoonGen – A Scriptable High-Speed Packet Generator

4/13

Chair for Network Architectures and Services

¨ Munchen Technische Universitat ¨

Design Goals

Design Goal of MoonGen Combine the advantages of both approaches while avoiding their disadvantages. I

Fast: DPDK for packet I/O, explicit multi-core support

I

Flexible: Craft all packets in user-controller Lua scripts

I

Timestamping: Utilize hardware features found on modern commodity NICs

I

Rate control: Hardware features and a novel software approach

Paul Emmerich

MoonGen – A Scriptable High-Speed Packet Generator

4/13

¨ Munchen Technische Universitat ¨

Chair for Network Architectures and Services

Lua VM

Userscript

Architecture Lua VM

Lua VM

Userscript Userscript

Userscript master

slave

spawn

MoonGen

config API

data API MoonGen Core

config API

data API DPDK

HW

Q0 ... Qn NIC

NIC Port

Paul Emmerich

MoonGen – A Scriptable High-Speed Packet Generator

5/13

Chair for Network Architectures and Services

¨ Munchen Technische Universitat ¨

Hardware Timestamping

I

NICs support PTP for precise clock synchronization

I

PTP support requires hardware timestamping capabilities

I

These can be (mis-)used for delay measurements Typical precision

I

I I

I

Some restrictions I I

Paul Emmerich

±6.4 ns (Intel 10 GbE chips) ±32 ns (Intel GbE chips) Packets must be UDP or PTP L2 protocol Minimum UDP packet size is 84 bytes

MoonGen – A Scriptable High-Speed Packet Generator

6/13

¨ Munchen Technische Universitat ¨

Chair for Network Architectures and Services

Hardware Timestamping Precision and Accuracy

I

Measure latencies of cables of various length

I

Calculate coding time k and propagation speed vp

NIC

t2m [ns] t8.5m [ns] t10m [ns] t20m [ns]

82599 (fiber) X540 (copper)

320 -

352 -

2252.8

k [ns]

vp

403.2 310.7 ± 3.9 0.72c ± 0.056c 2310.4 2195.2 ± 9.6 0.59c ± 0.065c

Timestamping Precision and Accuracy The linear behaviour and reasonable results for coding time and propagation speed show that MoonGen can measure latency with sub-microsecond precision and accuracy.

Paul Emmerich

MoonGen – A Scriptable High-Speed Packet Generator

7/13

¨ Munchen Technische Universitat ¨

Chair for Network Architectures and Services

Software Rate Control in Existing Packet Generators Loadgen p5

DuT NIC

p5

Qmemory

NIC p4

QNIC

p3

p2

p1 p0

Wire

I

Try to push single packets to the NIC

I

Queues cannot be used, no batch processing

I

NICs work with an asynchronous push-pull model

I

Can lead to micro-bursts

I

Unreliable, imprecise, and bad performance

Paul Emmerich

MoonGen – A Scriptable High-Speed Packet Generator

8/13

¨ Munchen Technische Universitat ¨

Chair for Network Architectures and Services

Hardware Rate Control

Loadgen p9 p9 p8 p7

Qmemory

HW rate control enabled

DuT

NIC p6 p5

QNIC

NIC p4

p3

p2

Wire

I

Modern NICs support rate control in hardware

I

Limited to constant bit rate and bursty traffic

I

Precision controlled by the hardware

I

High performance as queues can be used

Paul Emmerich

p1 p0

MoonGen – A Scriptable High-Speed Packet Generator

9/13

¨ Munchen Technische Universitat ¨

Chair for Network Architectures and Services

Evaluation: 500 kpps MoonGen

Probability [%]

40 0 Pktgen-DPDK 20 0 zsend 20 0 0.5 Paul Emmerich

1

1.5 2 2.5 3 Inter-Arrival Time [µs] MoonGen – A Scriptable High-Speed Packet Generator

3.5

4 10/13

¨ Munchen Technische Universitat ¨

Chair for Network Architectures and Services

Evaluation: 1,000 kpps MoonGen

Probability [%]

20 0 Pktgen-DPDK 20 0 zsend 40 0 0.5 Paul Emmerich

1 1.5 Inter-Arrival Time [µs] MoonGen – A Scriptable High-Speed Packet Generator

2 11/13

¨ Munchen Technische Universitat ¨

Chair for Network Architectures and Services

Software Rate Control Based on Invalid Packets Loadgen p6 p6

p4i

Qmemory

p5

HW rate control disabled

DuT

NIC p3i

QNIC

NIC p4

p2i

p3 p1i p2

p0i

p1 p0

Wire

I

Fill gaps with invalid packets pi (e.g. bad CRC)

I

NIC in the DuT drops invalid packets without side-effects

I

Combines advantages of both approaches

I

Precision limited by byte rate (0.8 ns) and minimum packet size (50 Byte)

Paul Emmerich

MoonGen – A Scriptable High-Speed Packet Generator

12/13

¨ Munchen Technische Universitat ¨

Chair for Network Architectures and Services

Q&A

Try MoonGen yourself!

https://github.com/emmericp/MoonGen

Questions? Paul Emmerich

MoonGen – A Scriptable High-Speed Packet Generator

13/13

¨ Munchen Technische Universitat ¨

Chair for Network Architectures and Services

[Backup Slide] Performance I: Lua Can be Faster Than C

Packet Rate [Mpps]

I

UDP packets from varying source IP addresses 15 10 MoonGen Pktgen-DPDK

5 1.2

1.3

1.4 1.5 1.6 1.7 1.8 CPU Frequency [GHz]

1.9

2

I

Pktgen-DPDK needs a complicated main loop that covers all possibilites

I

MoonGen can use a tight inner loop

Paul Emmerich

MoonGen – A Scriptable High-Speed Packet Generator

14/13

¨ Munchen Technische Universitat ¨

Chair for Network Architectures and Services

[Backup Slide] Performance II: Heavy Workload and Multi-Core Scaling Generate random UDP packets on 2 10 GBit NICs

I

8 calls to Lua’s standard math.random per packet

I

CPUs artificially clocked down to 1.2 GHz Packet Rate [Mpps]

I

30 20 10 0 1

Paul Emmerich

5 7 2 3 4 6 Number of 1.2 GHz CPU Cores

MoonGen – A Scriptable High-Speed Packet Generator

8

15/13

¨ Munchen Technische Universitat ¨

Chair for Network Architectures and Services

[Backup Slide] HW/SW Rate Control Details

Bursts ±64 ns ±128 ns ±256 ns ±512 ns

Rate

Software

500 kpps

MoonGen 0.02% Pktgen-DPDK 0.01% zsend 28.6%

49.9% 37.7% 3.9%

74.9% 72.3% 5.4%

99.8% 92% 6.4%

99.8% 94.5% 13.8%

MoonGen 1.2% 1000 kpps Pktgen-DPDK 14.2% zsend 52%

50.5% 36.7% 4.6%

52% 58% 7.9%

97% 70.6% 24.2%

100% 95.9% 88.1%

Paul Emmerich

MoonGen – A Scriptable High-Speed Packet Generator

16/13

¨ Munchen Technische Universitat ¨

Chair for Network Architectures and Services

[Backup Slide] Effects of Bad Rate Control I

Interrupt rate of an Open vSwitch packet forwarder

Interrupt Rate [Hz]

·105 1.5

Load generated with MoonGen Load generated with zsend

1 0.5 0

0

0.5

1 1.5 Offered Load [Mpps]

2

I

Micro-bursts confuse dynamic interrupt throttling

I

This affects latency (cannot be measured with zsend)

Paul Emmerich

MoonGen – A Scriptable High-Speed Packet Generator

17/13

¨ Munchen Technische Universitat ¨

Chair for Network Architectures and Services

[Backup Slide] Effects of Invalid Packets Median latency of an Open vSwitch packet forwarder

I

Packet rate controlled by hardware vs. invalid frames Deviation [%]

I

I

Paul Emmerich

4 2 0 −2 −4

0

0.5 1 1.5 Offered Load [Mpps]

2

Minor modifications to the DuT (e.g. an active SSH session) result in a deviation of up to 15% with the same rate control mechanism MoonGen – A Scriptable High-Speed Packet Generator

18/13

¨ Munchen Technische Universitat ¨

Chair for Network Architectures and Services

[Backup Slide] Poisson Traffic CBR (Median) CBR (25th/75th Percentile) Poisson (Median) Poisson (25th/75th Percentile)

Latency [µs]

150

100

50

0

Paul Emmerich

0

0.5

1 1.5 Offered Load [Mpps]

MoonGen – A Scriptable High-Speed Packet Generator

2

19/13