Modifying existing applications for 100 Gigabit Ethernet Jelte - - PowerPoint PPT Presentation

Modifying existing applications for 100 Gigabit Ethernet

Jelte Fennema

University of Amsterdam

29th June 2016

Introduction Approach Results Conclusion

Introduction

◮ 100 Gigabit Ethernet (100GbE) is becoming common ◮ Measuring the network speed is important ◮ iperf3 is unable to saturate a 100GbE link

◮ Can only reach ~45Gbit/s ◮ CPU core is being maxed out

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 2/25

Introduction Approach Results Conclusion

DPDK as a possible solution

◮ The Linux networking stack is too slow ◮ Possible solution: Data Plane Development Kit (DPDK)

◮ Developed by Intel for very fast network I/O ◮ Includes special high performance drivers

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 3/25

Introduction Approach Results Conclusion

Linux networking

Send/receive data Linux kernel Application Network card System calls and interupts network drivers and

Figure 1: Normal Linux networking

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 4/25

Introduction Approach Results Conclusion

DPDK networking

Send/receive data Application, Network card network drivers DPDK and

Figure 2: DPDK networking

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 5/25

Introduction Approach Results Conclusion

Current DPDK packet generators

◮ Moongen

◮ Achieved 120Gbit/s over multiple 10GbE interfaces ◮ Doesn’t support our Network Interface Card (NIC)

◮ Pktgen

◮ Developed by Intel as official DPDK application

◮ Both have not been tested on 100GbE NICs

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 6/25

Introduction Approach Results Conclusion

Research questions

• 1. Can current DPDK packet generators saturate a 100GbE link?
• 2. What is necessary to modify iperf3 to use DPDK?
• 3. What throughput improvements can be be achieved by

modifying iperf3 to use DPDK?

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 7/25

Introduction Approach Results Conclusion

Setup

Switch Server 1 100GbE Server 2 100GbE

Figure 3: The test setup

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 8/25

Introduction Approach Results Conclusion

Accelerated Network Stack

◮ iperf3 uses regular TCP connections ◮ DPDK itself can only be used for sending raw packets

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 9/25

Introduction Approach Results Conclusion

Accelerated Network Stack

◮ iperf3 uses regular TCP connections ◮ DPDK itself can only be used for sending raw packets ◮ ANS is a FreeBSD networking stack modified for DPDK

◮ Contains support for popular network protocols

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 9/25

Introduction Approach Results Conclusion

New iperf3 versions

Two new iperf3 versions are created:

◮ One modified to use ANS ◮ A Linux version with comparable modifications

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 10/25

Introduction Approach Results Conclusion

Focus

◮ TCP ◮ Single stream

◮ Single core

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 11/25

Introduction Approach Results Conclusion

Performance settings

◮ Setting CPU affinity ◮ isolcpus ◮ Disable hyperthreading

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 12/25

Introduction Approach Results Conclusion

DPDK baseline

◮ Pktgen could reach 86Gbit/s ◮ This is for raw packets

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 13/25

Introduction Approach Results Conclusion

iperf3 multi process baseline

1 2 3 4

Number of iperf3 processes

40 50 60 70 80 90 100

Total Gbit/s

Figure 4: iperf3 speedtest with multiple processes

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 14/25

Introduction Approach Results Conclusion

Modifications iperf3

◮ Event loop conversion from select to epoll style ◮ Removal of synchronous network I/O

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 15/25

Introduction Approach Results Conclusion

Modifications to iperf3

Three iperf3 versions:

• 1. Regular
• 2. Epoll
• 3. ANS

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 16/25

Introduction Approach Results Conclusion

Initial performance tests

Regular Epoll ANS

iperf3 version

10 20 30 40 50 60

Gbit/s

Figure 5: Initial performance comparison

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 17/25

Introduction Approach Results Conclusion

Missing performance features

◮ TCP window scaling ◮ Jumbo frames are broken ◮ Offloading to the NIC

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 18/25

Introduction Approach Results Conclusion

Performance with more streams

1 2 3 4 5

Number of streams

10 20 30 40 50 60

Total Gbit/s

iperf3 version

Regular Epoll ANS Figure 6: Performance with multiple TCP streams

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 19/25

Introduction Approach Results Conclusion

Full impact of missing features

1 2 3 4 5

Number of streams

10 20 30 40 50 60

Total Gbit/s

iperf3 version

Regular Epoll ANS Figure 7: Performance comparison without missing ANS features

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 20/25

Introduction Approach Results Conclusion

Final weird result

◮ Multiple streams improve single stream performance 1 2 3 4 5

Number of streams

10 20 30 40 50 60

Total Gbit/s

iperf3 version

Regular Epoll ANS

Figure 8: Performance comparison without missing ANS features

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 21/25

Introduction Approach Results Conclusion

Modified transmit buffer length

Regular Epoll ANS New ANS

iperf3 version

2 4 6 8 10 12 14

Gbit/s

Figure 9: A single TCP stream with performance features disabled

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 22/25

Introduction Approach Results Conclusion

Modified transmit buffer length

1 2 3 4 5

Number of streams

10 20 30 40 50 60

Total Gbit/s

iperf3 version

Regular Epoll ANS New ANS Figure 10: Multiple TCP streams with performance features disabled

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 23/25

Introduction Approach Results Conclusion

Conclusion

◮ Pktgen was not able to fully fill the 100GbE link

◮ But it was much faster than iperf3

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 24/25

Introduction Approach Results Conclusion

Conclusion

◮ Pktgen was not able to fully fill the 100GbE link

◮ But it was much faster than iperf3

◮ Modifying existing applications for DPDK is relatively easy by

using ANS

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 24/25

Introduction Approach Results Conclusion

Conclusion

◮ Pktgen was not able to fully fill the 100GbE link

◮ But it was much faster than iperf3

◮ Modifying existing applications for DPDK is relatively easy by

using ANS

◮ iperf3 speeds with ANS are currently slower than with Linux

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 24/25

Introduction Approach Results Conclusion

Conclusion

◮ Pktgen was not able to fully fill the 100GbE link

◮ But it was much faster than iperf3

◮ Modifying existing applications for DPDK is relatively easy by

using ANS

◮ iperf3 speeds with ANS are currently slower than with Linux ◮ When missing ANS features are disabled for Linux ANS is faster

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 24/25

Introduction Approach Results Conclusion

Conclusion

◮ Pktgen was not able to fully fill the 100GbE link

◮ But it was much faster than iperf3

◮ Modifying existing applications for DPDK is relatively easy by

using ANS

◮ iperf3 speeds with ANS are currently slower than with Linux ◮ When missing ANS features are disabled for Linux ANS is faster ◮ For multiple streams using multiple cores is probably easier

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 24/25

Introduction Approach Results Conclusion

Future work

◮ Compare iperf3 performance after features have been

implemented in ANS

◮ Investigate performance of Moongen on 100GbE

Jelte Fennema

Modifying existing applications for 100 Gigabit Ethernet 25/25