Comparison of Network Interface Controllers for Software Packet - - PowerPoint PPT Presentation

▶
comparison of network interface controllers for software
SMART_READER_LITE
LIVE PREVIEW

Comparison of Network Interface Controllers for Software Packet - - PowerPoint PPT Presentation

Aug 20, 2022 307 likes •586 views

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Comparison of Network Interface Controllers for Software Packet Processing (Final Talk) Alexander Frank Advisors: Paul Emmerich, Sebastian

slide-1
SLIDE 1

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Comparison of Network Interface Controllers for Software Packet Processing (Final Talk)

Alexander Frank

Advisors: Paul Emmerich, Sebastian Gallenmüller, Dominik Scholz Supervisor: Prof. Dr.-Ing. Georg Carle October 2, 2017 Chair of Network Architectures and Services Department of Informatics Technical University of Munich

slide-2
SLIDE 2

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Contents

Motivation Hardware Comparison Software Comparison Future Work Bibliography

  • A. Frank

– Network Controller Comparison 2

slide-3
SLIDE 3

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Motivation

  • A. Frank

– Network Controller Comparison 3

slide-4
SLIDE 4

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Motivation - General

Used software:

  • MoonGen - High-speed packet generator based on Lua scripts
  • libmoon - Lua wrapper for DPDK
  • DPDK (Data Plane Development Kit) - Provides drivers and libraries

for fast packet processing Hardware support:

  • DPDK supports NICs from Intel, Mellanox, Broadcom, Chelsio, ...
  • MoonGen/libmoon (technically) support all hardware supported by

DPDK

  • MoonGen/libmoon only tested against Intel hardware
  • A. Frank

– Network Controller Comparison 4

slide-5
SLIDE 5

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Motivation - Introduction to MoonGen/libmoon

DPDK NIC NIC MoonGen libmoon script MoonGen script

config API config API

HW Software  DPDK libraries  Drivers  Lua Wrapper for DPDK  Custom Drivers  Rate Control  Timestamping libmoon

Figure 1: MoonGen/libmoon software stack

  • A. Frank

– Network Controller Comparison 5

slide-6
SLIDE 6

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Motivation - Goals of the Thesis

Programming: Integrate Mellanox NICs into libmoon/MoonGen

  • Change build system to automatically handle Mellanox drivers and

dependencies

  • Enable hardware filtering
  • Uniform packet counting and statistics

Research: Compare three selected NICs in terms of features and ca- pabilities

  • Hardware level - interfaces, offloads, timestamping
  • Software level - integration with DPDK, filter support
  • A. Frank

– Network Controller Comparison 6

slide-7
SLIDE 7

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Hardware Comparison

  • A. Frank

– Network Controller Comparison 7

slide-8
SLIDE 8

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Tested NICs

Vendor NIC driver Intel Ethernet Controller X550T ixgbe Intel Ethernet Controller XL710 i40e Mellanox ConnectX-4 Lx mlx5

Table 1: Investigated Intel and Mellanox NICs

All adapters shown here are compatible with Ethernet For testing we used 10 Gb/s versions of the NICs

  • A. Frank

– Network Controller Comparison 8

slide-9
SLIDE 9

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Hardware Interfaces

All three NICs provide similar types of interfaces Relevant for throughput: network interface and PCIe interface NIC Series Ports Port Speed PCIe Interface X550 1/2 10 Gb/s v3.0 x4 @ 31.504 Gb/s 710 1/2/4* 40 Gb/s v3.0 x8 @ 63.016 Gb/s ConnectX-4 1/2 100 Gb/s v3.0 x16 @ 126.032 Gb/s

Table 2: Investigated Intel and Mellanox NIC series. Port speed refers to usage of a single port [3], [6], [4]

*XL710 provides 4 MAC interfaces for 2 physical ports. Can be utilized with breakout cable

  • A. Frank

– Network Controller Comparison 9

slide-10
SLIDE 10

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Timestamping

Timestamping is used by MoonGen for high precision latency mea- surements Drawbacks of software timestamping:

  • Inaccurate as software does not know when exactly a packet leaves

the port

  • Might in itself influence the timing characteristics

→ Hardware timestamping desirable

  • A. Frank

– Network Controller Comparison 10

slide-11
SLIDE 11

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Timestamping - Intel

  • X550 - 80 MHz clock with precision ± 12.5 ns
  • 710 - 625 MHz clock with precision ± 0.8 ns (Values depend on

configured link speed, 40 Gb/s in this case)

1 1 1 1 1 1 1 1 bit time Ethernet Start of Frame Delimiter First Octet Following Start of Frame Preamble Octet Message Timestamp Point Figure 2: Timestamping with X550 and 710 series NICs [6]

  • A. Frank

– Network Controller Comparison 11

slide-12
SLIDE 12

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Timestamping - Mellanox

Software SQ

(Send Work Queue)

RQ

(Recieve Work Queue)

  • 1. Post WQEs

(Work Queue Entries)

Hardware WQ

  • 2. Pass WQE ownership

to hardware

  • 3. Access/Execute

WQEs CQ

(completion queue)

  • 4. Post

CQE

  • 5. Read CQE

(Completion Queue Entry)

CQE contains timestamp Figure 3: Timestamps with ConnectX-4 based devices [4]

  • A. Frank

– Network Controller Comparison 12

slide-13
SLIDE 13

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Timestamping - Summary

Point in time to generate timestamp:

  • Intel’s NICs timestamp is generated when a certain bit leaves the

port

  • Mellanox’ NICs include a timestamp of CQE generation
  • When is a CQE exactly generated? Deterministic?

Clock frequencies of Mellanox cards must be queried from hardware and are not documented

  • A. Frank

– Network Controller Comparison 13

slide-14
SLIDE 14

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Software Comparison

  • A. Frank

– Network Controller Comparison 14

slide-15
SLIDE 15

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

DPDK

DPDK libraries mlx5 PMD Kernel Verbs User Verbs (libibverbs+libmlx5) ConnectX-4/5 mlx5 verbs provider (mlx5_ib) mlx5_core PRM Data Path MoonGen/libmoon

User Kernel Hardware

Control Path DPDK libraries I40E PMD/IXGBE Driver XL710/X550T igb_uio

  • r

uio_pci_generic Data Path MoonGen/libmoon

User Kernel Hardware

Control Path

Figure 4: DPDK and driver relation, Mellanox left, Intel right [2], [5]

  • A. Frank

– Network Controller Comparison 15

slide-16
SLIDE 16

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

DPDK with Intel

  • DPDK

was

  • riginally

developed by Intel

  • Usage of UIO modules

→ driver completely in userspace

  • Registers of hardware

are directly mapped into userspace

DPDK libraries IXGBE Driver X550T igb_uio

  • r

uio_pci_generic Data Path MoonGen/libmoon

User Kernel Hardware

Control Path

Figure 5: DPDK with ixgbe [2], [5]

  • A. Frank

– Network Controller Comparison 16

slide-17
SLIDE 17

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

DPDK with Mellanox 1

Mellanox uses the InfiniBand software stack. Why? Mellanox Background:

  • Portfolio is centered around InfiniBand capable hardware
  • Adapter types: VPI (InfiniBand and Ethernet), EN (Ethernet only)
  • We use EN types. Software stack is the same for VPI and EN

InfiniBand Background:

  • InfiniBand provides very low latencies and is mainly used for HPC
  • Verbs are abstract definitions of NIC operations defined in the In-

finiBand standard

  • Verbs are implemented by third party verbs providers
  • A. Frank

– Network Controller Comparison 17

slide-18
SLIDE 18

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

DPDK with Mellanox 2

  • mlx5 PMD is the userspace

driver and interfaces with the verbs layer

  • mlx5_core is the kernel level

driver

  • Kernel Verbs and the

mlx5 verbs provider implement the InfiniBand verbs

  • This setup allows to receive

and send packets via the OS’ network stack on ports which are not used by DPDK

DPDK libraries mlx5 PMD Kernel Verbs User Verbs (libibverbs+libmlx5) ConnectX-4/5 mlx5 verbs provider (mlx5_ib) mlx5_core PRM Data Path MoonGen/libmoon

User Kernel Hardware

Control Path

Figure 6: DPDK with mlx5 [2], [5]

  • A. Frank

– Network Controller Comparison 18

slide-19
SLIDE 19

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Benefits of InfiniBand - RDMA

Buffer Application Buffer Transport Protocol Driver Buffer NIC Driver Buffer Sockets Buffer Application Buffer Transport Protocol Driver Buffer NIC Driver Buffer Sockets Buffer NIC Buffer NIC Buffer Application NIC Driver Sockets Buffer Application NIC Driver Sockets Buffer RNIC Buffer RNIC

Traditional Interconnect RDMA Zero-Copy Interconnect

Figure 7: Comparison of data flow with and without RDMA [1]

  • A. Frank

– Network Controller Comparison 19

slide-20
SLIDE 20

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

DPDK support

Timestamping:

  • The ixgbe and i40e drivers implement timestamping
  • The mlx5 driver does currently not support timestamping
  • Patchwork DPDK is checking a patch to solve this issue

HW-Filtering:

  • Intel devices support DPDK’s old filtering framework
  • Mellanox supports this framework only partially
  • DPDK 17 introduces new framework with better support by Mel-

lanox

  • A. Frank

– Network Controller Comparison 20

slide-21
SLIDE 21

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

MoonGen/libmoon Improvements

Task Status Build process for mlx5 and mlx4 done Update to DPDK 17.05 done Integrate DPDK’s generic flow API done Statistics and uniform packet counting done Mellanox installation instructions done Update to DPDK 17.08 done temporary fix

  • A commit in DPDK 17.08 introduced a bug which prevents our

ConnectX-4 Lx from receiving packets

  • Temporary workaround is to disable the faulty feature (vectorized

burst)

  • A. Frank

– Network Controller Comparison 21

slide-22
SLIDE 22

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Future Work

  • A. Frank

– Network Controller Comparison 22

slide-23
SLIDE 23

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Future Work

Extend support of MoonGen:

  • Add support for mlx4 devices (ConnectX-3 Pro) or NICs from other

vendors (e.g. Terminator 5 from Chelsio)

  • Faster NICs (> 10 Gb/s)
  • Observe development of timestamping capabilities (Mellanox)

Compare the NICs’ architecture more in-depth, e.g., the different work flows of the NICs

  • A. Frank

– Network Controller Comparison 23

slide-24
SLIDE 24

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Bibliography

  • A. Frank

– Network Controller Comparison 24

slide-25
SLIDE 25

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

[1] InfiniBand FAQ. http://www.mellanox.com/related-docs/whitepapers/InfiniBandFAQ_FQ_100. pdf, 2014. Revision 1.3, accessed on 09/29/2017. [2] Mellanox DPDK. https://community.mellanox.com/docs/DOC-1502, 2014. Updated on 07/26/2017, accessed on 09/29/2017. [3] Intel Ethernet Controller X550 Datasheet. http://www.intel.com/content/www/us/en/embedded/products/networking/ ethernet-x550-datasheet.html, 2016. Revision 2.2, accessed on 09/29/2017.

  • A. Frank

– Network Controller Comparison 25

slide-26
SLIDE 26

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

[4] Mellanox Adapters Programmer’s Reference Manual (PRM), Supporting ConnectX-4 and ConnectX-4 Lx. http://www.mellanox.com/related-docs/user_manuals/Ethernet_Adapters_ Programming_Manual.pdf, 2016. Revision 0.40, accessed on 09/29/2017. [5] DPDK, Network Interface Controller Drivers. http://dpdk.org/doc/guides/nics/index.html, 2017. Release 17.11.0, accessed on 09/29/2017. [6] Intel Ethernet Controller 710 Series Datasheet. http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/ xl710-10-40-controller-datasheet.pdf, 2017. Revision 3.2, accessed on 09/29/2017.

  • A. Frank

– Network Controller Comparison 26