Enab abling DP DPDK DK/SR-IOV f for r November 2017 2017 - - PowerPoint PPT Presentation

Enab abling DP DPDK DK/SR-IOV f for r contai ainer erized zed V Virtual al Net Network Func unctions w ns with h Zun Zun

Bin Zhou [NFV Researcher, Lenovo] Hongbin Lu [Zun PTL,Huawei] Yaguang Tang [NFV Researcher, Lenovo] Shunli Zhou [Zun Core, Fiberhome]

November 2017 2017

➡Introduction to Zun ➡Zun Container for NFV

• Challenges & Gaps
• SR-IOV support in Zun
• Container with DPDK

➡Performance Benchmark Testing

• Setup
• Results

➡Demo ➡Conclusion

Age genda da

Which E Emerging Technologies I Interest st Open enStac ack Us User ers?

• Containers are the

most interesting emerging technologies.

• 75% of OpenStack

users interests in containers.

➡How to use containers on OpenStack? ➡Existing solutions

• Integrate containers into Nova
• Example: Nova-docker, Nova-lxd
• Install Container Orchestration Engine (COEs) on VMs.
• Example: Magnum, Kubespray
• OpenStack Container service: Zun

Introduce Zun Zun

• OpenStack Container service
• Provide API for provisioning and

managing containers without VMs ○ Speed ○ Simplicity

• Arbitrary memory and vCPUs
• Containers as first class resource

○ Keystone RBAC for individual container ○ Neutron port(s) for each container ○ Cinder volume(s) bind-mount

Introduce Zun Zun

VMs Containers Create List Delete Run Exec ... SSH Migrate ... Nova Zun

➡Nova-docker

• Use Nova to manage

containers

• Suitable if VMs and

containers are the same ➡Obstacles

• VMs and containers are

different

• Container specified features

are not exposed

Introduce Zun Zun

Baremetal Tenant 1 Virtualization Tenant 2 Tenant 3 COE Baremetal Tenant 1 Virtualization (optional) Tenant 2 Tenant 3 Contain ers Zun COE COE Contain ers Contain ers Contain ers Contain ers Contain ers Magnum Zun

➡Magnum

• Provision Nova instances
• Install a COE
• Run containers on the

COE ➡Pros:

• Strong Isolation

➡Cons:

• Low resource utilization
• Virtualization penalty

Introduce Zun Zun

➡Concepts:

• Contai

ainer er: A single container

• create, update, delete, start, stop, kill, …
• network-attach, add-security-group, …
• attach, exec, commit, log, ...
• Cap

apsule e (Experimental): A group of containers that are co- located, have shared network and volumes.

• create, list, delete, …

Introduce Zun Zun

Introduce Zun Zun

➡Zun API

• Provide REST APIs
• Manage all compute nodes
• Scheduling containers

➡Zun Compute

• Compute node agent
• Manage local containers
• Track compute resources

➡Kuryr

• Bind neutron ports to

containers

Zun API Zun Compute Docker Keystone Kuryr Neutron Cinder

➡Introduction to Zun ➡Zun Container for NFV

• Challenges & Gaps
• SR-IOV support in Zun
• Container with DPDK

➡Performance Benchmark Testing

• Setup
• Results

➡Demo ➡Conclusion

Age genda da

➡What is NFV

• A new way to design, deploy and manage network services
• Replace hardware with software
• Move network functions to commodity hardware

➡Benefits of NFV

• Fast provisioning
• Quick scale up and down
• Easy upgrade and relocate
• Reduce cost
• No vendor hardware locked-in

Co Contai ainer f for NF NFV

➡VM or Containers?

• Time to provision: container boots faster
• Resource consumption: container has less memory footprint
• Package management: Docker makes it easy
• Configurability: container is better
• Portability: container image is smaller
• Security: VM provides better isolation
• Use Clear Container to improve security

Co Contai ainer f for NF NFV

Challe lleng nges & Gaps of usin ing containe iners

NFV Req features VM Container SR-IOV Yes Weak DPDK Yes Weak CPU pinning Yes Weak NUMA Yes Weak Hugepage Yes Weak

➡Lack of supports of NFV required features in container ecosystem

• Container runtime
• Container orchestration
• OpenStack integration

➡Use Zun to reduce the gaps

Enab able S e SR-IOV i in Zu Zun

➡What is SR-IOV?

• A standardized mechanism to virtualize PCIe devices
• Make a single PCIe Ethernet controller (PF) to appear as multiple PCIe

devices (VF)

• PF: Physical Function
• VF: Virtual Function
• Passthrough VF to container
• Bypass virtual switch layer

Enab able S e SR-IOV i in Zu Zun

➡Enable SR-IOV in Zun

• Create VFs in compute nodes
• Configure Neutron
• Configure Zun
• Whitelist PCI devices (e.g. pci_passthrough_whitelist = { "devname":

"eth3", "physical_network": "physnet2"})

• Enable PCI filters (e.g. enabled_filters = ...,PciPassthroughFilter)
• Configure Kuryr
• Enable SR-IOV driver

Enab able S e SR-IOV i in Zu Zun

1.Create a SR-IOV port 2.Create a container 3.Pick a host that has available VFs 4.Assign a VF to the port 5.Create a container 6.Docker calls its network plugin (Kuryr) to setup the network 7.Kuryr retrieve VF’s information from the neutron port and perform port binding

Zun API Zun Compute Kuryr Neutron Docker User 1 2 3 5 6 7 4

➡Introduction to Zun ➡Zun Container for NFV

• Challenges & Gaps
• SR-IOV support in Zun
• Container with DPDK

➡Performance Benchmark Testing

• Setup
• Results

➡Demo ➡Conclusion

Age genda da

Co Contai ainer w with DP DPDK DK

DPDK PMD

• physical nic

○ igb_uio ○ vfio-pci

• virtual hardware

○ virtio_user vhost software

• net_pcap (kernel stack)

Host kernel Container Container VF VF PF PF driver

Host kernel

Container DPDK DPDK DPDK

DP DPDK DK & & S SR-IOV for c container er

SR-IOV in userland

SR-IOV in kernel

VF VF

VF driver VF driver

Container

netns ETHx netns ETHx Passthrough

Case 1 (non DPDK)

• Zun Container with SR-IOV
• Zun Container with OVS

networking

Per erforman ance Ben enchmark T Tes esting

Case 2

• Container with SR-IOV &

DPDK (kernel land)

• Container with SR-IOV &

DPDK (user land)

Role Hardware OS network CPU

Controller Think system x3650 M5 Ubuntu 16.04.3 82599ES 10Gb Intel(R) E5- 2680 v3 @ 2.50GHz compute Think system x3650 M5 Ubuntu 16.04.3 82599ES 10Gb Intel(R) E5- 2680 v3 @ 2.50GHz

Software version

• ther

DPDK 17.05 Openvswitch 2.8.1

Tes esting s set etup

• L2FWD as containerized

VNF

• RFC 2544 standard

throughput testing

• DPDK-pktgen as packet

generator DPDK Testing non DPDK Testing

• iperf3 with udp

zun-compute Server1 zun-compute Server2 O V S O V S container container container container Linux bridge Linux bridge

PF PF

Zu Zun networking w witho hout SR-IOV OV

zun-compute Server1 zun-compute Server2 container container container container VF VF VF VF

Zu Zun n networking with S SR-IOV OV

Container n network B Benchmarking

• Hugepage size
• PCIe NUMA
• Isolate CPU cores for tx/rx pktgen
• Disable isolated cpu core interrupts

BOOT_IMAGE=/vmlinuz-4.4.0-87-generic root=/dev/mapper/docker2--vg-root ro default_hugepagesz=1G hugepagesz=2M hugepagesz=1G hugepages=8 iommu=pt intel_iommu=on isolcpus=5,6,7,8,9,10 nohz=on nohz_full=5,6,7,8,9,10 rcu_nocbs=5,6,7,8,9,10

DPDK testing ng tuning

Server1 Server2 VF1 VF2 pktgen VNF l2fwd VF1 VF2 VF1

Te Testing scenar ario 1 1

• Userland SR-IOV used by container
• DPDK application l2fwd inside container

Container

dpdk-devbind --bind=igb_uio 0000:06:10.2 docker run -v /dev/hugepages/:/dev/hug epages --net=none -- privileged --name test2

• dit 14ce48b74dd9

l2fwd -l 5-6 -n 4 --huge-dir /dev/hugepages --socket- mem 1024,1024 -- -q 8 -p 1

Server1 Server2 VF1 VF2 pktgen VNF l2fwd VF1 VF2 VF1

Te Testing scenar ario 2 2

• containers using SR-IOV by kernel netns
• DPDK application l2fwd inside container

NETNS

Container $ neutron port-create sriov -- name sriov_port -- binding:vnic_type direct $ zun run --net port=sriov_port dpdk-test l2fwd -l 5-6 -n 4 --huge-dir /dev/hugepages --socket-mem 1024,1024 -- vdev=’eth_pcap0,iface=eth0’ --

• q 8 -p 1

Container DPDK Benchmarking

ht https://yout utu. u.be/EwghP hPOVZLq Lq0

Dem Demo

SR-IOV & DPDK can accelerate container networking performance Benefits High throughput Low latency Deterministic networking

Conclusi sion

• DPDK & SR-IOV for container user land approaching physical

server performance

• multi-tenancy issue
• security issue
• Container with SR-IOV for

high throughput non DPDK application

• unified management of VF

@ O @ OpenStack

Q&A

Thank you!

• pe
• penstack
• pe
• penstack

OpenStac ackFoundat ation