Scaling NFV - Are containers the answer? Azhar Sayeed - - - PowerPoint PPT Presentation

Scaling NFV - Are containers the answer?

Azhar Sayeed - asayeed@redhat.com Doug Smith - dosmith@redhat.com

2

Acknowledgements

This is a result of mul7ple efforts in Red Hat on Containers and Container Networking. We would like thank everyone who helped us put this POC, demo and presenta7on together. A big thank you to

• Dan Williams (dcbw@redhat.com) and Rashid Khan (rkhan@redhat.com) for listening,

being pa7ent with us and for building a prototype that is really powerful

• Dan for whipping up slides and code in a maKer of 8 weeks.
• Ajay Simha (asmiha@redhat.com) for his review and contribu7ons to the presenta7on

and the work he was doing with Doug Smith to build a POC

• Tomofumi Hayashi for his work on koko (Container Connector) basis for the demo

3

Agenda

• Introduc7on
• Telco Requirements for NFV scale
• Containers - how can they help ?
• Scale ques7ons
• Do they solve the problem ?
• Issues and Challenges
• Demo
• Summary

Virtualiza7on Progression

Bare metal Virtualized Apps in - VMs Virtualized Apps in Containers Applica7ons and Network Func7ons Containers in VMs and VMs in Containers?

NFV - Use cases and scale

5

vCPE - Residen7al vCPE/SDWAN vEPC/vIMS/ VoLTE vGiLAN Mobile Wireline Consumer Business

6

NFV Use case - vCPE

Enterprise vCPE Virtualized Central Office OR Data Center Internet Residen7al vCPE Enterprise vCPE

NFVO SDN Controller VNFM(s) VIM

Residen7al NID Security & Firewall Quality of Service (QoS) Traffic Shaping Device Management

Security & Firewall Parental Control Quota Management Home Automa7on

CPE virtualiza-on is not just about cost reduc-on but providing new services to customers at the pace of innova-on and Scale

• Flexibility of IP address assignment - Public IP, Private IP, IPv4 and IPv6 etc - many VNFs

require no NAT ○ DHCP based address assignment

• Mul7ple Interface assignment - Rou7ng, Metering etc
• Mul7-Tenancy and Management of overlays
• Packet Forwarding Performance requirements - All workloads are not equal

○ NIC bonding ○ NUMA affinity - container scheduling ○ Huge Page Support ○ CPU pinning or par77oning ○ Jumbo frames support

• Hybrid VNFs (container and VMs)
• Mixed topologies containers and VMs
• Load sharing
• Elas7city - Orchestra7on

NFV requirements

Generic NFV Workload Requirements

A Mul7-dimensional problem for Telcos

8

Scale metrics and factors

• Total number of Sessions, subscribers scale
• Service Density - VMs, Apps
• Throughput scale
• Orchestra7on scale
• Number of comple7ons (Adds moves and deletes)
• Management and troubleshoo7ng scale
• Visibility and Traceability at scale
• Audit Trail of Transac7ons
• Development environments CI/CD
• Introduc7on of new func7onality

It is not just about scale but also the speed of scale

Example: vCPE For Residen7al Services

9

Scale metrics and factors

• Footprint - Subscriber density
• Typical BNG Router serves 300K IP Sessions - Half rack dedicated hardware
• Adding QoS and other bells and whistles => 150-200K IP Sessions
• Throughput per subscriber
• 10Gbps connec7ons common - but simultaneous users and subscribers average to

<0.5Gbps per subscriber

• 50K ac7ve subscribers => 25 x 100Gbps sustained throughput
• Number of VMs per server - VNF Requirements on CPU, Memory and IO
• Number of Subs per VMs
• Number of Servers
• Number of cores needed to serve that throughput using OVS+Accelera7on or VPP

etc

HOW CAN WE SCALE THIS TO EVEN HIGHER DESNITIES

• Low virtualiza7on overhead per VNF (applica7on)
• Low memory footprint
• Instant restart 7me
• Low Latency - due to a shared memory model
• Higher density per server/socket than VMs
• Encapsula7on of microservices
• Portability
• Determinis7c packaging
• Reasonable Isola7on can be accomplished easily

Why Containers?

Containers: Sonware packaging concept that include an applica7on and all its run7me dependencies

Comparing VMs and Containers

Hardware Hardware Host OS Host OS Hypervisor Docker Engine OS OS App1 App2 App2 App1 VM Stack Container Stack

Virtual Machine

VMs

• Guest OS is needed per VM
• Each Virtual Machine is isolated by the hypervisor
• Interface and hardware emulated by the Hypervisor
• Distribu7on of app 7ed to OS
• OS commonly tuned to deliver app performance

Containers

• There is no hypervisor in the container stack
• Docker Engine acts as the “hypervisor”
• Each applica7on runs as a process in user space
• Isola7on through cnames
• Considered “lightweight” compared to VMs
• Packet forwarding performance dependent on

kernel stack

• Orchestra7on via Kubernetes
• Scale - >10x

Libs Libs

VMs and Containers – Telco Evolu7on viewpoint

12

HW HOST OS HYPERVISOR

Guest OS Guest OS

Libs & Run7me Libs & Run7me

App App

HW HOST OS Libs & Run7me

App App

HW HOST OS HYPERVISOR

Guest OS Guest OS

Libs & Run7me Libs & Run7me

App App App App

VM Containers Containers in VM (Tenant Isola7on) VMs

HW HOST OS

Hypervisor

Guest OS

Libs & Run7me

App App App App

Containers & VMs

Libs & Run7me

Containers & NFV

13

• Use containers ala “VM”
• Leverage dockeriza7on of some func7ons - such as DHCP, IPAM, NAT, FW etc
• Not really separa7ng components within Network Func7ons (ala Microservices) as

the network func7ons themselves are virtualized

• Intui7ve to apply and hence assumed easy to make it happen

Containers & NFV

Applicability

• Flexibility of IP address assignment to containers - Public IP, Private IP, IPv4 and

IPv6 etc - many VNFs require no NAT

• Mul7ple Interface assignment to a container
• Mul7-Tenancy and Management of overlays
• Performance requirements - All workloads are not equal

○ NIC bonding ○ NUMA affinity - scheduling ○ Huge Page Support ○ CPU pinning or par77oning

• Hybrid VNFs (container and VMs)
• Mixed topologies containers and VMs
• Load sharing and scale

Revisi7ng the NFV requirements

With Containers - How do they fare?

? ? ? ?

• Control plane heavy VNFs

○ High session count or control traffic ○ Low data forwarding ○ Latency and availability sensi7ve for network convergence ○ Examples - signaling, subscriber policy, control protocols

• Data plane heavy VNFs

○ Require large memory alloca7on ○ Large footprint applica7ons (CPU, memory, I/O) ○ High forwarding rate requirements ○ High volume of traffic ○ Examples - PGW, ePDG, DPI etc

Containers and NFV

Telco provided defini7on Candidates for Containeriza7on

• Simple VNF - vRouter with 2 interfaces

○ Simple IGP and BGP Configura7on ○ Stock images - Vyos distribu7on ○ Memory needed to run the VM with basic alloca7ons - 387MB

• Containers

○ Same configura7on ○ Stock Container image ○ Run using Docker ○ Per container - 34MB

• vCPU alloca7ons per VM

○ Core processing for DPDK ○ 6-12 cores for VNFs like vEPC, BNG ○ 16-32GB of memory

Sizing

NFV Example Notes: Smaller configs result in smaller containers - Only 1 BGP session and an IGP results in 28MB per container 6-10X density

• Use namespaces to isolate network func7ons
• Network namespaces for containers to see their resources
• Kernel performance becomes important
• Sonware switch - like macvlan
• Assign SR-IOV to network namespace
• Using DPDK accelera7on?

Forwarding performance with containers

NFV = Line Rate Performance Requirement

• Kubernetes - Scale is Proven - Openshin

○ Today operates largest of DCs with millions of containers ○ Enterprise IT and OTT

• Scaling number of pods and nodes

○ Common to find 100 Nodes and 3000 Pods for VNF deployments

• Kolla - Ansible playbooks with Docker Containers to provide produc7on ready containers for
• penstack clouds
• NFV special requirements

○ Constraints on Kubernetes/Openshin ○ What about OAM management, Traceability, Performance, conformance, audit trail

Container Orchestra7on

Scale of Orchestra7on

OPENSHIFT – PLATFORM FOR CONTAINERS

Kubernetes based OrchestraDon Docker Container Format

Atomic Host Networking Telemetry Security Automa7on Clustering Storage

• Run Openshin/Kubernetes on Openstack
• Kuryr
• Magnum
• Run Openstack services in containers

○ Kolla

Openstack and Containers

Managing containers in Openstack Environments

22

• Containers at the remote site or central data center
• S7tched together for service chaining –
• same host –IPC
• different hosts -VLAN/VXLAN
• Port mapping architecture can be made to work here
• Will this impact NSH or dynamic SFC?

Subscriber Service Chaining – with Containers

HFC/GPON

Service Func7on Chain

Demarca7on point Cloud Boundary

VxLAN

L2 NID

AAA

Configura7on Policy

Applica7on or Content Cache Parental Control Quota Management Firewall & NATing CPE ⲙVNF VxLAN or IPSec Central Office or DC

How does SFC work with containers?

vOLT

Proof of Concept

23

Containers NFV (Needs / Requests)

• Mul7ple networks/interfaces in a container

○ Op7onal SDN for management ○ Physical NICs and SR-IOV interfaces ○ Storage networks, legacy servers

• DPDK-enabled applica7ons
• Flexible IP addressing

○ Public/administrator-defined container IP addresses ○ Overlapping IP networks & mul7-tenancy

• Flat architecture for line-rate processing and low latency

○ Reduced number of interfaces between wire and applica7on

• Determinis7c CPU and memory resources

○ Pin containerized func7ons to specific CPUs and NUMA nodes

• Coordina7ng VFs in widely separated premises
• IPv6 support, especially in public cloud
• Provide exis7ng orchestra7on and op7onal micro-service features

○ Enable new distributed applica7on architectures

vCPE Server

SFC

NIC NIC NIC Customer Metro/GPON

NID Demarc Point GPON Operator Datacenter

VF Container VF Container Management

Container NFV Proof-of-Concept Using OpenShin

VF Container

PoC demonstrates a containerized mul7-VF vCPE on a customer premises, using a vRouter and vFirewall connected via simple Service Func7on Chain, directly connected to both the customer network and the provider network.

Addi7onal NIC Addi7onal NIC

To next hop in SFC on the same node

Management SDN

NIC or SR-IOV

Container NFV PoC: Inside the Container

veth SFC endpoint SDN endpoint

Administrator defined IP address Administrator defined IP address OpDonal SDN provided IP address and micro- services

Virtual FuncDon (vRouter, vFirewall, etc)

Addi7onal NIC

DPDK (op7onal) Kernel Networking

• Sta7c IPv6 support
• Simple Service Func7on Chains
• Flexible, administrator defined IP addressing
• No bridging or SDN in packet fast-paths
• Mul7ple interfaces per container (NIC, VLAN, SDN)
• Container VF CPU affinity

Features:

Container NFV PoC: Next Steps

• Determine feasibility of overlapping IP networks
• Inves7gate addi7onal features and requirements:

○ NUMA affinity ○ IPv6 SLAAC addressing ○ Dynamic Service Func7on Chains ○ Network Service Headers ○ Service support for addi7onal interfaces

• Implement support for SR-IOV interfaces
• Gather more requirements and use-cases
• Work with upstream Kubernetes community to standardize these features

Demo Doug Smith & Ajay Simha

28

Host A - OpenStack

192.168.2.100

quagga_a

ID: 2.2.2.2 192.168.2.101 192.168.3.100 192.168.4.100 192.168.3.101 192.168.4.101

centos_a

ID:1.1.1.1

centos_b

ID: 4.4.4.4

quagga_b

ID: 3.3.3.3

in1 in2 mid1

• ut1

mid2

• ut2

Host B - OpenShin on AWS

vxlan vxlan

Legend

Host / Guest Docker Container Network path Network Interface

WAN

Containerized Router Demo - over Hybrid Cloud

OSPF StaDc StaDc

30

Suppor7ng Technologies

KoKo - Network namespace u7lity Available on GitHub Ratchet - CNI Plugin Available on GitHub

31

KoKo func7onality

veth/vxlan veth/vxlan

KoKo

• Expose System resources to containers - such as PCI devices, NUMA Nodes, Kernel Modules,

○ Higher security risk - poten7ally larger aKack surface when compared to VMs ○ Kernel op7miza7ons become cri7cal for forwarding performance - no offload techniques ○ Inter-container communica7on uses IPC instead of Ethernet/IP => Advantage and a disadvantage

• Achieving mul7-tenancy is considered harder wrt containers than VMs

○ Namespaces and cgroups

• Adding OAM capabili7es for network func7ons in containers increases the size of the micro-

service - Newer OAM architectures needed to define services for mul7ple containers versus replica7ng for each containers

• May Require some service re-architec7ng
• Interface limita7ons (OS issues )

Challenges with using Containers for VNFs

• High scale needed for different use cases

○ IoT, vCPE Residen7al etc ○ Millions of sessions map to thousands of servers

• Containers can provide much higher scale than VMs (>10x)

○ Smaller footprint - when compared to VM on OSP

• VNFs may need to be re-wriKen to take advantage of Container/Micro-services models
• Kubernetes scales well for massive applica7on farms => adopt to NFV
• Container networking a subject of more work and interest in the industry for NFV

○ Kuryr provides a networking model with Openstack

• Na7ve networking for containers using Kubernetes/Openshin

○ POC and Code available for tes7ng - Goal to commit upstream as part of Kubernetes SDN enhancements

• For NFV with Containers - s7ll more work needed

○ Dynamic Service Chains, NUMA Affinity etc

Summary

THANK YOU

INSERT DESIGNATOR, IF NEEDED 35

Abstract

Containers are the biggest hype today - In the latest Heavy Reading Survey (October 2016), 68.4% of the responders said they will use containers for NFV. While containers provide high scale, low latency and a low startup 7me, however, no one really understands the complete impact of containers on how it changes the virtualiza7on model for NFV and what impact it has on the networking and orchestra7on model for NFV. Containers are well designed for scale out applica7ons but for containers to work with NFV we need the ability to assign public IP addresses to

• containers. That is not so easy as it sounds.

In this presenta7on, we will discuss the NFV architecture with containers in detail. In par7cular we will discuss topics like Kuryr (Containers and Openstack), Container networking, Container instan7a7on with Openstack, Scale, Performance (Latency and Throughput) and data path accelera7on for containers. What can I expect to learn?

• Container and how they can be used for NFV
• Limita7ons of containers
• Orchestra7on NFV with VMs and Containers using Openstack