Distributed Systems CS6421 Networking: SDN and NFV Prof. Tim Wood - - PowerPoint PPT Presentation

▶
distributed systems cs6421
SMART_READER_LITE
LIVE PREVIEW

Distributed Systems CS6421 Networking: SDN and NFV Prof. Tim Wood - - PowerPoint PPT Presentation

Sep 11, 2022 138 likes •903 views

Distributed Systems CS6421 Networking: SDN and NFV Prof. Tim Wood SDN + NFV Networks are changing - Trying to achieve the same level of customization, flexibility, and automation found in the cloud Software-based Networks - SDN: Software

slide-1
SLIDE 1

Distributed Systems CS6421

Networking: SDN and NFV

  • Prof. Tim Wood
slide-2
SLIDE 2

Tim Wood - The George Washington University - Department of Computer Science

SDN + NFV

Networks are changing

  • Trying to achieve the same level of customization, flexibility, and

automation found in the cloud

Software-based Networks

  • SDN: Software Defined Networking - control plane
  • NFV: Network Function Virtualization - data plane

2

slide-3
SLIDE 3

Software Defined Networks: Overview

Adapted from slides by

  • K. K. Ramakrishnan, UC Riverside

(with thanks to many people’s material that he re-used: David Koll, Univ. of Goettingen, Germany, Jennifer Rexford, Princeton, Nick Mckeown, Stanford and others).

3

slide-4
SLIDE 4

Cloud Scalability

“The average cloud environment might have 50 dedicated servers to one admin, and what you really need to get to is 500 servers to one admin, or what happened in the case of Microsoft, 10,000 servers. Without automation we don't have speed and scale - the very reason we want to go to the cloud.” (Microsoft)

  • Virtualization and automation software helped

solve these problems for the cloud…

  • What about the network?

4

slide-5
SLIDE 5

Network Scalability?

“Even simple topologies take days or weeks to create. Workload placement and mobility are restricted by physical network limitations and hardware dependencies require vendor-specific expertise. Network configuration is performed manually and maintenance is both expensive and resource- intensive.” (VMWare)

  • Mainly a manual processes: have to manually

configure each device with physical presence!

5

slide-6
SLIDE 6

Switch: Match on Destination MAC

  • MAC addresses are location independent

– Assigned by the vendor of the interface card – Cannot be aggregated across hosts in LAN

6

mac1 mac2 mac3 mac4 mac5

host host host ...

mac1 mac2 mac3

switch host host

mac5 mac4

slide-7
SLIDE 7

Router: Match on IP Prefix

  • IP addresses grouped into common subnets

– Allocated by ICANN, regional registries, ISPs, and within individual organizations – Variable-length prefix identified by a mask length

7

host host host LAN 1 ... host host host LAN 2 ... router router router WAN WAN

1.2.3.4 1.2.3.7 1.2.3.156 5.6.7.8 5.6.7.9 5.6.7.212 1.2.3.0/24 5.6.7.0/24

forwarding table Prefixes may be nested. Routers identify the longest matching prefix.

slide-8
SLIDE 8

Forwarding vs. Routing

  • Forwarding: data plane

– Directing a data packet to an outgoing link – Individual router using a forwarding table

  • Routing: control plane

– Computing paths the packets will follow – Routers talking amongst themselves – Individual router creating a forwarding table

8

slide-9
SLIDE 9

Example: Shortest-Path Routing

  • Compute: path costs to all nodes

– From a source u to all other nodes – Cost of the path through each link – Next hop along least-cost path to s

9

3 2 2 1 1 4 1 4 5 3

u s 6

v (u,v) w (u,w) x (u,w) y (u,v) z (u,v) link s (u,w) t (u,w)

v w y x t z

slide-10
SLIDE 10

Distributed Control Plane

  • Link-state routing: OSPF

, IS-IS

– Flood the entire topology to all nodes – Each node computes shortest paths – Dijkstra’s algorithm

10 18

v (u,v) w (u,w) x (u,w) y (u,v) z (u,v) link s (u,w) t (u,w)

3 2 2 1 1 4 1 4 5 3

u v w x y z s t

slide-11
SLIDE 11

Tim Wood - The George Washington University

Flexibility Problem

  • All packets arriving at a switch/router are treated the

same

  • Only consider the destination IP/MAC address to decide path
  • Prevents customizing the network for different cloud

tenants!

3 2 2 1 1 4 1 4 2 3

u v w x y z s t

VM-a VM-b VM-c VM-d

Green VMs are paying more for higher bandwidth networking so we would like to support different paths!

slide-12
SLIDE 12

Traffic Engineering Problem

  • Management plane: setting the weights

– Inversely proportional to link capacity? – Proportional to propagation delay? – Network-wide optimization based on traffic?

12

3 2 2 1 1 3 1 4 5 3 3

slide-13
SLIDE 13

Traffic Engineering: Optimization

  • Inputs

– Network topology – Link capacities – Traffic matrix

  • Output

– Link weights

  • Objective

– Minimize max-utilized link – Or, minimize a sum of link congestion

13

3 2 2 1 1 3 1 4 5 3

slide-14
SLIDE 14

Transient Routing Disruptions

  • Topology changes

– Link weight change – Node/link failure or recovery

  • Routing convergence

– Nodes temporarily disagree how to route – Leading to transient loops and blackholes

14

1 4 5 3 1 4 10 3 1 4 10 3

slide-15
SLIDE 15

Management Plane Challenges

  • Indirect control

– Changing weights instead of paths – Complex optimization problem

  • Uncoordinated control

– Cannot control which router updates first

  • Interacting protocols and mechanisms

– Routing and forwarding – Naming and addressing – Access control – Quality of service – …

15

slide-16
SLIDE 16

Software Defined Networking

  • Solution: Software-Defined-Networking (SDN)

– Decouples the control plane from the data plane

Images taken from materials of the Open Networking Foundation: https:// www.opennetworking.org/

16

slide-17
SLIDE 17

Software Defined Networking

  • SDN makes the network programmable
  • OSPF, DiffServ, IntServ, MPLS, RSVP?

– All such protocols can be done in software, controlled by a central instance – Scalable, easily manageable, better interoperability

17

slide-18
SLIDE 18

SDN Components at a glance

18

slide-19
SLIDE 19

SDN Components at a glance

  • Programmable Open APIs:

– Connects applications with control plane – Allows for programming of routing, QoS, etc.

  • Standard Communication Interface (e.g., OpenFlow):

– Between control and data planes – Allows direct access to forwarding plane

  • Network Controller (logically centralized):

–Sets up rules, actions, etc. for the network devices –Core element of SDN

19

slide-20
SLIDE 20

SDN Benefits

  • SDN further allows for…

– elastic resource allocation (e.g., to match QoS agreements) – distribution of the load on links (e.g., between backbone and application servers in SaaS) – scalability (no need to manually configure each of thousands (or even millions?) of devices) – overhead reduction – …and more

20

slide-21
SLIDE 21

OpenFlow – The SDN Protocol

  • Communication between the controller and the

network devices (i.e., switches)

Specification by the Open Networking Foundation: https://www.opennetworking.org/ images/stories/downloads/sdn-resources/onf-specifications/openflow/openflow-spec- v1.3.4.pdf (March 2014)

21

slide-22
SLIDE 22

OpenFlow Protocol

Data Path (Hardware) Control Path OpenFlow

Ethernet Switch

Network OS

Control Program A Control Program B

OpenFlow Basics

22

slide-23
SLIDE 23

Control Program A Control Program B

Network OS

OpenFlow Basics

Packet Forwarding Packet Forwarding Packet Forwarding Flow Table(s) “If header = p, send to port 4” “If header = ?, send to me” “If header = q, overwrite header with r, 
 add header s, and send to ports 5,6”

23

slide-24
SLIDE 24

Plumbing Primitives


<Match, Action>

Match arbitrary bits in headers:

– Match on any header field, but not data – Allows ‘any’ flow granularity

Action

– Forward to port(s), drop, send to controller – Overwrite header with mask, push or pop – Forward at specific bit-rate

24

Header Data Match: 1000x01xx0101001x

slide-25
SLIDE 25

OpenFlow – Switches

  • Incoming packets are matched against rule tables
  • Find highest priority match and execute actions
  • Send out port
  • Forward to another table
  • Drop
  • Rate limit
  • etc…

25

slide-26
SLIDE 26

OpenFlow – Switches

  • If no match in table: table miss
  • Handling: depends on table configuration –

might be drop packet, forward to other table, forward to controller

  • Forward to controller allows to set up a flow

entry (i.e., at the beginning of a flow)

  • Based on a program!

26

slide-27
SLIDE 27

Table Miss

  • What can the controller do if there is a miss?
  • What happens to subsequent packets?
  • Why only send to controller on miss?
  • Why not every packet?

27

slide-28
SLIDE 28

Examples

Switching - can customize based on known MAC addresses

* Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport Action * 00:1f:.. * * * * * * * port6

Flow Switching - fine grained switching for each TCP connection

port3 Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport Action 00:20.. 00:1f.. 0800 vlan1 1.2.3.4 5.6.7.8 4 17264 80 port6

Firewall - not just switching, but also dropping/rate limiting/etc

* Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport Action * * * * * * * * 22 drop

28

slide-29
SLIDE 29

OpenFlow - Example

29

slide-30
SLIDE 30

OpenFlow - Example

SRC: H2 DST: H4

30

slide-31
SLIDE 31

OpenFlow - Example

SRC: H2 DST: H4 ?

31

slide-32
SLIDE 32

OpenFlow - Example

SRC: H2 DST: H4

Packet-IN

32

slide-33
SLIDE 33

OpenFlow - Example

SRC: H2 DST: H4

Packet-OUT Action: eth2

33

slide-34
SLIDE 34

OpenFlow - Example

SRC: H2 DST: H4

34

slide-35
SLIDE 35

OpenFlow - Example

SRC: H2 DST: H4

35

slide-36
SLIDE 36

OpenFlow - Example

SRC: H2 DST: H4?

36

slide-37
SLIDE 37

OpenFlow - Example

SRC: H2 DST: H4!

37

slide-38
SLIDE 38

OpenFlow - Example

SRC: H2 DST: H4

38

slide-39
SLIDE 39

Tim Wood - The George Washington University - Department of Computer Science

SDN Workflow

Data plane (switches): maintains a flow table

  • Flow = one point-to-point connection (Src/Dest IP and Port)
  • Action = how switch should process the packet

Control plane: defines flow table rules for switches

  • Can be based on business logic
  • Select next hop, drop, mirror, etc.

39

Client 1 Host 1 SDN Switch SDN Controller SDN Switch SDN Switch

slide-40
SLIDE 40

Tim Wood - The George Washington University - Department of Computer Science

Network Function Virtualization

Make an efficient, customizable data plane

  • routers, switches, firewalls,

proxies, IDS, DPI, etc

Run network functions (NFs) in virtual machines

  • More flexible than hardware
  • Isolates functionality, easy to

deploy and manage

  • Slower than hardware…

40

Router Firewall

Virtualization Layer

Commodity Server

Router Switch

slide-41
SLIDE 41

Tim Wood - The George Washington University - Department of Computer Science

Network Data Plane

Perform network functionality on custom ASICs

Fast, expensive, inflexible

41

slide-42
SLIDE 42

Tim Wood - The George Washington University - Department of Computer Science

Software-Based Data Plane

Hardware Routers and Switches

  • Expensive, single purpose
  • Controllable with SDNs, but not flexible

PacketShader [Han, SIGCOMM ’10]

  • Use commodity servers and GPUs
  • 39 Gbps processing rates

Netmap [Rizzo, ATC ’12] and DPDK

  • Libraries to provide zero-copy network


processing on commodity 10gbps NICs

ClickOS [Martins, NSDI ’14] and NetVM [Hwang, NSDI ’14]

  • VM based network services
  • Flexible deployment and composition

42

slide-43
SLIDE 43

Tim Wood - The George Washington University - Department of Computer Science

Network Functions (NFs)

Switches, routers, firewalls, NAT

  • Simple packet header analysis and forwarding

Intrusion Detection Systems (IDS)

  • Deep packet inspection (DPI) beyond header to detect threats
  • Must have high scalability to observe full packet flows

Intrusion Prevention Systems (IPS)

  • Similar to IDS, but deployed in-line, so it can actively manipulate

traffic flows

  • Must be efficient to avoid adding delay

Cellular functions (Evolved Packet Core - EPC)

  • Mobility management, accounting, security, etc.

Proxies, caches, load balancers, etc.

43

AKA “middleboxes"

slide-44
SLIDE 44

Tim Wood - The George Washington University - Department of Computer Science

Linux Packet Processing

Traditional networking:

  • NIC uses DMA to copy data into kernel buffer
  • Interrupt when packets arrive
  • Copy packet data from kernel space to user space
  • Use system call to send data from user space

44

Linux User Applications H/W Platform Packet copy Interrupt Handling Systemcalls

Can you handle being interrupted 60 million times per second?

slide-45
SLIDE 45

Tim Wood - The George Washington University - Department of Computer Science

User Space Packet Processing

Recent NICs and OS support allow user space apps to directly access packet data

  • NIC uses DMA to copy data into kernel user space buffer
  • Interrupt use polling to find when packets arrive
  • Copy packet data from kernel space to user space
  • Use system regular function call to send data from user space

45

slide-46
SLIDE 46

Tim Wood - The George Washington University - Department of Computer Science

Data Plane Development Kit

High performance I/O library Poll mode driver reads packets from NIC Packets bypass the OS and are copied directly into user space memory Low level library... does not provide:

  • Support for multiple network functions
  • SDN-based control
  • Interrupt-driven NFs
  • State management
  • TCP stack

46

slide-47
SLIDE 47

Tim Wood - The George Washington University - Department of Computer Science

Data Plane Development Kit

Where to find it:

  • http://dpdk.org/

What to use it for:

  • Applications that need high speed access to low-level packet

data

Why try it:

  • One of the best documented open source projects I've ever seen

Alternatives:

  • netmap
  • PF_RING

47

slide-48
SLIDE 48

Network Platforms Group

Packet Size 64 bytes 40G Packets/second 59.5 Million each way Packet arrival rate 16.8 ns 2 GHz Clock cycles

33 cycles

Typical Server Packet Sizes Network Infrastructure Packet Sizes

Packet Size (B)

Packets per second

15,000,000 30,000,000 45,000,000 60,000,000 64 224 384 544 704 864 1024 1184 13441504

What is “line rate”?

Packet Size 1024 bytes 40G Packets/second 4.8 Million each way Packet arrival rate 208.8 ns 2 GHz Clock cycles

417 cycles

40 Gbps Line Rate (or 4x10G) Process Packet

Tx

Rx

48

slide-49
SLIDE 49

Network Platforms Group

How to Eliminate / Hide Overheads?

49

Interrupt Context Switch Overhead Kernel User Overhead

Core To Thread Scheduling Overhead

Polling User Mode Driver Pthread Affinity

4K Paging Overhead

PCI Bridge I/O Overhead

Huge Pages Lockless Inter-core Communication High Throughput Bulk Mode I/O calls

slide-50
SLIDE 50

Tim Wood - The George Washington University - Department of Computer Science

Network Interrupts

Very distracting! Have to stop doing useful work to handle incoming packets Coalescing interrupts helps, but still causes problems

  • Interrupts can arrive during critical sections!
  • Interrupts can be delivered to the wrong CPU core!
  • Still must pay context switch cost

50

App Kernel App Kernel App App Kernel

Interrupt Context Switch Overhead

slide-51
SLIDE 51

Tim Wood - The George Washington University - Department of Computer Science

Polling

Continuously loop looking for new packet arrivals Trade-off?

51

Interrupt Context Switch Overhead

App Kernel App Kernel App

Interrupts help share the CPU

App App

Polling can be wasteful

Busy wait

slide-52
SLIDE 52

Tim Wood - The George Washington University - Department of Computer Science

Kernel-User Overhead

NIC Driver operates in kernel mode

  • Reads packets into kernel

memory

  • Stack pulls data out of

packets

  • Data is copied into user

space for application

  • Application uses system

calls to interface with OS

Why is copying so bad?

52

Kernel
 Space

Driver

User
 Space

NIC

Applications

Stack

System Calls

CSRs

Interrupts

Memory (RAM)

Packet Data

Copy

Kernel Space Driver

Configuration Descriptors

DMA

Descriptor
 Rings Socket Buffers (skb’s)

1 2 3

Kernel User Overhead

From Intel DPDK University Lecture

slide-53
SLIDE 53

Tim Wood - The George Washington University - Department of Computer Science

Kernel Bypass

User-mode Driver

  • Kernel only sets up basic

access to NIC

  • User-space driver tells NIC

to DMA data directly into user-space memory

  • No extra copies
  • No in-kernel processing
  • No context switching

53

Kernel User Overhead Kernel
 Space

UIO Driver

User
 Space

NIC

DPDK PMD

Stack

System Calls

CSRs

Memory (RAM)

Packet Data

User Space Driver

Configuration

Descriptors

DMA

Descriptor
 Rings

1 2

From Intel DPDK University Lecture DPDK Application

Descriptors

slide-54
SLIDE 54

Tim Wood - The George Washington University - Department of Computer Science

Linux Kernel Map

54

slide-55
SLIDE 55

Tim Wood - The George Washington University - Department of Computer Science

Networking

Linux networking
 stack has a lot of
 extra components For NFV middlebox
 we don’t use all of this:

  • TCP

, UDP , sockets

NFV middle boxes just need packet data

  • Need it fast!

55

Application

Kernel User Overhead

slide-56
SLIDE 56

Tim Wood - The George Washington University - Department of Computer Science

CPU Core Affinity

Linux Scheduler can move threads between cores

  • Context switches :(
  • Cache locality :(

56

Core To Thread Scheduling Overhead

App App

Core 1

App App

Core 2

App

slide-57
SLIDE 57

Tim Wood - The George Washington University - Department of Computer Science

CPU Core Affinity

Pin threads and dedicate cores

  • Trade-offs?

57

Core To Thread Scheduling Overhead

App

Core 1

App

Core 3 Core 2

App

slide-58
SLIDE 58

Tim Wood - The George Washington University - Department of Computer Science

Trade-off?

Paging Overhead

4KB Pages

  • 4 packets per page
  • 14 million pps
  • 3.6 million page 


table entries every
 second

  • Sandy Bridge: 64 entries

1GB Huge Pages

  • 1 million packets/page
  • 14 page table entries


every second!

58

4K Paging Overhead

https://courses.cs.washington.edu/courses/cse378/00au/CSE378-00.Lec28/sld004.htm

  • Packet ~= 1KB

How big is the TLB?

slide-59
SLIDE 59

Tim Wood - The George Washington University - Department of Computer Science

Locks

Thread synchronization is expensive

  • Tens of nanoseconds to take an uncontested lock
  • 10Gbps -> 68ns per packet

Producer/Consumer architecture

  • Gather packets from NIC (producer) and ask worker to process

them (consumer)

Lock-free communication

  • Ring-buffer based message queues

59

slide-60
SLIDE 60

Tim Wood - The George Washington University - Department of Computer Science

Bulk Operations

PCIe bus uses messaging protocols for CPU to interact with devices (NICs) Each message incurs some overhead Better to make larger bulk requests over PCIe DPDK helps batch requests into bulk operations

  • Retrieve a batch (32) of packet descriptors received by NIC
  • Enqueue/dequeue beaches of packet descriptors onto rings

Trade-offs?

60

PCI Bridge I/O Overhead

slide-61
SLIDE 61

Tim Wood - The George Washington University - Department of Computer Science

Limitations

DPDK provides efficient I/O… but that’s about it Doesn’t help with NF management or orchestration

61

slide-62
SLIDE 62

Tim Wood - The George Washington University - Department of Computer Science

Service Chains

Chain together functionality to build more complex services

  • Need to move packets through chain efficiently

62

Firewall NAT Router

Server

slide-63
SLIDE 63

Tim Wood - The George Washington University - Department of Computer Science

Service Chains

Chain together functionality to build more complex services

  • Need to move packets through chain efficiently

63

Firewall IDS Router NAT Cache Transcoder DPI

Mirror

  • Can be complex with multiple paths!
slide-64
SLIDE 64

Tim Wood - The George Washington University

OpenNetVM NFV Platform

  • DPDK: provides underlying I/O engine
  • NFs: run inside Docker container, use NFlib API
  • Manager: tracks which NFs are active, organizes chains
  • Shared memory: efficient communication between NFs


SDN-aware: Controller can dictate flow rules for NFs

  • http://sdnfv.github.io/

64

Container 1

Shared Memory

(packets, flow tables, service chains, ring buffers)

Packet

NF Manager (DPDK)

R T

NFlib

NF RX

Container 2

R T

NF

NFlib 3rd party

library

Container 3

R T

NF

NFlib Container 4

R T

NF

NFlib

custom distro

NIC 1 NIC 2 TX1 TX2

FT

Packet

Mgr

R T

User Space

M a d e a t G W !

slide-65
SLIDE 65

Tim Wood - The George Washington University - Department of Computer Science

Limitations

DPDK only helps 
 with raw packet IO Doesn’t provide any 
 protocol stacks!

  • No IP
  • No TCP or UPD
  • No socket interface

65

slide-66
SLIDE 66

Tim Wood - The George Washington University - Department of Computer Science

TCP in Linux

Linux TCP stack is not designed for high performance

  • Especially for short flows
  • Poor scalability, bad locality, etc
  • Same problems we saw with DPDK

66

  • –

–

0.0 0.5 1.0 1.5 2.0 2.5 1 2 4 6 8 Connections/sec (x 105) Number of CPU Cores

TCP Connection Setup Performance

Linux: 3.10.16 Intel Xeon E5-2690 Intel 10Gbps NIC Performance meltdown

Figures from Jeong’s mTCP talk at NSDI 14 83% of CPU usage spent inside kernel!

  • Kernel

(without TCP/IP) 45% Packet I/O 4% TCP/IP 34% Application 17%

Web server (Lighttpd) Serving a 64 byte file Linux-3.10

slide-67
SLIDE 67

Tim Wood - The George Washington University - Department of Computer Science

mTCP [Jeong, NSDI ’14]

User space TCP stack

  • Built on DPDK/netmap (and now OpenNetVM!)

Key Ideas:

  • Eliminate shared resources by partitioning flows to independent

threads

  • Use batching to minimize overheads
  • Epoll interface to support existing end-point applications

67

3 6 9 12 15 2 4 6 8 Transactions/sec (x 105) Number of CPU Cores Linux REUSEPORT MegaPipe mTCP

1

  • *[OSDI’12]

Shared fd in process Shared listen socket

* [OSDI’12] MegaPipe: A New Programming Interface for Scalable Network I/O, Berkeley

Inefficient small packet processing in Kernel

3 6 9 12 15 3 6 9 12 15 3 6 9 12 15 3 6 9 12 15

Linux: 3.10.12 Intel Xeon E5-2690 32GB RAM Intel 10Gbps NIC

Figure from Jeong’s mTCP talk at NSDI 14

slide-68
SLIDE 68

Tim Wood - The George Washington University - Department of Computer Science

mTCP Kernel Bypass

Responding to a packet arrival only incurs a context switch, not a full system call

68

Packet I/O Kernel TCP Application thread BSD socket LInux epoll User-level packet I/O library mTCP thread Application Thread NIC device driver mTCP socket mTCP epoll Kernel User Linux TCP mTCP

System call Context switching

Figure from Jeong’s mTCP talk at NSDI 14

slide-69
SLIDE 69

Tim Wood - The George Washington University - Department of Computer Science

69

Performance Improvement on Ported Applications

Web Server (Lighttpd)

  • Real traffic workload: Static file

workload from SpecWeb2009 set

  • 3.2x faster than Linux
  • 1.5x faster than MegaPipe

SSL Proxy (SSLShader)

  • Performance Bottleneck in TCP
  • Cipher suite

1024-bit RSA, 128-bit AES, HMAC- SHA1

  • Download 1-byte object via HTTPS
  • 18% ~ 33% better on SSL handshake

20 1.24 1.79 2.69 4.02 1 2 3 4 5

Linux REUSEPORT MegaPipe mTCP

Throughput (Gbps)

26,762 28,208 27,725 31,710 36,505 37,739 5 10 15 20 25 30 35 40 4K 8K 16K Transactions/sec (x 103) # Concurrent Flows

Linux mTCP

Slide from Jeong’s mTCP talk, NSDI ‘14

slide-70
SLIDE 70

Tim Wood - The George Washington University - Department of Computer Science

70

47% 63% 67% 0% 25% 50% 75% Web security gateway Mail security gateway Web application firewall

Virtual Appliances Deployed in Service Provider Data Centers

Most Middleboxes Deal with TCP Traffic

  • TCP dominates the Internet
  • 95+% of traffic is TCP
  • Top 3 middleboxes in service providers rely on L4/L7 semantics

2 [1]

TCP UDP etc

[1] “Comparison of Caching Strategies in Modern Cellular Backhaul Networks”, ACM MobiSys 2013.

95.7%

[2] IHS Infonetics Cloud & Data Center Security Strategies & Vendor Leadership: Global Service Provider Survey, Dec. 2014.

[2]

Slide from Jamshed’s mOS talk, NSDI ‘17

slide-71
SLIDE 71

Tim Wood - The George Washington University - Department of Computer Science

mOS [Jamshed, NSDI ‘17]

What if your middle box (not end point server) needs TCP processing? Proxies, L4/L7 load balancers, DPI, IDS, etc

  • TCP state transitions
  • Byte stream reconstruction

71

  • 50K~100K code lines tightly coupled

with their IDS logic

Borrow code from open-source IDS (e.g., snort, suricata)

  • Designed for TCP end host
  • Different from middlebox semantics

Borrow code from open-source kernel (e.g., Linux/FreeBSD)

  • Complex and error-prone
  • Repeat it for every custom middlebox

Implement your own flow management code

Table from Jamshed’s mOS talk, NSDI ‘17

slide-72
SLIDE 72

Tim Wood - The George Washington University - Department of Computer Science

mOS [Jamshed, NSDI ‘17]

Reusable protocol stack for middle boxes Key Idea: Allow customizable processing based on flow-level “events” Separately track client and server side state

72

mOS stack emulation

TCP server Server side TCP stack

TCP state SYN

SYN/ACK

LISTEN CLOSED SYN_SENT

Client side TCP stack

TCP state SYN_RCVD ESTABLISHED

DATA/ACK

Receive buffer ESTABLISHED

TCP client

Figure from Jamshed’s mOS talk at NSDI 17

slide-73
SLIDE 73

Tim Wood - The George Washington University - Department of Computer Science

mOS [Jamshed, NSDI ‘17]

Base Events

  • TCP connection start/end, packet arrival, retransmission, etc

User Events

  • Base event + a filter function (executable code) run in mOS stack

73

2104 765 615 0K 10K 20K 30K 40K 50K 60K 70K 80K 90K Snort3 nDPI PRADS Lines Modified Total Lines

1.4 1.2 4.1 3.2 5.0 4.5 16.7 11.6 22.8 21.7 53.0 42.5 10 20 30 40 50 60 1 4 16 1 4 16 Throughput (Gbps) (# of CPU cores) Counting packets Searching for a string 64B file 8KB file

1 4 16 1 4 16

Figures from Jamshed’s mOS talk at NSDI 17

slide-74
SLIDE 74

Tim Wood - The George Washington University - Department of Computer Science

TCP + OpenNetVM

We have ported mOS/mTCP to run on OpenNetVM Allows deployment of mixed NFs and endpoints Allows several different mTCP endpoints on same host

74

NF Manager mTCP NF NF lighttpd mOS Proxy

M a d e a t G W !

slide-75
SLIDE 75

Tim Wood - The George Washington University - Department of Computer Science

TCP + OpenNetVM

Mixed NFs + endpoints blurs the line of the application and the network

  • NF services could expose APIs to work with endpoints

75

NF Manager mTCP NF NF lighttpd mOS Proxy

M a d e a t G W !

slide-76
SLIDE 76

Tim Wood - The George Washington University - Department of Computer Science

Microboxes

76

Guyue Liu – George Washington University

Conclusion

üConsolidate Stack Processing üCustomizable Stack Modules üUnified Event Interface

Microboxes = µStack + µEvent

= stack snapshot + parallel stacks + parallel events + event hierarchy + publish/subscribe interface

  • Redundant Stack Processing
  • A Monolithic Stack
  • Separate Stacks/Interfaces

NF NF NF NF NF

µStack µStack µStack µStack µStack µStack µStack µEvent µEvent µEvent µEvent

M a d e a t G W !