Network virtualisation using Crossbow Technology Uro Nedi, MSc - - PowerPoint PPT Presentation

Network virtualisation using Crossbow Technology

Uroš Nedić, MSc

OpenSolaris Contributor

2

• Crossbow Technical Background
• Network Machines
• Network in a Box

3

Crossbow Features

• Virtualization

> IP Instances > NIC Virtualization - VNICs > Service Virtualization - Flows

• Resource partitioning

> QoS/Diffserv (without performance penalties) > SLA on a per connection basis > Class of Service Support

• Performance & Observability

> Dynamic polling > H/W and S/W fanout to multiple cores > Real Time Usage and History

Better Defense against DDOS attacks

4

Virtual Stacks

Services and Protocols

Compute Resources

NIC 1 CPU 1

VIRTUAL SQUEUE

CPU 2

VIRTUAL SQUEUE

CPU 'n'

VIRTUAL SQUEUE

CPU 1 Virtual Squeue CPU 2 Virtual Squeue

VOIP

SQUEUE

HTTPS

SQUEUE

DEFAULT

SQUEUE

TCP

SQUEUE

UDP

SQUEUE

DEFAULT

SQUEUE

Virtual NIC Virtual NIC Virtual NIC Virtual NIC Virtual NIC Virtual NIC Memory Partition Memory Partition Memory Partition Memory Partition Memory Partition Memory Partition Flow Classifier Flow Classifier NIC 2

5

Virtual Stacks

Services and Protocols

CPU 1

VIRTUAL SQUEUE

CPU 2

VIRTUAL SQUEUE

CPU 'n'

VIRTUAL SQUEUE

CPU 1 Virtual Squeue

VOIP

SQUEUE

HTTPS

SQUEUE

DEFAULT

SQUEUE

Virtual NIC Virtual NIC Virtual NIC The VNICs are in the control path only. The data link layer is bypassed.

The Squeue switches the MSI interrupt per stack between interrupt and polling mode and controls the rate of packet arrival for the virtual stack.

Memory Partition Memory Partition Memory Partition Memory Partition Memory Partition Memory Partition Flow Classifier Flow Classifier NIC 1 NIC 2

Compute Resources

6

Crossbow Architecture

• Partition the NIC Hardware, kernel queues, and CPU

and allow creation of Virtual NICs

• Use dynamic polling on Virtual NICs to schedule rate of

packet arrival per VNIC

• Effect of dynamic polling

Mpstat (older driver)

intr ithr csw icsw migr smtx srw syscl usr sys wt idl 10818 8607 4558 1547 161 1797 289 19112 17 69 0 12

Mpstat (Crossbow based driver)

intr ithr csw icsw migr smtx srw syscl usr sys wt idl 2823 1489 875 151 93 261 1 19825 15 57 0 27

• Use Dynamic polling for B/W partitioning and isolation

without any support from switches and routers

7

Parallelized Stack: Made for Cores/Threads

Multi-Core Processor Crossbow Virtualized Network Streams & VNICs 1000’s of Logical Connections Application Threads Or Solaris Containers Or Virtual Machines C L A S S I F I E R VNIC

Rx/Tx DMA

Kernel Qs and threads

VNIC

Rx/Tx DMA

Kernel Qs and threads

Squeue Squeue Neptune 2 x 10 Gbe and/or 4 x 1Gbe

Application NIC Network Kernel CPU

Dedicated Lane VNIC

Rx/Tx DMA

Kernel Qs and threads

Squeue

8

Virtual Network Containers

Flow Classifier

Exclusive IP Instance

Rx/Tx

DMA

Rx/Tx

DMA

Rx/Tx

DMA

NIC bge0

VNIC1 (100Mbps) VNIC2 (200Mbps)

Exclusive IP Instance

Virtual

SQUEUE

Virtual

SQUEUE

Zone

xb1-z1

Zone

xb1-z2

Client

xb2

Client

xb3

Solaris Global Zone Virtualization

• Exclusive IP Stack
• Virtual NICs
• Virtual Switches

Resource Control

• Bandwidth Limits & Gauarantees
• Priority
• Dedicated CPUs

Observability

• Watch real time usage for each

VNIC

• Snmp and Kstat per VNIC
• History at no cost

9

Virtual Machines

Solaris Guest OS 1 Solaris Guest OS 2 Solaris Host OS Host OS

VIRTUAL SQUEUE All Traffic NIC Virtualization Engine NIC Virtualization Engine NIC Virtualization Engine

Guest OS 1

VIRTUAL SQUEUE

Guest OS 2

VIRTUAL SQUEUE All Traffic Host OS VNIC Guest OS 2 VNIC

NIC

H/W Flow Classifier

HTTP SQUEUE HTTPS SQUEUE

DEFAULT SQUEUE

Virtual NIC Virtual NIC Virtual NIC

Host OS All traffic Guest OS 1 HTTP Guest OS 1 HTTPS Guest OS 1 DEFAULT Guest OS 2 All Traffic

10

Network Machines

DMZ Apache

Solaris N2/NIU

Router/Firewall DNS Server

IP IP TCP/ UDP IP Virtual NIC A Virtual NIC A Virtual NIC B Virtual NIC B TCP/ UDP TCP/ UDP Rx/Tx

DMA

Rx/Tx

DMA

Rx/Tx

DMA

Rx/Tx

DMA

Rx/Tx

DMA

Rx/Tx

DMA

Flow Classifier Flow Classifier

NIC A NIC B

Internet Intranet

APIs for ISVs at each layer Dedicated CPUs

11

Anatomy of a Typical Router

Major cost paid on bringing the packet in and route lookup etc. Packets are dropped on Xmit Queues

PORT 0 PORT 1 PORT 2 PORT n Route Lookup Engine Xmit Q 1 Xmit Q 2 Xmit Q 3 Xmit Q m Xmit Port 0 Xmit Port 1 Xmit Port 2 Xmit Port m

12

Anatomy of Crossbow Router

OpenSolaris Network Stack Network Interface Compute Interface

Policy-based packet drop on ingress—zero cost drop.

• Explicit mapping of Compute Threads or Cores to Network Threads creating virtual

“express lanes”

• Dedicated BW, Priority, and CPU/Threads assigned to each lane
• High performance packet classification

DMA 0 DMA 1 DMA 2 DMA 16 VNIC VNIC VNIC VNIC HTTPS Data Voice Video T/C T/C T/C T/C Packet Classifier

1

Crossbow Network in a Box

Switch 3

Router

Port3 10.0.0.3 Port2 10.0.0.2 Port1 10.0.0.1 Port6 20.0.0.3 Port9 20.0.0.1

Switch 1

Host 2 Host 1

Client 100Mbps

Physical Network

1 Gbps 1 Gbps 1 Gbps 1 Gbps

1

Example VNIC Usage

• Done using dladm(1M), as with other data-link

interface administration

# dladm create-vnic -d bge1 vnic1 # dladm create-vnic -d bge1 -m random -p maxbw=100M -p cpus=4,5,6 vnic2 # dladm create-etherstub vswitch1 # dladm show-etherstub LINK vswitch1 # dladm create-vnic -d vswitch1 -p maxbw=1000M vnic3 # dladm show-vnic LINK OVER MACTYPE MACVALUE BANDWIDTH CPUS vnic1 bge1 factory 0:1:2:3:4:5 - - vnic2 bge1 random 2:5:6:7:8:9 max=100M 4,5,6 vnic3 vswitch1 random 4:3:4:7:0:1 max=1000M

1

Join Us. ...

• Our communities and projects are open on

OpenSolaris.org:

> CrossBow: http://opensolaris.org/os/project/crossbow > VNM: http://opensolaris.org/os/project/vnm > Networking:

http://opensolaris.org/os/community/networking

• Where you will find:

> Lively discussions, design docs, FAQs, source code

drops, binary releases, etc...