NFV Testing Jose Lausuch (Ericsson) Trevor Cooper (Intel) Agenda - - PowerPoint PPT Presentation

▶
nfv testing
SMART_READER_LITE
LIVE PREVIEW

NFV Testing Jose Lausuch (Ericsson) Trevor Cooper (Intel) Agenda - - PowerPoint PPT Presentation

Jan 07, 2024 253 likes •553 views

NFV Testing Jose Lausuch (Ericsson) Trevor Cooper (Intel) Agenda NFV Testing with Open Source Projects Role of CI Performance Testing Intro to OPNFV OPNFV testing and integration project Additions Enhancements OPNFV testing and

slide-1
SLIDE 1

NFV Testing

Jose Lausuch (Ericsson) Trevor Cooper (Intel)

slide-2
SLIDE 2

Agenda

NFV Testing with Open Source Projects Role of CI Performance Testing

slide-3
SLIDE 3

Intro to OPNFV

slide-4
SLIDE 4
slide-5
SLIDE 5

Enhancements Additions

OPNFV – testing and integration project

slide-6
SLIDE 6

PLATFORM SDN COMPONENTS FEATURES SFC

BPGPVN

IPv6

Hugepages Fault management

OPNFV – testing and integration project

slide-7
SLIDE 7

OPNFV – testing and integration project

Infrastructure Compute Storage Network

Virtualization Cloud Management

NFV apps NFV orchestration OSS/BSS

slide-8
SLIDE 8

NFV Testing with Open Source Projects

slide-9
SLIDE 9

Test Ecosystem in OPNFV

slide-10
SLIDE 10

Types of testing

Functional Performance Stress

Security

Infrastructure Features Components VNF MANO … Network Storage Compute Virtual layer Traffic gen. … Network Storage Compute …

Upgradability Backup&Restore

. . .

slide-11
SLIDE 11

The concept of scenarios

scenario X scenario Y scenario Z SFC

slide-12
SLIDE 12

The role of CI

Continuous Integration

slide-13
SLIDE 13

Infrastructure – Distributed Pharos Labs

4/3/2017 LF Infra Collab

slide-14
SLIDE 14

Pharos Architecture

slide-15
SLIDE 15

Role of CI

Jumphost Initiate Run for certain Scenario Download ISO Deploy Functest Yardstick Pull Down Functest Image Pull Down Yardstick Image Labs connected 24x7 using JNLP Google Cloud Storage Docker Hub Docker Hub

4/3/2017 LF Infra Collab Pharos POD (SUT)

slide-16
SLIDE 16

CI workflow

yes

new changes?

no Clone repo Poll changes

Build Build Build

Build Server Upload ISO Push Image Google Cloud Storage Docker Hub Patchset Verification Verified +1/-1 Patch

slide-17
SLIDE 17

Working with upstream communities

contribute contribute contribute contribute

pull from master, deploy, test and/or verify patchset, post feedback

slide-18
SLIDE 18

NFV Performance Testing

1. Meanings 2. Approaches 3. Tools 4. Examples 5. Ambitions

“When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind.” … Lord Kelvin

slide-19
SLIDE 19

Meanings of NFV Performance Testing

How do vendor A & B compare?

  • Can the platform deliver required networking performance?
  • Evaluating switching, acceleration technology, hardware platform, NFVI, VNF, full solution?

What Tools and Methods are needed?

  • We have relied on trusted proprietary tools
  • Traditional network performance methodologies are useful but not sufficient

NFV => general purpose compute elements in a heterogeneous compute environment

  • Benchmarking virtual network functions is different to physical functions
  • System configurations are complex and few deployments are the same (test infrastructures also differ)

Performance Testing Tools and Methods are not just for Developers and Test-Experts

Performance Testing Infrastructure Test Frameworks Test Dashboards Test Databases Traffic Profiles Test Methodologies KPIs / Metrics Traffic Generators

slide-20
SLIDE 20

Approaches to NFV Performance Testing

System Under Test, Workloads and Stimuli

  • VNF / NFVI
  • Pre-deployment / In-service
  • Topology / SW versions / configurations
  • Data Plane vs Control Plane + Data Plane
  • Component / sub-system / platform / network service
  • Traffic profiles, Workloads
  • Deployment automation vs control

Test Objectives

  • How Fast? How much Latency? How much Capacity?
  • How many Sessions?
  • How Quick is Recovery?

Test Methods and Metrics

  • Throughput, back2back, frame-loss
  • Packet and Frame Delay Distribution
  • Scalability according to number of flows, active ports, etc.
  • Stream Type with realistic profiles / flows: L2, L3, L4
  • Control and Data Path Coupling
  • CPU and Memory Consumption
  • “Soak” tests capture transient changes in performance

COVERAGE SPEED ACCURACY RELIABILITY SCALABILITY Activation Operation De-activation

IETF NFV Benchmarking Internet Drafts Data Centre Benchmarking Methodology

draft-ietf-bmwg-dcbench-methodology-02

Data Centre Benchmarking Terminology

draft-ietf-bmwg-dcbench-terminology-05

Benchmarking Methodology for SDN Controller Performance

draft-ietf-bmwg-sdn-controller-benchmark-meth-01

Terminology for Benchmarking SDN Controller Performance

draft-ietf-bmwg-sdn-controller-benchmark-term-01

Benchmarking VNFs and Their Infrastructure

draft-ietf-bmwg-virtual-net-02

Considerations for Benchmarking High Availability of NFVI

draft-kim-bmwg-ha-nfvi-01

Benchmarking Methodology for EVPN

draft-kishjac-bmwg-evpntest-00

VNF Benchmarking Methodology

draft-rosa-bmwg-vnfbench-00

Benchmarking Virtual Switches in OPNFV

draft-vsperf-bmwg-vswitch-opnfv-02

ETSI NFV Test Specifications NFV Pre-deployment Testing (Spec)

http://www.etsi.org/technologies-clusters/technologies/nfv

NFV Interoperability Testing Methodology (Spec) Use cases & recommendations for VNF Snapshot (Spec) NFV Issue Tracker – to report feedback on NFV DRAFTs

http://nfvwiki.etsi.org/index.php?title=NFV_Issue_Tracker

Path-implementations among NS Endpoints & VNFs

https://docbox.etsi.org/isg/nfv/open/drafts/

Capture of VNF data and VNF/VNFC snapshots DevOps and CI/CD - test procedures and metrics Interop test guidelines VNF, MANO, VIM-NFVI Key performance NFVI metrics at different layers

slide-21
SLIDE 21

NFV Performance Testing Tools

Test Infrastructure

  • https://wiki.opnfv.org/display/pharos
  • https://wiki.opnfv.org/display/pharos/Community+Labs
  • Video Tutorial: Introduction to Pharos

Test Frameworks

  • Network Service Performance (MANO)
  • VNF Characterization (Control + Data Plane)
  • NFVI Performance (Data Plane)
  • Components and Sub-system Performance
  • Staging Manager and Bottleneck Analysis
  • Analytics (post-process test results from CI/CD pipeline)
  • Integrated statistics / events monitoring

Traffic Generators

  • Vary in measurement capability and suitability
  • Vary in complexity and cost

Traffic Generator IXIA (physical and virtual) Proprietary SW/HW Spirent (physical and virtual) Proprietary SW/HW Xena Proprietary SW/HW Moongen Open Source SW TREX Open Source SW Pktgen Open Source SW LAN Tornado RFC 2544 Proprietary SW Seagull Open Source SW PROX Open Source SW

slide-22
SLIDE 22

NFV Test Tools - VSPERF

Modular test framework, combining traffic generation, virtual switches, VNFs, and network configuration test cases

IETF Internet Draft: https://tools.ietf.org/html/draft-ietf-bmwg-vswitch-opnfv-01 Test-cases: configure and program vSwitch, deploy and terminate VNFs, execute a traffic generator, modify a VSPERF configuration, execute external commands, etc. http://docs.opnfv.org/en/latest/submodules/vswitchperf/docs/testing/user/userguide/teststeps.html Metrics: Throughput, Packet delay, Packet delay variation, Packet loss, Burst behavior, Packet re-ordering, Packet correctness, Availability and capacity of the DUT http://docs.opnfv.org/en/latest/submodules/vswitchperf/docs/testing/developer/requirements/vswitchperf_ltd.html Results

  • OPNFV community lab resources https://wiki.opnfv.org/display/pharos/VSPERF+in+Intel+Pharos+Lab+-+Pod+12
  • Scenario example http://docs.opnfv.org/en/latest/submodules/vswitchperf/docs/testing/developer/results/scenario.html
  • CI daily results: https://build.opnfv.org/ci/view/vswitchperf/ https://build.opnfv.org/ci/view/vswitchperf/job/vswitchperf-daily-danube/lastBuild/console
  • Visualization (proto): http://testresults.opnfv.org/proto/index-vsperf.html

Speed Accuracy Reliability Scale

Activation

  • Address Learning Rates

and Caching Capacities

  • Packet Processing

Latencies and Variations

  • Flow

Addition

  • RFC2544.

System RecoveryTime

  • RFC2544.

ResetTime

  • RFC2889.

AddressCaching Capacity Operation

  • RFC2544.PacketLossRatio
  • RFC2544.PacketLossRate FrmMod
  • RFC2544.BackToBack Frames
  • RFC2889.MaxForwarding Rate
  • RFC2889.ForwardPressure
  • RFC2889.BroadcastFrame Forwarding
  • RFC2889.BroadcastFrame Latencytest
  • CPU.RFC2544.0PacketLoss
  • RFC2544.WorstN-BestN
  • InterPAcketDelayVariation RFC5481
  • Frame

Filtering

  • RFC2889.

Soak

  • RFC2889.

SoakFrame Modifcation

  • PacketDelay

Variation. RFC3393. Soak

  • Scalability.RFC25

44.0PacketLoss

  • MemoryBandwid

th.RFC2544.0Pac ketLoss.Scalability

slide-23
SLIDE 23

NFV Test Tools - Yardstick

Yardstick is used in for verifying the OPNFV infrastructure KPIs and some OPNFV features

  • Test configurations in YAML … parsed and converted into an internal model … converted into a Heat template and deployed
  • Scenario runners execute commands in VMs with SSH
  • Output written as json records to a file or influxdb, results shown with Grafana

Test Cases: http://docs.opnfv.org/en/latest/submodules/yardstick/docs/testing/user/userguide/15-list-of-tcs.html Metrics: http://docs.opnfv.org/en/latest/submodules/yardstick/docs/testing/user/userguide/02-methodology.html#metrics Results: http://testresults.opnfv.org/grafana/

slide-24
SLIDE 24

1.43 24.90

COLORADO (FUEL) LATEST (KOLLA) DANUBE

25

Test Example - TCP VM Performance

slide-25
SLIDE 25

Test Example – vSwitch Flow Rules

  • OpenStack deployed with Fuel/Apex uses

single logical tenant network (two tenant networks possible with DevStack/Kolla)

  • Two OVS bridges by default. Br-prv is
  • utward facing bridge. Br-int is inner bridge

where VMs are logically connected

  • VLAN segmentation means additional modify

VLAN ID required to and from VMs

  • OVS flow rules are created by OpenStack

Neutron

  • OVS flow action NORMAL - FDB table

updated per packet

slide-26
SLIDE 26

27

Colorado deployment with OVS normal mode (MAC learning) comparing throughput when UDP traffic has varying source MAC address and varying UDP ports

Varying source MAC Varying UDP port

1 2 3 4 5 6 70 128 256 512 1,024 1,280 1,518

Millions of Packets per Second

1 Flow 100 Flows 1000 Flows 1 2 3 4 5 6 70 128 256 512 1,024 1,280 1,518

Millions of Packets per Second

1 Flow 100 Flows 1000 Flows

Test Example - Traffic Flow Comparison

slide-27
SLIDE 27

Stable

Integrate Deploy Test Deploy Test

28

Stand-alone Previous Master Future? Developer Testing Test Cases Test Tools Configurations Performance Ranges

Reference Scenarios

Stress Tests OPNFV Test Suites User Testing

Vendor Platform Tests

Automated Performance Test Suite Test Data

NFV Platform Performance Testing

Testing Artifacts Performance Reference Scenarios

Good Data + Good Explanations -> Good Configurations -> Good Performance

“NFV Performance Standards” Analytics

Release Testing

slide-28
SLIDE 28

NFV Testing - Conclusions NFV is more challenging than Cloud Computing …

  • Just working is not good enough

Test areas in evolution are …

  • Advancing CI and integration test toolchains
  • Integrating test capabilities with upstream communities
  • Performance Testing automation
slide-29
SLIDE 29

You’re Invited! Free Beer, Basketball, and Networking

  • Open Source Community Reception
  • Tonight from 6:00 – 10:00 PM
  • Levi’s Stadium, Yahoo! Fantasy Football Lounge (a short walk)
  • Sponsored by Serro, Juniper, and Inocybe
  • Meet community members from OPNFV, OpenDaylight,

OpenStack, Fd.io and ONAP. Light food and drinks will be provided.

  • We'll also be showing the NCAA Men’s Basketball Final Game!
  • RSVP Required via OPNFV website here:
  • https://www.opnfv.org/event/open-networking-summit