VNF Benchmarking Methodology (draft-rosa-bmwg-vnfbench-00.txt) BMWG - - PowerPoint PPT Presentation

▶
vnf benchmarking methodology draft rosa bmwg vnfbench 00
SMART_READER_LITE
LIVE PREVIEW

VNF Benchmarking Methodology (draft-rosa-bmwg-vnfbench-00.txt) BMWG - - PowerPoint PPT Presentation

Sep 11, 2022 7 likes •327 views

VNF Benchmarking Methodology (draft-rosa-bmwg-vnfbench-00.txt) BMWG - IETF 95 Rosa, Raphael V. Rothenberg, Christian E. Szabo, Robert FEEC/UNICAMP and Ericsson Research Hungary April 7, 2016 1 / 17 Motivation New

slide-1
SLIDE 1

VNF Benchmarking Methodology (draft-rosa-bmwg-vnfbench-00.txt)

BMWG - IETF 95 Rosa, Raphael V.†‡ Rothenberg, Christian E.‡ Szabo, Robert†

‡FEEC/UNICAMP and †Ericsson Research Hungary

April 7, 2016

1 / 17

slide-2
SLIDE 2

Motivation

◮ New paradigms of network services envisioned by NFV bring VNFs as software

based entities, which can be deployed in virtualized environments

Figure : NFV Architectural Framework 2 / 17

slide-3
SLIDE 3

Motivation

◮ Virtualized environment (e.g., NFVI PoP) changes frequently in different places

(e.g., platforms, hardware acceleration)

Figure : Use of acceleration abstraction layer (AAL) to enable fully portable VNFC code across servers with different accelerators Figure : VNF Usage of Accelerators http://www.etsi.org/deliver/etsi gs/NFV-IFA/001 099/001/01.01.01 60/gs NFV-IFA001v010101p.pdf 3 / 17

slide-4
SLIDE 4

Motivation

◮ VNFs need continuous development/integration ◮ VNF Descriptors can specify performance profiles containing metrics (e.g.,

throughput) associated with allocated resources (e.g., vCPU)

Figure : VNF Environment Examples http://www.etsi.org/deliver/etsi gs/NFV-EVE/001 099/004/01.01.01 60/gs NFV-EVE004v010101p.pdf 4 / 17

slide-5
SLIDE 5

Motivation

◮ Process for metrics extraction can be automated - on-going work VBaaS -

https://datatracker.ietf.org/doc/draft-rorosz-nfvrg-vbaas/

Figure : NFV MANO and VBaaS

5 / 17

slide-6
SLIDE 6

Motivation

◮ Analysis with and without instrumentation showed interesting results (e.g., vCDN)

Figure : Bytes worked on per millisecond ration of vCDN Figure : NFV Testing Framework: a) no instrumentation; b) embedded instrumentation An Instrumentation and Analytics Framework for Optimal and Robust NFV Deployment, IEEE Comm Magazine 2015 6 / 17

slide-7
SLIDE 7

Assumptions

,----. ,----. ( VNF2 ) {VNF1: {10Mbps,200ms}{ ( VNF1 ) ‘----’ {{2CPU, 8GB}@PoP1} ‘----’ {{8CPU, 16GB}@PoP2} +---------+ {{4CPU, 4GB}@PoP3}}} |Customers| {20Mbps,300ms}...} +-----+---+ {VNF2:{10Mbps,200ms}{ | {{8CPU, 16GB}@PoP1} | ...}} +-----+-------+ ,---------------. | | ( VNF-Profiles )<--->| NFVO / VNFM | ‘---------------’ | | +-+----+----+-+ / | \ V V V +------+ +------+ +------+ | VIM1 | | VIM2 | | VIM3 | +-+----+ +-+----+ +-+----+ | | | NFVI *-------+--------+--------+-------* | | | | | +------+ SAPs | +-----+-+ +---+---+ +-+-----+ | SAPs +------+ |Agents|==>O--+-| PoP 1 |--| PoP 2 |--| PoP 3 |-+--O==>|Agents| +------+ | +-------+ +-------+ +-------+ | +------+ | PoP1 PoP2 PoP3 | | Container Enhanced Baremetal| | OS Hypervisor | *---------------------------------* 7 / 17

slide-8
SLIDE 8

Assumptions

Problem to be solved:

◮ Gain information about VNFs’

performance metrics with given reserved resources at given VIM (NFVI PoP).

,----. ,----. ( VNF2 ) {VNF1: {10Mbps,200ms}{ ( VNF1 ) ‘----’ {{2CPU, 8GB}@PoP1} ‘----’ {{8CPU, 16GB}@PoP2} +---------+ {{4CPU, 4GB}@PoP3}}} |Customers| {20Mbps,300ms}...} +-----+---+ {VNF2:{10Mbps,200ms}{ | {{8CPU, 16GB}@PoP1} | ...}} +-----+-------+ ,---------------. | | ( VNF-Profiles )<--->| NFVO / VNFM | ‘---------------’ | | +-+----+----+-+ / | \ V V V +------+ +------+ +------+ | VIM1 | | VIM2 | | VIM3 | +-+----+ +-+----+ +-+----+ | | | NFVI *-------+--------+--------+-------* | | | | | +------+ SAPs | +-----+-+ +---+---+ +-+-----+ | SAPs +------+ |Agents|==>O--+-| PoP 1 |--| PoP 2 |--| PoP 3 |-+--O==>|Agents| +------+ | +-------+ +-------+ +-------+ | +------+ | PoP1 PoP2 PoP3 | | Container Enhanced Baremetal| | OS Hypervisor | *---------------------------------* 7 / 17

slide-9
SLIDE 9

Assumptions

Problem to be solved:

◮ Gain information about VNFs’

performance metrics with given reserved resources at given VIM (NFVI PoP).

An important usage:

◮ Orchestration (e.g., NFVO) needs to

know throughput, latency, among

  • ther metrics, performance values for

a given resource allocation (cpu, memory, storage) of a VNF at a VIM.

,----. ,----. ( VNF2 ) {VNF1: {10Mbps,200ms}{ ( VNF1 ) ‘----’ {{2CPU, 8GB}@PoP1} ‘----’ {{8CPU, 16GB}@PoP2} +---------+ {{4CPU, 4GB}@PoP3}}} |Customers| {20Mbps,300ms}...} +-----+---+ {VNF2:{10Mbps,200ms}{ | {{8CPU, 16GB}@PoP1} | ...}} +-----+-------+ ,---------------. | | ( VNF-Profiles )<--->| NFVO / VNFM | ‘---------------’ | | +-+----+----+-+ / | \ V V V +------+ +------+ +------+ | VIM1 | | VIM2 | | VIM3 | +-+----+ +-+----+ +-+----+ | | | NFVI *-------+--------+--------+-------* | | | | | +------+ SAPs | +-----+-+ +---+---+ +-+-----+ | SAPs +------+ |Agents|==>O--+-| PoP 1 |--| PoP 2 |--| PoP 3 |-+--O==>|Agents| +------+ | +-------+ +-------+ +-------+ | +------+ | PoP1 PoP2 PoP3 | | Container Enhanced Baremetal| | OS Hypervisor | *---------------------------------* 7 / 17

slide-10
SLIDE 10

VNF Benchmarking Considerations

Adopt VNF benchmarking considerations draft

Follow additional considerations proposed by ETSI documents (e.g., pre-deployment testing draft)

Black-Box SUT with Black-Box Benchmarking Agents

In virtualization environments neither the VNF instance nor the underlying virtualization environment nor the agents specifics may be known by the entity managing abstract resources. This implies black box testing with black box functional components, which are configured by opaque configuration parameters defined by the VNF developers or alike for the benchmarking entity (e.g., NFVO)

Considerations for Benchmarking Virtual Network Functions and Their Infrastructure https://datatracker.ietf.org/doc/draft-morton-bmwg-virtual-net/ 8 / 17

slide-11
SLIDE 11

Testing Methodologies

Benchmarking

To measure VNF’s throughput, latency, frame loss rate metrics for given cpu, memory, storage reservation at given VIM.

9 / 17

slide-12
SLIDE 12

Testing Methodologies

Benchmarking

To measure VNF’s throughput, latency, frame loss rate metrics for given cpu, memory, storage reservation at given VIM.

Dimensioning

To determine cpu, memory, storage reservation metrics for given VNF at given VIM for target throughput, latency, frame loss rate parameters.

9 / 17

slide-13
SLIDE 13

Testing Methodologies

Benchmarking

To measure VNF’s throughput, latency, frame loss rate metrics for given cpu, memory, storage reservation at given VIM.

Dimensioning

To determine cpu, memory, storage reservation metrics for given VNF at given VIM for target throughput, latency, frame loss rate parameters.

Verification

To assess if given throughput, latency, frame loss rate metrics of a VNF is met with given cpu, memory, storage reservation at given VIM.

9 / 17

slide-14
SLIDE 14

Testing Methodologies

Benchmarking

To measure VNF’s throughput, latency, frame loss rate metrics for given cpu, memory, storage reservation at given VIM.

Dimensioning

To determine cpu, memory, storage reservation metrics for given VNF at given VIM for target throughput, latency, frame loss rate parameters.

Verification

To assess if given throughput, latency, frame loss rate metrics of a VNF is met with given cpu, memory, storage reservation at given VIM.

Observation

Dimensioning and verification boil down to benchmarking operation(s).

9 / 17

slide-15
SLIDE 15

VNF Benchmarking Methodology

Approach

◮ Definition of VNF-BPs for each testing procedure and its consequent output,

VNF-Profile

◮ Information about Benchmarking Methodology for Network Interconnect Devices

(RFC2544)

◮ IP Performance Metrics (IPPM) Framework (RFC2330)

10 / 17

slide-16
SLIDE 16

VNF Benchmarking Methodology

VNF Benchmarking Profile

The specification how to measure a VNF Profile. VNF-BP may be specific to a VNF or applicable to several VNF types. The specification includes structural and functional instructions, and variable parameters (metrics) at different abstractions (e.g., vCPU, memory, throughput, latency; session, transaction, tenants, etc.).

VNF Profile

Is a mapping between virtualized resources (e.g., vCPU, memory) and VNF performance (e.g., throughput, latency between in/ out ports) at a given NFVI PoP. An

  • rchestration function can use the VNF Profile to select a host (NFVI PoP) for a VNF

and to allocate necessary resources to deliver the required performance characteristics.

11 / 17

slide-17
SLIDE 17

Throughput

Objective

Provide, for a particular set of resources allocated, the throughput among two or more VNF ports, expressed in VNF-BP

12 / 17

slide-18
SLIDE 18

Throughput

Objective

Provide, for a particular set of resources allocated, the throughput among two or more VNF ports, expressed in VNF-BP

Prerequisite

VNF (SUT) must be deployed and stable and its allocated resources collected. VNF must be reachable by agents. The frame size to be used for agents must be defined in the VNF- BP

12 / 17

slide-19
SLIDE 19

Throughput

Procedure

  • 1. Establish connectivity between agents and VNF ports
  • 2. Agents initiate source of traffic, specifically designed for VNF test, increasing rate

periodically

  • 3. Throughput is measured when traffic rate is achieved without frame losses

13 / 17

slide-20
SLIDE 20

Throughput

Procedure

  • 1. Establish connectivity between agents and VNF ports
  • 2. Agents initiate source of traffic, specifically designed for VNF test, increasing rate

periodically

  • 3. Throughput is measured when traffic rate is achieved without frame losses

Reporting Format

Must contain VNF allocated resources and throughput measured (aka throughput in [rfc2544])

13 / 17

slide-21
SLIDE 21

Latency

Objective

Provide, for a particular set of resources allocated, the latency among two or more VNF ports, expressed in VNF-BP

14 / 17

slide-22
SLIDE 22

Latency

Objective

Provide, for a particular set of resources allocated, the latency among two or more VNF ports, expressed in VNF-BP

Prerequisite

VNF (SUT) must be deployed and stable and its allocated resources collected. VNF must be reachable by agents. The frame size and respective throughput to be used for agents must be defined in the VNF-BP

14 / 17

slide-23
SLIDE 23

Latency

Procedure

  • 1. Establish connectivity between agents and VNF ports
  • 2. Agents initiate source of traffic, throughput and frame size specifically designed

for VNF test

  • 3. Latency is measured when throughput is achieved for the period of time specified

in VNF-BP

15 / 17

slide-24
SLIDE 24

Latency

Procedure

  • 1. Establish connectivity between agents and VNF ports
  • 2. Agents initiate source of traffic, throughput and frame size specifically designed

for VNF test

  • 3. Latency is measured when throughput is achieved for the period of time specified

in VNF-BP

Reporting Format

Must contain VNF allocated resources, throughput used for stimulus and latency measurement (aka latency in [rfc2544])

15 / 17

slide-25
SLIDE 25

Frame Loss Rate

Objective

Provide, for a particular set of resources allocated, the frame loss rate among two or more VNF ports, expressed in VNF-BP

16 / 17

slide-26
SLIDE 26

Frame Loss Rate

Objective

Provide, for a particular set of resources allocated, the frame loss rate among two or more VNF ports, expressed in VNF-BP

Prerequisite

VNF (SUT) must be deployed and stable, its allocated resources collected specifying any particular feature of the underlying VNF virtualized environment, provided by NFVO/VIM or independently extracted. VNF must be reachable by agents. Rate of source traffic and frame size used for agents stimulus must be defined in VNF-BP

16 / 17

slide-27
SLIDE 27

Frame Loss Rate

Procedure

  • 1. Establish connectivity between agents and VNF ports
  • 2. Agents initiate source of traffic, specifically designed for VNF test, achieving rate
  • f source traffic defined in VNF-BP
  • 3. Frame loss rate is measured when pre-defined traffic rate is achieved for period of

time established in VNF-BP

17 / 17

slide-28
SLIDE 28

Frame Loss Rate

Procedure

  • 1. Establish connectivity between agents and VNF ports
  • 2. Agents initiate source of traffic, specifically designed for VNF test, achieving rate
  • f source traffic defined in VNF-BP
  • 3. Frame loss rate is measured when pre-defined traffic rate is achieved for period of

time established in VNF-BP

Reporting Format

Must contain VNF allocated resources, rate of source traffic used as stimulus and frame loss rate measurement (aka frame loss rate in [rfc2544])

17 / 17

slide-29
SLIDE 29

Summary

Messages

  • 1. Open discussions about VNF benchmarking methodology
  • 2. Interesting to work on separate documents: terminology and applicability

Next Steps T

his work is supported by Ericsson Research Brazil This work is supported by FP7 UNIFY, a research project partially funded by the European Community under the Seventh Framework Program (grant agreement no. 619609). The views expressed here are those of the authors only. The European Commission is not liable for any use that may be made of the information in this document 18 / 17

slide-30
SLIDE 30

Summary

Messages

  • 1. Open discussions about VNF benchmarking methodology
  • 2. Interesting to work on separate documents: terminology and applicability

Next Steps

◮ VNF scaling

T

his work is supported by Ericsson Research Brazil This work is supported by FP7 UNIFY, a research project partially funded by the European Community under the Seventh Framework Program (grant agreement no. 619609). The views expressed here are those of the authors only. The European Commission is not liable for any use that may be made of the information in this document 18 / 17

slide-31
SLIDE 31

Summary

Messages

  • 1. Open discussions about VNF benchmarking methodology
  • 2. Interesting to work on separate documents: terminology and applicability

Next Steps

◮ VNF scaling ◮ VNF instantiation time

T

his work is supported by Ericsson Research Brazil This work is supported by FP7 UNIFY, a research project partially funded by the European Community under the Seventh Framework Program (grant agreement no. 619609). The views expressed here are those of the authors only. The European Commission is not liable for any use that may be made of the information in this document 18 / 17

slide-32
SLIDE 32

Summary

Messages

  • 1. Open discussions about VNF benchmarking methodology
  • 2. Interesting to work on separate documents: terminology and applicability

Next Steps

◮ VNF scaling ◮ VNF instantiation time ◮ Improve VNF Benchmark Profile concept (e.g., topology)

T

his work is supported by Ericsson Research Brazil This work is supported by FP7 UNIFY, a research project partially funded by the European Community under the Seventh Framework Program (grant agreement no. 619609). The views expressed here are those of the authors only. The European Commission is not liable for any use that may be made of the information in this document 18 / 17