How to use OPNFV Testing tools to test/validate an Openstack - - PowerPoint PPT Presentation

How to use OPNFV Testing tools to test/validate an Openstack provided by a vendor

Morgan Richomme

NFV architect at Orange Labs OPNFV ambassador ONAP & OPNFV Functest committer

Speakers

Valentin Boucher

Software engineer at Kontron Canada OPNFV Functest committer:

https://wiki.opnfv.org/display/functest

Agenda

• OPNFV testing overview
• OPNFV Testing chain
• OPNFV Testing tools
• Reusing those tools
• Kontron
• Orange

OPNFV Testing Overview Testing chain and tools …..

OPNFV CI complexity

Addons / Features SDN Controller Installer / Deployer Lab/POD SFC BGPVPN DPDK FDio APEX Compass Daisy Joid Linux Foundation Intel Ericsson Huawei OpenDayLight ONOS OpenContrail Fuel XCI

OPNFV Testing chain

Scenario

HA/noHA, SDN, DPDK etc..

Deployer

APEX, Compass, Daisy, Fuel and JOID

Pod

LF, Huawei, Intel etc ..

Version

Master, Fraser

Deploy

Installer project

Report

Test api, reporting

Test

Functest, Yardstick etc

Test result

OPNFV Testing ecosystem

Functional testing -> Functest Functest

• Answer the question: “Does my infrastructure simply work as expected?”
• Provide tests from basic connectivity (<1s) to VNF testing (>1h)
• Leverage Upstream test suites from the integrated components

(OpenStack/Rally & Tempest, Kubernetes/Default test suites, OpenDaylight, …)

• Allow “A la carte” test combination through rich, light and modular Alpine

based framework, adapted to CI/CD chains

• PTL: C.Ollivier, Orange

Functest testcases for OpenStack

Complexity

Gating Official Release validation Target validation Full stack

Performance testing -> Yardstick & cie Yardstick

• Answer the question: “Qualify my NFVI and give me valuable figures?”
• Qualify Compute, Network and Storage domains through elementary tests
• Leverage OPNFv community to build baseline and define acceptable

thresholds for Telco Cloud solution

• Allow lab owner to compare their Telco Cloud KPI with the other OPNFV labs
• In line with ETSI NFV
• PTL: R. Brattain, Intel

Yardstick

More than 50 available tests covering different domains

(compute, network, storage, HA, IPv6, KVM, ..)

Src: http://docs.opnfv.org/en/stable-fraser/submodules/yardstick/docs/testing/user/userguide/15-list-of-tcs.html

Yardstick

Src: Alassane Samba (Orange) https://wiki.opnfv.org/display/testing/R+post-processing+of+the+Yardstick+results

TC_012 post processing

Yardstick testing capacity

• Generic test cases
• Compute
• Storage
• Network
• HA test cases
• Service HA
• Node HA
• Function test cases
• SRIOV
• Dpdk
• Hugepages

Certification -> Dovetail Dovetail

• Answer to question: “Do these products conform to the interfaces,

behaviors and functionalities defined by OPNFV?”

• Main objective: reduce VNF onboarding cost
• Do not implement test cases in their own code base but reuse the test

cases from the OPNFV test projects (Functest, Yardstick etc..)

• Allow vendors to use the OPNFV verified logo
• PTL: Georg Kunz, Ericsson

Dovetail – Test scope

Whole OPNFV testsuite OPNFV Verified 2018.01 Mandatory test cases

• OpenStack API tests (205 tests)
• Layer 2 packet forwarding (2 tests)
• OpenStack control service HA (8 tests)

Optional test cases

• IPv6 tenant networks (25 tests)
• BGPVPNs (4 tests)
• Fundamental VIM capabilities (30 tests)

Future test case candidates

• Neutron Trunk ports
• BGPVPN Tempest tests
• SDN Controller HA
• VNF lifecycle management
• ….

Future ..

Certification workflow

• 1. Submission of participation form
• 2. Testing
• 3. Submission of results
• 4. Notification of reviewers
• 5. Review of test results
• 6. Grant of use of program marks

Reuse those tools

Kontron communications unit (Canada): part ODM, part integrator SYMKLOUD hw and solution portfolio: integrate open source with high density converged platforms + cloud servers + white box switches Used for: NFVi at Edge + Core, SDN, DC clouds for the telco and video/broadcast, and cloud providers We make it easy: removed complexities to integrate software (CI/CD open source or not) on Symkloud hw; plus can ‘mix-n-match’ composable modules (GPU, FPGA, Crypto, Storage). Ramp up services faster and with lower risk: easy to Configure, Deploy and Manage services on

• ur platforms (automated switch configuration and

deployment, integrate addons,etc.)

Context

Problematics

How to test/validate that our deployment of Openstack or Kubernetes on SymKloud is fine ?

• Different SymKloud Platforms

○ Different type of switches (40Gb, 100Gb/SDN-ready etc..) ○ Different CPU (12, 16 .. cores) ○ Servers with/without GPU or FPGA etc…

• Differents software type and version

(OpenStack Newton/Queens, Kubernetes 1.9/1.10 etc..)

• Differents add-ons

(6wind network acceleration for OpenStack, OpenStack Ironic support etc..)

More features can be added in the future (New deployer, SDN controller, New hardware)

Current CI

Scenario

HA/noHA, ironic, 6wind etc..

Cloud software

OpenStack, Kubernetes

Hardware Version

Arnold, Britney, Develop etc..

Deploy

MAAS/Juju

Report

Testrail

Test

Homemade test suite using Behave

Experimental CI

Scenario

HA/noHA, ironic, 6wind etc..

Cloud software

OpenStack, Kubernetes

Hardware Version

Arnold, Britney, Develop etc..

Deploy

MAAS/Juju

Report

Test api, reporting Testrail

Test

Test result

Test

Homemade test suite Functest, Yardstick

Kontron OPNFV tools

Reporting

Our goals

Why reuse those tools in addition of our internal QA test ?

• Our QA test suite has a more diverse test coverage than the OPNFV

tools (not only focusing on the NFVI+VIM test but also installer, add-ons, configuration and hardware test)

• The OPNFV test suite has more tests related to NFV so it completes
• ur test coverage
• The goal is to be better and close to the customer target usage
• We ensure that next Dovetail verification will work if all opnfv tests pass
• n our platform.

The Orange CI/CD view (Preproduction)

CI/CD viewn ONAP OpenLAB

Bare Metal O/S management Infrastructure Installation ONAP Installation Openstack BIFROST / OPNFV XCI Kolla Ansible ONAP Heat Kubespray Functest Xtesting (Healthchecks/moke/vnf // smoke/conformance) ONAP OOM Functest Xtesting/onap-robot

Run daily Run daily Test DB

Test API ONAP Heat (Community lab)

OPNFV Orange Integration Center Installers 5: Apex, Compass, Daisy, Fuel, Joid Preproduction: 1 (RedHat Director) Orange ONAP OpenLab: 1 Kolla Ansible Scenarios Mores than 30 Preproduction: 1 OpenStack + Contrail Orange ONAP OpenLab: 2 OpenStack / Kubernetes + ONAP Tests Tests (automation): relying on several test projects:

• Functest* for functional testing
• Yardstick* for performance testing
• Qtip for bench HW
• Storeperf for ceph
• Vsperf for Vswitch qualification
• Bottleneck to optimize Yardstick runs

* used for release validation Preproduction: Functest + Yardstick Orange ONAP OpenLab: Functest + ONAP Robot test suites

Reuse of OPNFV tooling for ONAP

Reuse of Functest Xtesting framework to onboard ONAP robot tests

• Only 3 files needed (Robot natively supported)
• Docker size from 1 Go to 100Mo (Alpines)
• Inherit Test reporting capabilities
• Consistent view to manage Infra and orchestration testing (reuse

DB, Test API)

Reuse of OPNFV tooling for ONAP

• Test framework
• Test API/Test DB
• CI/CD methodology

Conclusions

• OPNFV testing tool is useful, evolutive, promoting Upstream First

(Openstack/ODL testing projects), it can easily complete existing vendor suites with a good focus on NFV

• Automation is key: Xtesting framework is very well adapted to this context

(light (Alpines), simple, modular)

• Tooling is easily reusable out of OPNFV context (ONAP,

Openstack/Kubernetes - non OPNFV - solutions) including frameworks, DB, test API

• OPNFV release/Dovetail criteria should be stronger as test frameworks
• ffer more than what is used as release criteria: infrastructure trustability is

key for Telco Cloud/NFV adoption

Thank You!

How does Functest work?

Yardstick Architecture

• User Interface
• Main Framework
• Context
• Scenario
• Runner
• Dispatcher
• Plugin
• Container Manager
• Services

OPNFV Test result API

http://artifacts.opnfv.org/releng/docs/testapi.html case_name: "cloudify_ims", pod_name: "huawei-pod2", version: "fraser", installer: "compass", scenario: "os-odl_l3-nofeature-ha", start_date: "2018-04-18 04:58:10", stop_date: "2018-04-18 05:38:50", criteria: “PASS” , build_tag: "jenkins-functest-baremetal-22", details: { } name: "cloudify", version: "4.0", status: "PASS", duration: 349 name: "clearwater" descriptor_version: "122", version: "107", status: "PASS", duration: 1330 name: "clearwater-live-test”, version: "1.0", status: "PASS", duration: 273, result: {failures: 0, skipped: 30, passed: 46}

• rchestrator

VNF VNF Test Result of each run in OPNFV test API

OPNFV Reporting

http://testresults.opnfv.org/reporting/ Trend, score, iteration by installer, version, scenario

OPNFV Reporting

http://testresults.opnfv.org/reporting/ Last test result by installer, version, scenario