FUNCTEST IN DEPTH Jose Lausuch (Ericsson) OPNFV Summit 2015 - - PowerPoint PPT Presentation

▶
functest in depth
SMART_READER_LITE
LIVE PREVIEW

FUNCTEST IN DEPTH Jose Lausuch (Ericsson) OPNFV Summit 2015 - - PowerPoint PPT Presentation

Sep 29, 2023 123 likes •643 views

FUNCTEST IN DEPTH Jose Lausuch (Ericsson) OPNFV Summit 2015 jose.lausuch@ericsson.com Agenda What is Functest? Test cases Repository structure CI & Automation Brahmaputra Release 09/11/2015 Functest in Depth, OPNFV Summit 2015 2

slide-1
SLIDE 1

FUNCTEST IN DEPTH

Jose Lausuch (Ericsson)

OPNFV Summit 2015

jose.lausuch@ericsson.com

slide-2
SLIDE 2

Agenda

09/11/2015 Functest in Depth, OPNFV Summit 2015 2

What is Functest? Test cases Repository structure CI & Automation Brahmaputra Release

slide-3
SLIDE 3

OPNFV Projects ecosystem

09/11/2015 Functest in Depth, OPNFV Summit 2015 3 TEST Projects

Yardstick Qtip VPerf

…

FEATURE Projects

VPN SFC SDN Contr. Resource mgmt.

…

PHAROS

Testbed

OCTOPUS (CI)

RELENG

INSTALLER Projects

Fuel Foreman JOID Compass

…

Functest

Testbed Testbed Testbed Testbed

slide-4
SLIDE 4

What is Functest?

09/11/2015 Functest in Depth, OPNFV Summit 2015 4

slide-5
SLIDE 5

What we do…

Focus on verifying the OpenStack deployment and the SDN Controllers Have a full integration and automation mechanism Provide comprehensive testing methodology

09/11/2015 Functest in Depth, OPNFV Summit 2015 5

Base System Functionality Testing

slide-6
SLIDE 6

09/11/2015 Functest in Depth, OPNFV Summit 2015 6

ETSI NFV Reference

WE DON’T: Performance tests HW tests Frameworks

slide-7
SLIDE 7

The gang…

09/11/2015 Functest in Depth, OPNFV Summit 2015 7

slide-8
SLIDE 8

The project…

09/11/2015 Functest in Depth, OPNFV Summit 2015 8

source: Bitergia (http://projects.bitergia.com/opnfv) date: November 2015

slide-9
SLIDE 9

Functest Testcases Arno

09/11/2015 Functest in Depth, OPNFV Summit 2015 9

slide-10
SLIDE 10

Functest Testcases

vPing test case

Create machines and verify connectivity

ODL test case

Robot framework, ODL functional testing

Rally bench tests

Benchmark the OpenStack deployment

Tempest test

OpenStack native tests (100+ smoke-tests)

09/11/2015 Functest in Depth, OPNFV Summit 2015 10

slide-11
SLIDE 11

Functest Testcases

vPing test case

Create machines and verify connectivity

09/11/2015 Functest in Depth, OPNFV Summit 2015 11

slide-12
SLIDE 12

VM2

vPing test case

09/11/2015 Functest in Depth, OPNFV Summit 2015 12

VM1

provisional Neutron network

slide-13
SLIDE 13

Functest Testcases

ODL test case

Robot framework, ODL functional testing

09/11/2015 Functest in Depth, OPNFV Summit 2015 13

slide-14
SLIDE 14

ODL test case

09/11/2015 Functest in Depth, OPNFV Summit 2015 14

Dashboard/API (Horizon) Compute Storage Networking (Neutron) ….. ML2 Plug-in Northbound API ODL – Neutron REST API

OVSDB

OVS Nodes

slide-15
SLIDE 15

ODL test cases

09/11/2015 Functest in Depth, OPNFV Summit 2015 15

Borrowed from ODL repository 4 basic test cases:

Check ODL Restconf Create/Delete/Check Network Create/Delete/Check Subnet Create/Delete/Check Port

slide-16
SLIDE 16

Functest Testcases

Rally bench tests

Benchmark the OpenStack deployment

09/11/2015 Functest in Depth, OPNFV Summit 2015 16

slide-17
SLIDE 17

Rally

09/11/2015 Functest in Depth, OPNFV Summit 2015 17

Rally is a benchmarking tool that answers the question: “How does OpenStack work at scale?”.

slide-18
SLIDE 18

Rally - scenario example

09/11/2015 Functest in Depth, OPNFV Summit 2015 18

Runners:

  • Serial: launch scenario N times in one thread
  • Constant: launch scenario N times in M parallel threads
  • RPS: generates N threads per second M times

Success criteria (SLA)

  • You can define a maximum time per iteration
  • It will skip the test if 1 iteration fails

Context:

  • Contexts needed for this scenario (tenants/users)
slide-19
SLIDE 19

Rally - output

09/11/2015 Functest in Depth, OPNFV Summit 2015 19

slide-20
SLIDE 20

Functest Testcases

Tempest test

OpenStack native tests (100+ smoke-tests)

09/11/2015 Functest in Depth, OPNFV Summit 2015 20

slide-21
SLIDE 21

Tempest test

09/11/2015 Functest in Depth, OPNFV Summit 2015 21

Tempest is a set of integration tests to be run against a live OpenStack cluster Rally performs the tempest installation and generation

  • f tempest.conf
slide-22
SLIDE 22

Functest Structure

today

09/11/2015 Functest in Depth, OPNFV Summit 2015 22

slide-23
SLIDE 23

Directory structure

09/11/2015 Functest in Depth, OPNFV Summit 2015 23

Main configuration script. Installs needed tools (Rally, Robot, …) Configuration parameters SDN Controllers test suites Openstack related tests (Rally, Tempest) vPing test case Dockerfile and automation scripts Format results to dashboard ready and other utils vIMS test case

slide-24
SLIDE 24

Configuration file

09/11/2015 Functest in Depth, OPNFV Summit 2015 24

Configuration yaml file

Predefined tested values Can be given as an input to Functest Possible to specify commit IDs of the used repositories Versioning control

slide-25
SLIDE 25

CI & Aut

09/11/2015 Functest in Depth, OPNFV Summit 2015 25

tion

slide-26
SLIDE 26

CI & Automation

09/11/2015 Functest in Depth, OPNFV Summit 2015 26

Functest Jenkins job

Triggered automatically after a successful deployment One macro per test case

You can execute a single test From Jenkins

slide-27
SLIDE 27

CI & Automation

09/11/2015 Functest in Depth, OPNFV Summit 2015 27

Daily verification of OPNFV platform

New Features

JENKINS

BUILD DEPLOYMENT FUNCTEST YARDSTICK ….

slide-28
SLIDE 28

Execution of tests

09/11/2015 Functest in Depth, OPNFV Summit 2015 28

Target POD Jumphost

Don’t worry sir, I’ll do it for you

jjob

slide-29
SLIDE 29

Brahmaputra release

09/11/2015 Functest in Depth, OPNFV Summit 2015

slide-30
SLIDE 30

Functest in Brahmaputra release

Dockerization Test Result collection Dashboard Virtual IMS test case + SDN Controllers tests Feature projects

09/11/2015 Functest in Depth, OPNFV Summit 2015 30

slide-31
SLIDE 31

Docker -ization

09/11/2015 Functest in Depth, OPNFV Summit 2015 31

slide-32
SLIDE 32

Functest Docker

09/11/2015 Functest in Depth, OPNFV Summit 2015 32

$docker run -t opnfv/functest $dir/run_test.sh

slide-33
SLIDE 33

Functest Docker

09/11/2015 Functest in Depth, OPNFV Summit 2015 33

Without Docker With Docker

  • Have to install needed libraries in

your jumphost

  • Maintain in the documentation

the needed libraries and dependencies

  • Possible OS dependencies
  • Not portable
  • Needed libraries preinstalled in

the docker image

  • Update image when there is a

new software dependency and push to the OPNFV Docker Hub

  • No OS dependencies
  • Portable
slide-34
SLIDE 34

Execution of tests

09/11/2015 Functest in Depth, OPNFV Summit 2015 34

Target POD Jumphost Docker Hub

Don’t worry sir, I’ll do it for you

jjob

slide-35
SLIDE 35

MRF PCRF CSCF HSS

vIMS test case

09/11/2015 Functest in Depth, OPNFV Summit 2015 35

slide-36
SLIDE 36

vIMS test case

Full automation of a core Open Source vIMS solution on OPNFV from Clearwater Deployment of an orchestrator (Cloudify) to manage the VNF Run suite of signaling tests on vIMS Successfully tested from Jenkins on LinuxFoundation POD2

09/11/2015 Functest in Depth, OPNFV Summit 2015 36

SIP

slide-37
SLIDE 37

vIMS test case

09/11/2015 Functest in Depth, OPNFV Summit 2015 37

1) Deploy the orchestrator 2) Create the tenant 3) Create the VNF (10 VMs) 4) Launch test suite (more than 100 signaling tests) 5) Collect results 6) Clean-up

slide-38
SLIDE 38

SDN Controllers

09/11/2015 Functest in Depth, OPNFV Summit 2015 38

slide-39
SLIDE 39

SDN Controllers

Extension of OpenDaylight test cases Integration of ONOS test framework Open Contrail test

09/11/2015 Functest in Depth, OPNFV Summit 2015 39

slide-40
SLIDE 40

Test results collection

09/11/2015 Functest in Depth, OPNFV Summit 2015 40

slide-41
SLIDE 41

Test results collection

09/11/2015 Functest in Depth, OPNFV Summit 2015 41

Test projects

Mongo DB REST API

VPerf Yardstick QTIP Bottlenecks Functest …

slide-42
SLIDE 42

Test results collection - API

Filter per test case

– http://<DB_Public_IP>/results?case=vPing

Filter per pod

– http://<DB_Public_IP>/results?pod=opnfv-jump-2

Filter per project

– http://<DB_Public_IP>/results?project=yardstick

Combination of filters

– http://<DB_Public_IP>/results?project=functest&case=Tempest&pod=opnfv-jump-2

09/11/2015 Functest in Depth, OPNFV Summit 2015 42

slide-43
SLIDE 43

Dashboard

09/11/2015 Functest in Depth, OPNFV Summit 2015 43

slide-44
SLIDE 44

Dashboard

Show all the results using simplistic graphs Possible to filter per project/test case/… Comparison of previous results with recent ones …

09/11/2015 Functest in Depth, OPNFV Summit 2015 44

slide-45
SLIDE 45

Dashboard Draft

09/11/2015 Functest in Depth, OPNFV Summit 2015 45

slide-46
SLIDE 46

Beyond Brahmaputra

09/11/2015 Functest in Depth, OPNFV Summit 2015 46

New functional test (vCPE, vHGW, vCDN, …) Results analytics Better Rally usage More SDN controllers test integration Broaden test coverage (security, O&M, …)

slide-47
SLIDE 47

Lessons learned in Arno

09/11/2015 Functest in Depth, OPNFV Summit 2015 47

Need for stable labs some weeks before the release Need to run the tests on different PODs from CI Complex troubleshooting Functest cannot solve all problems

slide-48
SLIDE 48

Challenges

09/11/2015 Functest in Depth, OPNFV Summit 2015 48

Different OpenStack installers Different SDN controllers Many features Lots of combinations to be tested !!

… …

slide-49
SLIDE 49

Summary

09/11/2015 Functest in Depth, OPNFV Summit 2015 49

First test project in OPNFV Validate a fresh OPNFV deployment Umbrella for testing projects Umbrella for feature projects We don’t bite, come to us with questions 

slide-50
SLIDE 50

09/11/2015 Functest in Depth, OPNFV Summit 2015 50

slide-51
SLIDE 51

Thank you!

09/11/2015 Functest in Depth, OPNFV Summit 2015 51