FUNCTEST IN DEPTH Jose Lausuch (Ericsson) OPNFV Summit 2015 - PowerPoint PPT Presentation

FUNCTEST IN DEPTH Jose Lausuch (Ericsson) OPNFV Summit 2015 jose.lausuch@ericsson.com

Agenda What is Functest? Test cases Repository structure CI & Automation Brahmaputra Release 09/11/2015 Functest in Depth, OPNFV Summit 2015 2

OPNFV Projects ecosystem OCTOPUS (CI) RELENG INSTALLER Projects FEATURE Projects TEST Projects Fuel VPN Functest Foreman SFC Yardstick JOID SDN Contr. Qtip Compass Resource mgmt. VPerf … … … Testbed Testbed Testbed Testbed Testbed PHAROS 09/11/2015 Functest in Depth, OPNFV Summit 2015 3

What is Functest? 09/11/2015 Functest in Depth, OPNFV Summit 2015 4

What we do… Base System Functionality Testing Focus on verifying the OpenStack deployment and the SDN Controllers Have a full integration and automation mechanism Provide comprehensive testing methodology 09/11/2015 Functest in Depth, OPNFV Summit 2015 5

ETSI NFV Reference WE DON’T: Performance tests HW tests Frameworks 09/11/2015 Functest in Depth, OPNFV Summit 2015 6

The gang… 09/11/2015 Functest in Depth, OPNFV Summit 2015 7

The project… source: Bitergia (http://projects.bitergia.com/opnfv) date: November 2015 09/11/2015 Functest in Depth, OPNFV Summit 2015 8

Functest Testcases Arno 09/11/2015 Functest in Depth, OPNFV Summit 2015 9

Functest Testcases vPing test case Create machines and verify connectivity ODL test case Robot framework, ODL functional testing Rally bench tests Benchmark the OpenStack deployment Tempest test OpenStack native tests (100+ smoke-tests) 09/11/2015 Functest in Depth, OPNFV Summit 2015 10

Functest Testcases vPing test case Create machines and verify connectivity 09/11/2015 Functest in Depth, OPNFV Summit 2015 11

vPing test case VM1 VM2 provisional Neutron network 09/11/2015 Functest in Depth, OPNFV Summit 2015 12

Functest Testcases ODL test case Robot framework, ODL functional testing 09/11/2015 Functest in Depth, OPNFV Summit 2015 13

ODL test case Dashboard/API (Horizon) Networking Compute Storage ….. (Neutron) ML2 Plug-in Northbound API ODL – Neutron REST API OVSDB 09/11/2015 Functest in Depth, OPNFV Summit 2015 OVS Nodes 14

ODL test cases Borrowed from ODL repository 4 basic test cases: Check ODL Restconf Create/Delete/Check Network Create/Delete/Check Subnet Create/Delete/Check Port 09/11/2015 Functest in Depth, OPNFV Summit 2015 15

Functest Testcases Rally bench tests Benchmark the OpenStack deployment 09/11/2015 Functest in Depth, OPNFV Summit 2015 16

Rally Rally is a benchmarking tool that answers the question: “How does OpenStack work at scale?”. 09/11/2015 Functest in Depth, OPNFV Summit 2015 17

Rally - scenario example Runners: • Serial : launch scenario N times in one thread • Constant : launch scenario N times in M parallel threads • RPS : generates N threads per second M times Context: • Contexts needed for this scenario (tenants/users) Success criteria (SLA) • You can define a maximum time per iteration • It will skip the test if 1 iteration fails 09/11/2015 Functest in Depth, OPNFV Summit 2015 18

Rally - output 09/11/2015 Functest in Depth, OPNFV Summit 2015 19

Functest Testcases Tempest test OpenStack native tests (100+ smoke-tests) 09/11/2015 Functest in Depth, OPNFV Summit 2015 20

Tempest test Tempest is a set of integration tests to be run against a live OpenStack cluster Rally performs the tempest installation and generation of tempest.conf 09/11/2015 Functest in Depth, OPNFV Summit 2015 21

Functest Structure today 09/11/2015 Functest in Depth, OPNFV Summit 2015 22

Directory structure Dockerfile and automation scripts SDN Controllers test suites Format results to dashboard ready and other utils Openstack related tests (Rally, Tempest) vIMS test case vPing test case Main configuration script. Installs needed tools (Rally, Robot, …) Configuration parameters 09/11/2015 Functest in Depth, OPNFV Summit 2015 23

Configuration file Configuration yaml file Predefined tested values Can be given as an input to Functest Possible to specify commit IDs of the used repositories Versioning control 09/11/2015 Functest in Depth, OPNFV Summit 2015 24

tion CI & Aut 09/11/2015 Functest in Depth, OPNFV Summit 2015 25

CI & Automation Functest Jenkins job Triggered automatically after a successful deployment One macro per test case You can execute a single test From Jenkins 09/11/2015 Functest in Depth, OPNFV Summit 2015 26

CI & Automation Daily verification of OPNFV platform JENKINS BUILD DEPLOYMENT FUNCTEST New Features YARDSTICK …. 09/11/2015 Functest in Depth, OPNFV Summit 2015 27

Execution of tests Target POD Don’t worry sir, I’ll do it for you jjob Jumphost 09/11/2015 Functest in Depth, OPNFV Summit 2015 28

Brahmaputra release 09/11/2015 Functest in Depth, OPNFV Summit 2015

Functest in Brahmaputra release Dockerization Test Result collection Dashboard Virtual IMS test case + SDN Controllers tests Feature projects 09/11/2015 Functest in Depth, OPNFV Summit 2015 30

Docker -ization 09/11/2015 Functest in Depth, OPNFV Summit 2015 31

Functest Docker $docker run -t opnfv/functest $dir/run_test.sh 09/11/2015 Functest in Depth, OPNFV Summit 2015 32

Functest Docker Without Docker With Docker • Have to install needed libraries in • Needed libraries preinstalled in your jumphost the docker image • Maintain in the documentation • Update image when there is a the needed libraries and new software dependency and dependencies push to the OPNFV Docker Hub • Possible OS dependencies • No OS dependencies • Not portable • Portable 09/11/2015 Functest in Depth, OPNFV Summit 2015 33

Execution of tests Don’t worry sir, Target POD I’ll do it for you Docker Hub jjob Jumphost 09/11/2015 Functest in Depth, OPNFV Summit 2015 34

HSS CSCF MRF PCRF vIMS test case 09/11/2015 Functest in Depth, OPNFV Summit 2015 35

vIMS test case Full automation of a core Open Source vIMS solution on OPNFV from Clearwater Deployment of an orchestrator (Cloudify) to manage the VNF Run suite of signaling tests on vIMS SIP Successfully tested from Jenkins on LinuxFoundation POD2 09/11/2015 Functest in Depth, OPNFV Summit 2015 36

vIMS test case 1) Deploy the orchestrator 2) Create the tenant 3) Create the VNF (10 VMs) 4) Launch test suite (more than 100 signaling tests) 5) Collect results 6) Clean-up 09/11/2015 Functest in Depth, OPNFV Summit 2015 37

SDN Controllers 09/11/2015 Functest in Depth, OPNFV Summit 2015 38

SDN Controllers Extension of OpenDaylight test cases Integration of ONOS test framework Open Contrail test 09/11/2015 Functest in Depth, OPNFV Summit 2015 39

Test results collection 09/11/2015 Functest in Depth, OPNFV Summit 2015 40

Test results collection Test projects Functest Yardstick QTIP VPerf Bottlenecks … REST API Mongo DB 09/11/2015 Functest in Depth, OPNFV Summit 2015 41

Test results collection - API Filter per test case – http://<DB_Public_IP>/results?case=vPing Filter per pod – http://<DB_Public_IP>/results?pod=opnfv-jump-2 Filter per project – http://<DB_Public_IP>/results?project=yardstick Combination of filters – http://<DB_Public_IP>/results?project=functest&case=Tempest&pod=opnfv-jump-2 09/11/2015 Functest in Depth, OPNFV Summit 2015 42

Dashboard 09/11/2015 Functest in Depth, OPNFV Summit 2015 43

Dashboard Show all the results using simplistic graphs Possible to filter per project/test case/… Comparison of previous results with recent ones … 09/11/2015 Functest in Depth, OPNFV Summit 2015 44

Dashboard Draft 09/11/2015 Functest in Depth, OPNFV Summit 2015 45

Beyond Brahmaputra New functional test (vCPE, vHGW, vCDN, …) Results analytics Better Rally usage More SDN controllers test integration Broaden test coverage (security, O&M, …) 09/11/2015 Functest in Depth, OPNFV Summit 2015 46

Lessons learned in Arno Need for stable labs some weeks before the release Need to run the tests on different PODs from CI Complex troubleshooting Functest cannot solve all problems 09/11/2015 Functest in Depth, OPNFV Summit 2015 47

Challenges … Different OpenStack installers Different SDN controllers … Many features Lots of combinations to be tested !! 09/11/2015 Functest in Depth, OPNFV Summit 2015 48

Summary First test project in OPNFV Validate a fresh OPNFV deployment Umbrella for testing projects Umbrella for feature projects We don’t bite, come to us with questions  09/11/2015 Functest in Depth, OPNFV Summit 2015 49

09/11/2015 Functest in Depth, OPNFV Summit 2015 50

Thank you! 09/11/2015 Functest in Depth, OPNFV Summit 2015 51