Tempest: Integrated OpenStack Testing Matthew Treinish - PowerPoint PPT Presentation

Tempest: Integrated OpenStack Testing Matthew Treinish mtreinish@kortar.org May 12, 2014

OpenStack is Complicated

Testing before Tempest ◮ Every project had its own unit tests ◮ Some projects had functional tests ◮ There was nothing to test the whole picture

What is Tempest? ◮ The official OpenStack integration test suite ◮ Only black-box testing, only interacts with OpenStack REST APIs ◮ Over 2300 tests covering every incubated and integrated project ◮ Used as the gating tests for every integrated project

Tempest Growth ◮ Started in late Diablo to solve the lack of integrated testing ◮ Became a gating test suite in the Folsom cycle ◮ Was one of the 5 most active projects in Icehouse ◮ Testing with tempest is now a project graduation requirement ◮ Being used in refstack as part of defcore validation

Growing Pains ◮ Huge continued increase in the number of projects and tests ◮ The common code to handle all the APIs becomes more complex ◮ Managing individual tests becomes very difficult ◮ Total run time continues to grow ◮ Tempest configuration becomes more involved to handle all the services and configurations

Design Principles ◮ Should be able to run against any OpenStack Cloud from devstack to a public cloud ◮ Everything explicit ◮ Only use public interfaces (REST APIs) ◮ Should be self cleaning ◮ Self Testing

Basic Tempest Architecture

API Tests ◮ Directly test the OpenStack REST APIs ◮ Uses a unified tempest REST client ◮ Fall into 2 categories positive and negative ◮ Negative tests are starting to be automatically generated from api schemas

Scenario Tests ◮ Through path tests of functionality ◮ Designed to simulate a typical end user action workflow ◮ Use official python-*clients ◮ Ideally will involve integration points between projects

CLI Tests ◮ Test the official client CLIs ◮ Quick sanity checks of functionality in an OpenStack environment ◮ Only perform read-only operations ◮ Mostly a matter of convenience for gating

Stress Tests ◮ Use tempest to generate load on a cloud ◮ Dedicated scenarios for generating load ◮ Can specify any existing tempest test to also be a stress generator ◮ All scenario tests are automatically added as stress tests ◮ Run stress tests serially or in parallel looping for a set period of time

3rd Party and Unit Tests ◮ 3rd Party Tests ◮ Tests for in-tree non-OpenStack APIs ◮ Currently only tests Nova’s EC2 implementation ◮ Unit Tests ◮ Tempest eventually reached a size where its operations were too complex to take for granted ◮ Started late in Havana because of bugs in wrapper scripts ”super passing” the gate ◮ Unit testing coverage was expanded significantly during Icehouse

Parallel Tempest ◮ Tempest is capable of running it’s tests in parallel ◮ More closely simulates real load on a cloud ◮ Greatly improves speed of running tests: ◮ Serially gate runs took 1 hr with < 1000 tests ◮ Current gate runs are 40min to 1 hr with 2300 tests

Issue with single user in parallel

Tenant Isolation and Parallel Execution ◮ Tenant isolation enables running each test class in isolation ◮ When enabled each test class creates an isolated user and tenant ◮ Execute tests in parallel at the class level

Branchless Tempest ◮ Starting with Icehouse tempest will no longer have stable branches ◮ For gating purposes we will use master tempest against stable branch proposed commits ◮ All proposed tempest commits will run against master on the other projects and the currently supported stable releases (Icehouse forward) ◮ This should help us ensure greater api consistency between releases

Tempest Configuration ◮ Around 200 config options currently ◮ Intent of config file is to tell tempest what can be run ◮ What should be run is the domain of the test runner ◮ Sample config file documents every option ◮ Intended to be able to cover any OpenStack configuration ◮ Should have sane defaults (ie match project defaults) ◮ To setup a basic config file: ◮ Configuring only basic auth and connection info should be needed ◮ Also specify flavor and image ids to use for creating servers ◮ List which services are available and any required configuration for those services

Future Work ◮ Expansion of autogenerated negative tests ◮ More Unit test coverage ◮ Expansion of feature flags in config file ◮ Configuration file validation and generation tooling ◮ Possibly a new service for running tempest

Where to get more information ◮ Tempest Repository ◮ Tempest Documentation ◮ IRC #openstack-qa channel on Freenode

Questions?