Upstream CI/CD tools for your production clouds That can operate - - PowerPoint PPT Presentation

▶
upstream ci cd tools for your production clouds
SMART_READER_LITE
LIVE PREVIEW

Upstream CI/CD tools for your production clouds That can operate - - PowerPoint PPT Presentation

Jul 02, 2023 227 likes •664 views

Upstream CI/CD tools for your production clouds That can operate 1,000+ test cycles in a day Slide URL: https://goo.gl/aZ3Tq3 Masayuki Igawa Core in Tempest, openstack-health, subunit2sql 2 Ghanshyam Mann Core in Tempest 3 Kenichi

slide-1
SLIDE 1

Upstream CI/CD tools for your production clouds

That can operate 1,000+ test cycles in a day

Slide URL: https://goo.gl/aZ3Tq3

slide-2
SLIDE 2

Masayuki Igawa Core in Tempest,

  • penstack-health,

subunit2sql

2

slide-3
SLIDE 3

Ghanshyam Mann Core in Tempest

3

slide-4
SLIDE 4

Ken’ichi Ohmichi QA PTL, Core in Nova

4

slide-5
SLIDE 5

Agenda

1. Introduction 2. Overview of QA tools 3. How to use QA tools for productions 4. Future works 5. Demo

5

slide-6
SLIDE 6

Introduction

Why we are here

  • We are working for OpenStack quality as upstream QA members
  • There are many tools for testing/debugging for the quality

What will you learn

  • How to keep/improve the quality at upstream development
  • How to use these tools for production clouds

6

slide-7
SLIDE 7

What’s the QA in OpenStack?

7

slide-8
SLIDE 8

Mission Statement of QA project

Develop, maintain, and initiate tools and plans to ensure the upstream stability and quality of OpenStack, and its release readiness at any point during the release cycle.

8

slide-9
SLIDE 9

Overview of gate tests

Every proposed code needs to be passed via multiple gate tests

  • pep8 (coding style)
  • python 27/34 (unit test)
  • gate-tempest-dsvm (integration test)

1,000+ gate test cycles run in most working days

  • Each gate test cycle consists of 1,000+ tests
  • That means 1,000,000+ tests run in a day

9

slide-10
SLIDE 10

Overview of upstream CI/CD

Testing the proposed code on temp env with the CI/CD

devstack-gate Virtual machine Node pool devstack Tempest logs gerrit developer

  • 1. Propose

code

  • 2. Kick
  • 4. Kick
  • 3. Allocate from pool
  • 5. Configure
  • 6. Deploy with

proposed code

  • 7. Test
  • 8. Store result and logs
  • 9. Developer checks

result on gerrit

QA projects

10

slide-11
SLIDE 11

Overview of QA tools

11

slide-12
SLIDE 12

QA tools

12

Testing: Tempest, os-testr, Grenade, Bashate, Hacking OpenStack specific: OpenStack-health, subunit2sql, elastic-recheck, DevStack, stackviz Generic: ElasticSearch/Logstash/Kibana, Gerrit, Jenkins, Graphite

https://wiki.openstack.org/wiki/QA

slide-13
SLIDE 13

QA tools

13

Run tests: Tempest Checks Results: OpenStack Health Dashboard Debug Failure: ElasticSearch/Logstash/ Kibana

slide-14
SLIDE 14

Tempest - OpenStack Integration Test Suite

  • This is a set of integration tests to be run against a live

OpenStack cluster.

  • Tempest is designed to be useful for a large number of

different environments. This includes being useful for gating commits to OpenStack core projects and used for testing the production Cloud.

14

slide-15
SLIDE 15

Tempest - Test types

  • API tests

○ API tests are validation tests for the OpenStack APIs. ○ Make sure stability and backward compatibility of APIs.

  • Scenario tests

○ Scenario tests are complex “through path” tests for OpenStack functionality. ○ Scenario tests interact with many services and tests cross projects features.

  • Stress tests - Deprecated, will be removed in Newton

○ Stress tests are designed to stress an OpenStack environment by running a high workload against it and seeing what breaks.

15

slide-16
SLIDE 16

Tempest - External Plugin

  • Tempest provides an external test

plugin interface to enables projects to implement tempest like tests and run those as part of tempest run. ~30 plugins

16

slide-17
SLIDE 17

OpenStack-Health

is a dashboard for visualizing test results of OpenStack CI jobs.

17

http://status.openstack.org/openstack-health/

slide-18
SLIDE 18

What is the OpenStack-Health?

18

Why do we need this? BigTents, numerous projects and test jobs, notify to ML doesn’t work Features Provide a dashboard :Graphs, Colorful table for analyzing our “BIG-DATA” Technically

OpenStack-Health

AngularJS 1(FrontEnd) + Flask(API server)

subunit2sql API + + Database

slide-19
SLIDE 19
  • Context color table: easy to understand the status
  • RSS feed: Alternative notification
  • Graph: “Gauge -> Bar” : space effective

OpenStack-Health: Recent Topics

19

slide-20
SLIDE 20

OpenStack-Health: Recent Topics (cont.)

Recent Topics

  • Regex Search filter: specify multiple projects (e.g. nova|glance|cinder…)
  • Pagination

20

slide-21
SLIDE 21

OpenStack-Health - Data flow

21

DataBase subunit2sql Subunit stream (Tempest, unit tests, etc.) Gearman worker O-H API server subunit2sql

slide-22
SLIDE 22

Kibana / Elasticsearch / Logstash

Log management system We are using them for searching similar cases to debug gate problems

debugger Kibana (logstash.openstack.org)

a) Search cases

Elasticsearch Logstash Gearman worker

  • 1. Fetch logs of gate test

and send them to Logstash

  • 2. Filter unnecessary logs out

and convert into useful format

  • 3. Store logs with indexing

b) Search logs

22

slide-23
SLIDE 23

How to use QA tools for productions

23

slide-24
SLIDE 24

How to use Tempest

2 Ways to Use Tempest

24

Tempest as a System Installed Program Tempest Source Code Repo

slide-25
SLIDE 25

How to use Tempest(cont.)

Install Tempest as system program:

25

$ git clone http://git.openstack.org/openstack/tempest $ pip install tempest/

/etc/tempest <prefix>/etc/tempest

slide-26
SLIDE 26

How to use Tempest(cont.)

Set up separate working directory

26

$ tempest init Cloud_01_Test $ tempest init Cloud_02_Test

slide-27
SLIDE 27

How to use Tempest(cont.)

Tempest Source Code Repo:

27

$ git clone http://git.openstack.org/openstack/tempest $ cd tempest

slide-28
SLIDE 28

How to use Tempest: Set up Config File

Tempest Configurations:

28

~80 config options

(Tempest & 6 Core Projects.)

+ Additional for other projects tests

(Tempest tree or in plugin).

slide-29
SLIDE 29

How to use Tempest: Set up Config File

Tempest Configuration Sample File.

$ tox –e genconfig OR $ oslo-config-generator --config-file etc/config-generator.tempest.conf

  • -output-file etc/tempest.conf

Best Reference for tempest.conf: Devstack generated one.

29

slide-30
SLIDE 30

How to use Tempest: Set up Config File(cont.)

30

Auth Configuration:

  • test_accounts_file
  • Use_dynamic_credentials

○ Admin_username ○ Admin_project_name ○ Admin_password ○ Admin_domain_name ○ Create_isolated_networks ○ default_credentials_domain_name

  • ...
slide-31
SLIDE 31

How to use Tempest: Set up Config File(cont.)

31

Identity Configuration:

  • uri
  • uri_v3
  • auth_version
  • catalog_type
  • region
  • v2_admin_endpoint_type
  • v2_public_endpoint_type
  • v3_endpoint_type
  • ...
slide-32
SLIDE 32

How to use Tempest: Set up Config File

Example: Compute Configuration

  • catalog_type
  • region
  • endpoint_type
  • api_extensions
  • image_ref
  • flavor_ref
  • resize
  • …..

32

slide-33
SLIDE 33

How to use Tempest:

All Set to Hit the Cloud

33

slide-34
SLIDE 34

How to use Tempest: Run Tests

$ testr run <test path>

tempest.api.compute.servers. test_servers.serversTestJSON. test_create_server_with_admin_password

$ testr run <test idempotent_id> b92d5ec7-b1dd-44a2-87e4-45e888c46ef0 $ testr run <test directory> tempest.api.compute $ testr run --parallel

34

slide-35
SLIDE 35

OpenStack-Health - Basic installation

  • Setup subunit2sql & Store your test data to your database

Pre condition: Setup your Database(e.g. dbname:subunit, username:subunit, pass:pass)

$ git clone git://git.openstack.org/openstack-infra/subunit2sql $ subunit2sql-db-manage --database-connection mysql://subunit:pass@127. 0.0.1/subunit upgrade head $ subunit2sql --database-connection mysql://subunit:pass@127.0.0.1/subunit SUBUNIT-V2-FILE

35

Usage: https://github.com/openstack-infra/subunit2sql#usage NOTE: * subunit2sql only takes a subunit v2 stream/file. So if you have only v1, you can convert it with using the subunit-1to2. e.g. cat v1-file | subunit-1to2 | subunit2sql --database-connection mysql://subunit:pass@127. 0.0.1/subunit * You may need to integrate subunit2sql with CI tool like Jenkins.

Must Read README.rst

slide-36
SLIDE 36

OpenStack-Health - Basic installation

  • Setup openstack-health

$ git clone git://git.openstack.org/openstack/openstack-health API Server: $ pip install -r requirements.txt (in virtualenv) Frontend:

$ sudo apt-get install nodejs npm nodejs-legacy $ sudo npm -g install npm@2 $ sudo npm -g config set prefix /usr/local $ sudo npm -g install npm $ sudo npm -g install gulp $ npm install

  • Run (in production)

API Server: $ sudo etc/openstack-health-api.conf /etc/openstack-health.conf $ uwsgi -s /tmp/uwsgi.sock --module openstack_health.api --callable app --http :5000 Frontend: $ gulp prod; (DEPLOY “build/*” to your env such as Apache)

36

Must Read README.rst

[default] db_uri = mysql+pymysql://query: query@logstash.openstack.

  • rg/subunit2sql

ignored_run_metadata_keys = build_change build_master build_node :

slide-37
SLIDE 37

Future Works

37

slide-38
SLIDE 38

Future work: Tempest

38

Resource config Move Common Interfaces to tempest/lib as stable interface Document for tempest plugin interface Extend plugin interface Cover API Microversion tests ...

slide-39
SLIDE 39

Future work: OpenStack-Health

39

Improve Data limitation: Detect/Store infra failure, check queue (not only gate/periodic) UI improvement: Improve messy/confusing UI “I accessed the o-h site, but didn’t understand it. So I don’t use it anymore” :( Framework update (AngularJS 1 -> 2)

slide-40
SLIDE 40

Demo

40

slide-41
SLIDE 41

Demo

41

  • Tempest
  • OpenStack-Health
slide-42
SLIDE 42

Demo: How to find necessary patches

42

1. Run Tempest 2. If failure, find Tempest change log which has added failed test 3. Find “Related-Bug” or something on the commit message 4. Find the bug report on OpenStack LP 5. Find the patch of the component patch which closed the bug E.g: test_create_servers_on_different_hosts_with_list_of_servers The bug report: https://bugs.launchpad.net/nova/+bug/1521928

slide-43
SLIDE 43

Useful Links

  • #openstack-qa on Freenode
  • penstack-dev@lists.openstack.org

Tempest

  • https://git.openstack.org/openstack/tempest

OpenStack-Health

  • http://status.openstack.org/openstack-health/
  • https://git.openstack.org/openstack-infra/subunit2sql
  • https://git.openstack.org/openstack/openstack-health
  • https://bugs.launchpad.net/openstack-health
  • OpenStack-Health Dashboard and Dealing with Data from the Gate: https://youtu.be/0__tD88oAlo

43