Zuul as a build system lessons learned in the Tungsten Fabric infra - - PowerPoint PPT Presentation

▶
zuul as a build system
SMART_READER_LITE
LIVE PREVIEW

Zuul as a build system lessons learned in the Tungsten Fabric infra - - PowerPoint PPT Presentation

Apr 27, 2023 216 likes •761 views

Zuul as a build system lessons learned in the Tungsten Fabric infra Jarosaw ukow and ukasz ukasiewicz Who we are About admins of Tungsten Fabric CI/CB system started deploying Zuul v3 in November 2017 we work at

slide-1
SLIDE 1

lessons learned in the Tungsten Fabric infra Jarosław Łukow and Łukasz Łukasiewicz

Zuul as a build system

slide-2
SLIDE 2

About

Who we are

2

  • admins of Tungsten Fabric CI/CB system
  • started deploying Zuul v3 in November 2017
  • we work at CodiLime

○ DevOps, SDN, NFV, Cloud-Native services ○ 200 engineers, 48 270 coffees/year*

  • contacts:

○ diabelko: lukasz@codilime.com ○ jluk: jaroslaw.lukow@codilime.com * at least the espresso machines say so

slide-3
SLIDE 3

Agenda

  • about
  • build system
  • reusing Zuul jobs
  • testing jobs
  • cool to see in Zuul
slide-4
SLIDE 4

Intro

What is Tungsten Fabric

4

  • formerly known as OpenContrail
  • multicloud, multistack SDN solution
  • integrates with OpenStack, Kubernetes, OpenShift, VMware
slide-5
SLIDE 5

Intro

Project specifics

5

  • C, Go, Python
  • single build of all components (30 repos)
  • Android Repo tool
  • services deployed as containers
  • platforms:

○ CentOS (mostly) ○ RHEL ○ Windows Server

slide-6
SLIDE 6

Agenda

  • about
  • build system
  • reusing Zuul jobs
  • testing jobs
  • cool to see in Zuul
slide-7
SLIDE 7

Build system

The starting point - Jenkins CB system

7

  • separate from CI (which was running on Zuul 2.5 at that time)
  • different locations of dependencies
  • different scripts
  • different slave pool
  • single-job pipeline
slide-8
SLIDE 8

Build system

Then comes Zuul v3

8

  • upgrade CI from 2.5
  • accent on openness
  • unify CI, build and release pipelines
slide-9
SLIDE 9

Build system

The pipeline

9

  • compile and package
  • containerize
  • publish
slide-10
SLIDE 10

Build system

The pipeline

10

  • compile and package
  • containerize
  • publish
slide-11
SLIDE 11

Build system

The pipeline

11

slide-12
SLIDE 12

Build system

The pipeline

12

slide-13
SLIDE 13

Build system

The pipeline

13

slide-14
SLIDE 14

Build system

The pipeline

14

slide-15
SLIDE 15

Build system

The pipeline

15

slide-16
SLIDE 16

Build system

The pipeline

16

slide-17
SLIDE 17

Build system

The pipeline

17

slide-18
SLIDE 18

Build system

The pipeline

18

slide-19
SLIDE 19

Build system

Surroundings

19

  • mirrors

○ RPMs, DEBs ○ PyPI ○ Maven

  • DockerHub cache
slide-20
SLIDE 20

Build system

Builder VM images

20

  • minimal approach

○ OS base ○ Zuul SSH key

  • all the dependencies are installed during build
  • the devs were disappointed with the fact that it's not a way to cache builds
slide-21
SLIDE 21

Build system

Triggering

21

  • scheduled (periodic)
  • n every merge
  • n-demand

trigger: gerrit:

  • event: ref-updated

trigger: timer:

  • time: "0 7 * * *"

zuul enqueue-ref --ref refs/heads/master ...

slide-22
SLIDE 22

Build system

Triggering

22

  • scheduled (periodic)
  • n every merge
  • n-demand

trigger: gerrit:

  • event: ref-updated

trigger: timer:

  • time: "0 7 * * *"

zuul enqueue-ref --ref refs/heads/master ...

daily builds docs, third party packages retrying builds

slide-23
SLIDE 23

Build system

Our extensions

23

  • consecutive build numbers
  • dumping exact commit information
  • dumping information about artifacts
  • generating lists of changes included in builds (changelog)
slide-24
SLIDE 24

Build system

Our extensions

24

  • consecutive build numbers

SQL DB buildset id master buildset id R5.0 420 138 custom Ansible module

slide-25
SLIDE 25
slide-26
SLIDE 26

dumping exact commit information

Our extensions

26

{"review.opencontrail.org/Juniper/contrail-controller": { "revisions": { "current": "f5d22c6", "previous": "42c7316" } "changes": [{ "title": "Replicate BGP EVPN Type-1 Routes...", "timestamp": 1542144758, "author": {...}, "bugs": [], "sha": "7d24140f16b6d066f9802e0547b41deb2a846893", "message": "...", "change": { "number": 47647, "id": "I4387030ca62495afe949f78b5fc391049f4783d5" } }, ...

slide-27
SLIDE 27

Build system

Our extensions

27

  • dumping exact commit information
slide-28
SLIDE 28

Build system

Build pipeline meets ‘check’

28

  • uses the same jobs as periodic pipeline
  • sanity jobs use containers built in previous jobs
  • publishing artifacts at the end is not needed
slide-29
SLIDE 29

Build system

Build pipeline meets ‘check’

29

slide-30
SLIDE 30

Build system

Build pipeline meets ‘check’

30

slide-31
SLIDE 31

Build system

Build pipeline meets ‘check’

31

slide-32
SLIDE 32

Build system

Build pipeline meets ‘check’

32

slide-33
SLIDE 33

Build system

Build pipeline meets ‘check’

33

slide-34
SLIDE 34

Build system

Build pipeline meets ‘check’

34

slide-35
SLIDE 35

Agenda

  • about
  • build system
  • reusing Zuul playbooks
  • testing jobs
  • cool to see in Zuul
slide-36
SLIDE 36

Reusing Zuul playbooks

Original idea

36

  • jobs are already shared by the CI and Build jobs
  • perhaps they can also be used in developer environment
  • so… let’s create Zuul-agnostic playbooks and roles
slide-37
SLIDE 37

Reusing Zuul playbooks

Why

37

  • so developers can reproduce the CI environment with ‘one-click’
  • to save us some time
  • because it’s cool to reuse stuff
slide-38
SLIDE 38

Reusing Zuul playbooks

The Zuul job dilemma

38

  • reusable playbooks vs. convenient usage of variables
  • good ARA visibility vs. single "shell" entrypoint
slide-39
SLIDE 39

Reusing Zuul playbooks

Why it failed

39

  • run playbook has to do all the work
  • can’t leverage pre- and post- playbooks
  • too hard to draw a strict line between Ansible and Zuul
  • too hard to mock Zuul outside of Zuul (you don't want to parse config on your own)
slide-40
SLIDE 40

Reusing Zuul playbooks

Aftermath

40

  • packaging/building logic inside the code, instead of the CI
  • simple Makefiles
  • still visible in ARA
slide-41
SLIDE 41

Reusing Zuul playbooks

Aftermath

41

CB run playbook pre- playbooks

make target-list make $target

post- playbooks (logs, pkg upload) dev environment

make all

slide-42
SLIDE 42

Agenda

  • about
  • build system
  • reusing Zuul jobs
  • testing jobs
  • cool to see in Zuul
slide-43
SLIDE 43

Testing jobs

CI of CI

43

  • your CI jobs are stored in a repo as code, so…
  • you should test them like everything else
  • but, some things are not testable in Zuul (for a good reason)
  • you can take the risk or...
slide-44
SLIDE 44

Testing jobs

Ideas for testing jobs

44

  • setting all pipelines as post-review (‘disable security’)
  • separate development environment (Zuul, Gerrit, Nodepool)
  • Zuul on a laptop
  • unit testing roles
  • running copies/mocks of jobs
slide-45
SLIDE 45

Testing jobs

Mocking your jobs

45

  • review to an untrusted repo
  • secrets as variables (dummy values)
  • changing Ansible host
slide-46
SLIDE 46

Agenda

  • about
  • build system
  • reusing Zuul jobs
  • testing jobs
  • cool to see in Zuul
slide-47
SLIDE 47

Cool to see in Zuul

Matching executor with its cloud

47

slide-48
SLIDE 48

Cool to see in Zuul

Matching executor with its cloud

48

slide-49
SLIDE 49

Cool to see in Zuul

Matrix build definitions

49

  • job:

name: contrail-build-containers-centos7-newton parent: contrail-build-containers-base vars:

  • penstack_version: newton
  • job:

name: contrail-build-containers-centos7-ocata parent: contrail-build-containers-base vars:

  • penstack_version: ocata
  • job:

name: contrail-build-containers-centos7-queens parent: contrail-build-containers-base vars:

  • penstack_version: queens
  • project:

name: Juniper/contrail-analytics check: jobs:

  • contrail-build-containers-centos7-newton
  • contrail-build-containers-centos7-ocata
  • contrail-build-containers-centos7-queens
  • job:

name: contrail-build-containers-centos7-{openstack_version} parent: contrail-build-containers-base

  • project:

name: Juniper/contrail-analytics check: jobs:

  • contrail-build-containers-centos7-{openstack_version}:

vars:

  • penstack_version: newton
  • contrail-build-containers-centos7-{openstack_version}:

vars:

  • penstack_version: ocata
  • contrail-build-containers-centos7-{openstack_version}:

vars:

  • penstack_version: queens
slide-50
SLIDE 50

Wrapping up

slide-51
SLIDE 51

Zuul as a build system

Takeaways

51

  • Tungsten Fabric has a cool CI/CB system
  • how to handle build artifacts with Zuul
  • reusing your jobs is the key
  • you can test your jobs not-in-the-production
slide-52
SLIDE 52

Zuul as a build system

Future plans

52

  • continuous upgrade of Zuul
  • running build and unittest jobs inside containers instead of VMs
  • supercedent pipeline manager
slide-53
SLIDE 53

Thank you