Building a Distributed Build System at Google Scale Aysylu - - PowerPoint PPT Presentation

▶
building a distributed build system at google scale
SMART_READER_LITE
LIVE PREVIEW

Building a Distributed Build System at Google Scale Aysylu - - PowerPoint PPT Presentation

Sep 25, 2023 158 likes •924 views

Building a Distributed Build System at Google Scale Aysylu Greenberg Aysylu Greenberg @aysylu22 Confidential + Proprietary Aysylu Greenberg @aysylu22 Confidential + Proprietary Building Distributed Build System at Google Scale

slide-1
SLIDE 1

Building a Distributed Build System at Google Scale

Aysylu Greenberg

slide-2
SLIDE 2

Confidential + Proprietary

Aysylu Greenberg

@aysylu22

slide-3
SLIDE 3

Confidential + Proprietary

Aysylu Greenberg

@aysylu22

slide-4
SLIDE 4

Confidential + Proprietary

Building Distributed Build System at Google Scale

slide-5
SLIDE 5

Confidential + Proprietary

Building Distributed Build System at Google Scale

slide-6
SLIDE 6

Confidential + Proprietary

WTH is "Google scale"?

slide-7
SLIDE 7

Confidential + Proprietary

Scale

  • Engineers: >30,000 developers in 40+ offices
slide-8
SLIDE 8

Confidential + Proprietary

Scale

  • Engineers: >30,000 developers in 40+ offices
  • Commits: 15K by humans + 30K by robots per day
slide-9
SLIDE 9

Confidential + Proprietary

Scale

  • Engineers: >30,000 developers in 40+ offices
  • Commits: 15K by humans + 30K by robots per day
  • Source code: 2 billion LOC
slide-10
SLIDE 10

Confidential + Proprietary

Scale

  • Engineers: >30,000 developers in 40+ offices
  • Commits: 15K by humans + 30K by robots per day
  • Source code: 2 billion LOC
  • Builds and tests: 5M per day through BuildRabbit
slide-11
SLIDE 11

Confidential + Proprietary

Scale

  • Engineers: >30,000 developers in 40+ offices
  • Commits: 15K by humans + 30K by robots per day
  • Source code: 2 billion LOC
  • Builds and tests: 5M per day through BuildRabbit
  • Petabytes of output artifacts
slide-12
SLIDE 12

Confidential + Proprietary

Scale

  • Engineers: >30,000 developers in 40+ offices
  • Commits: 15K by humans + 30K by robots per day
  • Source code: 2 billion LOC
  • Builds and tests: 5M per day through BuildRabbit
  • Petabytes of output artifacts
  • 1 repository
slide-13
SLIDE 13

Confidential + Proprietary

Working in One Repository

slide-14
SLIDE 14

Confidential + Proprietary

  • Linear revision history

Working in One Repository

slide-15
SLIDE 15

Confidential + Proprietary

  • Linear revision history
  • Everything is cross-referenced

Working in One Repository

slide-16
SLIDE 16

Confidential + Proprietary

  • Linear revision history
  • Everything is cross-referenced
  • Components for library releases

○ = Git subtree or Git subcomponents to separate release from WIP versions

Working in One Repository

slide-17
SLIDE 17

Confidential + Proprietary

  • Linear revision history
  • Everything is cross-referenced
  • Components for library releases
  • Repository of artifacts vs build from source:

○ Predictable, repeatable builds from source ○ Optimizations to avoid compiling same artifacts ○ Decouple each team's processes as much as possible

Working in One Repository

slide-18
SLIDE 18

Confidential + Proprietary

Building Distributed Build System at Google Scale

slide-19
SLIDE 19

Confidential + Proprietary

Building Distributed Build System at Google Scale

slide-20
SLIDE 20

Confidential + Proprietary

Towards Distributed Build System

slide-21
SLIDE 21

Confidential + Proprietary

Towards Distributed Build System

slide-22
SLIDE 22

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

slide-23
SLIDE 23

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

slide-24
SLIDE 24

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

slide-25
SLIDE 25

Confidential + Proprietary

Build Scenario:

Project with dependencies Find dependencies Build project with the dependencies Download build artifacts

slide-26
SLIDE 26

Confidential + Proprietary

Test Scenario:

Project with dependencies Find dependencies Build project with the dependencies Download build artifacts Run the test Get the results of the test

slide-27
SLIDE 27

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

slide-28
SLIDE 28

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

slide-29
SLIDE 29

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

slide-30
SLIDE 30

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

slide-31
SLIDE 31

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

slide-32
SLIDE 32

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

slide-33
SLIDE 33

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

slide-34
SLIDE 34

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

slide-35
SLIDE 35

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

slide-36
SLIDE 36

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

slide-37
SLIDE 37

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

slide-38
SLIDE 38

Confidential + Proprietary

Towards Distributed Build System

http://opencv.org/

slide-39
SLIDE 39

Confidential + Proprietary

Towards Distributed Build System

http://opencv.org/

slide-40
SLIDE 40

Confidential + Proprietary

BuildRabbit: Distributed Build System

slide-41
SLIDE 41

Confidential + Proprietary

BuildRabbit: Distributed Build System

IN THE CLOUD

slide-42
SLIDE 42

Confidential + Proprietary

What does BuildRabbit do?

slide-43
SLIDE 43

Confidential + Proprietary

Continuous integration system BuildRabbit Blaze Source System Google engineers & teams Release infrastructure Integration testing infrastructure Build artifact storage

slide-44
SLIDE 44

Confidential + Proprietary

Building Distributed Build System at Google Scale

slide-45
SLIDE 45

Confidential + Proprietary

Building Distributed Build System at Google Scale

slide-46
SLIDE 46

Confidential + Proprietary

Evolution of BuildRabbit: From push to pull

slide-47
SLIDE 47

Confidential + Proprietary

Evolution of BuildRabbit: From Push to Pull

  • Experimental project -> key piece of Google's

developer infrastructure

slide-48
SLIDE 48

Confidential + Proprietary

Evolution of BuildRabbit: From Push to Pull

  • Experimental project -> key piece of Google's

developer infrastructure

  • Push model

Client for User

BuildRabbit Scheduler BuildRabbit Worker

slide-49
SLIDE 49

Confidential + Proprietary

Evolution of BuildRabbit: From Push to Pull

  • Experimental project -> key piece of Google's

developer infrastructure

  • Push model
  • Pull model: build service
slide-50
SLIDE 50

Confidential + Proprietary

Build Service

User Persistent Queue BuildRabbit Worker Build Artifacts Build Progress Info RPC event stream

slide-51
SLIDE 51

Confidential + Proprietary

Replacing Jet Engine In-Flight

slide-52
SLIDE 52

Confidential + Proprietary

Replacing Jet Engine In- Flight

slide-53
SLIDE 53

Confidential + Proprietary

slide-54
SLIDE 54

Confidential + Proprietary

Client for User

BuildRabbit Scheduler BuildRabbit Worker

slide-55
SLIDE 55

Confidential + Proprietary

Client for User

BuildRabbit Scheduler BuildRabbit Worker User Persistent Queue BuildRabbit Worker Build Artifacts Build Progress Info

slide-56
SLIDE 56

Confidential + Proprietary

Client for User

BuildRabbit Scheduler BuildRabbit Worker User Persistent Queue BuildRabbit Worker Build Artifacts Build Progress Info

slide-57
SLIDE 57

Confidential + Proprietary

Implementing Re-architected Build System

slide-58
SLIDE 58

Confidential + Proprietary

Migrate backends first

implementing re-architected build system

slide-59
SLIDE 59

Confidential + Proprietary

User Persistent Queue BuildRabbit Worker Build Artifacts Build Progress Info RPC event stream

slide-60
SLIDE 60

Confidential + Proprietary

User Persistent Queue BuildRabbit Worker Build Artifacts Build Progress Info RPC event stream

slide-61
SLIDE 61

Confidential + Proprietary

Throw-away code is needed to prove identical functionality

implementing re-architected build system

slide-62
SLIDE 62

Confidential + Proprietary

Throw-away code is needed to prove identical functionality

>>> Compatibility window is several months

implementing re-architected build system

slide-63
SLIDE 63

Confidential + Proprietary

Target launch-friendly clients before transitioning everyone

implementing re-architected build system

slide-64
SLIDE 64

Confidential + Proprietary

User Persistent Queue BuildRabbit Worker Build Artifacts Build Progress Info RPC event stream

slide-65
SLIDE 65

Confidential + Proprietary

User Persistent Queue BuildRabbit Worker Build Artifacts Build Progress Info RPC event stream

slide-66
SLIDE 66

Confidential + Proprietary

Decouple launch of services

implementing re-architected build system

slide-67
SLIDE 67

Confidential + Proprietary

User Persistent Queue BuildRabbit Worker Build Artifacts Build Progress Info RPC event stream

slide-68
SLIDE 68

Confidential + Proprietary

User Persistent Queue BuildRabbit Worker Build Artifacts Build Progress Info RPC event stream

slide-69
SLIDE 69

Confidential + Proprietary

Maximum visibility into system state

implementing re-architected build system

slide-70
SLIDE 70

Confidential + Proprietary

Practice before launch

implementing re-architected build system

slide-71
SLIDE 71

Confidential + Proprietary

Practice before launch, a lot

implementing re-architected build system

slide-72
SLIDE 72

Confidential + Proprietary

Practice before launch, a lot Have a solid rollback plan

implementing re-architected build system

slide-73
SLIDE 73

Confidential + Proprietary

References

  • Why Google Stores Billions of Lines of Code in a Single Repository https://www.

youtube.com/watch?v=W71BTkUbdqE

  • Build artifacts storage http://google-engtools.blogspot.com/2011/10/build-in-

cloud-distributing-build.html

  • Distributed build actions http://google-engtools.blogspot.com/2011/09/build-in-

cloud-distributing-build-steps.html

  • Continuous integration testing http://dl.acm.org/citation.cfm?id=2635910
slide-74
SLIDE 74

Confidential + Proprietary

Gratitude

Vince Noel Scott Zawalski Rob Siemborski BuildRabbit team Caitie McCaffrey David Greenberg

slide-75
SLIDE 75