building a distributed build system at google scale
play

Building a Distributed Build System at Google Scale Aysylu - PowerPoint PPT Presentation

Sep 25, 2023 •158 likes •921 views

Building a Distributed Build System at Google Scale Aysylu Greenberg Aysylu Greenberg @aysylu22 Confidential + Proprietary Aysylu Greenberg @aysylu22 Confidential + Proprietary Building Distributed Build System at Google Scale

  1. Building a Distributed Build System at Google Scale Aysylu Greenberg

  2. Aysylu Greenberg @aysylu22 Confidential + Proprietary

  3. Aysylu Greenberg @aysylu22 Confidential + Proprietary

  4. Building Distributed Build System at Google Scale Confidential + Proprietary

  5. Building Distributed Build System at Google Scale Confidential + Proprietary

  6. WTH is "Google scale"? Confidential + Proprietary

  7. Scale ● Engineers: >30,000 developers in 40+ offices Confidential + Proprietary

  8. Scale ● Engineers: >30,000 developers in 40+ offices ● Commits: 15K by humans + 30K by robots per day Confidential + Proprietary

  9. Scale ● Engineers: >30,000 developers in 40+ offices ● Commits: 15K by humans + 30K by robots per day ● Source code: 2 billion LOC Confidential + Proprietary

  10. Scale ● Engineers: >30,000 developers in 40+ offices ● Commits: 15K by humans + 30K by robots per day ● Source code: 2 billion LOC ● Builds and tests: 5M per day through BuildRabbit Confidential + Proprietary

  11. Scale ● Engineers: >30,000 developers in 40+ offices ● Commits: 15K by humans + 30K by robots per day ● Source code: 2 billion LOC ● Builds and tests: 5M per day through BuildRabbit ● Petabytes of output artifacts Confidential + Proprietary

  12. Scale ● Engineers: >30,000 developers in 40+ offices ● Commits: 15K by humans + 30K by robots per day ● Source code: 2 billion LOC ● Builds and tests: 5M per day through BuildRabbit ● Petabytes of output artifacts ● 1 repository Confidential + Proprietary

  13. Working in One Repository Confidential + Proprietary

  14. Working in One Repository ● Linear revision history Confidential + Proprietary

  15. Working in One Repository ● Linear revision history ● Everything is cross-referenced Confidential + Proprietary

  16. Working in One Repository ● Linear revision history ● Everything is cross-referenced ● Components for library releases ○ = Git subtree or Git subcomponents to separate release from WIP versions Confidential + Proprietary

  17. Working in One Repository ● Linear revision history ● Everything is cross-referenced ● Components for library releases ● Repository of artifacts vs build from source: ○ Predictable, repeatable builds from source ○ Optimizations to avoid compiling same artifacts ○ Decouple each team's processes as much as possible Confidential + Proprietary

  18. Building Distributed Build System at Google Scale Confidential + Proprietary

  19. Building Distributed Build System at Google Scale Confidential + Proprietary

  20. Towards Distributed Build System Confidential + Proprietary

  21. Towards Distributed Build System Confidential + Proprietary

  22. Towards Distributed Build System http://opencv.org/ Confidential + Proprietary

  23. Towards Distributed Build System http://opencv.org/ Confidential + Proprietary

  24. Towards Distributed Build System http://opencv.org/ Confidential + Proprietary

  25. Build Scenario: Project with dependencies Find dependencies Build project with the dependencies Download build artifacts Confidential + Proprietary

  26. Test Scenario: Project with dependencies Find dependencies Build project with the dependencies Download build artifacts Run the test Get the results of the test Confidential + Proprietary

  27. Towards Distributed Build System http://opencv.org/ Confidential + Proprietary

  28. Towards Distributed Build System http://opencv.org/ Confidential + Proprietary

  29. Towards Distributed Build System http://opencv.org/ Confidential + Proprietary

  30. Towards Distributed Build System http://opencv.org/ Confidential + Proprietary

  31. Towards Distributed Build System http://opencv.org/ Confidential + Proprietary

  32. Towards Distributed Build System http://opencv.org/ Confidential + Proprietary

  33. Towards Distributed Build System http://opencv.org/ Confidential + Proprietary

  34. Towards Distributed Build System http://opencv.org/ Confidential + Proprietary

  35. Towards Distributed Build System http://opencv.org/ Confidential + Proprietary

  36. Towards Distributed Build System http://opencv.org/ Confidential + Proprietary

  37. Towards Distributed Build System http://opencv.org/ Confidential + Proprietary

  38. Towards Distributed Build System http://opencv.org/ Confidential + Proprietary

  39. Towards Distributed Build System http://opencv.org/ Confidential + Proprietary

  40. BuildRabbit: Distributed Build System Confidential + Proprietary

  41. BuildRabbit: Distributed Build System IN THE CLOUD Confidential + Proprietary

  42. What does BuildRabbit do? Confidential + Proprietary

  43. Continuous integration system Source System Release infrastructure BuildRabbit Blaze Google engineers & teams Build artifact Integration storage testing infrastructure Confidential + Proprietary

  44. Building Distributed Build System at Google Scale Confidential + Proprietary

  45. Building Distributed Build System at Google Scale Confidential + Proprietary

  46. Evolution of BuildRabbit: From push to pull Confidential + Proprietary

  47. Evolution of BuildRabbit: From Push to Pull ● Experimental project -> key piece of Google's developer infrastructure Confidential + Proprietary

  48. Evolution of BuildRabbit: From Push to Pull ● Experimental project -> key piece of Google's developer infrastructure ● Push model BuildRabbit Client Scheduler for BuildRabbit User Worker Confidential + Proprietary

  49. Evolution of BuildRabbit: From Push to Pull ● Experimental project -> key piece of Google's developer infrastructure ● Push model ● Pull model: build service Confidential + Proprietary

  50. Build Service Persistent BuildRabbit Queue Worker User Build Progress Info Build Artifacts RPC event stream Confidential + Proprietary

  51. Replacing Jet Engine In-Flight Confidential + Proprietary

  52. Replacing Jet Engine In- Flight Confidential + Proprietary

  53. Confidential + Proprietary

  54. BuildRabbit Client Scheduler for BuildRabbit User Worker Confidential + Proprietary

  55. BuildRabbit Client Scheduler for BuildRabbit User Worker Persistent BuildRabbit Queue Worker User Build Progress Info Build Artifacts Confidential + Proprietary

  56. BuildRabbit Client Scheduler for BuildRabbit User Worker Persistent BuildRabbit Queue Worker User Build Progress Info Build Artifacts Confidential + Proprietary

  57. Implementing Re-architected Build System Confidential + Proprietary

  58. Migrate backends first implementing re-architected build system Confidential + Proprietary

  59. Persistent BuildRabbit Queue Worker User Build Progress Info Build Artifacts RPC event stream Confidential + Proprietary

  60. Persistent BuildRabbit Queue Worker User Build Progress Info Build Artifacts RPC event stream Confidential + Proprietary

  61. Throw-away code is needed to prove identical functionality implementing re-architected build system Confidential + Proprietary

  62. Throw-away code is needed to prove identical functionality >>> Compatibility window is several months implementing re-architected build system Confidential + Proprietary

  63. Target launch-friendly clients before transitioning everyone implementing re-architected build system Confidential + Proprietary

  64. Persistent BuildRabbit Queue Worker User Build Progress Info Build Artifacts RPC event stream Confidential + Proprietary

  65. Persistent BuildRabbit Queue Worker User Build Progress Info Build Artifacts RPC event stream Confidential + Proprietary

  66. Decouple launch of services implementing re-architected build system Confidential + Proprietary

  67. Persistent BuildRabbit Queue Worker User Build Progress Info Build Artifacts RPC event stream Confidential + Proprietary

  68. Persistent BuildRabbit Queue Worker User Build Progress Info Build Artifacts RPC event stream Confidential + Proprietary

  69. Maximum visibility into system state implementing re-architected build system Confidential + Proprietary

  70. Practice before launch implementing re-architected build system Confidential + Proprietary

  71. Practice before launch, a lot implementing re-architected build system Confidential + Proprietary

  72. Practice before launch, a lot Have a solid rollback plan implementing re-architected build system Confidential + Proprietary

  73. References ● Why Google Stores Billions of Lines of Code in a Single Repository https://www. youtube.com/watch?v=W71BTkUbdqE Build artifacts storage http://google-engtools.blogspot.com/2011/10/build-in- ● cloud-distributing-build.html ● Distributed build actions http://google-engtools.blogspot.com/2011/09/build-in- cloud-distributing-build-steps.html Continuous integration testing http://dl.acm.org/citation.cfm?id=2635910 ● Confidential + Proprietary

  74. Gratitude Vince Noel Scott Zawalski Rob Siemborski BuildRabbit team Caitie McCaffrey David Greenberg Confidential + Proprietary

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Desired Properties in a Storage System (For building large-scale, geographically-distributed

Desired Properties in a Storage System (For building large-scale, geographically-distributed

Desired Properties in a Storage System (For building large-scale, geographically-distributed services) Jeff Dean Google Fellow jeff@google.com Desired File System Characteristics Single global namespace across many geographically dist.

225 views • 4 slides
How to Replace a Jet Engine of Your System In-Flight Aysylu Greenberg, Software Engineer, Google

How to Replace a Jet Engine of Your System In-Flight Aysylu Greenberg, Software Engineer, Google

How to Replace a Jet Engine of Your System In-Flight Aysylu Greenberg, Software Engineer, Google Aysylu Greenberg @aysylu22 Photo by imanka / CC BY Build System Story Towards Distributed Build System Towards Distributed Build System Towards

827 views • 46 slides
THE WAY WE USED TO BUILD THE WAY WE BUILD NOW WE CAN DO BETTER MCDONALDS HARMONY WITH

THE WAY WE USED TO BUILD THE WAY WE BUILD NOW WE CAN DO BETTER MCDONALDS HARMONY WITH

THE WAY WE USED TO BUILD THE WAY WE BUILD NOW WE CAN DO BETTER MCDONALDS HARMONY WITH SURROUNDINGS STREET W ALLS PLACEMENT ON THE SITE SCALE SCALE SCALE MASSING MASSING ROOF BUILDING FAADE Vertical Elements BUILDING FAADE

602 views • 34 slides
Building a Skyscraper with Legos: The Anatomy of a Distributed System Tyler McMullen

Building a Skyscraper with Legos: The Anatomy of a Distributed System Tyler McMullen

Building a Skyscraper with Legos: The Anatomy of a Distributed System Tyler McMullen @tbmcmullen ok, whats up? Lets build a distributed system! The Project 50+ Datacenters Thousands of bare metal servers up to 64 Servers per

1.05k views • 69 slides
Building Large-Scale Internet Services Jeff Dean jeff@google.com Plan for Today Googles

Building Large-Scale Internet Services Jeff Dean jeff@google.com Plan for Today Googles

Building Large-Scale Internet Services Jeff Dean jeff@google.com Plan for Today Googles computational environment hardware system software stack & its evolution Techniques for building large-scale systems decomposition

1.27k views • 80 slides
Build Build Build Build System building The process of compiling and linking software

Build Build Build Build System building The process of compiling and linking software

Build Build Build Build System building The process of compiling and linking software components into an executable system Different systems are built from different combinations of components Invariably supported by automated

557 views • 20 slides
PAT HELLAND AND ME HOW TO BUILD STATEFUL DISTRIBUTED APPLICATIONS THAT CAN SCALE ALMOST INFINITELY

PAT HELLAND AND ME HOW TO BUILD STATEFUL DISTRIBUTED APPLICATIONS THAT CAN SCALE ALMOST INFINITELY

PAT HELLAND AND ME HOW TO BUILD STATEFUL DISTRIBUTED APPLICATIONS THAT CAN SCALE ALMOST INFINITELY PAT HELLAND AND ME SEAN T. ALLEN VP OF ENGINEERING AT WALLAROO LABS MEMBER OF THE PONY CORE TEAM AUTHOR OF STORM APPLIED LOVER OF FRENCH

876 views • 49 slides
Workshop: Build a Distributed Serverless Application Viruual PyCon 2020 April 2020 Charles

Workshop: Build a Distributed Serverless Application Viruual PyCon 2020 April 2020 Charles

Workshop: Build a Distributed Serverless Application Viruual PyCon 2020 April 2020 Charles Engelke (engelke@google.com) Laurie White (lauriewhite@google.com) Google Cloud Developer Relations Slides and exercises online at

1.03k views • 64 slides
Distributed Relational Databases Thomas Schwarz, SJ Why? Parallelism is a simple way to

Distributed Relational Databases Thomas Schwarz, SJ Why? Parallelism is a simple way to

Distributed Relational Databases Thomas Schwarz, SJ Why? Parallelism is a simple way to scale up performance Example: Google web searches implemented via the Google File System Every file is stored at at least three servers (~2010)

1.48k views • 112 slides
Searching for Build Debt Managing Technical Debt at Google J.D. Morgenthaler, M. Gridnev, R.

Searching for Build Debt Managing Technical Debt at Google J.D. Morgenthaler, M. Gridnev, R.

Searching for Build Debt Managing Technical Debt at Google J.D. Morgenthaler, M. Gridnev, R. Sauciuc and S. Bhansali Google Confidential and Proprietary Google's Build System Overview Single, Monolithic Source Repository Single,

175 views • 13 slides
Secrets at Planet-Scale: Engineering the Internal Google Key Management System (KMS) Anvita

Secrets at Planet-Scale: Engineering the Internal Google Key Management System (KMS) Anvita

Secrets at Planet-Scale: Engineering the Internal Google Key Management System (KMS) Anvita Pandit Google LLC QCon San Francisco 2019, Nov 11-13 Anvita Pandit - Software engineer in Data Protection / Security and Privacy org in Google for

1.63k views • 78 slides
TensorFlow: A system for large-scale machine learning Martn Abadi et. al, 2016 Presented by

TensorFlow: A system for large-scale machine learning Martn Abadi et. al, 2016 Presented by

TensorFlow: A system for large-scale machine learning Martn Abadi et. al, 2016 Presented by Harrison Brown for R244 Background Originally built by Google engineers as successor to proprietary system for distributed training called

730 views • 13 slides
Building an IDE on top of a Build System The tale of a Haskell IDE How to write a compiler? +

Building an IDE on top of a Build System The tale of a Haskell IDE How to write a compiler? +

Building an IDE on top of a Build System The tale of a Haskell IDE How to write a compiler? + 1000s of papers, on every single aspect + A course at most universities + Blog posts galore How to write an IDE? Base it on a build system! The

484 views • 33 slides
Spanner A distributed database system Presented by Yue Xia Background - Developed by Google

Spanner A distributed database system Presented by Yue Xia Background - Developed by Google

Spanner A distributed database system Presented by Yue Xia Background - Developed by Google initially as a key-value storage system - Developers want traditional database features like query language - Evolved to a full featured SQL

718 views • 48 slides
Large-Scale Deep Learning with TensorFlow for Building Intelligent Systems Jeff Dean Google

Large-Scale Deep Learning with TensorFlow for Building Intelligent Systems Jeff Dean Google

Large-Scale Deep Learning with TensorFlow for Building Intelligent Systems Jeff Dean Google Brain Team g.co/brain In collaboration with many other people at Google We can now store and perform computation on large datasets, using things like

1.49k views • 127 slides
Hadoop Distributed File System (HDFS) 10/05/2018 1 HDFS Overview A distributed file system

Hadoop Distributed File System (HDFS) 10/05/2018 1 HDFS Overview A distributed file system

Hadoop Distributed File System (HDFS) 10/05/2018 1 HDFS Overview A distributed file system Built on the architecture of Google File System (GS) Shares a similar architecture to many other common distributed storage engines such as Amazon

1.72k views • 34 slides
RIDING THE JET STREAMS FUAD MALIKOV | HAZELCAST JAVA 8 STREAM API WHAT IS IT? JAVA 8 PRE JAVA

RIDING THE JET STREAMS FUAD MALIKOV | HAZELCAST JAVA 8 STREAM API WHAT IS IT? JAVA 8 PRE JAVA

RIDING THE JET STREAMS FUAD MALIKOV | HAZELCAST JAVA 8 STREAM API WHAT IS IT? JAVA 8 PRE JAVA 8 HISTORIC TIMES Collections Iterators For loops If-else statements WORD COUNT CODE Pre Java 8 Stream examples Get the unique

489 views • 19 slides
Societal Risks and the Law Stats/CS/PoliSci C79 Philip Stark, David Wagner, Jasjeet Sekhon,

Societal Risks and the Law Stats/CS/PoliSci C79 Philip Stark, David Wagner, Jasjeet Sekhon,

Societal Risks and the Law Stats/CS/PoliSci C79 Philip Stark, David Wagner, Jasjeet Sekhon, Cathryn Carson, Nicholas Jewell, Stephen Mahin Tuesday, January 22, 13 Which is riskier? (a) a 1000-mile car trip (b) flying 1000 miles by jet plane

432 views • 42 slides
Lecture 8 Aircraft Mission Text: Constraints analysis Introduction Concept of Constraints

Lecture 8 Aircraft Mission Text: Constraints analysis Introduction Concept of Constraints

Aircraft Mission Lecture 8 Aircraft Mission Text: Constraints analysis Introduction Concept of Constraints Motori Aeronautici Mathematical model Mar. 22, 2016 Aerodynamic Polar Throttle Lapse Flight phases Mission analysis Introduction

672 views • 46 slides
Question: Airplanes As you ride in a jet airplane, the clouds are passing you at 600 mph. The

Question: Airplanes As you ride in a jet airplane, the clouds are passing you at 600 mph. The

Airplanes 1 Airplanes 2 Question: Airplanes As you ride in a jet airplane, the clouds are passing you at 600 mph. The air just in front of one of the huge jet engine intake ducts is traveling much faster than 600 mph. much slower

512 views • 3 slides
Simulations on GPU Mohamed H. Aissa 1,2 1 Turbomachinery Department , Dr. Tom Verstraete 1 Von

Simulations on GPU Mohamed H. Aissa 1,2 1 Turbomachinery Department , Dr. Tom Verstraete 1 Von

Acceleration of Turbomachinery steady CFD Simulations on GPU Mohamed H. Aissa 1,2 1 Turbomachinery Department , Dr. Tom Verstraete 1 Von Karman Institute, Belgium 2 Delft Institute of Applied Mathematics, Prof. Cornelis Vuik 2 TU Delft, the

625 views • 37 slides
Data Structures and Besides reusing existing classes for new applications, OOP allows

Data Structures and Besides reusing existing classes for new applications, OOP allows

Inheritance Data Structures and Besides reusing existing classes for new applications, OOP allows definition of new Algorithms classes that extend existing ones to provide additional functionality Subclass inherits data fields and

392 views • 4 slides
Blazars: Can they (sometimes) be much faster than we thought? Markos Georganopoulos 1,2

Blazars: Can they (sometimes) be much faster than we thought? Markos Georganopoulos 1,2

Blazars: Can they (sometimes) be much faster than we thought? Markos Georganopoulos 1,2 Demosthenes Kazanas 2 Eric Perlman 3 Adam Higuera 1 1 University of Maryland, Baltimore County 2 NASA/Goddard Space Flight Center 3 Florida Institute of

503 views • 32 slides
Magnetic fields at the base of AGN jets: the case of M87 AGN M87

Magnetic fields at the base of AGN jets: the case of M87 AGN M87

Magnetic fields at the base of AGN jets: the case of M87 AGN M87 (Kogakuin U/NAOJ) 2017/12/21 @ Kagoshima U Contents l Introduction l Black

718 views • 45 slides

More recommend