Cluster management at Google with Borg - coping with scale 2015-11 - - PowerPoint PPT Presentation

▶
cluster management at google with borg coping with scale
SMART_READER_LITE
LIVE PREVIEW

Cluster management at Google with Borg - coping with scale 2015-11 - - PowerPoint PPT Presentation

Jan 13, 2023 15 likes •428 views

Cluster management at Google with Borg - coping with scale 2015-11 john wilkes / johnwilkes@google.com Principal Software Engineer Derived from EuroSys'15 paper (http://goo.gl/1C4nuo) Cluster management the system we internally call at

slide-1
SLIDE 1

Cluster management at Google with Borg - coping with scale

2015-11

john wilkes / johnwilkes@google.com Principal Software Engineer Derived from EuroSys'15 paper (http://goo.gl/1C4nuo)

slide-2
SLIDE 2

Cluster management at Google with Borg - coping with scale

2015-11

john wilkes / johnwilkes@google.com Principal Software Engineer Derived from EuroSys'15 paper (http://goo.gl/1C4nuo)

the system we internally call

slide-3
SLIDE 3

Borg contributors

Core: Abhishek Rai, Abhishek Verma, Andy Zheng, Ashwin Kumar, Ben Smith, Beng-Hong Lim, Bin Zhang, Bolu Szewczyk, Brad Strand, Brian Budge, Brian Grant, Brian Wickman, Chengdu Huang, Chris Colohan, Cliff Stein, Cynthia Wong, Daniel Smith, Dave Bort, David Oppenheimer, David Wall, Divyesh Shah, Dawn Chen, Eric Haugen, Eric Tune, Eric Wilcox, Ethan Solomita, Gaurav Dhiman, Geeta Chaudhry, Greg Roelofs, Grzegorz Czajkowski, James Eady, Jarek Kusmierek, Jaroslaw Przybylowicz, Jason Hickey, Javier Kohen, Jeff Dean, Jeremy Dion, Jeremy Lau, Jerzy Szczepkowski, Joe Hellerstein, John Wilkes, Jonathan Wilson, Joso Eterovic, Jutta Degener, Kai Backman, Kamil Yurtsever, Ken Ashcraft, Kenji Kaneda, Kevan Miller, Kurt Steinkraus, Leo Landa, Liza Fireman, Madhukar Korupolu, Maricia Scott, Mark Logan, Mark Vandevoorde, Markus Gutschke, Matt Sparks, Maya Haridasan, Michael Abd- El-Malek, Michael Kenniston, Ming-Yee Iu, Monika Henzinger, Mukesh Kumar, Nate Calvin, Onufry Wojtaszczyk, Olcan Sercinoglu, Paul Menage, Patrick Johnson, Pavanish Nirula, Pedro Valenzuela, Percy Liang, Piotr Witusowski, Praveen Kallakuri, Rafal Sokolowski, Rajmohan Rajaraman, Richard Gooch, Rishi Gosalia, Rob Radez, Robert Hagmann, Robert Jardine, Robert Kennedy, Rohit Jnagal, Roy Bryant, Rune Dahl, Scott Garriss, Scott Johnson, Sean Howarth, Sheena Madan, Smeeta Jalan, Stan Chesnutt, Temo Arobelidze, Tim Hockin, Todd Wang, Tomasz Blaszczyk, Tomasz Wozniak, Tomek Zielonka, Victor Marmol, Vish Kannan, Vrigo Gokhale, Walfredo Cirne, Walt Drummond, Weiran Liu, Xiaopan Zhang, Xiao Zhang, Ye Zhao, and Zohaib Maya. SRE: Adam Rogoyski, Alex Milivojevic, Anil Das, Cody Smith, Cooper Bethea, Folke Behrens, Matt Liggett, James Sanford, John Millikin, Matt Brown, Miki Habryn, Peter Dahl, Robert van Gent, Seppi Wilhelmi, Seth Hettich, Torsten Marek, and Viraj Alankar. BCL and borgcfg: Marcel van Lohuizen and Robert Griesemer. Reviewers: Christos Kozyrakis, Eric Brewer, Malte Schwarzkopf, and Tom Rodeheffer.

slide-4
SLIDE 4

http://www.google.com/about/datacenters/inside/locations/index.html

slide-5
SLIDE 5

http://googleasiapacific.blogspot.se/2015/06/growing-our-data-center-in-singapore.html

slide-6
SLIDE 6

Image by Connie Zhou

slide-7
SLIDE 7

job hello_world = { runtime = { cell = 'ic' } // Cell (cluster) to run in binary = '.../hello_world_webserver' // Program to run args = { port = '%port%' } // Command line parameters requirements = { // Resource requirements (optional) ram = 100M disk = 100M cpu = 0.1 } replicas = 5 // Number of tasks }

10000

User view

slide-8
SLIDE 8

User view

slide-9
SLIDE 9

What just happened?

web browsers BorgMaster link shard UI shard BorgMaster link shard UI shard BorgMaster link shard UI shard BorgMaster link shard UI shard Cell Scheduler borgcfg web browsers scheduler Borglet Borglet Borglet Borglet BorgMaster link shard read/UI shard Config file

persistent store (Paxos)

Binary

User view

slide-10
SLIDE 10

Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world!

Image by Connie Zhou

User view

Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world!

slide-11
SLIDE 11

User view

slide-12
SLIDE 12

task-eviction rates and causes

13

Failures

slide-13
SLIDE 13

Images by Connie Zhou

A 2000-machine service will have >10 task exits per day

This is not a problem: it's normal

Failures

slide-14
SLIDE 14

Advanced bin- packing algorithms

Experimental placement

  • f production VM

workload, July 2014

Efficiency

stranded resources available resources

  • ne

machine

slide-15
SLIDE 15

tasks per machine

Multiple applications per machine

CPI^2 paper, EuroSys 2013

Efficiency

slide-16
SLIDE 16

17

Sharing clusters between prod/batch helps

Segregating them would need more machines

Efficiency

shared cell (original) shared cell (compacted) non-prod load (compacted) prod-only load (compacted)

# machines

slide-17
SLIDE 17

# machines

18

Sharing clusters between prod/batch helps

Segregating them would need more machines

Efficiency

shared cell (original) shared cell (compacted) non-prod load (compacted) prod-only load (compacted)

  • verhead
slide-18
SLIDE 18

Waste

Sharing clusters between prod/batch helps

Segregating them would need more machines 15 production cells from a larger pool, omitting small

  • nes (<5000 machines)

19

Efficiency

slide-19
SLIDE 19

20

Efficiency

Smaller cells would need more machines

slide-20
SLIDE 20

Bucketing to next- largest power of 2 would need more machines

prod only, starting from 0.5 cores, 0.5GiB

21

Efficiency

slide-21
SLIDE 21

There are no

  • bvious resource

bucket sizes

  • cf. cloud VMs

22

nice round numbers gaming the system

Efficiency

slide-22
SLIDE 22

potentially reusable resources

Resource reclamation

23

Efficiency

time

limit: amount of resource requested usage: actual resource consumption reservation: estimate of future usage

slide-23
SLIDE 23

Resource reclamation could be more aggressive

Nov/Dec 2013

24

Efficiency

slide-24
SLIDE 24

Resource reclamation could be more aggressive

Nov/Dec 2013

25

Efficiency

slide-25
SLIDE 25

web browsers BorgMaster link shard UI shard BorgMaster link shard UI shard BorgMaster link shard UI shard BorgMaster link shard UI shard Cell Scheduler borgcfg web browsers scheduler Borglet Borglet Borglet Borglet BorgMaster link shard read/UI shard Config file

persistent store (Paxos)

A few other moving parts

slide-26
SLIDE 26

app

agent

master

job config

A few other moving parts

slide-27
SLIDE 27

app

agent

master

system config monitoring security accounting/planning binaries + data distribution job config storage

Diagram from an original by Cody Smith.

A few other moving parts

slide-28
SLIDE 28

app

agent master

system config monitoring security accounting/billing binaries + data distribution job config storage

A few other moving parts

Diagram from an original by Cody Smith.

slide-29
SLIDE 29

κυβερνήτης: pilot

  • r helmsman of a ship

http://kubernetes.io

Kubernetes

slide-30
SLIDE 30

Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent

Kubernetes

Web server Log roller

slide-31
SLIDE 31

Log roller Web server

Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent

Kubernetes master/scheduler

Pods

slide-32
SLIDE 32

FE FE FE FE FE BE BE BE BE BE BE BE BE BE Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent

Kubernetes master/scheduler

Labels

slide-33
SLIDE 33

FE FE FE FE FE BE BE BE BE BE BE BE BE BE Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent

Kubernetes master/scheduler

Label selectors

labels: role: frontend

slide-34
SLIDE 34

Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent

Kubernetes master/scheduler

FE FE FE FE FE BE BE BE BE BE BE BE BE BE

Label selectors

labels: role: frontend stage: production

slide-35
SLIDE 35

FE FE FE

replicas: 3 template: ... labels: role: frontend

Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent

Kubernetes - Master/Scheduler

Replica controller

slide-36
SLIDE 36

FE FE FE FE

replicas: 4 template: ... labels: role: frontend

Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent

Kubernetes - Master/Scheduler

Replica controller

slide-37
SLIDE 37

id: frontend-service port: 9000 labels: role: frontend

frontend-service FE FE FE FE Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent

Kubernetes - Master/Scheduler

Service

slide-38
SLIDE 38

Kubernetes

Direct Borg analogues:

  • Borg containers => Docker containers
  • alloc (task group) => pod (container group)
  • Borglet => Kubelet
  • persistent, declarative specs
  • reconciliation loops
slide-39
SLIDE 39

New / improved:

  • labels + label queries
  • service abstraction
  • composable microservices

○ replication controller ○ horizontal autoscaler

  • IP per pod

Kubernetes

slide-40
SLIDE 40

johnwilkes@google.com http://kubernetes.io

http://goo.gl/1C4nuo (Borg paper)

Images by Connie Zhou

Observations:

  • 1. Resiliency is achieved only by

ruthless attention to detail

a. ubiquitous software fault tolerance b. persistent, declarative specs

  • 2. We get efficiency by:

a. sharing resources b. reclaiming unused allocations

  • 3. Containers make users more

productive