Building a Skyscraper with Legos: The Anatomy of a Distributed - - PowerPoint PPT Presentation

▶
building a skyscraper with legos
SMART_READER_LITE
LIVE PREVIEW

Building a Skyscraper with Legos: The Anatomy of a Distributed - - PowerPoint PPT Presentation

Dec 03, 2022 346 likes •1.05k views

Building a Skyscraper with Legos: The Anatomy of a Distributed System Tyler McMullen @tbmcmullen ok, whats up? Lets build a distributed system! The Project 50+ Datacenters Thousands of bare metal servers up to 64 Servers per

slide-1
SLIDE 1

Building a Skyscraper with Legos:

The Anatomy

  • f a Distributed

System

slide-2
SLIDE 2

Tyler McMullen

@tbmcmullen

slide-3
SLIDE 3
slide-4
SLIDE 4
  • k, what’s up?
slide-5
SLIDE 5

Let’s build a distributed system!

slide-6
SLIDE 6

The Project

slide-7
SLIDE 7
slide-8
SLIDE 8
slide-9
SLIDE 9
slide-10
SLIDE 10
slide-11
SLIDE 11
slide-12
SLIDE 12
slide-13
SLIDE 13
slide-14
SLIDE 14
slide-15
SLIDE 15
slide-16
SLIDE 16
slide-17
SLIDE 17
slide-18
SLIDE 18

50+ Datacenters Thousands of bare metal servers up to 64 Servers per Datacenter Ten of Thousands of Origin Servers

slide-19
SLIDE 19

Recap

slide-20
SLIDE 20

Still with me?

slide-21
SLIDE 21

One Possible Solution

slide-22
SLIDE 22
slide-23
SLIDE 23

(Too) Strong Consistency

slide-24
SLIDE 24

Eventual Consistency

slide-25
SLIDE 25

Forward Progress

slide-26
SLIDE 26
slide-27
SLIDE 27

Ownership

slide-28
SLIDE 28
slide-29
SLIDE 29

Rendezvous Hashing

slide-30
SLIDE 30
slide-31
SLIDE 31

h(Sn,O) = Wn

hash function we decided upon set of live servers

  • rigin server we’re

deciding on the owner of the weight or priority

slide-32
SLIDE 32

priorities(O1) = [S2, S3, S1, S4, …]

slide-33
SLIDE 33

priorities(O2) = [S4, S2, S3, S1, …]

slide-34
SLIDE 34

Failure Detection

slide-35
SLIDE 35
slide-36
SLIDE 36
slide-37
SLIDE 37

u up? 1 2 3 4 5

✅

slide-38
SLIDE 38

1 2 3 4 5

✅

slide-39
SLIDE 39

1 2 3 4 5

❌

slide-40
SLIDE 40

Memberlist

slide-41
SLIDE 41

Gossip

slide-42
SLIDE 42

Push and Pull

slide-43
SLIDE 43

Convergence

slide-44
SLIDE 44

Causality

slide-45
SLIDE 45

Burgers happened-before Calzone Burgers happened-before Daal

Jack Jill Arugula Burgers Calzone Daal

Arugula happened-before Burgers Calzone is-concurrent-with Daal

slide-46
SLIDE 46

Lattices

slide-47
SLIDE 47
slide-48
SLIDE 48

Causality

slide-49
SLIDE 49

Version Vectors

slide-50
SLIDE 50

S1 S2 S3 [1,0,0] [1,0,0] [0,0,1] [1,1,0] [1,2,1] [1,1,1]

slide-51
SLIDE 51

[0,0,0] [1,0,0] [0,1,0] [0,0,1] [1,1,0] [1,1,1] [1,1,2] [2,1,1]

slide-52
SLIDE 52

Coordination-free Distributed Map

slide-53
SLIDE 53

Version Delta Merge

slide-54
SLIDE 54

Δ-CRDT Map

slide-55
SLIDE 55

type SharedMap struct { storage map[Key]SharedMapRecord v clock.VersionVector } type SharedMapRecord struct { value Value dot clock.VVDot }

slide-56
SLIDE 56

Send_Version: Send our Version Vector.

slide-57
SLIDE 57

Received_Version(V): For each record(R) in our map: If (V happened-before R.Dot) OR (V is-concurrent-with R.Dot): Add R to Delta. Send Delta.

slide-58
SLIDE 58

Received_Delta(D): V = Our Version Vector For each record(R) in D: If R.Dot happened-before V: Skip it. R’ = Local Record If R’.Dot happened-before D.Version: Merge it. R and R’ are concurrent: ✨

slide-59
SLIDE 59

✨ = Rendesvous

slide-60
SLIDE 60

Delta-state CRDT Map

slide-61
SLIDE 61
slide-62
SLIDE 62

Ok, why?

slide-63
SLIDE 63

Edge Compute

slide-64
SLIDE 64

Coordination-free Distributed Systems

slide-65
SLIDE 65

Single System Image

slide-66
SLIDE 66

We posit that striving for distributed systems that provide “single system image” semantics is fundamentally flawed and at odds with how systems

  • perate in the physical world.
slide-67
SLIDE 67

We need new metaphors.

slide-68
SLIDE 68

We need new intuition.

slide-69
SLIDE 69

Thank You.

@tbmcmullen