Distributed Systems in Practice, in Theory Aysylu Greenberg June - - PowerPoint PPT Presentation

▶
distributed systems in practice in theory
SMART_READER_LITE
LIVE PREVIEW

Distributed Systems in Practice, in Theory Aysylu Greenberg June - - PowerPoint PPT Presentation

Aug 26, 2022 206 likes •1.33k views

Distributed Systems in Practice, in Theory Aysylu Greenberg June 14, 2016 How I got into reading papers as a practitioner in industry Computer Science Research In Distributed Systems Industry Operating systems research Operating systems

slide-1
SLIDE 1

Aysylu Greenberg June 14, 2016

Distributed Systems in Practice, in Theory

slide-2
SLIDE 2

How I got into reading papers as a practitioner in industry

slide-3
SLIDE 3

Computer Science Research In Distributed Systems Industry

slide-4
SLIDE 4

Operating systems research

slide-5
SLIDE 5

Operating systems research

slide-6
SLIDE 6

Operating systems research Concurrency

slide-7
SLIDE 7

Operating systems research Concurrency Concurrency primitives: mutex & semaphore

slide-8
SLIDE 8

Operating systems research Concurrency Concurrency primitives: mutex & semaphore Processes execute at different speeds

slide-9
SLIDE 9

Time in distributed systems

https://www.flickr.com/photos/national_archives_of_norway/6263353228

slide-10
SLIDE 10

Time in distributed systems

slide-11
SLIDE 11

Time in distributed systems Pipelining

slide-12
SLIDE 12

1980

slide-13
SLIDE 13

1980

slide-14
SLIDE 14

Internet 1980

slide-15
SLIDE 15

Internet Distributed consensus 1980

slide-16
SLIDE 16

Internet Distributed consensus 1980

slide-17
SLIDE 17

Internet Distributed consensus 1980

slide-18
SLIDE 18

Paxos Internet Distributed consensus 1980

slide-19
SLIDE 19

Reconsider large systems

slide-20
SLIDE 20

Reconsider large systems Shared infrastructure ...

slide-21
SLIDE 21

CS Research is Timeless

Inform decisions Mitigate technical risk

slide-22
SLIDE 22

* 2

2

Aysylu Greenberg

@aysylu22

slide-23
SLIDE 23

Papers We Love NYC

slide-24
SLIDE 24

Papers We Love SF

slide-25
SLIDE 25

* 2

5

Aysylu Greenberg

@aysylu22

slide-26
SLIDE 26

Today

  • Staged Event-Driven Architecture
slide-27
SLIDE 27

Today

  • Staged Event-Driven Architecture
  • Leases
slide-28
SLIDE 28

Today

  • Staged Event-Driven Architecture
  • Leases
  • Inaccurate Computations
slide-29
SLIDE 29

Staged Event Driven Architecture & Deep Pipelines

2001

slide-30
SLIDE 30

Hardware to Data Pipelines

slide-31
SLIDE 31

Hardware to Data Pipelines

https://en.wikipedia.org/wiki/Graphics_pipeline

slide-32
SLIDE 32
slide-33
SLIDE 33

Staged Event Driven Architecture

slide-34
SLIDE 34

Staged Event Driven Architecture

+

slide-35
SLIDE 35

Single-machine pipeline generalizes to distributed pipelines Staged Event Driven Architecture

slide-36
SLIDE 36

Search Indexing Pipelines

slide-37
SLIDE 37

Search Indexing Pipelines

slide-38
SLIDE 38

Search Indexing Pipelines

slide-39
SLIDE 39

Search Indexing Pipelines

slide-40
SLIDE 40

Search Indexing Pipelines

slide-41
SLIDE 41

Search Indexing Pipelines

slide-42
SLIDE 42

Search Indexing Pipelines

slide-43
SLIDE 43

Search Indexing Pipelines

slide-44
SLIDE 44

Search Indexing Pipelines

+

slide-45
SLIDE 45

Leases as Heart Beat in Distributed Systems

1989

slide-46
SLIDE 46
slide-47
SLIDE 47

Leases

  • Distributed locking
slide-48
SLIDE 48

Leases

  • Distributed locking
  • Lease term tradeoffs

○ short

slide-49
SLIDE 49

Leases

  • Distributed locking
  • Lease term tradeoffs

○ short vs long

slide-50
SLIDE 50

Leases

  • Distributed locking
  • Lease term tradeoffs

○ short vs long

  • Use of leases in modern applications

○ Leader election TTL (in etcd)

slide-51
SLIDE 51

Leases

  • Distributed locking
  • Lease term tradeoffs

○ short vs long

  • Use of leases in modern applications

○ Leader election TTL (in etcd) ○ Liveness detection

slide-52
SLIDE 52
slide-53
SLIDE 53

Leases in Build System: Success Scenario

slide-54
SLIDE 54

Build my project

Build System

slide-55
SLIDE 55

Build my project

Build System

OK

slide-56
SLIDE 56

Build my project

Build System

OK Waiting for the results

slide-57
SLIDE 57

Build my project

Build System

OK Waiting for the results Build is in progress

slide-58
SLIDE 58

Build my project

Build System

OK Waiting for the results Build is in progress Waiting for the results

slide-59
SLIDE 59

Build my project

Build System

OK Waiting for the results Build is in progress Waiting for the results Build is finished

slide-60
SLIDE 60

Leases in Build System: Failure Scenario

slide-61
SLIDE 61

Leases in Build System

slide-62
SLIDE 62

Leases in Build System

slide-63
SLIDE 63

Leases in Build System

slide-64
SLIDE 64

Leases in Build System

slide-65
SLIDE 65

Leases in Build System

slide-66
SLIDE 66

Leases in Build System

slide-67
SLIDE 67

Using etcd leases for heartbeat

$ curl http://server.com/v2/keys/foo -XPUT -d\ value=bar -d ttl=300

slide-68
SLIDE 68

{ "action": "set", "node": { "createdIndex": 2, "expiration":"2016-06-14T16:15:00", "key": "/foo", "modifiedIndex": 2, "ttl": 300, "value": "bar" } }

slide-69
SLIDE 69

Using etcd leases for heartbeat

$ curl http://server.com/v2/keys/foo -XPUT -d \ value=bar -d ttl=300

… 3 minutes later...

slide-70
SLIDE 70

Using etcd leases for heartbeat

$ curl http://server.com/v2/keys/foo -XPUT -d \ value=bar -d ttl=300 $ curl \ http://server.com/v2/keys/foo?prevValue=bar \

  • XPUT -d ttl=300 -d refresh=true -d \

prevExist=true

slide-71
SLIDE 71

{ "action": "update", "node": { "createdIndex": 2, "expiration":"2016-06-14T16:18:00", "key": "/foo", "modifiedIndex": 3, "ttl": 300, "value": "bar" } "prevNode": {...} }

slide-72
SLIDE 72

{ "action": "update", "node": { "createdIndex": 2, "expiration":"2016-06-14T16:18:00", "key": "/foo", "modifiedIndex": 3, "ttl": 300, "value": "bar" } "prevNode": {...} }

"prevNode": { "createdIndex": 2, "expiration":"2016-06-14T16:15:00", "key": "/foo", "modifiedIndex": 2, "ttl": 120, "value": "bar" }

slide-73
SLIDE 73

Leases for heartbeat: How long should the lease term be?

slide-74
SLIDE 74

Inaccurate Computations & Serving Search Results

slide-75
SLIDE 75

From Accurate to "Good Enough"

slide-76
SLIDE 76

[Trade off] Inaccuracy for Performance

slide-77
SLIDE 77
slide-78
SLIDE 78
slide-79
SLIDE 79
slide-80
SLIDE 80

[Trade off] Inaccuracy for Resilience

slide-81
SLIDE 81
slide-82
SLIDE 82
slide-83
SLIDE 83

Reduce Map Input Map Input Map Input

slide-84
SLIDE 84

Inaccuracy for Resilience

  • 1. Task decomposition
slide-85
SLIDE 85
slide-86
SLIDE 86

Inaccuracy for Resilience

  • 1. Task decomposition
  • 2. Baseline for correctness
slide-87
SLIDE 87
slide-88
SLIDE 88

Inaccuracy for Resilience

  • 1. Task decomposition
  • 2. Baseline for correctness
  • 3. Criticality Testing
slide-89
SLIDE 89
slide-90
SLIDE 90
slide-91
SLIDE 91
slide-92
SLIDE 92
slide-93
SLIDE 93
slide-94
SLIDE 94

Inaccuracy for Resilience

  • 1. Task decomposition
  • 2. Baseline for correctness
  • 3. Criticality Testing
  • 4. Distortion and timing models
slide-95
SLIDE 95

Distortion Model

slide-96
SLIDE 96

Timing Model

slide-97
SLIDE 97

[In production] Inaccuracy for Performance & Resilience

slide-98
SLIDE 98

Jeff Dean "Building Software Systems at Google and Lessons Learned", Stanford, 2010

slide-99
SLIDE 99
slide-100
SLIDE 100
slide-101
SLIDE 101

[Designing with] Inaccuracy for Performance & Resilience

slide-102
SLIDE 102

[Designing with] Inaccuracy for Performance & Resilience simplified implementation focus on observability applicable to some problem domains

slide-103
SLIDE 103

[Designing with] Inaccuracy for Performance & Resilience fuzz testing generative testing simplified implementation fault injection testing focus on observability applicable to some problem domains

slide-104
SLIDE 104

References

  • T. Wurthinger, C. Wimmer et al. "One VM to Rule Them

All"

  • M. Rinard "Probabilistic Accuracy Bounds for Fault-

Tolerant Computations that Discard Tasks"

  • F. Corbato, M. Daggett, R. Daley "An Experimental Time-

Sharing System"

  • E. Dijkstra "Cooperating Sequential Processes"
  • L. Lamport "Time, Clocks, and the Ordering of Events in a

Distributed System"

  • http://blinkdb.org/
slide-105
SLIDE 105

References

  • B. Oki, B. Liskov "Viewstamped Replication: A New Primary Copy

Method to Support Highly-Available Distributed Systems"

  • L. Lamport "The Part-Time Parliament"
  • M. Welsh, D. Culler, E. Brewer "SEDA: An Architecture for Well-

Conditioned, Scalable Internet Services"

  • C. Gray, D. Cheriton "Leases: An Efficient Fault-Tolerant

Mechanism for Distributed File Cache Consistency"

  • S. Agarwal, B. Mozafari et al. "BlinkDB: Queries with Bounded

Errors and Bounded Response Times on Very Large Data"

slide-106
SLIDE 106

Gratitude

Ines Sombra David Greenberg Karan Parikh Matt Welsh Erran Berger

slide-107
SLIDE 107

Robust & scalable pipelines

slide-108
SLIDE 108

Robust & scalable pipelines Leases for sharing & heartbeat

slide-109
SLIDE 109

Robust & scalable pipelines Leases for sharing & heartbeat Inaccuracy for resilience & performance

slide-110
SLIDE 110

Robust & scalable pipelines Leases for sharing & heartbeat Inaccuracy for resilience & performance

CS research is timeless: use it to mitigate risk

slide-111
SLIDE 111

Aysylu Greenberg June 14, 2016

Distributed Systems in Practice, in Theory

@aysylu22