Consistency without consensus: CRDTs in production at SoundCloud - - PowerPoint PPT Presentation

▶
consistency without consensus crdts in production at
SMART_READER_LITE
LIVE PREVIEW

Consistency without consensus: CRDTs in production at SoundCloud - - PowerPoint PPT Presentation

Feb 27, 2023 245 likes •1.12k views

Consistency without consensus: CRDTs in production at SoundCloud Consistency without consensus: CRDTs in production at SoundCloud Me Some guy Embedded, sensor networks Distributed systems SoundCloud infrastructure Theory Distributed

slide-1
SLIDE 1

Consistency without consensus:
 CRDTs in production at SoundCloud

slide-2
SLIDE 2

Consistency without consensus:
 CRDTs in production at SoundCloud

slide-3
SLIDE 3

Me

Some guy Embedded, sensor networks Distributed systems SoundCloud infrastructure

slide-4
SLIDE 4

Theory

slide-5
SLIDE 5

Distributed programming

“The art of solving the same problem that 
 you can solve on a single computer
 using multiple computers.”
 —book.mixu.net

slide-6
SLIDE 6

Distributed programming

“Generally a bad idea,
 best avoided.”
 —me

slide-7
SLIDE 7

>>> x = 1 >>> print x 1

slide-8
SLIDE 8

$ curl -XPOST -d'{"val": 1}' http://db/vars/x HTTP 502 Bad Gateway $ curl -XGET http://db/vars/x HTTP 503 Service Unavailable

slide-9
SLIDE 9

Idioms

slide-10
SLIDE 10

1980s — RPC

slide-11
SLIDE 11

1990s — CORBA

slide-12
SLIDE 12

2000 — CAP

slide-13
SLIDE 13

Partition tolerance

“The system continues to operate despite message loss due to network and/or node failure.”
 —book.mixu.net

slide-14
SLIDE 14

Partition
 tolerance Consistency Availability

✗

CP AP

slide-15
SLIDE 15

CP

Chubby, Doozer — Paxos ZooKeeper — Zab Consul, etcd — Raft ? — Viewstamped Replication

slide-16
SLIDE 16

AP

Cassandra Riak Mongo Couch

slide-17
SLIDE 17

Message failure

Delayed Dropped Delivered out-of-order Duplicated

slide-18
SLIDE 18

CALM principle

Consistency As Logical Monotonicity

slide-19
SLIDE 19

ACID 2.0

Associative Commutative Idempotent Distributed, sure, whatever

slide-20
SLIDE 20

CRDT

Conflict-free Replicated Data Type

slide-21
SLIDE 21

Increment-only
 counter

slide-22
SLIDE 22

A' A → A'

slide-23
SLIDE 23

+

(1 + 2) + 3 = 1 + (2 + 3) 1 + 2 = 2 + 1 1 + 1 ≠ 1

slide-24
SLIDE 24

∪

( 1 ∪ 2 ) ∪ 3 = 1 ∪ ( 2 ∪ 3 ) 1 ∪ 2 = 2 ∪ 1 1 ∪ 1 = 1

slide-25
SLIDE 25

{ } { } { }

slide-26
SLIDE 26

{ } { } { }

123

slide-27
SLIDE 27

{ } { } { }

123

123

slide-28
SLIDE 28

{ } { } { }

123 123

slide-29
SLIDE 29

{ } { } { }

123 123 123

slide-30
SLIDE 30

{ } { } { }

123 123 123

slide-31
SLIDE 31

{ } { } { }

123 123 123

456

slide-32
SLIDE 32

{ } { } { }

123 123 123, 456

456

slide-33
SLIDE 33

{ } { } { }

123 123 123, 456

✘

slide-34
SLIDE 34

{ } { } { }

123 123, 456 123, 456

slide-35
SLIDE 35

{ } { } { }

123 123, 456 123, 456

slide-36
SLIDE 36

Read

{123, 456} ∪ {123} ∪ {123, 456} = {123, 456}
 {123, 456} ∆ {123} ∆ {123, 456} = {456}

slide-37
SLIDE 37

{ } { } { }

123 123, 456 123, 456

456

slide-38
SLIDE 38

{ } { } { }

123, 456 123, 456 123, 456

slide-39
SLIDE 39

{ } { } { }

123, 456 123, 456 123, 456

slide-40
SLIDE 40

{ } { } { }

123, 456 123, 456 123, 456

slide-41
SLIDE 41

Interlude —
 Bending the problem

slide-42
SLIDE 42

CRDTs in production

slide-43
SLIDE 43
slide-44
SLIDE 44

Event

Timestamp Actor Verb Thing

slide-45
SLIDE 45

Event

2014-04-01T15:16:17.187Z snoopdogg reposted theeconomist/election-day

slide-46
SLIDE 46

Fan out on write

༼⍢༽ ●

  • ●●● (•ི̛ᴗ•̛)ྀ
  • ●● ༼•͟ ͜
  • ༽
  • ●●●● (ಠ_ಠ)
slide-47
SLIDE 47

Fan in on read

༼⍢༽ ••••• (•ི̛ᴗ•̛)ྀ ༼•͟ ͜

  • ༽

(ಠ_ಠ) [¬º-°] •••

slide-48
SLIDE 48

Unique events — use a set

G-set — can’t delete 2P-set — add, remove once OR-set — storage overhead

slide-49
SLIDE 49

A wild set appears

slide-50
SLIDE 50

Roshi set

S+ { A/1 B/2 C/3 } S– { D/4 } S { A B C }

slide-51
SLIDE 51

Roshi set

S = actor’s outbox key
 snoopdogg·outbox A/B/C/D = actor+verb+thing
 snoopdogg·repost·theeconomist/election-day 1/2/3 = timestamp
 2014-04-01T15:16:17.187Z

slide-52
SLIDE 52

Reading is easy

slide-53
SLIDE 53

Writing is interesting

slide-54
SLIDE 54

Insert

  • If either key+ or key– already contains element,


and the existing score >= score,
 no-op and exit.

  • Insert (element, score) into add set key+.
  • Delete (element) from remove set key–.
slide-55
SLIDE 55

Delete

  • If either key+ or key– already contains element,


and the existing score >= score,
 no-op and exit.

  • Insert (element, score) into add set key–.
  • Delete (element) from remove set key+.
slide-56
SLIDE 56

Example

slide-57
SLIDE 57

S+ { A/1 B/2 } S– { C/3 }

slide-58
SLIDE 58

S+ { A/1 B/2 } S– { C/3 }

Insert D/4

slide-59
SLIDE 59

S+ { A/1 B/2 D/4 } S– { C/3 }

Insert D/4

slide-60
SLIDE 60

S+ { A/1 B/2 D/4 } S– { C/3 }

slide-61
SLIDE 61

S+ { A/1 B/2 D/4 } S– { C/3 }

Insert D/4

slide-62
SLIDE 62

S+ { A/1 B/2 D/4 } S– { C/3 }

Insert D/4

slide-63
SLIDE 63

S+ { A/1 B/2 D/4 } S– { C/3 }

slide-64
SLIDE 64

S+ { A/1 B/2 D/4 } S– { C/3 }

Delete D/3

slide-65
SLIDE 65

S+ { A/1 B/2 D/4 } S– { C/3 }

Delete D/3

slide-66
SLIDE 66

S+ { A/1 B/2 D/4 } S– { C/3 }

slide-67
SLIDE 67

S+ { A/1 B/2 D/4 } S– { C/3 }

Delete D/5

slide-68
SLIDE 68

S+ { A/1 B/2 D/4 } S– { C/3 D/5 }

Delete D/5

slide-69
SLIDE 69

S+ { A/1 B/2 } S– { C/3 D/5 }

slide-70
SLIDE 70

S+ { A/1 B/2 } S– { C/3 D/5 }

Delete D/6

slide-71
SLIDE 71

S+ { A/1 B/2 } S– { C/3 D/6 }

Delete D/6

slide-72
SLIDE 72

S+ { A/1 B/2 } S– { C/3 D/6 }

slide-73
SLIDE 73

Making it real

slide-74
SLIDE 74
slide-75
SLIDE 75

Pool Cluster

slide-76
SLIDE 76

Pool Cluster Pool Cluster Pool Cluster Farm

slide-77
SLIDE 77

Writing is easy

slide-78
SLIDE 78

Reading is interesting

slide-79
SLIDE 79

Pool Cluster Pool Cluster Pool Cluster Farm

slide-80
SLIDE 80

Pool Cluster Pool Cluster Pool Cluster Farm

slide-81
SLIDE 81

Cluster Cluster Cluster

{A B C} {A C} {A B C} ∪ = {A B C} ∆ = {B}

slide-82
SLIDE 82

Pool Cluster Pool Cluster Pool Cluster Farm

slide-83
SLIDE 83

github.com/soundcloud/roshi

slide-84
SLIDE 84

In conclusion,

slide-85
SLIDE 85

Consistency without consensus = CRDT.
 Embrace your invariants.
 Maybe bend your problem, not your solution.

slide-86
SLIDE 86

Thanks!

soundcloud.com/jobs @peterbourgon ☞ ☜