Cloud to the edge Just-Right Consistency Static analysis for - - PowerPoint PPT Presentation

Just-Right Consistency

Static analysis for minimal synchronisation

Marc Shapiro

Inria & Sorbonne-Universités UPMC-LIP6

[Just-Right Consistency]

Cloud to the edge

2

Social, web, e-commerce: shared mutable data Scalability ⇒ replication ⇒ consistency issues

[Just-Right Consistency]

Part I: Consistency vs. performance

Part I: Consistency vs. performance

• Geo-replicated cloud databases
• Consistency models
• Some partial solutions

Part II

• Just-right consistency

3 [Just-Right Consistency]

Serrano-SI P-Store-SER GMU-US Jessy2pc-NMSI RC Walter-PSI SDUR-SER Credit: Masoud Saeida Ardekani

Models matter

4

3× 4.5×

Termination Latency of Update Transactions (ms) 90% Read-only transactions; Disaster Tolerant 70% Read-only transactions; Disaster Tolerant

[Just-Right Consistency]

Strongest: Strict Serialisability

5

T1 T1 R1 R2 R3 Invariant

client

T1 Invariant Invariant T3 T3 T2 T2

[Just-Right Consistency]

Weakest: Eventual consistency

6

Op1 R1 R2 R3 Op2

[Just-Right Consistency]

The problem(s) of consistency

Same object:

• Safe: updates, state satisfy specification,

internal invariants

• Replicas converge to same state

Separate objects: maintain relations

• Multi-object invariants
• Different kinds ⟹ different mechanisms

ACID transactions mix all this; often too strong

7 [Just-Right Consistency]

• Seq. consistency

examples

Bank account

• debit(amt), credit(amt), accrueInterest(amt)
• Invariant: “balance ≥ 0”
• { amt ≤ balance ∧ Inv } debit(amt) { Inv }

File system

• mkdir, rmdir, create, write, rm, ls, etc.
• Invariant: Tree
• { Tree ∧ ¬ x/…/y } mv(x,y) { Tree }

8

[Just-Right Consistency]

CAP

Sequential Consistency: total order of

• perations ⟹ replicas identical
• Consensus: “Who’s next?”
• Requires communication

CAP Theorem: “When network can Partition,

• either sequential Consistency,
• or Availability;
• can’t have both!”

Availability related to performance

• Parallelise
• More implementation choices

9 [Just-Right Consistency]

Consistency issues under EC

Updates delivered in different orders: not identical, do not converge Lost updates (LWW: by design) No causality: updates received out of order No transactions: inter-object invariants violated Compensating at application level: very challenging Solution: Spanner?

10 [Just-Right Consistency]

Operation

u: state ⤻ (retval, (state ⤻ state)) Prepare (@origin) u?; deliver u! Read one, write all (ROWA) Deferred-update replication (DUR)

11

• rigin

replica

u! u! u?

client

u

replica

uPRE u!

replica

v? v!

[Just-Right Consistency]

Concurrency

Concurrent, Multi-master Strong: total order, identical state Weak: concurrent, interleaving, no global state

12

Convergence? Safety?

u! u! u? u

• rigin

replica client

• ther

replica

v? v! v! v! u!

[Just-Right Consistency]

Anomalies of concurrent updates

Bank:

• σinit = 100€
• Alice: credit(20) = { σ ≔ 120 }
• Bob: debit (60) = { σ ≔ 40 }
• σ = ???

File system:

• σinit = “/“
• Alice: mkdir (“/foo”); mkdir (“/foo/bar”)
• Bob: receives mkdir (“/foo/bar”)
• σ = ???

13 [Just-Right Consistency]

access (Bob, photo) ⟹ ACL (Bob, photo) v observed effects of u ⟹ v should be delivered after u Available: doesn’t slow down sender

Not causal

14 u u

v v v u

Bob Alice @home Alice @phone

Don’t show photos to Bob post photo Bob sees photo

[Just-Right Consistency]

Causal-order delivery (1) Causal consistency

15 u u

v v v u

Bob Alice @home Alice @phone

access (Bob, photo) ⟹ ACL (Bob, photo) v observed effects of u ⟹ v should be delivered after u Available: doesn’t slow down sender

[Just-Right Consistency]

(2) Conflict-free replicated data types

Data type

• Encapsulates issues

Replicated

• At multiple nodes

Available

• Update my replica without coordination
• Convergence guaranteed (formal properties)
• Decentralised, peer-to-peer

16

[Just-Right Consistency]

Commute ⟹ converge

Bank account:

• credit(amt)! = { local_balance += amt }
• debit(amt)! = { local_balance –= amt }
• interest()! =

{ local_balance += origin_balance*.05 } File system:

• write(f)! = { local_f ⊔ f }

17 [Just-Right Consistency]

CRDT design concept

Backward-compatible with sequential datatype Commute ⟹ concurrent is same

• add(e); rm(f) = rm(f); add(e) ≜ add(e) || rm (f)

Otherwise, concurrency semantics

• Example: add(e) || rm (e)
• Deterministic, similar to sequential
• ≈ rm(e);add(e) or ≈ add(e); rm(e)
• Merge, don’t lose updates
• Result doesn't depend on order received
• Stable preconditions

18 [Just-Right Consistency]

CRDT design concept

Backward-compatible with sequential datatype Commute ⟹ concurrent is same

• add(e); rm(f) = rm(f); add(e) ≜ add(e) || rm (f)

Otherwise, concurrency semantics

• Example: add(e) || rm (e)
• Deterministic, similar to sequential
• ≈ rm(e);add(e) or ≈ add(e); rm(e)
• Merge, don’t lose updates
• Result doesn't depend on order received
• Stable preconditions

19 [Just-Right Consistency]

CRDT design concept

Backward-compatible with sequential datatype Commute ⟹ concurrent is same

• add(e); rm(f) = rm(f); add(e) ≜ add(e) || rm (f)

Otherwise, concurrency semantics

• Example: add(e) || rm (e)
• Deterministic, similar to sequential
• ≈ rm(e);add(e) or ≈ add(e); rm(e)
• Merge, don’t lose updates
• Result doesn't depend on order received
• Stable preconditions

20

[Just-Right Consistency]

CRDT concept

Innovation and research: systems, algorithms, databases Consistency of shared mutable data

21 [CRDTs in practice] 22 RA RB RC {1} (1, {a}, {}) add(1) {1} (1, {c}, {}) add(1) {} (1, {c}, {c}) remove(1) {1} {1} {1} (1, {a, c}, {c}) (1, {a, c}, {c}) (1, {a, c}, {c})

Add-Wins Set CRDT

[Just-Right Consistency]

CRDT types

Converge concurrent updates Encapsulate replication & resolution Re-usable data types Correct by construction

23

Register

• Last-Writer Wins
• Multi-Value

Set

• Grow-Only
• 2P
• Observed-Remove

Map Counter

• Unlimited
• Restricted ≥0

Graph

• Directed
• Monotonic DAG
• Edit graph

Sequence

[Just-Right Consistency]

(3) Bounded Counter CRDT

Replicated Counter: inc(), dec() Invariant: bounded “x≥0” Credit per replica: ∑ crediti ≤ bound Asynchronous:

• { crediti ≥ 1 } dec!() ={ ctr –= 1; crediti –= 1 }
• transfer (crediti, crediti)

Synchronized

• acquire(crediti)

24

[Just-Right Consistency] 25 [Just-Right Consistency]

SwiftCloud edge +cloud

26

Update, commit shared store locally Availability + consistency: DC switch Causal + transactional 3000+ client replicas

DC DC DC C C C C

T r a n s m i t partial database app Process request
 & store update Transmit Transmit

f a i l

• v

e r

full database Transmit

[Just-Right Consistency]

Antidote

SyncFree EU project High performance, sharded, transactional, causal Aims to scale to 100s of DCs

• Very modular
• Partial replication
• Small but safe metadata (vector clock)

In DC: strong consistency, physical clocks (Clock-SI) Industrial apps: Virtual Wallet, SocialApp, configuration management, FMK

27 [Just-Right Consistency]

(4) NMSI: strong, parallel

28

T1 T2 x y x? Wait-Free Queries y? T2 x? Forward Freshness

• Mini. Commit.

Synch + Genuine Partial Repl. T3 y? x? Non- Monotonic Snapshot

Read from causal snapshot Scalability properties:

• Wait-Free Queries
• Forward Freshness
• Mini. Commitment Synchronisation
• Genuine Partial Replication

[Just-Right Consistency]

Three dimensions

29

Eventual Consistency Snapshot Isolation HAT

Gen1 / Total Order E Q / C

• m

p

• s

i t i

• n

PO / Visibility

Causal Linearisability Serialisability Strict Serialisability

CAP

[Just-Right Consistency]

Part II: Just-right consistency

Part I: Consistency vs. performance

• Geo-replicated cloud databases
• Consistency models
• Some partial solutions

Part II

• Just-right consistency

30 [Just-Right Consistency]

Application invariants

South ⨄ Boat ⨄ North = { sheep, dog, wolf } carryNorth(S) ⟹ 1 ≤ |S| ≤ 2 carrySouth(S) ⟹ 1 ≤ |S| ≤ 2 ∀S ∈ {South, Boat, North} : sheep ∈ S ∧ wolf ∈ S ⟹ dog ∈ S Hard to tease invariants out

• Silent invariants

31 [Just-Right Consistency]

• Seq. consistency

examples

Bank account

• debit(amt), credit(amt), accrueInterest(amt)
• Invariant: “balance ≥ 0”
• { amt ≤ balance ∧ Inv } debit(amt) { Inv }

File system

• mkdir, rmdir, create, write, rm, ls, etc.
• Invariant: Tree
• { Tree ∧ ¬ x/…/y } mv(x,y) { Tree }

32

[Just-Right Consistency]

Just-Right Consistency

CRDT geo-replicated database

• Lots of internal parallelism
• Transactional, causal consistency by default

Specification of application updates, invariant

• CISE: do all state transitions preserve invariant?
• If not, fix: adjust
• either specification
• or synchronisation
• Repeat until safe

App / synch co-design: Minimal synchronisation

33 [Just-Right Consistency] 34

Asynchronous, replicated updates

• State σ
• Invariant I
• Prepare: read one, generate effector
• Update all, deferred: deliver effector

Converge? Invariant OK?

σ: I u! u! u? σ: I v! v! I ? I ?

100 € ≥ 0 100 € ≥ 0 accrue 5% +5 € +5 € debit(100) 5 € ≥ 0 5 € ≥ 0 –100 –100

[Just-Right Consistency] 35

CISE Rules 1: Sequential correctness

• Any single operation maintains the invariant

2: Convergence

• Concurrent effectors commute

3: Precondition Stability

• Every precondition is stable under every

concurrent operation If satisfied: invariant is guaranteed

σ: I u! u! u? σ: I v! v! I ? I ?

[Just-Right Consistency]

Simple example: bank account

Operations: credit(amount), debit(amount) Invariant: balance ≥ 0

• Start with weak specification
• Rule 1 ⟶ strengthen precondition for debit
• Rule 2: OK
• Rule 3 ⟶ debit || debit unsafe, fixed with

concurrency control

36

[Just-Right Consistency] 37

CISE Rules 1: Sequential correctness

• Any single operation maintains the invariant

2: Convergence

• Concurrent effectors commute

3: Precondition Stability

• Every precondition is stable under every

concurrent operation If satisfied: invariant is guaranteed

u! u! u? I I uPRE uPRE

• σ: I

σ: I

[Just-Right Consistency] 38

CISE Rules 1: Sequential correctness

• Any single operation maintains the invariant

2: Convergence

• Concurrent effectors commute

3: Precondition Stability

• Every precondition is stable under every

concurrent operation If satisfied: invariant is guaranteed

u! u! u? v! v! σ: I σ: I

[Just-Right Consistency] 39

CISE Rules 1: Sequential correctness

• Any single operation maintains the invariant

2: Convergence

• Concurrent effectors commute

3: Precondition Stability

• Every precondition is stable under every

concurrent operation If satisfied: invariant is guaranteed

u! u! u? uPRE uPRE ? I

• v!
• σ: I

σ: I

[Just-Right Consistency] 40

CISE Rules 1: Sequential correctness

• Any single operation maintains the invariant

2: Convergence

• Concurrent effectors commute

3: Precondition Stability

• Every precondition is stable under every

concurrent operation If satisfied: invariant is guaranteed

balance = 1 balance − 1 debit(1) debitPRE {1 ≤ 1} debitPRE { 1 ≤ 0 } ? balance = 1 balance − 1 balance − 1 debit(1)

Fix: concurrency control

balance = –1

[Just-Right Consistency]

Advanced example: file system

Operations: mkdir, rmdir, mv, update, etc. Invariant: Tree

• Rule 1 ⟶ precondition on mv
• “May not move node under self”
• Rule 2 ⟶ Use CRDTs for update || update
• Rule 3 ⟶ mv || mv precondition unstable

41 [Just-Right Consistency] 42

CISE Rules 1: Sequential correctness

• Any single operation maintains the invariant

2: Convergence

• Concurrent effectors commute

3: Precondition Stability

• Every precondition is stable under every

concurrent operation If satisfied: invariant is guaranteed

mv /B, /A mvPRE {¬B/.../A} mvPRE {¬B/.../A} ? mv /A, /B

Fix: concurrency control

root B A root B A

root B A root B A

• You can

have your cake and eat it too

[Just-Right Consistency]

Applying the logic

Only O(n2): no need to consider all possible interleavings We use a tool

• You can apply the same logic manually

43 [Just-Right Consistency]

Conclusion & future work

3-D decomposition

• Deconstruct hierarchy
• Classes of invariants / primitive mechanisms

CISE tool

• Synthesize synchronisation

CISE assumes causal, transactional

• Constructive: use insights for designing

apps, building mechanisms

• Deconstruct / weaker / chopping transactions
• Selective application of causality

44