GEO-REPLICATION GEO-REPLICATION 150 ms SYNC DC2 DC2 DC1 DC1 - - PowerPoint PPT Presentation

▶
geo replication geo replication
SMART_READER_LITE
LIVE PREVIEW

GEO-REPLICATION GEO-REPLICATION 150 ms SYNC DC2 DC2 DC1 DC1 - - PowerPoint PPT Presentation

Sep 21, 2023 181 likes •310 views

INTERNET SERVICES NOWADAYS Services operate on a global scale. PUTTING CONSISTENCY BACK An unprecedented number of people are using INTO EVENTUAL CONSISTENCY internet services. Valter Balegas, Srgio Duarte, Carla Ferreira, Rodrigo

slide-1
SLIDE 1

PUTTING CONSISTENCY BACK INTO EVENTUAL CONSISTENCY

Valter Balegas, Sérgio Duarte, Carla Ferreira, Rodrigo Rodrigues, Nuno Preguiça NOVA LINCS / U. Nova de Lisboa Mahsa Najafzadeh, Marc Shapiro INRIA, LIP6

INTERNET SERVICES NOWADAYS

  • Services operate on a global scale.
  • An unprecedented number of people are using

internet services.

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

2

GEO-REPLICATION

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

3

20 ms 150 ms 20 ms SYNC DC2 DC1

GEO-REPLICATION

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

4

DC2 DC1

slide-2
SLIDE 2

GEO-REPLICATION

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

5

Tournament A B Player Sonic Pac-man Mario Tournament A B Player Sonic Pac-man Mario

DC2 DC1

GEO-REPLICATION

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

6

enroll(Pac-man, A)

Player Sonic Pac-man Mario Player Sonic Pac-man Mario

DC2 DC1

Tournament A B Tournament A B

GEO-REPLICATION

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

7

enroll(Mario, A)

Player Sonic Pac-man Mario Player Sonic Pac-man Mario

DC2 DC1

Tournament A B Tournament A B

GEO-REPLICATION

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

8

removeTournament(A)

Player Sonic Pac-man Mario Player Sonic Pac-man Mario

DC2 DC1

Tournament A B Tournament A B

slide-3
SLIDE 3

GEO-REPLICATION

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

9

removeTournament(A)

Player Sonic Pac-man Mario Player Sonic Pac-man Mario

DC2 DC1

Tournament A B Tournament B

GEO-REPLICATION

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

10

SYNC

Player Sonic Pac-man Mario Player Sonic Pac-man Mario

DC2 DC1

Tournament A B Tournament B

GEO-REPLICATION

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

11

Mario is enrolled in tournament that was concurrently removed.

DC2 DC1

Player Sonic Pac-man Mario Player Sonic Pac-man Mario Tournament B Tournament B

STRONG CONSISTENCY

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

12

DC1 DC2

enroll(Pac-man, A) SYNC Ordering of operations: enroll(Mario, A) removeTournament(A) enroll(Pac-man, A) SYNC enroll(Mario, A) removeTournament(A) SYNC

slide-4
SLIDE 4

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

13

IS COORDINATION NEEDED?

DC1 DC2

enroll(Pac-man, A) Ordering of operations: enroll(Mario, A) removeTournament(A) enroll(Pac-man, A) enroll(Mario, A) removeTournament(A) SYNC Local operation Low latencyJ

OUTLINE

  • Background
  • Explicit Consistency
  • Indigo
  • Evaluation
  • Conclusion

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

14

EXPLICIT CONSISTENCY

  • Programmer specifies application invariant.
  • System ensures that every state transition

preserves the invariant.

  • Opportunity to improve performance by not

restricting execution ordering.

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

15

A METHODOLOGY FOR EXPLICIT CONSISTENCY

  • Identify I-offenders

– Static analysis identifies operations that may break invariants when executed concurrently.

  • Choose reservations

– Efficient mechanism to execute I-offenders avoiding coordination.

  • Instrument application code with selected

mechanism.

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

16

slide-5
SLIDE 5

A METHODOLOGY FOR EXPLICIT CONSISTENCY

  • Identify I-offenders

– Static analysis identifies operations that may break invariants when executed concurrently.

  • Choose reservations

– Efficient mechanism to execute I-offenders avoiding coordination.

  • Instrument application code with selected

mechanism.

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

17

STATIC ANALYSIS: APPLICATION MODEL

  • Programmer specifies:

– Invariant:

“Players can only participate in existing tournaments.”

– Operations’ side effects:

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

18

Inv = enrolled(p,t) player(p) tournament(t)

{enrolled(p,t) := true } enroll (p,t): removeTournament(t): { tournament(t) := false}

STATIC ANALYSIS: ALGORITHM

Inv = enrolled(p,t) player(p) tournament(t)

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

19

enroll(p,t) removeTournament(t)

Inv = true Inv = false Inv = true player(p) tournament(t) Use SMT Solver to test all pairs of

  • perations.

Inv = enrolled(p,t) false Inv = true false

A METHODOLOGY FOR EXPLICIT CONSISTENCY

  • Identify I-offenders

– Static analysis identifies operations that may break invariants when executed concurrently.

  • Choose reservations

– Efficient mechanism to execute I-offenders avoiding coordination.

  • Instrument application code with selected

mechanism.

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

20

slide-6
SLIDE 6

RESERVATIONS

  • Mechanisms to control the execution of I-offenders

without breaking invariants.

  • Coordination outside the operation flow.
  • Different reservations for different invariants:

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

21

Invariant type Reservation Generic Multi-level Lock Numeric Escrow Referential Integrity Multi-level Lock Uniqueness UID Generator Disjunction Multi-level Mask Range partition Partition Lock

RESERVATIONS: MULTI-LEVEL LOCK

  • Protects the execution of conflicting
  • perations.
  • Only allow the execution of one type of
  • peration at a time.
  • Operation can be executed by multiple clients

that hold the lock.

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

22

ENROLL(*,A)

RESERVATIONS: EXAMPLE

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

23

Tournament A B Player Sonic Pac-man Mario Player Sonic Pac-man Mario

DC2 DC1

Tournament A B DC1 DC2 ENROLL(*,A) ENROLL(*,A) DC1 DC2 ENROLL(*,A)

RESERVATIONS: EXAMPLE

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

24

enroll(Pac-man, A)

Player Sonic Pac-man Mario Player Sonic Pac-man Mario

DC2 DC1

Tournament A B Tournament A B DC1 DC2 ENROLL(*,A) DC1 DC2 ENROLL(*,A) ENROLL(*,A) ENROLL(*,A)

slide-7
SLIDE 7

RESERVATIONS: EXAMPLE

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

25

Player Sonic Pac-man Mario Player Sonic Pac-man Mario

DC2 DC1

Tournament A B Tournament A B

enroll(Pac-man, A)

DC1 DC2 ENROLL(*,A) DC1 DC2 ENROLL(*,A) ENROLL(*,A) ENROLL(*,A)

RESERVATIONS: EXAMPLE

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

26

enroll(Mario, A)

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

26

Player Sonic Pac-man Mario Player Sonic Pac-man Mario

DC2 DC1

Tournament A B Tournament A B DC1 DC2 ENROLL(*,A) DC1 DC2 ENROLL(*,A) ENROLL(*,A) ENROLL(*,A)

RESERVATIONS: EXAMPLE

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

27

enroll(Mario, A)

Player Sonic Pac-man Mario Player Sonic Pac-man Mario

DC2 DC1

Tournament A B Tournament A B DC1 DC2 ENROLL(*,A) DC1 DC2 ENROLL(*,A) ENROLL(*,A) ENROLL(*,A)

RESERVATIONS: EXAMPLE

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

28

removeTournament(A)

Player Sonic Pac-man Mario Player Sonic Pac-man Mario

DC2 DC1

Tournament A B Tournament A B DC1 DC2 ENROLL(*,A) DC1 DC2 ENROLL(*,A) ENROLL(*,A) ENROLL(*,A)

slide-8
SLIDE 8

RESERVATIONS: EXAMPLE

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

29

Player Sonic Pac-man Mario Player Sonic Pac-man Mario

DC2 DC1

Tournament A B Tournament A B

STOP(enroll(*, A)) removeTournament(A) Waiting…

DC1 DC2 ENROLL(*,A) DC1 DC2 ENROLL(*,A) ENROLL(*,A) ENROLL(*,A)

RESERVATIONS: EXAMPLE

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

30

Player Sonic Pac-man Mario Player Sonic Pac-man Mario

DC2 DC1

Tournament A B Tournament A B

removeTournament(A) Waiting…

DC1 DC2 ENROLL(*,A) DC1 DC2 ENROLL(*,A)

  • ENROLL(*,A)

ENROLL(*,A)

  • RESERVATIONS: EXAMPLE

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

31

Player Sonic Pac-man Mario Player Sonic Pac-man Mario

SYNC DC2 DC1

Tournament A B Tournament A B

removeTournament(A) Waiting…

DC1 DC2 DC1 DC2 ENROLL(*,A)

  • ENROLL(*,A)

REMOVE_TRNMT(A)

  • RESERVATIONS: EXAMPLE

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

32

Player Sonic Pac-man Mario Player Sonic Pac-man Mario

DC2 DC1

Tournament A B Tournament A B

removeTournament(A) Waiting…

DC1 DC2 DC1 DC2

  • DO(removeTournament(A))

ENROLL(*,A) ENROLL(*,A)

slide-9
SLIDE 9

REMOVE_TRNMT(A)

  • Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

33

Player Sonic Pac-man Mario Player Sonic Pac-man Mario

RESERVATIONS: EXAMPLE

DC2 DC1

Tournament A B Tournament B

removeTournament(A)

DC1 DC2 DC1 DC2

  • ENROLL(*,A)

INDIGO

  • Middleware that provides Explicit consistency
  • n top of KV-Stores.
  • Requires only properties that are known to be

efficient.

  • Can be extended with new reservations.

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

34

EVALUATION

  • How well does the system scale?
  • What is the latency of operations?
  • Behavior with more reservations per
  • peration?
  • Applicability of the solution.

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

35

DEPLOYMENT

  • Data-centers deployed in AWS:

– 3 Regions (EU, US-EAST/WEST); – N app-servers connect to local DBs; – Clients submit operations to the app-server in close loop.

  • Compare performance:

– Causal Consistency – Strong Consistency (Writes to single server) – Red-Blue Consistency (Causal + Writes to single server) – Explicit Consistency (Causal + Reservations)

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

36

slide-10
SLIDE 10

AD-SERVICE

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

37

1000 ads; 100% I-offenders

TOURNAMENT

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

38

82% reads; 4% safe writes; 14% I-offenders

TOURNAMENT: OPERATIONS LATENCY

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

39

Detailed Operations Latency

CONCLUSIONS

  • Explicit Consistency successfully reduces

coordination:

– Programmers provide simple annotations; – Static analysis detects conflicting operations; – Low-latency operations with reservations.

  • Performance comparable to Causal consistency.

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

40

slide-11
SLIDE 11

QUESTIONS?

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

41

Adding more reservations

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

42 5 10 15 20 25 30 35 40 45 50 500 1000 1500 2000 2500 3000 Latency [ms] Throughput [TP/s] Indigo R5-W1 Indigo R5-W2 Indigo R5-W3 Weak R5-W1 Weak R5-W2 Weak R5-W3

(b) Peak throughput with an increasing number

  • f invariants (ad counter application).

Latency over time

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

43

50 100 150 200 250 300 20 40 60 80 100 120 140 160 180 Latency [ms] Time [seconds] US-WEST

(c) Latency of individual operations of US-W datacenter (ad counter application).

Overhead with increasing contention

Valter Balegas – NOVA LINCS, FCT-UNL - Putting Consistency Back Into Eventual Consistency @ Eurosys'15

44

200 400 600 800 1000 1200 1400 1600 1 10 100 1000 10000 Throughput [ TP/Second ] # Keys Indigo Causal

(a) Peak throughput with increasing contention (ad counter application).