[PPT] - Failure Detectors Concurrency Trilogy Part IV Announcements PowerPoint Presentation

SLIDE 1

Failure Detectors

Concurrency Trilogy Part IV

SLIDE 2

Announcements

Project proposals are due tonight, unless you got an extension.

SLIDE 3

Announcements

Project proposals are due tonight, unless you got an extension.
Only a few hours left to submit something or seek an extension.

SLIDE 4

Announcements

Project proposals are due tonight, unless you got an extension.
Only a few hours left to submit something or seek an extension.
No quiz next week.

SLIDE 5

Announcements

Project proposals are due tonight, unless you got an extension.
Only a few hours left to submit something or seek an extension.
No quiz next week.
Should have gotten results for last week's quiz.

SLIDE 6

RSMs All Over Again

SLIDE 7

Revisiting RSMs

Application Application Application Application Ordering Ordering Ordering Ordering

SLIDE 8

Revisiting RSMs

Application Application Application Application Ordering Ordering Ordering Ordering

Client Client Client Client

SLIDE 9

Revisiting RSMs

Application Application Application Application Ordering Ordering Ordering Ordering

Client Client Client Client

SLIDE 10

Revisiting RSMs

Application Application Application Application Ordering Ordering Ordering Ordering

Client Client Client Client

SLIDE 11

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client

SLIDE 12

Revisiting RSMs

Raft Raft Raft Raft

Client Client Client Client

SLIDE 13

Revisiting RSMs

Raft Raft Raft Raft

Client Client Client Client

SLIDE 14

Revisiting RSMs

Application Application Application Application Raft Raft Raft Raft

Client Client Client Client The act of executing a command at the application is destructive. Cannot undo a command.

SLIDE 15

Revisiting RSMs

Application Application Application Application Raft Raft Raft Raft

Client Client Client Client Requirement: All application replicas end up in the same state.

SLIDE 16

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client

set(x, 5)

1

M0 M1 M2 M3

SLIDE 17

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client

set(x, 5)

1 2 2 2 AppendEntries

M0 M1 M2 M3

SLIDE 18

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client

set(x, 5)

1 2 2 2 AppendEntries

set(x, 5)

3

M0 M1 M2 M3

SLIDE 19

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client

set(x, 5)

1 2 2 2 AppendEntries

set(x, 5)

3 4

M0 M1 M2 M3

SLIDE 20

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client

set(x, 5)

1 2 2 2 AppendEntries

set(x, 5)

3 4

success

5

M0 M1 M2 M3

SLIDE 21

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client

set(x, 5)

1 2 2 2 AppendEntries

set(x, 5)

3

success

5 4

M0 M1 M2 M3

set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1

SLIDE 22

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client M0 M1 M2 M3

M0 M1 M2 M3

set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1

For which replicas is x=5?

SLIDE 23

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client M0 M1 M2 M3

M0 M1 M2 M3

set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1

When?

SLIDE 24

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client M0 M1 M2 M3

M0 M1 M2 M3

set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1

Term = 2

SLIDE 25

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client M0 M1 M2 M3

M0 M1 M2 M3

set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1

Term = 2

SLIDE 26

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client M0 M1 M2 M3

M0 M1 M2 M3

set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1

Term = 2

set(x, 5)

6

SLIDE 27

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client M0 M1 M2 M3

M0 M1 M2 M3

set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1

Term = 2

set(x, 5)

6

Is this safe?

SLIDE 28

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client M0 M1 M2 M3

M0 M1 M2 M3

set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1

Term = 2

set(x, 5)

6

Is this safe?

SLIDE 29

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client M0 M1 M2 M3

M0 M1 M2 M3

set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1

Term = 2

get(x)

a leaderCommit =-1

SLIDE 30

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client M0 M1 M2 M3

M0 M1 M2 M3

set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1

Term = 2

get(x)

a

AppendEntries

b

get(x), 1, 2

leaderCommit =-1

SLIDE 31

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client M0 M1 M2 M3

M0 M1 M2 M3

set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1

Term = 2

get(x)

a

get(x), 1, 2 get(x), 1, 2 get(x), 1, 2 get(x), 1, 2

AppendEntries

b leaderCommit =-1

SLIDE 32

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client M0 M1 M2 M3

M0 M1 M2 M3

set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1

Term = 2

get(x)

a

get(x), 1, 2 get(x), 1, 2 get(x), 1, 2 get(x), 1, 2

AppendEntries

b

get(x)

c leaderCommit =-1

SLIDE 33

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client M0 M1 M2 M3

M0 M1 M2 M3

set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1

Term = 2

get(x)

a

get(x), 1, 2 get(x), 1, 2 get(x), 1, 2 get(x), 1, 2

AppendEntries

b

get(x)

c

Is this correct?

leaderCommit =-1

SLIDE 34

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client M0 M1 M2 M3

M0 M1 M2 M3

set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1

Term = 2

get(x)

a

get(x), 1, 2 get(x), 1, 2 get(x), 1, 2 get(x), 1, 2

AppendEntries

b

get(x)

c

Is this correct?

leaderCommit =-1

SLIDE 35

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client M0 M1 M2 M3

M0 M1 M2 M3

set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1

Term = 2

get(x)

a

get(x), 1, 2 get(x), 1, 2 get(x), 1, 2 get(x), 1, 2

AppendEntries

b

set(x,5)

c leaderCommit = 0

SLIDE 36

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client M0 M1 M2 M3

M0 M1 M2 M3

set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1

Term = 2

get(x)

a

get(x), 1, 2 get(x), 1, 2 get(x), 1, 2 get(x), 1, 2

AppendEntries

b

set(x,5)

c

get(x)

d

KeyValue(x, 5)

e leaderCommit = 1

SLIDE 37

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client M0 M1 M2 M3

M0 M1 M2 M3

set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1

Term = 2

get(x)

a

get(x), 1, 2 get(x), 1, 2 get(x), 1, 2 get(x), 1, 2

AppendEntries

b

set(x,5)

c

get(x)

d

KeyValue(x, 5)

e

KeyValue(x, 5)

f leaderCommit = 1

SLIDE 38

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client M0 M1 M2 M3

M0 M1 M2 M3

set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1

Term = 2

cas(x, 5, 4)

g

get(x), 1, 2 get(x), 1, 2 get(x), 1, 2 get(x), 1, 2

cas(x, 5, 4)

i leaderCommit = 2

SLIDE 39

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client M0 M1 M2 M3

M0 M1 M2 M3

set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1

Term = 2

cas(x, 5, 4)

g

get(x), 1, 2 get(x), 1, 2 get(x), 1, 2 get(x), 1, 2

cas(x, 5, 4)

i

cas(x,5,4), 2, 2

leaderCommit = 2

SLIDE 40

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client M0 M1 M2 M3

M0 M1 M2 M3

set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1

Term = 2

cas(x, 5, 4)

g

get(x), 1, 2 get(x), 1, 2 get(x), 1, 2 get(x), 1, 2

AppendEntries

h

cas(x, 5, 4)

i

cas(x,5,4), 2, 2

leaderCommit = 2

SLIDE 41

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client M0 M1 M2 M3

M0 M1 M2 M3

set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1

Term = 2

cas(x, 5, 4)

g

get(x), 1, 2 get(x), 1, 2 get(x), 1, 2 get(x), 1, 2

AppendEntries

h

cas(x, 5, 4)

i

cas(x,5,4), 2, 2 cas(x,5,4), 2, 2 cas(x,5,4), 2, 2 cas(x,5,4), 2, 2

leaderCommit = 2

SLIDE 42

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client M0 M1 M2 M3

M0 M1 M2 M3

set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1

Term = 2

cas(x, 5, 4)

g

get(x), 1, 2 get(x), 1, 2 get(x), 1, 2 get(x), 1, 2

AppendEntries

h

cas(x, 5, 4)

i

cas(x,5,4), 2, 2 cas(x,5,4), 2, 2 cas(x,5,4), 2, 2 cas(x,5,4), 2, 2

Is this correct?

leaderCommit = 2

SLIDE 43

Revisiting RSMs

KVStore KVStore KVStore KVStore Raft Raft Raft Raft

Client Client Client Client M0 M1 M2 M3

M0 M1 M2 M3

set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1 set(x, 5), 0, 1

Term = 2

cas(x, 5, 4)

g

get(x), 1, 2 get(x), 1, 2 get(x), 1, 2 get(x), 1, 2

AppendEntries

h

cas(x, 5, 4)

i

cas(x,5,4), 2, 2 cas(x,5,4), 2, 2 cas(x,5,4), 2, 2 cas(x,5,4), 2, 2

KeyValue(x, 4)

j

k leaderCommit = 2

SLIDE 44

Configuration Change

SLIDE 45

Why?

Want to be able to change the set of servers.

SLIDE 46

Why?

Want to be able to change the set of servers.
Take down servers for maintenance.

SLIDE 47

Why?

Want to be able to change the set of servers.
Take down servers for maintenance.
Add new servers to replace failed ones.

SLIDE 48

Why?

Want to be able to change the set of servers.
Take down servers for maintenance.
Add new servers to replace failed ones.
Other reasons.

SLIDE 49

How?

Use a special log message which contains the set of servers.

Term Config Index

SLIDE 50

How?

Use a special log message which contains the set of servers.
Use Raft to replicate this to everyone.

Term Config Index

SLIDE 51

How Special?

All peers use configuration as soon as logged.
Why safe?
We know how to revert this change.

Term Config Index

SLIDE 52

Protocol

set(x, 5) set(x, 5) set(x, 6)

1

set(x, 6)

1

set(x, 5) set(x, 6)

1

...

2

...

3

...

4

...

2

SLIDE 53

Protocol

set(x, 5) set(x, 5) set(x, 6)

1

set(x, 6)

1

set(x, 5) set(x, 6)

1

...

2

...

3

...

4

...

2

SLIDE 54

Protocol

set(x, 5) set(x, 5) set(x, 6)

1

set(x, 6)

1

set(x, 5) set(x, 6)

1

...

2

...

3

...

4

...

2

C

5

SLIDE 55

Protocol

set(x, 5) set(x, 5) set(x, 6)

1

set(x, 6)

1

set(x, 5) set(x, 6)

1

...

2

...

3

...

4

...

2

C

5

...

3

...

4

C

5

...

2

...

3

...

4

C

5

set(x, 5) set(x, 6)

1

...

2

...

3

...

4

C

5

set(x, 5) set(x, 6)

1

...

2

...

3

...

4

C

5

SLIDE 56

Protocol

set(x, 5) set(x, 5) set(x, 6)

1

set(x, 6)

1

set(x, 5) set(x, 6)

1

...

2

...

2

...

3

...

4

C

5

SLIDE 57

Protocol

set(x, 5) set(x, 5) set(x, 6)

1

set(x, 6)

1

set(x, 5) set(x, 6)

1

...

2

...

2

...

3

...

4

C

5

SLIDE 58

Protocol

set(x, 5) set(x, 5) set(x, 6)

1

set(x, 6)

1

set(x, 5) set(x, 6)

1

...

2

...

2

...

3

...

4

C

5 1

...

3

SLIDE 59

Protocol

set(x, 5) set(x, 5) set(x, 6)

1

set(x, 6)

1

set(x, 5) set(x, 6)

1

...

2

...

2

...

3

...

4

C

5 1

...

3

...

2 1

...

3

SLIDE 60

Protocol

set(x, 5) set(x, 5) set(x, 6)

1

set(x, 6)

1

set(x, 5) set(x, 6)

1

...

2

...

2 1

...

3

...

2 1

...

3

SLIDE 61

Protocol

set(x, 5) set(x, 5) set(x, 6)

1

set(x, 6)

1

set(x, 5) set(x, 6)

1

...

2

...

2 1

...

3

...

2 1

...

3

...

2 1

...

3

SLIDE 62

Protocol

set(x, 5) set(x, 5) set(x, 6)

1

set(x, 6)

1

set(x, 5) set(x, 6)

1

...

2

...

2 1

...

3

...

2 1

...

3

...

2 1

...

3

What happens now?

SLIDE 63

Protocol

set(x, 5) set(x, 5) set(x, 6)

1

set(x, 6)

1

set(x, 5) set(x, 6)

1

...

2

...

3

...

4

...

2

SLIDE 64

Protocol

set(x, 5) set(x, 5) set(x, 6)

1

set(x, 6)

1

set(x, 5) set(x, 6)

1

...

2

...

3

...

4

...

2

SLIDE 65

Protocol

set(x, 5) set(x, 5) set(x, 6)

1

set(x, 6)

1

set(x, 5) set(x, 6)

1

...

2

...

3

...

4

...

2

C-all

5

SLIDE 66

Protocol

set(x, 5) set(x, 5) set(x, 6)

1

set(x, 6)

1

set(x, 5) set(x, 6)

1

...

2

...

3

...

4

...

2

C-all

5

...

3

...

4

...

2

...

3

...

4

C-all

5

set(x, 5) set(x, 6)

1

...

2

...

3

...

4

C-all

5

set(x, 5) set(x, 6)

1

...

2

...

3

...

4

C-all

5

SLIDE 67

Protocol

set(x, 5) set(x, 5) set(x, 6)

1

set(x, 6)

1

set(x, 5) set(x, 6)

1

...

2

...

3

...

4

...

2

C-all

5

...

3

...

4

...

2

...

3

...

4

C-all

5

set(x, 5) set(x, 6)

1

...

2

...

3

...

4

C-all

5

set(x, 5) set(x, 6)

1

...

2

...

3

...

4

C-all

5

C-new

5

SLIDE 68

Protocol

set(x, 5) set(x, 5) set(x, 6)

1

set(x, 6)

1

...

2

...

3

...

4

C-all

5

...

2

...

3

...

4

C-all

5

set(x, 5) set(x, 6)

1

...

2

...

3

...

4

C-all

5

set(x, 5) set(x, 6)

1

...

2

...

3

...

4

C-all

5

C-new

5

C-new

5

C-new

5

C-new

5

SLIDE 69

Failure Detectors

SLIDE 70

What Problem?

We have been depending on random timeouts, etc. to build consensus.

SLIDE 71

What Problem?

We have been depending on random timeouts, etc. to build consensus.
Based on partial synchrony: the network is not always behaving at its worse.

SLIDE 72

What Problem?

We have been depending on random timeouts, etc. to build consensus.
Based on partial synchrony: the network is not always behaving at its worse.
Tedious to model (for proofs) and tune (for deployment).

SLIDE 73

What Problem?

We have been depending on random timeouts, etc. to build consensus.
Based on partial synchrony: the network is not always behaving at its worse.
Tedious to model (for proofs) and tune (for deployment).
Abstract them away with failure detectors.

SLIDE 74

Failure Detector

Application Failure Detector

SLIDE 75

Failure Detector

Application Failure Detector

suspect p0 is failed.

SLIDE 76

Failure Detector

Application Failure Detector

suspect p0 is failed. suspect p0, p1 are failed.

SLIDE 77

Failure Detector

Application Failure Detector

suspect p0 is failed. suspect p0, p1 are failed. suspect p1, p2 are failed.

SLIDE 78

Failure Detector

Application Failure Detector

suspect p0 is failed. suspect p0, p1 are failed. suspect p1, p2 are failed. suspect p1 is failed.

SLIDE 79

Reasoning about Detectors

Completeness Accuracy

SLIDE 80

Reasoning about Detectors

Completeness Accuracy Failed nodes:

When are they detected?
Who detects them?

SLIDE 81

Reasoning about Detectors

Completeness Accuracy Failed nodes:

When are they detected?
Who detects them?

Live nodes:

When can they be suspected?

SLIDE 82

Reasoning about Detectors

Completeness Accuracy Strong Weak

SLIDE 83

Reasoning about Detectors

Completeness Accuracy Strong Weak Every failed node is eventually detected by all correct nodes.

SLIDE 84

Reasoning about Detectors

Completeness Accuracy Strong Weak Every failed node is eventually detected by all correct nodes. Every failed node is eventually detected by some correct nodes.

SLIDE 85

Reasoning about Detectors

Completeness Accuracy Strong Weak Every failed node is eventually detected by all correct nodes. Every failed node is eventually detected by some correct nodes. No correct node is ever suspected.

SLIDE 86

Reasoning about Detectors

Completeness Accuracy Strong Weak Every failed node is eventually detected by all correct nodes. Every failed node is eventually detected by some correct nodes. No correct node is ever suspected. Some correct node is never suspected by any node.

SLIDE 87

Reasoning about Detectors

Accuracy Eventual Not Eventual

Strong Weak

SLIDE 88

Reasoning about Detectors

Accuracy Eventual Not Eventual

Strong Weak Eventually No correct node is ever suspected.

SLIDE 89

Reasoning about Detectors

Accuracy Eventual Not Eventual

Strong Weak Eventually No correct node is ever suspected. No correct node is ever suspected.

SLIDE 90

Reasoning about Detectors

Accuracy Eventual Not Eventual

Strong Weak Eventually No correct node is ever suspected. No correct node is ever suspected. Eventually some correct node is never suspected by any node.

SLIDE 91

Reasoning about Detectors

Accuracy Eventual Not Eventual

Strong Weak Eventually No correct node is ever suspected. No correct node is ever suspected. Eventually some correct node is never suspected by any node. Some correct node is never suspected by any node.

SLIDE 92

Types of Detectors

Strong completeness, strong accuracy: Perfect detector (P)

SLIDE 93

Types of Detectors

Strong completeness, strong accuracy: Perfect detector (P)
Strong completeness, weak accuracy: Strong detector (S)

SLIDE 94

Types of Detectors

Strong completeness, strong accuracy: Perfect detector (P)
Strong completeness, weak accuracy: Strong detector (S)
Strong completeness, eventual strong accuracy: ♢P

SLIDE 95

Types of Detectors

Strong completeness, strong accuracy: Perfect detector (P)
Strong completeness, weak accuracy: Strong detector (S)
Strong completeness, eventual strong accuracy: ♢P
Strong completeness, eventual weak accuracy: ♢S or Ω

SLIDE 96

Types of Detectors

SLIDE 97

Types of Detectors

Weak completeness, strong accuracy: Q

SLIDE 98

Types of Detectors

Weak completeness, strong accuracy: Q
Weak completeness, weak accuracy: Weak Detector (W)

SLIDE 99

Types of Detectors

Weak completeness, strong accuracy: Q
Weak completeness, weak accuracy: Weak Detector (W)
Weak completeness, eventual strong accuracy: ♢Q

SLIDE 100

Types of Detectors

Weak completeness, strong accuracy: Q
Weak completeness, weak accuracy: Weak Detector (W)
Weak completeness, eventual strong accuracy: ♢Q
Weak completeness, eventual weak accuracy: ♢W

SLIDE 101

How to use Failure Detectors?

SLIDE 102