Distributed Snapshot One-dollar bank 2 (2,0) (1,2) 1 0 (0,1) - - PowerPoint PPT Presentation

Distributed Snapshot

One-dollar bank

1 2 (0,1) (1,2) (2,0)

Let a $1 coin circulate in a network of a million banks. How can someone count the total $ in circulation? If not counted “properly,” then one may think the total $ in circulation to be one million.

Importance of snapshots

Major uses in

• deadlock detection
• termination detection
• rollback recovery
• global predicate computation

Example 1

• Suppose you want to take a picture of a

scenic view

– Your camera cannot fit the entire scene in

• ne picture

– Take several pictures – Combine them to get overall picture

Example 2

• Suppose you want to take a picture of

basketball game

– Your camera cannot fit the entire scene in

• ne picture

– Take several pictures – Combine them to get overall picture

• Care needs to be taken to ensure that the

several pictures you took are consistent

– E.g., the same player cannot be in two places

Example: Distributed Systems

• You want to take a picture (global

snapshot) of the distributed system

– You can take a picture (local snapshot) of

• ne process at a time

– Need to combine these local snapshots – Need for consistency

Example: Distributed Systems

• Local snapshot

– Can be viewed in terms of the last event on the process

• When we combine such snapshots, we call it a

global snapshot

– Can be viewed in terms of the last event and all preceding events on a process

• When we combine such snapshots, we call it a

(global) cut

Consistent cut

(a ∈ consistent cut C) ∧ (b happened before a) ⇒ b ∈ C a b c d g m e f k i h j Cut 1 Cut 2 A cut is a set of events.

(Not consistent) (Consistent)

P1 P2 P3

Consistent snapshot

The set of states immediately following a consistent cut forms a consistent snapshot

• f a distributed system.
• A snapshot that is of practical interest is the

most recent one. Let C1 and C2 be two consistent cuts and C1 ⊂ C2. Then C2 is more recent than C1.

• Assumption: The cut lines do not go through

any event

Consistent snapshot

How to record a consistent snapshot? Note that 1. The recording must be non-invasive 2. Recording must be done on-the-fly. You cannot stop the system.

Revisit Ring Based Termination Detection

• Ring based termination detection

– Took a snapshot where each process was passive – Snapshot contained c value – Sum of c values was used to detect a property – Color indicated if the snapshot maybe incosnsitent

• Yellow = consistent
• Purple = maybe inconsistent
• We took snapshots until we found one that

was consistent (after system had terminated)

– Here the goal is to take the snapshot immediately upon demand

Chandy-Lamport Algorithm

Works on a (1) strongly connected graph (2) each channel is FIFO. An initiator initiates the algorithm by sending out a marker ( )

White and red processes

Initially every process is white. When a process receives a marker, it turns red if it has not already done so. Every action by a process, and every message sent by a process gets the color of that process.

Two steps

Step 1. In one atomic action, the initiator (a) Turns red

(b) Records its own state (c) sends a marker along all outgoing channels Step 2. Every other process, upon receiving a marker for the first time (and before doing anything else) (a) Turns red (b) Records its own state (c) sends markers along all outgoing channels

The algorithm terminates when (1) every process turns red, and (2) Every process has received a marker through each incoming channel.

Why does it work?

Lemma 1. No red message is received in a white

action.

Why does it work?

• Theorem. The global state recorded by Chandy-Lamport

algorithm is equivalent to the ideal snapshot state SSS.

• Hint. A pair of act ions (a, b) can be scheduled

in any order, if t here is no causal order bet ween t hem, so (a; b) is equivalent t o (b; a)

SSS

Easy conceptualization of the snapshot state All white All red

Why does it work?

Let an observer observe the following actions:

w[i] w[k] r[k] w[j] r[i] w[l] r[j] r[l] … ≡ w[i] w[k] w[j] r[k] r[i] w[l] r[j] r[l] … [Lemma 1] ≡ w[i] w[k] w[j] r[k] w[l] r[i] r[j] r[l] … [Lemma 1] ≡ w[i] w[k] w[j] w[l] r[k] r[i] r[j] r[l] … [done!]

Recorded state

Understanding snapshot

The observed state is a feasible state that is reachable from the initial configuration. It may not actually be visited during a specific execution. The final state of the original computation is always reachable from the observed state.

Discussions

What good is a snapshot if that state has never been visited by the system?

• It is relevant for the detection of stable

predicates.

• Useful for checkpointing.

Discussions

What if the channels are not FIFO? Study how Lai-Yang algorithm works. It does not use any marker

• LY1. The initiator records its own state. When it needs to send a

message m to another process, it sends a message (m, red).

• LY2. When a process receives a message (m, red), it records its

state if it has not already done so, and then accepts the message m. Question 1. Why will it work? Question 1 Are there any limitations of this approach?

Another related problem

Distributed snapshot = distributed read. Distributed reset = distributed write

Global state collection

Some applications

• computing network topology
• termination detection
• deadlock detection

Chandy Lamport algorithm does a partial job. Each process collects a fragment of the global state, but these pieces have to be stitched together to form a global state. All to all broadcast can be achieved via computation similar to diffusing computation

Recall: Global State

• The global state of a system consists of

– One local state for each process

• Contains all the messages sent and received

upto a point in computation

• A local state could be specified by the

`last’ event on the respective process

Consistency in Global State

• Consistent iff

– If reception of any message is recorded in the global state then the corresponding send is also recorded

• If global snapshot is consistent then

what is the causal relation between the `last’ events of respective processes?

– Why?

Snapshot with Time

• Everyone take a local snapshot a 5pm

– Is this consistent?

Snapshot with Time

• Suppose we use hybrid logical clocks
• Consider the algorithm

– Everyone take a snapshot when the HLC value reaches l = 100, c = 0

• Is it consistent?

– What if process goes from

• l=99 to l = 100,c>1?

Snapshot with Logical Clocks

• Suppose we use Lamport’s clocks

– Take a snapshot when logical clock = 100

• Is it consistent?
• What is the problem?

Application of Global State Detection

• Termination detection
• Checkpointing and recovery