[R EPLICATION & C ONSISTENCY ] Shrideep Pallickara Computer - - PDF document

▶
r eplication c onsistency
SMART_READER_LITE
LIVE PREVIEW

[R EPLICATION & C ONSISTENCY ] Shrideep Pallickara Computer - - PDF document

May 11, 2023 375 likes •646 views

CS555: Distributed Systems [Fall 2019] Dept. Of Computer Science , Colorado State University CS 555: D ISTRIBUTED S YSTEMS [R EPLICATION & C ONSISTENCY ] Shrideep Pallickara Computer Science Colorado State University CS555: Distributed

slide-1
SLIDE 1

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.1

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS 555: DISTRIBUTED SYSTEMS

[REPLICATION & CONSISTENCY]

Shrideep Pallickara Computer Science Colorado State University

November 5, 2019

L21.1 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.2 Professor: SHRIDEEP PALLICKARA

Frequently asked questions from the previous class survey

November 5, 2019

slide-2
SLIDE 2

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.2

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.3 Professor: SHRIDEEP PALLICKARA

Topics covered in this lecture

¨ Replica Management ¨ Consistency protocols ¨ Primary based protocols ¨ Replicated write protocols

November 5, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

REPLICA MANAGEMENT

November 5, 2019

L21.4

slide-3
SLIDE 3

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.3

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.5 Professor: SHRIDEEP PALLICKARA

Key issues in replication within a distributed system

¨ Placement of replicas ¤ When, where and by whom ¨ Placement of content ¨ Mechanisms to keep replicas consistent

November 5, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.6 Professor: SHRIDEEP PALLICKARA

Replica-Server Placement

¨ K out of N possible locations (K < N) ¨ Optimization problem ¤ Computationally complex ¨ Solved only through heuristics

November 5, 2019

slide-4
SLIDE 4

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.4

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.7 Professor: SHRIDEEP PALLICKARA

Replica-Server placements: Distance between client and locations

November 5, 2019

¨ In terms of latency or bandwidth ¨ If k servers have been placed, there are N-k locations ¤ Select server one at a time ¤ Minimize average distance between server and its client

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.8 Professor: SHRIDEEP PALLICKARA

Replica Server placements Use Internet topology

November 5, 2019

¨ Autonomous Systems ¤ Network in which all nodes run the same protocol ¤ About 20,000 AS ¨ Within an AS place server on router with largest NIC/links ¤ Recursively repeat for other AS

slide-5
SLIDE 5

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.5

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.9 Professor: SHRIDEEP PALLICKARA

Problems with these approaches

¨ Computing placements is expensive ¨ Does not work well with flash-crowds

November 5, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.10 Professor: SHRIDEEP PALLICKARA

Arriving at quick replica placements

¨ Region/cell is a collection of nodes accessing the same content ¤ Inter-node latency is low ¨ Select most demanding regions ¤ With largest number of nodes ¤ One of the nodes acts as a replica

November 5, 2019

slide-6
SLIDE 6

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.6

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.11 Professor: SHRIDEEP PALLICKARA

Quick replica server placements

November 5, 2019

¨ Nodes are in an m-dimensional geometric space ¨ Identify K largest clusters ¤ Assign node from each cluster to host replica

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

TYPES OF REPLICAS

November 5, 2019

L21.12

slide-7
SLIDE 7

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.7

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.13 Professor: SHRIDEEP PALLICKARA

Types of Replicas

Permanent Replicas

Server-initiated Replicas Client initiated Replicas

November 5, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.14 Professor: SHRIDEEP PALLICKARA

Permanent Replicas

¨ Initial set of replicas that comprise data store ¤ Usually a small set ¨ Files stored across servers at a single location ¤ Request forwarded using round-robin strategy ¨ Files copied to mirror sites ¤ Geographically dispersed

November 5, 2019

slide-8
SLIDE 8

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.8

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.15 Professor: SHRIDEEP PALLICKARA

Server initiated replicas

¨ Copies that exist to enhance performance ¨ Created at the initiative of the owner of data store

November 5, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.16 Professor: SHRIDEEP PALLICKARA

Server initiated replicas: Example

¨ Web server in NYC ¤ Can handle dissemination loads effectively ¨ Bursts of traffic over 2-3 days may come in ¤ From some specific location (or set of locations) ¨ Install temporary replicas in regions where requests originate

November 5, 2019

slide-9
SLIDE 9

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.9

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.17 Professor: SHRIDEEP PALLICKARA

Server initiated replicas: Issues in dynamic replications

¨ Replication takes place to reduce load at server ¨ Specific files on server migrated/replicated to servers in proximity of

requesting clients

November 5, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.18 Professor: SHRIDEEP PALLICKARA

Dynamic replication: Migrating/replicating files

¨ Each server tracks access counts per file ¤ And also who initiates accesses ¨ Given a client C ¤ Each server can determine which of the servers is closest to C

November 5, 2019

slide-10
SLIDE 10

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.10

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.19 Professor: SHRIDEEP PALLICKARA

Counting access requests from clients: C1 and C2 share closest server P

C2 C1

File F Server Q Without copy of F Server P

  • Accesses from C1, C2 for file F at server Q are

registered as if they are from P

§ countQ(P, F)

November 5, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.20 Professor: SHRIDEEP PALLICKARA

Replication threshold: rep(S, F)

¨ Indicates number of requests for file is high ¨ Might be worth replicating it

November 5, 2019

slide-11
SLIDE 11

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.11

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.21 Professor: SHRIDEEP PALLICKARA

Deletion thresholds

November 5, 2019

¨ When requests for file F at server S drops below deletion threshold,

del(S,F)

¤ File F removed from S ¨ Number of replicas reduce ¨ Higher loads at the other servers ¨ Ensure at least one copy of file continues to exist

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.22 Professor: SHRIDEEP PALLICKARA

More on replication and deletion thresholds

¨ rep(S, F) always chosen to be higher than the del(S, F) ¨ If a number of requests lie between deletion and replication threshold ¤ File can only be migrated ¤ Number of replicas for file should be the same

November 5, 2019

slide-12
SLIDE 12

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.12

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.23 Professor: SHRIDEEP PALLICKARA

Reevaluating the placement of files at a server Q

November 5, 2019

¨ Check access count for each file ¨ If number of accesses < del(Q, F) ? ¤ File deleted unless it is the last copy ¨ For some server P, if countQ(P, F) is more than ½ of requests for F at

Q ?

¤ Server P is requested to take over copy of F ¤ Migration

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.24 Professor: SHRIDEEP PALLICKARA

Migration/replication of a file may not always succeed

¨ Server P might already be heavily overloaded ¨ Q will then attempt to replicate F elsewhere § Number of access > rep(Q, F) ¨ If countQ(R, F) exceeds a certain fraction of all requests for F at Q ¤ Try to replicate at R

November 5, 2019

slide-13
SLIDE 13

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.13

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.25 Professor: SHRIDEEP PALLICKARA

Client initiated replicas: Client cache

¨ Temporarily store data that was just requested ¤ Could be on client’s machine or nearby machine ¨ Used to improve access times ¨ Data kept in cache for a limited time ¤ Avoid stale data problem ¤ Make room for other data ¨ To improve cache hits; cache may be shared between clients

November 5, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CONSISTENCY PROTOCOLS

November 5, 2019

L21.26

slide-14
SLIDE 14

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.14

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.27 Professor: SHRIDEEP PALLICKARA

Consistency Protocols

¨ Describes an implementation of a consistency model ¨ Models we looked at ¤ Data centric ¤ Client centric

November 5, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.28 Professor: SHRIDEEP PALLICKARA

Continuous consistency protocols: Bounding

¨ Numerical deviations ¨ Staleness deviations ¨ Ordering deviations

November 5, 2019

slide-15
SLIDE 15

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.15

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.29 Professor: SHRIDEEP PALLICKARA

Continuous consistency: Bounding numerical deviations

¨ Writes on a single data item x ¨ Each write is submitted to one of N available servers

November 5, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.30 Professor: SHRIDEEP PALLICKARA

Several submitted writes need to be propagated to all servers

¨ TW[i, j] : write executed by server Si that originated at Sj ¨ TW[i, i] aggregate writes submitted to Si ¤ By clients

November 5, 2019

slide-16
SLIDE 16

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.16

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.31 Professor: SHRIDEEP PALLICKARA

Contrasting the values of x

November 5, 2019

¨ Actual value v of x at time t: v(t) § v(0) is the initial value of x ¨ v(t) is determined by all the submitted writes

v(t) = v(0) + TW [k,k]

k=1 N

∑

vi = v(0) + TW [i,k]

k=1 N

∑

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.32 Professor: SHRIDEEP PALLICKARA

Bounding numerical deviations

November 5, 2019

¨ The upper bound on deviations that needs to be enforced

v(t) – vi ≤ δi

¨ Server Si propagates updates from Sj to Sk ¤ Sk maintains a view TWk[i, j] ¨ When Sk notices Si has not been keeping up with updates ¤ Propagate updates from its log

slide-17
SLIDE 17

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.17

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.33 Professor: SHRIDEEP PALLICKARA

Bounding staleness deviation

¨ Maintain real-time vector clock ¨ RVCk[i] = T(i) ¤ Sk has seen all writes at Si up to time Ti ¤ Ti is time local to Si ¨ At Sk when T(k) – RVCk[i] > staleness_limit § Pull writes at Si with timestamp > RVCk[i]

November 5, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.34 Professor: SHRIDEEP PALLICKARA

Bounding ordering deviations

November 5, 2019

¨ Each server has a local copy of tentative writes ¤ Actual order in which they should be applied is not yet determined ¨ Limit on the maximum length of the queue

slide-18
SLIDE 18

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.18

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.35 Professor: SHRIDEEP PALLICKARA

When the maximum queue length limit is reached

November 5, 2019

¨ No new writes will be accepted ¨ Attempt to commit pending writes ¤ Determine order of these writes from other servers

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

PRIMARY BASED PROTOCOLS

November 5, 2019

L21.36

slide-19
SLIDE 19

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.19

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.37 Professor: SHRIDEEP PALLICKARA

Primary based protocols

November 5, 2019

¨ Used in enforcing sequential consistency ¨ Each data item x has an associated primary ¤ Responsible for coordinating writes on x ¨ Primary may be ¤ Fixed at a remote server ¤ Moved to process where write operations are performed

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.38 Professor: SHRIDEEP PALLICKARA

Primary-based protocols: Remote-Write {Primary backup}

November 5, 2019

¨ To perform write operation on x ¨ Forward write to primary-server for x ¨ Primary performs ¤ Update on its local copy of x ¤ Forwards update to the backup servers

slide-20
SLIDE 20

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.20

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.39 Professor: SHRIDEEP PALLICKARA

Remote-write protocols: Primary back-up protocol

Data Store Primary Server for item x W1 W2 W3 W3 W3 W4 R1 R2

November 5, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.40 Professor: SHRIDEEP PALLICKARA

Local-write protocols

¨ Primary copy migrates between process ¤ From current server to the one wanting to update ¨ To perform write operation on x ¤ Locate primary copy for x ¤ Move primary copy to its own location

November 5, 2019

slide-21
SLIDE 21

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.21

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.41 Professor: SHRIDEEP PALLICKARA

Local-write protocol: Primary back-up protocol

Data Store Old primary for item x R1 W4 W2 W4 R2 W1 W3 New primary for item x W4

November 5, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

REPLICATED WRITE PROTOCOLS

November 5, 2019

L21.42

slide-22
SLIDE 22

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.22

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.43 Professor: SHRIDEEP PALLICKARA

Replicated write protocols

November 5, 2019

¨ Write operations are carried out at multiple replicas ¤ Not just 1 (or primary) ¨ Active Replication ¤ Operation forwarded to all replicas ¨ Quorum-based ¤ Based on majority voting

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.44 Professor: SHRIDEEP PALLICKARA

Active Replication

November 5, 2019

¨ Operation is sent to each replica ¨ Must be carried out in same order everywhere ¤ Lamport’s clocks ¤ Use of a central coordinator: Sequencer n Could start to resemble primary-based protocols

slide-23
SLIDE 23

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.23

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.45 Professor: SHRIDEEP PALLICKARA

Quorum-based protocols:

Clients must request and acquire permissions

¨ From multiple servers ¨ Before reading and writing replicated data items

November 5, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.46 Professor: SHRIDEEP PALLICKARA

Quorum-based protocols: Distributed File System example {Write}

November 5, 2019

¨ File is replicated on N servers ¨ To update a file ¤ Client must contact at least (N/2 + 1) servers n Majority ¤ Get them to agree to do the update ¨ Upon agreement ¤ File is changed and version number incremented

slide-24
SLIDE 24

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.24

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.47 Professor: SHRIDEEP PALLICKARA

Quorum-based protocols: Reading a replicated file

November 5, 2019

¨ Client must contact at least (N/2 + 1) servers ¤ Ask them for version numbers of file ¨ If version numbers agree … most recent version ¨ With N=5, and ¤ Clients see 3 responses with version-8 ¤ Then getting 2 responses with verison-9? n Impossible, because update to version-9 needs 3 to agree

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.48 Professor: SHRIDEEP PALLICKARA

Quorum-based protocols: When there are N replicas

November 5, 2019

¨ Read quorum NR ¨ To modify a file write-quorum NW ¨ NR + NW > N ¤ Prevent read-write conflict ¨ NW > N/2 ¤ Prevent write-write conflict

slide-25
SLIDE 25

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.25

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.49 Professor: SHRIDEEP PALLICKARA

A B C D E F G H I J K L

Quorum-based protocols: Example 1

NR=3 NW=10

A B C D E F G H I J K L

NR=7 NW=6

J

L

Write-write conflict

Concurrent writes to {A, B, C, E, F, G} and {D, H, I, J, K, L} will be accepted Read Quorum: Write Quorum:

November 5, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.50 Professor: SHRIDEEP PALLICKARA

A B C D E F G H I J K L

Quorum-based protocols: Example 2

NR=1 NW=12

J

Read Quorum: Write Quorum:

November 5, 2019

slide-26
SLIDE 26

SLIDES CREATED BY: SHRIDEEP PALLICKARA L21.26

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L21.51 Professor: SHRIDEEP PALLICKARA

The contents of this slide-set are based on the following references

November 5, 2019 ¨ Distributed Systems: Principles and Paradigms. Andrew S. Tanenbaum and Maarten Van

  • Steen. 2nd Edition. Prentice Hall. ISBN: 0132392275/978-0132392273. [Chapter 7]