Leases and Cache Coherence Leases Lease - a time-limited right to - - PowerPoint PPT Presentation

▶
leases and cache coherence leases
SMART_READER_LITE
LIVE PREVIEW

Leases and Cache Coherence Leases Lease - a time-limited right to - - PowerPoint PPT Presentation

Jan 04, 2024 258 likes •1.33k views

Leases and Cache Coherence Leases Lease - a time-limited right to do something - can be renewed - unlike Paxos, depends on loosely synchronized clocks Lease fault tolerance - if lease holder or network fails, wait for lease to expire - plus

slide-1
SLIDE 1

Leases and Cache Coherence

slide-2
SLIDE 2

Leases

Lease - a time-limited right to do something

  • can be renewed
  • unlike Paxos, depends on loosely synchronized clocks

Lease fault tolerance

  • if lease holder or network fails, wait for lease to expire
  • plus epsilon to account for clock drift
  • hand lease to someone new
slide-3
SLIDE 3

Paxos as Lease Server

Paxos group as fault tolerant view server

  • grant lease to primary
  • primary serves requests
  • revoke lease if not renewed
  • grant lease to new primary

Design pattern used in GFS, BigTable, …

slide-4
SLIDE 4

Primary election in Chubby, Zookeeper

x = Open(“/BigTable/primary”) if (TryAcquire(x) == success) { // I'm the primary, tell everyone SetContents(x, my-address) } else { // I'm not the primary, find out who is primary = GetContents(x) // also set up notifications // in case the primary changes }

slide-5
SLIDE 5

Example

Paxos Chubby App App Client

slide-6
SLIDE 6

Example

Paxos Chubby App App TryAcquire Client

slide-7
SLIDE 7

Example

Paxos Chubby App App OK Client

slide-8
SLIDE 8

Example

Paxos Chubby App Primary Client

slide-9
SLIDE 9

Example

Paxos Chubby App Primary Client TryAcquire

slide-10
SLIDE 10

Example

Paxos Chubby App Primary Client Nope

slide-11
SLIDE 11

Example

Paxos Chubby Backup Primary Client

slide-12
SLIDE 12

Example

Paxos Chubby Backup Primary Client GetContents

slide-13
SLIDE 13

Example

Paxos Chubby Backup Primary Client Primary

slide-14
SLIDE 14

Example

Paxos Chubby Backup Primary Client

slide-15
SLIDE 15

Example

Paxos Chubby Backup Primary Client Requests

slide-16
SLIDE 16

What if Primary Fails?

Paxos Chubby Backup Primary Client

X

slide-17
SLIDE 17

What if Primary Fails?

Paxos Chubby Backup Primary Client

X

slide-18
SLIDE 18

What if Primary Fails?

Paxos Chubby Backup Primary Client

X

TryAcquire

slide-19
SLIDE 19

What if Primary Fails?

Paxos Chubby Backup Primary Client

X

OK

slide-20
SLIDE 20

What if Primary Fails?

Paxos Chubby Primary Primary Client

X

slide-21
SLIDE 21

Primary Backup With Leases

What if the old primary didn’t crash? Client sends request to old primary What keeps old primary from performing op?

slide-22
SLIDE 22

Primary Backup With Leases

What if the old primary didn’t crash? Client sends request to old primary What keeps old primary from performing op? Old primary demotes itself if it doesn’t renew lease

slide-23
SLIDE 23

Primary Backup with Leases

No possibility of split brain Reads can occur at the primary!

  • no need to talk to backup

Writes can be logged to storage layer

  • on failure, new primary reads latest changes from

storage layer

  • backup is optimization to speed recovery
slide-24
SLIDE 24

Fault Tolerant Caching with Leases

Linearizability with caches is another use of leases Cache obtains lease (ex: read-only) No one can modify data until lease expires or is revoked Once lease expires, ok for server to change

slide-25
SLIDE 25

Caching With Leases

Paxos Chubby Cache 2 Cache 1 Client Client Client Client

slide-26
SLIDE 26

Caching With Leases

Server Cache 2 Cache 1 Client lease Client Client Client Client

slide-27
SLIDE 27

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client

slide-28
SLIDE 28

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Get x

slide-29
SLIDE 29

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Get x

slide-30
SLIDE 30

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1 has x, for t x=3, for t

slide-31
SLIDE 31

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1 has x, for t x= 3, for t

slide-32
SLIDE 32

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1 has x, for t x= 3, for t Get x

slide-33
SLIDE 33

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1 has x, for t x= 3, for t Get x

slide-34
SLIDE 34

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1,2 has x, for t x= 3, for t x=3, for t

slide-35
SLIDE 35

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1,2 has x, for t x= 3, for t x= 3, for t

slide-36
SLIDE 36

Caching With Leases

Why give out cache leases with same values of t? Why give out cache leases with different values of t?

slide-37
SLIDE 37

Caching With Leases

Why give out cache leases with same values of t?

  • less state at server
  • can reclaim leases at same time

Why give out cache leases with different values of t?

  • caches all ask for new lease at same time
slide-38
SLIDE 38

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1,2 has x, for t x= 3, for t x= 3, for t Get x

slide-39
SLIDE 39

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1,2 has x, for t x= 3, for t x= 3, for t x=3

slide-40
SLIDE 40

Caching With Leases

Can clients cache values too?

slide-41
SLIDE 41

Caching With Leases

Can clients cache values too? Yes! Leases can be delegated. Caches keep track as to which clients have which data.

slide-42
SLIDE 42

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1,2 has x, for t x= 3, for t x= 3, for t Put x=4

slide-43
SLIDE 43

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1,2 has x, for t x= 3, for t Put x=4 x= 3, for t

slide-44
SLIDE 44

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1,2 has x, for t x= 3, for t x= 3, for t

slide-45
SLIDE 45

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1,2 has x, for t x= 3, for t x= 3, for t Get x

slide-46
SLIDE 46

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1,2 has x, for t x= 3, for t x= 3, for t x=3

slide-47
SLIDE 47

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client No one has copy of x Ok to change x

slide-48
SLIDE 48

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1,2 has x, for t x= 3, for t x= 3, for t Put x=4

OR

slide-49
SLIDE 49

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1,2 has x, for t x= 3, for t Put x=4

slide-50
SLIDE 50

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 2 has x, for t x= 3, for t

slide-51
SLIDE 51

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 2 has x, for t x= 3, for t Revoke x

slide-52
SLIDE 52

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 2 has x, for t

slide-53
SLIDE 53

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 2 has x, for t OK!

slide-54
SLIDE 54

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client No one has copy of x Ok to change x

slide-55
SLIDE 55

Caching With Leases

Why can’t we leave the old value on cache 1 while we shoot down other copies? Why can’t we just update the old value on cache 1 and then shoot down the other copies?

slide-56
SLIDE 56

Caching With Leases

Why can’t we leave the old value on cache 1 while we shoot down other copies? Why can’t we just update the old value on cache 1 and then shoot down the other copies? Linearizability: as if there is only one copy

  • implement by having only one copy for updates
  • many copies ok when no one is updating
slide-57
SLIDE 57

Caching with Invalidation

Cache obtains lease (read-only) No one can modify data until lease expires or is revoked Server gets update Forwards invalidation (revoke) to every node with copy Wait for response from all (or timeout) OK to proceed with change

slide-58
SLIDE 58

Terminology

Cache coherence: keeping caches up to date

  • can be linearizable, or weaker semantics

Write through: caches hold read-only data

  • write sent to store, store revokes copies

Write back: caches can hold read-only or modified data

  • write to cache, cache asks store to revoke
  • subsequent writes faster
slide-59
SLIDE 59

MSI

Three cache states:

  • Modified: this is the only copy, it’s dirty
  • Shared: this is one of many copies, it’s clean
  • Invalid

Allowed states between pairs of caches:

M S I M ü S ü ü I ü ü ü

slide-60
SLIDE 60

Write Back Fault Tolerance?

Write back: caches can hold modified data What happens when cache fails? Lose data? Option 1: checkpoint/restart if any cache fails

  • appropriate for background computations
  • CPU cache coherence is write-back

Option 2: log local changes to replicas

  • identical to lease to a primary (primary logs changes),

except fine-grained leases

slide-61
SLIDE 61

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Get x

slide-62
SLIDE 62

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Get x

slide-63
SLIDE 63

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1 has x, t, shared x=3, t, shared

slide-64
SLIDE 64

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1 has x, t, shared x= 3, t, shared

slide-65
SLIDE 65

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1 has x, t, shared x= 3, t, shared Put x, 4

slide-66
SLIDE 66

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1 has x, t, shared x= 3, t, shared Put x, 4

slide-67
SLIDE 67

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1 has x, t, shared x= 3, t, shared Need x, dirty

slide-68
SLIDE 68

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1 has x, t, dirty x= 3, t, shared x, dirty

slide-69
SLIDE 69

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1 has x, t, dirty x= 4, t, dirty

slide-70
SLIDE 70

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1 has x, t, dirty x= 4, t, dirty

  • k!
slide-71
SLIDE 71

Caching With Leases

Why does cache 1 wait until other copies are revoked and write is applied before returning ok to client?

slide-72
SLIDE 72

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1 has x, t, dirty x= 4, t, dirty Put x, 5

slide-73
SLIDE 73

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1 has x, t, dirty x= 5, t, dirty

slide-74
SLIDE 74

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1 has x, t, dirty x= 5, t, dirty

  • k!
slide-75
SLIDE 75

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1 has x, t, dirty x= 5, t, dirty get x

slide-76
SLIDE 76

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1 has x, t, dirty x= 5, t, dirty get x

slide-77
SLIDE 77

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1 has x, t, dirty x= 5, t, dirty Revoke x shared

slide-78
SLIDE 78

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1 has x, t, dirty x= 5, t, shared

slide-79
SLIDE 79

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1 has x, t, dirty x= 5, t, shared

  • k! x=5
slide-80
SLIDE 80

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1,2 has x, t, shared x= 5, t, shared

  • k! x= 5
slide-81
SLIDE 81

Caching With Leases

Server Cache 2 Cache 1 Client Client Client Client Client Cache 1,2 has x, t, shared x= 5, t, shared x= 5, t, shared

slide-82
SLIDE 82

Questions

While a write to x is waiting on invalidations, can other clients read old values of x from their caches?

slide-83
SLIDE 83

Questions

While a write to x is waiting on invalidations, can the server perform a read to y != x?

slide-84
SLIDE 84

Questions

While a write to x is waiting on invalidations, can the server perform a write (from another cache) to y != x?

slide-85
SLIDE 85

Questions

While a write to x is waiting on invalidations, can the server perform a write (from another cache) to y = x?

slide-86
SLIDE 86

Write Back Cache Coherence

On a write:

  • Send invalidations to all caches
  • Each cache invalidates, responds


(possibly with updated data)

  • Wait for all invalidations
  • Return

Reads can proceed when there is a local copy Order requests carefully at server, avoid deadlock

slide-87
SLIDE 87

MSI

Invalid Shared Modified

slide-88
SLIDE 88

MSI

Invalid Shared Modified

slide-89
SLIDE 89

MSI

Invalid Shared Modified Read miss

slide-90
SLIDE 90

MSI

Invalid Shared Modified

slide-91
SLIDE 91

MSI

Invalid Shared Modified Write miss

slide-92
SLIDE 92

MSI

Invalid Shared Modified

slide-93
SLIDE 93

MSI

Invalid Shared Modified Local write

slide-94
SLIDE 94

MSI

Invalid Shared Modified

slide-95
SLIDE 95

MSI

Invalid Shared Modified Remote write

slide-96
SLIDE 96

MSI

Invalid Shared Modified

slide-97
SLIDE 97

MSI

Invalid Shared Modified Remote write

slide-98
SLIDE 98

MSI

Invalid Shared Modified

slide-99
SLIDE 99

MSI

Invalid Shared Modified Write back / Remote read

slide-100
SLIDE 100

MESI

Motivation:

  • Common pattern: read, then write
  • MSI inefficient when doing a read and then a write
  • If no one else has a copy, can “claim” it with the read

Four cache states:

  • Modified: this is the only copy, it’s dirty
  • Exclusive: this is the only copy, it’s clean
  • Shared: this is one of many copies, it’s clean
  • Invalid
slide-101
SLIDE 101

MESI allowed states

M E S I M ü E ü S ü ü I ü ü ü ü

slide-102
SLIDE 102

False Sharing

Expensive to keep track of MESI for every memory location Instead, coarse-grained record-keeping

  • CPUs: cache line granularity
  • File systems: file/file block granularity

What if two clients try to modify different memory locations in same block, concurrently?

  • Cache line can only be “dirty" in one at a time
  • Correct behavior, but slow
slide-103
SLIDE 103

Atomic Read-Modify-Write

RMW needed to implement spinlocks and other sync Request cache line exclusive/modified Delay concurrent remote read/write misses until entire

  • peration completes
slide-104
SLIDE 104

Multi-key Transactions

Often want to read/modify multiple keys atomically Acquire cache lines in MESI state If remote miss during transaction

  • Abort, erase modifications, and try again
  • Or delay until done

If reach end of transaction without remote miss

  • Success!
slide-105
SLIDE 105

Weak leases

Cache valid until lease expires Allow writes, other reads simultaneously Semantics?

slide-106
SLIDE 106

Weak leases

Examples: NFS, DNS, web browsers Advantages

  • Stateless at server (don’t care who is caching)
  • Reads, writes always processed immediately

Disadvantages

  • Consistency model (!!!)
  • Overhead of revalidations
  • Synchronized revalidations