Persistence Semantics for Weak Memory Integrating Epoch Persistency - - PowerPoint PPT Presentation

β–Ά
persistence semantics for weak memory
SMART_READER_LITE
LIVE PREVIEW

Persistence Semantics for Weak Memory Integrating Epoch Persistency - - PowerPoint PPT Presentation

Persistence Semantics for Weak Memory Integrating Epoch Persistency with the TSO Memory Model Azalea Raad Viktor Vafeiadis Max Planck Institute for Software Systems (MPI-SWS) Thursday 8 November OOPSLA 2018 Boston, USA azalea@mpi-sws.org


slide-1
SLIDE 1

Persistence Semantics for Weak Memory

Integrating Epoch Persistency with the TSO Memory Model

Azalea Raad Viktor Vafeiadis

Max Planck Institute for Software Systems (MPI-SWS)

Thursday 8 November OOPSLA 2018 Boston, USA

azalea@mpi-sws.org @azalearaad SoundAndComplete.org

slide-2
SLIDE 2

History

time Difficulty Sequential

πŸ˜‹

slide-3
SLIDE 3

History

time Difficulty Sequential

πŸ˜‹

SC

πŸ˜‘

slide-4
SLIDE 4

History

time Difficulty Sequential

πŸ˜‹

SC

πŸ˜‘

WMC

☹

slide-5
SLIDE 5

History

time Difficulty Sequential

πŸ˜‹

SC

πŸ˜‘

WMC

☹ 😀

Persistent WMC

slide-6
SLIDE 6

Volatile memory

x := 1 // x = 1 // x = 0

What is Persistent Memory?

3

// x = v : reading x yields v

slide-7
SLIDE 7

Volatile memory

x := 1 // x = 1 // x = 0

What is Persistent Memory?

3

// x = 0 // no recovery // x = v : reading x yields v

slide-8
SLIDE 8

Volatile memory

x := 1 // x = 1 // x = 0

What is Persistent Memory?

3

// x = 0 // no recovery // x = v : reading x yields v

Persistent memory

x := 1 // x = 1 // x = 0 // x = 0 OR x = 1 // recovery routine

slide-9
SLIDE 9

Volatile memory

x := 1 // x = 1 // x = 0

What is Persistent Memory?

3

// x = 0 // no recovery // x = v : reading x yields v

Persistent memory

x := 1 // x = 1 // x = 0 // x = 0 OR x = 1 // recovery routine

persists are asynchronous (buffered): may not persist immediately

slide-10
SLIDE 10

(Sequential) Hardware

slide-11
SLIDE 11

CPU (Volatile) Memory

(Sequential) Hardware

slide-12
SLIDE 12

CPU (Volatile) Memory

(Sequential) Hardware

x:=1 : adds x:=1 to memory

slide-13
SLIDE 13

CPU (Volatile) Memory

(Sequential) Hardware

x:=1 : adds x:=1 to memory a:=x : reads x from memory

slide-14
SLIDE 14

CPU (Volatile) Memory

(Sequential) Hardware

x:=1 : adds x:=1 to memory a:=x : reads x from memory memory lost

slide-15
SLIDE 15

Persistence Buffer CPU (Persistent) Memory CPU (Volatile) Memory

(Sequential) Hardware

x:=1 : adds x:=1 to memory a:=x : reads x from memory memory lost

slide-16
SLIDE 16

Persistence Buffer CPU (Persistent) Memory CPU (Volatile) Memory

(Sequential) Hardware

x:=1 : adds x:=1 to memory a:=x : reads x from memory memory lost x:=1 : adds x:=1 to p-buffer

slide-17
SLIDE 17

Persistence Buffer CPU (Persistent) Memory CPU (Volatile) Memory

(Sequential) Hardware

x:=1 : adds x:=1 to memory a:=x : reads x from memory memory lost x:=1 : adds x:=1 to p-buffer a:=x : if p-buffer contains x, reads latest entry else reads from memory

slide-18
SLIDE 18

Persistence Buffer CPU (Persistent) Memory CPU (Volatile) Memory

(Sequential) Hardware

x:=1 : adds x:=1 to memory a:=x : reads x from memory memory lost x:=1 : adds x:=1 to p-buffer a:=x : if p-buffer contains x, reads latest entry else reads from memory p-buffer lost; memory retained

slide-19
SLIDE 19

Persistence Buffer CPU (Persistent) Memory CPU (Volatile) Memory

(Sequential) Hardware

x:=1 : adds x:=1 to memory a:=x : reads x from memory memory lost x:=1 : adds x:=1 to p-buffer a:=x : if p-buffer contains x, reads latest entry else reads from memory p-buffer lost; memory retained unbuffer* : p-buffer to memory * at non-deterministic times

slide-20
SLIDE 20
  • Memory consistency model describes: 


the order writes are made visible to other threads


e.g. SC, TSO, …

What is Memory Persistency Model?

slide-21
SLIDE 21
  • Memory consistency model describes: 


the order writes are made visible to other threads


e.g. SC, TSO, …

  • Memory persistency model describes: 


the order writes are persisted to memory


e.g. Epoch Persistency

What is Memory Persistency Model?

slide-22
SLIDE 22
  • Memory consistency model describes: 


the order writes are made visible to other threads


e.g. SC, TSO, …

  • Memory persistency model describes: 


the order writes are persisted to memory


e.g. Epoch Persistency

What is Memory Persistency Model?

Problem Formal Epoch Persistency Model for Mainstream Hardware (Weak Memory Models)

slide-23
SLIDE 23

x := 1; // recovery routine // x=0;y=0 y := 1; // x=0;y=0 OR x=1;y=1 OR x=1;y=0 OR x=0;y=1

What Can Go Wrong?

6

slide-24
SLIDE 24

x := 1; // recovery routine // x=0;y=0 y := 1; // x=0;y=0 OR x=1;y=1 OR x=1;y=0 OR x=0;y=1

What Can Go Wrong?

6

!! Writes may persist out of order

slide-25
SLIDE 25

x := 1; // recovery routine // x=0;y=0 y := 1; // x=0;y=0 OR x=1;y=1 OR x=1;y=0 OR x=0;y=1

What Can Go Wrong?

6

!! Writes may persist out of order β˜› persistent fence pfence

slide-26
SLIDE 26

7

x := 1; // recovery routine // x=0;y=0 y := 1; // x=0;y=0 OR x=1;y=1 OR x=1;y=0 OR x=0;y=1 pfence;

β˜›

Persistent Fence

slide-27
SLIDE 27

Persistent Fence

8

x := 1; z := 4; pfence; y := 2; x := 3;

a b c d

slide-28
SLIDE 28

Persistent Fence

8

  • writes on same locations persist in execution order

x := 1; z := 4; pfence; y := 2; x := 3;

a b c d a c

persists before

slide-29
SLIDE 29

Persistent Fence

8

  • writes on same locations persist in execution order
  • writes on different locations are unordered

x := 1; z := 4; pfence; y := 2; x := 3;

a b c d a b

may persist in any order

a c

persists before

slide-30
SLIDE 30

Persistent Fence

8

  • writes on same locations persist in execution order
  • writes on different locations are unordered
  • pfence adds a new epoch

x := 1; z := 4; pfence; y := 2; x := 3;

a b c d a b

may persist in any order

a c

persists before

epoch 1 epoch 2

slide-31
SLIDE 31

Persistent Fence

8

  • writes on same locations persist in execution order
  • writes on different locations are unordered
  • pfence adds a new epoch
  • writes persist in epoch order

x := 1; z := 4; pfence; y := 2; x := 3;

a b c d

persist before

a b c d a b

may persist in any order

a c

persists before

epoch 1 epoch 2

slide-32
SLIDE 32

// x=0;y=0 OR x=1;y=1 OR x=1;y=0 x := 1; // recovery routine // x=0;y=0 y := 1; pfence;

9

What Can Go Wrong (Continued)?

!! Execution continues ahead of persistence

asynchronous (buffered)

slide-33
SLIDE 33

// x=0;y=0 OR x=1;y=1 OR x=1;y=0 x := 1; // recovery routine // x=0;y=0 y := 1; pfence;

9

What Can Go Wrong (Continued)?

!! Execution continues ahead of persistence β˜› persistent sync psync

asynchronous (buffered)

slide-34
SLIDE 34

// x=0;y=0 OR x=1;y=1 OR x=1;y=0 x := 1; // recovery routine // x=0;y=0 y := 1; pfence;

9

What Can Go Wrong (Continued)?

!! Execution continues ahead of persistence β˜› persistent sync psync

C1; psync; C2

  • same persist-ordering as pfence
  • C2 executed only when all C1 writes have persisted

asynchronous (buffered)

slide-35
SLIDE 35

x := 1; // recovery routine // x=0;y=0 y := 1; // x=0;y=0 OR x=1;y=1 OR x=1;y=0 psync;

10

Persistent Sync

!! Execution continues ahead of persistence β˜› persistent sync psync

C1; psync; C2

  • same persist-ordering as pfence
  • C2 executed only when all C1 writes have persisted

β˜›

slide-36
SLIDE 36

(Sequential) Hardware

x:=1 : adds x:=1 to p-buffer a:=x : if p-buffer contains x, reads latest entry else reads from memory p-buffer lost; memory retained

11

CPU (Persistent) Memory

epoch n epoch 2 pfence pfence . . . pfence epoch 1

slide-37
SLIDE 37

(Sequential) Hardware

x:=1 : adds x:=1 to p-buffer a:=x : if p-buffer contains x, reads latest entry else reads from memory unbuffer* : p-buffer to memory (in epoch order) p-buffer lost; memory retained * at non-deterministic times

11

CPU (Persistent) Memory

epoch n epoch 2 pfence pfence . . . pfence epoch 1

slide-38
SLIDE 38

(Sequential) Hardware

x:=1 : adds x:=1 to p-buffer a:=x : if p-buffer contains x, reads latest entry else reads from memory unbuffer* : p-buffer to memory (in epoch order) p-buffer lost; memory retained * at non-deterministic times pfence : introduces a new epoch in p-buffer

11

CPU (Persistent) Memory

epoch n epoch 2 pfence pfence . . . pfence epoch 1

slide-39
SLIDE 39

(Sequential) Hardware

x:=1 : adds x:=1 to p-buffer a:=x : if p-buffer contains x, reads latest entry else reads from memory unbuffer* : p-buffer to memory (in epoch order) p-buffer lost; memory retained * at non-deterministic times pfence : introduces a new epoch in p-buffer psync : flushes the entire p-buffer to memory

11

CPU (Persistent) Memory

epoch n epoch 2 pfence pfence . . . pfence epoch 1

slide-40
SLIDE 40

What about Concurrency?

TSO POWER ARMv8

…

12

slide-41
SLIDE 41

What about Concurrency?

TSO POWER ARMv8

…

12

slide-42
SLIDE 42

Contributions

13

slide-43
SLIDE 43
  • PTSO: First formal epoch persistency semantics under 


mainstream hardware

  • Operational model
  • Declarative model
  • Equivalence of the two models

Contributions

13

slide-44
SLIDE 44
  • PTSO: First formal epoch persistency semantics under 


mainstream hardware

  • Operational model
  • Declarative model
  • Equivalence of the two models

Contributions

13

slide-45
SLIDE 45
  • PTSO: First formal epoch persistency semantics under 


mainstream hardware

  • Operational model
  • Declarative model
  • Equivalence of the two models
  • Verifying programs under PTSO
  • PTSO programming pattern
  • Correctness condition: persistent linearisability
  • Verified several examples under PTSO

Contributions

13

slide-46
SLIDE 46

Total Store Ordering (TSO)

14

slide-47
SLIDE 47

Total Store Ordering (TSO)

14

Thread2 Buffer (Volatile) Memory Thread1 Buffer

slide-48
SLIDE 48

Total Store Ordering (TSO)

14

Thread2 Thread1

x = 0; y = 0;

x := 1; a := y;

Thread1

y := 1; c := x;

Thread2

β˜› β˜›

Store Buffering (SB)

slide-49
SLIDE 49

Total Store Ordering (TSO)

14

Thread2 Thread1

x = 0; y = 0;

x := 1; a := y;

Thread1

y := 1; c := x;

Thread2

x = 1

β˜› β˜›

Store Buffering (SB)

slide-50
SLIDE 50

Total Store Ordering (TSO)

14

Thread2 Thread1

x = 0; y = 0;

x := 1; a := y;

Thread1

y := 1; c := x;

Thread2

x = 1 y = 1

β˜› β˜›

Store Buffering (SB)

slide-51
SLIDE 51

Total Store Ordering (TSO)

14

Thread2 Thread1

x = 0; y = 0;

x := 1; a := y;

Thread1

y := 1; c := x;

Thread2

x = 1 y = 1

β˜› β˜›

Store Buffering (SB)

slide-52
SLIDE 52

Total Store Ordering (TSO)

14

Thread2 Thread1

x = 0; y = 0;

x := 1; a := y;

Thread1

y := 1; c := x;

Thread2

x = 1 y = 1

β˜› β˜›

Store Buffering (SB)

slide-53
SLIDE 53

Total Store Ordering (TSO)

14

Thread2 Thread1

x = 0; y = 0;

x := 1; a := y;

Thread1

y := 1; c := x;

Thread2

x = 1 y = 1

β˜› β˜›

// 0 Store Buffering (SB)

slide-54
SLIDE 54

Total Store Ordering (TSO)

14

Thread2 Thread1

x = 0; y = 0;

x := 1; a := y;

Thread1

y := 1; c := x;

Thread2

x = 1 y = 1

β˜› β˜›

// 0 Store Buffering (SB)

slide-55
SLIDE 55

Total Store Ordering (TSO)

14

Thread2 Thread1

x = 0; y = 0;

x := 1; a := y;

Thread1

y := 1; c := x;

Thread2

x = 1 y = 1

β˜› β˜›

// 0 Store Buffering (SB)

slide-56
SLIDE 56

Total Store Ordering (TSO)

14

Thread2 Thread1

x = 0; y = 0;

x := 1; a := y;

Thread1

y := 1; c := x;

Thread2

x = 1 y = 1

β˜› β˜›

// 0 // 0 Store Buffering (SB)

slide-57
SLIDE 57

Total Store Ordering (TSO)

14

Thread2 Thread1

x := 1; a := y;

Thread1

y := 1; c := x;

Thread2

x = 1 y = 1

β˜› β˜›

// 0 // 0 Store Buffering (SB)

; y = 0;

slide-58
SLIDE 58

Total Store Ordering (TSO)

14

Thread2 Thread1

x := 1; a := y;

Thread1

y := 1; c := x;

Thread2

x = 1

β˜› β˜›

// 0 // 0 Store Buffering (SB)

; y = 1;

slide-59
SLIDE 59

Persistent TSO (PTSO)

15

(Volatile) Memory Thread1 Buffer Thread2 Buffer (Volatile) Memory CPU

slide-60
SLIDE 60

Persistent TSO (PTSO)

15

(Persistent) Memory CPU Persistence Buffer (Volatile) Memory Thread1 Buffer Thread2 Buffer (Volatile) Memory CPU

slide-61
SLIDE 61

Persistent TSO (PTSO)

15

(Persistent) Memory Thread1 Buffer Persistence Buffer Thread2 Buffer (Persistent) Memory CPU Persistence Buffer (Volatile) Memory Thread1 Buffer Thread2 Buffer (Volatile) Memory CPU

slide-62
SLIDE 62
  • PTSO: First formal epoch persistency semantics under 


mainstream hardware

  • Operational model
  • Declarative model
  • Equivalence of the two models

Contributions

16

slide-63
SLIDE 63
  • PTSO: First formal epoch persistency semantics under 


mainstream hardware

  • Operational model
  • Declarative model
  • Equivalence of the two models

Contributions

16

  • Verifying programs under PTSO
  • PTSO programming pattern
  • Correctness condition: persistent linearisability
  • Verified several examples under PTSO
slide-64
SLIDE 64
  • PTSO: First formal epoch persistency semantics under 


mainstream hardware

  • Operational model
  • Declarative model
  • Equivalence of the two models

Contributions

16

  • Verifying programs under PTSO
  • PTSO programming pattern
  • Correctness condition: persistent linearisability
  • Verified several examples under PTSO
slide-65
SLIDE 65

Verifying programs under PTSO

The persistent variant of the Michael-Scott queue and its recovery mechanism

slide-66
SLIDE 66

Verifying programs under PTSO

The persistent variant of the Michael-Scott queue and its recovery mechanism

What constitutes a correct persistent implementation?

slide-67
SLIDE 67

Linearisability

18

thread 1 thread 2 enq(1)

a

deq(1)

c

enq(2)

b

time

slide-68
SLIDE 68

Linearisability

  • Define happens-before relation hb
  • (e1, e2) ∈ hb ⟺ e1.end <time e2.begin

18

thread 1 thread 2 enq(1)

a

deq(1)

c

enq(2)

b

time

slide-69
SLIDE 69

Linearisability

  • Define happens-before relation hb
  • (e1, e2) ∈ hb ⟺ e1.end <time e2.begin

18

thread 1 thread 2 enq(1)

a

deq(1)

c

enq(2)

b

time

  • - e.g. ( , ) ∈ hb ( , ) βˆ‰ hb

a b a c

slide-70
SLIDE 70

Linearisability

  • Define happens-before relation hb
  • (e1, e2) ∈ hb ⟺ e1.end <time e2.begin

18

thread 1 thread 2 enq(1)

a

deq(1)

c

enq(2)

b

time

  • - e.g. ( , ) ∈ hb ( , ) βˆ‰ hb

a b a c

  • H respects hb
  • H is a legal sequence (library-specific)
  • Linearisable ⟺ βˆƒ H. H totally orders events
slide-71
SLIDE 71

Linearisability

  • Define happens-before relation hb
  • (e1, e2) ∈ hb ⟺ e1.end <time e2.begin

18

thread 1 thread 2 enq(1)

a

deq(1)

c

enq(2)

b

time

  • - e.g. ( , ) ∈ hb ( , ) βˆ‰ hb

a b a c

  • H respects hb
  • H is a legal sequence (library-specific)
  • - e.g. FIFO sequences for queue
  • Linearisable ⟺ βˆƒ H. H totally orders events

a c b

βœ”

c a b

✘

a b c

βœ”

slide-72
SLIDE 72

Linearisability

  • Define happens-before relation hb
  • (e1, e2) ∈ hb ⟺ e1.end <time e2.begin

18

thread 1 thread 2 enq(1)

a

deq(1)

c

enq(2)

b

time

  • - e.g. ( , ) ∈ hb ( , ) βˆ‰ hb

a b a c

  • H respects hb
  • H is a legal sequence (library-specific)
  • - e.g. FIFO sequences for queue
  • Linearisable ⟺ βˆƒ H. H totally orders events

a c b

βœ”

c a b

✘

a b c

βœ”

linearisable

slide-73
SLIDE 73

Linearisability

  • Define happens-before relation hb
  • (e1, e2) ∈ hb ⟺ e1.end <time e2.begin

19

thread 1 thread 2 enq(1)

a

deq(2)

c

enq(2)

b

time

  • - e.g. ( , ) ∈ hb ( , ) βˆ‰ hb

a b a c

  • H respects hb
  • H is a legal sequence (library-specific)
  • - e.g. FIFO sequences for queue
  • Linearisable ⟺ βˆƒ H. H totally orders events
slide-74
SLIDE 74

Linearisability

  • Define happens-before relation hb
  • (e1, e2) ∈ hb ⟺ e1.end <time e2.begin

19

thread 1 thread 2 enq(1)

a

deq(2)

c

enq(2)

b

time

  • - e.g. ( , ) ∈ hb ( , ) βˆ‰ hb

a b a c

  • H respects hb
  • H is a legal sequence (library-specific)
  • - e.g. FIFO sequences for queue
  • Linearisable ⟺ βˆƒ H. H totally orders events

a c b c a b a b c

slide-75
SLIDE 75

Linearisability

  • Define happens-before relation hb
  • (e1, e2) ∈ hb ⟺ e1.end <time e2.begin

19

thread 1 thread 2 enq(1)

a

deq(2)

c

enq(2)

b

time

  • - e.g. ( , ) ∈ hb ( , ) βˆ‰ hb

a b a c

  • H respects hb
  • H is a legal sequence (library-specific)
  • - e.g. FIFO sequences for queue
  • Linearisable ⟺ βˆƒ H. H totally orders events

a c b

✘

c a b

✘

a b c

✘

non-linearisable (not legal)

slide-76
SLIDE 76
  • Define happens-before relation hb
  • (e1, e2) ∈ hb ⟺ e1.end <time e2.begin

20

  • - e.g. ( , ) ∈ hb ( , ) βˆ‰ hb

a b a c

thread 1 thread 2 enq(1)

a

deq(1)

c

time enq(2)

b

Persistent Linearisability

slide-77
SLIDE 77
  • Define happens-before relation hb
  • (e1, e2) ∈ hb ⟺ e1.end <time e2.begin

20

  • - e.g. ( , ) ∈ hb ( , ) βˆ‰ hb

a b a c

  • H respects hb
  • H is a legal sequence
  • Persistently linearisable ⟺ βˆƒ H. H totally orders a subset S of events

thread 1 thread 2 enq(1)

a

deq(1)

c

time enq(2)

b

Persistent Linearisability

slide-78
SLIDE 78
  • Define happens-before relation hb
  • (e1, e2) ∈ hb ⟺ e1.end <time e2.begin

20

  • - e.g. ( , ) ∈ hb ( , ) βˆ‰ hb

a b a c

  • H respects hb
  • H is a legal sequence
  • Persistently linearisable ⟺ βˆƒ H. H totally orders a subset S of events

thread 1 thread 2 enq(1)

a

deq(1)

c

time enq(2)

b

  • - persists are asynchronous: only a prefix may persist after a crash

Persistent Linearisability

  • S is hb-prefix-closed : (a, b) ∈ hb and b ∈ S β‡’ a ∈ S
slide-79
SLIDE 79
  • Define happens-before relation hb
  • (e1, e2) ∈ hb ⟺ e1.end <time e2.begin

20

  • - e.g. ( , ) ∈ hb ( , ) βˆ‰ hb

a b a c

  • H respects hb
  • H is a legal sequence
  • Persistently linearisable ⟺ βˆƒ H. H totally orders a subset S of events

a c

βœ”

Persistently linearisable thread 1 thread 2 enq(1)

a

deq(1)

c

time enq(2)

b

  • - persists are asynchronous: only a prefix may persist after a crash

Persistent Linearisability

  • S is hb-prefix-closed : (a, b) ∈ hb and b ∈ S β‡’ a ∈ S
slide-80
SLIDE 80

21

What about Multiple Crashes?

execution

time

recovery recovery execution execution

slide-81
SLIDE 81

21

What about Multiple Crashes?

execution

time

recovery recovery execution execution

no crashes

slide-82
SLIDE 82

21

What about Multiple Crashes?

G1 Gn G2

execution

time

recovery recovery execution execution

no crashes

slide-83
SLIDE 83

21

  • H i persistently linearises G i β€” as before
  • H1 ++ … ++ Hn is a legal sequence
  • A chain G1 … Gn is persistently linearisable ⟺ βˆƒ H1 … Hn .

What about Multiple Crashes?

G1 Gn G2

execution

time

recovery recovery execution execution

no crashes

slide-84
SLIDE 84

Conclusions

  • PTSO: First formal epoch persistency semantics under


mainstream hardware

  • Operational model
  • Declarative model
  • Equivalence of the two models
  • Verifying programs under PTSO
  • PTSO programming pattern
  • Correctness condition: persistent linearisability
  • Verified several examples under PTSO
slide-85
SLIDE 85

Conclusions

  • PTSO: First formal epoch persistency semantics under


mainstream hardware

  • Operational model
  • Declarative model
  • Equivalence of the two models
  • Verifying programs under PTSO
  • PTSO programming pattern
  • Correctness condition: persistent linearisability
  • Verified several examples under PTSO

Thank you for listening!

azalea@mpi-sws.org @azalearaad SoundAndComplete.org

slide-86
SLIDE 86

Programming Pattern

  • 1. // log progress
  • 2. pfence
  • 3. // do the work
  • 4. pfence
slide-87
SLIDE 87

Programming Pattern

  • 1. // log progress
  • 2. pfence
  • 3. // do the work
  • 4. pfence

Log at most one step ahead of work

slide-88
SLIDE 88

Programming Pattern

  • 1. // log progress
  • 2. pfence
  • 3. // do the work
  • 4. pfence

1. 2. 3. 4.

Log at most one step ahead of work