Persistence Semantics for Weak Memory Integrating Epoch Persistency - - PowerPoint PPT Presentation

▶

persistence semantics for weak memory

Persistence Semantics for Weak Memory Integrating Epoch Persistency - - PowerPoint PPT Presentation

Feb 13, 2023 328 likes •1.22k views

Persistence Semantics for Weak Memory Integrating Epoch Persistency with the TSO Memory Model Azalea Raad Viktor Vafeiadis Max Planck Institute for Software Systems (MPI-SWS) Thursday 8 November OOPSLA 2018 Boston, USA azalea@mpi-sws.org

slide-1

SLIDE 1

Persistence Semantics for Weak Memory

Integrating Epoch Persistency with the TSO Memory Model

Azalea Raad Viktor Vafeiadis

Max Planck Institute for Software Systems (MPI-SWS)

Thursday 8 November OOPSLA 2018 Boston, USA

azalea@mpi-sws.org @azalearaad SoundAndComplete.org

slide-2

SLIDE 2

History

time Difficulty Sequential

😋

slide-3

SLIDE 3

History

time Difficulty Sequential

😋

SC

😑

slide-4

SLIDE 4

History

time Difficulty Sequential

😋

SC

😑

WMC

☹

slide-5

SLIDE 5

History

time Difficulty Sequential

😋

SC

😑

WMC

☹ 😤

Persistent WMC

slide-6

SLIDE 6

Volatile memory

x := 1 // x = 1 // x = 0

What is Persistent Memory?

3

// x = v : reading x yields v

slide-7

SLIDE 7

Volatile memory

x := 1 // x = 1 // x = 0

What is Persistent Memory?

3

// x = 0 // no recovery // x = v : reading x yields v

slide-8

SLIDE 8

Volatile memory

x := 1 // x = 1 // x = 0

What is Persistent Memory?

3

// x = 0 // no recovery // x = v : reading x yields v

Persistent memory

x := 1 // x = 1 // x = 0 // x = 0 OR x = 1 // recovery routine

slide-9

SLIDE 9

Volatile memory

x := 1 // x = 1 // x = 0

What is Persistent Memory?

3

// x = 0 // no recovery // x = v : reading x yields v

Persistent memory

x := 1 // x = 1 // x = 0 // x = 0 OR x = 1 // recovery routine

persists are asynchronous (buffered): may not persist immediately

slide-10

SLIDE 10

(Sequential) Hardware

slide-11

SLIDE 11

CPU (Volatile) Memory

(Sequential) Hardware

slide-12

SLIDE 12

CPU (Volatile) Memory

(Sequential) Hardware

x:=1 : adds x:=1 to memory

slide-13

SLIDE 13

CPU (Volatile) Memory

(Sequential) Hardware

x:=1 : adds x:=1 to memory a:=x : reads x from memory

slide-14

SLIDE 14

CPU (Volatile) Memory

(Sequential) Hardware

x:=1 : adds x:=1 to memory a:=x : reads x from memory memory lost

slide-15

SLIDE 15

Persistence Buffer CPU (Persistent) Memory CPU (Volatile) Memory

(Sequential) Hardware

x:=1 : adds x:=1 to memory a:=x : reads x from memory memory lost

slide-16

SLIDE 16

Persistence Buffer CPU (Persistent) Memory CPU (Volatile) Memory

(Sequential) Hardware

x:=1 : adds x:=1 to memory a:=x : reads x from memory memory lost x:=1 : adds x:=1 to p-buffer

slide-17

SLIDE 17

Persistence Buffer CPU (Persistent) Memory CPU (Volatile) Memory

(Sequential) Hardware

x:=1 : adds x:=1 to memory a:=x : reads x from memory memory lost x:=1 : adds x:=1 to p-buffer a:=x : if p-buffer contains x, reads latest entry else reads from memory

slide-18

SLIDE 18

Persistence Buffer CPU (Persistent) Memory CPU (Volatile) Memory

(Sequential) Hardware

x:=1 : adds x:=1 to memory a:=x : reads x from memory memory lost x:=1 : adds x:=1 to p-buffer a:=x : if p-buffer contains x, reads latest entry else reads from memory p-buffer lost; memory retained

slide-19

SLIDE 19

Persistence Buffer CPU (Persistent) Memory CPU (Volatile) Memory

(Sequential) Hardware

x:=1 : adds x:=1 to memory a:=x : reads x from memory memory lost x:=1 : adds x:=1 to p-buffer a:=x : if p-buffer contains x, reads latest entry else reads from memory p-buffer lost; memory retained unbuffer* : p-buffer to memory * at non-deterministic times

slide-20

SLIDE 20

Memory consistency model describes:

the order writes are made visible to other threads 

e.g. SC, TSO, …

What is Memory Persistency Model?

slide-21

SLIDE 21

Memory consistency model describes:

the order writes are made visible to other threads 

e.g. SC, TSO, …

Memory persistency model describes:

the order writes are persisted to memory 

e.g. Epoch Persistency

What is Memory Persistency Model?

slide-22

SLIDE 22

Memory consistency model describes:

the order writes are made visible to other threads 

e.g. SC, TSO, …

Memory persistency model describes:

the order writes are persisted to memory 

e.g. Epoch Persistency

What is Memory Persistency Model?

Problem Formal Epoch Persistency Model for Mainstream Hardware (Weak Memory Models)

slide-23

SLIDE 23

x := 1; // recovery routine // x=0;y=0 y := 1; // x=0;y=0 OR x=1;y=1 OR x=1;y=0 OR x=0;y=1

What Can Go Wrong?

6

slide-24

SLIDE 24

x := 1; // recovery routine // x=0;y=0 y := 1; // x=0;y=0 OR x=1;y=1 OR x=1;y=0 OR x=0;y=1

What Can Go Wrong?

6

!! Writes may persist out of order

slide-25

SLIDE 25

x := 1; // recovery routine // x=0;y=0 y := 1; // x=0;y=0 OR x=1;y=1 OR x=1;y=0 OR x=0;y=1

What Can Go Wrong?

6

!! Writes may persist out of order ☛ persistent fence pfence

slide-26

SLIDE 26

7

x := 1; // recovery routine // x=0;y=0 y := 1; // x=0;y=0 OR x=1;y=1 OR x=1;y=0 OR x=0;y=1 pfence;

☛

Persistent Fence

slide-27

SLIDE 27

Persistent Fence

8

x := 1; z := 4; pfence; y := 2; x := 3;

a b c d

slide-28

SLIDE 28

Persistent Fence

8

writes on same locations persist in execution order

x := 1; z := 4; pfence; y := 2; x := 3;

a b c d a c

persists before

slide-29

SLIDE 29

Persistent Fence

8

writes on same locations persist in execution order
writes on different locations are unordered

x := 1; z := 4; pfence; y := 2; x := 3;

a b c d a b

may persist in any order

a c

persists before

slide-30

SLIDE 30

Persistent Fence

8

writes on same locations persist in execution order
writes on different locations are unordered
pfence adds a new epoch

x := 1; z := 4; pfence; y := 2; x := 3;

a b c d a b

may persist in any order

a c

persists before

epoch 1 epoch 2

slide-31

SLIDE 31

Persistent Fence

8

writes on same locations persist in execution order
writes on different locations are unordered
pfence adds a new epoch
writes persist in epoch order

x := 1; z := 4; pfence; y := 2; x := 3;

a b c d

persist before

a b c d a b

may persist in any order

a c

persists before

epoch 1 epoch 2

slide-32

SLIDE 32

// x=0;y=0 OR x=1;y=1 OR x=1;y=0 x := 1; // recovery routine // x=0;y=0 y := 1; pfence;

9

What Can Go Wrong (Continued)?

!! Execution continues ahead of persistence

asynchronous (buffered)

slide-33

SLIDE 33

// x=0;y=0 OR x=1;y=1 OR x=1;y=0 x := 1; // recovery routine // x=0;y=0 y := 1; pfence;

9

What Can Go Wrong (Continued)?

!! Execution continues ahead of persistence ☛ persistent sync psync

asynchronous (buffered)

slide-34

SLIDE 34

// x=0;y=0 OR x=1;y=1 OR x=1;y=0 x := 1; // recovery routine // x=0;y=0 y := 1; pfence;

9

What Can Go Wrong (Continued)?

!! Execution continues ahead of persistence ☛ persistent sync psync

C1; psync; C2

same persist-ordering as pfence
C2 executed only when all C1 writes have persisted

asynchronous (buffered)

slide-35

SLIDE 35

x := 1; // recovery routine // x=0;y=0 y := 1; // x=0;y=0 OR x=1;y=1 OR x=1;y=0 psync;

10

Persistent Sync

!! Execution continues ahead of persistence ☛ persistent sync psync

C1; psync; C2

same persist-ordering as pfence
C2 executed only when all C1 writes have persisted

☛

slide-36

SLIDE 36

(Sequential) Hardware

x:=1 : adds x:=1 to p-buffer a:=x : if p-buffer contains x, reads latest entry else reads from memory p-buffer lost; memory retained

11

CPU (Persistent) Memory

epoch n epoch 2 pfence pfence . . . pfence epoch 1

slide-37

SLIDE 37

(Sequential) Hardware

x:=1 : adds x:=1 to p-buffer a:=x : if p-buffer contains x, reads latest entry else reads from memory unbuffer* : p-buffer to memory (in epoch order) p-buffer lost; memory retained * at non-deterministic times

11

CPU (Persistent) Memory

epoch n epoch 2 pfence pfence . . . pfence epoch 1

slide-38

SLIDE 38

(Sequential) Hardware

x:=1 : adds x:=1 to p-buffer a:=x : if p-buffer contains x, reads latest entry else reads from memory unbuffer* : p-buffer to memory (in epoch order) p-buffer lost; memory retained * at non-deterministic times pfence : introduces a new epoch in p-buffer

11

CPU (Persistent) Memory

epoch n epoch 2 pfence pfence . . . pfence epoch 1

slide-39

SLIDE 39

(Sequential) Hardware

x:=1 : adds x:=1 to p-buffer a:=x : if p-buffer contains x, reads latest entry else reads from memory unbuffer* : p-buffer to memory (in epoch order) p-buffer lost; memory retained * at non-deterministic times pfence : introduces a new epoch in p-buffer psync : flushes the entire p-buffer to memory

11

CPU (Persistent) Memory

epoch n epoch 2 pfence pfence . . . pfence epoch 1

slide-40

SLIDE 40

What about Concurrency?

TSO POWER ARMv8

…

12

slide-41

SLIDE 41

What about Concurrency?

TSO POWER ARMv8

…

12

slide-42

SLIDE 42

Contributions

13

slide-43

SLIDE 43

PTSO: First formal epoch persistency semantics under

mainstream hardware

Operational model
Declarative model
Equivalence of the two models

Contributions

13

slide-44

SLIDE 44

PTSO: First formal epoch persistency semantics under

mainstream hardware

Operational model
Declarative model
Equivalence of the two models

Contributions

13

slide-45

SLIDE 45

PTSO: First formal epoch persistency semantics under

mainstream hardware

Operational model
Declarative model
Equivalence of the two models
Verifying programs under PTSO
PTSO programming pattern
Correctness condition: persistent linearisability
Verified several examples under PTSO

Contributions

13

slide-46

SLIDE 46

Total Store Ordering (TSO)

14

slide-47

SLIDE 47

Total Store Ordering (TSO)

14

Thread2 Buffer (Volatile) Memory Thread1 Buffer

slide-48

SLIDE 48

Total Store Ordering (TSO)

14

Thread2 Thread1

x = 0; y = 0;

x := 1; a := y;

Thread1

y := 1; c := x;

Thread2

☛ ☛

Store Buffering (SB)

slide-49

SLIDE 49

Total Store Ordering (TSO)

14

Thread2 Thread1

x = 0; y = 0;

x := 1; a := y;

Thread1

y := 1; c := x;

Thread2

x = 1

☛ ☛

Store Buffering (SB)

slide-50

SLIDE 50

Total Store Ordering (TSO)

14

Thread2 Thread1

x = 0; y = 0;

x := 1; a := y;

Thread1

y := 1; c := x;

Thread2

x = 1 y = 1

☛ ☛

Store Buffering (SB)

slide-51

SLIDE 51

Total Store Ordering (TSO)

14

Thread2 Thread1

x = 0; y = 0;

x := 1; a := y;

Thread1

y := 1; c := x;

Thread2

x = 1 y = 1

☛ ☛

Store Buffering (SB)

slide-52

SLIDE 52

Total Store Ordering (TSO)

14

Thread2 Thread1

x = 0; y = 0;

x := 1; a := y;

Thread1

y := 1; c := x;

Thread2

x = 1 y = 1

☛ ☛

Store Buffering (SB)

slide-53

SLIDE 53

Total Store Ordering (TSO)

14

Thread2 Thread1

x = 0; y = 0;

x := 1; a := y;

Thread1

y := 1; c := x;

Thread2

x = 1 y = 1

☛ ☛

// 0 Store Buffering (SB)

slide-54

SLIDE 54

Total Store Ordering (TSO)

14

Thread2 Thread1

x = 0; y = 0;

x := 1; a := y;

Thread1

y := 1; c := x;

Thread2

x = 1 y = 1

☛ ☛

// 0 Store Buffering (SB)

slide-55

SLIDE 55

Total Store Ordering (TSO)

14

Thread2 Thread1

x = 0; y = 0;

x := 1; a := y;

Thread1

y := 1; c := x;

Thread2

x = 1 y = 1

☛ ☛

// 0 Store Buffering (SB)

slide-56

SLIDE 56

Total Store Ordering (TSO)

14

Thread2 Thread1

x = 0; y = 0;

x := 1; a := y;

Thread1

y := 1; c := x;

Thread2

x = 1 y = 1

☛ ☛

// 0 // 0 Store Buffering (SB)

slide-57

SLIDE 57

Total Store Ordering (TSO)

14

Thread2 Thread1

x := 1; a := y;

Thread1

y := 1; c := x;

Thread2

x = 1 y = 1

☛ ☛

// 0 // 0 Store Buffering (SB)

; y = 0;

slide-58

SLIDE 58

Total Store Ordering (TSO)

14

Thread2 Thread1

x := 1; a := y;

Thread1

y := 1; c := x;

Thread2

x = 1

☛ ☛

// 0 // 0 Store Buffering (SB)

; y = 1;

slide-59

SLIDE 59

Persistent TSO (PTSO)

15

(Volatile) Memory Thread1 Buffer Thread2 Buffer (Volatile) Memory CPU

slide-60

SLIDE 60

Persistent TSO (PTSO)

15

(Persistent) Memory CPU Persistence Buffer (Volatile) Memory Thread1 Buffer Thread2 Buffer (Volatile) Memory CPU

slide-61

SLIDE 61

Persistent TSO (PTSO)

15

(Persistent) Memory Thread1 Buffer Persistence Buffer Thread2 Buffer (Persistent) Memory CPU Persistence Buffer (Volatile) Memory Thread1 Buffer Thread2 Buffer (Volatile) Memory CPU

slide-62

SLIDE 62

PTSO: First formal epoch persistency semantics under

mainstream hardware

Operational model
Declarative model
Equivalence of the two models

Contributions

16

slide-63

SLIDE 63

PTSO: First formal epoch persistency semantics under

mainstream hardware

Operational model
Declarative model
Equivalence of the two models

Contributions

16

Verifying programs under PTSO
PTSO programming pattern
Correctness condition: persistent linearisability
Verified several examples under PTSO

slide-64

SLIDE 64

PTSO: First formal epoch persistency semantics under

mainstream hardware

Operational model
Declarative model
Equivalence of the two models

Contributions

16

Verifying programs under PTSO
PTSO programming pattern
Correctness condition: persistent linearisability
Verified several examples under PTSO

slide-65

SLIDE 65

Verifying programs under PTSO

The persistent variant of the Michael-Scott queue and its recovery mechanism

slide-66

SLIDE 66

Verifying programs under PTSO

The persistent variant of the Michael-Scott queue and its recovery mechanism

What constitutes a correct persistent implementation?

slide-67

SLIDE 67

Linearisability

18

thread 1 thread 2 enq(1)

a

deq(1)

c

enq(2)

b

time

slide-68

SLIDE 68

Linearisability

Define happens-before relation hb
(e1, e2) ∈ hb ⟺ e1.end <time e2.begin

18

thread 1 thread 2 enq(1)

a

deq(1)

c

enq(2)

b

time

slide-69

SLIDE 69

Linearisability

Define happens-before relation hb
(e1, e2) ∈ hb ⟺ e1.end <time e2.begin

18

thread 1 thread 2 enq(1)

a

deq(1)

c

enq(2)

b

time

- e.g. ( , ) ∈ hb ( , ) ∉ hb

a b a c

slide-70

SLIDE 70

Linearisability

Define happens-before relation hb
(e1, e2) ∈ hb ⟺ e1.end <time e2.begin

18

thread 1 thread 2 enq(1)

a

deq(1)

c

enq(2)

b

time

- e.g. ( , ) ∈ hb ( , ) ∉ hb

a b a c

H respects hb
H is a legal sequence (library-specific)
Linearisable ⟺ ∃ H. H totally orders events

slide-71

SLIDE 71

Linearisability

Define happens-before relation hb
(e1, e2) ∈ hb ⟺ e1.end <time e2.begin

18

thread 1 thread 2 enq(1)

a

deq(1)

c

enq(2)

b

time

- e.g. ( , ) ∈ hb ( , ) ∉ hb

a b a c

H respects hb
H is a legal sequence (library-specific)
- e.g. FIFO sequences for queue
Linearisable ⟺ ∃ H. H totally orders events

a c b

✔

c a b

✘

a b c

✔

slide-72

SLIDE 72

Linearisability

Define happens-before relation hb
(e1, e2) ∈ hb ⟺ e1.end <time e2.begin

18

thread 1 thread 2 enq(1)

a

deq(1)

c

enq(2)

b

time

- e.g. ( , ) ∈ hb ( , ) ∉ hb

a b a c

H respects hb
H is a legal sequence (library-specific)
- e.g. FIFO sequences for queue
Linearisable ⟺ ∃ H. H totally orders events

a c b

✔

c a b

✘

a b c

✔

linearisable

slide-73

SLIDE 73

Linearisability

Define happens-before relation hb
(e1, e2) ∈ hb ⟺ e1.end <time e2.begin

19

thread 1 thread 2 enq(1)

a

deq(2)

c

enq(2)

b

time

- e.g. ( , ) ∈ hb ( , ) ∉ hb

a b a c

H respects hb
H is a legal sequence (library-specific)
- e.g. FIFO sequences for queue
Linearisable ⟺ ∃ H. H totally orders events

slide-74

SLIDE 74

Linearisability

Define happens-before relation hb
(e1, e2) ∈ hb ⟺ e1.end <time e2.begin

19

thread 1 thread 2 enq(1)

a

deq(2)

c

enq(2)

b

time

- e.g. ( , ) ∈ hb ( , ) ∉ hb

a b a c

H respects hb
H is a legal sequence (library-specific)
- e.g. FIFO sequences for queue
Linearisable ⟺ ∃ H. H totally orders events

a c b c a b a b c

slide-75

SLIDE 75

Linearisability

Define happens-before relation hb
(e1, e2) ∈ hb ⟺ e1.end <time e2.begin

19

thread 1 thread 2 enq(1)

a

deq(2)

c

enq(2)

b

time

- e.g. ( , ) ∈ hb ( , ) ∉ hb

a b a c

H respects hb
H is a legal sequence (library-specific)
- e.g. FIFO sequences for queue
Linearisable ⟺ ∃ H. H totally orders events

a c b

✘

c a b

✘

a b c

✘

non-linearisable (not legal)

slide-76

SLIDE 76

Define happens-before relation hb
(e1, e2) ∈ hb ⟺ e1.end <time e2.begin

20

- e.g. ( , ) ∈ hb ( , ) ∉ hb

a b a c

thread 1 thread 2 enq(1)

a

deq(1)

c

time enq(2)

b

Persistent Linearisability

slide-77

SLIDE 77

Define happens-before relation hb
(e1, e2) ∈ hb ⟺ e1.end <time e2.begin

20

- e.g. ( , ) ∈ hb ( , ) ∉ hb

a b a c

H respects hb
H is a legal sequence
Persistently linearisable ⟺ ∃ H. H totally orders a subset S of events

thread 1 thread 2 enq(1)

a

deq(1)

c

time enq(2)

b

Persistent Linearisability

slide-78

SLIDE 78

Define happens-before relation hb
(e1, e2) ∈ hb ⟺ e1.end <time e2.begin

20

- e.g. ( , ) ∈ hb ( , ) ∉ hb

a b a c

H respects hb
H is a legal sequence
Persistently linearisable ⟺ ∃ H. H totally orders a subset S of events

thread 1 thread 2 enq(1)

a

deq(1)

c

time enq(2)

b

- persists are asynchronous: only a prefix may persist after a crash

Persistent Linearisability

S is hb-prefix-closed : (a, b) ∈ hb and b ∈ S ⇒ a ∈ S

slide-79

SLIDE 79

Define happens-before relation hb
(e1, e2) ∈ hb ⟺ e1.end <time e2.begin

20

- e.g. ( , ) ∈ hb ( , ) ∉ hb

a b a c

H respects hb
H is a legal sequence
Persistently linearisable ⟺ ∃ H. H totally orders a subset S of events

a c

✔

Persistently linearisable thread 1 thread 2 enq(1)

a

deq(1)

c

time enq(2)

b

- persists are asynchronous: only a prefix may persist after a crash

Persistent Linearisability

S is hb-prefix-closed : (a, b) ∈ hb and b ∈ S ⇒ a ∈ S

slide-80

SLIDE 80

21

What about Multiple Crashes?

execution

time

recovery recovery execution execution

slide-81

SLIDE 81

21

What about Multiple Crashes?

execution

time

recovery recovery execution execution

no crashes

slide-82

SLIDE 82

21

What about Multiple Crashes?

G1 Gn G2

execution

time

recovery recovery execution execution

no crashes

slide-83

SLIDE 83

21

H i persistently linearises G i — as before
H1 ++ … ++ Hn is a legal sequence
A chain G1 … Gn is persistently linearisable ⟺ ∃ H1 … Hn .

What about Multiple Crashes?

G1 Gn G2

execution

time

recovery recovery execution execution

no crashes

slide-84

SLIDE 84

Conclusions

PTSO: First formal epoch persistency semantics under

mainstream hardware

Operational model
Declarative model
Equivalence of the two models
Verifying programs under PTSO
PTSO programming pattern
Correctness condition: persistent linearisability
Verified several examples under PTSO

slide-85

SLIDE 85

Conclusions

PTSO: First formal epoch persistency semantics under

mainstream hardware

Operational model
Declarative model
Equivalence of the two models
Verifying programs under PTSO
PTSO programming pattern
Correctness condition: persistent linearisability
Verified several examples under PTSO

Thank you for listening!

azalea@mpi-sws.org @azalearaad SoundAndComplete.org

slide-86

SLIDE 86

Programming Pattern

1. // log progress
2. pfence
3. // do the work
4. pfence

slide-87

SLIDE 87

Programming Pattern

1. // log progress
2. pfence
3. // do the work
4. pfence

Log at most one step ahead of work

slide-88

SLIDE 88

Programming Pattern

1. // log progress
2. pfence
3. // do the work
4. pfence

1. 2. 3. 4.

Log at most one step ahead of work