Argosy: Verifying layered storage systems with recovery refinement - - PowerPoint PPT Presentation

Argosy: Verifying layered storage systems with recovery refinement

Tej Chajed, Joseph Tassarotti, Frans Kaashoek, Nickolai Zeldovich MIT

2

disk1 disk2 logical disk Bob writes a replication system

2

disk1 disk2 write1 logical disk write2 Bob writes a replication system

3

disk1 disk2 write1 logical disk Bob writes a replication system

3

disk1 disk2 write1 logical disk

?

Bob writes a replication system

3

disk1 disk2 write1

rep_recover

logical disk

?

and implements its recovery procedure Bob writes a replication system

3

disk1 disk2 write1

rep_recover

logical disk

?

recovery restores invariants and implements its recovery procedure Bob writes a replication system

4

replication

read write rep_recover

read and write are atomic if you run rep_recover after every crash

Disk interface Two-disk interface Bob is careful and writes a
 machine-checked proof of correctness

Transactions

5

write-ahead logging

log_recover

…

Disk interface

Transactions

5

write-ahead logging

log_recover

…

• ps are atomic if you run

log_recover after every crash

Disk interface

6

write-ahead log replication logging + replication

Transactions Disk interface Two-disk interface

?

6

write-ahead log replication logging + replication

Transactions Disk interface Two-disk interface

rep_recover ; log_recover

?

7

rep_recover ; log_recover

?

log_recover

under crashes

rep_recover

under crashes

Challenge: crashes during composed recovery

how do we prove correctness under crashes using the existing proofs?

8

Prior work cannot handle multiple recovery procedures

CHL [SOSP ’15] not modular Yggdrasil [OSDI ’16] single recovery Flashix [SCP ’16] restricted recovery procedures

write-ahead log replication

9

Argosy supports modular recovery proofs

developer proves developer proves

write-ahead log replication

Transactions Disk interface Two-disk interface

9

Argosy supports modular recovery proofs

logging + replication

Argosy proves

write-ahead log replication

Transactions Disk interface Two-disk interface

10

Contributions

Recovery refinement for modular proofs

10

Contributions

see paper see paper

Recovery refinement for modular proofs CHL for proving recovery refinement Verified example: logging + replication

10

Contributions

see paper see paper see code

Recovery refinement for modular proofs CHL for proving recovery refinement Verified example: logging + replication Machine-checked proofs in Coq

11

Preview: recovery refinement

replication

Disk interface Two-disk interface

• 1. Normal execution correctness


using refinement

• 2. Crash and recovery correctness


using recovery refinement

Refinement

12

Background

Background

13

Disk interface Two-disk interface

replication

Background

13

Disk interface Two-disk interface

replication write write1 write2 write_impl

Background

13

Disk interface Two-disk interface

replication write write1 write2 write_impl

Background

13

Disk interface

read

Two-disk interface

replication write write1 write2 write_impl read_impl read1 read2

Background

13

Disk interface

read

Two-disk interface

replication write write1 write2 write_impl read_impl read1 read2 code code_impl write write_impl read read_impl

correctness is based on how we use replication : run code using Disk interface on top of two disks

Background

14

Correctness: trace inclusion

replication

Disk interface Two-disk interface

code code_impl

⊇

spec’s
 behaviors running code’s behaviors

Background

15

Proving correctness with an abstraction relation

disk1 disk2 logical disk

R

1. developer provides
 abstraction relation R spec state

Background

15

Proving correctness with an abstraction relation

disk1 disk2 logical disk

R

write1 write2

1. developer provides
 abstraction relation R spec state

Background

15

Proving correctness with an abstraction relation

disk1 disk2 logical disk

R

write1 write2 write

1. developer provides
 abstraction relation R

• 2. prove spec execution exists

spec state

Background

15

Proving correctness with an abstraction relation

disk1 disk2 logical disk

R

write1 write2 write

R

1. developer provides
 abstraction relation R

• 2. prove spec execution exists
• 3. and abstraction relation is preserved

spec state

Recovery refinement

16

17

Disk interface

read

Two-disk interface replication

write write1 write2 write_impl read_impl read1 read2

17

Disk interface

read

Two-disk interface replication

write write1 write2 write_impl rep_recover read_impl read1 read2

17

Disk interface

read

Two-disk interface replication

write write1 write2 write_impl rep_recover read_impl read1 read2

18

replication

Disk interface Two-disk interface

Extending trace inclusion with recovery

code code_impl

⊇ ⊇

specification for crash behavior crash & recovery behavior

18

replication

Disk interface Two-disk interface

Extending trace inclusion with recovery

code code_impl

⊇ ⊇

specification for crash behavior crash & recovery behavior ? crash semantics

recover

? recovery semantics

19

replication

Disk interface Two-disk interface

code code_impl

⊇

…

|

• p1

|

• p1
• p2

|

• ne of these

:=

code

⊇

crash & recovery behavior

recover

? recovery semantics

19

replication

Disk interface Two-disk interface

code code_impl

⊇

…

|

• p1

|

• p1
• p2

|

:=

code

⊇

crash & recovery behavior

recover

? recovery semantics

20

replication

Disk interface Two-disk interface

code code_impl

⊇ ⊇

code recover code_impl

21

replication

Disk interface Two-disk interface

code code_impl

⊇ ⊇

code recover code_impl

zero-or-more iterations

recover ⋆

21

replication

Disk interface Two-disk interface

code code_impl

⊇ ⊇

code recover code_impl recover ⋆

22

replication

Disk interface Two-disk interface

code code_impl

⊇ ⊇

code recover code_impl recover ⋆

Trace inclusion, with recovery

23

• p1_impl

recover

• p2_impl

recover

⋆

Proving trace inclusion, with recovery

23

• p1_impl

recover

• p2_impl

recover

⋆

crash must occur during some operation

Proving trace inclusion, with recovery

23

• p1_impl

recover

• p2_impl

recover

⋆

Proving trace inclusion, with recovery

23

• p1_impl

recover

• p2_impl

recover

⋆

• p1

R R

Proving trace inclusion, with recovery

23

recover

• p2_impl

recover

⋆ R

Proving trace inclusion, with recovery

23

recover

• p2_impl

recover

⋆ R

• p2

| R

Proving trace inclusion, with recovery

24

• p

R

• p_impl

R

non-crash execution crash and recovery execution

R

recover

• p_impl

recover

⋆

• p

|

R

Recovery refinement

24

• p

R

• p_impl

R

non-crash execution crash and recovery execution

R

recover

• p_impl

recover

⋆

• p

|

R

Recovery refinement

implies

Trace inclusion

specification behavior

⊇

running code behavior

Composition theorem

25

26

• p1
• p2

|

• p

r

⋆

Kleene algebra for transition relations

expression

26

• p1
• p2
• p1
• p2
• p

|

• p

r

⋆ …

r r r

Kleene algebra for transition relations

expression matching transitions

27

Theorem: recovery refinements compose

write-ahead log

log_recover …

replication

rep_recover …

If

Transactions Two-disk interface Disk interface

27

Theorem: recovery refinements compose

write-ahead log

log_recover …

replication

rep_recover …

then If

logging + replication

… rep_recover; log_recover

Transactions Two-disk interface Transactions Two-disk interface Disk interface

28

Goal: prove composed recovery correct

rep_recover ; log_recover

?

log_recover

under crashes

rep_recover

under crashes

29

under crashes

log rep

under crashes

rep log

;

?

Goal: prove composed recovery correct

rep_recover log_recover

30

log log

⋆

rep rep

⋆

30

log log

⋆

rep rep

⋆

rep log rep

|

( )

rep log

⋆

30

log log

⋆

rep rep

⋆

rep log rep

|

( )

rep log

⋆ how to re-use recovery proofs here?

31

Using Kleene algebra for reasoning

rep log rep

|

( )

rep log

⋆

31

Using Kleene algebra for reasoning

rep log rep

|

( )

rep log

⋆

after de-nesting (p ∣ q)⋆ = p⋆(qp⋆)⋆

31

Using Kleene algebra for reasoning

rep log rep

|

( )

rep log

⋆

rep rep log rep

( )

rep log

⋆

=

⋆ ⋆ after de-nesting (p ∣ q)⋆ = p⋆(qp⋆)⋆

31

Using Kleene algebra for reasoning

rep log rep

|

( )

rep log

⋆

rep rep log rep

( )

rep log

⋆

=

⋆ ⋆ after de-nesting (p ∣ q)⋆ = p⋆(qp⋆)⋆

rep rep log rep (

)

rep log

⋆

⋆

⋆

=

(pq)⋆p = p(qp)⋆

after sliding

32

rep rep log rep (

)

rep log

⋆

⋆

⋆

After rewrite both proofs apply

replication proof

32

rep rep log rep (

)

rep log

⋆

⋆

⋆

rep invariants restored

After rewrite both proofs apply

replication proof

32

rep rep log rep (

)

rep log

⋆

⋆

⋆

rep invariants restored

log

behaves like

After rewrite both proofs apply

write-ahead log proof replication proof

32

rep rep log rep (

)

rep log

⋆

⋆

⋆

rep invariants restored

log

behaves like

log log

⋆ log invariants restored

After rewrite both proofs apply

33

Argosy is implemented and verified in Coq

github.com/mit-pdos/argosy

3,200 lines for framework 4,000 lines for verified example (logging + replication) Example extracts to Haskell and runs

34

Argosy: modular proofs of layered storage systems

34

Argosy: modular proofs of layered storage systems

rep rep

|

( )⋆

log

Kleene algebra

34

Argosy: modular proofs of layered storage systems

r impl r

⋆

• p

|

recovery refinement

rep rep

|

( )⋆

log

Kleene algebra

34

Argosy: modular proofs of layered storage systems

r impl r

⋆

• p

|

recovery refinement modular proofs

rep rep

|

( )⋆

log

Kleene algebra

34

Argosy: modular proofs of layered storage systems

r impl r

⋆

• p

|

recovery refinement modular proofs

rep rep

|

( )⋆

log

Kleene algebra

come find us after! Tej and Joe