Argosy: Verifying layered storage systems with recovery refinement - PowerPoint PPT Presentation

Argosy: Verifying layered storage systems with recovery refinement Tej Chajed , Joseph Tassarotti, Frans Kaashoek, Nickolai Zeldovich MIT

logical disk disk 1 disk 2 Bob writes a replication system � 2

logical disk write 1 write 2 disk 1 disk 2 Bob writes a replication system � 2

logical disk write 1 disk 1 disk 2 Bob writes a replication system � 3

? logical disk write 1 disk 1 disk 2 Bob writes a replication system � 3

? logical disk write 1 rep_recover disk 1 disk 2 Bob writes a replication system and implements its recovery procedure � 3

? logical disk write 1 rep_recover recovery restores invariants disk 1 disk 2 Bob writes a replication system and implements its recovery procedure � 3

Bob is careful and writes a 
 machine-checked proof of correctness Disk interface replication read and write are atomic if you run rep_recover after every crash read rep_recover write Two-disk interface � 4

Transactions write-ahead logging … log_recover Disk interface � 5

Transactions write-ahead logging ops are atomic if you run log _recover after every crash … log_recover Disk interface � 5

Transactions write-ahead log logging + ? replication Disk interface replication Two-disk interface � 6

Transactions write-ahead log logging + ? replication Disk interface rep_recover ; log_recover replication Two-disk interface � 6

Challenge: crashes during composed recovery rep_recover under crashes under crashes log_recover how do we prove correctness rep_recover ; log_recover ? under crashes using the existing proofs? � 7

Prior work cannot handle multiple recovery procedures CHL [SOSP ’15] not modular write-ahead log Yggdrasil [OSDI ’16] single recovery replication restricted recovery Flashix [SCP ’16] procedures � 8

Argosy supports modular recovery proofs Transactions developer proves write-ahead log Disk interface replication developer proves Two-disk interface � 9

Argosy supports modular recovery proofs Transactions Argosy write-ahead log proves logging + Disk interface replication replication Two-disk interface � 9

Contributions Recovery refinement for modular proofs � 10

Contributions Recovery refinement for modular proofs CHL for proving recovery refinement see paper Verified example: logging + replication see paper � 10

Contributions Recovery refinement for modular proofs CHL for proving recovery refinement see paper Verified example: logging + replication see paper Machine-checked proofs in Coq see code � 10

Preview: recovery refinement Disk interface 1. Normal execution correctness 
 using refinement replication 2. Crash and recovery correctness 
 using recovery refinement Two-disk interface � 11

Background Refinement � 12

Background Disk interface replication Two-disk interface � 13

Background Disk interface write replication write_impl Two-disk interface write 1 write 2 � 13

Background Disk interface write replication write_impl Two-disk interface write 1 write 2 � 13

Background Disk interface read write replication read_impl write_impl Two-disk interface write 1 write 2 read 1 read 2 � 13

Background Disk interface read write correctness is based on how we use replication : run code using Disk interface on top of two disks replication code read read_impl read_impl write write_impl code_impl write_impl Two-disk interface write 1 write 2 read 1 read 2 � 13

Background Correctness: trace inclusion Disk interface spec’s 
 code behaviors replication ⊇ running code’s code_impl behaviors Two-disk interface � 14

Background Proving correctness with an abstraction relation spec state logical disk 1. developer provides 
 abstraction relation R R disk 1 disk 2 � 15

Background Proving correctness with an abstraction relation spec state logical disk 1. developer provides 
 abstraction relation R R disk 1 write 1 write 2 disk 2 � 15

Background Proving correctness with an abstraction relation spec state write logical disk 1. developer provides 
 abstraction relation R R 2. prove spec execution exists disk 1 write 1 write 2 disk 2 � 15

Background Proving correctness with an abstraction relation spec state write logical disk 1. developer provides 
 abstraction relation R R R 2. prove spec execution exists 3. and abstraction relation is preserved disk 1 write 1 write 2 disk 2 � 15

Recovery refinement � 16

Disk interface read write replication read_impl write_impl Two-disk interface write 1 write 2 read 1 read 2 � 17

Disk interface read write replication read_impl rep_recover write_impl Two-disk interface write 1 write 2 read 1 read 2 � 17

Disk interface read write replication read_impl rep_recover write_impl Two-disk interface write 1 write 2 read 1 read 2 � 17

Extending trace inclusion with recovery code ⊇ Disk interface code_impl replication specification for crash behavior Two-disk interface ⊇ crash & recovery behavior � 18

Extending trace inclusion with recovery code ⊇ Disk interface code_impl replication crash semantics ? specification for crash behavior Two-disk interface ⊇ recovery semantics ? recover crash & recovery behavior � 18

code ⊇ Disk interface one of these code_impl replication | | | := op 1 op 1 op 2 code … Two-disk interface ⊇ recovery semantics ? recover crash & recovery behavior � 19

code ⊇ Disk interface code_impl replication | | | := op 1 op 1 op 2 code … Two-disk interface ⊇ recovery semantics ? recover crash & recovery behavior � 19

code ⊇ Disk interface code_impl replication code Two-disk interface ⊇ code_impl recover � 20

code ⊇ Disk interface code_impl replication code Two-disk interface ⊇ ⋆ code_impl recover recover zero-or-more iterations � 21

code ⊇ Disk interface code_impl replication code Two-disk interface ⊇ ⋆ code_impl recover recover � 21

Trace inclusion, with recovery code ⊇ Disk interface code_impl replication code Two-disk interface ⊇ ⋆ code_impl recover recover � 22

Proving trace inclusion, with recovery ⋆ op1_impl op2_impl recover recover � 23

Proving trace inclusion, with recovery ⋆ op1_impl op2_impl recover recover crash must occur during some operation � 23

Proving trace inclusion, with recovery ⋆ op1_impl op2_impl recover recover � 23

Proving trace inclusion, with recovery op 1 R R ⋆ op1_impl op2_impl recover recover � 23

Proving trace inclusion, with recovery R ⋆ op2_impl recover recover � 23

Proving trace inclusion, with recovery op 2 | R R ⋆ op2_impl recover recover � 23

Recovery refinement non-crash execution crash and recovery execution | op op R R R R ⋆ op_impl recover recover op_impl � 24

Recovery refinement non-crash execution crash and recovery execution | op op R R R R ⋆ op_impl recover recover op_impl Trace inclusion implies specification behavior ⊇ running code behavior � 24

Composition theorem � 25

Kleene algebra for transition relations expression op 1 op 2 op | ⋆ r � 26

Kleene algebra for transition relations expression matching transitions op 1 op 2 op 1 op 2 op | op ⋆ r r r r … � 26

Theorem: recovery refinements compose Transactions write-ahead log … log_recover If Disk interface replication … rep_recover Two-disk interface � 27

Theorem: recovery refinements compose Transactions Transactions write-ahead log … log_recover logging + replication If then Disk interface … rep_recover; log_recover replication … rep_recover Two-disk interface Two-disk interface � 27

Goal: prove composed recovery correct rep_recover under crashes under crashes log_recover rep_recover ; log_recover ? � 28

Goal: prove composed recovery correct rep_recover rep under crashes log_recover under crashes log rep log ; ? � 29

⋆ rep rep ⋆ log log � 30

⋆ rep rep ⋆ log log ( ) ⋆ rep rep log rep log | � 30

⋆ rep rep ⋆ log log ( ) ⋆ rep rep log rep log | how to re-use recovery proofs here? � 30

Using Kleene algebra for reasoning ( ) ⋆ rep rep log rep log | � 31

Using Kleene algebra for reasoning ( ) ⋆ rep rep log rep log | after de-nesting ( p ∣ q ) ⋆ = p ⋆ ( qp ⋆ ) ⋆ � 31

Using Kleene algebra for reasoning ( ) ⋆ rep rep log rep log | after de-nesting ( p ∣ q ) ⋆ = p ⋆ ( qp ⋆ ) ⋆ ⋆ ⋆ ⋆ ( ) = rep log rep rep log rep � 31

Using Kleene algebra for reasoning ( ) ⋆ rep rep log rep log | after de-nesting ( p ∣ q ) ⋆ = p ⋆ ( qp ⋆ ) ⋆ ⋆ ⋆ ⋆ ( ) = rep log rep rep log rep ( pq ) ⋆ p = p ( qp ) ⋆ after sliding rep ( ⋆ ⋆ ) ⋆ = log rep rep rep log � 31