Push-Button Verification of File Systems via Crash Refinement Helgi - - PowerPoint PPT Presentation

▶
push button verification of file systems
SMART_READER_LITE
LIVE PREVIEW

Push-Button Verification of File Systems via Crash Refinement Helgi - - PowerPoint PPT Presentation

Apr 28, 2023 361 likes •802 views

Push-Button Verification of File Systems via Crash Refinement Helgi Sigurbjarnarson , James Bornholt, Emina Torlak, Xi Wang University of Washington 1 / 24 File systems are hard to get right Complex hierarchical on-disk data structures

slide-1
SLIDE 1

Push-Button Verification of File Systems

via Crash Refinement Helgi Sigurbjarnarson, James Bornholt, Emina Torlak, Xi Wang

University of Washington

1 / 24

slide-2
SLIDE 2

File systems are hard to get right Complex hierarchical on-disk data structures Maintain consistency in the face of crashes

2 / 24

slide-3
SLIDE 3

Example: Missing flush in ext4 Bugs are hard to reproduce Hard to test fixes It’s unlikely this will be necessary [..] but we need this in order to guarantee correctness.

File System Developers

3 / 24

slide-4
SLIDE 4

Verification: Effective at eliminating bugs, but costly Prove the absence of bugs

◮ BilbyFS [ASPLOS 2016] ◮ FSCQ [SOSP 2015]

Requires expertise and are labor intensive impl proof 2,000 20,000

4 / 24

slide-5
SLIDE 5

Goal: Push-button verification No manual annotation / proof of implementation Get a concrete test case for any bug Our approach: System design to leverage a state-of-the-art automated theorem prover, Z3.

5 / 24

slide-6
SLIDE 6

Push-button verification: Challenges Need a formalization of correctness

◮ Needs to capture crash & recovery ◮ Needs to be automatically verifiable

Too many states to exhaust

◮ Disks are large ◮ Many execution paths ◮ Non-determinism

Prior work focused on bug finding

◮ eXplode [OSDI ’06], EXE [CCS ’06] ◮ Useful, but incomplete 6 / 24

slide-7
SLIDE 7

Contributions and outlines Yggdrasil [’yg:,drasil:]: A toolkit for building verified file systems Crash refinement

◮ A new definition of file-system correctness ◮ Enable modularity to scale verification

Case study: The Yxv6 file system

◮ Similar to ext3 and xv6, but guarantees

functional correctness and crash safety

7 / 24

slide-8
SLIDE 8

Yggdrasil Overview (Green – trusted components)

specification implementation consistency invariants verifier compiler visualizer C code for file system counterexample fail pass

8 / 24

slide-9
SLIDE 9

Yggdrasil Overview (Green – trusted components)

specification implementation consistency invariants verifier compiler visualizer C code for file system counterexample fail pass

8 / 24

slide-10
SLIDE 10

Yggdrasil Overview (Green – trusted components)

specification implementation consistency invariants verifier compiler visualizer C code for file system counterexample fail pass

class TxnDisk(BaseSpec): def begin_tx(self): self._txn = [] def write_tx(self, bid, data): self._cache = self._cache.update(bid, data) self._txn.append((bid, data)) def commit_tx(self): with self._mach.transaction(): for bid, data in self._txn: self._disk = self._disk.update(bid, data)

8 / 24

slide-11
SLIDE 11

Yggdrasil Overview (Green – trusted components)

specification implementation consistency invariants verifier compiler visualizer C code for file system counterexample fail pass

8 / 24

slide-12
SLIDE 12

Yggdrasil Overview (Green – trusted components)

specification implementation consistency invariants verifier compiler visualizer C code for file system counterexample fail pass

8 / 24

slide-13
SLIDE 13

Yggdrasil Overview (Green – trusted components)

specification implementation consistency invariants verifier compiler visualizer C code for file system counterexample fail pass

8 / 24

slide-14
SLIDE 14

Yggdrasil Overview (Green – trusted components)

specification implementation consistency invariants verifier compiler visualizer C code for file system counterexample fail pass

8 / 24

slide-15
SLIDE 15

Summary of the Yggdrasil toolkit Easy to use, no complex logic required Useful test-cases for bugs Limitations

◮ No concurrency ◮ Unverified Python to C compiler and FUSE 9 / 24

slide-16
SLIDE 16

Straw-man approach to verify FS Model FS as a state machine with a set of

  • perations {mknod, rename, etc.}.

S0 S1 I0 I1 I2 I3 spec impl

Limitation: Doesn’t capture crashes

10 / 24

slide-17
SLIDE 17

Crash refinement: Intuition

S0 S1 I0 I1 I2 I3 spec impl I4 I5

Formalize this intuition

1

Capture crashes explicitly with a crash schedule

2

Use the crash schedule to define correctness

11 / 24

slide-18
SLIDE 18

Crash refinement 1/2: Crash schedule Explicit crash-schedule

◮ A set of boolean variables ◮ Captures crashes and disk reorderings 12 / 24

slide-19
SLIDE 19

Crash refinement 1/2: Crash schedule Explicit crash-schedule

◮ A set of boolean variables ◮ Captures crashes and disk reorderings

Operation Schedule Disk state write(a1, v1) write(a2, v2)

12 / 24

slide-20
SLIDE 20

Crash refinement 1/2: Crash schedule Explicit crash-schedule

◮ A set of boolean variables ◮ Captures crashes and disk reorderings

Operation Schedule Disk state ⇒ write(a1, v1) write(a2, v2)

12 / 24

slide-21
SLIDE 21

Crash refinement 1/2: Crash schedule Explicit crash-schedule

◮ A set of boolean variables ◮ Captures crashes and disk reorderings

Operation Schedule Disk state ⇒ write(a1, v1) {b1} write(a2, v2)

12 / 24

slide-22
SLIDE 22

Crash refinement 1/2: Crash schedule Explicit crash-schedule

◮ A set of boolean variables ◮ Captures crashes and disk reorderings

Operation Schedule Disk state ⇒ write(a1, v1) {b1} d [a1 → if b1 then v1 else old(a1)] write(a2, v2)

12 / 24

slide-23
SLIDE 23

Crash refinement 1/2: Crash schedule Explicit crash-schedule

◮ A set of boolean variables ◮ Captures crashes and disk reorderings

Operation Schedule Disk state write(a1, v1) {b1} d [a1 → if b1 then v1 else old(a1)] ⇒ write(a2, v2)

12 / 24

slide-24
SLIDE 24

Crash refinement 1/2: Crash schedule Explicit crash-schedule

◮ A set of boolean variables ◮ Captures crashes and disk reorderings

Operation Schedule Disk state write(a1, v1) {b1} d [a1 → if b1 then v1 else old(a1)] ⇒ write(a2, v2) {b1, b2}

12 / 24

slide-25
SLIDE 25

Crash refinement 1/2: Crash schedule Explicit crash-schedule

◮ A set of boolean variables ◮ Captures crashes and disk reorderings

Operation Schedule Disk state write(a1, v1) {b1} d [a1 → if b1 then v1 else old(a1)] ⇒ write(a2, v2) {b1, b2} d a1 → if b1 then v1 else old(a1) a2 → if b2 then v2 else old(a2)

  • 12 / 24
slide-26
SLIDE 26

Crash refinement 1/2: Crash schedule Explicit crash-schedule

◮ A set of boolean variables ◮ Captures crashes and disk reorderings

Operation Schedule Disk state write(a1, v1) {b1} d [a1 → if b1 then v1 else old(a1)] write(a2, v2) {b1, b2} d a1 → if b1 then v1 else old(a1) a2 → if b2 then v2 else old(a2)

  • Note: this program can produce 4 possible states

12 / 24

slide-27
SLIDE 27

Crash refinement 2/2: Definition

1 Augment each op in FS with an explicit crash

schedule: op(disk, inp, sched) → disk

2 For each op ∈ FS, prove:

∀disk, inp, schedimpl. ∃schedspec.

  • pspec(disk, inp, schedspec) = opimpl(disk, inp, schedimpl)

Z3 is good at solving this particular form

13 / 24

slide-28
SLIDE 28

Crash refinement 2/2: Definition

1 Augment each op in FS with an explicit crash

schedule: op(disk, inp, sched) → disk

2 For each op ∈ FS, prove:

∀disk, inp, schedimpl. implementation states ∃schedspec.

  • pspec(disk, inp, schedspec) = opimpl(disk, inp, schedimpl)

Z3 is good at solving this particular form

13 / 24

slide-29
SLIDE 29

Crash refinement 2/2: Definition

1 Augment each op in FS with an explicit crash

schedule: op(disk, inp, sched) → disk

2 For each op ∈ FS, prove:

∀disk, inp, schedimpl. ∃schedspec. specification states

  • pspec(disk, inp, schedspec) = opimpl(disk, inp, schedimpl)

Z3 is good at solving this particular form

13 / 24

slide-30
SLIDE 30

Crash refinement summary Amenable to automatic verification using Z3 Enables modular, scalable verification Example: Decouple logical / physical data layout

◮ Verify a simple layout first (ex. one inode per block) ◮ Prove a separate crash-refinement for efficient layout

Example: Stacking of layered abstractions

14 / 24

slide-31
SLIDE 31

Verifying multiple layers: Straw-man approach

regular files, symbolic links, and directories xv6 files xv6 inodes write-ahead logging disk specification implementation

15 / 24

slide-32
SLIDE 32

Verifying multiple layers: Straw-man approach

regular files, symbolic links, and directories xv6 files xv6 inodes write-ahead logging disk specification implementation

15 / 24

slide-33
SLIDE 33

Yxv6 file system: Stack of layered abstractions Each layer has a specification Each layer builds upon a lower layer specification Limit verification to a single layer at a time

regular files, symbolic links, and directories Yxv6 files inodes Yxv6 inodes transactional disk write-ahead logging disk specification implementation Crash refinement Crash refinement Crash refinement

16 / 24

slide-34
SLIDE 34

Implementation using Python and Z3 Two Yxv6 variants

◮ Yxv6+sync: similar to xv6, FSCQ and ext4+sync ◮ Yxv6+group_commit: an optimized Yxv6+sync

component specification implementation consistency inv Yxv6 250 1,500 5 infrastructure – 1,500 – FUSE stub – 250 –

Also built: YminLFS, Ycp and Ylog No manual proofs!

17 / 24

slide-35
SLIDE 35

Yxv6 evaluation

1 How long does it take to verify? 2 Is the implementation actually correct? 3 What is the development effort for Yxv6? 4 Is the performance of Yxv6 reasonable? 18 / 24

slide-36
SLIDE 36

Yxv6 evaluation 1/4: Verification time Let’s see how many days it takes to verify Yxv6+sync

19 / 24

slide-37
SLIDE 37

Yxv6 evaluation 2/4: Correctness Passed Linux testing project’s fsstress Passed SibylFS [SOSP ’15] Posix compliance test

◮ Except for incomplete features (ex. hard links, acl)

Passed manual crash and inspection tests Self hosting its development on Linux

20 / 24

slide-38
SLIDE 38

Yxv6 evaluation 3/4: Development effort 4 months exploring ways to scale verification 2-3 months building Yxv6+sync until self hosting Past 6 months: Experiment with optimizations

21 / 24

slide-39
SLIDE 39

Yxv6 evaluation 4/4: Runtime performance Yxv6+sync similar to FSCQ and ext4+sync Yxv6+group_commit

◮ 3–150× faster than ext4+sync ◮ Within 10× of ext4+default 0.001 0.01 0.1 1 10 100 1000 Make Bash Mailbench Running time in seconds fscq ext4+sync yxv6+sync 22 / 24

slide-40
SLIDE 40

Yxv6 evaluation 4/4: Runtime performance Yxv6+sync similar to FSCQ and ext4+sync Yxv6+group_commit

◮ 3–150× faster than ext4+sync ◮ Within 10× of ext4+default 0.001 0.01 0.1 1 10 100 1000 Make Bash Mailbench Running time in seconds fscq ext4+sync yxv6+sync yxv6+group_commit ext4+default 22 / 24

slide-41
SLIDE 41

Yxv6 evaluation 1/4: Verification time revisited

23 / 24

slide-42
SLIDE 42

Yxv6 evaluation 1/4: Verification time revisited Yxv6+sync takes about a minute to verify Yxv6+group_commit

◮ Larger log, longer verification time. ◮ 1.6 hours using 24 cores 23 / 24

slide-43
SLIDE 43

Conclusion Push-button verification is feasible for FS

◮ No manual proofs on implementation ◮ Generate test-cases for bugs

Design systems to enable automatic, modular verification Towards integration of verification into daily development

24 / 24