Verifying a high-performance crash-safe file system using a tree - - PowerPoint PPT Presentation

Verifying a high-performance crash-safe file system using a tree specifica6on

Haogang Chen, Tej Chajed, Stephanie Wang, Alex Konradi, Atalay İleri, Adam Chlipala, M. Frans Kaashoek, Nickolai Zeldovich

File systems are difficult to make correct

• Complicated implementa6ons
• on-disk layout
• in-memory data structures
• Computer can crash at any 6me

2

Despite much effort, file systems have bugs

• File systems s6ll have subtle bugs
• Well documented [Lu, TOS ’14] [Min, SOSP ’15]
• Example from ext4:


combina6on of two op6miza6ons allows data to leak from one file to another on crash

• Discovered a[er 6 years [Kara 2014]

3

Approach: formal verifica6on

• Write a specifica6on
• Prove implementa6on meets the specifica6on
• Ensures implementa6on handles all corner cases
• Proof assistant (Coq) ensures proof is correct
• Avoid large class of bugs

4

Exis6ng verified file systems

5

FSCQ [SOSP ’15] BilbyFS [ASPLOS ’16] Yggdrasil [OSDI ’16] ext4 btrfs ZFS

correctness performance

verified file systems

Goal: verified high-performance file system

6

FSCQ [SOSP ’15] BilbyFS [ASPLOS ’16] Yggdrasil [OSDI ’16] ext4 btrfs ZFS

correctness performance

?

verified file systems

Strawman: op6mize FSCQ

7

correctness performance

FSCQ code

Strawman: op6mize FSCQ

7

correctness performance

FSCQ code spec proof

Strawman: op6mize FSCQ

7

correctness performance

FSCQ code spec proof fast FSCQ proof?

Problem: specifica6on incompa6ble with high performance

• Achieving high performance requires op6miza6ons
• Some op6miza6ons change file-system behavior
• Requires changes to specifica6on

8

Example op6miza6on: deferred commit

• Deferred commit: buffer system calls un6l fsync
• FSCQ’s specifica6on: “if create(f) has returned

and computer crashes, f exists”

• Deferred commit requires a new specifica6on

9

Op6miza6ons that change crash behavior

• Deferred commit: buffer system calls un6l fsync
• Log-bypass writes: skip log for data writes
• Buffer cache: cache data un6l fdatasync
• Exis*ng specifica*ons do not support these
• p*miza*ons

10

Contribu6on: DFSCQ file system

• Precise specifica6on for a subset of POSIX
• supports deferred commit and log-bypass writes
• Verified, crash-safe file system
• Tradi6onal journalling file-system design
• Implements most of ext4’s op6miza6ons
• Machine-checked proof that implementa6on meets specifica6on
• Performance on par with ext4 (but DFSCQ has fewer features)

11

Specifying a file system

• Design abstract state

12

Specifying a file system

• Design abstract state
• Describe how system calls execute

12

Specifying a file system

• Design abstract state
• Describe how system calls execute
• Describe effect of crashes

12

Star6ng point: tree as abstract state

Trees are a simplified abstrac6on of a file system

13

g f

Specifica6on abstracts implementa6on details

14

g f

abstract state implementa6on’s state

Specify how system calls affect abstract state

15

unlink(g) g f f unlink(g)

specifica6on describes transi6on

Challenges in specifying crash behavior

• Op6miza6ons mean crashes can be complex
• Problem 1: deferred commit
• Problem 2: log-bypass writes
• Problem 3: caching

16

Problem 1: deferred commit leads to many crash states

17

unlink(g) g f f

Problem 1: deferred commit leads to many crash states

17

unlink(g) g f f

crash: reset memory

Problem 1: deferred commit leads to many crash states

17

unlink(g) g f f

crash: reset memory

g f f

How do we specify crash outcomes with deferred commit?

18

f g f

How do we specify crash outcomes with deferred commit?

18

f g f crash

tree sequence

Specify deferred commit using tree sequences

19

f g

tree sequence

Specify deferred commit using tree sequences

• Abstract state is a sequence of trees

19

f g

tree sequence

Specify deferred commit using tree sequences

• Abstract state is a sequence of trees
• Always read from the latest tree

19

f g

Specify deferred commit using tree sequences

• Metadata updates add new trees in the specifica6on
• Always read from the latest tree

20

unlink(g) f g f f g

Specify deferred commit using tree sequences

• Metadata updates add new trees in the specifica6on
• Always read from the latest tree

21

f g f

Specify deferred commit using tree sequences

• Metadata updates add new trees in the specifica6on
• Always read from the latest tree

22

truncate(f,2) f g f f f g f

Specify deferred commit using tree sequences

• Metadata updates add new trees in the specifica6on
• Always read from the latest tree

23

f g f f

Specify deferred commit using tree sequences

• Metadata updates add new trees in the specifica6on
• Always read from the latest tree

24

f g f f rename(f,/) f f g f f

tree sequence

Behavior of tree sequences on crash

• What about crash behavior?

25

f g f f f

tree sequence

Behavior of tree sequences on crash

• What about crash behavior?

25

f g post-crash tree sequence

crash

f g f f f

Crash specifica6on allows background commits

26

post-crash states: f f f g f tree sequence f g f f f crash

Specifica6on for fsync

27

f g f f f f fsync("/")

Problem 2: log-bypass writes may reorder updates

• Log-bypass writes: update file data blocks in place, skipping log

28

f f

rename

f

write

Problem 2: log-bypass writes may reorder updates

• Log-bypass writes: update file data blocks in place, skipping log
• Effect: data writes and metadata updates can be reordered on crash

28

f f

rename

f

write crash

f

Log-bypass writes

29

f g f f

At minimum, writes to latest tree

f g f f f f write(f,…)

Log-bypass writes

30

Affects the same file in earlier trees

f g f f f f g f f f write(f,…)

Specify that other files are unaffected

31

f g f f

Puts an obliga6on on the implementa6on to avoid block re-use within a tree sequence

b21 b21 b21 f g f f f f

?

write(f,…)

Specify that other files are unaffected

32

f g f f b21 b21 f g f f f f

Puts an obliga6on on the implementa6on to avoid block re-use within a tree sequence

write(f,…) b21

Specify that other files are unaffected

32

f g f f b21 b21 b51 f g f f f f

Puts an obliga6on on the implementa6on to avoid block re-use within a tree sequence

write(f,…) b21 b51

Problem 3: data writes are cached

• Write-back buffer cache

33

f f

write

f

crash

Problem 3: data writes are cached

• Write-back buffer cache
• Data can be persisted in any order

33

f f

write

f f f

crash

f

Specifying data caching: block sets

34

f g f

uncached two possible values: old ( ) and new ( )

f f

f

Behavior of block sets on crash

f f g f f f g f f crash

Behavior of block sets on crash

f f g f f f g f f f f f

two degrees of non-determinism in crash states:

crash

Behavior of block sets on crash

f f g f f f g f f f f f

specifica6on allows metadata and data updates to be reordered

two degrees of non-determinism in crash states:

crash

Specifica6on for fdatasync

37

f g f f f fdatasync(f)

Specifica6on for fdatasync

38

f g f f fdatasync(f) f g f f

fdatasync specifica6on says block sets collapse in every tree

f f

Summary: DFSCQ’s tree-based specifica6on

• metadata opera6ons add a new tree
• fsync collapses to latest tree
• writes update blocksets in every tree
• fdatasync collapses blocksets in every tree

39

Prove implementa6on meets specifica6on

40

length: 2 type: file … stat(g) g f g f length: 2 type: file … stat(g)

Prove implementa6on meets specifica6on

40

length: 2 type: file … stat(g) g f

return values match

g f length: 2 type: file … stat(g)

Prove implementa6on meets specifica6on

40

unlink(g) length: 2 type: file … stat(g) g f

return values match

g f g f f length: 2 type: file … stat(g) unlink(g)

Prove implementa6on meets specifica6on

40

unlink(g) length: 2 type: file … stat(g) g f

return values match disk con6nues to relate to abstract state

g f g f f length: 2 type: file … stat(g) unlink(g)

DFSCQ Design

41

buffer cache logging checksums deferred commit log-bypass API block allocator free-bit cache avoid re-use inode k-indirect blocks dirty blocks directory name cache

Many single-layer op6miza6ons

42

• Affect only proof of single layer

buffer cache logging checksums deferred commit log-bypass API block allocator free-bit cache avoid re-use inode k-indirect blocks dirty blocks directory name cache

Many single-layer op6miza6ons

42

• Affect only proof of single layer

buffer cache logging checksums deferred commit log-bypass API block allocator free-bit cache avoid re-use inode k-indirect blocks dirty blocks directory name cache cache free blocks

Many single-layer op6miza6ons

42

• Affect only proof of single layer

buffer cache logging checksums deferred commit log-bypass API block allocator free-bit cache avoid re-use inode k-indirect blocks dirty blocks directory name cache improves performance with no change to abstrac6on cache free blocks

Cross-layer op6miza6ons

43

buffer cache logging checksums deferred commit log-bypass API block allocator free-bit cache avoid re-use inode k-indirect blocks dirty blocks directory name cache

• Break abstrac6on

boundaries

• Complicate proofs
• Good for performance

Cross-layer op6miza6ons

43

buffer cache logging checksums deferred commit log-bypass API block allocator free-bit cache avoid re-use inode k-indirect blocks dirty blocks directory name cache

• Break abstrac6on

boundaries

• Complicate proofs
• Good for performance

track dirty blocks in the cache

Cross-layer op6miza6ons

43

buffer cache logging checksums deferred commit log-bypass API block allocator free-bit cache avoid re-use inode k-indirect blocks dirty blocks directory name cache

• Break abstrac6on

boundaries

• Complicate proofs
• Good for performance

records dirent offset from inode layer track dirty blocks in the cache

Implementa6on and proof

• Extend FSCQ [SOSP ’15]
• 75,000 lines of Coq (compared to 31,000 in FSCQ)

44

specifica6on code proofs

Coq

OK

Coq proof checker

Running DFSCQ

45

code

Coq code extrac6on

implementa6on

Haskell

Running DFSCQ

45

code

Coq

DFSCQ FUSE server

GHC code extrac6on

implementa6on

Haskell

FUSE interface

Haskell

+

Performance evalua6on

• Several workloads
• micro benchmarks
• applica6on workloads
• Compare with ext4 in default mode
• Running on an SSD on a desktop

46

(see paper for more results)

DFSCQ is compe66ve with ext4

47

files/s 80 160 240 320 400 smallfile FSCQ DFSCQ ext4

DFSCQ is compe66ve with ext4

47

files/s 80 160 240 320 400 smallfile FSCQ DFSCQ ext4 MB/s 36 72 108 144 180 largefile

x

DFSCQ is compe66ve with ext4

• DFSCQ s6ll has high CPU overhead compared to ext4
• Haskell code allocates large amounts of memory

47

files/s 80 160 240 320 400 smallfile FSCQ DFSCQ ext4 MB/s 36 72 108 144 180 largefile

x

DFSCQ outperforms ext4 on mailbench

48

msgs/s 14 28 42 56 70 mailbench FSCQ DFSCQ ext4

DFSCQ outperforms ext4 on mailbench

48

msgs/s 14 28 42 56 70 mailbench FSCQ DFSCQ ext4

• mailbench simulates a qmail-like mail server
• metadata and fsync-heavy workload

SQLite on DFSCQ is compe66ve with ext4

49

txns/s 16 32 48 64 80 TPC-C on SQLite FSCQ DFSCQ ext4

x

SQLite on DFSCQ is compe66ve with ext4

49

txns/s 16 32 48 64 80 TPC-C on SQLite FSCQ DFSCQ ext4

x

• Write-heavy database workload
• DFSCQ issues less I/O, but has higher CPU overhead

Future work

• Reduce CPU overhead
• Concurrency

50

Summary

• DFSCQ: verified, efficient, crash-safe file system
• Precise tree-based specifica*on of


deferred commit and log-bypass writes

• Proof that implementa6on meets specifica6on
• Performance on par with Linux ext4

51

hsps:/ /github.com/mit-pdos/fscq

Backup slides

• ext4 async commit + log-bypass bug
• verifica6on architecture diagram
• write-ahead logging
• group commit
• log-bypass writes
• deferred commit and log-bypass perf
• spec example
• fsync(2)
• atomic write
• FUSE architecture
• LOC

52

Op6miza6ons are hard to implement correctly

Subtle interac6on between op6miza6ons

• bug where crash could leak data in Linux ext4
• discovered a[er 6 years

ext4 now forbids both op6miza6ons

53 Author: Jan Kara <jack@suse.cz> Date: Tue Nov 25 20:19:17 2014 -0500 ext4: forbid journal_async_commit in data=ordered mode [...]

Approach to avoid bugs: verifica6on

54

disk hardware file system applica6on

specifica6on specifica*on

verify FS correct

specifica6on

verify applica6ons

Write-ahead logging

• System calls can update mul6ple disk blocks

55

create(‘d/a’)

(address, block)

Write-ahead logging

• System calls can update mul6ple disk blocks
• Logging ensures all updates are persisted or none

even if computer crashes

55

disk log data create(‘d/a’)

(address, block)

disk

Deferred commit enables high throughput

56

log data

disk

Deferred commit enables high throughput

56

memory log data

• 1. Buffer system calls in memory

disk

Deferred commit enables high throughput

56

➡ mkdir(‘d’) ➡ create(‘d/a’) ➡ rename(‘d/a’, ‘d/b’)

, ,

memory log data

• 1. Buffer system calls in memory

disk

Deferred commit enables high throughput

56

➡ mkdir(‘d’) ➡ create(‘d/a’) ➡ rename(‘d/a’, ‘d/b’) ➡ fsync(‘d’)

, ,

memory log data

• 1. Buffer system calls in memory
• 2. fsync() flushes cached transac6ons

to the on-disk log in a batch

disk

Deferred commit enables high throughput

56

➡ mkdir(‘d’) ➡ create(‘d/a’) ➡ rename(‘d/a’, ‘d/b’) ➡ fsync(‘d’) memory log data

• 1. Buffer system calls in memory
• 2. fsync() flushes cached transac6ons

to the on-disk log in a batch

Log-bypass writes avoid doubling data writes

57

disk log data ➡ mkdir(‘d’) ➡ create(‘d/a’)

• 1. Record metadata updates in log as

usual

transacCon cache

,

Log-bypass writes avoid doubling data writes

57

disk log data ➡ mkdir(‘d’) ➡ create(‘d/a’) ➡ write(‘d/a’,...)

• 1. Record metadata updates in log as

usual

transacCon cache

,

Log-bypass writes avoid doubling data writes

57

disk log data ➡ mkdir(‘d’) ➡ create(‘d/a’) ➡ write(‘d/a’,...)

• 1. Record metadata updates in log as

usual

• 2. Bypass log for file data

transacCon cache

,

Log-bypass writes avoid doubling data writes

57

disk log data ➡ mkdir(‘d’) ➡ create(‘d/a’) ➡ write(‘d/a’,...)

• 1. Record metadata updates in log as

usual

• 2. Bypass log for file data

transacCon cache

,

Deferred commit and log bypass maser in prac6ce

58

ext4 performance largefile synchronous 120 MB/s + deferred commit 150 MB/s + log-bypass 300 MB/s

fdatasync every 10 MB to an SSD

Specifica6ons

59

SPEC unlink(cwd_ino, pathname)
 PRE disk: tree_rep(tree_seq)
 POST disk: tree_rep(tree_seq ++ [new_tree]) /\ new_tree = tree_prune(tree_seq.latest, cwd_ino, pathname) CRASH disk: tree_intact(tree_seq ++ [new_tree])

POSIX manual gives complicated specifica6on

• not clear enough about crash behavior

60

fsync() flushes modified buffer cache pages for fd to the disk device so that all changed informa<on can be retrieved even a=er the system crashes or is rebooted. fsync() also flushes metadata informa<on associated with the file (see inode(7)). fdatasync() is similar to fsync(), but does not flush modified metadata unless that metadata is needed in order to allow a subsequent data retrieval to be correctly handled.

paraphrase of fsync(2) manpage

Evalua6ng the specifica6on: atomic write pasern

61

f tmpfile

rename()

Goal: on crash f either:

• doesn’t exist
• or contains

Proved atomic write pasern crash safe

62

def atomic_write(data, name): with open(tmpfile, "cw") as f: ftruncate(f, len(data)) write(f, data) fdatasync(f) rename(tmpfile, name) fsync(dirname(name))

prepare tmpfile persist data move to des6na6on persist metadata

Proved atomic write pasern crash safe

62

def atomic_write(data, name): with open(tmpfile, "cw") as f: ftruncate(f, len(data)) write(f, data) fdatasync(f) rename(tmpfile, name) fsync(dirname(name))

prepare tmpfile persist data move to des6na6on persist metadata

Specifica6on is sufficient to prove applica6on-level proper6es

Atomic write is correct

63

/tmp name

Specifica6on: on crash, name either does not exist

• r contains data

/tmp name

crash states:

(just a[er rename)

DFSCQ runs ordinary Linux programs using FUSE

64

DFSCQ FUSE server

userspace Linux kernel

$ mv src dst $ git clone FUSE

Effort to implement DFSCQ

• Total of 75,000 lines of verified code, specs, and proofs in Coq
• Compared to FSCQ’s 31,000 lines
• 4,800 lines of implementa6on
• Took 5 authors 2 years (but less than 10 person years)

65

10% 12% 43% 35%

CHL infrastructure FS impl and proofs Top-level API Tree sequences