Finding Semantic Bugs in File Systems with an Extensible Fuzzing - - PowerPoint PPT Presentation

Finding Semantic Bugs in File Systems with an Extensible Fuzzing Framework

Seulbae Kim, Meng Xu*, Sanidhya Kashyap*, Jungyeon Yoon, Wen Xu, Taesoo Kim

* On the job market

Fuzzing F2FS in Linux v5.0-rc7 for crash consistency Result at the end of the talk!

Demonstration

2

Question: Can file systems be bug-free?

3

Can file systems be bug-free?

• Code base is massive

4

• Code base is massive

Can file systems be bug-free? Not likely

5

39 KLoC 98 KLoC 94 KLoC

+ common VFS layer (53 KLoC)!

Can file systems be bug-free? Not likely

• Code base is massive and evolving

39 KLoC 98 KLoC 94 KLoC

6

Can file systems be bug-free? Not likely

• Code base is massive and evolving

7

100+ ext4, Btrfs, XFS bugs were reported in 2019

39 KLoC 98 KLoC 94 KLoC

File system bugs are devastating

• Bugs and effects

8

Crash consistency bug Specification violation Logic bug Memory error Data loss / corruption

!

Unexpected runtime error

! !

DoS / Privilege escalation

!

Incorrect result

Previous approaches to find FS bugs

9

Regression Testing Model Checking Verified File System Fuzzing

Linux Test Project xfstests fsck FSCQ Yggdrasil DFSCQ SFSCQ (SOSP’15) (OSDI’16) (SOSP’17) (OSDI’18) FiSC eXplode Juxta Ferrite B3 (OSDI’04) (OSDI’06) (SOSP’15) (ASPLOS’16) (OSDI’18) Syzkaller kAFL Janus (Google) (Security’17) (S&P’19)

Previous approaches to find FS bugs

10

Regression Testing Model Checking Verified File System Fuzzing

Linux Test Project xfstests fsck Only test known cases FSCQ Yggdrasil DFSCQ SFSCQ (SOSP’15) (OSDI’16) (SOSP’17) (OSDI’18) FiSC eXplode Juxta Ferrite B3 (OSDI’04) (OSDI’06) (SOSP’15) (ASPLOS’16) (OSDI’18) Syzkaller kAFL Janus (Google) (Security’17) (S&P’19)

Previous approaches to find FS bugs

11

Regression Testing Model Checking Verified File System Fuzzing

Linux Test Project xfstests fsck Only test known cases High false positive Limited to known test cases FSCQ Yggdrasil DFSCQ SFSCQ (SOSP’15) (OSDI’16) (SOSP’17) (OSDI’18) FiSC eXplode Juxta Ferrite B3 (OSDI’04) (OSDI’06) (SOSP’15) (ASPLOS’16) (OSDI’18) Syzkaller kAFL Janus (Google) (Security’17) (S&P’19)

Previous approaches to find FS bugs

12

Regression Testing Model Checking Verified File System Fuzzing

Linux Test Project xfstests fsck Only test known cases High false positive Limited to known test cases

Large unverified parts (buggy)

FSCQ Yggdrasil DFSCQ SFSCQ (SOSP’15) (OSDI’16) (SOSP’17) (OSDI’18) FiSC eXplode Juxta Ferrite B3 (OSDI’04) (OSDI’06) (SOSP’15) (ASPLOS’16) (OSDI’18) Syzkaller kAFL Janus (Google) (Security’17) (S&P’19)

Previous approaches to find FS bugs

13

Regression Testing Model Checking Verified File System Fuzzing

Linux Test Project xfstests fsck FiSC eXplode Juxta Ferrite B3 Only test known cases High false positive Limited to known test cases

Large unverified parts (buggy) ?

(OSDI’04) (OSDI’06) (SOSP’15) (ASPLOS’16) (OSDI’18) FSCQ Yggdrasil DFSCQ SFSCQ (SOSP’15) (OSDI’16) (SOSP’17) (OSDI’18) FiSC eXplode Juxta Ferrite B3 (OSDI’04) (OSDI’06) (SOSP’15) (ASPLOS’16) (OSDI’18) Syzkaller kAFL Janus (Google) (Security’17) (S&P’19)

• Feedback-driven fuzzing is a complementary solution

○ Produces effective test cases on-the-fly ○ Proven to be scalable in practice ○

• Known file system fuzzers

○ VM-based kernel fuzzers

■ kAFL (Security’17), Syzkaller (Google)

○ LibOS-based fuzzer

■ Janus (S&P’19) - our previous work!

Our approach: Fuzzing file systems

14

🙃 🙃

• Feedback-driven fuzzing is a complementary solution

○ Produces effective test cases on-the-fly ○ Proven to be scalable in practice ○

• Known file system fuzzers

○ VM-based kernel fuzzers

■ kAFL (Security’17), Syzkaller (Google)

○ LibOS-based fuzzer

■ Janus (S&P’19) - our previous work!

Our approach: Fuzzing file systems

15

🙃 🙃 Janus discovered 90 memory-safety bugs

from file systems in 2018 🙃

• Feedback-driven fuzzing is a complementary solution

○ Produces effective test cases on-the-fly ○ Proven to be scalable in practice ○

• Known file system fuzzers

○ VM-based kernel fuzzers

■ kAFL (Security’17), Syzkaller (Google)

○ LibOS-based fuzzer

■ Janus (S&P’19) - our previous work!

Our approach: Fuzzing file systems

16

🙃 🙃 Janus discovered 90 memory-safety bugs

from file systems in 2018 🙃 However, existing file system fuzzers focus only on memory-safety bugs 🙂

File system bugs in various flavors

• Memory-safety bugs

(focus of existing fuzzers)

17

12 % (219) 88 % (1786)

*Reference: Lu, Lanyue, et al. “A study of Linux file system evolution.” FAST’13

File system bugs in various flavors

• Memory-safety bugs

(focus of existing fuzzers)

• Semantic bugs

○ Crash consistency bug ○ Specification violation ○ Logic bug ○ ...

18

*Reference: Lu, Lanyue, et al. “A study of Linux file system evolution.” FAST’13

12 % (219) 88 % (1786)

File system bugs in various flavors

• Memory-safety bugs

(focus of existing fuzzers)

• Semantic bugs

○ Crash consistency bug ○ Specification violation ○ Logic bug ○ ...

19

*Reference: Lu, Lanyue, et al. “A study of Linux file system evolution.” FAST’13

12 % (219) 88 % (1786)

We’d like to take advantage of fuzzing for finding semantic bugs

Challenge: Semantic bugs are harder to detect

• Key idea in fuzzing: “Crashes” are feedback to fuzzers

20

FUZZER Target program

Fuzzing for memory-safety bugs

Challenge: Semantic bugs are harder to detect

• Key idea in fuzzing: “Crashes” are feedback to fuzzers

21

FUZZER Target program input

Fuzzing for memory-safety bugs

Challenge: Semantic bugs are harder to detect

• Key idea in fuzzing: “Crashes” are feedback to fuzzers

22

FUZZER Target program input

Fuzzing for memory-safety bugs

if BUG, crash

Challenge: Semantic bugs are harder to detect

• Key idea in fuzzing: “Crashes” are feedback to fuzzers

23

FUZZER Target program input

Fuzzing for memory-safety bugs

feedback (e.g., SIGSEGV)

Detected!

if BUG, crash

Challenge: Semantic bugs are harder to detect

• Problem: Semantic bugs fail SILENTLY (i.e., no feedback)

24

FUZZER Target program input

Fuzzing for memory-safety bugs

feedback (e.g., SIGSEGV) FUZZER Target program if BUG, crash

Fuzzing for semantic bugs (e.g., spec. violation)

Detected!

Challenge: Semantic bugs are harder to detect

• Problem: Semantic bugs fail SILENTLY (i.e., no feedback)

25

FUZZER Target program input

Fuzzing for memory-safety bugs

feedback (e.g., SIGSEGV) FUZZER Target program input if BUG, crash

Fuzzing for semantic bugs (e.g., spec. violation)

Detected!

Challenge: Semantic bugs are harder to detect

• Problem: Semantic bugs fail SILENTLY (i.e., no feedback)

26

FUZZER Target program input

Fuzzing for memory-safety bugs

feedback (e.g., SIGSEGV) FUZZER Target program input

Fuzzing for semantic bugs (e.g., spec. violation)

if BUG, function returns a wrong value internally if BUG, crash

Detected!

?

Challenge: Semantic bugs are harder to detect

• Problem: Semantic bugs fail SILENTLY (i.e., no feedback)

27

FUZZER Target program input

Fuzzing for memory-safety bugs

if BUG, crash feedback (e.g., SIGSEGV) FUZZER Target program

Fuzzing for semantic bugs (e.g., spec. violation)

input if BUG, function returns a wrong value internally

Detected! Not detected 🙂

?

Challenge: Semantic bugs are harder to detect

• Problem: Semantic bugs fail SILENTLY (i.e., no feedback)

28

FUZZER Target program input

Fuzzing for memory-safety bugs

if BUG, crash feedback (e.g., SIGSEGV) FUZZER Target program

Fuzzing for semantic bugs (e.g., spec. violation)

input

!

Detected! Detected 🙃

Checker if BUG, function returns a wrong value internally feedback

Challenge: Semantic bugs are harder to detect

• Problem: Semantic bugs fail SILENTLY (i.e., no feedback)

29

FUZZER Target program input

Fuzzing for memory-safety bugs

if BUG, crash feedback (e.g., SIGSEGV) FUZZER Target program

Fuzzing for semantic bugs (e.g., spec. violation)

input

!

Detected! Detected :)

Retval checker if BUG, function returns a wrong value internally signal

Accurate checker for each bug type needs to be integrated to fuzzing!

Proposed solution: Hydra

A turnkey solution for file system fuzzing

30

HYDRA overview (high-level)

31

Input generator Test case LibOS-based Test Executor Checker BUG! Feedback

HYDRA overview - Input generator

32

Input generator Test case LibOS-based Test Executor Checker BUG! Feedback AFL variant*

* Fuzzing File Systems via Two-Dimensional Input Space Exploration - IEEE S&P 2019

HYDRA overview - Test case

33

Input generator Test case LibOS-based Test Executor Checker BUG! Feedback

* Fuzzing File Systems via Two-Dimensional Input Space Exploration - IEEE S&P 2019

AFL variant* FS image + System calls

HYDRA overview - LibOS-based test executor

34

Input generator Test case LibOS-based Test Executor Checker BUG! Feedback

* Fuzzing File Systems via Two-Dimensional Input Space Exploration - IEEE S&P 2019

Mount img, exec syscalls AFL variant* FS image + System calls

HYDRA overview - Checker

35

Input generator Test case LibOS-based Test Executor Checker BUG! Feedback

* Fuzzing File Systems via Two-Dimensional Input Space Exploration - IEEE S&P 2019

FS image + System calls Check for bug AFL variant* Mount img, exec syscalls

HYDRA overview - Feedback

36

Input generator Test case LibOS-based Test Executor Checker BUG! Feedback

* Fuzzing File Systems via Two-Dimensional Input Space Exploration - IEEE S&P 2019

FS image + System calls Check for bug AFL variant* Mount img, exec syscalls

• FS-specific code coverage
• Checker-defined signal

Hydra framework takes care of

37

Input generator Test case LibOS-based Test Executor Checker BUG! Feedback

• Automated input space exploration
• Test execution
• Incorporation of checkers, ...
• Develop and

plug-in a bug checker

In the meantime.. a tester can

38

Input generator Test case LibOS-based Test Executor Checker BUG! Feedback

• Automated input space exploration
• Test execution
• Incorporation of checkers, ...
• Develop and plug-in

a specialized bug checker

Separation of concern!

39

Input generator Test case LibOS-based Test Executor Checker BUG! Feedback

• Develop and plug-in

a specialized bug checker

• Automated input space exploration
• Test execution
• Incorporation of checkers, ...

Developers may focus solely on describing the bugs of their own interests

• Through pluggable checkers

Hydra is extensible!

40

Input generator Test case LibOS-based Test Executor BUG! Feedback Crash consistency bug Consistency checker e.g., SymC3

• Spec. Violation

POSIX checker e.g., SibylFS Logic bug Built-in FS checks Memory safety bug Address sanitizer e.g., KASAN In-house developed checker Existing oracle, with few lines for integration In-kernel checker, used as is

• Through pluggable checkers

Hydra is extensible!

41

Input generator Test case LibOS-based Test Executor BUG! Feedback Crash consistency bug Consistency checker e.g., SymC3

• Spec. Violation

POSIX checker e.g., SibylFS Logic bug Built-in FS checks Memory safety bug Address sanitizer e.g., KASAN In-house developed checker Existing oracle, with few lines for integration In-kernel checker, used as is

Readily extensible to other types of bugs by plugging in relevant checkers

Hydra in action

42

Finding crash consistency bug utilizing SymC3 checker with Hydra

Hydra in action - Crash consistency testing

• SymC3: Symbolically evaluate crashing states

(i.e., keeping in-memory and on-disk states, like real FS implementation)

○ Input : a list of system calls, initial state ○ Output: a list of legitimate post-crash states

43

Hydra in action - Crash consistency testing

• SymC3: Symbolically evaluate crashing states

(i.e., keeping in-memory and on-disk states, like real FS implementation)

○ Input : a list of system calls, initial state ○ Output: a list of legitimate post-crash states

• Checking errors:

44

Test case

Hydra in action - Crash consistency testing

• SymC3: Symbolically evaluate crashing states

(i.e., keeping in-memory and on-disk states, like real FS implementation)

○ Input : a list of system calls, initial state ○ Output: a list of legitimate post-crash states

• Checking errors:

45

Test case LibOS Executor

execute & crash

𝛿: Crash-recovered concrete state

Hydra in action - Crash consistency testing

• SymC3: Symbolically evaluate crashing states

(i.e., keeping in-memory and on-disk states, like real FS implementation)

○ Input : a list of system calls, initial state ○ Output: a list of legitimate post-crash states

• Checking errors:

46

Test case LibOS Executor SymC3

(states contain symbols)

execute & crash emulate

𝚻: Set of legit. states {state1, state2, …} 𝛿: Crash-recovered concrete state

Hydra in action - Crash consistency testing

• SymC3: Symbolically evaluate crashing states

(i.e., keeping in-memory and on-disk states, like real FS implementation)

○ Input : a list of system calls, initial state ○ Output: a list of legitimate post-crash states

• Checking errors:

47

Test case LibOS Executor SymC3 𝚻: Set of legit. states {state1, state2, …} 𝛿: Crash-recovered concrete state

(states contain symbols)

𝛿 ∈ 𝚻 ?

execute & crash emulate

Not bug Bug

True False

Hydra in action - Fuzzer-generated test case

• Simplest test case (but it was a real bug in F2FS!)

48

1 mkdir “A” 0775 2 sync 3 chmod “A” 0600 4 fsync “A”

Hydra in action - Initial emulator states

49

1 mkdir “A” 0775 2 sync 3 chmod “A” 0600 4 fsync “A”

Tree i0 . In-memory i0.dents=[.] On-disk i0.dents=[.] i0 . Snapshots

Initial states in the emulator Store possible inode hierarchy

Hydra in action - Emulation of test case

50

Tree i0 . i1 A i0 . i0 . i1 A In-memory i0.dents=[., A] i1.dents=[.] i1.mode =[0775] On-disk i0.dents=[.] Snapshots

new inode created in memory Store new tree-snapshot

1 mkdir “A” 0775 2 sync 3 chmod “A” 0600 4 fsync “A”

Hydra in action - Emulation of test case

51

Tree i0 . i1 A i0 . i0 . i1 A In-memory i0.dents=[., A] i1.dents=[.] i1.mode =[0775] On-disk i0.dents=[., A] i1.dents=[.] i1.mode =[0775] Snapshots

All metadata flushed to disk

1 mkdir “A” 0775 2 sync 3 chmod “A” 0600 4 fsync “A”

Hydra in action - Emulation of test case

52

Tree i0 . i1 A i0 . i0 . i1 A In-memory i0.dents=[., A] i1.dents=[.] i1.mode =[0775,0600] On-disk i0.dents=[., A] i1.dents=[.] i1.mode =[0775] Snapshots

1 mkdir “A” 0775 2 sync 3 chmod “A” 0600 4 fsync “A”

New metadata is written History of metadata changes is maintained

Hydra in action - Emulation of test case

53

Tree i0 . i1 A i0 . i0 . i1 A In-memory i0.dents=[., A] i1.dents=[.] i1.mode =[0600] On-disk i0.dents=[., A] i1.dents=[.] i1.mode =[0600] Snapshots

1 mkdir “A” 0775 2 sync 3 chmod “A” 0600 4 fsync “A”

i1’s metadata flushed to disk

Hydra in action - End of test case emulation

54

Tree i0 . i1 A i0 . i0 . i1 A In-memory i0.dents=[., A] i1.dents=[.] i1.mode =[0600] On-disk i0.dents=[., A] i1.dents=[.] i1.mode =[0600] Snapshots

1 mkdir “A” 0775 2 sync 3 chmod “A” 0600 4 fsync “A”

Enumerate legitimate post-crash states

Hydra in action - Enumerating legitimate states

55

Snapshots i0 . i0 . i1 A [S1] [S0] In-memory i0.dents=[., A] i1.dents=[.] i1.mode =[0600] On-disk i0.dents=[., A] i1.dents=[.] i1.mode =[0600]

Hydra in action - Enumerating legitimate states

56

i0 . i0 . i1 A [S1] [S0] In-memory i0.dents=[., A] i1.dents=[.] i1.mode =[0600] On-disk i0.dents=[., A] i1.dents=[.] i1.mode =[0600] Drop S0 (i1 is persisted) Snapshots ./A must exist!

1. Check validity of snapshots

Hydra in action - Enumerating legitimate states

57

i0 . i0 . i1 A [S1] [S0] In-memory i0.dents=[., A] i1.dents=[.] i1.mode =[0600] On-disk i0.dents=[., A] i1.dents=[.] i1.mode =[0600] Snapshots

1. Check validity of snapshots

S1 is valid (does not violate persisted state)

Hydra in action - Enumerating legitimate states

58

i0 . i0 . i1 A [S1] [S0] In-memory i0.dents=[., A] i1.dents=[.] i1.mode =[0600] On-disk i0.dents=[., A] i1.dents=[.] i1.mode =[0600] [Post-crash state 1] i0 - name: . i1 - name: ./A mode: 0600 Snapshots

2. Generate possible crash states from valid snapshots

Hydra in action - Bug checking

59

[Post-crash state 1] i0 - name: . i1 - name: ./A mode: 0600 Crashed F2FS image from Executor (𝛿) $ cd mnt_point $ stat A Access: (0775/drwxrwxr-x)

𝚻

?

∋

3. Check if the set of legitimate states 𝚻 has crashed state 𝛿 as a member

Hydra in action - Bug found

60

[Post-crash state 1] i0 - name: . i1 - name: ./A mode: 0600 Crashed F2FS image from Executor (𝛿) $ cd mnt_point $ stat A Access: (0775/drwxrwxr-x)

3. Check if the set of legitimate states 𝚻 has crashed state 𝛿 as a member

𝚻

∌

None of the states have A’s mode as 0775. This is a bug! (reported and patched)

Evaluation

Effectiveness and performance as a fuzzing framework

61

Evaluation - Hydra is effective

• Hydra found 36 new semantic bugs (+ 33 memory errors)

○ including a crash consistency bug in FSCQ, a verified file system

62

File System

(checker)

Crash Consistency

(SymC3)

Logic Bugs

(In-kernel checks)

• Spec. Violation

(SibylFS)

ext4 1 1 Btrfs 4 7 2 F2FS 3 16 1 FSCQ 1

• Total

9 23 4

Evaluation - Hydra is effective

• Hydra found 36 new semantic bugs (+ 33 memory errors)

○ including a crash consistency bug in FSCQ, a verified file system

63

File System

(checker)

Crash Consistency

(SymC3)

Logic Bugs

(In-kernel checks)

• Spec. Violation

(SibylFS)

ext4 1 1 Btrfs 4 7 2 F2FS 3 16 1 FSCQ 1

• Total

9 23 4 Bug: dir is lost upon crash, if another file is truncated Dev: “ftruncate was broken, and used an unverified helper function”

Evaluation - Hydra quickly explores input space

• Performance of Hydra’s state exploration with checkers

64

Logic bugs (102.8 exec/sec) Memory safety (98.4 exec/sec) Crash consistency (11.4 exec/sec) POSIX conformance (4.5 exec/sec) Checker overhead (ms) Throughput (exec/sec)

Evaluation - Hydra quickly explores input space

• Faster than VM-based kernel fuzzing

65

Logic bugs (102.8 exec/sec) Memory safety (98.4 exec/sec) Crash consistency (11.4 exec/sec) POSIX conformance (4.5 exec/sec) Checker overhead (ms) Throughput (exec/sec) VM-based approach (0.7 exec/sec)

Evaluation - Hydra generates better test cases

• ext4 code coverage of Hydra vs kernel fuzzers

66

Code coverage (12 hours) Hydra Syzkaller kAFL

Evaluation - Hydra generates better test cases

• Hydra reaches more code paths

67

Code coverage (12 hours) Hydra Syzkaller kAFL 1.6x 8.7x

Evaluation - Hydra test cases vs B3 test suite

• B3 generates test cases by enumerating FS operations

○ Limits input space with bounds (e.g., #ops <= 3)

68

Hydra B3

(used up all B3 test cases)

Code coverage (12 hours)

Evaluation - Hydra test cases vs B3 test suite

• B3 generates test cases by enumerating FS operations

○ Limits input space with bounds (e.g., #ops <= 3)

69

Hydra B3

(used up all B3 test cases)

Code coverage (12 hours)

B3’s enumerated test cases explore less code Hydra generates test cases on-the-fly

Evaluation - Hydra test cases vs B3 test suite

• B3 generates test cases by enumerating FS operations

○ Limits input space with bounds (e.g., #ops <= 3)

70

Hydra B3

(used up all B3 test cases)

Code coverage (12 hours)

B3’s enumerated test cases explore less code

B3 missed all of the crash consistency bugs found by Hydra & SymC3

Hydra generates test cases on-the-fly

Summary

• Hydra is an extensible fuzzing framework for one-stop testing
• n multiple aspects of file systems

○ Open-sourced at https://github.com/sslab-gatech/hydra

71

Summary

• Hydra is an extensible fuzzing framework for one-stop testing
• n multiple aspects of file systems

○ Open-sourced at https://github.com/sslab-gatech/hydra

• Discovered hard-to-detect semantic bugs (& memory bugs)

○ 9 crash consistency bugs (1 in verified file system, FSCQ) ○ 4 POSIX violations, 23 Logic bugs, and 33 memory bugs

72

Summary

• Hydra is an extensible fuzzing framework for one-stop testing
• n multiple aspects of file systems

○ Open-sourced at https://github.com/sslab-gatech/hydra

• Discovered hard-to-detect semantic bugs (& memory bugs)

○ 9 crash consistency bugs (1 in verified file system, FSCQ) ○ 4 POSIX violations, 23 Logic bugs, and 33 memory bugs

• Further extensions as future work

○ More bug checkers, e.g., data race checker ○ Support for distributed file systems

73

Demonstration - fuzzing for 10 mins

Wait, the fuzzing result?

Thank you!

Q & A

This research is supported by