Finding Semantic Bugs in File Systems with an Extensible Fuzzing - PowerPoint PPT Presentation

Finding Semantic Bugs in File Systems with an Extensible Fuzzing Framework Seulbae Kim, Meng Xu * , Sanidhya Kashyap * , Jungyeon Yoon, Wen Xu, Taesoo Kim * On the job market

Demonstration Fuzzing F2FS in Linux v5.0-rc7 for crash consistency Result at the end of the talk! 2

Question: Can file systems be bug-free? 3

Can file systems be bug-free? ● Code base is massive 4

Can file systems be bug-free? Not likely ● Code base is massive 39 KLoC 98 KLoC 94 KLoC + common VFS layer (53 KLoC)! 5

Can file systems be bug-free? Not likely ● Code base is massive and evolving 39 KLoC 98 KLoC 94 KLoC 6

Can file systems be bug-free? Not likely ● Code base is massive and evolving 39 KLoC 98 KLoC 94 KLoC 100+ ext4, Btrfs, XFS bugs were reported in 2019 7

File system bugs are devastating ● Bugs and effects Crash consistency bug Data loss / corruption ! Specification violation Unexpected runtime error ! Logic bug Incorrect result ! Memory error DoS / Privilege escalation ! 8

Previous approaches to find FS bugs Regression Model Verified Fuzzing Testing Checking File System FiSC (OSDI’04) FSCQ (SOSP’15) Linux Test Project eXplode (OSDI’06) Syzkaller (Google) Yggdrasil (OSDI’16) xfstests Juxta kAFL (SOSP’15) (Security’17) DFSCQ (SOSP’17) fsck Ferrite (ASPLOS’16) Janus (S&P’19) SFSCQ (OSDI’18) B3 (OSDI’18) 9

Previous approaches to find FS bugs Regression Model Verified Fuzzing Testing Checking File System FiSC (OSDI’04) FSCQ (SOSP’15) Linux Test Project eXplode (OSDI’06) Syzkaller (Google) Yggdrasil (OSDI’16) xfstests Juxta kAFL (SOSP’15) (Security’17) DFSCQ (SOSP’17) fsck Ferrite (ASPLOS’16) Janus (S&P’19) SFSCQ (OSDI’18) B3 (OSDI’18) Only test known cases 10

Previous approaches to find FS bugs Regression Model Verified Fuzzing Testing Checking File System FiSC (OSDI’04) FSCQ (SOSP’15) Linux Test Project eXplode (OSDI’06) Syzkaller (Google) Yggdrasil (OSDI’16) xfstests Juxta kAFL (SOSP’15) (Security’17) DFSCQ (SOSP’17) fsck Ferrite (ASPLOS’16) Janus (S&P’19) SFSCQ (OSDI’18) B3 (OSDI’18) High false positive Only test Limited to known known cases test cases 11

Previous approaches to find FS bugs Regression Model Verified Fuzzing Testing Checking File System FiSC (OSDI’04) FSCQ (SOSP’15) Linux Test Project eXplode (OSDI’06) Syzkaller (Google) Yggdrasil (OSDI’16) xfstests Juxta kAFL (SOSP’15) (Security’17) DFSCQ (SOSP’17) fsck Ferrite (ASPLOS’16) Janus (S&P’19) SFSCQ (OSDI’18) B3 (OSDI’18) High false positive Only test Large unverified parts Limited to known known cases (buggy) test cases 12

Previous approaches to find FS bugs Regression Model Verified Fuzzing Testing Checking File System FiSC FiSC (OSDI’04) (OSDI’04) FSCQ (SOSP’15) Linux Test Project eXplode eXplode (OSDI’06) (OSDI’06) Syzkaller (Google) Yggdrasil (OSDI’16) xfstests Juxta Juxta kAFL (SOSP’15) (SOSP’15) (Security’17) DFSCQ (SOSP’17) fsck Ferrite Ferrite (ASPLOS’16) (ASPLOS’16) Janus (S&P’19) SFSCQ (OSDI’18) B3 B3 (OSDI’18) (OSDI’18) High false positive Only test Large unverified parts Limited to known ? known cases (buggy) test cases 13

Our approach: Fuzzing file systems ● Feedback-driven fuzzing is a complementary solution ○ Produces effective test cases on-the-fly 🙃 ○ Proven to be scalable in practice 🙃 ○ ● Known file system fuzzers ○ VM-based kernel fuzzers ■ kAFL (Security’17), Syzkaller (Google) ○ LibOS-based fuzzer ■ Janus (S&P’19) - our previous work! 14

Our approach: Fuzzing file systems ● Feedback-driven fuzzing is a complementary solution ○ Produces effective test cases on-the-fly 🙃 🙃 Janus discovered 90 memory-safety bugs ○ Proven to be scalable in practice from file systems in 2018 🙃 ○ ● Known file system fuzzers ○ VM-based kernel fuzzers ■ kAFL (Security’17), Syzkaller (Google) ○ LibOS-based fuzzer ■ Janus (S&P’19) - our previous work! 15

Our approach: Fuzzing file systems ● Feedback-driven fuzzing is a complementary solution ○ Produces effective test cases on-the-fly 🙃 🙃 Janus discovered 90 memory-safety bugs ○ Proven to be scalable in practice from file systems in 2018 🙃 ○ ● Known file system fuzzers ○ VM-based kernel fuzzers ■ kAFL (Security’17), Syzkaller (Google) However, existing file system fuzzers ○ LibOS-based fuzzer ■ focus only on memory-safety bugs 🙂 Janus (S&P’19) - our previous work! 16

File system bugs in various flavors ● Memory-safety bugs (focus of existing fuzzers) 12 % (219) 88 % (1786) *Reference: Lu, Lanyue, et al. “A study of Linux file system evolution.” 17 FAST’13

File system bugs in various flavors ● Memory-safety bugs (focus of existing fuzzers) 12 % (219) ● Semantic bugs ○ Crash consistency bug 88 % ○ Specification violation (1786) ○ Logic bug ○ ... *Reference: Lu, Lanyue, et al. “A study of Linux file system evolution.” 18 FAST’13

File system bugs in various flavors ● Memory-safety bugs (focus of existing fuzzers) 12 % (219) ● Semantic bugs ○ Crash consistency bug 88 % We’d like to take advantage of fuzzing ○ Specification violation (1786) ○ Logic bug for finding semantic bugs ○ ... *Reference: Lu, Lanyue, et al. “A study of Linux file system evolution.” 19 FAST’13

Challenge: Semantic bugs are harder to detect ● Key idea in fuzzing: “Crashes” are feedback to fuzzers Fuzzing for memory-safety bugs Target FUZZER program 20

Challenge: Semantic bugs are harder to detect ● Key idea in fuzzing: “Crashes” are feedback to fuzzers Fuzzing for memory-safety bugs input Target FUZZER program 21

Challenge: Semantic bugs are harder to detect ● Key idea in fuzzing: “Crashes” are feedback to fuzzers Fuzzing for memory-safety bugs input Target FUZZER program if BUG, crash 22

Challenge: Semantic bugs are harder to detect ● Key idea in fuzzing: “Crashes” are feedback to fuzzers Fuzzing for memory-safety bugs input Target FUZZER program if BUG, crash feedback (e.g., SIGSEGV) Detected! 23

Challenge: Semantic bugs are harder to detect ● Problem: Semantic bugs fail SILENTLY (i.e., no feedback) Fuzzing for semantic bugs Fuzzing for memory-safety bugs (e.g., spec. violation) input Target Target FUZZER FUZZER program program if BUG, crash feedback (e.g., SIGSEGV) Detected! 24

Challenge: Semantic bugs are harder to detect ● Problem: Semantic bugs fail SILENTLY (i.e., no feedback) Fuzzing for semantic bugs Fuzzing for memory-safety bugs (e.g., spec. violation) input input Target Target FUZZER FUZZER program program if BUG, crash feedback (e.g., SIGSEGV) Detected! 25

Challenge: Semantic bugs are harder to detect ● Problem: Semantic bugs fail SILENTLY (i.e., no feedback) Fuzzing for semantic bugs Fuzzing for memory-safety bugs (e.g., spec. violation) input input Target Target FUZZER FUZZER program program if BUG, function returns ? if BUG, crash feedback a wrong value internally (e.g., SIGSEGV) Detected! 26

Challenge: Semantic bugs are harder to detect ● Problem: Semantic bugs fail SILENTLY (i.e., no feedback) Fuzzing for semantic bugs Fuzzing for memory-safety bugs (e.g., spec. violation) input input Target Target FUZZER FUZZER program program if BUG, function returns ? if BUG, crash feedback a wrong value internally (e.g., SIGSEGV) Not detected 🙂 Detected! 27

Challenge: Semantic bugs are harder to detect ● Problem: Semantic bugs fail SILENTLY (i.e., no feedback) Fuzzing for semantic bugs Fuzzing for memory-safety bugs (e.g., spec. violation) input input Target Target FUZZER FUZZER program program if BUG, function returns ! if BUG, crash feedback a wrong value internally (e.g., SIGSEGV) Detected! Checker feedback Detected 🙃 28

Challenge: Semantic bugs are harder to detect ● Problem: Semantic bugs fail SILENTLY (i.e., no feedback) Fuzzing for semantic bugs Fuzzing for memory-safety bugs (e.g., spec. violation) input input Target Target FUZZER FUZZER program program if BUG, function returns ! if BUG, crash feedback a wrong value internally (e.g., SIGSEGV) Accurate checker for each bug type Retval Detected! checker signal needs to be integrated to fuzzing! Detected :) 29

Proposed solution: Hydra A turnkey solution for file system fuzzing 30

HYDRA overview (high-level) LibOS-based Test case BUG! Input generator Checker Test Executor Feedback 31

HYDRA overview - Input generator AFL variant* LibOS-based Test case BUG! Input generator Checker Test Executor Feedback 32 * Fuzzing File Systems via Two-Dimensional Input Space Exploration - IEEE S&P 2019

HYDRA overview - Test case FS image + AFL variant* System calls LibOS-based Test case BUG! Input generator Checker Test Executor Feedback 33 * Fuzzing File Systems via Two-Dimensional Input Space Exploration - IEEE S&P 2019