Finding Race Conditions in Kernels from fuzzing to symbolic - PowerPoint PPT Presentation

Checking data races - ordering (causality) Syscall Timer Workqueue W delayed_work <timer start> <timer end> queue_work <work start> R Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 30

Bring out data races explicitly with a checker Signaled? Data race checker Data race Syscall Program Crashed? Test case generator executor Memory error Feedback code coverage Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 31

A slightly complicated data race G[ … ] is all null at initialization sys_readlink (path, ...): sys_truncate (size, ...): global A = 1; global A = 0; local y; local x; if (size > 4096) { if (IS_DIR(path)) { x = A + 1; y = A * 2; if (! G[ x ] ) if (! G[ y ] ) G[ x ] = kmalloc(...); G[ y ] = kmalloc(...); } } Thread 1 Thread 2 *Assume sequential consistency. Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 32

A slightly complicated data race G[ … ] is all null at initialization sys_readlink (path, ...): sys_truncate (size, ...): global A = 1; global A = 0; local y; local x; if (size > 4096) { if (IS_DIR(path)) { x = A + 1; y = A * 2; if (! G[ x ] ) if (! G[ y ] ) G[ x ] = kmalloc(...); G[ y ] = kmalloc(...); } } Thread 1 Thread 2 *Assume sequential consistency. Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 33

Case simpli fi ed A = 1; A = 0; x = A + 1; y = A * 2; Thread 1 Thread 2 Can we reach x == y? Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 34

Case simpli fi ed A = 1; A = 1; A = 1; x = A + 1; A = 0; A = 0; A = 0; x = A + 1; y = A * 2; y = A * 2; y = A * 2; x = A + 1; A = 1; A = 0; x = A + 1; x = 2, y = 0 x = 1, y = 0 x = 1, y = 0 y = A * 2; Thread 1 Thread 2 A = 0; A = 0; A = 0; Can we reach x == y? y = A * 2; A = 1; A = 1; x = A + 1; A = 1; y = A * 2; x = A + 1; x = A + 1; y = A * 2; x = 2, y = 0 x = 2, y = 2 x = 2, y = 2 Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 35

All interleavings yield to the same code coverage! A = 1; A = 1; A = 1; global A = 1; local x; x = A + 1; A = 0; A = 0; if (IS_DIR(path)) A = 0; x = A + 1; y = A * 2; x = A + 1; y = A * 2; y = A * 2; x = A + 1; if (! G[ x ] ) x = 2, y = 0 x = 1, y = 0 x = 1, y = 0 G[ x ] = kmalloc(...); ... A = 0; A = 0; A = 0; global A = 0; local y; y = A * 2; A = 1; A = 1; if (size > 4096) x = A + 1; A = 1; y = A * 2; y = A * 2; x = A + 1; x = A + 1; y = A * 2; if (! G[ y ] ) G[ y ] = kmalloc(...); x = 2, y = 0 x = 2, y = 2 x = 2, y = 2 ... Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 36

Incompleteness of CFG edge coverage Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 37

A multi-dimensional view of coverage in fuzzing Edge-coverage only Krace Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 38

Visualizing the concurrency dimension Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 39

Visualizing the concurrency dimension Edge-coverage only Krace Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 40

Bring fuzzing to the concurrency dimension Signaled? Data race checker Data race Syscall Program Crashed? Test case generator executor Memory error Feedback code coverage Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 41

Bring fuzzing to the concurrency dimension Signaled? Data race checker Data race Syscall Program Crashed? Test case generator executor Memory error Feedback code coverage concurrency coverage Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 42

Bring fuzzing to the concurrency dimension Signaled? Data race Interleaving checker Data race generator Syscall Program Crashed? Test case generator executor Memory error Feedback code coverage concurrency coverage Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 43

Concurrency coverage tracking Signaled? Data race Interleaving checker Data race generator Syscall Program Crashed? Test case generator executor Memory error Feedback code coverage concurrency coverage Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 44

A straw-man solution sys_truncate (size, ...): sys_readlink (path, ...): global A = 0; global A = 1; i7 i1 local y; i8 local x; i2 if (size > 4096) { i9 if (IS_DIR(path)) { i3 y = A * 2; x = A + 1; i10 i4 if ( G[ y ] ) if ( G[ x ] ) i11 i5 kmalloc(...); i12 kmalloc(...); i6 } } Thread 1 Thread 2 Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 45

A straw-man solution global A = 1; sys_truncate (size, ...): sys_readlink (path, ...): i1 global A = 0; i7 global A = 0; local y; global A = 1; i7 i8 i1 local x; local y; i8 local x; i2 i2 if (IS_DIR(path)) { i3 if (size > 4096) { if (size > 4096) { i9 if (IS_DIR(path)) { i9 i3 x = A + 1; y = A * 2; x = A + 1; i4 i10 i4 y = A * 2; if ( G[ y ] ) if ( G[ x ] ) i10 i11 i5 if( G[ x ] ) kmalloc(...); i5 i12 kmalloc(...); i6 } if ( G[ y ] ) } i11 kmalloc(...); i12 Thread 1 Thread 2 } kmalloc(...); i6 } A possible interleaving Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 46

A straw-man solution global A = 1; sys_truncate (size, ...): sys_readlink (path, ...): i1 global A = 0; i7 global A = 0; local y; global A = 1; i7 i8 i1 local x; local y; i8 local x; i2 i2 if (IS_DIR(path)) { if (IS_DIR(path)) { i3 if (size > 4096) { if (size > 4096) { if (size > 4096) { i9 if (IS_DIR(path)) { i9 i3 x = A + 1; y = A * 2; x = A + 1; i4 i10 i4 y = A * 2; if ( G[ y ] ) if ( G[ x ] ) i10 i11 i5 if( G[ x ] ) kmalloc(...); i5 i12 kmalloc(...); i6 } if ( G[ y ] ) } i11 kmalloc(...); i12 Thread 1 Thread 2 } kmalloc(...); i6 } Hash(i1, i7, i8, i2, i3 , i9 , i4, i10, i5, i11, i12, i6) = 7825 Hash(i1, i7, i8, i2, i9 , i3 , i4, i10, i5, i11, i12, i6) = 1356 Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 47

A straw-man solution global A = 1; sys_truncate (size, ...): sys_readlink (path, ...): i1 global A = 0; i7 Number of possible interleavings of two threads global A = 0; local y; global A = 1; i7 i8 i1 local x; local y; i8 local x; i2 i2 If two threads have and instructions respectively, m n if (IS_DIR(path)) { i3 then the number interleavings between them is given by: if (size > 4096) { if (size > 4096) { i9 if (IS_DIR(path)) { i9 i3 x = A + 1; y = A * 2; x = A + 1; i4 ( m + n )! i10 i4 y = A * 2; if ( G[ y ] ) if ( G[ x ] ) i10 i11 i5 m ! × n ! if( G[ x ] ) kmalloc(...); i5 i12 kmalloc(...); i6 } if ( G[ y ] ) } i11 m = n = 2 m = n = 4 m = n = 8 m = n = 16 kmalloc(...); i12 6 70 13 K 601 M Thread 1 Thread 2 } kmalloc(...); i6 } A possible interleaving Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 48

Observations on practical interleaving tracking Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 49

Observations on practical interleaving tracking Thread 1 Thread 2 Only interleaved accesses to shared memory matters - In an extreme case where two threads do not shared memory, they interleaving does not matter at all. Only interleaved read-write accesses to shared memory locations matters - In an extreme case where two threads only read from shared memory, they interleaving does not matter at all. Thread interleaving alters the def-use relation of memory locations! Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 50

Observations on practical interleaving tracking Thread 1 Thread 2 Only interleaved accesses to shared memory matters - In an extreme case where two threads do not shared memory, they interleaving does not matter at all. Only interleaved read-write accesses to y = A * 2 R shared memory locations matters - In an extreme case where two threads only read from x = A + 1 R shared memory, they interleaving does not matter at all. Thread interleaving alters the def-use relation of memory locations! Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 51

Observations on practical interleaving tracking Thread 1 Thread 2 Only interleaved accesses to shared memory matters - In an extreme case where two threads do not shared A = 1 W memory, they interleaving does not matter at all. A = 0 W Only interleaved read-write accesses to shared memory locations matters - In an extreme case where two threads only read from x = A + 1 R shared memory, they interleaving does not matter at all. Thread interleaving alters the def-use relation of memory locations! Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 52

Observations on practical interleaving tracking Thread 1 Thread 2 Only interleaved accesses to shared memory matters - In an extreme case where two threads do not shared A = 1 W memory, they interleaving does not matter at all. Interleaving approximation A = 0 W Only interleaved read-write accesses to Track cross-thread write-to-read (def-to-use) edges! shared memory matters - In an extreme case where two threads only read from x = A + 1 R shared memory, they interleaving does not matter at all. Thread interleaving alters the def-use relation of memory locations! Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 53

Observations on practical interleaving tracking Interleaving approximation Track cross-thread write-to-read (def-to-use) edges! Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 54

Aliased-instruction coverage Thread 1 Thread 2 A = 1 i1 W i2 i3 → A = 0 i2 W x = A + 1 i3 R Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 55

Aliased-instruction coverage Thread 1 Thread 2 A = 1 i1 W i2 i5, i4 i3 → → A = 0 i4 W B = 2 i2 W x = A + 1 y = B * 4 i3 i5 R R Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 56

Aliased-instruction coverage Thread 1 Thread 2 A = 1 i1 W Concurrency coverage bitmap size A = 0 i4 W During our experiment, we observed 63,590 unique cross-thread, write-to-read edges. B = 2 i2 i2 i5, i4 i3 W → → a bitmap size of 128KB will be su ffi cient. → x = A + 1 y = B * 4 i3 i5 R R Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 57

Concurrency coverage tracking Signaled? Data race Interleaving checker Data race generator Syscall Program Crashed? Test case generator executor Memory error Feedback code coverage concurrency coverage Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 58

Interleaving exploration Signaled? Data race Interleaving checker Data race generator Syscall Program Crashed? Test case generator executor Memory error Feedback code coverage concurrency coverage Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 59

Active interleaving exploration - ideal case A = 1; A = 1; A = 1; x = A + 1; A = 0; A = 0; A = 0; x = A + 1; y = A * 2; y = A * 2; y = A * 2; x = A + 1; x = 2, y = 0 x = 1, y = 0 x = 1, y = 0 A = 1; A = 0; i3 i1 i3 i2 i3 i2 <nil> x = A + 1; → → y = A * 2; i2 i4 Thread 1 Thread 2 A = 0; A = 0; A = 0; y = A * 2; A = 1; A = 1; x = A + 1; A = 1; y = A * 2; x = A + 1; x = A + 1; y = A * 2; x = 2, y = 0 x = 2, y = 2 x = 2, y = 2 <nil> i1 i4 i1 i4 → → Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 60

Active interleaving exploration - ideal case A = 1; A = 1; A = 1; x = A + 1; A = 0; A = 0; A = 0; x = A + 1; y = A * 2; y = A * 2; y = A * 2; x = A + 1; Enumerating all interleaving among all kernel threads is impossible x = 2, y = 0 x = 1, y = 0 x = 1, y = 0 A = 1; A = 0; i3 i1 i3 i2 i3 i2 <nil> x = A + 1; → → y = A * 2; i2 i4 During our experiment, we observed at maximum 60 threads running concurrently. Thread 1 Thread 2 A = 0; A = 0; A = 0; ⟶ 10 60 Assume each thread have only 10 shared memory accesses possibilities. y = A * 2; A = 1; A = 1; x = A + 1; A = 1; y = A * 2; x = A + 1; x = A + 1; y = A * 2; x = 2, y = 0 x = 2, y = 2 x = 2, y = 2 <nil> i1 i4 i1 i4 → → Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 61

Active interleaving exploration through delay injection i1 T1 R Concurrency coverage i2 T2 W i3 T3 W i4 T4 R Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 62

Active interleaving exploration through delay injection d(669) i1 T1 R R d(300) Concurrency coverage i2 T2 W W d(273) i3 T3 W W i4 T4 R R d(20) Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 63

Active interleaving exploration through delay injection d(669) i1 T1 R d(300) Concurrency coverage i2 T2 W d(273) i3 T3 W i4 T4 R d(20) Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 64

Interleaving exploration Signaled? Data race Interleaving checker Data race generator Syscall Program Crashed? Test case generator executor Memory error Feedback code coverage concurrency coverage Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 65

Bring them all together Signaled? Data race Interleaving checker Data race generator Syscall Program Crashed? Test case generator executor Memory error Feedback code coverage concurrency coverage Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 66

QEMU-based implementation Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 67

Alias coverage growth will be saturating Btrfs Ext4 But fi le systems that are higher in concurrency level saturates much slower! Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 68

Edge and alias coverage goes generally in synchronization Btrfs Ext4 But there will be time when the edge coverage saturates but alias coverage keeps fi nding new thread interleaving Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 69

Slightly more branch coverage than Syzkaller Btrfs Ext4 This maybe due to the fact that we give each seed more chances (if they make progresses in alias coverage) Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 70

Evaluation Bugs Bugs found by Krace File system # data races # harmful con fi rmed Btrfs 11 8 Ext4 4 1 VFS 8 2 Total 23 11 Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 71

Comparison with related works Structured input Application [Google] Syzkaller [CCS’17] SlowFuzz [SP’19] Janus [ICSE’19] DifFuzz [ICSE’19] SLF [VLDB’20] Apollo …… …… Seed selection Coverage metric [CCS’16] AFLFast [SP’18] Angora [ASE’18] FairFuzz [RAID’19] Benchmark [SP’20] Krace [FSE’19] Fudge …… Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 72

Conclusion and contribution Structured input Application [Google] Syzkaller [CCS’17] SlowFuzz [SP’19] Janus [ICSE’19] DifFuzz [ICSE’19] SLF [VLDB’20] Apollo …… …… Seed selection Coverage metric [CCS’16] AFLFast [SP’18] Angora [ASE’18] FairFuzz [RAID’19] Benchmark [SP’20] Krace [FSE’19] Fudge …… Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 73

Common criticism about the fuzzing approach 1. How strong is the security guarantee? e.g., 1 hour of fuzzing without any bugs, what does that mean? 2. Can you deterministically replay the bugs found? Most likely not, because we do not directly control the scheduler. 3. How easy can these data races be triggered in reality? Hard to argue, because fuzzing is essentially a probabilistic search. 4. What are the consequences out of these data races? e.g., can you really alter the control fl ow or change some sensitive data? Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 74

Agenda 1. What are race conditions? c 2. Finding their presence with fuzzing ? [SP’20] Data races in fi le systems 3. Towards a more systematic methodology? [SP’18] Symbolic race checking c 4. Up to the extreme of completeness and soundness ? [WIP (CAV’21)] C to SMT transpilation Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 75

Agenda 1. What are race conditions? c 2. Finding their presence with fuzzing ? [SP’20] Data races in fi le systems 3. Towards a more systematic methodology? [SP’18] Symbolic race checking 4. Up to the extreme of completeness and soundness ? [WIP (CAV’21)] C to SMT transpilation Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 76

SMT and symbolic execution • SMT stands for satis fi ability modulo theories • Given some basic math theories about Booleans, Integers, etc, decide whether some constraints are satis fi able or not ==> constraint solvers. • Example: what are the values of and that satisfy both constraints: x y • x + y = 10 • x + 2 y = 20 • Manual solving: x = 0, y = 10 Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 77

An SMT script is a problem description • To describe the problem to an automated solver, • we need some standardized format ==> an SMT script • Continued from the example: (declare-const x Int) (declare-const y Int) sat • x + y = 10 (model (assert (= (+ x y) 10)) (define-fun y () Int 10) (assert (= (+ x (* 2 y)) 20)) (define-fun x () Int 0) • x + 2 y = 20 ) (check-sat) (get-model) The problem we have in mind The SMT script we formulated The answer given by Z3 SMT solver Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 78

Analogy with bug fi nding Will variable “ s” over fl ow in the program? (declare-const x Int) (declare-const y Int) int loop(int x) { sat • int s = 1; x + y = 10 (model ??? ??? (assert (= (+ x y) 10)) for (int i=1; i<=x; i++) { (define-fun y () Int 10) (assert (= (+ x (* 2 y)) 20)) s *= i; (define-fun x () Int 0) • x + 2 y = 20 } ) (check-sat) return s; (get-model) } The problem we have in mind The SMT script we formulated The answer given by Z3 SMT solver Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 79

Analogy with bug fi nding Translating bug description — focus of the SP’18 paper Symbolizing program — focus of the ongoing CAV’21 paper Will variable “ s” over fl ow in the program? int loop(int x) { int s = 1; ??? ??? for (int i=1; i<=x; i++) { s *= i; } return s; } The problem we have in mind The SMT script we formulated The answer given by Z3 SMT solver Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 80

What is a double-fetch bug? static int perf_copy_attr( struct perf_event_attr __user *uattr, User program struct perf_event_attr *kattr) { address space u32 size; /* first fetch */ if (copy_from_user(&size, &uattr->size, 4)) return -EFAULT; /* sanity checks */ if (size > PAGE_SIZE || size < PERF_ATTR_SIZE_VER0) Kernel return -EFAULT; address space /* second fetch */ if (copy_from_user(attr, &uattr->size, size)) return -EFAULT; ...... } /* BUG: when attr->size is used later */ copy_from_user(uattr, attr, attr->size)) Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 81

What is a double-fetch bug? static int perf_copy_attr( struct perf_event_attr __user *uattr, User program struct perf_event_attr *kattr) { address space u32 size; /* first fetch */ if (copy_from_user(&size, &uattr->size, 4)) return -EFAULT; /* sanity checks */ if (size > PAGE_SIZE || size < PERF_ATTR_SIZE_VER0) Kernel return -EFAULT; address space /* second fetch */ if (copy_from_user(attr, &uattr->size, size)) return -EFAULT; ...... } /* BUG: when attr->size is used later */ copy_from_user(uattr, attr, attr->size)) Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 82

What is a double-fetch bug? ?? bytes static int perf_copy_attr( struct perf_event_attr __user *uattr, User program struct perf_event_attr *kattr) { address space u32 size; /* first fetch */ if (copy_from_user(&size, &uattr->size, 4)) return -EFAULT; /* sanity checks */ if (size > PAGE_SIZE || size < PERF_ATTR_SIZE_VER0) Kernel return -EFAULT; address space /* second fetch */ if (copy_from_user(attr, &uattr->size, size)) return -EFAULT; ...... } /* BUG: when attr->size is used later */ copy_from_user(uattr, attr, attr->size)) Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 83

What is a double-fetch bug? ?? bytes static int perf_copy_attr( struct perf_event_attr __user *uattr, User program 30 struct perf_event_attr *kattr) { address space 4 bytes u32 size; /* first fetch */ if (copy_from_user(&size, &uattr->size, 4)) return -EFAULT; /* sanity checks */ if (size > PAGE_SIZE || size < PERF_ATTR_SIZE_VER0) Kernel return -EFAULT; address space /* second fetch */ if (copy_from_user(attr, &uattr->size, size)) return -EFAULT; ...... } /* BUG: when attr->size is used later */ copy_from_user(uattr, attr, attr->size)) Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 84

What is a double-fetch bug? ?? bytes static int perf_copy_attr( struct perf_event_attr __user *uattr, User program 30 struct perf_event_attr *kattr) { address space 4 bytes u32 size; /* first fetch */ 30 if (copy_from_user(&size, &uattr->size, 4)) return -EFAULT; /* sanity checks */ if (size > PAGE_SIZE || size < PERF_ATTR_SIZE_VER0) Kernel return -EFAULT; address space /* second fetch */ if (copy_from_user(attr, &uattr->size, size)) return -EFAULT; ...... } /* BUG: when attr->size is used later */ copy_from_user(uattr, attr, attr->size)) Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 85

What is a double-fetch bug? ?? bytes static int perf_copy_attr( struct perf_event_attr __user *uattr, User program 30 struct perf_event_attr *kattr) { address space 4 bytes u32 size; /* first fetch */ 30 if (copy_from_user(&size, &uattr->size, 4)) return -EFAULT; /* sanity checks */ if (size > PAGE_SIZE || size < PERF_ATTR_SIZE_VER0) Kernel return -EFAULT; address space /* second fetch */ if (copy_from_user(attr, &uattr->size, size)) return -EFAULT; ...... } /* BUG: when attr->size is used later */ copy_from_user(uattr, attr, attr->size)) Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 86

What is a double-fetch bug? 30 bytes static int perf_copy_attr( struct perf_event_attr __user *uattr, User program 30 struct perf_event_attr *kattr) { address space 4 bytes u32 size; /* first fetch */ 30 if (copy_from_user(&size, &uattr->size, 4)) return -EFAULT; /* sanity checks */ if (size > PAGE_SIZE || size < PERF_ATTR_SIZE_VER0) Kernel return -EFAULT; address space /* second fetch */ if (copy_from_user(attr, &uattr->size, size)) return -EFAULT; ...... } /* BUG: when attr->size is used later */ copy_from_user(uattr, attr, attr->size)) Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 87

What is a double-fetch bug? 30 bytes static int perf_copy_attr( struct perf_event_attr __user *uattr, User program 30 struct perf_event_attr *kattr) { address space 4 bytes u32 size; /* first fetch */ 30 if (copy_from_user(&size, &uattr->size, 4)) return -EFAULT; /* sanity checks */ if (size > PAGE_SIZE || size < PERF_ATTR_SIZE_VER0) Kernel return -EFAULT; address space /* second fetch */ 30 if (copy_from_user(kattr, uattr, size)) return -EFAULT; ...... } /* BUG: when attr->size is used later */ copy_from_user(uattr, attr, attr->size)) Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 88

What is a double-fetch bug? 30 bytes static int perf_copy_attr( struct perf_event_attr __user *uattr, User program 30 struct perf_event_attr *kattr) { address space 4 bytes u32 size; /* first fetch */ 30 if (copy_from_user(&size, &uattr->size, 4)) return -EFAULT; /* sanity checks */ if (size > PAGE_SIZE || size < PERF_ATTR_SIZE_VER0) Kernel return -EFAULT; address space /* second fetch */ 30 if (copy_from_user(kattr, uattr, size)) return -EFAULT; ...... } Copy 30 bytes back to user copy_to_user(uattr, attr, attr->size)) Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 89

The assumption failure 30 bytes static int perf_copy_attr( struct perf_event_attr __user *uattr, User program 30 struct perf_event_attr *kattr) { address space 4 bytes u32 size; /* first fetch */ 30 if (copy_from_user(&size, &uattr->size, 4)) What can go wrong in this process? return -EFAULT; Data atomicity during syscall execution is not guaranteed! /* sanity checks */ if (size > PAGE_SIZE || size < PERF_ATTR_SIZE_VER0) Kernel return -EFAULT; address space /* second fetch */ 30 if (copy_from_user(kattr, attr, size)) return -EFAULT; ...... } Copy 30 bytes back to user copy_to_user(uattr, attr, attr->size)) Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 90

Up until the fi rst fetch… ?? bytes static int perf_copy_attr( struct perf_event_attr __user *uattr, User program 30 struct perf_event_attr *kattr) { address space 4 bytes u32 size; /* first fetch */ 30 if (copy_from_user(&size, &uattr->size, 4)) return -EFAULT; /* sanity checks */ if (size > PAGE_SIZE || size < PERF_ATTR_SIZE_VER0) Kernel return -EFAULT; address space /* second fetch */ if (copy_from_user(attr, &uattr->size, size)) return -EFAULT; ...... } /* BUG: when attr->size is used later */ copy_from_user(uattr, attr, attr->size)) Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 91

Wrong assumption: atomicity in syscall ?? bytes static int perf_copy_attr( struct perf_event_attr __user *uattr, User program 60000 struct perf_event_attr *kattr) { address space 4 bytes u32 size; /* first fetch */ 30 if (copy_from_user(&size, &uattr->size, 4)) return -EFAULT; /* sanity checks */ if (size > PAGE_SIZE || size < PERF_ATTR_SIZE_VER0) Kernel return -EFAULT; address space /* second fetch */ if (copy_from_user(attr, &uattr->size, size)) return -EFAULT; ...... } /* BUG: when attr->size is used later */ copy_from_user(uattr, attr, attr->size)) Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 92

Wrong assumption: atomicity in syscall 30 bytes static int perf_copy_attr( struct perf_event_attr __user *uattr, User program 60000 struct perf_event_attr *kattr) { address space 4 bytes u32 size; /* first fetch */ 30 if (copy_from_user(&size, &uattr->size, 4)) return -EFAULT; /* sanity checks */ if (size > PAGE_SIZE || size < PERF_ATTR_SIZE_VER0) Kernel return -EFAULT; address space /* second fetch */ 60000 if (copy_from_user(kattr, uattr, size)) return -EFAULT; ...... } /* BUG: when attr->size is used later */ copy_from_user(uattr, attr, attr->size)) Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 93

When the exploit happens 30 bytes static int perf_copy_attr( struct perf_event_attr __user *uattr, User program 60000 struct perf_event_attr *kattr) { address space 4 bytes u32 size; /* first fetch */ 30 if (copy_from_user(&size, &uattr->size, 4)) return -EFAULT; /* sanity checks */ if (size > PAGE_SIZE || size < PERF_ATTR_SIZE_VER0) Kernel return -EFAULT; address space /* second fetch */ 60000 if (copy_from_user(kattr, uattr, size)) return -EFAULT; ...... } /* BUG: when attr->size is used later */ Kernel information leak! copy_to_user(uattr, kattr, kattr->size)) Meng Xu (Georgia Tech) Finding Race Conditions in Kernels July 16, 2020 94

Modeling a double-fetch bug Fetch : a pair , where ( A , S ) - the starting address of the fetch A - the size of memory copied into kernel S Meng Xu (Georgia Tech) Finding Semantic Bugs in Kernels March 18, 2020 95

Modeling a double-fetch bug Fetch : a pair , where ( A , S ) - the starting address of the fetch A - the size of memory copied into kernel S Overlap : two fetches, and , that satisfy ( A 1 , S 1 ) ( A 2 , S 2 ) A 1 ≤ A 2 < ( A 1 + S 1 ) || A 2 ≤ A 1 < ( A 2 + S 2 ) Meng Xu (Georgia Tech) Finding Semantic Bugs in Kernels March 18, 2020 96

Modeling a double-fetch bug Fetch : a pair , where ( A , S ) - the starting address of the fetch A - the size of memory copied into kernel S Overlap : two fetches, and , that satisfy ( A 1 , S 1 ) ( A 2 , S 2 ) A 1 ≤ A 2 < ( A 1 + S 1 ) || A 2 ≤ A 1 < ( A 2 + S 2 ) Dependency : such that controls whether and how the ∃ V ∈ 𝙿𝚠𝚏𝚜𝚖𝚋𝚚 V second fetch might take place. Meng Xu (Georgia Tech) Finding Semantic Bugs in Kernels March 18, 2020 97

Modeling a double-fetch bug Fetch : a pair , where ( A , S ) - the starting address of the fetch A - the size of memory copied into kernel S Overlap : two fetches, and , that satisfy ( A 1 , S 1 ) ( A 2 , S 2 ) A 1 ≤ A 2 < ( A 1 + S 1 ) || A 2 ≤ A 1 < ( A 2 + S 2 ) Dependency : such that controls whether and how the ∃ V ∈ 𝙿𝚠𝚏𝚜𝚖𝚋𝚚 V second fetch might take place. Precaution : check that from the second fetch equals from the fi rst fetch. V ′ V Meng Xu (Georgia Tech) Finding Semantic Bugs in Kernels March 18, 2020 98

Model illustration static int perf_copy_attr( struct perf_event_attr __user *uattr, struct perf_event_attr *kattr) { u32 size; /* first fetch */ if (copy_from_user(&size, &uattr->size, 4)) return -EFAULT; /* sanity checks */ if (size > PAGE_SIZE || size < PERF_ATTR_SIZE_VER0) return -EFAULT; /* second fetch */ if (copy_from_user(kattr, uattr, size)) return -EFAULT; ...... } /* BUG: when attr->size is used later */ copy_to_user(uattr, kattr, kattr->size)) Meng Xu (Georgia Tech) Finding Semantic Bugs in Kernels March 18, 2020 99

Model illustration static int perf_copy_attr( struct perf_event_attr __user *uattr, struct perf_event_attr *kattr) { u32 size; /* first fetch */ size if (copy_from_user(&size, &uattr->size, 4)) return -EFAULT; /* sanity checks */ if (size > PAGE_SIZE || size < PERF_ATTR_SIZE_VER0) return -EFAULT; /* second fetch */ if (copy_from_user(kattr, uattr, size)) return -EFAULT; ...... } /* BUG: when attr->size is used later */ copy_to_user(uattr, kattr, kattr->size)) Meng Xu (Georgia Tech) Finding Semantic Bugs in Kernels March 18, 2020 100