Finding and Understanding Bugs in Software Model Checkers Chengyu - PowerPoint PPT Presentation

Approach II: Enumerative Counting Reachability (ECR) int main(void){ int main(void){ int a[3] = {1}; int a[3] = {1}; int i = 0; int i = 0; int br = 0; int br = 0; while(i < 3){ while(i < 3){ if(a[i] == 1) { br++; if(a[i] == 1) { br++; …… …… } } i++; i++; } } GetValue(br) GetValue(br) return 0; return 0; } } Actual execution: br=1 Actual execution: br=3 � 37

Approach II: Enumerative Counting Reachability (ECR) int main(void){ int main(void){ int a[3] = {1}; int a[3] = {1}; int i = 0; int i = 0; int br = 0; int br = 0; while(i < 3){ while(i < 3){ if(a[i] == 1) { br++; if(a[i] == 1) { br++; …… …… } } i++; i++; } } if(br != 1) if(br != 3) __VERIFIER_error(); __VERIFIER_error(); return 0; return 0; } } Actual execution: br=1 Actual execution: br=3 � 37

Approach II: Enumerative Counting Reachability (ECR) int main(void){ int main(void){ int a[3] = {1}; int a[3] = {1}; int i = 0; int i = 0; int br = 0; int br = 0; while(i < 3){ while(i < 3){ if(a[i] == 1) { br++; if(a[i] == 1) { br++; …… …… } } i++; i++; } } if(br != 1) if(br != 3) __VERIFIER_error(); __VERIFIER_error(); return 0; return 0; } } Test oracle: safe Test oracle: safe � 38

Approach II: Enumerative Counting Reachability (ECR) int main(void){ int main(void){ int a[3] = {1}; int a[3] = {1}; int i = 0; int i = 0; int br = 0; int br = 0; while(i < 3){ while(i < 3){ if(a[i] == 1) { br++; if(a[i] == 1) { br++; …… …… } } i++; i++; } } if(br != 1) if(br != 3) __VERIFIER_error(); __VERIFIER_error(); return 0; return 0; } } Test oracle: safe Test oracle: safe Model checker: unsafe Model checker: safe � 39

Approach • Approach I: Enumerative Reachability (ER) • Approach II: Enumerative Counting Reachability (ECR) � 40

Approach III: Fused Counting Reachability (FCR) int main(void){ int a[3] = {1}; int i = 0; while(i < 3){ if(a[i] == 1) { …… } i++; } return 0; } � 41

Approach III: Fused Counting Reachability (FCR) int main(void){ int a[3] = {1}; int i = 0; int br = 0;int br2 = 0; while(i < 3){ br1++; if(a[i] == 1) { br2++; …… } i++; } GetValue(br1) GetValue(br2) return 0; } Actual execution: br1=1; br2=3 � 41

Approach III: Fused Counting Reachability (FCR) int main(void){ int a[3] = {1}; int i = 0; int br = 0;int br2 = 0; while(i < 3){ br1++; if(a[i] == 1) { br2++; …… } i++; } if(br1 != 3 || br2 != 1) __VERIFIER_error(); return 0; } � 41

Approach III: Fused Counting Reachability (FCR) int main(void){ int a[3] = {1}; int i = 0; int br = 0;int br2 = 0; while(i < 3){ br1++; if(a[i] == 1) { br2++; …… } i++; } if(br1 != 3 || br2 != 1) __VERIFIER_error(); return 0; Test oracle: safe } � 41

Approach III: Fused Counting Reachability (FCR) int main(void){ int a[3] = {1}; int i = 0; int br = 0;int br2 = 0; while(i < 3){ br1++; if(a[i] == 1) { br2++; …… } i++; } if(br1 != 3 || br2 != 1) __VERIFIER_error(); return 0; Test oracle: safe } Model checker: unsafe � 41

Approach • Approach I: Enumerative Reachability (ER) • Approach II: Enumerative Counting Reachability (ECR) • Approach III: Fused Counting Reachability (FCR) � 42

Approach find more kinds of bugs • Approach I: Enumerative Reachability (ER) • Approach II: Enumerative Counting Reachability (ECR) • Approach III: Fused Counting Reachability (FCR) � 42

Approach find more kinds of bugs • Approach I: Enumerative Reachability (ER) • Approach II: Enumerative Counting Reachability (ECR) • Approach III: Fused Counting Reachability (FCR) save more time � 42

Evaluation Setup � 43

Evaluation Setup GCC test suite � 43

Evaluation Setup GCC test suite 4,609 Files 219,636 Loc � 43

Evaluation Setup IC3 based GCC test suite 4,609 Files 219,636 Loc � 43

Evaluation Setup IC3 based GCC test suite CEGAR based 4,609 Files 219,636 Loc � 43

Evaluation Setup IC3 based GCC test suite CEGAR based 4,609 Files 219,636 Loc BMC based � 43

Evaluation Setup IC3 based GCC test suite CEGAR based 4,609 Files 219,636 Loc BMC based � 43

RQ1: Can our approaches find bugs? � 44

RQ1: Can our approaches find bugs? � 45

RQ1: Can our approaches find bugs? � 46

RQ1: Can our approaches find bugs? � 47

RQ1: Can our approaches find bugs? � 48

RQ1: Can our approaches find bugs? � 49

RQ1: Can our approaches find bugs? � 50

RQ1: Can our approaches find bugs? � 51

RQ2: How many bugs can be found by each approach? � 52

RQ2: How many bugs can be found by each approach? Approach I 52 � 53

RQ2: How many bugs can be found by each approach? Approach II Approach I 61 52 � 54

RQ2: How many bugs can be found by each approach? Approach II/III Approach I 61 52 � 55

RQ2: How many bugs can be found by each approach? Approach II/III Approach I 61 52 � 56

RQ2: How many bugs can be found by each approach? Approach II/III Approach I 61 52 � 56

RQ2: How many bugs can be found by each approach? Approach II/III Approach I 10 51 1 � 57

Bug#529 in CPAchecker (False negative in approach II/III) void main() { i:0 int i = 0; *(&i):1 while (1) { if (i > 0) { break; } if (i == 0){ *(&i) = *(&i) + 1; } } } � 58

Bug#529 in CPAchecker (False negative in approach II/III) void main() { i:0 int i = 0; int br = 0; *(&i):1 while (1) { if (i > 0) { br++; break; } if (i == 0){ *(&i) = *(&i) + 1; } } Test oracle: safe if(br != 1) __VERIFIER_error(); } � 58

Bug#529 in CPAchecker (False negative in approach II/III) void main() { i:0 int i = 0; int br = 0; *(&i):1 while (1) { if (i > 0) { br++; break; } if (i == 0){ *(&i) = *(&i) + 1; } } Test oracle: safe if(br != 1) Buggy model checker: safe __VERIFIER_error(); } � 58

RQ3: How much time does each approach consume? � 59

RQ3: How much time does each approach consume? � 60

RQ3: How much time does each approach consume? � 61

RQ3: How much time does each approach consume? Save 89% of time � 62

RQ3: How much time does each approach consume? Approach II/III Approach I 10 51 1 � 63

Assorted Bug Samples • C standard library • Front-end • Language feature • Memory model • Configuration • Pointer alias • Third-party component https://github.com/MCFuzzer/MCFuzz/issues � 64

Example: Front-end related bug in CPAchecker void f(int a, int b){ if (a == b) __VERIFIER_error(); } int main(){ int d = 0; int c = 4; int e = 2; f (d=c&&e, 1); return 0; } � 65

Example: Front-end related bug in CPAchecker void f(int a, int b){ if (a == b) d = c&&e = 1 __VERIFIER_error(); } Test oracle: unsafe int main(){ int d = 0; int c = 4; int e = 2; f (d=c&&e, 1); return 0; } � 66

Example: Front-end related bug in CPAchecker void f(int a, int b){ if (a == b) __VERIFIER_error(); __CPAchecker_TMP_0 = c&&e } d = __CPAchecker_TMP_0 int main(){ int d = 0; Test oracle: unsafe int c = 4; int e = 2; f (d=c&&e, 1); return 0; } � 67

Example: Front-end related bug in CPAchecker void f(int a, int b){ if (a == b) __VERIFIER_error(); d = __CPAchecker_TMP_0 } __CPAchecker_TMP_0 = c&&e int main(){ int d = 0; Test oracle: unsafe int c = 4; int e = 2; f (d=c&&e, 1); return 0; } � 67

Example: Front-end related bug in CPAchecker void f(int a, int b){ if (a == b) __VERIFIER_error(); d = __CPAchecker_TMP_0 } __CPAchecker_TMP_0 = c&&e int main(){ int d = 0; Test oracle: unsafe int c = 4; int e = 2; : safe f (d=c&&e, 1); return 0; } � 67

Example: Language feature related bug in CBMC x struct { int a:4; int :4; 2 u 3 4 int b:4; int c:4; } x = { 2,3,4 }; a _ b c int main (){ if (x.b != 3) “Unnamed members of objects of __VERIFIER_error(); structure type do not return 0; participate in initialization.” } —— C standard Test oracle: safe � 68

Example: Language feature related bug in CBMC x struct { int a:4; int :4; 2 3 4 u int b:4; int c:4; } x = { 2,3,4 }; a _ b c int main (){ if (x.b != 3) “Unnamed members of objects of __VERIFIER_error(); structure type do not return 0; participate in initialization.” } —— C standard Test oracle: safe � 69

Example: Language feature related bug in CBMC x struct { int a:4; int :4; 2 3 4 u int b:4; int c:4; } x = { 2,3,4 }; a _ b c int main (){ if (x.b != 3) “Unnamed members of objects of __VERIFIER_error(); structure type do not return 0; participate in initialization.” } —— C standard Test oracle: safe : unsafe � 69

Example: Configuration related bug in Seahorn 7/6 == 1 void test(int x,int y, -7/-6 == 1 int q){ if ((x / y) != q ) __VERIFIER_error(); Test oracle: safe } int main (){ test (7, 6, 1); test (-7, -6, 1); return 0; } � 70

Example: Configuration related bug in Seahorn 7/6 == 1 void test(int x,int y, -7/-6 == 1 int q){ if ((x / y) != q ) __VERIFIER_error(); Test oracle: safe } int main (){ test (7, 6, 1); sea pf file.c test (-7, -6, 1); return 0; } � 71

Example: Configuration related bug in Seahorn 7/6 == 1 void test(int x,int y, -7/-6 == 1 int q){ if ((x / y) != q ) __VERIFIER_error(); Test oracle: safe } int main (){ test (7, 6, 1); sea pf file.c test (-7, -6, 1); return 0; } : unsafe � 71

Example: Configuration related bug in Seahorn 7/6 == 1 void test(int x,int y, -7/-6 == 1 int q){ if ((x / y) != q ) __VERIFIER_error(); Test oracle: safe } int main (){ sea pf test (7, 6, 1); test (-7, -6, 1); —inline file.c return 0; } � 72

Example: Configuration related bug in Seahorn 7/6 == 1 void test(int x,int y, -7/-6 == 1 int q){ if ((x / y) != q ) __VERIFIER_error(); Test oracle: safe } int main (){ sea pf test (7, 6, 1); test (-7, -6, 1); —inline file.c return 0; } : safe � 72

Evaluation on SV-COMP benchmarks � 73