Badger : Complexity Analysis with Fuzzing and Symbolic Execution - - PowerPoint PPT Presentation

▶
badger complexity analysis with fuzzing and symbolic
SMART_READER_LITE
LIVE PREVIEW

Badger : Complexity Analysis with Fuzzing and Symbolic Execution - - PowerPoint PPT Presentation

Nov 02, 2023 220 likes •407 views

Badger : Complexity Analysis with Fuzzing and Symbolic Execution Yannic Noller Rody Kersten Corina S. Pasareanu yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018 ! 1 Problem Solution Example

slide-1
SLIDE 1

Badger: Complexity Analysis with Fuzzing and Symbolic Execution

Yannic Noller Rody Kersten Corina S. Pasareanu

!1 yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018

slide-2
SLIDE 2

Complexity Analysis

Problem Related Solution Example Summary

yannic.noller@hu-berlin.de !2 International Symposium on Software Testing and Analysis (ISSTA) 2018

discover vulnerabilities related to worst-case time/ space complexity, e.g., Denial-of-Service

0 public void sort (int[] a) { 1 int N = a.length; 2 for (int i = 1; i < N; i++) { 3 int j = i - 1; 4 int x = a[i]; 5 while ((j >= 0) && (a[j] > x)) { 6 a[j + 1] = a[j]; 7 j--; 8 } 9 a[j + 1] = x; 10 } 11 }

Insertion Sort find worst-case input:
 automated + fast + concrete

  • worst-case complexity:

O(n2)

  • e.g. a=[8, 7, 6] (n=3)

Evaluation

slide-3
SLIDE 3

Our Contributions

!3 yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018

  • combine fuzzing and symbolic execution to

find algorithmic complexity vulnerabilities

Problem Related Solution Example Summary Evaluation

  • Badger, a framework for analysis of Java

applications

  • analysis parameterized by a cost metric
  • handling of user-defined cost
slide-4
SLIDE 4

Badger

!4 yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018

fuzzer symbolic execution

exchange interesting
 inputs KelinciWCA (based on AFL) based on Symbolic PathFinder (SPF) fuzzer and symbolic execution run in parallel

Problem Related Solution Example Summary Evaluation

increased coverage or increased cost

slide-5
SLIDE 5
  • based on AFL, extends Kelinci
  • mutation-based greybox fuzzing
  • cost-guided fuzzer: coverage + cost

KelinciWCA

!5 yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018

[Kersten2017]

  • maintain current highscore
  • cost metrics: timing / memory / user-defined

Problem Related Solution Example Summary Evaluation

slide-6
SLIDE 6

SymExe with SPF

!6 yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018

import inputs

fuzzer

export inputs

interesting input

SymExe

Trie Extension /
 Input Assessment worst-case analysis concolic execution

includes

Exploration Input
 Generation

most promising node

trie-guided symbolic execution bounded symbolic execution model generation input generation

new input 1 2 3 4 5 path condition

Problem Related Solution Example Summary Evaluation

slide-7
SLIDE 7

!7 yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018

0 public void sort (int[] a) { 1 int N = a.length; 2 for (int i = 1; i < N; i++) { 3 int j = i - 1; 4 int x = a[i]; 5 while ((j >= 0) && (a[j] > x)) { 6 a[j + 1] = a[j]; 7 j--; 8 } 9 a[j + 1] = x; 10 } 11 }

Insertion Sort

initial input
 a=[37, 42, 48]

id=2 line=5 choice=0
 score=7.0 id=0
 ROOT score=7.0 id=1
 line=5 choice=0
 score=7.0

Trie extension with initial input. The most promising node get selected.

Example

Problem Related Solution Example Summary Evaluation

slide-8
SLIDE 8

!8 yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018

0 public void sort (int[] a) { 1 int N = a.length; 2 for (int i = 1; i < N; i++) { 3 int j = i - 1; 4 int x = a[i]; 5 while ((j >= 0) && (a[j] > x)) { 6 a[j + 1] = a[j]; 7 j--; 8 } 9 a[j + 1] = x; 10 } 11 }

Insertion Sort

id=0
 ROOT score=7.0

Exploration and input generation.

id=2 line=5 choice=0
 score=7.0 id=3 line=5 choice=1
 score=? id=1
 line=5 choice=0
 score=7.0

pc = sym_0 ≤ sym_1 ∧ sym_1 > sym_2

new input
 a=[0, 1, 0]

Example

Problem Related Solution Example Summary Evaluation

slide-9
SLIDE 9

Example

!9 yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018

0 public void sort (int[] a) { 1 int N = a.length; 2 for (int i = 1; i < N; i++) { 3 int j = i - 1; 4 int x = a[i]; 5 while ((j >= 0) && (a[j] > x)) { 6 a[j + 1] = a[j]; 7 j--; 8 } 9 a[j + 1] = x; 10 } 11 }

Insertion Sort

Assessment of new input and extension of the trie. New most promising node gets selected.

id=2 line=5 choice=0
 score=7.0

new input
 a=[0, 1, 0]

id=4 line=5 choice=0
 score=10 id=1
 line=5 choice=0
 score=8.5 id=0
 ROOT score=8.5 id=3 line=5 choice=1
 score=10

Problem Related Solution Example Summary Evaluation

slide-10
SLIDE 10

Research Questions

yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018

RQ1: Since Badger combines fuzzing and symbolic execution, is it better than each part on their own in terms of: (a) Quality of worst-case, and (b) Speed? RQ2: Is KelinciWCA better than Kelinci in terms of: (a) Quality of worst-case, and (b) Speed? RQ3: Can Badger reveal worst-case vulnerabilities?

Problem Related Solution Example Summary Evaluation

!10

slide-11
SLIDE 11

Experiments

!11 yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018

ID Subject 1 Insertion Sort 2 Quicksort 3a Regular Expression (fixed input) 3b Regular Expression (fixed regex) 4 Hash Table 5 Compression 6 Image Processor 7 Smart Contract

we report the average values
 (our full data set is available

  • nline)

each experiment for 5 hours and 5 times

Problem Related Solution Example Summary Evaluation

slide-12
SLIDE 12

!12 yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018

Insertion Sort (N=64)

costs (# jumps) 2500 5000 7500 10000 time (minutes) 1 21 41 61 81 101 121 141 161 181 201 221 241 261 281 300

Kelinci KelinciWCA SymExe Badger

9533 18.73x 6701 3025 Badger after 20min: 9305 KelinciWCA 9305 after 2.85 hours initial input score: 509 9850 19.35x

Problem Related Solution Example Summary Evaluation

slide-13
SLIDE 13

!13 yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018

Quicksort (N=64)

costs (# jumps) 950 1900 2850 3800 time (minutes) 1 21 41 61 81 101 121 141 161 181 201 221 241 261 281 300

Kelinci KelinciWCA SymExe Badger

3683 1.30x 3719 1.31x 3161 2970 initial input score: 2829 no significant difference between Badger and KelinciWCA

Problem Related Solution Example Summary Evaluation

slide-14
SLIDE 14

!14 yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018

Image Processor (2x2 JPEG)

costs (# jumps) 100000 200000 300000 400000 time (minutes) 1 21 41 61 81 101 121 141 161 181 201 221 241 261 281 300

Kelinci KelinciWCA SymExe Badger

193,730 22.24x 349,438 40.11x 291,384 188,719 initial input score: 8712

Problem Related Solution Example Summary Evaluation

slide-15
SLIDE 15

Existing Solutions

!15 yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018

  • Fuzzing


e.g. SlowFuzz

  • Symbolic Execution


e.g. WISE , SPF-WCA

  • Fuzzing + Symbolic Execution


e.g. Driller

[Petsios2017] [Stephens2016] [Luckow2017] [Burnim2009]

Problem Related Solution Example Summary Evaluation

slide-16
SLIDE 16

!16 yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018

git clone https://github.com/isstac/badger.git

Badger: Complexity Analysis with Fuzzing and Symbolic Execution

Problem Related Solution Example Summary Evaluation

slide-17
SLIDE 17

References

!17 yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018

[AFL] Website. american fuzzy lop (AFL). http://lcamtuf.coredump.cx/afl/. 
 
 [Burnim2009] J. Burnim, S. Juvekar, and K. Sen. 2009. WISE: Automated test generation for worst-case complexity. In 2009 IEEE 31st International Conference on Software Engineering. 463–473. 
 
 [Clarke1976] L. A. Clarke, "A System to Generate Test Data and Symbolically Execute Programs," in IEEE Transactions on Software Engineering,

  • vol. SE-2, no. 3, pp. 215-222, Sept. 1976.



 [Godefroid2005] Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: directed automated random testing. In Proceedings a the 2005 ACM SIGPLAN conference on Programming language design and implementation (PLDI '05). ACM, New York, NY, USA, 213-223.
 
 [Kersten2017] Rody Kersten, Kasper Luckow, and Corina S. Păsăreanu. 2017. POSTER: AFL-based Fuzzing for Java with Kelinci. In Proceedings

  • f the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS '17).



 [King1976] James C. King. 1976. Symbolic execution and program testing. Commun. ACM 19, 7 (July 1976), 385-394.
 
 [Luckow2017] Kasper Luckow, Rody Kersten, and Corina Pasareanu. 2017. Symbolic Complexity Analysis using Context-preserving Histories. In Proceedings of the 10th IEEE International Conference on Software Testing, Verification and Validation (ICST 2017). 58–68.
 
 [Miller1990] Barton P . Miller, Louis Fredriksen, and Bryan So. 1990. An empirical study of the reliability of UNIX utilities. Commun. ACM 33, 12 (December 1990), 32-44.
 
 [Petsiois2017] Theofilos Petsios, Jason Zhao, Angelos D. Keromytis, and Suman Jana. 2017. SlowFuzz: Automated Domain-Independent Detection of Algorithmic Complexity Vulnerabilities. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS '17). ACM, New York, NY, USA, 2155-2168.
 
 [Sen2005] Koushik Sen, Darko Marinov, and Gul Agha. 2005. CUTE: a concolic unit testing engine for C. In Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering (ESEC/ FSE-13). ACM, New York, NY, USA, 263-272.
 
 [Stephens2016] Stephens N, Grosen J, Salls C, Dutcher A, Wang R, Corbetta J, Shoshitaishvili Y, Kruegel C, Vigna G. Driller: Augmenting Fuzzing Through Selective Symbolic Execution. InNDSS 2016 Feb (Vol. 16, pp. 1-16).