Efficient Symbolic Execution for Software Testing Johannes Kinder - - PowerPoint PPT Presentation

▶
efficient symbolic execution for software testing
SMART_READER_LITE
LIVE PREVIEW

Efficient Symbolic Execution for Software Testing Johannes Kinder - - PowerPoint PPT Presentation

Dec 01, 2023 333 likes •2.34k views

Efficient Symbolic Execution for Software Testing Johannes Kinder Royal Holloway, University of London Joint work with: Stefan Bucur, George Candea, Volodymyr Kuznetsov @ EPFL Symbolic Execution Automatically explore program paths

slide-1
SLIDE 1

Efficient Symbolic Execution for Software Testing

Johannes Kinder Royal Holloway, University of London Joint work with: Stefan Bucur, George Candea, Volodymyr Kuznetsov @ EPFL

slide-2
SLIDE 2

Symbolic Execution

  • Automatically explore program paths
  • Execute program on “symbolic” input values
  • “Fork” execution at each branch
  • Record branching conditions
  • Constraint solver
  • Decides path feasibility
  • Generates test cases for paths and bugs

2

slide-3
SLIDE 3

Symbolic Execution

  • (Very brief) history
  • Test generation by SE in 70s [King ‘75] [Boyer et al. ’75]
  • SAT / SMT solvers lead to boom in 2000s

[Godefroid et al. ‘05][Cadar et al. ’06]

  • Many successful tools
  • KLEE, SAGE, PEX, SPF, CREST, Cloud9, S2E, …
  • Specific advantages
  • No false positives, useful partial results
  • Reduces need for modeling
slide-4
SLIDE 4

Outline

  • Symbolic Execution for Testing
  • State Merging – Fighting Path Explosion
  • Interpreted High-Level Code
slide-5
SLIDE 5

Outline

  • Symbolic Execution for Testing
  • State Merging – Fighting Path Explosion
  • Interpreted High-Level Code
slide-6
SLIDE 6

6

slide-7
SLIDE 7

7

Symbolic program state

slide-8
SLIDE 8

8

Symbolic program state Input symbol

slide-9
SLIDE 9

9

Symbolic program state Path condition Input symbol

slide-10
SLIDE 10

10

slide-11
SLIDE 11

11

✓ ✓

slide-12
SLIDE 12

12

slide-13
SLIDE 13

13

slide-14
SLIDE 14

14

slide-15
SLIDE 15

15

slide-16
SLIDE 16

16

slide-17
SLIDE 17

17

slide-18
SLIDE 18

18

slide-19
SLIDE 19

19

slide-20
SLIDE 20

20

Satisfying assignments: Test cases:

slide-21
SLIDE 21

Finding Bugs

  • Symbolic execution enumerates paths
  • Runs into bugs that trigger whenever path executes
  • Assertions, buffer overflows, division by zero, etc.,

require specific conditions

  • Error conditions
  • Treat assertions as conditions
  • Creates explicit error paths

21

slide-22
SLIDE 22

Finding Bugs

  • Instrument program with properties
  • Translate any safety property to reachability
  • Division by zero

22

slide-23
SLIDE 23

Finding Bugs

  • Instrument program with properties
  • Translate any safety property to reachability
  • Division by zero
  • Buffer overflows

23

slide-24
SLIDE 24

Finding Bugs

  • Instrument program with properties
  • Translate any safety property to reachability
  • Division by zero
  • Buffer overflows
  • Implementation is usually implicit

24

slide-25
SLIDE 25

Symbolic Execution Algorithms

  • Static symbolic execution
  • Simulate execution on program source code
  • Computes strongest postconditions from entry point
  • Dynamic symbolic execution (DSE)
  • Run / interpret the program with concrete state
  • Symbolic state computed in parallel (“concolic”)
  • Solver generates new concrete state
  • DSE-Flavors
  • EXE-style [Cadar et al. ‘06] vs. DART [Godefroid et al. ‘05]

25

slide-26
SLIDE 26

EXE

26

slide-27
SLIDE 27

EXE

27

Symbolic program state

slide-28
SLIDE 28

EXE

28

Symbolic program state Concrete state

slide-29
SLIDE 29

EXE

29

slide-30
SLIDE 30

EXE

30

slide-31
SLIDE 31

EXE

31

✓

slide-32
SLIDE 32

EXE

32

slide-33
SLIDE 33

EXE

33

slide-34
SLIDE 34

EXE

34

slide-35
SLIDE 35

EXE

35

slide-36
SLIDE 36

EXE

36

slide-37
SLIDE 37

EXE

37

slide-38
SLIDE 38

EXE

38

slide-39
SLIDE 39

EXE

39

Satisfying assignments: Test cases:

slide-40
SLIDE 40

DART

40

Path condition: Test cases:

slide-41
SLIDE 41

DART

41

Path condition: Test cases:

slide-42
SLIDE 42

DART

42

Path condition: Test cases:

slide-43
SLIDE 43

DART

43

Path condition: Test cases:

slide-44
SLIDE 44

DART

44

Path condition: Test cases:

slide-45
SLIDE 45

DART

45

✓

Path condition: Test cases:

slide-46
SLIDE 46

DART

46

✓

Path condition: Test cases:

slide-47
SLIDE 47

DART

47

Path condition: Test cases:

slide-48
SLIDE 48

DART

48

Path condition: Test cases:

slide-49
SLIDE 49

DART

49

Path condition: Test cases:

✓

slide-50
SLIDE 50

DART

50

Path condition: Test cases:

✓

slide-51
SLIDE 51

DART

51

Path condition: Test cases:

slide-52
SLIDE 52

DART

52

Path condition: Test cases:

slide-53
SLIDE 53

DART

53

Path condition: Test cases:

slide-54
SLIDE 54

DART

54

Path condition: Test cases:

slide-55
SLIDE 55

DART

55

Path condition: Test cases:

slide-56
SLIDE 56

DART vs. EXE

  • Complete execution

from 1st step

  • Deep exploration
  • One query per run
  • Offline SE possible
  • Follow recorded trace

56

  • Fine-grained control of

execution

  • Shallow exploration
  • Many queries early on
  • Online SE
  • SE and interpretation

in lockstep

slide-57
SLIDE 57

Concretization

  • Parts of input space can be kept concrete
  • Reduces complexity
  • Focuses search
  • Expressions can be concretized at runtime
  • Avoid expressions outside of SMT solver theories

(non-linear etc.)

  • Sound but incomplete

57

slide-58
SLIDE 58

Concretization (Example)

58

slide-59
SLIDE 59

Concretization (Example)

59

slide-60
SLIDE 60

Concretization (Example)

60

slide-61
SLIDE 61

Concretization (Example)

61

slide-62
SLIDE 62

Concretization (Example)

62

slide-63
SLIDE 63

Concretization (Example)

63

slide-64
SLIDE 64

Concretization (Example)

64

slide-65
SLIDE 65

Concretization (Example)

65

Solution diverges from expected path! (e.g., X = 2)

slide-66
SLIDE 66

Concretization (Example)

66

Concretization constraint

slide-67
SLIDE 67

Soundness & Completeness

  • Conceptually, each path is exact
  • Strongest postcondition in predicate transformer

semantics

  • No over-approximation, no under-approximation
  • Globally, SE under-approximates
  • Explores only subset of paths in finite time
  • “Eventual” completeness

67

slide-68
SLIDE 68

Soundness & Completeness

  • Conceptually, each path is exact
  • Strongest postcondition in predicate transformer

semantics

  • No over-approximation, no under-approximation
  • Globally, SE under-approximates
  • Explores only subset of paths in finite time
  • “Eventual” completeness

68

Explored symbolically State Space

slide-69
SLIDE 69

Soundness & Completeness

  • Symbolic Execution = Underapproximates
  • Soundness = does not include infeasible behavior
  • Completeness = explores all behavior
  • Concretization restricts state covered by path
  • Remains sound
  • Loses (eventual) completeness

69

Explored symbolically State Space

slide-70
SLIDE 70

Soundness & Completeness

  • Symbolic Execution = Underapproximates
  • Soundness = does not include infeasible behavior
  • Completeness = explores all behavior
  • Concretization restricts state covered by path
  • Remains sound
  • Loses (eventual) completeness

70

Explored symbolically State Space

slide-71
SLIDE 71

Concretization

  • Key strength of dynamic symbolic execution
  • Enables external calls
  • Concretize call arguments
  • Callee executes concretely
  • Concretization constraints can be omitted
  • Sacrifices soundness (original DART)
  • Deal with divergences by random restarts

71

slide-72
SLIDE 72

Outline

  • Symbolic Execution for Testing
  • State Merging – Fighting Path Explosion
  • Interpreted High-Level Code
slide-73
SLIDE 73

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

“echo” Coreutil

slide-74
SLIDE 74

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

“echo” Coreutil

slide-75
SLIDE 75

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

“echo” Coreutil

slide-76
SLIDE 76

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

“echo” Coreutil

slide-77
SLIDE 77

Symbolic Execution

77

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

slide-78
SLIDE 78

Symbolic Execution

78

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

slide-79
SLIDE 79

Symbolic Execution

79

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

slide-80
SLIDE 80

Symbolic Execution

80

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

slide-81
SLIDE 81

Symbolic Execution

81

✓ ✓

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

slide-82
SLIDE 82

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

Problem: Path Explosion

82

slide-83
SLIDE 83

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

Problem: Path Explosion

83

slide-84
SLIDE 84

if (argv[i][0] == 'n') { r = 0; ++i; }

Solution (?): State Merging

else then

84

slide-85
SLIDE 85

if (argv[i][0] == 'n') { r = 0; ++i; }

Solution (?): State Merging

  • Use disjunctions to represent

state at join points

  • ite(x, y, z) : if x then y else z

else then

85

slide-86
SLIDE 86

if (argv[i][0] == 'n') { r = 0; ++i; }

Solution (?): State Merging

  • Use disjunctions to represent

state at join points

  • ite(x, y, z) : if x then y else z

else then

86

slide-87
SLIDE 87

if (argv[i][0] == 'n') { r = 0; ++i; }

Solution (?): State Merging

  • Use disjunctions to represent

state at join points

  • ite(x, y, z) : if x then y else z
  • SE tree becomes a DAG
  • Whole program can be turned into
  • ne verification condition (BMC)

else then

87

slide-88
SLIDE 88

Symbolic Execution vs. BMC

  • Complexity does not disappear
  • Work moved from the SE engine to the solver
  • SE: set of conjunctive queries, BMC: 1 query with

nested disjunctions

  • Complete merging sacrifices advantages of SE
  • No dynamic mode
  • No continuous progress
  • No quick reaching of coverage goals
  • Try to get the best of both worlds
slide-89
SLIDE 89

1 formula / path 1 formula / CFG DART (SAGE)

[Godefroid, PLDI’05]

EXE (KLEE)

[Cadar et al., CCS’06]

Symbolic Execution Verification Condition Generation

F-Soft

[Ivancic et al., CAV’05]

CBMC

[Clarke et al., TACAS’04]

slide-90
SLIDE 90

1 formula / path 1 formula / CFG DART (SAGE)

[Godefroid, PLDI’05]

EXE (KLEE)

[Cadar et al., CCS’06]

Symbolic Execution Verification Condition Generation

F-Soft

[Ivancic et al., CAV’05]

CBMC

[Clarke et al., TACAS’04]

Boogie

[Barnett et al., FMCO’05]

slide-91
SLIDE 91

1 formula / path 1 formula / CFG Compositional SE / Summaries

[Godefroid, POPL’07]

DART (SAGE)

[Godefroid, PLDI’05]

EXE (KLEE)

[Cadar et al., CCS’06]

Symbolic Execution Verification Condition Generation

F-Soft

[Ivancic et al., CAV’05]

CBMC

[Clarke et al., TACAS’04]

Boogie

[Barnett et al., FMCO’05]

slide-92
SLIDE 92

1 formula / path 1 formula / CFG Compositional SE / Summaries

[Godefroid, POPL’07]

BMC slicing

[Ganai&Gupta, DAC’08]

DART (SAGE)

[Godefroid, PLDI’05]

EXE (KLEE)

[Cadar et al., CCS’06]

Symbolic Execution Verification Condition Generation

F-Soft

[Ivancic et al., CAV’05]

CBMC

[Clarke et al., TACAS’04]

Boogie

[Barnett et al., FMCO’05]

slide-93
SLIDE 93

1 formula / path 1 formula / CFG Compositional SE / Summaries

[Godefroid, POPL’07]

BMC slicing

[Ganai&Gupta, DAC’08]

State joining

[Hansen et al., RV’09]

DART (SAGE)

[Godefroid, PLDI’05]

EXE (KLEE)

[Cadar et al., CCS’06]

Symbolic Execution Verification Condition Generation

F-Soft

[Ivancic et al., CAV’05]

CBMC

[Clarke et al., TACAS’04]

Boogie

[Barnett et al., FMCO’05]

slide-94
SLIDE 94

1 formula / path 1 formula / CFG Compositional SE / Summaries

[Godefroid, POPL’07]

BMC slicing

[Ganai&Gupta, DAC’08]

State joining

[Hansen et al., RV’09]

DART (SAGE)

[Godefroid, PLDI’05]

EXE (KLEE)

[Cadar et al., CCS’06]

Symbolic Execution Verification Condition Generation

F-Soft

[Ivancic et al., CAV’05]

CBMC

[Clarke et al., TACAS’04]

Boogie

[Barnett et al., FMCO’05]

Dynamic State Merging

slide-95
SLIDE 95

1 formula / path 1 formula / CFG Compositional SE / Summaries

[Godefroid, POPL’07]

BMC slicing

[Ganai&Gupta, DAC’08]

State joining

[Hansen et al., RV’09]

DART (SAGE)

[Godefroid, PLDI’05]

EXE (KLEE)

[Cadar et al., CCS’06]

Symbolic Execution Verification Condition Generation

F-Soft

[Ivancic et al., CAV’05]

CBMC

[Clarke et al., TACAS’04]

Boogie

[Barnett et al., FMCO’05]

Query Count Estimation Dynamic State Merging [KKBC PLDI’12]

slide-96
SLIDE 96

Merging Increases Solving Cost

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

96

slide-97
SLIDE 97

Merging Increases Solving Cost

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

97

slide-98
SLIDE 98

Merging Increases Solving Cost

Condition becomes symbolic, extra check required.

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

98

slide-99
SLIDE 99

Merging Increases Solving Cost

✓

Condition becomes symbolic, extra check required.

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

99

slide-100
SLIDE 100

Merging Increases Solving Cost

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

100

slide-101
SLIDE 101

Merging Increases Solving Cost

Query becomes more complex.

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

101

slide-102
SLIDE 102

Merging Increases Solving Cost

Query becomes more complex.

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

Should not merge after checking 1st argument!

102

slide-103
SLIDE 103

Query Count Estimation (QCE)

Intuition

  • Estimate the extra burden on the solver
  • Merge only when merging amortizes extra cost

Cost ≈ number of solver queries

103

slide-104
SLIDE 104

Applying QCE

  • 1. Estimate query counts from each program location
  • Total number of queries
  • n all paths
  • For each variable, number of dependent queries
slide-105
SLIDE 105

Applying QCE

  • 1. Estimate query counts from each program location
  • Total number of queries
  • n all paths
  • For each variable, number of dependent queries

Static

slide-106
SLIDE 106

Applying QCE

  • 1. Estimate query counts from each program location
  • Total number of queries
  • n all paths
  • For each variable, number of dependent queries
  • 2. Determine “hot” variables

Static

slide-107
SLIDE 107

Applying QCE

  • 1. Estimate query counts from each program location
  • Total number of queries
  • n all paths
  • For each variable, number of dependent queries
  • 2. Determine “hot” variables
  • 3. Symbolic Execution
  • Do not merge two candidate states if they differ in hot

variables

  • Avoids creating ite expressions

Static Dynamic

slide-108
SLIDE 108

Merging not Beneficial

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

i is “hot”, leads to many extra queries

108

slide-109
SLIDE 109

Merging not Beneficial

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

i is “hot”, leads to many extra queries

109

slide-110
SLIDE 110

Merging not Beneficial

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

i is “hot”, leads to many extra queries

110

slide-111
SLIDE 111

111

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

Merging Beneficial

slide-112
SLIDE 112

112

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

Merging Beneficial

slide-113
SLIDE 113

113

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

Merging Beneficial

slide-114
SLIDE 114

114

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

Merging Beneficial

slide-115
SLIDE 115

115

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

Merging Beneficial

slide-116
SLIDE 116

116

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

Merging Beneficial

slide-117
SLIDE 117

117

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

Merging Beneficial

slide-118
SLIDE 118

118

void main(int argc, char **argv) { int r = 1, i = 1; if (i < argc) { if (argv[i][0] == 'n') { r = 0; ++i; } } for (; i < argc; ++i) { for (int j = 0; argv[i][j] != 0; ++j) { putchar(argv[i][j]); } } if (r) { putchar('\n'); } }

Merging Beneficial

Largc states reduced to 1

(L: maximum argument length)

slide-119
SLIDE 119

Search Strategies

… … …

  • SE usually incomplete
  • 100% path coverage is

impractical

  • Uses search strategy to

reach a coverage goal

  • Search strategy chooses

states from a worklist

119

slide-120
SLIDE 120

Search Strategies

… … …

  • SE usually incomplete
  • 100% path coverage is

impractical

  • Uses search strategy to

reach a coverage goal

  • Search strategy chooses

states from a worklist

120

slide-121
SLIDE 121

Merging Breaks Search Strategies

… … …

  • Naïve state merging

blocks states at join points

  • Allows earlier states to

“catch up”

  • Thwarts strategy in

reaching its goal!

121

slide-122
SLIDE 122

Merging Breaks Search Strategies

… … …

  • Naïve state merging

blocks states at join points

  • Allows earlier states to

“catch up”

  • Thwarts strategy in

reaching its goal!

122

slide-123
SLIDE 123

Merging Breaks Search Strategies

… … …

  • Naïve state merging

blocks states at join points

  • Allows earlier states to

“catch up”

  • Thwarts strategy in

reaching its goal!

123

slide-124
SLIDE 124

Merging Breaks Search Strategies

… … …

  • Naïve state merging

blocks states at join points

  • Allows earlier states to

“catch up”

  • Thwarts strategy in

reaching its goal!

Ask QCE

124

slide-125
SLIDE 125

Merging Breaks Search Strategies

… … …

  • Naïve state merging

blocks states at join points

  • Allows earlier states to

“catch up”

  • Thwarts strategy in

reaching its goal!

125

slide-126
SLIDE 126

Dynamic State Merging

… … …

  • Maintain bounded history
  • f predecessor states

126

slide-127
SLIDE 127

Dynamic State Merging

… … …

  • Maintain bounded history
  • f predecessor states

127

slide-128
SLIDE 128

Dynamic State Merging

… … …

  • Maintain bounded history
  • f predecessor states

128

slide-129
SLIDE 129

Dynamic State Merging

… … …

  • Maintain bounded history
  • f predecessor states
  • QCE compares a state

to predecessors of

  • thers

Ask QCE

129

slide-130
SLIDE 130

Dynamic State Merging

… … …

  • Maintain bounded history
  • f predecessor states
  • QCE compares a state

to predecessors of

  • thers
  • Matching states are

prioritized

  • Original search strategy

controls remaining worklist

130

slide-131
SLIDE 131

Dynamic State Merging

… … …

  • Maintain bounded history
  • f predecessor states
  • QCE compares a state

to predecessors of

  • thers
  • Matching states are

prioritized

  • Original search strategy

controls remaining worklist

Ask QCE

131

slide-132
SLIDE 132

Dynamic State Merging

… … …

  • Maintain bounded history
  • f predecessor states
  • QCE compares a state

to predecessors of

  • thers
  • Matching states are

prioritized

  • Original search strategy

controls remaining worklist

132

slide-133
SLIDE 133

Evaluation

  • Our prototype
  • Based on state-of-the-art engine KLEE
  • QCE implemented in LLVM – static analysis

completes in seconds

  • Analysis Targets
  • 96 GNU Coreutils (echo, ls, dd, who, …)
  • 2’000 – 10’000 executable lines of code per tool
  • 72 KLOC in total

133

slide-134
SLIDE 134

Time for Exhaustive Exploration

Triangle: KLEE time-out (> 2h) QCE + SSM vs. KLEE. Static state merging (SSM) follows topological order.

Consistent speedups

10 100 1000 7200 (Timeout) 10 100 1000 7200 (Timeout) SSM Completion Time (TSSM) in seconds KLEE Completion Time (T Klee) in seconds No Speedup (TKlee = TSSM)

slide-135
SLIDE 135

Speedup vs. Input-Size (Exhaustive)

QCE + SSM vs. KLEE.

0.1 1 10 100 1000 10000 5 10 15 20 25 30 35 40 45 Speedup (TKlee / TSSM) Input Size (# of symbolic bytes) echo link nice basename

Exponential speedups in symbolic input size

135

slide-136
SLIDE 136

# Paths in Incomplete Exploration

0.0001 0.01 1 100 10000 1e+06 1e+08 1e+10 1e+12

dd pwd unlink uptime id whoami users sync readlink comm sleep hostid tsort logname link tty rmdir mkfifo mktemp mknod rm pinky su uname dircolors printf base64 ln mv tac runcon date mkdir who setuidgid seq chcon cp stty tee uniq dir shuf fold cat printenv split nice pr shred du tail ptx paste chown tr echo join cut basename dirname nohup chroot expand chgrp cksum sum false true yes head csplit

  • d

fmt chmod factor stat sort unexpand touch wc env nl sha1sum md5sum

Path Ratio (PDSM / PKlee) Tool name QCE + DSM vs. KLEE (1h time bound).

Up to 11 orders of magnitude more paths explored

136

Target Coreutil

slide-137
SLIDE 137

Combating Path Explosion

  • Dynamic merging (DSM + QCE)
  • Static merging of small structures [Avgerinos et al.,

ICSE’14]

  • Procedure summaries [Godefroid, POPL’07]
  • Parallelization [Bucur et al, EuroSys’11]

137

slide-138
SLIDE 138

Outline

  • Symbolic Execution for Testing
  • State Merging – Fighting Path Explosion
  • Interpreted High-Level Code
slide-139
SLIDE 139

Programming Languages Symbolic Execution Engines

slide-140
SLIDE 140

Programming Languages Symbolic Execution Engines

Scala Java C C++

slide-141
SLIDE 141

Programming Languages Symbolic Execution Engines

Java Bytecode LLVM x86 ARM Scala Java C C++

slide-142
SLIDE 142

Programming Languages Symbolic Execution Engines

Java Bytecode LLVM x86 ARM Scala Java C C++ BitBlaze KLEE JPF SAGE S2E

slide-143
SLIDE 143

Programming Languages Symbolic Execution Engines

Java Bytecode LLVM x86 ARM Scala Java C C++ Python Ruby Lua JavaScript Bash Perl BitBlaze KLEE JPF SAGE S2E

slide-144
SLIDE 144

Programming Languages Symbolic Execution Engines

Java Bytecode LLVM x86 ARM Scala Java C C++ Python Ruby Lua JavaScript Bash Perl BitBlaze KLEE JPF SAGE S2E

?

slide-145
SLIDE 145

def parse_file(file_name): with open(file_name, "r") as f: data = f.read() return json.loads(data, encoding="utf-8")

Interpreted Languages

slide-146
SLIDE 146

def parse_file(file_name): with open(file_name, "r") as f: data = f.read() return json.loads(data, encoding="utf-8")

Interpreted Languages

Complex semantics

+

Ambiguity in specifications

+

Evolving language

+

Large standard library

+

Widespread native methods

Since Python 2.5 Complete File Read Incomplete Specification

slide-147
SLIDE 147

def parse_file(file_name): with open(file_name, "r") as f: data = f.read() return json.loads(data, encoding="utf-8")

Interpreted Languages

Complex semantics

+

Ambiguity in specifications

+

Evolving language

+

Large standard library

+

Widespread native methods

Since Python 2.5 Complete File Read Incomplete Specification

Too much work

slide-148
SLIDE 148

“Consequently, if you were coming from Mars and tried to re-implement Python from this document alone, you might have to guess things and in fact you would probably end up implementing quite a different language.”

  • The Python Language Reference
slide-149
SLIDE 149

How can we efficiently obtain a correct symbolic execution engine?

slide-150
SLIDE 150

Key idea: Use the language interpreter as executable specification

[Bucur, K, Candea, ASPLOS’14]

slide-151
SLIDE 151

Symbolic Execution Engine for Language X

Chef

Key idea: Use the language interpreter as executable specification

Language X Interpreter

[Bucur, K, Candea, ASPLOS’14]

slide-152
SLIDE 152

Symbolic Execution Engine for Language X

Chef

Program + Symbolic Tests Test Cases

Key idea: Use the language interpreter as executable specification

Language X Interpreter

[Bucur, K, Candea, ASPLOS’14]

slide-153
SLIDE 153

Chef Overview

  • Built on top of the S2E symbolic execution

engine for x86

  • Relies on lightweight interpreter

instrumentation + optimizations

  • Prototyped engines for Python and Lua in 5 +

3 person-days

slide-154
SLIDE 154

Testing Interpreted Programs

Naive approach: Run interpreter in stock SE engine

slide-155
SLIDE 155

Testing Interpreted Programs

def validateEmail(email): pos = email.find("@") if pos < 1: raise InvalidEmailError() if email.rfind(".") < pos: raise InvalidEmailError()

Naive approach: Run interpreter in stock SE engine

slide-156
SLIDE 156

Testing Interpreted Programs

def validateEmail(email): pos = email.find("@") if pos < 1: raise InvalidEmailError() if email.rfind(".") < pos: raise InvalidEmailError()

Naive approach: Run interpreter in stock SE engine

Python Interpreter

./python program.py

slide-157
SLIDE 157

Testing Interpreted Programs

def validateEmail(email): pos = email.find("@") if pos < 1: raise InvalidEmailError() if email.rfind(".") < pos: raise InvalidEmailError()

x86 Symbolic Execution Engine

(S2E)

Naive approach: Run interpreter in stock SE engine

Python Interpreter

./python program.py

slide-158
SLIDE 158

Py_LOCAL_INLINE(Py_ssize_t) fastsearch(const STRINGLIB_CHAR* s, Py_ssize_t n, const STRINGLIB_CHAR* p, Py_ssize_t m, Py_ssize_t maxcount, int mode) { unsigned long mask; Py_ssize_t skip, count = 0; Py_ssize_t i, j, mlast, w; w = n - m; if (w < 0 || (mode == FAST_COUNT && maxcount == 0)) return -1; /* look for special cases */ if (m <= 1) { if (m <= 0) return -1; /* use special case for 1-character strings */ if (mode == FAST_COUNT) {

pos = email.find("@")

Naive approach: Run interpreter in stock symbolic execution engine

slide-159
SLIDE 159

pos = email.find("@")

Naive approach: Run interpreter in stock symbolic execution engine

slide-160
SLIDE 160

pos = email.find("@")

Naive approach: Run interpreter in stock symbolic execution engine Path Explosion

slide-161
SLIDE 161

pos = email.find("@")

Gets lost in the details of the implementation Naive approach: Run interpreter in stock symbolic execution engine Path Explosion

slide-162
SLIDE 162

High-level Execution Paths

def validateEmail(email): pos = email.find("@") if pos < 1: raise InvalidEmailError() if email.rfind(".") < pos: raise InvalidEmailError()

High-level execution tree

slide-163
SLIDE 163

High-level Execution Paths

def validateEmail(email): pos = email.find("@") if pos < 1: raise InvalidEmailError() if email.rfind(".") < pos: raise InvalidEmailError()

High-level execution tree Low-level (x86) execution tree

slide-164
SLIDE 164

High-level Execution Paths

def validateEmail(email): pos = email.find("@") if pos < 1: raise InvalidEmailError() if email.rfind(".") < pos: raise InvalidEmailError()

High-level execution tree Low-level (x86) execution tree

slide-165
SLIDE 165

High-level Execution Paths

def validateEmail(email): pos = email.find("@") if pos < 1: raise InvalidEmailError() if email.rfind(".") < pos: raise InvalidEmailError()

High-level execution tree Low-level (x86) execution tree

slide-166
SLIDE 166

High-level Execution Paths

def validateEmail(email): pos = email.find("@") if pos < 1: raise InvalidEmailError() if email.rfind(".") < pos: raise InvalidEmailError()

HL/LL path ratio is low due to path explosion

3 HL paths 10 LL paths

High-level execution tree Low-level (x86) execution tree

slide-167
SLIDE 167

Goal: Prioritize the low-level paths that maximize the HL/LL path ratio.

slide-168
SLIDE 168

High-level Execution Paths

Alternative approach: Select states at high-level branches

slide-169
SLIDE 169

High-level Execution Paths

Fork (low-level) Divergence (high-level)

Alternative approach: Select states at high-level branches

slide-170
SLIDE 170

High-level Execution Paths

Fork (low-level) Divergence (high-level)

High-level fork points are unpredictable Alternative approach: Select states at high-level branches

slide-171
SLIDE 171

Reducing Path Explosion

Program

slide-172
SLIDE 172

Reducing Path Explosion

Fork points clustered in hot spots Program

Low High Selection probability

slide-173
SLIDE 173

Reducing Path Explosion

Fork points clustered in hot spots Clusters grow bigger ⇒ Slower overall progress Program

“Fork bomb”

(e.g., input dependent loop)

Low High Selection probability

Global DFS / BFS / randomized strategy

slide-174
SLIDE 174

Reducing Path Explosion

Fork points clustered in hot spots Clusters grow bigger ⇒ Slower overall progress Program

“Fork bomb”

(e.g., input dependent loop)

Low High Selection probability

Reduced state diversity Global DFS / BFS / randomized strategy

slide-175
SLIDE 175

Class-Uniform Path Analysis

Program Idea: Partition the state space into groups

slide-176
SLIDE 176

Class-Uniform Path Analysis

Program Idea: Partition the state space into groups

slide-177
SLIDE 177

Class-Uniform Path Analysis

Program Idea: Partition the state space into groups

Select group Select state from group

slide-178
SLIDE 178

Class-Uniform Path Analysis

Program Idea: Partition the state space into groups

Select group Select state from group

Faster progress across all groups

slide-179
SLIDE 179

Class-Uniform Path Analysis

Program Idea: Partition the state space into groups

Select group Select state from group

Faster progress across all groups Increased state diversity

slide-180
SLIDE 180

Class-Uniform Path Analysis

slide-181
SLIDE 181

Class-Uniform Path Analysis

slide-182
SLIDE 182

Class-Uniform Path Analysis

States arranged in a class hierarchy Class 1 Class 2

slide-183
SLIDE 183

Class-Uniform Path Analysis

States arranged in a class hierarchy Class 1 Class 2

slide-184
SLIDE 184

Partitioning High-level Paths

slide-185
SLIDE 185

Partitioning High-level Paths

High-level Instruction

(Bytecode Instruction)

slide-186
SLIDE 186

Partitioning High-level Paths

High-level Instruction

(Bytecode Instruction)

slide-187
SLIDE 187

Partitioning High-level Paths

High-level Instruction

(Bytecode Instruction) High-level Program Counter

slide-188
SLIDE 188

Partitioning High-level Paths

High-level Instruction

(Bytecode Instruction)

Low-level x86 PC

High-level Program Counter

slide-189
SLIDE 189

Partitioning High-level Paths

High-level Instruction

(Bytecode Instruction)

Low-level x86 PC

High-level Program Counter

1st CUPA Class

slide-190
SLIDE 190

Partitioning High-level Paths

High-level Instruction

(Bytecode Instruction)

Low-level x86 PC

High-level Program Counter

1st CUPA Class 2nd CUPA Class Reconstruct high-level execution tree

slide-191
SLIDE 191

CUPA Classes

  • 1. High-level PC
  • Uniform HL instruction exploration
  • Obtained via instrumentation
  • 2. x86 PC
  • Uniform native method exploration
  • Approximated as the PC of fork point
slide-192
SLIDE 192

switch (opcode) { case LOAD: ... case STORE: ... case CALL_FUNCTION: ... ... } hlpc++; }

Interpreter Loop Instrumentation

while (true) { fetch_instr(hlpc, &opcode, &params);

slide-193
SLIDE 193

switch (opcode) { case LOAD: ... case STORE: ... case CALL_FUNCTION: ... ... } hlpc++; }

Interpreter Loop Instrumentation

while (true) { fetch_instr(hlpc, &opcode, &params);

Reconstruct high-level execution tree and CFG

chef_log_hlpc(hlpc, opcode);

slide-194
SLIDE 194

Interpreter Optimizations

  • Simple changes to

interpreter source

  • “Anti-optimizations” in

linear performance...

  • ... but exponential gains

in symbolic mode

static long string_hash(PyStringObject *a) { #ifdef SYMBEX_HASHES return 0; #else register Py_ssize_t len; register unsigned char *p; register long x; len = Py_SIZE(a); p = (char *) a->ob_sval; x = _Py_HashSecret.prefix; x ^= *p << 7; while (--len >= 0) x = (1000003*x) ^ *p++; x ^= Py_SIZE(a); x ^= _Py_HashSecret.suffix; if (x == -1) x = -2; return x; #endif }

Hash neutralization

slide-195
SLIDE 195

Program + Symbolic Tests Symbolic Execution Engine for Language X

Chef Summary

Language X Interpreter

(+instrumentation) CHEF API HL Tree Reconstr. CUPA State Selection S2E x86 Symbolic Execution

Chef

slide-196
SLIDE 196

Chef-Prototyped Engines

Python 5 person-days 321 LoC Lua 3 person-days 277 LoC

slide-197
SLIDE 197

Testing Python Packages

xlrd simplejson argparse HTMLParser ConfigParser unicodecsv 6 Popular Packages 10.9K lines of Python code 30 min. / package > 7,000 tests generated 4 undocumented exceptions found

slide-198
SLIDE 198

Efficiency

Package 0.1 1 10 100 1000 10000 Path Ratio (P / PBaseline) CUPA + Optimizations Baseline

slide-199
SLIDE 199

Efficiency

Package 0.1 1 10 100 1000 10000 Path Ratio (P / PBaseline) CUPA + Optimizations Optimizations Only CUPA Only Baseline

slide-200
SLIDE 200

Symbolic Execution

  • Path-wise under-approximate program analysis
  • Mixes concrete and symbolic reasoning
  • Automatic test case-generation
  • Major-challenge: path explosion
  • Solutions:
  • State merging
  • Domain-specific optimizations
  • …