SLIDE 1 Testing, Abstraction, Theorem Proving: Better Together
Authors: Greta Yorsh Thomas Ball Mooly Sagiv Presenter: Michael Rudolph
Seminar Program Analysis and Software Testing
University of Freiburg 2016
SLIDE 2 Motivation
Test set Program Execute tests No Proof Real error no error error
Dynamic Analysis
SLIDE 3 Motivation
Program Abstraction Analysis Proof Potential error no error error
Static Analysis
SLIDE 4 False error
Unreachable concrete states Reachable concrete states false error
Abstract state = +
SLIDE 5
Motivation
Dynamic Analysis No Proof Real error Static Analysis Proof Potential error
SLIDE 6
Motivation
Dynamic Analysis No Proof Real error Static Analysis Proof Potential error solved by
SLIDE 7
Motivation
Dynamic Analysis No Proof Real error Static Analysis Proof Potential error solved by solved by
SLIDE 8
Motivation
⇓ Idea: Combine both approaches.
SLIDE 9 Table of Contents
1 Motivation 2 Overview method 3 Real error 4 Proof 5 False error 6 Conclusion
SLIDE 10 Example
Algorithm: foo(int x, int y)
I int ∗ px = NULL; A x = x + 1; B if x < 4 then C
px = &x;
D if px == &y then E
x = x + 1;
F if x < 5 then G
∗px = ∗px + 1;
(foo algorithm, G. Yorsh , T. Ball, and M. Sagiv , 2006.)
SLIDE 11 Example
Algorithm: foo(3, 0)
I int ∗ px = NULL;
// (pc=I, x=3, y=0, px=NULL)
SLIDE 12 Example
Algorithm: foo(3, 0)
I int ∗ px = NULL;
// (pc=I, x=3, y=0, px=NULL)
A x = x + 1;
// (pc=A, x=3, y=0, px=NULL)
SLIDE 13 Example
Algorithm: foo(3, 0)
I int ∗ px = NULL;
// (pc=I, x=3, y=0, px=NULL)
A x = x + 1;
// (pc=A, x=3, y=0, px=NULL)
B if x < 4
// (pc=B, x=4, y=0, px=NULL) then
C
px = &x;
SLIDE 14 Example
Algorithm: foo(3, 0)
I int ∗ px = NULL;
// (pc=I, x=3, y=0, px=NULL)
A x = x + 1;
// (pc=A, x=3, y=0, px=NULL)
B if x < 4
// (pc=B, x=4, y=0, px=NULL) then
C
px = &x;
D if px == &y
// (pc=D, x=4, y=0, px=NULL) then
E
x = x + 1; // Dead code
SLIDE 15 Example
Algorithm: foo(3, 0)
I int ∗ px = NULL;
// (pc=I, x=3, y=0, px=NULL)
A x = x + 1;
// (pc=A, x=3, y=0, px=NULL)
B if x < 4
// (pc=B, x=4, y=0, px=NULL) then
C
px = &x;
D if px == &y
// (pc=D, x=4, y=0, px=NULL) then
E
x = x + 1; // Dead code
F if x < 5
// (pc=F, x=4, y=0, px=NULL) then
G
∗px = ∗px + 1; // (pc=G, x=4, y=0, px=NULL)
SLIDE 16 Example
Algorithm: foo(3, 0)
I int ∗ px = NULL;
// (pc=I, x=3, y=0, px=NULL)
A x = x + 1;
// (pc=A, x=3, y=0, px=NULL)
B if x < 4
// (pc=B, x=4, y=0, px=NULL) then
C
px = &x;
D if px == &y
// (pc=D, x=4, y=0, px=NULL) then
E
x = x + 1; // Dead code
F if x < 5
// (pc=F, x=4, y=0, px=NULL) then
G
∗px = ∗px + 1; // (pc=G, x=4, y=0, px=NULL) = ⇒ Null pointer dereference error
SLIDE 17 Execute
program P test set T Execute potential error
(Figure 1, G. Yorsh , T. Ball, and M. Sagiv , 2006.)
SLIDE 18 Execute
program P test set T Execute potential error
(Figure 1, G. Yorsh , T. Ball, and M. Sagiv , 2006.)
SLIDE 19 Execute
Program P foo Test set foo(2,0) = (pc=I, x=2, y=0, px=NULL)
An example input.
SLIDE 20 Execute
Algorithm: foo(2, 0)
I int ∗ px = NULL;
// (pc=I, x=2, y=0, px=NULL)
SLIDE 21 Execute
Algorithm: foo(2, 0)
I int ∗ px = NULL;
// (pc=I, x=2, y=0, px=NULL)
A x = x + 1;
// (pc=A, x=2, y=0, px=NULL)
SLIDE 22 Execute
Algorithm: foo(2, 0)
I int ∗ px = NULL;
// (pc=I, x=2, y=0, px=NULL)
A x = x + 1;
// (pc=A, x=2, y=0, px=NULL)
B if x < 4
// (pc=B, x=3, y=0, px=NULL) then
C
px = &x; // (pc=C, x=3, y=0, px=NULL)
SLIDE 23 Execute
Algorithm: foo(2, 0)
I int ∗ px = NULL;
// (pc=I, x=2, y=0, px=NULL)
A x = x + 1;
// (pc=A, x=2, y=0, px=NULL)
B if x < 4
// (pc=B, x=3, y=0, px=NULL) then
C
px = &x; // (pc=C, x=3, y=0, px=NULL)
D if px == &y
// (pc=D, x=3, y=0, px=¬NULL) then
E
x = x + 1;
SLIDE 24 Execute
Algorithm: foo(2, 0)
I int ∗ px = NULL;
// (pc=I, x=2, y=0, px=NULL)
A x = x + 1;
// (pc=A, x=2, y=0, px=NULL)
B if x < 4
// (pc=B, x=3, y=0, px=NULL) then
C
px = &x; // (pc=C, x=3, y=0, px=NULL)
D if px == &y
// (pc=D, x=3, y=0, px=¬NULL) then
E
x = x + 1;
F if x < 5
// (pc=F, x=3, y=0, px=¬NULL) then
G
∗px = ∗px + 1; // (pc=G, x=3, y=0, px=¬NULL)
SLIDE 25
Execute
Execute test set ⇒ No Error. but Execute foo(3,0) ⇒ Error ⇓ Execute phase gives no proof.
SLIDE 26
Abstract
program P test set T Execute potential error Abstract CT
SLIDE 27 Abstract
Unreachable concrete states Reachable concrete states
Abstract state = +
SLIDE 28 Abstract α(C) = {(pc, x < 5, px = NULL)|(pc, x, y, px) ∈ C}
Algorithm: foo(2,0)
I int ∗ px = NULL;
CT AT pc=I, x=2, y=0, px=NULL I, t, t
SLIDE 29 Abstract α(C) = {(pc, x < 5, px = NULL)|(pc, x, y, px) ∈ C}
Algorithm: foo(2,0)
I int ∗ px = NULL; A x = x + 1;
CT AT pc=I, x=2, y=0, px=NULL I, t, t pc=A, x=2, y=0, px=NULL A, t, t
SLIDE 30 Abstract α(C) = {(pc, x < 5, px = NULL)|(pc, x, y, px) ∈ C}
Algorithm: foo(2,0)
I int ∗ px = NULL; A x = x + 1; B if x < 4 then C
px = &x; CT AT pc=I, x=2, y=0, px=NULL I, t, t pc=A, x=2, y=0, px=NULL A, t, t pc=B, x=3, y=0, px=NULL B, t, t
SLIDE 31 Abstract α(C) = {(pc, x < 5, px = NULL)|(pc, x, y, px) ∈ C}
Algorithm: foo(2,0)
I int ∗ px = NULL; A x = x + 1; B if x < 4 then C
px = &x; CT AT pc=I, x=2, y=0, px=NULL I, t, t pc=A, x=2, y=0, px=NULL A, t, t pc=B, x=3, y=0, px=NULL B, t, t pc=C, x=3, y=0, px=NULL C, t, t
SLIDE 32 Abstract α(C) = {(pc, x < 5, px = NULL)|(pc, x, y, px) ∈ C}
Algorithm: foo(2,0)
I int ∗ px = NULL; A x = x + 1; B if x < 4 then C
px = &x;
D if px == &y then E
x = x + 1; CT AT pc=I, x=2, y=0, px=NULL I, t, t pc=A, x=2, y=0, px=NULL A, t, t pc=B, x=3, y=0, px=NULL B, t, t pc=C, x=3, y=0, px=NULL C, t, t pc=D, x=3, y=0, px=¬NULL D, t, f
SLIDE 33 Abstract α(C) = {(pc, x < 5, px = NULL)|(pc, x, y, px) ∈ C}
Algorithm: foo(2,0)
I int ∗ px = NULL; A x = x + 1; B if x < 4 then C
px = &x;
D if px == &y then E
x = x + 1;
F if x < 5 then G
∗px = ∗px +1; CT AT pc=I, x=2, y=0, px=NULL I, t, t pc=A, x=2, y=0, px=NULL A, t, t pc=B, x=3, y=0, px=NULL B, t, t pc=C, x=3, y=0, px=NULL C, t, t pc=D, x=3, y=0, px=¬NULL D, t, f pc=F, x=3, y=0, px=¬NULL F, t, f
SLIDE 34 Abstract α(C) = {(pc, x < 5, px = NULL)|(pc, x, y, px) ∈ C}
Algorithm: foo(2,0)
I int ∗ px = NULL; A x = x + 1; B if x < 4 then C
px = &x;
D if px == &y then E
x = x + 1;
F if x < 5 then G
∗px = ∗px +1; CT AT pc=I, x=2, y=0, px=NULL I, t, t pc=A, x=2, y=0, px=NULL A, t, t pc=B, x=3, y=0, px=NULL B, t, t pc=C, x=3, y=0, px=NULL C, t, t pc=D, x=3, y=0, px=¬NULL D, t, f pc=F, x=3, y=0, px=¬NULL F, t, f pc=G, x=3, y=0, px=¬NULL G, t, f
SLIDE 35
Check adequacy
program P test set T Execute potential error Abstract Check adequacy CT AT
SLIDE 36 Check adequacy
What are successor states ? Let’s look at our example. Algorithm: part of foo(int x, int y)
A x = x + 1; B if x < 4 then C
px = &x; // If B is true.
D if px == &y
// If B is false. then
E
x = x + 1;
F if x < 5 then G
∗px = ∗px + 1; Successor states of B
SLIDE 37 Check adequacy
A set of tests T is adequate under a given abstraction ⇔ for all concrete states which are represented by AT it holds that their successor states are covered by AT, too.
- AT: abstract states covered by T
concrete state c successor states of c
SLIDE 38
Check adequacy
A set of tests T is adequate under a given abstraction ⇔ for all concrete states which are represented by AT it holds that their successor states are covered by AT, too. AT: abstract states covered by T concrete state c successor states of c
SLIDE 39 Check adequacy α(C) = {(pc, x < 5, px = NULL)|(pc, x, y, px) ∈ C}
Algorithm: foo(int x, int y)
I int ∗ px = NULL; A x = x + 1; B if x < 4 then C
px = &x; // (pc=C, x<4, px=Null)
D if px == &y
then
E
x = x + 1;
F if x < 5 then G
∗px = ∗px + 1; AT I, t, t A, t, t B, t, t C, t, t D, t, f F, t, f G, t, f
SLIDE 40 Check adequacy α(C) = {(pc, x < 5, px = NULL)|(pc, x, y, px) ∈ C}
Algorithm: foo(int x, int y)
I int ∗ px = NULL; A x = x + 1; B if x < 4 then C
px = &x;
D if px == &y
// (pc=D, x>=4, px=NULL) then
E
x = x + 1;
F if x < 5 then G
∗px = ∗px + 1; AT I, t, t A, t, t B, t, t C, t, t D, t, f F, t, f G, t, f
SLIDE 41
Check adequacy
The successor state (D,4,0,NULL) is not covered by AT ⇓ adequacy check fails.
SLIDE 42 Fabricate test
program P test set T Execute potential error Abstract Check adequacy Fabricate test CT AT failed T
′
SLIDE 43
Fabricate test
The successor state (D,4,0,NULL) is not covered by AT ⇓ adequacy check fails. ⇓ model generator fabricates a pair of concrete states, e.g. (B,4,0,NULL), (D,4,0,NULL) concrete states need not to be reachable. ⇓ extend test set T by the generated concrete states.
SLIDE 44 Check safety property
program P test set T Execute potential error Abstract Check adequacy Fabricate test Check safety properties CT AT failed adequate T
′
verified potential error
SLIDE 45 Check safety property
What is a safety property ? Let’s look at our example. Algorithm: part of foo(int x, int y)
A x = x + 1; B if x < 4 then C
px = &x;
D if px == &y then E
x = x + 1;
F if x < 5 then G
∗px = ∗px + 1; // error if px = NULL ⇒ safety property: abstract error state (G,t,t) / ∈ AT
SLIDE 46 Table of Contents
1 Motivation 2 Overview method 3 Real error 4 Proof 5 False error 6 Conclusion
SLIDE 47 Input
Program P foo Test set T {(pc = I, x = 2, y = 0, px = NULL)
, (pc = I, x = 6, y = 0, px = NULL)
, (pc = I, x = 11, y = 0, px = NULL)
}
SLIDE 48 Run until check adequacy
program P test set T Execute potential error Abstract Check adequacy Fabricate test Check safety properties CT AT failed adequate T
′
verified potential error
SLIDE 49 Run until check adequacy
program P test set T Execute potential error Abstract Check adequacy Fabricate test Check safety properties CT AT failed adequate T
′
verified potential error
SLIDE 50 Run until check adequacy
program P test set T Execute potential error Abstract Check adequacy Fabricate test Check safety properties CT AT failed adequate T
′
verified potential error
SLIDE 51 Algorithm: foo(int x, int y)
I int ∗ px = NULL; A x = x + 1; B if x < 4 then C
px = &x;
D if px == &y then E
x = x + 1;
F if x < 5 then G
∗px = ∗px + 1;
(Figure 2(a), G. Yorsh , T. Ball, and M. Sagiv , 2006.)
SLIDE 52 Fabricate test
program P test set T Execute potential error Abstract Check adequacy Fabricate test Check safety properties CT AT adequate failed T
′
verified potential error
SLIDE 53 Fabricate test
Algorithm: fixedFoo(int x, int y)
I int ∗ px = NULL; A x = x + 1; B if x < 5 then C
px = &x;
D if px == &y then E
x = x + 1;
F if x < 5 then G
∗px = ∗px + 1; Fabricate tests (pc=B, x=4, y=0, px=NULL) (pc=D, x=4, y=0, px=NULL) Remark: Both tests are reachable.
SLIDE 54 Execute
program P test set T Execute potential error Abstract Check adequacy Fabricate test Check safety properties CT AT failed adequate T
′
verified potential error
SLIDE 55 Execute (pc=B, x=4, y=0, px=NULL)
Algorithm: foo(int x, int y)
I int ∗ px = NULL; A x = x + 1; B if x < 4
// (pc=B, x=4, y=0, px=NULL) then
C
px = &x;
D if px == &y
// (pc=D, x=4, y=0, px=NULL) then
E
x = x + 1;
F if x < 5
// (pc=F, x=4, y=0, px=NULL) then
G
∗px = ∗px + 1; // (pc=G, x=4, y=0, px=NULL)
SLIDE 56 Execute (pc=B, x=4, y=0, px=NULL)
Algorithm: foo(int x, int y)
I int ∗ px = NULL; A x = x + 1; B if x < 4
// (pc=B, x=4, y=0, px=NULL) then
C
px = &x;
D if px == &y
// (pc=D, x=4, y=0, px=NULL) then
E
x = x + 1;
F if x < 5
// (pc=F, x=4, y=0, px=NULL) then
G
∗px = ∗px + 1; // (pc=G, x=4, y=0, px=NULL) = ⇒ Null pointer dereference error
SLIDE 57 Report potential error
program P test set T Execute potential error Abstract Check adequacy Fabricate test Check safety properties CT AT failed adequate T
′
verified potential error
SLIDE 58 Table of Contents
1 Motivation 2 Overview method 3 Real error 4 Proof 5 False error 6 Conclusion
SLIDE 59 Modify example foo
Algorithm: fixedFoo(int x, int y)
I int ∗ px = NULL; A x = x + 1; B if x < 5
// instead of x < 4 then
C
px = &x;
D if px == &y then E
x = x + 1;
F if x < 5 then G
∗px = ∗px + 1;
(fixed foo algorithm, G. Yorsh , T. Ball, and M. Sagiv , 2006.)
SLIDE 60 Input
Program P fixedFoo Test set T {(pc = I, x = 2, y = 0, px = NULL)
, (pc = I, x = 6, y = 0, px = NULL)
, (pc = I, x = 11, y = 0, px = NULL)
}
SLIDE 61 Run until check adequacy
program P test set T Execute potential error Abstract Check adequacy Fabricate test Check safety properties CT AT failed adequate T
′
verified potential error
SLIDE 62 Run until check adequacy
program P test set T Execute potential error Abstract Check adequacy Fabricate test Check safety properties CT AT failed adequate T
′
verified potential error
SLIDE 63 Run until check adequacy
program P test set T Execute potential error Abstract Check adequacy Fabricate test Check safety properties CT AT failed adequate T
′
verified potential error
SLIDE 64 Algorithm: fixedFoo(int x, int y)
I int ∗ px = NULL; A x = x + 1; B if x < 5 then C
px = &x;
D if px == &y then E
x = x + 1;
F if x < 5 then G
∗px = ∗px + 1;
(Figure 2(b), G. Yorsh , T. Ball, and M. Sagiv , 2006.)
SLIDE 65 Fabricate test
program P test set T Execute potential error Abstract Check adequacy Fabricate test Check safety properties CT AT adequate failed T
′
verified potential error
SLIDE 66 Fabricate test
Algorithm: fixedFoo(int x, int y)
I int ∗ px = NULL; A x = x + 1; B if x < 5 then C
px = &x;
D if px == &y then E
x = x + 1;
F if x < 5 then G
∗px = ∗px + 1; Fabricate tests (pc=D, x=4, y=0, px=&y) (pc=E, x=4, y=0, px=&y) Remark: Both tests are not reachable, since px never assigned to &y.
SLIDE 67 Run until check adequacy
program P test set T Execute potential error Abstract Check adequacy Fabricate test Check safety properties CT AT failed adequate T
′
verified potential error
SLIDE 68 Run until check adequacy
program P test set T Execute potential error Abstract Check adequacy Fabricate test Check safety properties CT AT failed adequate T
′
verified potential error
SLIDE 69 Run until check adequacy
program P test set T Execute potential error Abstract Check adequacy Fabricate test Check safety properties CT AT failed adequate T
′
verified potential error
SLIDE 70 (Figure 2(b), G. Yorsh , T. Ball, and M. Sagiv , 2006.)
SLIDE 71 Check safety properties
program P test set T Execute potential error Abstract Check adequacy Fabricate test Check safety properties CT AT failed adequate T
′
verified potential error
SLIDE 72 Error state (G,t,t) /
∈
(Figure 2(b), G. Yorsh , T. Ball, and M. Sagiv , 2006.)
SLIDE 73 Proof
program P test set T Execute potential error Abstract Check adequacy Fabricate test Check safety properties CT AT failed adequate T
′
verified potential error
SLIDE 74 Table of Contents
1 Motivation 2 Overview method 3 Real error 4 Proof 5 False error 6 Conclusion
SLIDE 75 Input
Program P fixedFoo Test set T {(pc = I, x = 2, y = 0, px = NULL)
, (pc = I, x = 6, y = 0, px = NULL)
, (pc = I, x = 11, y = 0, px = NULL)
} abstraction function: α = {(pc, x < 10, px = NULL)|(pc, x, y, px) ∈ C}
SLIDE 76 Run until check adequacy
program P test set T Execute potential error Abstract Check adequacy Fabricate test Check safety properties CT AT failed adequate T
′
verified potential error
SLIDE 77 Run until check adequacy
program P test set T Execute potential error Abstract Check adequacy Fabricate test Check safety properties CT AT failed adequate T
′
verified potential error
SLIDE 78 Run until check adequacy
program P test set T Execute potential error Abstract Check adequacy Fabricate test Check safety properties CT AT failed adequate T
′
verified potential error
SLIDE 79 Algorithm: fixedFoo(int x, int y)
I int ∗ px = NULL; A x = x + 1; B if x < 5 then C
px = &x;
D if px == &y then E
x = x + 1;
F if x < 5 then G
∗px = ∗px + 1;
(Figure 2(c), G. Yorsh , T. Ball, and M. Sagiv , 2006.)
SLIDE 80 Fabricate test
program P test set T Execute potential error Abstract Check adequacy Fabricate test Check safety properties CT AT failed adequate T
′
verified potential error
SLIDE 81 Fabricate test
Algorithm: fixedFoo(int x, int y)
I int ∗ px = NULL; A x = x + 1; B if x < 5 then C
px = &x;
D if px == &y then E
x = x + 1;
F if x < 5 then G
∗px = ∗px + 1; Fabricate tests (pc=F, x=4, y=0, px=NULL) (pc=G, x=4, y=0, px=NULL) Remark: Both tests are not reachable, since px = &x ⇔ x < 5.
SLIDE 82 Execute
program P test set T Execute potential error Abstract Check adequacy Fabricate test Check safety properties CT AT failed adequate T
′
verified potential error
SLIDE 83 Execute (pc=F, x=4, y=0, px=NULL)
Algorithm: fixedFoo(int x, int y)
I int ∗ px = NULL; A x = x + 1; B if x < 5 then C
px = &x;
D if px == &y then E
x = x + 1;
F if x < 5
// (pc=F, x=4, y=0, px=NULL) then
G
∗px = ∗px + 1; // (pc=G, x=4, y=0, px=NULL)
SLIDE 84 Execute (pc=F, x=4, y=0, px=NULL)
Algorithm: fixedFoo(int x, int y)
I int ∗ px = NULL; A x = x + 1; B if x < 5 then C
px = &x;
D if px == &y then E
x = x + 1;
F if x < 5
// (pc=F, x=4, y=0, px=NULL) then
G
∗px = ∗px + 1; // (pc=G, x=4, y=0, px=NULL) = ⇒ Null pointer dereference error
SLIDE 85 False error
program P test set T Execute potential error Abstract Check adequacy Fabricate test Check safety properties CT AT failed adequate T
′
verified potential error
SLIDE 86 Table of Contents
1 Motivation 2 Overview method 3 Real error 4 Proof 5 False error 6 Conclusion
SLIDE 87
Conclusion
Given an abstraction function and safety property the procedure can proof the absence of errors in programs. Approach cannot distinguish between false and real errors like a dynamic approach. Idea of presented approach: Combining the pros of dynamic and static program analysis.
SLIDE 88
Conclusion
Given an abstraction function and safety property the procedure can proof the absence of errors in programs. Approach cannot distinguish between false and real errors like a dynamic approach. Idea of presented approach: Combining the pros of dynamic and static program analysis.
Is this true ?
SLIDE 89
Conclusion question
Dynamic Analysis No Proof Real error Static Analysis Proof Potential error solved by solved by?
