An Integrated Approach to Assertion-Based Random Testing an 1,4 e F. - - PowerPoint PPT Presentation
An Integrated Approach to Assertion-Based Random Testing an 1,4 e F. Morales 1 Ignacio De Casso San Rom Jos a 1,2 Manuel V. Hermenegildo 1,4 Pedro L opez Garc 1 IMDEA Software Institute, 2 Spanish Research Council (CSIC), and 4
Ignacio De Casso San Rom´ an1,4 Jos´ e F. Morales1 Pedro L´
ıa1,2 Manuel V. Hermenegildo1,4
1 IMDEA Software Institute, 2 Spanish Research Council (CSIC), and 4 Technical University of Madrid (UPM)
MFoC – November 26, 2019
De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 1 / 19
Introduction
Objective Random testing based on assertion properties.
◮ Integrated in an assertion-based verification and debugging framework.
Combined with static analysis. Unified assertion language.
◮ Mostly automatic.
Language agnostic:
◮ Translate programs into an IR: (Constraint) Horn Clauses. ◮ Perform analysis and assertion-based testing on that IR.
Tools The Ciao System:
◮ Built up from a logic-based (CHC) kernel. ◮ Rich assertion language and analysis/testing framework.
De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 2 / 19
Horn Clauses as IR
javac soot + Ciao transform.
Transformation Analysis / Testing
Java parser xcc / clang / Ciao (Horn clauses) IR − CFG Ciao Source Java Source Java Bytecode ISA / LLVM / ... XC Source Sharing Shapes/sizes CHA ... Domains Analysis Resources Assertions Testing
Transformation: ◮ Source: Program P in LP + Semantics of LP ◮ Target: A (C) Horn Clause program capturing [[P]]. Block-based CFG. Each block represented as a Horn clause. Allows supporting multiple languages.
De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 3 / 19
Horn Clauses as IR
Adding numbers from 1 to n
source public static int r(int n) { int ans=0; while(n>0) { ans+=n; n-=1; } return ans; }
De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 4 / 19
Horn Clauses as IR
Adding numbers from 1 to n
source public static int r(int n) { int ans=0; while(n>0) { ans+=n; n-=1; } return ans; } Horn clause IR
✞
:- pred r/2 : num * var. r(N,Ret) :- Ans=0, r_1(N, Ans, Ret). r_1(N, Ans, Ret) :- N>0, add(Ans, N, Ans1), N1 is N - 1, r_1(N1, Ans1, Ret). r_1(N, Ans, Ret):- N=<0, Ret=Ans.
✝ ✆
De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 4 / 19
Horn Clauses as IR
int fact(int N) { if (N <= 0) return 1; return N * fact(N - 1); } 1 <fact >: 2 0x01: entsp 0x2 3 0x02: stw r0, sp[0x1] 4 0x03: ldw r1, sp[0x1] 5 0x04: ldc r0, 0x0 6 0x05: lss r0, r0, r1 7 0x06: bf r0, <0x08> 10 0x07: bu <0x10> 11 0x0a: ldw r0, sp[0x1] 12 0x0b: sub r0, r0, 0x1 13 0x0c: bl <fact> 15 0x0d: ldw r1, sp[0x1] 16 0x0e: mul r0, r1, r0 17 0x0f: retsp 0x2 19 0x08: mkmsk r0, 0x1 20 0x09: retsp 0x2 1 fact(R0,R0_3):- 2 entsp(0x2), 3 stw(R0,Sp0x1), 4 ldw(R1,Sp0x1), 5 ldc(R0_1,b0x0), 6 lss(R0_2,bR0_1,R1), 7a bf(R0_2 ,0x8), 7b fact_aux(R0_2,Sp0x1,R0_3,R1_1). 9 fact_aux(1,Sp0x1,R0_4,R1):- 10 bu(0x10), 11 ldw(R0_1,Sp0x1), 12 sub(R0_2,R0_1 ,0x1), 13a bl(fact), 13b fact(R0_2,R0_3), 15 ldw(R1,Sp0x1), 16 mul(R0_4,R1,R0_3), 17 retsp(0x2). 18 fact_aux(0,Sp0x1,R0,R1):- 19 mkmsk(R0,0x1), 20 retsp(0x2). De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 5 / 19
The Ciao Model
:- pred Pred [:Precond] [=> Postcond] [+ Comp-formula ] .
Each typically a “mode” of use; the set covers the valid calls.
:- pred quicksort(X,Y) : (list(X,int), var(Y)) => sorted l(Y) + (is det,not fails). :- pred quicksort(X,Y) : (var(X), list(Y,int)) => ground(X) + non det.
Properties; from libraries or user defined (in the source language):
:- regtype list(X,L). list( ,[]). list(X,[H|T]) :- X(H), list(X,T). :- prop sorted l(L). sorted l([]). sorted l([ ]). sorted l([X,Y|Z]) :- X > Y, sorted l([Y|Z]).
Types/shapes, cost, data sizes, aliasing, termination, determinacy, non-failure, ...
Program-point Assertions: Inlined with code:
..., check( int(X), X>0 ), Z is Y/X, ....
Assertions optional, can be added at any time. Provide partial spec. Part of the programming language and “runnable.” Used everywhere: itf with analyzer, doc gen., foreign itf, debug ...
De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 6 / 19
The Ciao Model
+ Comparator (Incl. VCgen) Normalizer & Lib Itf. Assertion Analysis Info [[P]] Program P :− trust I Builtins/ Libs
RT Check verification warning compile−time error
verified
(ACC) (optimized) code certificate Unit Test
Analysis Static
possible run−time error
:− check :− false :− checked :− texec :− check :− test PREPROCESSOR
The Ciao Debugging Framework Abstract interpretation-based static analysis tries to verify assertions.
◮ Compile-time errors if violated, warnings if unable to verify.
Run-time checks generated for parts of asserts not verified statically.
◮ Run-time errors (exceptions) if violated.
User-defined unit tests.
De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 7 / 19
The Ciao Model
+ Comparator (Incl. VCgen) Normalizer & Lib Itf. Assertion Analysis Info [[P]] Program P :− trust I Builtins/ Libs
RT Check verification warning compile−time error
verified
(ACC) (optimized) code certificate Unit Test
Analysis Static
possible run−time error
:− check :− false :− checked :− texec :− check :− test PREPROCESSOR
Goal: random unit tests derived automatically from assertions. Components and ideas already present in Ciao model:
◮ Assertion language. ◮ Unit tests and runtime-checking framework. ◮ (Runnable) properties as generators → test cases from assrt. precondition.
De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 8 / 19
Test Case Generation
Objective: generating test cases from calls fields of assertions pred q(X1,...,Xn) : (pre1(X1,...,Xn),...,prer(X1,...,Xn)) [=> Postcond(X1,...,Xn)] [+ Comp-formula] In principle, properties can be run as efficient generators.
◮ Using free variables as arguments, ◮ and a suitable resolution strategy.
E.g.: ✞
list([]). list([_|X]) :- list(X).
✝ ✆
?- list(X).
− →
X=[];X=[_];X=[_,_]; X=[_,_,_];...
In practice, not that simple:
◮ Termination/efficiency trade-offs. ◮ Inappropriate test case distribution.
We study several cases.
De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 9 / 19
Test Case Generation
HC+Herbrand properties − → already generators if run
◮ e.g.,
?- list(X) −
→ X=[];X=[_];X=[_,_];...
Depth-first resolution strategy not always well suited:
◮ Efficient, but termination problems. ◮ Skewed distribution of values. ◮ Incomplete ( list_peano(X) → [],[0],[0,0] ... but [s(0)] never generated).
Solution: alternative search strategies.
◮ E.g., breadth-first, iterative deepening, etc. ◮ Already available in Ciao using ’packages’ mechanism.
Our tool → added also randomized search strategy
◮ Clauses chosen at random.
✞
solve_goal(G) :- random_clause(G,Body), solve_body(Body). ... solve_body([G|Gs]) :- solve_goal(G), solve_body(Gs).
✝ ✆
◮ Growth control, not repeated solutions, size parameter ...
Already enough for regular types.
De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 10 / 19
✞
list([]). list([_|Xs]) :- list(Xs).
✝ ✆
Test Case Generation
Proposal: Constraint-based random search Random search collecting constraints for each derivation
◮ E.g.,
X<Y interpreted as X.<.Y in CLP(Q).
◮ Solvers ensure constrains are satisfiable during all steps of generation → failure and backtracking otherwise.
E.g.,
sorted_l([N,M|Ms]):-N =<M, sorted_l([M|Ms]). sorted_l([_]). sorted_l([]). ?- sorted_l(L) → ...; L = [X,Y,Z], X.=<.Y, Y.=<.Z;....
Last step: random sampling or enumeration of constraints. E.g., L = [X,Y,Z], X.=<.Y, Y.=<.Z;... X=1, Y=4, Z=9 L=[1,4,9] Supported domains:
◮ Arithmetic → CLP(Q) and CLP(FD) domains + random sampler: ◮ Modes and sharing → new constraints domain + random sampler:
e.g.,
(list(X,var), list(Y,var), indep(X,Y)) X=[A,B,A],Y=[C,D]
◮ Generate-and-check or user-defined generators for other domains. ◮ Random instantiation of unconstrained free variables.
e.g.,
list(X) X=[_,_,_,_] X=[f(a),0,_,[1,2]]
De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 11 / 19
Integration with Static Analysis
Almost for free thanks to unified assertion framework for testing and analysis:
◮ Assertions can be verified statically (no need for testing). ◮ Assertions can be simplified statically (simpler testing).
Static analysis can guide test case generation. Example: using inferred calls patterns Suppose we have: ✞
:- check comp q(X,Y) + not_fails. p(X) :- q(X,Y), ... % q/2 not exported
✝ ✆ Analysis infers: ✞
:- true calls q(X,Y) : var(Y).
✝ ✆ So we can check instead: ✞
:- check comp q(X,Y) : var(Y) + not_fails.
✝ ✆
De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 12 / 19
Integration with Static Analysis
✞
:- check pred p(X) + not_fails.
✝ ✆ We could try to verify the following with the analyzer: ✞
:- check pred p(X): ground(X) + not_fails. :- check pred p(X): var(X) + not_fails. :- check pred p(X): (nonground(X), nonvar(X)) + not_fails.
✝ ✆ And then we would only need to test values outside those calls. The algorithm
start with entry A ∈ Dα.
1
(1) Try to prove assertion statically for A. If that fails, go to (2).
2
(2) test assertion recursively for set of abstract values S ⊆ {B|B ∈ Dα, B ⊑ A}. Go to (1) for each B.
3
(3) generate random test cases in γ(A) \ γ(B). Requirements:
◮ Suitable enumeration sampling function in the domain ◮ Transformation scheme between abstract values and properties. ◮ Rich generators for properties.
Analysis can still be treated as a black box otherwise. Additional advantage: it ensures coverage.
De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 13 / 19
Shrinking
Flaw of random testing: failed cases may be too complex → hard to debug ✞
:- check pred quicksort(Xs,Ys): (list(Xs,int), var(Ys)) + is_det. ... partition([],_,[],[]). partition([X|Xs],P,[X|L],R) :- X =< P, partition(Xs,P,L,R). partition([X|Xs],P,L,[X|R]) :- X >= P, partition(Xs,P,L,R). % should be >
✝ ✆ Assertion failed for quicksort([5,9,-2,12,-4,9,3],Ys) Solution (QuickCheck): shrinking
◮ Generate simpler test cases from found counter-examples
Testing quicksort([9,-2,12,-4,9,3],Ys) (fails too) Testing quicksort([-2,12,-4,9,3],Ys) (does not fail) Testing quicksort([9,12,-4,9,3],Ys) (fails too) ... Assertion failed for quicksort([9,9],Ys) Our tool → Automatic shrinking based on sizes (related to resource analysis/domains [SizedTypes]).
De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 14 / 19
Shrinking
Mirrors generation
◮ Can be seen as generation with further constraints. ◮ Reuses part of the generation framework.
No longer random → exhaustive, greedy search HC + Herbrand
◮ Use subtraces of generation trace. E.g.: X=[a,b] X=[a]
.
list(X) X=[Y|Ys] elem(Y) Y=a list(Ys) Y=[Z|Zs] elem(Z) Z=b list(Zs) Zs=[] list(X) X=[Y|Ys] elem(Y) Y=a list(Ys) Ys=[]
Other domains:
◮ New constraints, e.g.,
Generation: [X,Y], X<Y [3,5]. Shrinking: [X,Y], X<Y, 0<=X<3, 0<=Y<5 [2,4].
◮ Normal sampling (not random).
De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 15 / 19
Case Study
We test the implementation of the sharing-freeness abstract domain:
◮ lattice structure (leq/2, lub/3, glb/3, abstract/2) ◮ handling of builtins (builtin success/3)
✞
:- pred leq_reflexive(X) : shfr(X,_) + not_fails. leq_reflexive(X) :- leq(X,X). :- pred lub(X,Y,Z) : (shfr(X,Vs), shfr(Y,Vs)) => (leq(X,Z), leq(Y,Z)). :- pred builtin_sucess(Blt,Call,Succ) : (builtin(F,A), Blt=F/A, length(Vs,A), shfr(Call,Vs)) + (not_fails , is_det, leaves_no_choicepoints , not_further_inst(Call))} :- pred builtin_monotonic(Blt, X, Y) : (builtin(Blt), Blt=F/A, length(Vs,A), shfr(X,Vs), shfr(Y,Vs), leq(X,Y)) + not_fails. builtins_monotonic(Blt,X,Y) :- builtin_success(Blt,X,XSucc), builtin_success(Blt,Y,YSucc), leq(X,Y). :- pred builtin_soundness(Blt, Args) : (builtin(Blt), Blt=F/A, length(Args,A), list(Args, term)) + not_fails. builtin_soundess(Blt,Args) :- ... % abstraction of Args after F(Args) is lower than the result computed by builtin_success/3 for abstraction of Args
✝ ✆
De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 16 / 19
Case Study
Bugs Found No bugs found for leq/2, lub/2, glb/2 (simple and commonly used). Minor and larger bugs in builtin success/2.
◮ Choicepoints left. ◮ Failed for ⊥ in some builtins (never called in Ciaopp with that value). ◮ Bugs for less commonly-used builtins. ◮ Bug for var/1. Handler failed for specific abstract substitutions. ◮ Large bug for =/2. Handler failed for unrealistic literal X=X. ◮ Large bug for ==/2. Handler failed for unrealistic literal f(X)==g(Y).
Observations Many interesting assertions are discharged by analysis. But assertions with complex user-defined properties are usually not. Complex properties defined using auxiliary predicate with not fails assertion. Not suitable for debugging analysis precision. Often the errors occur in the assertions themselves and not the code.
De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 17 / 19
Conclusions
Summary We have presented an assertion-based random testing tool [LOPSTR’19]: Integrated with the Ciao assertion model. Fully automatic. With a very expressive assertion language. With rich test-case generation.
◮ Based on running properties in generation mode (random search strategy). ◮ With support for different constraint domains.
Enhanced by static analysis. Proven effective (used to debug static analyzer). Can be applied to programs in other languages by translating them to (constraint) Horn Clauses.
De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 18 / 19
Conclusions
De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 19 / 19