Verifying Test Hypotheses - HOL/TestGen An Experiment in Test and - - PowerPoint PPT Presentation
Verifying Test Hypotheses - HOL/TestGen Verifying Test Hypotheses - HOL/TestGen An Experiment in Test and Proof Thomas Malcher January 20, 2014 1 / 20 Verifying Test Hypotheses - HOL/TestGen HOL/TestGen Outline Introduction Test Hypotheses
Verifying Test Hypotheses - HOL/TestGen
An Experiment in Test and Proof Thomas Malcher January 20, 2014
1 / 20
Verifying Test Hypotheses - HOL/TestGen
Outline
2 / 20
Verifying Test Hypotheses - HOL/TestGen
What it does?
builds test case and test data from a formal/logical test specification the Model is a logical expression (test specification) dynamic testing/blackbox testing generates functional test strategy
3 / 20
Verifying Test Hypotheses - HOL/TestGen
What it is?
semi-automated test tool for specification based tests Academic tool to get insight in the connection between test and verification
What is the nature of the relation between test and proof? Does a test approximate a verification? Does testing contribute to verification?
4 / 20
Verifying Test Hypotheses - HOL/TestGen
Foundations
Isabelle/HOL Higher-Order-Logic (HOL)
like a functional language extended by logical quantifiers Test Hypotheses
1 i s s o r t e d ( [ ] ) = True 2 i s s o r t e d ( x#xs ) = ( case xs
[ ] − − > True | 3 y#ys − − > ( ( x<y ) or ( x=y ) ) and i s s o r t e d ( xs ) )
5 / 20
Verifying Test Hypotheses - HOL/TestGen
The Core
Decompose a test specification (TS), into a semantically equivalent set of Test cases TCi and Test hypotheses Hi [TC1 . . . TCn, H1 . . . Hm] = ⇒ TS If the system under test passes all test cases TCi successfully, and if it satisfies all test hypothesis Hi, it is correct with respect to the test specification TS. Or:
validation by test + proof of test hypotheses → validation by proof
6 / 20
Verifying Test Hypotheses - HOL/TestGen
7 / 20
Verifying Test Hypotheses - HOL/TestGen
We can distinguish between 2 types of test hypotheses: uniformity hypotheses ∃x : P(x) = ⇒ ∀x : P(x) regularity hypotheses ∀x : |x| <= k ∧ P(x) = ⇒ ∀x : k < |x| ∧ P(x) e.g: input 32bit integer strong uniformity hypothesis: input ”1” passes → its correct for all integers weak uniformity hypothesis: tests for all integers
8 / 20
Verifying Test Hypotheses - HOL/TestGen
Although we always use Test Hypotheses they are never explicit explicit Test Hypotheses explicit Test Hypotheses formalize common test practices and express the gab beetween test sucess and correctness [4] A test set should never be presented independently of its hypotheses [5]
9 / 20
Verifying Test Hypotheses - HOL/TestGen
Workflow/System
Four phases: writing the test specification TS
(e.g: is sorted(SUT(list)))
generation of test cases TC and test hyptheses for the TS generation of test data TD (variables replaced by ground instance) the test execution phase
10 / 20
Verifying Test Hypotheses - HOL/TestGen
HOL/TestGen generates test cases toghether with the explicit hypotheses They give information about the quality of the test cases Test Hypotheses can be seen as a proof obligation for verification
validation by test + proof of test hypotheses = ⇒ validation by proof
11 / 20
Verifying Test Hypotheses - HOL/TestGen
uniformity hypothesis TS : if 0 ≤ x then SUT(x) else SUT(-x) test theorem 1 : 0 ≤?X1 → SUT(?X1) 2: THYP: (∃x.0 ≤ x → SUT(x)) = ⇒ (∀x.0 ≤ x → SUT(x)) 3: ?X2 < 0 → SUT(−?X2) 4: THYP: (∃x.x < 0 → SUT(−x)) = ⇒ (∀x.x < 0 → SUT(−x))
data generation phase will easily generate instances of the test cases e.g., SUT(3) and SUT(−(−4)), (satisfying the constraints)
12 / 20
Verifying Test Hypotheses - HOL/TestGen
regularity hypothesis TS : SUT(list l) test theorem 1 : SUT([]) 2: SUT([?X1]) 3: SUT([?X2,?X3]) 4: THYP: ∀x.2 < |x| = ⇒ SUT(x)
depth is user defined; in this example 2 uniformity hypotheses omitted
13 / 20
Verifying Test Hypotheses - HOL/TestGen
example ”List” from HOL/TestGen
14 / 20
Verifying Test Hypotheses - HOL/TestGen
refeed test hypothesis into HOL/Testgen again generates more testcases and weakens the hypothesis. if we get rid of all hypotheses we would have to test the program with all possible inputs.
15 / 20
Verifying Test Hypotheses - HOL/TestGen
uniformity hypotheses
proofed over real code all test cases passed + all hypotheses proofed → SUT verified uniformity hypothesis. ∃x, xa.is sorted(SUT([xa, x])) = ⇒ ∀x, xa.is sorted(SUT([xa, x]))
SUT replaced by program sort, can be written as:
x, xa, x′, xa′′ : is sorted(sort([xa′, x′])) = ⇒ is sorted(sort([xa, x])) The only way to proceed is by discarding the assumption x, xa : is sorted(sort([xa, x])) .....
16 / 20
Verifying Test Hypotheses - HOL/TestGen
uniformity hypotheses
This shows that the test case is irrelevant for the proof of uniformity as the assumption (=test case) of the hypothesis had to be discharged test contributes to the proof only if it covers input space then the test is the verification
17 / 20
Verifying Test Hypotheses - HOL/TestGen
regularity hypotheses
The uniformity hypotheses can be combined to ∀l.|l| ≤ 3 = ⇒ is sorted(sort(l)) is sorted(l) → is sorted(ins(a, l)) regularity hypothesis by induction. anchor : a, l.|l| = 3 = ⇒ is sorted(ins(a, sort(l))) step : a, l.is sorted(sort(l)) = ⇒ is sorted(ins(a, sort(l)))
18 / 20
Verifying Test Hypotheses - HOL/TestGen
tests and (post-hoc) verifications can work together Test hypotheses can be seen as proof obligation that is proven in later stages of validation if needed Test hypotheses give information about the quality of tests testing test hypotheses can be used to approximate verification (covering input space) no hope for the desire to use tests to simplify proofs
19 / 20
Verifying Test Hypotheses - HOL/TestGen
Brucker, Br¨ ugger, Krieger, and Wolff. Hol-testgen 1.7.0 user guide. Laboratoire en Recherche en Infromatique (LRI),Universit e Paris-Sud, 2012. Brucker, Br¨ ugger, and Wolff. Verifying test-hypotheses - an experiment in test and proof. In Proceedings of the Fourth Workshop on Model Based Testing (MBT 2008), 2008. Brucker and Wolff. Interactive testing with hol-testgen. Formal Approaches to Software Testing 5th International Workshop, FATES, 2005. M.-C. Gaudel G. Bernot and B. Marre. Software testing based on formal specifications: a theory and a tool. Software Engineering Journal, 6(6):387-405, 1991. Marie-Claude Gaudel. Testing can be formal, too. Laboratoire en Recherche en Infromatique (LRI),Universit e Paris-Sud, 1995. 20 / 20