The Practical Assessment of Test Sets with Inductive Inference - PowerPoint PPT Presentation

The Practical Assessment of Test Sets with Inductive Inference Techniques Neil Walkinshaw Department of Computer Science University of Leicester September 4, 2010

B ACKGROUND Test Adequacy ◮ Assessing the ability of a test set to identify faults ◮ Successful execution of an adequate test set should imply that there are no faults in a tested program ◮ How do you know if a test set is adequate? ◮ Numerous adequacy criteria have been developed ◮ Statement / branch / path / data-flow, . . .

B ACKGROUND Test Adequacy ◮ Assessing the ability of a test set to identify faults ◮ Successful execution of an adequate test set should imply that there are no faults in a tested program ◮ How do you know if a test set is adequate? ◮ Numerous adequacy criteria have been developed ◮ Statement / branch / path / data-flow, . . . Problem ◮ Criteria based on syntax are often a poor approximation for actual adequacy

U SING I NFERENCE TO A SSESS T EST S ET A DEQUACY Program inputs T est input System under test generator

U SING I NFERENCE TO A SSESS T EST S ET A DEQUACY Inference engine Observations of Program inputs test executions T est input System under test generator

U SING I NFERENCE TO A SSESS T EST S ET A DEQUACY Hypothesis Inference engine Observations of Program inputs test executions T est input System under test generator

U SING I NFERENCE TO A SSESS T EST S ET A DEQUACY Hypothesis Equivalence implies test set adequacy Inference engine Observations of Program inputs test executions T est input System under test generator

U SING I NFERENCE TO A SSESS T EST S ET A DEQUACY Rationale: Hypothesis Only a sufficiently thorough test set will provide an adequate basis to infer an exact hypothesis. Equivalence implies test set adequacy Inference engine Observations of Program inputs test executions T est input System under test generator

U SING I NFERENCE TO A SSESS T EST S ET A DEQUACY Weyuker 1983 Lisp program Equivalence implies test set adequacy Inference engine Observations of Program inputs test executions T est input System under test generator

U SING I NFERENCE TO A SSESS T EST S ET A DEQUACY Bergadano and Gunetti 1996 Prolog program Equivalence implies test set adequacy Inference engine Observations of Program inputs test executions T est input System under test generator

U SING I NFERENCE TO A SSESS T EST S ET A DEQUACY Harder et al. 2003 X>0 Xie, Notkin 2003 Y < (A+B) Invariants Daikon Equivalence implies test set adequacy Inference engine Observations of Program inputs test executions T est input System under test generator

U SING I NFERENCE TO A SSESS T EST S ET A DEQUACY Berg et al. 2005 Raffelt, Steffen 2006 Bollig et al. 2008 Shahbaz, Li, Groz 2006 FSM Walkinshaw et al. 2009 Angluin State-merging Equivalence implies test set adequacy Inference engine Observations of Program inputs test executions T est input System under test generator

U SING I NFERENCE TO A SSESS T EST S ET A DEQUACY X>0 Y < (A+B) Hypothesis Undecidable Inference engine Observations of Program inputs test executions T est input System under test generator

U SING I NFERENCE TO A SSESS T EST S ET A DEQUACY X>0 Y < (A+B) Hypothesis Lots of random tests Inference engine W/WP-method (for FSMs) Observations of Program inputs test executions T est input System under test generator

adequacy tests P ROBLEM Based on exact results - no flexibility ◮ The inferred model is either equivalent to the subject system or not. ◮ The corresponding test set is either adequate or not. ◮ In reality, there is bound to be a certain degree of error. ◮ A test set may result in a model that is 99% correct, with only small, trivial errors accuracy examples

P ROBLEM Based on exact results - no flexibility ◮ The inferred model is either equivalent to the subject system or not. ◮ The corresponding test set is either adequate or not. ◮ In reality, there is bound to be a certain degree of error. ◮ A test set may result in a model that is 99% correct, with only small, trivial errors accuracy adequacy examples tests

T HE P ROBABLY A PPROXIMATELY C ORRECT (PAC) FRAMEWORK Setting ◮ There exists an instance space X ◮ The learning target is a concept c ⊂ X ◮ For any element x ∈ X , c ( x ) = 1 or 0 ◮ There is a selection procedure EX ( c , D ) that randomly selects elements in X ◮ The probability of them belonging to c is determined by some static distribution D (not necessarily known) ◮ Given a labelled set of examples selected by EX , it is the goal of the learning procedure to infer c

T HE P ROBABLY A PPROXIMATELY C ORRECT (PAC) FRAMEWORK Assessing a Learner ◮ Two problems 1. Can only guarantee accurate result if supplied with every possible instance in X . 2. Given that samples are a random subset, there is the chance that EX will supply a misleading sample. ◮ To address these issues, the success of a learner is characterised as follows: ◮ δ - probability that the hypothesis will meet the success conditions ◮ ε - allowable degree of error

T HE P ROBABLY A PPROXIMATELY C ORRECT (PAC) FRAMEWORK Evaluator Ex(c,D) Hypothesis example set A classifications Inference engine

T HE P ROBABLY A PPROXIMATELY C ORRECT (PAC) FRAMEWORK classifications ε δ example set B Evaluator Ex(c,D) hypothesis classifications Hypothesis probably approximately correct (or not)

U SING PAC TO A SSESS T EST A DEQUACY Evaluator Ex(c,D) Hypothesis example set A classifications Inference engine

U SING PAC TO A SSESS T EST A DEQUACY X>0 Evaluator Y < (A+B) T est input Hypothesis generator test set A test outcomes Inference engine

U SING PAC TO A SSESS T EST A DEQUACY test outcomes ε δ test set B Evaluator X>0 Y < (A+B) hypothesis T est input outcomes Hypothesis generator probably approximately adequate (or not)

U SING PAC TO A SSESS T EST A DEQUACY Assumptions ◮ Validity of final outcome must be interpreted with care ◮ Test set is being evaluated against itself ◮ Size of sets A and B must be sufficiently large and distinct ◮ Test set generator must be capable of (eventually) exhaustively exercising the SUT

C ONCLUSIONS ◮ Inferring models from tests gives us a ’test-eye view’ of the system ◮ Test adequacy can be assessed by measuring model accuracy ◮ This can be achieved with established ML techniques ◮ For a given type of system (e.g. state-based) the PAC approach can be used to assess and compare the general performance of testing techniques. Challenge Find the best combination of machine-learner and test-set generator.