Jüri Vain Dept. of Computer Science Tallinn University of Technology J.Vain “...Reactive Planning 1 Testing...”Abo, Feb 3, 2012
Vain, Jüri; Kääramees, Marko; Markvardt, Maili Online testing of nondeterm inistic system s w ith reactive planning tester. In: Petre, L.; Sere, K.; Troubitsyna, E. (Eds.). Dependability and Computer Engineering : Concepts for Software-Intensive Systems. Hershey, PA: IGI Global (2012), pages 1 1 3 -1 5 0 . J.Vain “...Reactive Planning 2 Testing...”Abo, Feb 3, 2012
Outline Preliminaries Model-Based Testing Online testing Reactive Planning Tester (RPT) Synthesis of the RPT Performance issues Test execution environment dTron Demo J.Vain “...Reactive Planning 3 Testing...”Abo, Feb 3, 2012
What is testing for? to check the quality (functionality, reliability, performance, … ) of an (software) object -by performing experiments -in a controlled way In avg. 10-20 errors per 1000 LOC 30-50 % of development time and cost in software 4 J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
What is a Test? Test Cases Output Test Data Correct Software result? under Test (SUT) Oracle 5 J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
Model- Based Testing (typically) J.Vain “...Reactive Planning 6 Testing...”Abo, Feb 3, 2012
Specifics of testing embedded systems Real environment in the loop! – non-determ inism – partial observability – RT constraints – dependability test coverage issues How to address them in testing? J.Vain “...Reactive Planning 7 Testing...”Abo, Feb 3, 2012
Option 1: Offline vs Online testing Offline testing Open “control” loop Test is not adaptive to outputs or timing of SUT Test planning – result analysis loop is long Online testing Flexible, test control is based on SUT feedback One test covers usually many test cases J.Vain “...Reactive Planning 8 Testing...”Abo, Feb 3, 2012
Online testing Group of test generation and execution algorithms that compute successive stimuli at runtime directed by the test purpose and the observed outputs of the SUT Tester SUT J.Vain “...Reactive Planning 9 Testing...”Abo, Feb 3, 2012
Online testing Advantages: The state-space explosion problem is reduced because only a limited part of the state-space needs to be kept track of at any point in time. Drawbacks: Exhaustive planning is diffcult due to the limitations of the available computational resources at the time of test execution. J.Vain “...Reactive Planning 10 Testing...”Abo, Feb 3, 2012
Option 2: Test scripting vs model-based generation Scripting: does not need SUT modelling + sensitive to human errors - inflexible, needs rewriting even with small changes of SUT specs - hard to achieve test coverage - Model-based generation: considerable SUT modelling effort - correctness of tests is verifiable + easy to modify and regenerate + clear characteristics of coverage + J.Vain “...Reactive Planning 11 Testing...”Abo, Feb 3, 2012
Model-Based Testing Given Model of the SUT specification System Under Test (SUT), The test goal (in terms of spec. model elements) Find If the SUT conforms to the specification in terms expressed in the test goal. NEEDED! - Sufficiently rich modelling formalism, - Supporting tool set. J.Vain “...Reactive Planning 12 Testing...”Abo, Feb 3, 2012
Model -Based Testing We assume formal specs as: UML State Charts Extended Finite State Machines MSC OCL etc. UPTA -Timed Automata (timing, parallelism, test data) J.Vain “...Reactive Planning 13 Testing...”Abo, Feb 3, 2012
Online MBT architecture : UppAal-TRON Spec = UppAal Timed Automata Network: Env | | IUT Input event ordering Expected system reaction ”Relativized Real-Tim e i/ o conform ance” Relation Timed Trace: i 1 .2½ .o 1 .3.o 2 .19.i 2. 5.i 3 [ www.uppaal.com] J.Vain “...Reactive Planning 16 Testing...”Abo, Feb 3, 2012
Bottleneck of online MBT: planning Spectrum of planning methods Random walk (RW): select test stimuli in random inefficient - based on random exploration of the state space leads to test cases that are unreasonably long may leave the test purpose unachieved RW with reinforcement learning (anti-ant) the exploration is guided by some reward function ........ Exploration with exhaustive planning MC provides possibly an optimal witness trace the size of the model is critical in explicit state MC state explosion in "combination lock" or deep loop models J.Vain “...Reactive Planning 17 Testing...”Abo, Feb 3, 2012
Bottleneck of online MBT: planning Spectrum of planning methods Random walk (RW): select test stimuli in random inefficient - based on random exploration of the state space leads to test cases that are unreasonably long Zero planning may leave the test purpose unachieved RW with reinforcement learning (anti-ant) 1 step ahead the exploration is guided by reward function ........ ??? Full Exploration with exhaustive planning space MC provides possibly an optimal witness trace the size of the model is critical in explicit state MC state explosion in "combination lock" or deep loop models J.Vain “...Reactive Planning 18 Testing...”Abo, Feb 3, 2012
Bottleneck of online MBT: planning Spectrum of planning methods Random walk (RW): select test stimuli in random inefficient - random exploration of the state space test cases that are unreasonably long Zero planning may leave the test purpose unachieved RW with reinforcement learning (anti-ant) 1 step ahead the exploration is guided by some reward function ........ Planning w ith adaptive horizon! Full Exploration with exhaustive planning space MC provides possibly an optimal witness trace the size of the model is critical in explicit state MC state explosion in "combination lock" or deep loop models J.Vain “...Reactive Planning 19 Testing...”Abo, Feb 3, 2012
Reactive Planning in a Nutshell (1) Instead of a complete plan, only a set of decision rules is derived The rules direct the system towards the test goal. Based on current situation evaluation just one subsequent input is computed at a time. Planning horizon is adjusable J.Vain “...Reactive Planning 20 Testing...”Abo, Feb 3, 2012
Reactive Planning: Planning cycle Identify current state of SUT: Observe the output (or history) of the SUT Pick the next move: Select one from unsatisfied test (sub-)goals G i Compute the best strategy for G i : Gain function guides the exploration of the model (choose the transition with the shortest path to the (sub-)goal G i ) J.Vain “...Reactive Planning 21 Testing...”Abo, Feb 3, 2012
Reactive Planning Tester in a Nutshell (2) Offline phase Online phase SUT State model of Test goal the IUT Stimuli / responses Reachability Reactive Planning Tester Analysis (RPT) implementation Reactive Planning Tester (RPT) model RPT autonomously generates stimuli to reach the goal RPT is another J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 22 state model 2012
Constructing the RPT model Test strategy parameters: -planning horizon - timeouts Synthesis of RPT Constructing Model of gain guards Extraction SUT (GG) of the Reduction Model control of GG of RPT Constructing strcture Test goal gain functions J.Vain “...Reactive Planning 23 Testing...”Abo, Feb 3, 2012
RPT: Key Assumptions Testing is guided by the (EFSM) model of the tester and the test goal. Decision rules of reactive planning are encoded in the guards of the transitions of the tester model. The SUT model is presented as an output observable nondeterministic EFSM in which all paths are feasible 1 . 1 - A. Y. Duale and M. U. Uyar. A method enabling feasible conformance test sequence generation for EFSM models. IEEE Trans. Comput.,53(5): 614–627, 2004. J.Vain “...Reactive Planning 24 Testing...”Abo, Feb 3, 2012
Example: Nondeterministic FSM e 1 : i 0 / o e 3 : e 0 : i 3 / o 1 i 0 / o s 1 s 2 3 e 2 : 0 i 2 / o 2 e 5 : i 5 / o 5 e 6 : e 7 : i 6 / o 6 e 4 : i 7 / o 7 i 3 / o 4 s 3 i 0 and i 3 are output observable nondeterministic inputs J.Vain “...Reactive Planning 25 Testing...”Abo, Feb 3, 2012
Encoding the Test Goal in SUT Model Trap - a boolean variable assignment attached to the transitions of the IUT model A trap variable is initially set to false . The trap update functions are executed (set to true ) when the transition is visited. J.Vain “...Reactive Planning 26 Testing...”Abo, Feb 3, 2012
Goal directed testing: The power of traps Several goals can be expressed all/ selected transitions transition sequences (traps with reference to other traps) advanced goals using auxiliary variables, consequent transitions, repeated pass, … traps with 1st order predicates data variables Properties not expressible by traps Assertions/ invariants – always/ never properties The model specifies only the allowed behaviours Åbo Akademi University, Dec 2011
Recommend
More recommend
