LEARNING-BASED TESTING: RECENT PROGRESS AND FUTURE PROSPECTS Karl - PowerPoint PPT Presentation

LEARNING-BASED TESTING: RECENT PROGRESS AND FUTURE PROSPECTS Karl Meinke Computer Science School KTH Stockholm

Overview of Talk Tes&ng ¡as ¡a ¡search ¡problem ¡ 1. Architecture ¡of ¡an ¡LBT ¡tool: ¡LBTest ¡ 2. General ¡Principles ¡of ¡LBT ¡ 3. Quan&fying ¡the ¡Complexity ¡of ¡Tes&ng ¡and ¡Learning ¡ 4. Regular ¡Inference ¡and ¡Over-­‑approxima&on ¡ 5. Querying ¡Power ¡of ¡Model ¡Checkers ¡ 6. Unifying theme of 3,4 and 5 is measuring model convergence since we rarely learn to completion Based on: ¡ K. Meinke: Learning-Based Testing: Recent Progress and Future Prospects , Proc. This Workshop? ¡

Testing as a Search Problem • Basic route to fully automated testing • Has analogies to machine learning • Finding access strings • Finding distinguishing strings • Random testing • Search-based Testing (optimise a cost function, SBSE) • Gradient-descent • Genetic algorithms • Simulated annealing • Model-based Testing • Constraint satisfaction on abstract models

1. LBTest Tool • LBTest implements black-box requirements testing for embedded systems with off-the-shelf and customised components • LBTest automates 3 processes: • Test Case Generation (ATCG), 3 Sources: • Active learning queries • Model checker generated counterexamples • Stochastic equivalence checker queries • Test execution (online testing) • Verdict construction (pass/fail/warning/exception) • Some configurations quickly achieve high model convergence

LBTest Architecture Communication wrapper observed test output case i n o n System under Test e.g. jar file i n Automaton n = 1, 2, … Learning Algorithm Stochastic model i n LTL equivalence abstraction M n Requirement checker Formula Req NuSMV counterexample i n Model Checker Verdict v n TCG and Oracle LBTest

Technical & Process Advantages • Well suited to agile development • Model is always synchronised to actual code • No false positives or false negatives due to wrong/ outdated models (C.f. model-based testing) • Avoid manual model construction and maintenance

Modular Structure • Learners • L*Mealy • Kearn’s algorithm • CGE, ICGE (term rewriting system representation) • MinSplit (NDFA representation) • Hybrid automaton learner HyCGE (infinite state systems) • Model checkers • NuSMV 2.5 • BDD checker • BMC/SAT solver • nuXmv 1.0 • Stochastic equivalence checker • First / longest / shortest difference (strategies)

Requirements Modeling • Modeling reactive systems needs a time concept • LBTest uses propositional linear temporal logic (PLTL) • PLTL = “Boolean logic + time” • Conventional model-based testing (conformance testing) is the next-only part of PLTL. • Could interface LTL to visual requirements modeling languages and pattern languages .

Approximate Models • Real-world SUTs are infinite state systems • LBTest constructs finite state approximations through finite partition sets . • Input partitioning is implemented in LBTest (test selection) • Output partitioning is implemented in SUT wrapper (equivalence class) • Gives a limited first-order extension to PLTL.

Verdict Construction (Oracle step) • On-the-fly verdict construction filters false negatives • Compares two behaviours: (1) a predicted behaviour from model (bad) (2) an observed behaviour in SUT • Prediction == Observation => Fail/Warning • Prediction != Observation => Pass • No Observation => Exception/Timeout error

RRWhl_rpm ABS:absRR ABS:absFR ABSTorq_RR GlobalBrakeController : Pedal: brake gbc Pedal: accel ABS:absRL ABS:absFL Case Study: Brake-by-Wire ECU

Fourteen Black-box Requirements REQ-4 If the brake pedal is pressed and the actual speed of the vehicle is larger than 10 km/h and the slippage sensor shows that the (front right) wheel is slipping, this implies that the corresponding brake torque at the (front right) wheel should very quickly be 0. G( BrakePedal = b & Motion = moving & SlipRR = slipping -> X( ABSBrakeTorqueRR = zero ) )

Model #3 I a i under1;still;zero;noSlip after 400 msec b b i under1;still;zero;noSlip i bi a under1;still;zero;noSlip biaaa a 30;moving;zero;noSlip bia i 10;moving;zero;noSlip a b biaa a i b 20;moving;zero;noSlip b a b biaab biaaa b a i 1;still;nonZero;slip 30;moving;zero;noSlip i b biaabi bb b a i b a i 1;still;zero;noSlip under1;still;nonZero;noSlip

2. ¡Abstract ¡LBT ¡Algorithm 1. M 0 ¡:= ¡getIni&alHypothesis(); ¡ ¡ 2. ¡For ¡each ¡ k ¡>= ¡0 ¡ do ¡ Model ¡check ¡ M k ¡against ¡ Req ¡ 1. Choose ¡ “ best ¡counterexample ” i k+1 ¡from ¡step ¡2.1 ¡ 2. Execute ¡ i k+1 ¡ ¡on ¡SUT ¡to ¡produce ¡ o k+1 ¡ 3. if ¡( i k+1 ¡ , ¡o k+1 ) ¡sa&sfies ¡! Req ¡ label ¡ i k+1 ¡as ¡an ¡error ¡ 4. If ¡ equivalent ¡(SUT, ¡M k+1, ¡convergenceBound) ¡break. ¡ 5. M k+1 ¡:= ¡getNextHypothesis ¡(i k+1 ¡ , ¡o k+1 ) ¡ ¡ 6. ¡ ¡

3. Quantifying the Complexity of Testing and Learning

4. Regular Inference by Over-Approximation

5. Querying Power of Model Checkers What is the querying power of model checkers? Do they accelerate convergence? (Lucent patent!) Answer: Switch the MC off … In: 2 automotive case studies using more than 20 different requirements and 3 learning algorithms we found no difference in convergence at all.

Conclusions • Advantages • Flexible (black-box) • High-volume, high coverage (active learning) • Metric coverage (stochastic equivalence checking) • Accurate test verdicts (formal requirements & model checking)

Future Prospects • Latency problems – distributed learning? • Faster learning – sparse models, more extrapolation? • Learning more expressive models? • No statistical methods here? • Implicit model techniques (NN) & model checking? • Multiple counter-examples (model checking)