killing bugs in a black box with model based mutation
play

Killing Bugs in a Black Box with Model-based Mutation Testing - PowerPoint PPT Presentation

Mar 24, 2023 •369 likes •1.49k views

Institute of Software Technology tugraz Killing Bugs in a Black Box with Model-based Mutation Testing Bernhard K. Aichernig Institute of Software Technology Graz University of Technology, Austria MT CPS Workshop Vienna, 11 Apr 2016 B.K.

  1. Institute of Software Technology tugraz Model-Based Mutation Testing Mutation Model Tool Test Case Generator: Conformance Checker Abstract Test Case SUT Test Driver B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 12 / 64

  2. Institute of Software Technology tugraz Model-Based Mutation Testing Mutation Model Model Mutant Tool Test Case Generator: Conformance Checker Abstract Test Case SUT Test Driver B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 12 / 64

  3. Institute of Software Technology tugraz Model-Based Mutation Testing Mutation Model Model Mutant Tool Test Case Generator: Conformance Checker if ¬ conforms Abstract Test Case then pass/fail SUT Test Driver B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 12 / 64

  4. Institute of Software Technology tugraz Model-Based Mutation Testing Mutation Model Model Mutant Tool Test Case Generator: Conformance Checker if ¬ conforms Abstract Test Case if conforms then fail SUT Test Driver B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 12 / 64

  5. Institute of Software Technology tugraz Model-Based Mutation Testing then ¬ conforms Mutation Model Model Mutant Tool Test Case Generator: Conformance Checker if ¬ conforms Abstract Test Case if conforms then fail SUT Test Driver B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 12 / 64

  6. Institute of Software Technology tugraz MoMuT Tools MoMuT ◮ is a family of tools implementing Model-based Mutation Testing. ◮ is jointly developed and maintained by AIT and TU Graz ◮ supports different modelling styles: ◮ MoMuT::UML (UML state machines) ◮ MoMuT::OOAS (OO Action Systems) ◮ MoMuT::QAS (Qualitative Action Systems) ◮ MoMuT::TA (Timed Automata) ◮ MoMuT::TAS (Timed Action Systems) ◮ MoMuT::REQs (Synchronous Requirement Interfaces) www.momut.org B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 13 / 64

  7. Institute of Software Technology tugraz Agenda ◮ Model-based Mutation Testing ◮ Real-Time Systems ◮ Hybrid Systems ◮ Discrete Systems B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 14 / 64

  8. Institute of Software Technology tugraz Conformance Relation of Timed Systems ... defines in a testing theory what constitutes a failure. Definition (Timed input-output conformance – tioco [Krichen&Tripakis09]) Given a timed automaton Model and a Mutant with inputs and outputs Mutant tioco Model iff ∀ σ ∈ L ( Model ) : out ( Mutant after σ ) ⊆ out ( Model after σ ) σ A after σ = { s ∈ S | s 0 − → s } S ... set of all states t s 0 ... initial state elapse(s) = { t > 0 | s − →} σ ... timed trace of labels a out ( s ) = { a ∈ Σ O | s − →} ∪ elapse(s) Σ O ... output labels out ( S ) = � s ∈ S out ( s ) B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 15 / 64

  9. Institute of Software Technology tugraz tioco and Language Inclusion start Theorem ([Krichen&Tripakis09]) a ? b ? c ≥ 2 L ( Mutant ) ⊆ L ( Model ) ⇒ Mutant tioco Model a ? Σ I ∪ Σ O c < 2 Theorem ([Krichen&Tripakis09]) b ? If Model is input-enabled , then Mutant tioco Model ⇒ L ( Mutant ) ⊆ L ( Model ) x ! a ? ... Demonic completion for deterministic TA For deterministic TA, reduce tioco check to language inclusion check (PSPACE-complete). B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 16 / 64

  10. Institute of Software Technology tugraz k-Bounded Language Inclusion ◮ Construct a formula ϕ k A I , A S that is satisfiable if L ( A I ) �⊆ L ( A S ) ◮ providing a timed trace as witness ϕ k ≡ A I , A S i = 1 ( d i ≥ 0 ∧ 1 ≤ α i ≤ | Σ | ) ∧ i ≥ 1 ∧ i ≤ k � k ∧ (delays and actions) 1 ≤ i ≤ k ∧ (in i steps) init A I ( X I , C I ) ∧ path 1 , i − 1 ( A , D , X I , C I ) ∧ (reach in mutant) A I init A S ( X S , C S ) ∧ path 1 , i − 1 ( A , D , X S , C S ) ∧ (reach in model) A S path i , i A I ( A , D , X I , C I ) ∧ ¬ path i , i A S ( A , D , X S , C S ) (failure) Variable sets: x i ∈ X ... location at step i α i ∈ A ... i th discrete action d i ∈ D ... i th time delay { c i , c ∗ , i } ⊆ C ... clock valuation after i th time and discrete step B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 17 / 64

  11. Institute of Software Technology tugraz Experimental Results I ◮ Bounded language inclusion check for deterministic Uppaal TA ◮ Implemented in Scala calling SMT solver Z3 ◮ Car alarm system characteristics: deterministic, ◮ 5 clock variables, 16 locations, 25 transitions. ◮ 8 mutation operators → 1,320 mutants ◮ Overall runtime: 30 minutes ( k = 12) Depth Bounded Model Checking Symbolic Execution Mean Median Max Min Mean Median Max Min 12 1 . 4 s 1 . 1 s 33 s 0 . 07 s Runtime details B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 18 / 64

  12. Institute of Software Technology tugraz Experimental Results I ◮ Bounded language inclusion check for deterministic Uppaal TA ◮ Implemented in Scala calling SMT solver Z3 ◮ Car alarm system characteristics: deterministic, ◮ 5 clock variables, 16 locations, 25 transitions. ◮ 8 mutation operators → 1,320 mutants ◮ Overall runtime: 30 minutes ( k = 12) Depth Bounded Model Checking Symbolic Execution Mean Median Max Min Mean Median Max Min 12 1 . 4 s 1 . 1 s 33 s 0 . 07 s Runtime details B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 18 / 64

  13. Institute of Software Technology tugraz Timed Action Systems 1 types { State = [ ... | Flash | FlashSound | Silent | SwitchOffAlarm | ... ]; } 2 state { 3 loc : State; } 4 clocks [Real]{ c;d;e; f ;g } 5 init { 6 loc := OpenAndUnlocked;} 7 invariant { 8 loc == Flash then e <= 0; if 9 if loc == FlashSound then e <= 30; 10 if loc == Silent then e <= 300; 11 ... } 12 actions { 13 !soundOn#1() if loc == Flash && e == 0 then { loc := FlashSound; }; 14 15 !soundOff#1() if loc == FlashSound && e == 30 then { loc := Silent ; }; 16 17 ?unlock#6() resets g if loc == FlashSound && e < 30 then { loc := SwitchOffAlarm; }; 18 ... } 19 B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 19 / 64

  14. Institute of Software Technology tugraz Symbolic Execution of Timed Action Systems path conditions . . . blue pc = ↼ � pc s 0 symbolic (clock) states . . . red q c = { e �→ ↼ � d , . . . } B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 20 / 64

  15. Institute of Software Technology tugraz Symbolic Execution of Timed Action Systems path conditions . . . blue pc = ↼ � pc s 0 symbolic (clock) states . . . red q c = { e �→ ↼ � d , . . . } ! soundOn pc = ↼ � pc ∧ Flash = Flash s 1 q = { loc �→ FlashSound , . . . } B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 20 / 64

  16. Institute of Software Technology tugraz Symbolic Execution of Timed Action Systems path conditions . . . blue pc = ↼ � pc s 0 symbolic (clock) states . . . red q c = { e �→ ↼ � d , . . . } ! soundOn pc = ↼ � ✭ pc ∧ ✭✭✭✭ Flash = Flash s 1 q = { loc �→ FlashSound , . . . } B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 20 / 64

  17. Institute of Software Technology tugraz Symbolic Execution of Timed Action Systems path conditions . . . blue pc = ↼ � pc s 0 symbolic (clock) states . . . red q c = { e �→ ↼ � d , . . . } ! soundOn pc = ↼ � pc s 1 q = { loc �→ FlashSound , . . . } delay ( d ) pc ∧ Flash = Flash → ↼ � pc = ↼ � d + d ≤ 30 ∧ . . . s 2 q c = { e �→ ↼ � d + d , . . . } B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 20 / 64

  18. Institute of Software Technology tugraz Symbolic Execution of Timed Action Systems path conditions . . . blue pc = ↼ � pc s 0 symbolic (clock) states . . . red q c = { e �→ ↼ � d , . . . } ! soundOn pc = ↼ � pc s 1 q = { loc �→ FlashSound , . . . } delay ( d ) pc ∧ ↼ � pc = ↼ � d + d ≤ 30 s 2 q c = { e �→ ↼ � d + d , . . . } B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 20 / 64

  19. Institute of Software Technology tugraz Symbolic Execution of Timed Action Systems path conditions . . . blue pc = ↼ � pc s 0 symbolic (clock) states . . . red q c = { e �→ ↼ � d , . . . } ! soundOn pc = ↼ � pc s 1 q = { loc �→ FlashSound , . . . } delay ( d ) pc ∧ ↼ � pc = ↼ � d + d ≤ 30 s 2 q c = { e �→ ↼ � d + d , . . . } ? unlock ! soundOff pc ∧ ↼ � pc ∧ ↼ � pc = ↼ � pc = ↼ � d + d < 30 d + d = 30 s 3 s 4 q c = { g �→ 0 } q = { loc �→ Silent } q = { loc �→ SwitchOffAlarm } delay ( d ′ ) delay ( d ′ ) B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 20 / 64

  20. Institute of Software Technology tugraz Symbolic Execution of Timed Action Systems path conditions . . . blue pc = ↼ � pc s 0 symbolic (clock) states . . . red q c = { e �→ ↼ � d , . . . } ! soundOn pc = ↼ � pc s 1 q = { loc �→ FlashSound , . . . } delay ( d ) pc ∧ ↼ � pc = ↼ � d + d ≤ 30 s 2 q c = { e �→ ↼ � d + d , . . . } ? unlock ! soundOff pc ∧ ↼ � pc ∧ ↼ � pc = ↼ � pc = ↼ � d + d < 30 d + d = 30 s 3 s 4 q c = { g �→ 0 } q = { loc �→ Silent } q = { loc �→ SwitchOffAlarm } delay ( d ′ ) delay ( d ′ ) Provides all symbolic timed traces through model! B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 20 / 64

  21. Institute of Software Technology tugraz Conformance Checking via Symbolic Execution ◮ Bounded implicit product graph exploration ◮ Simultaneous symbolic execution of all model traces ◮ Non-conformance checks (stioco) of the form: pc q . . . path condition of symbolic state q B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 21 / 64

  22. Institute of Software Technology tugraz Conformance Checking via Symbolic Execution ◮ Bounded implicit product graph exploration ◮ Simultaneous symbolic execution of all model traces ◮ Non-conformance checks (stioco) of the form: ∃ q fail ∈ ModelStates � �� � all symbolic states after current trace pc q fail � �� � state reachable (model) pc q . . . path condition of symbolic state q B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 21 / 64

  23. Institute of Software Technology tugraz Conformance Checking via Symbolic Execution ◮ Bounded implicit product graph exploration ◮ Simultaneous symbolic execution of all model traces ◮ Non-conformance checks (stioco) of the form: ∃ q fail ∈ ModelStates , ∃ λ ∈ Observations : � �� � all symbolic states after current trace � � � pc q fail ∧ pc s ∧ guards λ [ state s ] ∧ � �� � s ∈ MutantStates state reachable (model) � �� � observation possible (mutant)   � ¬ pc q ∧ guards λ [ state q ]   q ∈ ModelStates � �� � observation not possible (model) pc q . . . path condition of symbolic state q B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 21 / 64

  24. Institute of Software Technology tugraz Experimental Results II ◮ Symbolic execution tioco check for deterministic Timed Action Systems ◮ Implemented in Scala calling SMT solver Z3 ◮ Car alarm system characteristics: deterministic, ◮ 5 clock variables, 16 locations, 25 transitions. ◮ 8 mutation operators → 986 mutants ◮ Overall runtime: 27.5 minutes ( k = 12) Depth Bounded Model Checking Symbolic Execution Mean Median Max Min Mean Median Max Min 12 1 . 4 s 1 . 1 s 33 s 0 . 07 s 1 . 7 s 0 . 02 s 38 . 83 s ∼ 0 s Runtime details B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 22 / 64

  25. Institute of Software Technology tugraz Experimental Results II ◮ Symbolic execution tioco check for deterministic Timed Action Systems ◮ Implemented in Scala calling SMT solver Z3 ◮ Car alarm system characteristics: deterministic, ◮ 5 clock variables, 16 locations, 25 transitions. ◮ 8 mutation operators → 986 mutants ◮ Overall runtime: 27.5 minutes ( k = 12) Depth Bounded Model Checking Symbolic Execution Mean Median Max Min Mean Median Max Min 12 1 . 4 s 1 . 1 s 33 s 0 . 07 s 1 . 7 s 0 . 02 s 38 . 83 s ∼ 0 s Runtime details B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 22 / 64

  26. Institute of Software Technology tugraz Experimental Results III g ≤ 0 unlock? e ≤ 300 ◮ Symbolic tioco checker also for g := 0 e ≤ 30 e < 30 e == 30 non-deterministic models unlock? soundOff! g := 0 soundOn! flashOn! ◮ Car Alarm System: silent transition e ≤ 0 e ≤ 0 with non-deterministic delay flashOn! soundOn! e == 300 ◮ Plus underspecification in switching soundOff! soundOff! e ≤ 0 on alarm e == 0 armedOff! ◮ 3 equivalent mutants timed out after e ≤ 0 open? 10min e := 0 g ≤ 0 e ≤ 300 d ≤ 0 f ≤ 0 unlock? d := 0 armedOn! c == 20 armedOn! c ≤ 20 0 < c < 2 close? armedOff! flashOff! f := 0 ǫ close? lock? c := 0 c := 0 open? unlock? flashOff! unlock? open? lock? close? unlock? start B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 23 / 64

  27. Institute of Software Technology tugraz Experimental Results III g ≤ 0 unlock? e ≤ 300 ◮ Symbolic tioco checker also for g := 0 e ≤ 30 e < 30 e == 30 non-deterministic models unlock? soundOff! g := 0 soundOn! flashOn! ◮ Car Alarm System: silent transition e ≤ 0 e ≤ 0 with non-deterministic delay flashOn! soundOn! e == 300 ◮ Plus underspecification in switching soundOff! soundOff! e ≤ 0 on alarm e == 0 armedOff! ◮ 3 equivalent mutants timed out after e ≤ 0 open? 10min e := 0 g ≤ 0 e ≤ 300 d ≤ 0 f ≤ 0 unlock? d := 0 armedOn! c == 20 armedOn! c ≤ 20 0 < c < 2 close? armedOff! flashOff! f := 0 ǫ close? lock? Depth Symbolic Execution c := 0 c := 0 Mean Median Max Min open? unlock? 12 0 . 79 s 0 . 06 s 360 . 84 s ∼ 0 s flashOff! unlock? open? lock? close? unlock? start B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 23 / 64

  28. Institute of Software Technology tugraz Experimental Results III g ≤ 0 unlock? e ≤ 300 ◮ Symbolic tioco checker also for g := 0 e ≤ 30 e < 30 e == 30 non-deterministic models unlock? soundOff! g := 0 soundOn! flashOn! ◮ Car Alarm System: silent transition e ≤ 0 e ≤ 0 with non-deterministic delay flashOn! soundOn! e == 300 ◮ Plus underspecification in switching soundOff! soundOff! e ≤ 0 on alarm e == 0 armedOff! ◮ 3 equivalent mutants timed out after e ≤ 0 open? 10min e := 0 g ≤ 0 e ≤ 300 d ≤ 0 f ≤ 0 unlock? d := 0 armedOn! c == 20 armedOn! c ≤ 20 0 < c < 2 close? armedOff! flashOff! f := 0 ǫ close? lock? Depth Symbolic Execution c := 0 c := 0 Mean Median Max Min open? unlock? 12 0 . 79 s 0 . 06 s 360 . 84 s ∼ 0 s flashOff! unlock? open? lock? close? ... and the bounded model checking? unlock? start B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 23 / 64

  29. Institute of Software Technology tugraz Bounded Determinisation of Timed Automata start coffee BREWING IDLE x = 1 refund q 3 q 0 x < 4 ǫ coin 1 < x < 2 q 4 { x } { x } beep EMPTY 0 < x < 3 beep q 2 q 1 x = 2 GRAINING HEATING B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 24 / 64

  30. Institute of Software Technology tugraz Bounded Determinisation of Timed Automata q 0 start coin start { x 1 } coffee BREWING IDLE x = 1 beep beep q 1 0 < x 1 < 3 x 1 = 2 refund q 3 q 0 { x 2 } { x 2 } x < 4 unfolding ǫ coin q 2 q 4 1 < x < 2 q 4 { x } ǫ refund { x } beep 1 < x 1 < 2 EMPTY x 1 < 4 0 < x < 3 { x 2 , 0 } { x 3 } beep q 2 q 1 q 3 q 5 x = 2 coffee GRAINING HEATING x 2 , 0 = 1 { x 3 } q 6 B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 24 / 64

  31. Institute of Software Technology tugraz Bounded Determinisation of Timed Automata q 0 start coin start { x 1 } coffee BREWING IDLE x = 1 beep beep q 1 0 < x 1 < 3 x 1 = 2 refund q 3 q 0 { x 2 } { x 2 } x < 4 unfolding ǫ coin q 2 q 4 1 < x < 2 q 4 { x } ǫ refund { x } beep 1 < x 1 < 2 EMPTY x 1 < 4 0 < x < 3 { x 2 , 0 } { x 3 } beep q 2 q 1 q 3 q 5 x = 2 coffee GRAINING HEATING x 2 , 0 = 1 { x 3 } q 6 ǫ -removal start q 0 coin beep { x 1 } beep 0 < x 1 < 3 x 1 = 2 q 1 ∧ x 1 < 2 { x 2 } { x 2 } q 3 q 4 coffee refund 2 < x 1 < 3 x 1 < 4 ∧ 1 < x 1 { x 3 } { x 3 } q 6 q 5 B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 24 / 64

  32. Institute of Software Technology tugraz Bounded Determinisation of Timed Automata q 0 start coin start { x 1 } coffee BREWING IDLE x = 1 beep beep q 1 0 < x 1 < 3 x 1 = 2 refund q 3 q 0 { x 2 } { x 2 } x < 4 unfolding ǫ coin q 2 q 4 1 < x < 2 q 4 { x } ǫ refund { x } beep 1 < x 1 < 2 EMPTY x 1 < 4 0 < x < 3 { x 2 , 0 } { x 3 } beep q 2 q 1 q 3 q 5 x = 2 coffee GRAINING HEATING x 2 , 0 = 1 { x 3 } q 6 ǫ -removal start q 0 start q 0 coin { x 1 } coin beep { x 1 } beep q 1 beep 0 < x 1 < 3 x 1 = 2 q 1 ( 0 < x 1 < 3 ∧ ∧ x 1 < 2 { x 2 } x 1 < 2 ) ∨ { x 2 } x 1 = 2 ∨ determinisation 0 < x 1 < 3 q 3 q 4 { x 2 } coffee coffee q ¬ acc refund 2 < x 1 < 3 2 < x 1 < 3 ∧ refund x 1 < 4 ∧ 1 < x 1 1 < x 1 ∧ x 1 < 4 ∧ { x 3 } { x 3 } 0 < x 1 − x 2 < 3 ∧ x 1 − x 2 = 2 q 6 q 5 x 1 − x 2 < 2 { x 3 } q 6 q 5 { x 3 } B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 24 / 64

  33. Institute of Software Technology tugraz Experimental Results IV ◮ Bounded determinization g ≤ 0 unlock? e ≤ 300 g := 0 e ≤ 30 → 13,545 locations (depth 12) e < 30 e == 30 unlock? soundOff! g := 0 soundOn! flashOn! → bounded model check fails e ≤ 0 e ≤ 0 flashOn! soundOn! ◮ Partial models! e == 300 soundOff! soundOff! e ≤ 0 e == 0 armedOff! e ≤ 0 c = 20 open? armedOn! e := 0 g ≤ 0 e ≤ 300 d ≤ 0 unlock? f ≤ 0 d := 0 armedOn! c == 20 0 < c < 2 armedOn! ǫ c ≤ 20 open? unlock? close? lock? { c } { c } ) 0 < c < 2 close? armedOff! flashOff! ǫ f := 0 close? c := 0 open? unlock? unlock? open? flashOff! lock? close? lock? unlock? start start Model D. Bounded Model Checking Symbolic Execution Mean Median Max Min Mean Median Max Min Partial 1 8 9 . 7 s 8 . 0 s 85 . 1 s 0 . 3 s 0 . 28 s 0 . 04 s 16 . 78 s ∼ 0 s Partial 2 12 1 . 6 s 1 . 63 s 37 . 3 s 0 . 08 s 0 . 08 s 0 . 03 s 2 . 28 s ∼ 0 s Complete 12 x x x x 0 . 79 s 0 . 06 s 360 . 84 s ∼ 0 s B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 25 / 64

  34. Institute of Software Technology tugraz Experimental Results IV ◮ Bounded determinization g ≤ 0 unlock? e ≤ 300 g := 0 e ≤ 30 → 13,545 locations (depth 12) e < 30 e == 30 unlock? soundOff! g := 0 soundOn! flashOn! → bounded model check fails e ≤ 0 e ≤ 0 flashOn! soundOn! ◮ Partial models! e == 300 soundOff! soundOff! e ≤ 0 e == 0 armedOff! e ≤ 0 c = 20 open? armedOn! e := 0 g ≤ 0 e ≤ 300 d ≤ 0 unlock? f ≤ 0 d := 0 armedOn! c == 20 0 < c < 2 armedOn! ǫ c ≤ 20 open? unlock? close? lock? { c } { c } ) 0 < c < 2 close? armedOff! flashOff! ǫ f := 0 close? c := 0 open? unlock? unlock? open? flashOff! lock? close? lock? unlock? start start Model D. Bounded Model Checking Symbolic Execution Mean Median Max Min Mean Median Max Min Partial 1 8 9 . 7 s 8 . 0 s 85 . 1 s 0 . 3 s 0 . 28 s 0 . 04 s 16 . 78 s ∼ 0 s Partial 2 12 1 . 6 s 1 . 63 s 37 . 3 s 0 . 08 s 0 . 08 s 0 . 03 s 2 . 28 s ∼ 0 s Complete 12 x x x x 0 . 79 s 0 . 06 s 360 . 84 s ∼ 0 s B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 25 / 64

  35. Institute of Software Technology tugraz Experimental Results IV ◮ Bounded determinization g ≤ 0 unlock? e ≤ 300 g := 0 e ≤ 30 → 13,545 locations (depth 12) e < 30 e == 30 unlock? soundOff! g := 0 soundOn! flashOn! → bounded model check fails e ≤ 0 e ≤ 0 flashOn! soundOn! ◮ Partial models! e == 300 soundOff! soundOff! e ≤ 0 e == 0 armedOff! e ≤ 0 c = 20 open? armedOn! e := 0 g ≤ 0 e ≤ 300 d ≤ 0 unlock? f ≤ 0 d := 0 armedOn! c == 20 0 < c < 2 armedOn! ǫ c ≤ 20 open? unlock? close? lock? { c } { c } ) 0 < c < 2 close? armedOff! flashOff! ǫ f := 0 close? c := 0 open? unlock? unlock? open? flashOff! lock? close? lock? unlock? start start Model D. Bounded Model Checking Symbolic Execution Mean Median Max Min Mean Median Max Min Partial 1 8 9 . 7 s 8 . 0 s 85 . 1 s 0 . 3 s 0 . 28 s 0 . 04 s 16 . 78 s ∼ 0 s Partial 2 12 1 . 6 s 1 . 63 s 37 . 3 s 0 . 08 s 0 . 08 s 0 . 03 s 2 . 28 s ∼ 0 s Complete 12 x x x x 0 . 79 s 0 . 06 s 360 . 84 s ∼ 0 s B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 25 / 64

  36. Institute of Software Technology tugraz Experimental Results V ◮ Adding data variable and parameters to ◮ deterministic Car Alarm System with one clock ◮ 3-digit PIN code for unlocking ◮ No negative effects, even with higher digit PIN codes ◮ Symbolic execution faster with 1 clock (0 . 24 s ) than with 5 clocks (1 . 7 s ) Depth Bounded Model Checking Symbolic Execution Mean Median Max Min Mean Median Max Min 8 1 . 46 s 0 . 28 s 59 . 41 s 0 . 12 s 0 . 07 s 0 . 05 s 0 . 82 s ∼ 0 s 12 4 . 12 s 0 . 35 s 35 . 41 s 0 . 13 s 0 . 24 s 0 . 05 s 3 . 67 s ∼ 0 s B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 26 / 64

  37. Institute of Software Technology tugraz Experimental Results V ◮ Adding data variable and parameters to ◮ deterministic Car Alarm System with one clock ◮ 3-digit PIN code for unlocking ◮ No negative effects, even with higher digit PIN codes ◮ Symbolic execution faster with 1 clock (0 . 24 s ) than with 5 clocks (1 . 7 s ) Depth Bounded Model Checking Symbolic Execution Mean Median Max Min Mean Median Max Min 8 1 . 46 s 0 . 28 s 59 . 41 s 0 . 12 s 0 . 07 s 0 . 05 s 0 . 82 s ∼ 0 s 12 4 . 12 s 0 . 35 s 35 . 41 s 0 . 13 s 0 . 24 s 0 . 05 s 3 . 67 s ∼ 0 s B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 26 / 64

  38. Institute of Software Technology tugraz Real-Time Systems Summary Symbolic execution (SE) seems to perform better, but no clear winner! ◮ Number of clocks: ◮ BMC: small impact (was faster in deterministic case) ◮ SE: high impact ◮ Non-determinism: is an obstacle for conformance checking ◮ BMC: state-space explosion → partial models ◮ SE: lowered performance (40 s vs. 6 min ) → 3 mutants timed out ◮ Statistical outliers: due to equivalent mutants ◮ BMC: runtime almost equal ◮ SE: extreme differences due to optimisations B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 27 / 64

  39. Institute of Software Technology tugraz Agenda ◮ Model-based Mutation Testing ◮ Real-Time Systems ◮ Hybrid Systems ◮ Discrete Systems B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 28 / 64

  40. Institute of Software Technology tugraz A Hybrid System: Two Tank System P2 G2 Requirements: out ◮ P1 starts pumping, if T2 below Full Reserve and T1 is full T2 Reserve ◮ until T1 is empty or T2 is full inout Empty ◮ P2 is controlled by button WaterRequest P1 G1 ◮ runs if there is water in T2. in ◮ Note: T1 may overflow Full T1 Empty P1, P2 . . . water pumps G1, G2 . . . water-level sensors B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 29 / 64

  41. Institute of Software Technology tugraz Related Work ◮ Hybrid Systems ◮ Hybrid Automata (Alur,Courcoubetis,Henzinger,Ho 93) ◮ Action Systems [Back,Kurki-Suonio 83] ◮ Hybrid Action Systems [Rönkkö,Ravn,Sere 03] ◮ Qualitative Reasoning [Kuipers 94] ◮ Testing ◮ Mutation Testing [Hamlet 77, De Millo et al. 78] ◮ Input-Output Conformance [Brinksma,Tretmans 92] B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 30 / 64

  42. Institute of Software Technology tugraz Abstraction 1: Action Systems Modeling the Controller Controller: Guards: ◮ g 1 = x 2 ≤ Reserve ∧ df | [ var P1_running , P2_running : Bool , x 1 = Full ∧ out* , inout* : Real ¬ P1_running • P1_running := false ; ◮ g 2 = P1_running ∧ df P2_running := false ; ( x 1 ≤ Empty ∨ x 2 = Full ) out := 0 ; inout := 0 ; do ◮ g 1 → P1_running := true ; inout := ( 0 , Max ] g 3 = WaterRequest ∧ df � ¬ P2_running ∧ g 2 → P1_running := false ; inout := 0 x 2 > Reserve � g 3 → P2_running := true ; out := ( 0 , Max ] ◮ g 4 = P2_running ∧ � df ( ¬ WaterRequest ∨ g 4 → P2_running := false ; out := 0 x 2 = Empty ) od ] | : WaterRequest , x 1 , x 2 B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 31 / 64

  43. Institute of Software Technology tugraz Abstraction 1: Action Systems Modeling the Controller Controller: Guards: ◮ g 1 = x 2 ≤ Reserve ∧ df | [ var P1_running , P2_running : Bool , x 1 = Full ∧ out* , inout* : Real ¬ P1_running • P1_running := false ; ◮ g 2 = P1_running ∧ df P2_running := false ; ( x 1 ≤ Empty ∨ x 2 = Full ) out := 0 ; inout := 0 ; do ◮ g 1 → P1_running := true ; inout := ( 0 , Max ] g 3 = WaterRequest ∧ df � ¬ P2_running ∧ g 2 → P1_running := false ; inout := 0 x 2 > Reserve � g 3 → P2_running := true ; out := ( 0 , Max ] ◮ g 4 = P2_running ∧ � df ( ¬ WaterRequest ∨ g 4 → P2_running := false ; out := 0 x 2 = Empty ) od ] | : WaterRequest , x 1 , x 2 B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 31 / 64

  44. Institute of Software Technology tugraz Why Action Systems? ◮ Well-suited for embedded systems modeling ◮ Action view maps naturally to LTS testing theories ◮ Solid foundation: ◮ precise semantics ◮ refinement ◮ Compositional modeling ◮ Many extensions available: ◮ object-orientation ◮ hybrid systems B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 32 / 64

  45. Institute of Software Technology tugraz Hybrid Action Systems Environment: | [ var x 1 * , x 2 * : Real • x 1 := 0 ; x 2 := 0 alt g 1 → . . . � . . . with ¬ ( g 1 ∨ . . . ) : → ˙ x 1 = ( in − inout ) / A 1 ∧ ˙ x 2 = ( inout − out ) / A 2 ] | : inout , out ◮ Rönkkö, M., Ravn, A.P., Sere, K.: Hybrid action systems. Theoretical Computer Science 290 (2003) 937–973. B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 33 / 64

  46. Institute of Software Technology tugraz Abstraction 2: Qualitative Flows v-abs.f.t f.t max high med t zero t-abs.f.t 2 2 t g.s 1 1 0 0 B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 34 / 64

  47. Institute of Software Technology tugraz Example Qualitative Flow of Water Tanks x 2 x 1 Full Full x 2 x 1 Reserve Empty Empty t P1 P2 P1/P2 OFF P1 ON OFF ON Full x 1 Empty Zero Full Reserve x 2 Empty Zero B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 35 / 64

  48. Institute of Software Technology tugraz Qualitative Reasoning (QR) ◮ QR originates from Artificial Intelligence ◮ Common sense reasoning about physical systems with possibly incomplete knowledge. ◮ Ordinary Differential Equations (ODE) → Qualitative Differential Equations (QDE): x 1 = ( in − inout ) / A 1 ˙ → d / dt ( x 1 , diff 1 ) ∧ add ( diff 1 , inout , in ) ◮ Arithmetic is reduced to sign algebra: 5 − 1 = 4 → [+] + [ − ] = [+] | [ − ] − 3 ∗ 2 = − 6 → [ − ] ∗ [+] = [ − ] B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 36 / 64

  49. Institute of Software Technology tugraz Qualitative Action Systems | [ var x 1 * , x 2 * : Real • x 1 := 0 ; x 2 := 0 alt g 1 → . . . � . . . with ¬ ( g 1 ∨ . . . ) : → d / dt ( x 1 , diff 1 ) ∧ d / dt ( x 2 , diff 2 ) ∧ add ( diff 2 , out , inout ) ∧ add ( diff 1 , inout , in ) ] | : inout , out B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 37 / 64

  50. Institute of Software Technology tugraz Qualitative Simulation ◮ Implementations: ◮ QSIM (Lisp) ◮ Garp3 (SWI-Prolog) ◮ ASIM (GNU-Prolog) B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 38 / 64

  51. Institute of Software Technology tugraz Model-based Mutation Testing Action System Model IOLTS S discriminating test case ioco? Mutants for every mutant IOLTS M ioco . . . input-output conformance B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 39 / 64

  52. Institute of Software Technology tugraz Model-based Mutation Testing Action System Model IOLTS S discriminating test case ioco? Mutants for every mutant IOLTS M ioco . . . input-output conformance B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 39 / 64

  53. Institute of Software Technology tugraz Model-based Mutation Testing Action System Model IOLTS S discriminating test case ioco? Mutants for every mutant IOLTS M ioco . . . input-output conformance B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 39 / 64

  54. Institute of Software Technology tugraz Model-based Mutation Testing Action System Model IOLTS S discriminating test case ioco? Mutants for every mutant IOLTS M ioco . . . input-output conformance B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 39 / 64

  55. Institute of Software Technology tugraz Conformance Checking ◮ Event-view: labeled actions ◮ Input and Output Labels Def. IOCO [Tretmans 96] ∀ σ ∈ Straces ( Model ) : out ( Mutant after σ ) ⊆ out ( Model after σ ) out ... outputs labels + quiescence after ... reachable states after trace ◮ ioco supports: partial, non-deterministic models ◮ ioco-checker Ulysses ◮ implemented in GNU Prolog ◮ explores discrete actions + qualitative flows ◮ builds synchronous product modulo ioco ◮ highly non-deterministic → on-the-fly determinization B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 40 / 64

  56. Institute of Software Technology tugraz Conformance Checking ◮ Event-view: labeled actions ◮ Input and Output Labels Def. IOCO [Tretmans 96] ∀ σ ∈ Straces ( Model ) : out ( Mutant after σ ) ⊆ out ( Model after σ ) out ... outputs labels + quiescence after ... reachable states after trace ◮ ioco supports: partial, non-deterministic models ◮ ioco-checker Ulysses ◮ implemented in GNU Prolog ◮ explores discrete actions + qualitative flows ◮ builds synchronous product modulo ioco ◮ highly non-deterministic → on-the-fly determinization B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 40 / 64

  57. Institute of Software Technology tugraz Generating a Testcase: Original Model System = | [ x 1 : T 1 , x 2 : T 2 , out , inout : FR , var diff 1 , diff 2 : NZP , p1_running , p2_running , wr : Bool • x 1 := ( 0 , 0 ); x 2 := ( 0 , 0 ); out := ( 0 , 0 ); inout := ( 0 , 0 ); wr := false p1_running := false ; p2_running := false obs pump1_on : g 1 → p1_running := true ; alt inout := ( 0 ..Max , 0 ) � obs pump1_off : g 2 → p1_running := false ; inout := ( 0 , 0 ) � obs pump2_on : g 3 → p2_running := true ; out := ( 0 ..Max , 0 ) � obs pump2_off : g 4 → p2_running := false ; out := ( 0 , 0 ) � ctr water_req(X) : g 5 → wr := X with ¬ ( g 1 ∨ g 2 ∨ g 3 ∨ g 4 ∨ g 5 ) : ⇁ add ( diff 2 , out , inout ) ∧ add ( diff 1 , inout , in ) ∧ d/dt ( x 1 , diff 1 ) ∧ d/dt ( x 2 , diff 2 ) ] | : in B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 41 / 64

  58. Institute of Software Technology tugraz Generating a Testcase II: Mutated Model System = | [ x 1 : T 1 , x 2 : T 2 , out , inout : FR , var diff 1 , diff 2 : NZP , p1_running , p2_running , wr : Bool • x 1 := ( 0 , 0 ); x 2 := ( 0 , 0 ); out := ( 0 , 0 ); inout := ( 0 , 0 ); wr := false p1_running := false ; p2_running := false obs pump1_on : g 1 → p1_running := true ; alt inout := ( 0 ..Max , 0 ) � obs pump1_off : g 2 → p1_running := true ; inout := ( 0 , 0 ) � obs pump2_on : g 3 → p2_running := true ; out := ( 0 ..Max , 0 ) � obs pump2_off : g 4 → p2_running := false ; out := ( 0 , 0 ) � ctr water_req(X) : g 5 → wr := X with ¬ ( g 1 ∨ g 2 ∨ g 3 ∨ g 4 ∨ g 5 ) : ⇁ add ( diff 2 , out , inout ) ∧ add ( diff 1 , inout , in ) ∧ d/dt ( x 1 , diff 1 ) ∧ d/dt ( x 2 , diff 2 ) ] | : in B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 42 / 64

  59. Institute of Software Technology tugraz Generating a Testcase III: Product Graph Part of the result of the conformance check 0 between the original and the mutated specification. obs qual([x1:full/inc,x2:zero/std]) 1 obs pump1_on 2 obs qual([x1:empty..full/inc,x2:full/inc]) obs qual([x1: ...,x2: ...]) 26 ... ctr water_req(1) obs pump1_off ... 27 34 obs pump2_on obs pump1_off ctr water_req(1) obs delta obs pump1_off 28 32 35 (pass) 36 (fail) obs pump1_off obs pump2_on obs pump1_off 29 33 (fail) obs qual([x1:empty..full/inc,x2:empty..reserve/dec]) obs pump1_off 30 (pass) 31 (fail) B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 43 / 64

  60. Institute of Software Technology tugraz Results Mut. No. Avg.Time Average No. = � = Op. Mutants [s] States Trans. No. Perc. ASO 10 13.9 64 117 7 3 30% ENO 6 7.6 68 120 5 1 17% ERO 20 12.9 62 110 20 0 0% LRO 13 12.8 93 168 9 4 31% MCO 16 12.8 70 126 10 6 38% RRO 12 12.0 40 73 10 2 17% Total 77 12.0 66 119 61 16 21% ASO ... Association Shift Operator LRO ... Logical Operator Replacement ENO ... Expression Negation Operator MCO ... Missing Condition Operator ERO ... Event Replacement Operator RRO ... Relational Replacement Operator B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 44 / 64

  61. Institute of Software Technology tugraz Generating a Testcase IV: Linear TC Selecting one path 0 for each unsafe state leading to failure. obs qual([x1:full/inc,x2:zero/std]) 1 obs out_pump1_on 2 obs qual([x1:empty..full/inc,x2:full/inc]) obs qual([x1: ...,x2: ...]) 98 ... (inconcl) obs out_pump1_off 99 ctr in_water_req(1) 102 obs out_pump2_on 104 obs qual([x1:empty..full/inc,x2:empty..reserve/dec]) 106 (pass) B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 45 / 64

  62. Institute of Software Technology tugraz Generating a Testcase V: Adaptive TC A test graph including all paths to a given unsafe state leading to failure. obs pa ss 13 obs out_pum p1_on 10 11 12 obs out_pum p2_off obs out_pum p1_off obs de lta 6 obs out_pum p1_on ctr in_wa te r_re q(1) ctr in_wa te r_re q(0) 8 9 obs inconc obs de lta 4 5 obs out_pum p2_on obs out_pum p1_off ctr in_wa te r_re q(1) 7 2 3 obs de lta obs out_pum p1_on obs out_pum p1_off 1 0 obs out_pum p1_on Qualitative events are internal (not visible). B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 46 / 64

  63. Institute of Software Technology tugraz Hybrid Systems Summary ◮ AI meets FM: qualitative reasoning ◮ Requirements → incomplete qualitative models ◮ Model exploration: controller (discrete) + environment (qualitative) ◮ TCG based on mutation testing and ioco conformance checking ◮ Different strategies for selecting test case B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 47 / 64

  64. Institute of Software Technology tugraz Agenda ◮ Model-based Mutation Testing ◮ Real-Time Systems ◮ Hybrid Systems ◮ Discrete Systems B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 48 / 64

  65. Institute of Software Technology tugraz Discrete Systems: MoMuT::UML Applications: ◮ Car Alarm System (Ford) ◮ Railway Interlocking System (Thales) ◮ Automotive Meassurement Device: Particle Counter (AVL) B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 49 / 64

  66. Institute of Software Technology tugraz SUT: AVL489 Particle Counter ◮ One of AVL’s automotive measurement devices ◮ Measures particle number concentrations in exhaust gas ◮ Focus: testing of the control logic ◮ AVL uses virtual test-beds with simulated devices for integration and regression testing. ◮ We tested a simulation of the particle counter: ◮ Matlab Simulink model compiled to real-time executable ◮ Same interface as real device! B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 50 / 64

  67. Institute of Software Technology tugraz SUT: AVL489 Particle Counter ◮ One of AVL’s automotive measurement devices ◮ Measures particle number concentrations in exhaust gas ◮ Focus: testing of the control logic ◮ AVL uses virtual test-beds with simulated devices for integration and regression testing. ◮ We tested a simulation of the particle counter: ◮ Matlab Simulink model compiled to real-time executable ◮ Same interface as real device! B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 50 / 64

  68. Institute of Software Technology tugraz SUT: AVL489 Particle Counter ◮ One of AVL’s automotive measurement devices ◮ Measures particle number concentrations in exhaust gas ◮ Focus: testing of the control logic ◮ AVL uses virtual test-beds with simulated devices for integration and regression testing. ◮ We tested a simulation of the particle counter: ◮ Matlab Simulink model compiled to real-time executable ◮ Same interface as real device! B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 50 / 64

  69. Institute of Software Technology tugraz UML Test Model of AVL489 LeakageTest, ResponseCheck, SetPurge, SetZeroPoint, StopIntegralMeasurement, SetStandby, StartMeasurement, StartIntegralMeasurement, SetPause, DilutionSelection AVL489 Active isBusy SetPurge [ not Busy and not Manual ] Pause_0 Purging_Pause_12 30 [ not (oclIsInState(Active::Response_14) or oclIsInState(Active::Purging_Standby_12) send SPUL_state /entry SetPause [ not Busy and not Manual ] 10 when Busy or oclIsInState(Active::Leakage_11) or oclIsInState(Active::ZeroGas_10) Integral_9 or oclIsInState(Active::Purging_Pause_12)) ] / set not Busy - send StatusReady send SPAU state /entry SetPause [ not Busy and not Manual ] send SINT_state /entry send StatusBusy; set Busy /exit DilutionSelection [ not Manual and not Busy ] / set Dilution send StatusBusy; set Busy /exit SetStandby [ not Busy and not Manual ] SetPause [ not Busy and not Manual ] StartIntegralMeasurement, StopIntegralMeasurement, StartMeasurement [ not Busy and not Manual ] StartIntegralMeasurement [ not Busy and not Manual ] LeakageTest, ResponseCheck [ not (oclIsInState(Standby_1 Measurement_2 StartMeasurement [ not Busy and not Manual ] send SMGA_state /entry SetPurge [ not (oclIsInState(Pause_0) or oclIsInState(Standby send StatusBusy; set Busy /exit SetZeroPoint [ not Busy and not Manual ] 10 SetZeroPoint [ not oclIsInState(Active::Measurement_2) and ZeroGas_10 SetStandby [ not Busy and not Manual ] send SNGA_state /entry send StatusBusy; set Busy /exit isReady Standby_1 StopIntegralMeasurement [ not oclIsInState(Active::Integral_ LeakageTest [ not Busy and not Manual ] Leakage_11 send SLEC_state /entry 10 StartMeasurement [ not (oclIsInState(Standby_1) or oclIsInState SetPurge [ not Busy and not Manual ] Purging_Standby_12 send SPUL_state /entry 10 StartIntegralMeasurement [ not (oclIsInState(Active::Measurement_2) ResponseCheck [ not Busy and not Manual ] Response_14 10 send SEGA_state /entry SetStandby [ not Busy and not Manual ] send STBY_state /entry send StatusBusy; set Busy /exit SetManual SetRemote / send Offline SetRemote / send Online Manual Remote SetManual / send Offline set Manual /entry unset Manual /entry DilutionSelection, LeakageTest, ResponseCheck, SetPurge, SetZeroPoint, StopIntegralMeasurement, SetPause, SetStandby, StartMeasurement, StartIntegralMeasurement / send RejectOF B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 51 / 64

  70. Institute of Software Technology tugraz MoMuT::UML ◮ Test-case generator of AIT and TU Graz ◮ Implementing model-based mutation testing for UML state machines MoMuT::UML( Java ! backend ! ( ( frontend ! ( Enumera3ve(TCG( ( Prolog ! UML2OOAS( OOAS2AS( ( AlarmSystem_StateMachine OpenAndUnlocked Unlock Alarm Open Close Lock Unlock Activate Alarms /entry Deactivate Alarms /exit ClosedAndUnlocked OpenAndLocked Java ! Java ! ( FlashAndSound Unlock Lock Close Open 30 / Deactivate Sound Symbolic(TCG( ClosedAndLocked Flash 20 Close ( Armed SilentAndOpen Unlock 300 Show Armed /entry Show Unarmed /exit Open Prolog ! abstract(test(cases( UML(model( ( ! Aldebaran aut format ! Papyrus MDT/ ! Visual Paradigm ! SMT!Solver! Z3! Architecture of the MoMuT::UML tool chain AS ... Action Systems [Back83] OOAS ... Object-Oriented Action Systems B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 52 / 64

  71. Institute of Software Technology tugraz Abstract Test Case of AVL489 Abstract test cases → concrete C# obs StatusReady(0) NUnit test cases. obs SPAU_state(0) obs Offline(0) ctr SetStandby(0) obs StatusBusy(0) obs STBY_state(0) obs Online(0) obs StatusReady(30) ctr StartMeasurement(0) obs StatusBusy(0) obs SMGA_state(0) obs StatusReady(30) ctr StartIntegralMeasurement(0) obs SINT_state(0) ctr SetStandby(0) ctr ... controllable event (input) obs STBY_state(0) obs ... observable event (output) pass B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 53 / 64

  72. Institute of Software Technology tugraz Test Execution on Particle Counter We found several bugs in the SUT: ◮ Forbidden changes of operating state while busy ◮ Pause → Standby ◮ Normal Measurement → Integral Measurement ◮ Ignoring high-frequent input without error-messages ◮ Loss of error messages in client for remote control of the device B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 54 / 64

  73. Institute of Software Technology tugraz Refinement + ioco Conformance Checking Refinement: Input-Output Conformance: ◮ state-based ◮ event-based ◮ predicative semantics ◮ io labelled transition systems Def. Refinement [Hoare & He 98] Def. IOCO [Tretmans 96] ∀ s , s ′ : Mutant ( s , s ′ ) ⇒ Model ( s , s ′ ) ∀ σ ∈ traces ( Model ) : out ( Mutant after σ ) ⊆ out ( Model after σ ) s ... state before s’ ... state after execution out ... outputs labels + quiescence after ... reachable states after trace New combined conformance checking: ◮ Refinement checker searches for faulty state (fast) ◮ Ioco checker looks if faulty state propagates to different observations B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 55 / 64

  74. Institute of Software Technology tugraz Refinement + ioco Conformance Checking Refinement: Input-Output Conformance: ◮ state-based ◮ event-based ◮ predicative semantics ◮ io labelled transition systems Def. Refinement [Hoare & He 98] Def. IOCO [Tretmans 96] ∀ s , s ′ : Mutant ( s , s ′ ) ⇒ Model ( s , s ′ ) ∀ σ ∈ traces ( Model ) : out ( Mutant after σ ) ⊆ out ( Model after σ ) s ... state before s’ ... state after execution out ... outputs labels + quiescence after ... reachable states after trace New combined conformance checking: ◮ Refinement checker searches for faulty state (fast) ◮ Ioco checker looks if faulty state propagates to different observations B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 55 / 64

  75. Institute of Software Technology tugraz Refinement + ioco Conformance Checking Refinement: Input-Output Conformance: ◮ state-based ◮ event-based ◮ predicative semantics ◮ io labelled transition systems Def. Refinement [Hoare & He 98] Def. IOCO [Tretmans 96] ∀ s , s ′ : Mutant ( s , s ′ ) ⇒ Model ( s , s ′ ) ∀ σ ∈ traces ( Model ) : out ( Mutant after σ ) ⊆ out ( Model after σ ) s ... state before s’ ... state after execution out ... outputs labels + quiescence after ... reachable states after trace New combined conformance checking: ◮ Refinement checker searches for faulty state (fast) ◮ Ioco checker looks if faulty state propagates to different observations B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 55 / 64

  76. Institute of Software Technology tugraz Symbolic Refinement Checking Is non-refinement reachable? ∃ s , s ′ , tr , tr ′ : reachable ( s , tr ) ∧ Mutant ( s , s ′ , tr , tr ′ ) ∧ ¬ Model ( s , s ′ , tr , tr ′ ) s ... state before s’ ... states after execution tr ... trace of labels before tr’ ... trace of labels after execution B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 56 / 64

  77. Institute of Software Technology tugraz TCG Particle Counter stles stles stles stles 68+ stles 111+ stles notYconformingY stles 189+ stles (nonZref.Y&YnotYioco)Y stles stles stles uniqueYTCsY conformingY stles stles (refining)Y stles duplicateYTCsY stles stles conformingY stles stles (nonZref.,YbutYioco)Y stles 928+ stles 817+ stles stles stles stles stles stles (a) Breakup into conforming and stles (b) Breakup into unique and stles not conforming model mutants. duplicate test cases. stles stles 30Y stles 26+ stles 25Y stles unique+test+cases+[#]+ 20Y stles stles 13+ 15Y 12+ 12+ 12+ stles 10+ 9+ stles 10Y stles 4+ 4+ 5Y 2+ 2+ stles 1+ 1+ 1+ 1+ 1+ stles 0Y stles 1Y 2Y 3Y 4Y 5Y 6Y 7Y 8Y 9Y 10Y 11Y 12Y 13Y 14Y 15Y 16Y stles length+ (c) Lengths of the unique test cases. B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 57 / 64

  78. Institute of Software Technology tugraz Fault Propagation 452' 5001 423' 4001 mutants'[#]' 3001 2001 1001 44' 6' 3' 01 11 21 31 41 51 ioco'depth' Figure: Number of steps from fault to failure (ioco depths) B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 58 / 64

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Black Hole Thermodynamics Robert M. Wald I. Black Holes; Event Horizons and Killing Horizons II.

Black Hole Thermodynamics Robert M. Wald I. Black Holes; Event Horizons and Killing Horizons II.

Black Hole Thermodynamics Robert M. Wald I. Black Holes; Event Horizons and Killing Horizons II. The First Law of Black Hole Mechanics and Black Hole Entropy III. Dynamic and Thermodynamic Stability of Black Holes IV. Quantum Aspects of Black

1.74k views • 116 slides
Black holes with a single Killing vector field Jorge E. Santos Cambridge University - DAMTP

Black holes with a single Killing vector field Jorge E. Santos Cambridge University - DAMTP

Black holes with a single Killing vector field Black holes with a single Killing vector field Jorge E. Santos Cambridge University - DAMTP Oxford - 23/02/2016 In collaboration with Oscar J. C. Dias, Ben Niehoff and Benson Way 1 / 20

1.05k views • 102 slides
1 Mutation pressure Let = the mutation rate from A a Let = the mutation rate from a

1 Mutation pressure Let = the mutation rate from A a Let = the mutation rate from a

Population Genetics 5: Mutation pressure Mutation pressure Table 1 : Estimates of per generation mutation rates for a range of organisms Organism Per nucleotide rate Genomic rate RNA GENOMES 1.97 10 -5 Poliovirus 0.15 1.10 10 -4

306 views • 7 slides
1 Mutation pressure Let = the mutation rate from A a Let = the mutation rate from a

1 Mutation pressure Let = the mutation rate from A a Let = the mutation rate from a

Population Genetics 5: Mutation pressure Mutation pressure Table 1 : Estimates of per generation mutation rates for a range of organisms Organism Per nucleotide rate Genomic rate RNA GENOMES 1.97 10 -5 Poliovirus 0.15 1.10 10 -4

350 views • 6 slides
CSE 331 Mutation and immutability slides created by Marty Stepp based on materials by M. Ernst,

CSE 331 Mutation and immutability slides created by Marty Stepp based on materials by M. Ernst,

CSE 331 Mutation and immutability slides created by Marty Stepp based on materials by M. Ernst, S. Reges, D. Notkin, R. Mercer, Wikipedia http://www.cs.washington.edu/331/ 1 Mutation mutation : A modification to the state of an object.

571 views • 29 slides
Defect Detection Thomas Zimmermann The First Bug September 9, 1947 More Bugs More Bugs More

Defect Detection Thomas Zimmermann The First Bug September 9, 1947 More Bugs More Bugs More

Defect Detection Thomas Zimmermann The First Bug September 9, 1947 More Bugs More Bugs More Bugs More Bugs More Bugs More Bugs More Bugs More Bugs More Bugs More Bugs More Bugs More Bugs Facts on Debugging Software bugs are

840 views • 63 slides
BINARY BLACK HOLES IN CIRCULAR ORBITS: AN HELICAL KILLING VECTOR APPROACH Eric Gourgoulhon

BINARY BLACK HOLES IN CIRCULAR ORBITS: AN HELICAL KILLING VECTOR APPROACH Eric Gourgoulhon

BINARY BLACK HOLES IN CIRCULAR ORBITS: AN HELICAL KILLING VECTOR APPROACH Eric Gourgoulhon Silvano Bonazzola Philippe Grandcl ement Phys. Rev. D. 65 , 044020 (2002). Phys. Rev. D. 65 , 044021 (2002). 3+1

667 views • 28 slides
Automating the Generation of Mutation-based Test Cases Mik ike e Pap apad adakis akis

Automating the Generation of Mutation-based Test Cases Mik ike e Pap apad adakis akis

Automating the Generation of Mutation-based Test Cases Mik ike e Pap apad adakis akis Department of Informatics Athens University of Economics and Business 1 Mutation Testing White-box, fault-based testing technique Considered as

540 views • 22 slides
MODEL-BASED, MUTATION-DRIVEN TEST CASE GENERATION VIA HEURISTIC-GUIDED BRANCHING SEARCH Andreas

MODEL-BASED, MUTATION-DRIVEN TEST CASE GENERATION VIA HEURISTIC-GUIDED BRANCHING SEARCH Andreas

MODEL-BASED, MUTATION-DRIVEN TEST CASE GENERATION VIA HEURISTIC-GUIDED BRANCHING SEARCH Andreas Fellner FMCAD Student Forum Wien, October 4th 2017 TEST CASE GENERATION WITH 1 Andreas Fellner TEST CASE GENERATION WITH Abstract Model UML /

358 views • 34 slides
Outline Bugs! 1 Avoiding and Finding bugs 2 Bugs still happen 3 Why do bugs still happen ?!

Outline Bugs! 1 Avoiding and Finding bugs 2 Bugs still happen 3 Why do bugs still happen ?!

Failure is not an option * A journey through software bugs Philippe Biondi Nov 20 th 2015 / GreHack Failure is not an option * Outline Bugs! 1 Avoiding and Finding bugs 2 Bugs still happen 3 Why do bugs still happen ?! 4 Living with bugs

966 views • 63 slides
pneumophila softly Killing Legionella pneumophila Killing Legionella pneumophila softly Bakent

pneumophila softly Killing Legionella pneumophila Killing Legionella pneumophila softly Bakent

Killing Legionella pneumophila softly Killing Legionella pneumophila Killing Legionella pneumophila softly Bakent Meds Legionella pneumophila G(-) bacteria Infectious agent of atypic pneumonia; Legionellosis case1study.wikispaces.com

371 views • 36 slides
Black Box Testing (A&amp;O Ch. 4) (2 nd Ed. Ch. 6) Course Software Testing &amp; Verification

Black Box Testing (A&amp;O Ch. 4) (2 nd Ed. Ch. 6) Course Software Testing &amp; Verification

Black Box Testing (A&amp;O Ch. 4) (2 nd Ed. Ch. 6) Course Software Testing &amp; Verification 2019/20 Wishnu Prasetya &amp; Gabriele Keller Plan Partitioning based testing (A&amp;O Ch 4/2 nd Ed Ch. 6). Model based testing Note: black

772 views • 39 slides
1 Mutation - selection equilibrium 1. Mutation pressure: ( ) ( ) = Let = the

1 Mutation - selection equilibrium 1. Mutation pressure: ( ) ( ) = Let = the

Population Genetics 7: transient verses equilibrium polymorphism Mutation - selection equilibrium Mutation pressure and selection can operate in opposite directions as a force for change in allele frequencies in populations. Note that

359 views • 9 slides
Finding Crash-Consistency Bugs with Bounded Black-Box Testing Jayashree Mohan , Ashlie Martinez,

Finding Crash-Consistency Bugs with Bounded Black-Box Testing Jayashree Mohan , Ashlie Martinez,

Finding Crash-Consistency Bugs with Bounded Black-Box Testing Jayashree Mohan , Ashlie Martinez, Soujanya Ponnapalli, Pandian Raju, Vijay Chidambaram Crashes I crashed This is very File saved! important File missing Image

878 views • 87 slides
A Mathematical Model of Cancer Stem Cell Lineage Population Dynamics with Mutation Accumulation

A Mathematical Model of Cancer Stem Cell Lineage Population Dynamics with Mutation Accumulation

Biological Preliminaries The Mathematical Model Discussion A Mathematical Model of Cancer Stem Cell Lineage Population Dynamics with Mutation Accumulation and Telomere Length Hierarchies Georgi Kapitanov Vanderbilt University Feb 17, 2012

337 views • 23 slides
Using Conditional Mutation to Increase the Efficiency of Mutation Analysis e Just 1 &amp; Gregory

Using Conditional Mutation to Increase the Efficiency of Mutation Analysis e Just 1 &amp; Gregory

Using Conditional Mutation to Increase the Efficiency of Mutation Analysis e Just 1 &amp; Gregory M. Kapfhammer 2 &amp; Franz Schweiggert 1 Ren 1 Ulm University, Germany 2 Allegheny College, USA 6th International Workshop on the Automation of

946 views • 73 slides
Morphosyntactic gambits (or: dont assume syntacticians will do your morphological washing)

Morphosyntactic gambits (or: dont assume syntacticians will do your morphological washing)

Morphosyntactic gambits (or: dont assume syntacticians will do your morphological washing) 1 Greville G. Corbett Surrey Morphology Group g.corbett@surrey.ac.uk Outline of the paper 1 The intellectual landscape 2 The issue 3

262 views • 15 slides
FroCoS13 Business Meeting LORIA &amp; INRIA Nancy Grand Est September 19, 2013

FroCoS13 Business Meeting LORIA &amp; INRIA Nancy Grand Est September 19, 2013

FroCoS13 Business Meeting LORIA &amp; INRIA Nancy Grand Est September 19, 2013 (LORIA-UHP-INRIA) FroCoS13 Business Meeting 1 / 15 Context: Colocation with Tableaux { FroCoS , Tableaux } IJCAR12 FroCoS13?

336 views • 15 slides
tr s tr

tr s tr

tr P st P rs sr t Prr r t t tr s tr

1.3k views • 85 slides
HEAD EAST FOR A BARGAIN CITY BREAK AS STERLING SLIDES Clean sweep for Eastern Europe in 2016

HEAD EAST FOR A BARGAIN CITY BREAK AS STERLING SLIDES Clean sweep for Eastern Europe in 2016

Mar 19, 2016 00:01 GMT HEAD EAST FOR A BARGAIN CITY BREAK AS STERLING SLIDES Clean sweep for Eastern Europe in 2016 City Costs Barometer: Warsaw heads cities that fill eight of the best value top ten (www.postoffice.co.uk/citycosts)

210 views • 7 slides
Advances in Machine Learning tools in High Energy Physics David Rousseau LAL-Orsay

Advances in Machine Learning tools in High Energy Physics David Rousseau LAL-Orsay

Advances in Machine Learning tools in High Energy Physics David Rousseau LAL-Orsay rousseau@lal.in2p3.fr LPSC Seminar, Tuesday 4th October 2016 Outline q Basics q ML software tools q ML techniques q ML in analysis q ML in

904 views • 73 slides
Inspector Gadget Automated Extraction of Proprietary Gadgets from Malware Binaries Clemens

Inspector Gadget Automated Extraction of Proprietary Gadgets from Malware Binaries Clemens

Int. Secure Systems Lab Vienna University of Technology Inspector Gadget Automated Extraction of Proprietary Gadgets from Malware Binaries Clemens KOLBITSCH Thorsten HOLZ Engin KIRDA Christopher KRUEGEL ck@iseclab.org Int.

622 views • 39 slides
Domain-specific front-end for virtual Domain-specific front-end for virtual system modeling

Domain-specific front-end for virtual Domain-specific front-end for virtual system modeling

Domain-specific front-end for virtual Domain-specific front-end for virtual system modeling Workshop on Graphical Modeling Language Development at ECMFA 2012 Janne Vatjus-Anttila, Jari Kreku, Kari Tiensyrj VTT Technical Research Centre of

320 views • 20 slides
Global Illumination CPSC 453 Fall 2018 Sonny Chan Outline for Today (and Thursday)

Global Illumination CPSC 453 Fall 2018 Sonny Chan Outline for Today (and Thursday)

Global Illumination CPSC 453 Fall 2018 Sonny Chan Outline for Today (and Thursday) Motivation Radiometry: foundations of physically-based rendering Surface reflectance The rendering equation Solutions to the rendering

886 views • 86 slides

More recommend