Chaining Test Cases for Reactive System Testing Peter Schrammel, Tom Melham and Daniel Kroening first.lastname@cs.ox.ac.uk The 25th IFIP International Conference on Testing Software and Systems (ICTSS’13) Nov 13-15, 2013, Istanbul, Turkey
Test Chains Context: Safety critical embedded software Often modelled as synchronous reactive system Safety standards: tool support for systematic testing desirable Problem: Often lengthy input sequences required to drive the system to a test goal Reset after each test case: serious problem in on-target testing Goal: Find a test case chain: a single test case that covers a set of test goals and minimises overall test execution time
Model-Based Testing requirements design model implementation check check
Model-Based Testing requirements design model implementation check check check
Model-Based Testing requirements design model implementation check check check test suite generator test suite
Model-Based Testing requirements design model implementation check check check test suite generator test suite
Model-Based Testing textual Simulink/ requirements Stateflow formalised generated implementation check properties C code check
Model-Based Testing textual Simulink/ requirements Stateflow formalised generated implementation check properties C code check test suite generator test suite
Model-Based Testing textual Simulink/ requirements Stateflow formalised generated implementation check properties C code check test suite generator test suite
Example: Cruise Control brake ∨ dec gas ∨ acc OFF,0,FALSE brake ∨ dec button button OFF,1,FALSE OFF,0,TRUE brake ∨ dec button gas ∨ acc acc ∨ dec ON,1,TRUE button gas ∨ acc gas brake brake brake ∨ acc ∨ dec ∨ dec brake DIS,2,TRUE DIS,0,TRUE ∨ dec button brake ∨ dec gas ∨ acc gas gas ∨ acc OFF,2,FALSE OFF,2,TRUE ∨ acc button button
Example: Generated C Code from Simulink void i n i t ( s t a t e t ∗ s ) { s − > mode = OFF ; s − > speed = 0; s − > enable = FALSE ; } ∗ i , ∗ s ) { void compute( i o t s t a t e t mode = s − > mode ; switch (mode) { case ON: i f ( i − > gas | | i − > brake ) s − > mode=DIS ; break ; case DIS : i f ( ( s − > speed==2 && ( i − > dec | | i − > brake ) ) | | ( s − > speed==0 && ( i − | | i − > acc > gas ) ) ) s − > mode=ON; break ; case OFF : i f ( s − > speed==0 && s − > enable && ( i − > gas | | i − > acc ) | | s − > speed==1 && i − > button | | s − > speed==2 && s − > enable && ( i − > brake | | i − > dec ) ) s − > mode=ON; break ; } i f ( i − > button ) s − > enable = ! s − > enable ; i f ( ( i − | | mode!=ON && i − > acc ) && s − > speed < 2) s − > gas > speed ++; i f ( ( i − > brake | | mode!=ON && i − > dec ) && s − > speed > 0) s − > speed −− ; }
Example: Generated C Code from Simulink void i n i t ( s t a t e t ∗ s ) { s − > mode = OFF ; s − > speed = 0; s − > enable = FALSE ; } ∗ i , ∗ s ) { void compute( i o t s t a t e t mode = s − > mode ; switch (mode) { case ON: i f ( i − > gas | | i − > brake ) s − > mode=DIS ; break ; case DIS : i f ( ( s − > speed==2 && ( i − > dec | | i − > brake ) ) | | ( s − > speed==0 && ( i − | | i − > acc > gas ) ) ) Formalised properties: s − > mode=ON; � � p 1 : G mode = ON ∧ speed = 1 ∧ dec ⇒ X ( speed = 1) break ; � � p 2 : G mode = DIS ∧ speed = 2 ∧ dec ⇒ X ( mode = ON ) case OFF : � � p 3 : G mode = ON ∧ brake ⇒ X ( mode = DIS ) i f ( s − > speed==0 && s − > enable && ( i − > gas | | i − > acc ) | | � � p 4 : G mode = OFF ∧ speed = 2 ∧ ¬ enable ∧ button ⇒ X enable s − > speed==1 && i − > button | | s − > speed==2 && s − > enable && ( i − > brake | | i − > dec ) ) s − > mode=ON; break ; } i f ( i − > button ) s − > enable = ! s − > enable ; i f ( ( i − | | mode!=ON && i − > acc ) && s − > speed < 2) s − > gas > speed ++; i f ( ( i − > brake | | mode!=ON && i − > dec ) && s − > speed > 0) s − > speed −− ; }
Example I = F brake ∨ dec gas ∨ acc OFF,0,FALSE brake ∨ dec button button OFF,1,FALSE OFF,0,TRUE button brake ∨ dec button gas ∨ acc acc ∨ dec ( p 1 ) ON,1,TRUE button gas ∨ acc gas brake brake gas brake ( p 3 ) ∨ acc ∨ dec ∨ dec ( p 2 ) brake DIS,2,TRUE DIS,0,TRUE ∨ dec button brake ∨ dec gas ∨ acc gas gas ∨ acc OFF,2,FALSE OFF,2,TRUE ∨ acc button ( p 4 ) button
Example I = F brake ∨ dec gas ∨ acc OFF,0,FALSE brake ∨ dec button button OFF,1,FALSE OFF,0,TRUE button brake ∨ dec button gas ∨ acc acc ∨ dec ( p 1 ) ON,1,TRUE button gas ∨ acc gas brake brake gas ∨ acc ∨ dec ∨ dec ( p 2 ) brake ( p 3 ) brake DIS,2,TRUE DIS,0,TRUE ∨ dec button brake ∨ dec gas ∨ acc gas gas ∨ acc OFF,2,FALSE OFF,2,TRUE ∨ acc button ( p 4 ) button
Preliminaries Program: State space Σ, input space Υ Initial states I ⊆ Σ Transition relation T ⊆ Σ × Υ × Σ
Preliminaries Program: State space Σ, input space Υ Initial states I ⊆ Σ Transition relation T ⊆ Σ × Υ × Σ Bounded Model Checking: Check the existence of a path � s 0 , s 1 , . . . , s K � of increasing length K from φ to φ ′ � T ( s k − 1 , i k − 1 , s k ) ∧ φ ′ ( s K ) φ ( s 0 ) ∧ 1 ≤ k ≤ K If SAT: satisfying assignment aka counterexample ( s 0 , i 0 , s 1 , i 1 , . . . , s K − 1 , i K − 1 , s K )
Preliminaries Program: State space Σ, input space Υ Initial states I ⊆ Σ Transition relation T ⊆ Σ × Υ × Σ Bounded Model Checking: Check the existence of a path � s 0 , s 1 , . . . , s K � of increasing length K from φ to φ ′ � T ( s k − 1 , i k − 1 , s k ) ∧ φ ′ ( s K ) φ ( s 0 ) ∧ 1 ≤ k ≤ K If SAT: satisfying assignment aka counterexample ( s 0 , i 0 , s 1 , i 1 , . . . , s K − 1 , i K − 1 , s K ) Test case generation: φ = I and test goal φ ′ Test case: input sequence � i 0 , . . . , i K − 1 � , expected outcome
Chaining Test Cases Temporal logic safety specification: Set of properties, e.g. , of type � � G mode = ON ∧ speed = 1 ∧ dec ⇒ X ( speed = 1) � �� � assumption ϕ
Chaining Test Cases Temporal logic safety specification: Set of properties, e.g. , of type � � G mode = ON ∧ speed = 1 ∧ dec ⇒ X ( speed = 1) � �� � assumption ϕ Test goals: set of assumptions ϕ (finite paths)
Chaining Test Cases Temporal logic safety specification: Set of properties, e.g. , of type � � G mode = ON ∧ speed = 1 ∧ dec ⇒ X ( speed = 1) � �� � assumption ϕ Test goals: set of assumptions ϕ (finite paths) Test chain : from initial states I via all ϕ s to final states F
Chaining Test Cases Temporal logic safety specification: Set of properties, e.g. , of type � � G mode = ON ∧ speed = 1 ∧ dec ⇒ X ( speed = 1) � �� � assumption ϕ Test goals: set of assumptions ϕ (finite paths) Test chain : from initial states I via all ϕ s to final states F Approach Abstraction: property reachability graph 1 Optimisation: shortest path 2 Concretisation: compute concrete test case 3
Abstraction: Property Reachability Graph Weighted, directed graph: Nodes: test goals ϕ Edges: from I to all ϕ s from all ϕ s to F pairwise links between ϕ s Edge weights: number of execution steps Incrementally build graph by reachability queries: ϕ 1 ϕ 3 I F ϕ 2 ϕ 4
Abstraction: Property Reachability Graph Weighted, directed graph: Nodes: test goals ϕ Edges: from I to all ϕ s from all ϕ s to F pairwise links between ϕ s Edge weights: number of execution steps Incrementally build graph by reachability queries: K = 1 ϕ 1 ϕ 3 1 I 1 F 1 ϕ 2 ϕ 4
Abstraction: Property Reachability Graph Weighted, directed graph: Nodes: test goals ϕ Edges: from I to all ϕ s from all ϕ s to F pairwise links between ϕ s Edge weights: number of execution steps Incrementally build graph by reachability queries: K = 2 2 ϕ 1 ϕ 3 1 2 2 I 1 2 2 F 2 1 2 ϕ 2 ϕ 4 2
Existence of a Covering Path Covering path: path that visits all nodes at least once. There is a covering path from I to F iff ϕ 1 (1) all nodes are reachable from I , (2) F is reachable from all nodes, I F and (3) for all pairs of nodes ( v 1 , v 2 ), ϕ 2 (a) v 2 is reachable from v 1 or (b) v 1 is reachable from v 2 . Reachability can be decided in constant time on the transitive closure of the graph.
Existence of a Covering Path Covering path: path that visits all nodes at least once. There is a covering path from I to F iff ϕ 1 (1) all nodes are reachable from I , (2) F is reachable from all nodes, I F and (3) for all pairs of nodes ( v 1 , v 2 ), ϕ 2 (a) v 2 is reachable from v 1 or (b) v 1 is reachable from v 2 . Reachability can be decided in constant time on the transitive closure of the graph.
Recommend
More recommend
