SLIDE 1 State-Based Testing
Part B – Error Identification Generating test cases for complex behaviour
- Reference:Robert V. Binder
Testing Object-Oriented Systems: Models, Patterns, and Tools
Addison-Wesley, 2000, Chapter 7
SLIDE 2 SEI–2
Flattening the statechart
Statecharts are great for communication, reducing
clutter etc.
- They might hide subtle bugs
e.g. entering a sub-state rather than a super-state
SLIDE 3 SEI–3
Flattening the statechart – 2
For testing we need to expand them to full transition
diagrams
Expansion makes implicit transitions explicit, so they
are not lost
- Expansion is a flat view
Includes everything from inheritance in OO and
sub-states in statecharts
SLIDE 4
SEI–4
Concurrent statechart
SLIDE 5 SEI–5
Concurrency Hides Problems
Concurrency hides implicit state combinations
Hides potential serious defects Arise from implicit state combinations
- Explicit violations of implicit prohibitions should be tested
SLIDE 6
SEI–6
Expanding the Example
!
SLIDE 7
SEI–7
Expanding the Example – 2
!
Events, guards and output for the automotive control FSM
SLIDE 8 SEI–8
Unspecified Event/State Pairs
State machine models will not include all events for all
states
- Implicit transitions may be
Illegal Ignored Or a specification omission
SLIDE 9 SEI–9
Unspecified Event/State Pairs – 2
Accepted illegal events lead to bugs called sneak paths
- For testing purposes, we cannot ignore implicit
behaviour
Develop a Response Matrix
SLIDE 10
SEI–10
Example statechart
SLIDE 11
SEI–11
Response
matrix
SLIDE 12
SEI–12
Possible responses to illegal events
SLIDE 13 SEI–13
Designing responses to illegal events
Abstract state should not change
Concrete state may change due to exception handling
- Illegal event design question
Handle with defensive programming
Uncooperative (defensive) systems
- Handle with precondition contracts
Cooperative systems
SLIDE 14 SEI–14
Designing responses to illegal events – 2
Possible responses
Raise exception Treat message as a noop Attempt error recovery Invoke abnormal termination
- Tester needs to decide expected responses so actual
responses can be evaluated
SLIDE 15 SEI–15
State model validation
Before it is used to generate test cases a state
model must be
Complete Consistent Correct
Passes checklists
SLIDE 16
Checklist questions
What is a checklist? Why do we use checklists? What checklists would be useful for statecharts? SEI–16
SLIDE 17
SEI–17
Validation checklists
We will look at five validation checklists
Structure checklist State name checklist Guarded transition checklist Robustness checklist Well-formed subclass behaviour checklist
SLIDE 18
Structure checklist question
What would you look for to verify the structure of a
statechart is correct?
SEI–18
SLIDE 19 SEI–19
Structure checklist
There is an initial state with only outbound transitions
- There is a final state with only inbound transitions
If not, explicit reason is needed
- Except for the initial and final states, every state has at
least one incoming and one outgoing transition
SLIDE 20 SEI–20
Structure checklist – 2
Every state is reachable from the initial state
- The final state is reachable from all states
- No equivalent states
SLIDE 21 SEI–21
Structure checklist – 3
Every defined event and every defined action appears in
at least one transition
- The events accepted in a particular state are
Unique Or differentiated by mutually exclusive guards
For every state, every event is accepted or rejected
Either explicitly or implicitly
SLIDE 22 SEI–22
State name checklist
Poor names are indications of
Incomplete design Or incorrect design
- Names must be meaningful in the context of the
application
Past participles are OK
SLIDE 23
SEI–23
State name checklist – 2
State names should be passive If a state is not necessary, leave it out
“Wait states” are often superfluous
SLIDE 24
Guarded transition checklist question
What would you look for to ensure the guards on
transitions are correct in a statechart?
SEI–24
SLIDE 25 SEI–25
Guarded transition checklist
Guard variables are visible
- The entire range of truth values for a particular event is
covered
- Each guard is mutually exclusive of all other guards
SLIDE 26 SEI–26
Guarded transition checklist – 2
Guards with three or more variables are modeled with a
decision table
- The evaluation of a guard does not cause side effects
SLIDE 27 SEI–27
Robustness checklist
There is an explicit spec for an error-handling or
exception-handling mechanism for implicitly rejected events
- Illegal events do not corrupt the machine
Preserve the last good state Reset to a valid state Or self-destruct safely
SLIDE 28 SEI–28
Robustness checklist – 2
Actions have no side effects on the resultant state
- For contract violations specify mechanism for
Explicit exception Error logging Recovery
SLIDE 29 SEI–29
Well-Formed Subclass Behaviour Checklist
Does not remove any superclass states
All transitions accepted in the superclass are
accepted in the subclass
- All guards on superclass transitions are
The same for subclass transitions Or weaker for subclass transitions
SLIDE 30 SEI–30
Well-Formed Subclass Behaviour Checklist – 2
Subclass does not weaken the state invariant of the
superclass
- Subclass may add an orthogonal state defined with
respect to its locally introduced instance variables
SLIDE 31 SEI–31
Well-Formed Subclass Behaviour Checklist – 3
All inherited actions are consistent with the subclass's
responsibilities
Verify name-scope sensitive or dynamic binding of
intraclass messages is correct
- All inherited accessor events are appropriate in the
context of the subclass
- Messages sent to objects that are variables in a guard
expression do not have side effects on the class under test
SLIDE 32 SEI–32
Control faults for state machines
An incorrect sequence of events is accepted
- An incorrect sequence of outputs is produced
SLIDE 33
State control faults question
What types of state control faults can occur in a
statechart?
SEI–33
SLIDE 34 SEI–34
Types of state control faults
Missing transition
- Incorrect transition
- Missing action
- Incorrect action
- Trap door
- Sneak path
- Corrupt state
- Illegal message failure
Occur individually or any nightmare combination Are all combinations possible?
SLIDE 35
State control faults
SEI–35
SLIDE 36
Missing transition question
How can you tell from the behaviour of a statechart
that there is a missing transition?
SEI–36
SLIDE 37 SEI–37
Missing transition
Implementation does not respond to a valid event-state
pair
- Resultant state is incorrect but not corrupt
SLIDE 38
SEI–38
Missing transition – 2
SLIDE 39
Incorrect transition question
How can you tell from the behaviour of a statechart
that there is an incorrect transition?
SEI–39
SLIDE 40 SEI–40
Incorrect transition
Implementation behaves as if an incorrect resultant state
has been reached
- Resultant state is incorrect but not corrupt
SLIDE 41
SEI–41
Incorrect transition – 2
SLIDE 42
Missing action question
How can you tell from the behaviour of a statechart
that there is a missing action?
SEI–42
SLIDE 43
SEI–43
Missing action
Implementation does not have an action for a transition
Can have incorrect output Later be in an incorrect state Wait forever for the missing action to occur
SLIDE 44
SEI–44
Missing action – 2
If simulateVolley() is missing, system hangs
SLIDE 45
Incorrect action question
How can you tell from the behaviour of a statechart
that there is an incorrect action?
SEI–45
SLIDE 46 SEI–46
Incorrect action
Implementation produces the wrong action for a
transition
Different case from incorrect output for an action
Have incorrect output Later be in an incorrect state
SLIDE 47
SEI–47
Incorrect action – 2
Instead of simulateVolley()
SLIDE 48
Trap door question
How can you tell from the behaviour of a statechart
that there is a trap door?
SEI–48
SLIDE 49 SEI–49
Trap door
Implementation accepts an entry event that is not
defined in the specification
Incorrect output Corrupt state Enter wrong state Or combination
SLIDE 50
SEI–50
Trap door – 2
Transition enters the system.
SLIDE 51 SEI–51
Trap door – 3
Can result from
Obsolete features that were not removed
- Inherited features that are inconsistent with the
requirements of the subclass
- "Undocumented" features added by the developer
for debugging purposes
- Sabotage for criminal or malicious purposes
SLIDE 52
Sneak path question
How can you tell from the behaviour of a statechart
that there is a sneak path?
SEI–52
SLIDE 53 SEI–53
Sneak path
Implementation for a state accepts an event that is
Illegal Or unspecified
Incorrect output Corrupt state Enter wrong state Or combination
SLIDE 54
SEI–54
Sneak path
Transition is within the system.
SLIDE 55 SEI–55
Sneak path – 3
Can result from
Obsolete features that were not removed
- Inherited features that are inconsistent with the
requirements of the subclass
- "Undocumented" features added by the developer
for debugging purposes
- Sabotage for criminal or malicious purposes
SLIDE 56
Corrupt state question
How can you tell from the behaviour of a statechart
that there is a corrupt state?
SEI–56
SLIDE 57 SEI–57
Corrupt state
Implementation computes a state that is not valid
- Either the class invariant or state invariant is violated
- Due to coding and / or design errors
SLIDE 58
SEI–58
Corrupt state – 2
Incorrect transition to corrupt state
SLIDE 59 SEI–59
Incorrect Composite Behaviour
Misuse of inheritance with modal classes can lead to
state control bugs
- Subclasses can conflict with sequential requirements
- f a superclass
- Need to test beyond the scope of one class
SLIDE 60 SEI–60
Incorrect Composite Behaviour – 2
Bugs occur for the following reasons
Missing or incorrect redefinition of a method
- Subclass extension of the local state conflicts with a
superclass state
- Subclass fails to retarget a superclass transition
Switches to an incorrect or undefined superclass
state
SLIDE 61 SEI–61
Incorrect Composite Behaviour – 3
Bugs occur for the following reasons – contʼd
Order of evaluation of guards and preconditions is
incorrect or sensitive to the order of evaluation
Guards behave as if an extra state exists
- Order of guard evaluation produces a side effect in the
subclass that is not present in the superclass
- Default name scope resolution results in guard
parameters being bound to the wrong subclass or superclass methods
