Model-Based Conformance Testing of Software Product Lines 23rd - PowerPoint PPT Presentation

Model-Based Conformance Testing of Software Product Lines 23rd CREST Open Workshop Malte Lochau, Real-Time Systems Lab, TU Darmstadt Nov. 20th, 2012

Model-Based Testing [UL07] test model (tm) conforms? i k , . . . , i 2 , i 1 iut o 1 , o 2 , . . . , o k Black-box assumption for implementation under test (iut) Automated derivation and application of test cases from a behavioral specification (test model) Lochau | SPL MBT | 2

Model-Based Conformance Testing [Tre99] test model (tm) conforms? i k , . . . , i 2 , i 1 iut o 1 , o 2 , . . . , o k Test Hypothesis observational equivalent? implementation model (im) Test Hypothesis for test result confidence and reproducibility [Ber91] Partial verification of the observable behavioral conformance [NH84] Lochau | SPL MBT | 3

Testing Preorder Relations Implementation relation – equivalent behaviors: impl ” spec Lochau | SPL MBT | 4

Testing Preorder Relations Implementation relation – equivalent behaviors: impl ” spec Preorder relation – implementation conforms specification: impl Ď spec Lochau | SPL MBT | 4

Testing Preorder Relations Implementation relation – equivalent behaviors: impl ” spec Preorder relation – implementation conforms specification: impl Ď spec Model-based testing – test model as behavioral specification: impl Ď tm Lochau | SPL MBT | 4

Testing Preorder Relations Implementation relation – equivalent behaviors: impl ” spec Preorder relation – implementation conforms specification: impl Ď spec Model-based testing – test model as behavioral specification: impl Ď tm Black-box assumption – imaginary implementation model: im Ď tm Lochau | SPL MBT | 4

Testing Preorder Relations Implementation relation – equivalent behaviors: impl ” spec Preorder relation – implementation conforms specification: impl Ď spec Model-based testing – test model as behavioral specification: impl Ď tm Black-box assumption – imaginary implementation model: im Ď tm Weakened implementation relation – testing equivalence: im Ď te tm Lochau | SPL MBT | 4

Testing Preorder Relations Implementation relation – equivalent behaviors: impl ” spec Preorder relation – implementation conforms specification: impl Ď spec Model-based testing – test model as behavioral specification: impl Ď tm Black-box assumption – imaginary implementation model: im Ď tm Weakened implementation relation – testing equivalence: im Ď te tm Parameterized implementation relation – finite set of behaviors: im Ď TC tm te Lochau | SPL MBT | 4

Labeled Transition Systems (LTS) Labeled State-Transition Graph p Proc , Act , Ñq LTS trace semantics tr “ p a 1 , a 2 , . . . , a n q P Tr p s 0 , lts q Ď Act ˚ , iff a 1 a 2 Ñ s 2 ¨ ¨ ¨ a n tr Ý Ñ s 1 Ý Ý Ñ s n “ s 0 Ý Ñ s n s 0 Lochau | SPL MBT | 5

Labeled Transition Systems (LTS) Labeled State-Transition Graph p Proc , Act , Ñq LTS trace semantics tr “ p a 1 , a 2 , . . . , a n q P Tr p s 0 , lts q Ď Act ˚ , iff a 1 a 2 Ñ s 2 ¨ ¨ ¨ a n tr Ý Ñ s 1 Ý Ý Ñ s n “ s 0 Ý Ñ s n s 0 Tr p s 0 , lts 1 q “ t a , ab , ac u s 0 a a s 1 s 4 c b s 3 s 5 Lochau | SPL MBT | 5

Labeled Transition Systems (LTS) Labeled State-Transition Graph p Proc , Act , Ñq LTS trace semantics tr “ p a 1 , a 2 , . . . , a n q P Tr p s 0 , lts q Ď Act ˚ , iff a 1 a 2 Ñ s 2 ¨ ¨ ¨ a n tr Ý Ñ s 1 Ý Ý Ñ s n “ s 0 Ý Ñ s n s 0 Tr p s 0 , lts 1 q “ t a , ab , ac u Trace Preorder as Testing Preorder Relation: s 0 a a im Ď T tm : ô Tr p s 0 , im q Ď Tr p s 0 , tm q s 1 s 4 c b s 3 s 5 Lochau | SPL MBT | 5

Labeled Transition Systems (LTS) Labeled State-Transition Graph p Proc , Act , Ñq LTS trace semantics tr “ p a 1 , a 2 , . . . , a n q P Tr p s 0 , lts q Ď Act ˚ , iff a 1 a 2 Ñ s 2 ¨ ¨ ¨ a n tr Ý Ñ s 1 Ý Ý Ñ s n “ s 0 Ý Ñ s n s 0 Tr p s 0 , lts 1 q “ t a , ab , ac u Trace Preorder as Testing Preorder Relation: s 0 a a im Ď T tm : ô Tr p s 0 , im q Ď Tr p s 0 , tm q s 1 s 4 Parameterized Testing Preorder Relation: c b im Ď TC tm : ô p Tr p s 0 , im qX TC q Ď p Tr p s 0 , tm qX TC q s 3 s 5 T where TC Ď Tr p s 0 , im q Lochau | SPL MBT | 5

Example s 0 s 0 s 0 a a a a a s 1 s 4 s 1 s 1 s 4 c c c b b b s 3 s 5 s 2 s 3 s 2 s 3 (e) lts 1 (f) lts 2 (g) lts 3 lts 1 ” T lts 2 ” T lts 3 Lochau | SPL MBT | 6

Example s 0 s 0 s 0 a a a a a s 1 s 4 s 1 s 1 s 4 c c c b b b s 3 s 5 s 2 s 3 s 2 s 3 (h) lts 1 (i) lts 2 (j) lts 3 lts 1 ” T lts 2 ” T lts 3 But: different behaviors after composition with environment emitting input action a . Lochau | SPL MBT | 6

Decorated Trace Semantics Trace equivalence is a weak equivalence Stricter notions of behavioral equivalence discriminate different decision structures within the state-transition graphs [Abr87] Lochau | SPL MBT | 7

Decorated Trace Semantics Trace equivalence is a weak equivalence Stricter notions of behavioral equivalence discriminate different decision structures within the state-transition graphs [Abr87] But: testing is limited to observable behaviors a initials p s q “ t a P Act | s Ý Ñ u Ď Act Lochau | SPL MBT | 7

Decorated Trace Semantics Trace equivalence is a weak equivalence Stricter notions of behavioral equivalence discriminate different decision structures within the state-transition graphs [Abr87] But: testing is limited to observable behaviors a initials p s q “ t a P Act | s Ý Ñ u Ď Act Example: Failures and Readies Lochau | SPL MBT | 7

Decorated Trace Semantics Trace equivalence is a weak equivalence Stricter notions of behavioral equivalence discriminate different decision structures within the state-transition graphs [Abr87] But: testing is limited to observable behaviors a initials p s q “ t a P Act | s Ý Ñ u Ď Act Example: Failures and Readies A pair p tr , X q with tr P Act ˚ and X Ď Act is a failure of state s 0 if tr Ý Ñ s n for some state s n and initials p s n q X X “ ∅ . s 0 Lochau | SPL MBT | 7

Decorated Trace Semantics Trace equivalence is a weak equivalence Stricter notions of behavioral equivalence discriminate different decision structures within the state-transition graphs [Abr87] But: testing is limited to observable behaviors a initials p s q “ t a P Act | s Ý Ñ u Ď Act Example: Failures and Readies A pair p tr , X q with tr P Act ˚ and X Ď Act is a failure of state s 0 if tr Ý Ñ s n for some state s n and initials p s n q X X “ ∅ . s 0 A pair p tr , X q with tr P Act ˚ and X Ď Act is a ready of state s 0 if tr Ý Ñ s n for some state s n and initials p s n q “ X . s 0 Lochau | SPL MBT | 7

Preorder Relation Inclusion Hierarchy [BFvG04] Ď R Ď B Ď RT Ď F Ď CT Ď FT Ď 1S Ď T Lochau | SPL MBT | 8

Example – Revisited s 0 s 0 s 0 a a a a a s 1 s 4 s 1 s 1 s 4 c c c b b b s 3 s 5 s 2 s 3 s 2 s 3 (k) lts 1 (l) lts 2 (m) lts 3 lts 3 has completed trace a lts 2 Ď F lts 1 lts 2 and lts 1 are incomparable under Ď R Lochau | SPL MBT | 9

Model-Based SPL Testing SPL Test Model Product Implementations conforms? tm SPL p F q under Test i k , . . . , i 2 , i 1 iut 1 o 1 , o 2 , . . . , o k iut 2 iut n tm 1 tm 2 tm n Reusable generic test model specification parameterized over features F Lochau | SPL MBT | 10

Model-Based SPL Testing SPL Test Model Product Implementations conforms? tm SPL p F q under Test i k , . . . , i 2 , i 1 iut 1 o 1 , o 2 , . . . , o k iut 2 iut n tm 1 tm 2 tm n Reusable generic test model specification parameterized over features F Reuse of test cases TC 1 Ď TC of product iut for product iut 1 if tm Ď TC 1 tm 1 te Lochau | SPL MBT | 10

Model-Based SPL Testing SPL Test Model Product Implementations conforms? tm SPL p F q under Test i k , . . . , i 2 , i 1 iut 1 o 1 , o 2 , . . . , o k iut 2 iut n im 1 im 2 im n Extending the Test Hypothesis to SPLs under test Lochau | SPL MBT | 11

Model-Based SPL Testing SPL Test Model Product Implementations conforms? tm SPL p F q under Test i k , . . . , i 2 , i 1 iut 1 o 1 , o 2 , . . . , o k iut 2 iut n im 1 im 2 im n Extending the Test Hypothesis to SPLs under test Reuse of test results TC 2 Ď TC 1 of product iut for product iut 1 if im Ď TC 2 im 1 te Lochau | SPL MBT | 11

Feature-Annotated LTS [CHSL11] s 0 a r f 1 s a r f 2 s s 1 s 4 c r� f 2 _ � f 3 s b r f 1 s c r f 2 ^ f 3 s s 2 s 3 s 5 Lochau | SPL MBT | 12

Feature-Annotated LTS [CHSL11] LTS with transition annotations σ p s , a , s 1 q P B p F q s 0 a r f 1 s a r f 2 s s 1 s 4 c r� f 2 _ � f 3 s b r f 1 s c r f 2 ^ f 3 s s 2 s 3 s 5 Lochau | SPL MBT | 12

Feature-Annotated LTS [CHSL11] LTS with transition annotations σ p s , a , s 1 q P B p F q Constraints by feature model fm P B p F q s 0 a r f 1 s a r f 2 s s 1 s 4 c r� f 2 _ � f 3 s b r f 1 s c r f 2 ^ f 3 s s 2 s 3 s 5 Lochau | SPL MBT | 12

Feature-Annotated LTS [CHSL11] LTS with transition annotations σ p s , a , s 1 q P B p F q Constraints by feature model fm P B p F q s 0 a r f 1 s a r f 2 s Product configuration Γ : F Ñ B s 1 s 4 (full, partial) c r� f 2 _ � f 3 s b r f 1 s c r f 2 ^ f 3 s s 2 s 3 s 5 Lochau | SPL MBT | 12