[PPT] - Model-Based Conformance Testing of Software Product Lines 23rd PowerPoint Presentation

SLIDE 1

Model-Based Conformance Testing of Software Product Lines

23rd CREST Open Workshop

Malte Lochau, Real-Time Systems Lab, TU Darmstadt

Nov. 20th, 2012

SLIDE 2

Model-Based Testing [UL07]

test model (tm) conforms? ik, . . . , i2, i1

1, o2, . . . , ok

iut

Black-box assumption for implementation under test (iut) Automated derivation and application of test cases from a behavioral specification (test model)

Lochau | SPL MBT | 2

SLIDE 3

Model-Based Conformance Testing [Tre99]

test model (tm) conforms? ik, . . . , i2, i1

1, o2, . . . , ok

iut

Test Hypothesis implementation model (im)

bservational

equivalent?

Test Hypothesis for test result confidence and reproducibility [Ber91] Partial verification of the observable behavioral conformance [NH84]

Lochau | SPL MBT | 3

SLIDE 4

Testing Preorder Relations

Implementation relation – equivalent behaviors: impl ” spec

Lochau | SPL MBT | 4

SLIDE 5

Testing Preorder Relations

Implementation relation – equivalent behaviors: impl ” spec Preorder relation – implementation conforms specification: impl Ď spec

Lochau | SPL MBT | 4

SLIDE 6

Testing Preorder Relations

Implementation relation – equivalent behaviors: impl ” spec Preorder relation – implementation conforms specification: impl Ď spec Model-based testing – test model as behavioral specification: impl Ď tm

Lochau | SPL MBT | 4

SLIDE 7

Testing Preorder Relations

Implementation relation – equivalent behaviors: impl ” spec Preorder relation – implementation conforms specification: impl Ď spec Model-based testing – test model as behavioral specification: impl Ď tm Black-box assumption – imaginary implementation model: im Ď tm

Lochau | SPL MBT | 4

SLIDE 8

Testing Preorder Relations

Implementation relation – equivalent behaviors: impl ” spec Preorder relation – implementation conforms specification: impl Ď spec Model-based testing – test model as behavioral specification: impl Ď tm Black-box assumption – imaginary implementation model: im Ď tm Weakened implementation relation – testing equivalence: im Ďte tm

Lochau | SPL MBT | 4

SLIDE 9

Testing Preorder Relations

Implementation relation – equivalent behaviors: impl ” spec Preorder relation – implementation conforms specification: impl Ď spec Model-based testing – test model as behavioral specification: impl Ď tm Black-box assumption – imaginary implementation model: im Ď tm Weakened implementation relation – testing equivalence: im Ďte tm Parameterized implementation relation – finite set of behaviors: im ĎTC

te

tm

Lochau | SPL MBT | 4

SLIDE 10

Labeled Transition Systems (LTS)

Labeled State-Transition Graph pProc, Act, Ñq LTS trace semantics tr “ pa1, a2, . . . , anq P Trps0, ltsq Ď Act˚, iff s0

a1

Ý Ñ s1

a2

Ý Ñ s2 ¨ ¨ ¨ an Ý Ñ sn “ s0

tr

Ý Ñ sn

Lochau | SPL MBT | 5

SLIDE 11

Labeled Transition Systems (LTS)

Labeled State-Transition Graph pProc, Act, Ñq LTS trace semantics tr “ pa1, a2, . . . , anq P Trps0, ltsq Ď Act˚, iff s0

a1

Ý Ñ s1

a2

Ý Ñ s2 ¨ ¨ ¨ an Ý Ñ sn “ s0

tr

Ý Ñ sn

s0 s4 s5 s1 s3 a a b c

Trps0, lts1q “ ta, ab, acu

Lochau | SPL MBT | 5

SLIDE 12

Labeled Transition Systems (LTS)

Labeled State-Transition Graph pProc, Act, Ñq LTS trace semantics tr “ pa1, a2, . . . , anq P Trps0, ltsq Ď Act˚, iff s0

a1

Ý Ñ s1

a2

Ý Ñ s2 ¨ ¨ ¨ an Ý Ñ sn “ s0

tr

Ý Ñ sn

s0 s4 s5 s1 s3 a a b c

Trps0, lts1q “ ta, ab, acu Trace Preorder as Testing Preorder Relation: im ĎT tm :ô Trps0, imq Ď Trps0, tmq

Lochau | SPL MBT | 5

SLIDE 13

Labeled Transition Systems (LTS)

Labeled State-Transition Graph pProc, Act, Ñq LTS trace semantics tr “ pa1, a2, . . . , anq P Trps0, ltsq Ď Act˚, iff s0

a1

Ý Ñ s1

a2

Ý Ñ s2 ¨ ¨ ¨ an Ý Ñ sn “ s0

tr

Ý Ñ sn

s0 s4 s5 s1 s3 a a b c

Trps0, lts1q “ ta, ab, acu Trace Preorder as Testing Preorder Relation: im ĎT tm :ô Trps0, imq Ď Trps0, tmq Parameterized Testing Preorder Relation: im ĎTC

T

tm :ô pTrps0, imqXTCq Ď pTrps0, tmqXTCq where TC Ď Trps0, imq

Lochau | SPL MBT | 5

SLIDE 14

Example

s0 s4 s5 s1 s3 a a b c

(e) lts1

s0 s1 s3 s2 a b c

(f) lts2

s0 s4 s1 s3 s2 a a b c

(g) lts3

lts1 ”T lts2 ”T lts3

Lochau | SPL MBT | 6

SLIDE 15

Example

s0 s4 s5 s1 s3 a a b c

(h) lts1

s0 s1 s3 s2 a b c

(i) lts2

s0 s4 s1 s3 s2 a a b c

(j) lts3

lts1 ”T lts2 ”T lts3 But: different behaviors after composition with environment emitting input action a.

Lochau | SPL MBT | 6

SLIDE 16

Decorated Trace Semantics

Trace equivalence is a weak equivalence Stricter notions of behavioral equivalence discriminate different decision structures within the state-transition graphs [Abr87]

Lochau | SPL MBT | 7

SLIDE 17

Decorated Trace Semantics

Trace equivalence is a weak equivalence Stricter notions of behavioral equivalence discriminate different decision structures within the state-transition graphs [Abr87] But: testing is limited to observable behaviors initialspsq “ ta P Act | s

a

Ý Ñ u Ď Act

Lochau | SPL MBT | 7

SLIDE 18

Decorated Trace Semantics

Trace equivalence is a weak equivalence Stricter notions of behavioral equivalence discriminate different decision structures within the state-transition graphs [Abr87] But: testing is limited to observable behaviors initialspsq “ ta P Act | s

a

Ý Ñ u Ď Act Example: Failures and Readies

Lochau | SPL MBT | 7

SLIDE 19

Decorated Trace Semantics

Trace equivalence is a weak equivalence Stricter notions of behavioral equivalence discriminate different decision structures within the state-transition graphs [Abr87] But: testing is limited to observable behaviors initialspsq “ ta P Act | s

a

Ý Ñ u Ď Act Example: Failures and Readies A pair ptr, Xq with tr P Act ˚ and X Ď Act is a failure of state s0 if s0

tr

Ý Ñ sn for some state sn and initialspsnq X X “ ∅.

Lochau | SPL MBT | 7

SLIDE 20

Decorated Trace Semantics

Trace equivalence is a weak equivalence Stricter notions of behavioral equivalence discriminate different decision structures within the state-transition graphs [Abr87] But: testing is limited to observable behaviors initialspsq “ ta P Act | s

a

Ý Ñ u Ď Act Example: Failures and Readies A pair ptr, Xq with tr P Act ˚ and X Ď Act is a failure of state s0 if s0

tr

Ý Ñ sn for some state sn and initialspsnq X X “ ∅. A pair ptr, Xq with tr P Act ˚ and X Ď Act is a ready of state s0 if s0

tr

Ý Ñ sn for some state sn and initialspsnq “ X.

Lochau | SPL MBT | 7

SLIDE 21

Preorder Relation Inclusion Hierarchy [BFvG04]

ĎB ĎRT ĎR ĎFT Ď1S ĎF ĎCT ĎT

Lochau | SPL MBT | 8

SLIDE 22

Example – Revisited

s0 s4 s5 s1 s3 a a b c

(k) lts1

s0 s1 s3 s2 a b c

(l) lts2

s0 s4 s1 s3 s2 a a b c

(m) lts3

lts3 has completed trace a lts2 ĎF lts1 lts2 and lts1 are incomparable under ĎR

Lochau | SPL MBT | 9

SLIDE 23

Model-Based SPL Testing

SPL Test Model tmSPLpFq conforms? ik, . . . , i2, i1

1, o2, . . . , ok

Product Implementations under Test

iut1 iut2 iutn

tm2 tm1 tmn

Reusable generic test model specification parameterized over features F

Lochau | SPL MBT | 10

SLIDE 24

Model-Based SPL Testing

SPL Test Model tmSPLpFq conforms? ik, . . . , i2, i1

1, o2, . . . , ok

Product Implementations under Test

iut1 iut2 iutn

tm2 tm1 tmn

Reusable generic test model specification parameterized over features F Reuse of test cases TC 1 Ď TC of product iut for product iut1 if tm ĎTC 1

te

tm1

Lochau | SPL MBT | 10

SLIDE 25

Model-Based SPL Testing

SPL Test Model tmSPLpFq conforms? ik, . . . , i2, i1

1, o2, . . . , ok

Product Implementations under Test

iut1 iut2 iutn

im2 im1 imn

Extending the Test Hypothesis to SPLs under test

Lochau | SPL MBT | 11

SLIDE 26

Model-Based SPL Testing

SPL Test Model tmSPLpFq conforms? ik, . . . , i2, i1

1, o2, . . . , ok

Product Implementations under Test

iut1 iut2 iutn

im2 im1 imn

Extending the Test Hypothesis to SPLs under test Reuse of test results TC 2 Ď TC 1 of product iut for product iut1 if im ĎTC2

te

im1

Lochau | SPL MBT | 11

SLIDE 27

Feature-Annotated LTS [CHSL11]

s0 s4 s5 s1 s3 s2 a rf1s a rf2s b rf1s c rf2 _ f3s c rf2 ^ f3s

Lochau | SPL MBT | 12

SLIDE 28

Feature-Annotated LTS [CHSL11]

s0 s4 s5 s1 s3 s2 a rf1s a rf2s b rf1s c rf2 _ f3s c rf2 ^ f3s

LTS with transition annotations σps, a, s1q P BpFq

Lochau | SPL MBT | 12

SLIDE 29

Feature-Annotated LTS [CHSL11]

s0 s4 s5 s1 s3 s2 a rf1s a rf2s b rf1s c rf2 _ f3s c rf2 ^ f3s

LTS with transition annotations σps, a, s1q P BpFq Constraints by feature model fm P BpFq

Lochau | SPL MBT | 12

SLIDE 30

Feature-Annotated LTS [CHSL11]

s0 s4 s5 s1 s3 s2 a rf1s a rf2s b rf1s c rf2 _ f3s c rf2 ^ f3s

LTS with transition annotations σps, a, s1q P BpFq Constraints by feature model fm P BpFq Product configuration Γ : F Ñ B (full, partial)

Lochau | SPL MBT | 12

SLIDE 31

Feature-Annotated LTS [CHSL11]

s0 s4 s5 s1 s3 s2 a rf1s a rf2s b rf1s c rf2 _ f3s c rf2 ^ f3s

LTS with transition annotations σps, a, s1q P BpFq Constraints by feature model fm P BpFq Product configuration Γ : F Ñ B (full, partial) Product space PCfm “ tΓ : F Ñ B | Γ | ù fmu

Lochau | SPL MBT | 12

SLIDE 32

Feature-Annotated LTS [CHSL11]

s0 s4 s5 s1 s3 s2 a rf1s a rf2s b rf1s c rf2 _ f3s c rf2 ^ f3s

LTS with transition annotations σps, a, s1q P BpFq Constraints by feature model fm P BpFq Product configuration Γ : F Ñ B (full, partial) Product space PCfm “ tΓ : F Ñ B | Γ | ù fmu Feature model refinement fm1 Ďfm fm is product space refinement

Lochau | SPL MBT | 12

SLIDE 33

Transition Modalities [LT88]

s0 s4 s5 s1 s3 s2 a rf1s a rf2s b rf1s c rf2 _ f3s c rf2 ^ f3s

fm “ f1 ^ pf2 _ f3q

Lochau | SPL MBT | 13

SLIDE 34

Transition Modalities [LT88]

s0 s4 s5 s1 s3 s2 a rf1s a rf2s b rf1s c rf2 _ f3s c rf2 ^ f3s

fm “ f1 ^ pf2 _ f3q may-transitions Ñmay ĎÑ, where s

a

Ý Ñmay s1 :ô DΓ P PCfm : Γ | ù σps, a, s1q

Lochau | SPL MBT | 13

SLIDE 35

Transition Modalities [LT88]

s0 s4 s5 s1 s3 s2 a rf1s a rf2s b rf1s c rf2 _ f3s c rf2 ^ f3s

fm “ f1 ^ pf2 _ f3q may-transitions Ñmay ĎÑ, where s

a

Ý Ñmay s1 :ô DΓ P PCfm : Γ | ù σps, a, s1q must-transitions Ñmust ĎÑ, where s

a

Ý Ñmust s1 :ô @Γ P PCfm : Γ | ù σps, a, s1q

Lochau | SPL MBT | 13

SLIDE 36

Transition Modalities [LT88]

s0 s4 s5 s1 s3 s2 a rf1s a rf2s b rf1s c rf2 _ f3s c rf2 ^ f3s

fm “ f1 ^ pf2 _ f3q may-transitions Ñmay ĎÑ, where s

a

Ý Ñmay s1 :ô DΓ P PCfm : Γ | ù σps, a, s1q must-transitions Ñmust ĎÑ, where s

a

Ý Ñmust s1 :ô @Γ P PCfm : Γ | ù σps, a, s1q prohibited-transitions Û Ď Proc ˆ Act ˆ Proc, where s

a

Û s1 :ô EΓ P PCfm : Γ | ù σps, a, s1q

Lochau | SPL MBT | 13

SLIDE 37

Transition Modalities [LT88]

s0 s4 s5 s1 s3 s2 a rf1s a rf2s b rf1s c rf2 _ f3s c rf2 ^ f3s

fm “ f1 ^ pf2 _ f3q may-transitions Ñmay ĎÑ, where s

a

Ý Ñmay s1 :ô DΓ P PCfm : Γ | ù σps, a, s1q must-transitions Ñmust ĎÑ, where s

a

Ý Ñmust s1 :ô @Γ P PCfm : Γ | ù σps, a, s1q prohibited-transitions Û Ď Proc ˆ Act ˆ Proc, where s

a

Û s1 :ô EΓ P PCfm : Γ | ù σps, a, s1q Ñmust Ď Ñmay Û X Ñmay “ H

Lochau | SPL MBT | 13

SLIDE 38

F-LTS Refinement

From fm1 ĎFM fm it follows that ltsΓ 1 ĎT ltsΓ Ñ1

may Ď Ñmay

Ñmust Ď Ñ1

must

Û Ď Û1

Lochau | SPL MBT | 14

SLIDE 39

F-LTS Refinement

From fm1 ĎFM fm it follows that ltsΓ 1 ĎT ltsΓ Ñ1

may Ď Ñmay

Ñmust Ď Ñ1

must

Û Ď Û1 But: this does not hold for decorated trace semantics Set of failures increases under refinement Set of readies is not subset closed ñ May-transitions may become failures as well as readies after refinement

Lochau | SPL MBT | 14

SLIDE 40

Decorated May-Trace Semantics

A pair ptr, Xq with s0

tr

Ý Ñ s and X Ď Act is a may-failure of state s0 if for each a P Act with s

a

Ý Ñmust s1 it holds that a R X

Lochau | SPL MBT | 15

SLIDE 41

Decorated May-Trace Semantics

A pair ptr, Xq with s0

tr

Ý Ñ s and X Ď Act is a may-failure of state s0 if for each a P Act with s

a

Ý Ñmust s1 it holds that a R X A pair ptr, Xq with s0

tr

Ý Ñ s and X Ď Act is a may-ready of state s0 if (1) for each a P Act with s

a

Ý Ñmust s1 it holds that a P X, and (2) for each a P Act with s

a

Û s1 it holds that a R X

Lochau | SPL MBT | 15

SLIDE 42

Decorated May-Trace Semantics

A pair ptr, Xq with s0

tr

Ý Ñ s and X Ď Act is a may-failure of state s0 if for each a P Act with s

a

Ý Ñmust s1 it holds that a R X A pair ptr, Xq with s0

tr

Ý Ñ s and X Ď Act is a may-ready of state s0 if (1) for each a P Act with s

a

Ý Ñmust s1 it holds that a P X, and (2) for each a P Act with s

a

Û s1 it holds that a R X From fm1 ĎFM fm it follows that ltsΓ 1 Ďte-may ltsΓ holds.

Lochau | SPL MBT | 15

SLIDE 43

Decorated May-Trace Semantics

A pair ptr, Xq with s0

tr

Ý Ñ s and X Ď Act is a may-failure of state s0 if for each a P Act with s

a

Ý Ñmust s1 it holds that a R X A pair ptr, Xq with s0

tr

Ý Ñ s and X Ď Act is a may-ready of state s0 if (1) for each a P Act with s

a

Ý Ñmust s1 it holds that a P X, and (2) for each a P Act with s

a

Û s1 it holds that a R X From fm1 ĎFM fm it follows that ltsΓ 1 Ďte-may ltsΓ holds. But: full product configurations are incomparable under Ďte-may

Lochau | SPL MBT | 15

SLIDE 44

Must-Trace Semantics

A pair ptr, Xq with s0

tr

Ý Ñ s, where si

a

Ý Ñmust si`1, for 0 ď i ă n, and X Ď Act is a must-failure of state s0 if for each a P Act with s

a

Ý Ñmay s1 it holds that a R X A pair ptr, Xq with s0

tr

Ý Ñ s, where si

a

Ý Ñmust si`1, for 0 ď i ă n, and X Ď Act is a must-ready of state s0 if (1) for each a P Act with s

a

Ý Ñmust s1 it holds that a P X, and (2) there is no a1 P Act with s

a1

Ý Ñmay and not s

a1

Ý Ñmust

Lochau | SPL MBT | 16

SLIDE 45

Must-Trace Semantics

A pair ptr, Xq with s0

tr

Ý Ñ s, where si

a

Ý Ñmust si`1, for 0 ď i ă n, and X Ď Act is a must-failure of state s0 if for each a P Act with s

a

Ý Ñmay s1 it holds that a R X A pair ptr, Xq with s0

tr

Ý Ñ s, where si

a

Ý Ñmust si`1, for 0 ď i ă n, and X Ď Act is a must-ready of state s0 if (1) for each a P Act with s

a

Ý Ñmust s1 it holds that a P X, and (2) there is no a1 P Act with s

a1

Ý Ñmay and not s

a1

Ý Ñmust ñ From fm1 ĎFM fm it follows that lts Γ Ďte-must lts Γ 1 holds.

Lochau | SPL MBT | 16

SLIDE 46

Must-Trace Semantics

A pair ptr, Xq with s0

tr

Ý Ñ s, where si

a

Ý Ñmust si`1, for 0 ď i ă n, and X Ď Act is a must-failure of state s0 if for each a P Act with s

a

Ý Ñmay s1 it holds that a R X A pair ptr, Xq with s0

tr

Ý Ñ s, where si

a

Ý Ñmust si`1, for 0 ď i ă n, and X Ď Act is a must-ready of state s0 if (1) for each a P Act with s

a

Ý Ñmust s1 it holds that a P X, and (2) there is no a1 P Act with s

a1

Ý Ñmay and not s

a1

Ý Ñmust ñ From fm1 ĎFM fm it follows that lts Γ Ďte-must lts Γ 1 holds. ñ From Γ 2 “ lubpΓ, Γ 1q and TC “ Trte-mustps0, f-lts 2q it follows that ltsΓ ĎTC

te ltsΓ1 holds.

Lochau | SPL MBT | 16

SLIDE 47

F-LTS Refinement Hierarchy

F–LTS 150% tm . . . Ă PC–LTS Ă PC–LTS . . . Ă PC–LTS . . . Ă PC–LTS Ă PC–LTS x PC–LTS x PC–LTS x PC–LTS x PC–LTS Ďte–may Ďte–must Lochau | SPL MBT | 17

SLIDE 48

FM-constraint May-Trace Semantics

A trace s0

tr

Ý Ñ sn is an fm-constraint may-trace if Ź

1ďiďn σpsi´1, ai, siq |

ù fm holds A may-failure ptr, Xq is an fm-constraint may-failure if (1) s0

tr

Ý Ñ sn is an FM-constraint may-trace, and (2) Ź

aPX σpsn, a, s1q |

ù fm holds A may-ready ptr, Xq is an fm-constraint may-ready if (1) s0

tr

Ý Ñ sn is an FM-constraint may-trace, and (2) Ź

aPX σpsn, a, s1q |

ù fm holds

Lochau | SPL MBT | 18

SLIDE 49

Conclusions & Future Work

Sample implementation for trace preorder semantics [LSKL12, LLSG12] Test result reuse via test model slicing [KLB12] Future Work Variability-aware test result reuse criteria Feature-Unit testing Testing Equivalences with τ-sensitivity Ñ pl-ioco Automated SPL test suite generation

Lochau | SPL MBT | 19

SLIDE 50

Thanks for Your Attention

Any Questions?

Lochau | SPL MBT | 20

SLIDE 51

References I

Samson Abramsky. Observation Equivalence as a Testing Equivalence. Theoretical Computer Science, 53:225–241, August 1987. Gilles Bernot. Testing against Formal Specifications: A Theoretical View. In S. Abramsky and T. Maibaum, editors, TAPSOFT ’91, volume 494 of Lecture Notes in Computer Science, pages 99–119. Springer Berlin , Heidelberg, 1991. 10.1007/354053981663. Bard Bloom, Wan Fokkink, and Rob J. van Glabbeek. Precongruence Formats for Decorated Trace Semantics. ACM Transactions on Computational Logic, 5(1):26–78, Jan. 2004. Andreas Classen, Patrick Heymans, Pierre-Yves Schobbens, and Axel Legay. Symbolic Model Checking of Software Product Lines. In ICSE, pages 321–330, 2011.

Lochau | SPL MBT | 21

SLIDE 52

References II

Jochen Kamischke, Malte Lochau, and Hauke Baller. Conditioned Model Slicing of Feature-Annotated State Machines. In 4th International Workshop on Feature-Oriented Software Development (FOSD), 2012. Sascha Lity, Malte Lochau, Ina Schaefer, and Ursula Goltz. Delta-oriented Model-based SPL Regression Testing. In 3rd International IEEE Workshop on Product Line Approaches in Software Engineering (PLEASE), 2012. Malte Lochau, Ina Schaefer, Jochen Kamischke, and Sascha Lity. Incremental Model-based Testing of Delta-oriented Software Product Lines. In 6th International Conference on Tests & Proofs, 2012. Kim Guldstrand Larsen and Bent Thomsen. A Modal Process Logic. In Logic in Computer Science, 1988. LICS ’88., Proceedings of the Third Annual Symposium on, pages 203 –210, Juli 1988.

Lochau | SPL MBT | 22

SLIDE 53

References III

Rocco De Nicola and Matthew. C. B. Hennessy. Testing Equivalences for Processes. Theoretical Computer Science, pages 83–133, 1984. Jan Tretmans. Testing Concurrent Systems: A Formal Approach. In Proceedings of the 10th International Conference on Concurrency Theory, CONCUR ’99, pages 46–65, London, UK, 1999. Springer-Verlag. Mark Utting and Bruno Legeard. Practical Model-Based Testing. A Tools Approach.

M. Kaufmann, 2007.

Lochau | SPL MBT | 23