Potential Synergies of Theorem Proving and Model Checking for - - PowerPoint PPT Presentation

▶
potential synergies of theorem proving and model checking
SMART_READER_LITE
LIVE PREVIEW

Potential Synergies of Theorem Proving and Model Checking for - - PowerPoint PPT Presentation

Jun 29, 2023 200 likes •956 views

Potential Synergies of Theorem Proving and Model Checking for Software Product Lines um 1 , Jens Meinicke 1 , Fabian Benduhn 1 , Thomas Th Martin Hentschel 2 , Alexander von Rhein 3 , Gunter Saake 1 May 7th, 2014 1 University of Magdeburg,

slide-1
SLIDE 1

Potential Synergies of Theorem Proving and Model Checking for Software Product Lines

Thomas Th¨ um1, Jens Meinicke1, Fabian Benduhn1, Martin Hentschel2, Alexander von Rhein3, Gunter Saake1

May 7th, 2014

1 University of Magdeburg, Germany 2 University of Darmstadt, Germany 3 University of Passau, Germany

slide-2
SLIDE 2

Potential Synergies of Theorem Proving and Model Checking for Software Product Lines

Thomas Th¨ um1, Jens Meinicke1, Fabian Benduhn1, Martin Hentschel2, Alexander von Rhein3, Gunter Saake1

May 7th, 2014

1 University of Magdeburg, Germany 2 University of Darmstadt, Germany 3 University of Passau, Germany

slide-3
SLIDE 3

Variability in Single-System Engineering

David W. Stefan Tassio

  • 1. Strategy: clone-and-own, copy-and-modify, branching, . . .

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 2

slide-4
SLIDE 4

Variability in Single-System Engineering

David W. Stefan Tassio

  • 1. Strategy: clone-and-own, copy-and-modify, branching, . . .

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 2

slide-5
SLIDE 5

Variability in Single-System Engineering

David W. Stefan Tassio

  • 1. Strategy: clone-and-own, copy-and-modify, branching, . . .

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 2

slide-6
SLIDE 6

Variability in Single-System Engineering

David W. Stefan Tassio

  • 1. Strategy: clone-and-own, copy-and-modify, branching, . . .

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 2

slide-7
SLIDE 7

Variability in Single-System Engineering

David W. Stefan Tassio

  • 1. Strategy: clone-and-own, copy-and-modify, branching, . . .

Problems: creation, bug fixes, extension, . . . [code-clones problems]

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 2

slide-8
SLIDE 8

Variability in Single-System Engineering

Max

  • 2. Strategy: runtime variability/parameters, all-in-one-solution,

swiss army knife (German: Eierlegende Wollmilchsau), . . .

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 3

slide-9
SLIDE 9

Variability in Single-System Engineering

Max

  • 2. Strategy: runtime variability/parameters, all-in-one-solution,

swiss army knife (German: Eierlegende Wollmilchsau), . . . Problems: footprint, performance, safety, security, . . . [unused functionality]

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 3

slide-10
SLIDE 10

Variability in Product-Line Engineering

Compile-time variability: components, plug-ins, feature modules, aspects, build scripts, preprocessors, virtual separation, . . .

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 4

slide-11
SLIDE 11

Variability in Product-Line Engineering

Compile-time variability: components, plug-ins, feature modules, aspects, build scripts, preprocessors, virtual separation, . . .

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 4

slide-12
SLIDE 12

Variability in Product-Line Engineering

Compile-time variability: components, plug-ins, feature modules, aspects, build scripts, preprocessors, virtual separation, . . .

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 4

slide-13
SLIDE 13

Variability in Product-Line Engineering

Compile-time variability: components, plug-ins, feature modules, aspects, build scripts, preprocessors, virtual separation, . . .

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 4

slide-14
SLIDE 14

Variability in Product-Line Engineering

Compile-time variability: components, plug-ins, feature modules, aspects, build scripts, preprocessors, virtual separation, . . .

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 4

slide-15
SLIDE 15

Variability in Product-Line Engineering

Compile-time variability: components, plug-ins, feature modules, aspects, build scripts, preprocessors, virtual separation, . . . Challenges: testing, verification, specification, . . .

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 4

slide-16
SLIDE 16

Transition between Variability Representations

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 5

slide-17
SLIDE 17

Transition between Variability Representations

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 5

slide-18
SLIDE 18

Transition between Variability Representations

David W. Stefan Daniela Sheng Tassio

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 5

slide-19
SLIDE 19

Transition between Variability Representations

David W. Stefan Daniela Sheng Tassio Max

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 5

slide-20
SLIDE 20

Transition between Variability Representations

David W. Stefan Daniela Sheng Tassio Max High manual effort

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 5

slide-21
SLIDE 21

Transition between Variability Representations

David W. Stefan Daniela Sheng Tassio Max Christoph High manual effort vs. automatic generation

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 5

slide-22
SLIDE 22

Transition between Variability Representations

David W. Stefan Daniela Sheng Tassio Max Christoph High manual effort vs. automatic generation

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 5

slide-23
SLIDE 23

Transition between Variability Representations

David W. Stefan Daniela Sheng Tassio Max Christoph Norbert Thomas High manual effort vs. automatic generation

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 5

slide-24
SLIDE 24

Transition between Variability Representations

David W. Stefan Daniela Sheng Tassio Max Christoph Norbert Thomas Mustafa Alex G. Olaf High manual effort vs. automatic generation

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 5

slide-25
SLIDE 25

Transition between Variability Representations

David W. Stefan Daniela Sheng Tassio Max Christoph Norbert Thomas Mustafa Alex G. Olaf

Fl´ avio Iago Bruno Sergiy Malte Bo Wolfram Christian Sarah Sven Johannes Claus Thorsten Sandro

High manual effort vs. automatic generation

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 5

slide-26
SLIDE 26

Transition between Variability Representations

David W. Stefan Daniela Sheng Tassio Max Christoph Norbert Thomas Mustafa Alex G. Olaf

Fl´ avio Iago Bruno Sergiy Malte Bo Wolfram Christian Sarah Sven Johannes Claus Thorsten Sandro

I ’ m s

  • r

r y f

  • r

: T i m F a b i a n D e a n A l e x v . R . M a r t i n D a v i d G . High manual effort vs. automatic generation

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 5

slide-27
SLIDE 27

Variability Encoding

Translating compile-time into run-time/load-time variability for:

◮ Model checking — Post and Sinz [2008], Apel et al. [2011],

Classen et al. [2011], Apel et al. [2013]

◮ Theorem proving — Th¨

um et al. [2012]

◮ Testing — K¨

astner et al. [2012]

◮ Predicting non-functional properties — Siegmund et al. [2013]

Norbert

◮ . . .

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 6

slide-28
SLIDE 28

Variability Encoding

Translating compile-time into run-time/load-time variability for:

◮ Model checking — Post and Sinz [2008], Apel et al. [2011],

Classen et al. [2011], Apel et al. [2013]

◮ Theorem proving — Th¨

um et al. [2012]

◮ Testing — K¨

astner et al. [2012]

◮ Predicting non-functional properties — Siegmund et al. [2013]

Norbert

◮ . . .

We can reuse tools from single-system engineering!

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 6

slide-29
SLIDE 29

Theorem Proving vs. Model Checking

◮ Deductive reasoning ◮ Code translated into

first-order logic

◮ Transformation of logic

formulas

◮ Methods in isolation ◮ Applicable to incomplete

code

◮ Theorem provers:

KeY, Coq, . . .

◮ Exhaustive search ◮ Specification translated into

runtime assertions

◮ Code (symbolically)

executed

◮ Test scenarios ◮ Applicable to incomplete

specifications

◮ Model checkers:

JPF, SPIN, . . .

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 7

slide-30
SLIDE 30

Theorem Proving vs. Model Checking

◮ Deductive reasoning ◮ Code translated into

first-order logic

◮ Transformation of logic

formulas

◮ Methods in isolation ◮ Applicable to incomplete

code

◮ Theorem provers:

KeY, Coq, . . .

◮ Exhaustive search ◮ Specification translated into

runtime assertions

◮ Code (symbolically)

executed

◮ Test scenarios ◮ Applicable to incomplete

specifications

◮ Model checkers:

JPF, SPIN, . . . What is more efficient/effective?

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 7

slide-31
SLIDE 31

Empirical Comparison

BankAccount DailyLimit Interest InterestEstimation Overdraft Logging TransactionLog CreditWorthiness Lock Transaction Legend: Optional Logging ∧ Transaction ⇔ TransactionLog

◮ Feature modules with feature-oriented contracts ◮ Dependent variables: verification time, effectiveness ◮ Independent variables: number of features, number of defects

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 8

slide-32
SLIDE 32

Automatic Generation of Defects

Typical mutations from mutation testing — Jia and Harman [2011] Source/Target Target/Source In Java In JML < > 6 <= >= 2 17 != == 39 && || 11 ==> <==> 27 +

  • 7

8 * / 11 +=

  • =

4 false true 27 1 To simulate different stages during development

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 9

slide-33
SLIDE 33

Effectiveness of Theorem Proving and Model Checking

2 4 6 8 10 20 40 60 80 100 1−3 Defects Number of effective runs (in %) 2 4 6 8 10 4−10 Defects KeY JPF KeY or JPF Key and JPF Number of features Number of features Number of features Number of features

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 10

slide-34
SLIDE 34

Performance of Theorem Proving and Model Checking

  • 0.2

1.0 5.0 50.0 500.0 KeY, 0 Defects (in s, logarithmic)

  • KeY, 1−3 Defects
  • KeY, 4−10 Defects
  • 1

3 5 7 9 0.5 0.6 0.8 1.0 JPF , 0 Defects Verification time

  • 1

3 5 7 9 JPF , 1−3 Defects Number of features

  • 1

3 5 7 9 JPF , 4−10 Defects

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 11

slide-35
SLIDE 35

Combining Theorem Proving and Model Checking

fix defect fix defect verified defect found no defect found

  • pen proof

all proofs closed

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 12

slide-36
SLIDE 36

Efficiency of Theorem Proving and Model Checking

2 4 6 8 10 50 100 150 200 1−3 Defects KeY JPF Synergy Effectiveness / verification time (in %/s) 2 4 6 8 10 4−10 Defects Number of features Number of features Number of features Number of features

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 13

slide-37
SLIDE 37

Conclusion

◮ Theorem proving and model checking are more effective and

efficient for many than for few defects

◮ Model checking is more efficient, but less effective ◮ Combination improves efficiency and effectiveness ◮ Combination especially more effective for few defects

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 14

slide-38
SLIDE 38

FOSD Meeting 2014

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 15

slide-39
SLIDE 39
slide-40
SLIDE 40

References I

Sven Apel, Hendrik Speidel, Philipp Wendler, Alexander von Rhein, and Dirk Beyer. Detection of Feature Interactions using Feature-Aware Verification. In Proc. Int’l

  • Conf. Automated Software Engineering (ASE), pages 372–375, Washington, DC,

USA, 2011. IEEE. Sven Apel, Alexander von Rhein, Philipp Wendler, Armin Gr¨

  • ßlinger, and Dirk Beyer.

Strategies for Product-Line Verification: Case Studies and Experiments. In Proc. Int’l Conf. Software Engineering (ICSE), pages 482–491, Piscataway, NJ, USA, May 2013. IEEE. Andreas Classen, Patrick Heymans, Pierre-Yves Schobbens, and Axel Legay. Symbolic Model Checking of Software Product Lines. In Proc. Int’l Conf. Software Engineering (ICSE), pages 321–330, New York, NY, USA, 2011. ACM. ISBN 978-1-4503-0445-0. doi: http://doi.acm.org/10.1145/1985793.1985838. Yue Jia and Mark Harman. An Analysis and Survey of the Development of Mutation

  • Testing. IEEE Trans. Software Engineering (TSE), 37(5):649–678, September
  • 2011. ISSN 0098-5589. doi: 10.1109/TSE.2010.62.

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 17

slide-41
SLIDE 41

References II

Christian K¨ astner, Alexander von Rhein, Sebastian Erdweg, Jonas Pusch, Sven Apel, Tillmann Rendel, and Klaus Ostermann. Toward Variability-Aware Testing. In

  • Proc. Int’l Workshop Feature-Oriented Software Development (FOSD), pages 1–8,

New York, NY, USA, September 2012. ACM. ISBN 978-1-4503-1309-4. doi: 10.1145/2377816.2377817. Hendrik Post and Carsten Sinz. Configuration Lifting: Software Verification meets Software Configuration. In Proc. Int’l Conf. Automated Software Engineering (ASE), pages 347–350, Washington, DC, USA, 2008. IEEE. Norbert Siegmund, Alexander von Rhein, and Sven Apel. Family-based Performance

  • Measurement. In Proc. Int’l Conf. Generative Programming and Component

Engineering (GPCE), pages 95–104, New York, NY, USA, 2013. ACM. ISBN 978-1-4503-2373-4. doi: 10.1145/2517208.2517209. Thomas Th¨ um, Ina Schaefer, Sven Apel, and Martin Hentschel. Family-Based Deductive Verification of Software Product Lines. In Proc. Int’l Conf. Generative Programming and Component Engineering (GPCE), pages 11–20, New York, NY, USA, September 2012. ACM. ISBN 978-1-4503-1129-8. doi: 10.1145/2371401.2371404.

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 18

slide-42
SLIDE 42

Product-Based Specification

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 19

slide-43
SLIDE 43

Product-Based Specification

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 19

slide-44
SLIDE 44

Product-Based Specification

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 19

slide-45
SLIDE 45

Product-Based Specification

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 19

slide-46
SLIDE 46

Product-Based Specification

Problems: specification clones, scalability

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 19

slide-47
SLIDE 47

Product-Based Specification

?

Problems: specification clones, scalability

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 19

slide-48
SLIDE 48

Feature-Based Specification

FASE’12, CSUR’14:

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 20

slide-49
SLIDE 49

Feature-Based Specification

FASE’12, CSUR’14:

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 20

slide-50
SLIDE 50

Feature-Based Specification

FASE’12, CSUR’14:

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 20

slide-51
SLIDE 51

Feature-Based Specification

FASE’12, CSUR’14:

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 20

slide-52
SLIDE 52

Feature-Based Specification

FASE’12, CSUR’14:

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 20

slide-53
SLIDE 53

Feature-Based Specification

FASE’12, CSUR’14:

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 20

slide-54
SLIDE 54

Feature-Based Specification

FASE’12, CSUR’14:

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 20

slide-55
SLIDE 55

Family-Based Specification

CSUR’14:

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 21

slide-56
SLIDE 56

Family-Based Specification

CSUR’14:

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 21

slide-57
SLIDE 57

Family-Based Specification

CSUR’14:

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 21

slide-58
SLIDE 58

Product-Based Analysis

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 22

slide-59
SLIDE 59

Product-Based Analysis

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 22

slide-60
SLIDE 60

Product-Based Analysis

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 22

slide-61
SLIDE 61

Product-Based Analysis

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 22

slide-62
SLIDE 62

Product-Based Analysis

Problems: redundant analysis, scalability

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 22

slide-63
SLIDE 63

Product-Based Analysis

?

Problems: redundant analysis, scalability

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 22

slide-64
SLIDE 64

Feature-Based Analysis

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 23

slide-65
SLIDE 65

Feature-Based Analysis

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 23

slide-66
SLIDE 66

Feature-Based Analysis

Limitation: only compositional properties

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 23

slide-67
SLIDE 67

Family-Based Analysis

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 24

slide-68
SLIDE 68

Family-Based Analysis

Automatic (!) transition of compile-time into runtime variability

  • nly for analysis

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 24

slide-69
SLIDE 69

Family-Based Analysis

Automatic (!) transition of compile-time into runtime variability

  • nly for analysis

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 24

slide-70
SLIDE 70

Family-Based Analysis

Automatic (!) transition of compile-time into runtime variability

  • nly for analysis

Enables reuse of analysis tools from single-system engineering

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 24

slide-71
SLIDE 71

Implementation vs. Specification vs. Analysis

Possible combinations of the strategies:

  • Impl. \ Spec.

Product-based Family-based Feature-based Product-based Family-based Feature-based

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 25

slide-72
SLIDE 72

Implementation vs. Specification vs. Analysis

Possible combinations of the strategies:

  • Impl. \ Spec.

Product-based Family-based Feature-based Product-based P P P Family-based P P P Feature-based P P P

Legend: P/F/f - product/family/feature-based analysis

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 25

slide-73
SLIDE 73

Implementation vs. Specification vs. Analysis

Possible combinations of the strategies:

  • Impl. \ Spec.

Product-based Family-based Feature-based Product-based P P P Family-based P P F P F Feature-based P P F P F

Legend: P/F/f - product/family/feature-based analysis

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 25

slide-74
SLIDE 74

Implementation vs. Specification vs. Analysis

Possible combinations of the strategies:

  • Impl. \ Spec.

Product-based Family-based Feature-based Product-based P P P Family-based P P F P F Feature-based P P F P F f

Legend: P/F/f - product/family/feature-based analysis

Thomas Th¨ um et al. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines 25