A Comfortable TestPlayer for Analyzing Statistical Usage Testing - - PowerPoint PPT Presentation

▶
a comfortable testplayer for analyzing
SMART_READER_LITE
LIVE PREVIEW

A Comfortable TestPlayer for Analyzing Statistical Usage Testing - - PowerPoint PPT Presentation

Jul 14, 2023 273 likes •708 views

A Comfortable TestPlayer for Analyzing Statistical Usage Testing Strategies Winfried Dulz (Department of Computer Science 7, University of Erlangen-Nuremberg, Germany) Outline Motivation Short History of Statistical Usage Testing

slide-1
SLIDE 1

A Comfortable TestPlayer for Analyzing Statistical Usage Testing Strategies

Winfried Dulz

(Department of Computer Science 7, University of Erlangen-Nuremberg, Germany)

slide-2
SLIDE 2

2

Outline

 Motivation

  • Short History of Statistical Usage Testing

 Tool Environment

  • yED
  • R/Eclipse StatET

 TestPlayer User Interface

  • Generation of Test Cases
  • Visualization of Test Cases

 Evaluation of Test Suites

  • Intra Test Suite Analysis
  • Inter Test Suite Analysis

 Final Remarks

Winfried Dulz FAU University Erlangen-Nuremberg

slide-3
SLIDE 3

3

Motivation

Short History of Statistical Usage Testing

  • J. A. Whittaker, J. H. Poore. Markov Analysis of Software Specications.

ACM Transactions on Software Engineering and Methodology, 1993.

  • Reviewer Leo G. Egan, Jr.: The authors introduce a new method to analyze

software specifications before design and coding begin. They assume all programmers and systems analysts are conversant with and comfortable with Markov techniques and higher mathematical analyses in general, which traditionally is not the case. I would not recommend this paper to all programmers and analysts. I would make it known and available to systems engineers who have the responsibility of designing systems that are hardware-intensive and especially software-intensive.

Winfried Dulz FAU University Erlangen-Nuremberg

slide-4
SLIDE 4

4

Motivation

Short History of Statistical Usage Testing (cont.)

 Stacy J. Prowell. Computations for Markov Chain Usage Models. Technical report, Software Engineering Institute, Carnegie-Mellon University, 2000.

  • … result of years of work by many different people, and very few results are
  • riginal. James Whittaker, Michael Thomason, and Jesse Poore did the
  • riginal work on Markov chain usage models. Gwen Walton’s research

applied mathematical programming techniques to set model probabilities under testing constraints . Jenny Morales and Dave Pearson investigated combining information across tests to improve reliability measurements. Kirk Sayre’s research provided many new and useful analytical results, and provided a framework for simulation and partition testing . Walter Gutjahr demonstrated how a Markov chain could be modified to bias test generation toward low-use critical function, and how the bias could be removed in the results .

Winfried Dulz FAU University Erlangen-Nuremberg

slide-5
SLIDE 5

5

Motivation

Short History of Statistical Usage Testing (cont.)

 Stacy J. Prowell. Jumbl: A tool for model-based statistical testing. HICSS, 2003.

  • … The J Usage Model Builder Library (JUMBL) is a Java class library and set of

command-line tools for working with usage models …

  • W. Dulz and F. Zhen. MaTeLo – Statistical Usage Testing by Annotated

Sequence Diagrams, Markov Chains and TTCN-3. IEEE QSIC, 2003.

  • … The main focus is on automatically generating a MCUM (Markov Chain

Usage Model) starting from an FDT (Formal Description Technique) in order to derive TTCN-3 (ETSI Testing and Test Control Notation version 3) compatible test case definitions …

Winfried Dulz FAU University Erlangen-Nuremberg

slide-6
SLIDE 6

6

Tool Environment

yED: MCUM editing and visualization

Winfried Dulz FAU University Erlangen-Nuremberg

slide-7
SLIDE 7

7

Tool Environment

R/Eclipse StatET

 Open source scripting language for statistical computing and graphics  Package pool to extend the basic functionality, e.g. for GUI programming

Winfried Dulz FAU University Erlangen-Nuremberg

slide-8
SLIDE 8

8

TestPlayer User Interface

TestPlayer GUI (Gtk+)

 Toolbar

  • <quit>
  • <ok>

 MCUM definition

  • <Model name>
  • <Number of test cases>
  • <Start state>
  • <End state>
  • <Profile usage>
  • <Profile name>

Winfried Dulz FAU University Erlangen-Nuremberg

slide-9
SLIDE 9

9

TestPlayer User Interface

TestPlayer GUI (cont.)

 MCUM extensions

  • <Generation.of.graph.

elements>

  • <node names>
  • <event names>
  • <probabilities>
  • <Closed.subgraph.

elements> in hierarchical models

Winfried Dulz FAU University Erlangen-Nuremberg

slide-10
SLIDE 10

10

TestPlayer User Interface

Running example

  • J. A. Whittaker, J. H.
  • Poore. Markov

Analysis of Software

  • Specications. ACM

Transactions on Software Engineering and Methodology, 1993.

Winfried Dulz FAU University Erlangen-Nuremberg

slide-11
SLIDE 11

11

TestPlayer User Interface

Uncomplete MCUM

Winfried Dulz FAU University Erlangen-Nuremberg

slide-12
SLIDE 12

12

TestPlayer User Interface

Extended MCUM

Winfried Dulz FAU University Erlangen-Nuremberg

slide-13
SLIDE 13

13

TestPlayer User Interface

Completed MCUM

Winfried Dulz FAU University Erlangen-Nuremberg

slide-14
SLIDE 14

14

TestPlayer User Interface

Open hierarchical MCUM

Winfried Dulz FAU University Erlangen-Nuremberg

slide-15
SLIDE 15

15

TestPlayer User Interface

Closed hierarchical MCUM

Winfried Dulz FAU University Erlangen-Nuremberg

slide-16
SLIDE 16

16

TestPlayer User Interface

Pruned hierarchical MCUM

Winfried Dulz FAU University Erlangen-Nuremberg

slide-17
SLIDE 17

17

TestPlayer User Interface

TestPlayer GUI (cont.)

 Test case/suite generation

  • <Generation.strategy>
  • <Coverage.strategy>
  • <nodes>
  • <transitions>
  • <Sorting.strategy>
  • <Start.number> and

<End.number> of generated test cases

Winfried Dulz FAU University Erlangen-Nuremberg

slide-18
SLIDE 18

18

TestPlayer User Interface

Winfried Dulz FAU University Erlangen-Nuremberg

Test case generation

 <Sorting.strategy>

  • <unsorted>
  • <frequency>
  • <length>
  • <p.mul>
  • <p.add>
  • <combination>

 Test case list <xyz>_<aaa>.tcl

  • <xyz>: model
  • <aaa>: number
  • f sorted test

cases

slide-19
SLIDE 19

19

TestPlayer User Interface

Winfried Dulz FAU University Erlangen-Nuremberg

Test case generation

slide-20
SLIDE 20

20

TestPlayer User Interface

Winfried Dulz FAU University Erlangen-Nuremberg

Test case visualization

slide-21
SLIDE 21

21

TestPlayer User Interface

Winfried Dulz FAU University Erlangen-Nuremberg

Test case visualization

slide-22
SLIDE 22

22

TestPlayer User Interface

Winfried Dulz FAU University Erlangen-Nuremberg

Test case visualization

slide-23
SLIDE 23

23

TestPlayer User Interface

TestPlayer GUI (cont.)

 Test suite evaluation

  • <Single.metrics>
  • <SSP>
  • <SSV>
  • <KL>
  • <SSP.N>, <SSV.N>, <KL.N>
  • <SSP.T>, <SSV.T>, <KL.T>
  • <MCUM statistics>
  • <Compare.metrics>

Winfried Dulz FAU University Erlangen-Nuremberg

slide-24
SLIDE 24

24

Evaluation of Test Suites

Winfried Dulz FAU University Erlangen-Nuremberg

 SSP

  • steady state probabilities
  • f the MCUM vs.

state frequencies of the test suite

 SSV

  • mean number of test

cases to visit a state once in the MCUM vs. mean number of test cases to visit a state once in the test suite

slide-25
SLIDE 25

25

Evaluation of Test Suites

Winfried Dulz FAU University Erlangen-Nuremberg

slide-26
SLIDE 26

26

Evaluation of Test Suites

Winfried Dulz FAU University Erlangen-Nuremberg

slide-27
SLIDE 27

27

Evaluation of Test Suites

Winfried Dulz FAU University Erlangen-Nuremberg

 KL

  • Kullback/Leibler

divergence (KL) between MCUM and test suite vs mean weighted transition deviation (MWT) between MCUM and test suite

slide-28
SLIDE 28

28

Evaluation of Test Suites

Winfried Dulz FAU University Erlangen-Nuremberg

<Sorting.strategy>=<unsorted>

slide-29
SLIDE 29

29

Evaluation of Test Suites

Winfried Dulz FAU University Erlangen-Nuremberg

<Sorting.strategy>=<unsorted> <Zoom factor>=<5>

slide-30
SLIDE 30

30

Evaluation of Test Suites

Winfried Dulz FAU University Erlangen-Nuremberg

<Sorting.strategy>=<length> <Zoom factor>=<5>

slide-31
SLIDE 31

31

Evaluation of Test Suites

Winfried Dulz FAU University Erlangen-Nuremberg

<Sorting.strategy>=<unsorted> <Number of test cases>=<100>

slide-32
SLIDE 32

32

Evaluation of Test Suites

Winfried Dulz FAU University Erlangen-Nuremberg

<Sorting.strategy>=< p.mul > <Number of test cases>=<100>

slide-33
SLIDE 33

33

Evaluation of Test Suites

Winfried Dulz FAU University Erlangen-Nuremberg

<Sorting.strategy>=< unsorted>< p.mul > <Number of test cases>=<100>

slide-34
SLIDE 34

34

Final Remarks

Winfried Dulz FAU University Erlangen-Nuremberg

 Other models

slide-35
SLIDE 35

35

Final Remarks

Winfried Dulz FAU University Erlangen-Nuremberg

 Other models

slide-36
SLIDE 36

36

Final Remarks

Winfried Dulz FAU University Erlangen-Nuremberg

 Other models

slide-37
SLIDE 37

37

Final Remarks

Winfried Dulz FAU University Erlangen-Nuremberg

 Other models

slide-38
SLIDE 38

38

Final Remarks

Winfried Dulz FAU University Erlangen-Nuremberg

 Other models

slide-39
SLIDE 39

39

Final Remarks

Winfried Dulz FAU University Erlangen-Nuremberg

 Other models

slide-40
SLIDE 40

40

Final Remarks

Winfried Dulz FAU University Erlangen-Nuremberg

 Other models

slide-41
SLIDE 41

41

Final Remarks

Winfried Dulz FAU University Erlangen-Nuremberg

 Thank you for the attention  Further information: www.testus.eu Questions ?