Systematic Analysis of testing-related Systematic Analysis of - - PowerPoint PPT Presentation

▶
systematic analysis of testing related systematic
SMART_READER_LITE
LIVE PREVIEW

Systematic Analysis of testing-related Systematic Analysis of - - PowerPoint PPT Presentation

Apr 17, 2023 311 likes •920 views

Systematic Analysis of testing-related Systematic Analysis of testing-related publications concerning reprocucibility publications concerning reprocucibility and comparability and comparability Bachelor's Thesis Defense by Artur Solomonik

slide-1
SLIDE 1

Systematic Analysis of testing-related Systematic Analysis of testing-related publications concerning reprocucibility publications concerning reprocucibility and comparability and comparability

Bachelor's Thesis Defense by Artur Solomonik Bachelor's Thesis Defense by Artur Solomonik

Referees: Prof. Dr. Norbert Siegmund, Prof. Dr. Martin Potthast Referees: Prof. Dr. Norbert Siegmund, Prof. Dr. Martin Potthast

slide-2
SLIDE 2

Software Testing Software Testing

slide-3
SLIDE 3

Software Testing Life Cycle Software Testing Life Cycle

slide-4
SLIDE 4

Software Testing Life Cycle Software Testing Life Cycle

slide-5
SLIDE 5

Software Testing Life Cycle Software Testing Life Cycle

slide-6
SLIDE 6

Software Testing Research Software Testing Research Generating test suites Exploration pinciples Mutation testing Executing generated test suites Prioritization and Reduction of Test Cases Automating test case creation, selection and execution Finding new approaches on organizing testing processes Testing Workow Decision Making Process When and What to Automate?

slide-7
SLIDE 7

Software Testing Research Software Testing Research Testing Levels Data-Flow Testing, Static Code Analysis | Unit Testing Backbone-, Client-Server-, Bottom-Up | Integration Testing GUI Testing, End-To-End Testing | System Testing Reliability and Stability, Chaos Testing | Acceptance Testing Execution Paradigms

slide-8
SLIDE 8

Test Execution Paradigms Test Execution Paradigms

slide-9
SLIDE 9

How do we know the testing system is working? How do we know the testing system is working?

slide-10
SLIDE 10

Empirical Software Evaluations Empirical Software Evaluations

slide-11
SLIDE 11

Evaluating result data Evaluating result data

Present the result data set and identify signicant values Connect hypotheses and results Compare related work and their ndings Argument the improvement or benets of the approach Apply suitable metrics

slide-12
SLIDE 12

Reproducibility Reproducibility

Goal: Provide the reader with every information and resource necessary to recreate the ndings presented in the paper

slide-13
SLIDE 13

Reproducibility Attributes Reproducibility Attributes

Reproduction score inuenced by data set attributes Identication: Explanation of where the data is and what it is called Description: Level of the of the explanation regarding the element Availability: Ease of accessing or obtaining the research elements Persistence: Condence in future state and availability of the elements Flexibility: Adaptability of the elements to new environments

slide-14
SLIDE 14

Reproducibility Attributes Reproducibility Attributes

Reproduction score inuenced by data set attributes Identication: Explanation of where the data is and what it is called Description: Level of the of the explanation regarding the element Availability: Ease of accessing or obtaining the research elements Persistence: Condence in future state and availability of the elements Flexibility: Adaptability of the elements to new environments

slide-15
SLIDE 15

Reproducibility Attributes Reproducibility Attributes

Reproduction score inuenced by data set attributes Identication: Explanation of where the data is and what it is called Description: Level of the of the explanation regarding the element Availability: Ease of accessing or obtaining the research elements Persistence: Condence in future state and availability of the elements Flexibility: Adaptability of the elements to new environments Varying data sources - Attributes not applicable to anything

slide-16
SLIDE 16

Comparability Comparability

Goal: Assess papers on whether empirical comparisons in the evaluation are appropriate or existent. Criteria for comprehensible evaluations Strategies of Comparison Connectivity to related work

slide-17
SLIDE 17

How can we understand the research strategies of How can we understand the research strategies of software testing publications in terms of software testing publications in terms of reproducibility and comparability? reproducibility and comparability?

slide-18
SLIDE 18

Paper Classication Paper Classication

slide-19
SLIDE 19

Data Source Data Source

Papers from 10 popular software engineering conferences (ASE, ICSE, ISSTA, ...) Additional publications from two journals (ESE, TOSEM) Frequently mentioned publications Papers from modication / renement phases

slide-20
SLIDE 20

Processed Data Set Processed Data Set

slide-21
SLIDE 21

Datei Bearbeiten Ansicht Einfügen Format Daten Tools Add-ons Hilfe

100% Nur Kommentierzugriff

year

testing papers and evaluation data

3

Freigeben Anmelden In Arbeit... paper_evaluation benchmark paper_benchmark papers

slide-22
SLIDE 22

Datei Bearbeiten Ansicht Einfügen Format Daten Tools Add-ons Hilfe

100% Nur Kommentierzugriff

year

testing papers and evaluation data

Freigeben Anmelden In Arbeit... paper_evaluation benchmark paper_benchmark papers

Raw Data Set Spreadsheet with 8060 registered papers of which 360 are classied by 23 columns 205 documented benchmarks Over 15000 bibliographic and semantic connections between records

slide-23
SLIDE 23

Classication Parameters Availability [open/closed] Data Set State [vanilla/modied] Selection Cause [...] Modication Cause [...] Sub-Check Systems [single/multiple] [named/unnamed]

slide-24
SLIDE 24

Classication Parameters Contribution [...] Choice of Metric [functionality/performance/both] Metrics [ ] Metrics

slide-25
SLIDE 25

Classication Parameters Error Creation [generation/real world/both] Error Annotation [TRUE/FALSE] Comparison [TRUE/FALSE] [former/foreign/parallel] [exclusive/inclusive]

slide-26
SLIDE 26
slide-27
SLIDE 27

Open Source vs. Closed Source Open Source vs. Closed Source

slide-28
SLIDE 28

Software Testing Evaluation Metrics Software Testing Evaluation Metrics

slide-29
SLIDE 29

Choice of Metric and Error Annotation Choice of Metric and Error Annotation

slide-30
SLIDE 30

Selection and modication causes of benchmarks Selection and modication causes of benchmarks

slide-31
SLIDE 31

Bibliographic Networks Bibliographic Networks

slide-32
SLIDE 32

Goal: Visualizing great amounts of bibliographic data, increasing the interactivity with a set of publications and creating dynamic, time-based insight on the netvork evolution.

slide-33
SLIDE 33

Current implementations of paper networks Current implementations of paper networks

Visualize the connection and inuence between authors Giving insight rather than specic values Connected over citations, bibliographic coupling, co-citations or co-authorship relations Color- and size-coding node information Geographic hierarchies

slide-34
SLIDE 34

Additions and Improvements Additions and Improvements

Benchmarks and software systems as their own entities in a network More insight on reproducibility Multidimensional graph data visualization without clutter Tailouring the visualization to a certain aspect of a publication (e.g. the evaluation)

slide-35
SLIDE 35

Visualizing bibliographic networks Visualizing bibliographic networks

slide-36
SLIDE 36

MATCH n = ({contribution: 'mutation testing'})-->() return n

 SEND  SAVE QUERY

Use Cypher Query Sort By Venue Temporary Highlighting Permanent Highlighting Color Nodes by Contribution

TESTING LITERATU OVERVIEW SYSTEM

This

test coverage test-su test generati race testing stematic testing fa mbolic execution regression testing

slide-37
SLIDE 37

MATCH n = ({contribution: 'mutation testing'})-->() return n

 SEND  SAVE QUERY

Use Cypher Query Sort By Venue Temporary Highlighting Permanent Highlighting Color Nodes by Contribution

TESTING LITERATU OVERVIEW SYSTEM

This

test coverage test-su test generati race testing stematic testing fa mbolic execution regression testing

TeLO-S D3 visualization of testing publications in a node-link force- directed graph

slide-38
SLIDE 38

MATCH n = ({contribution: 'mutation testing'})-->() return n

 SEND  SAVE QUERY

Use Cypher Query Sort By Venue Temporary Highlighting Permanent Highlighting Color Nodes by Contribution

TESTING LITERATU OVERVIEW SYSTEM

This

test coverage test-su test generati race testing stematic testing fa mbolic execution regression testing

TeLO-S D3 visualization of testing publications in a node-link force- directed graph

Cypher Query Input and Conguration Selecting sepecic nodes from the Neo4J graph data base and manipulating the layout and color-coding

slide-39
SLIDE 39

MATCH n = ({contribution: 'mutation testing'})-->() return n

 SEND  SAVE QUERY

Use Cypher Query Sort By Venue Temporary Highlighting Permanent Highlighting Color Nodes by Contribution

TESTING LITERATU OVERVIEW SYSTEM

This

test coverage test-su test generati race testing stematic testing fa mbolic execution regression testing

TeLO-S D3 visualization of testing publications in a node-link force- directed graph

Cypher Query Input and Conguration Selecting sepecic nodes from the Neo4J graph data base and manipulating the layout and color-coding Contribution Plot Immediate assessement

  • f proportions of

contribution representatives

slide-40
SLIDE 40

MATCH n = ({contribution: 'mutation testing'})-->() return n

 SEND  SAVE QUERY

Use Cypher Query Sort By Venue Temporary Highlighting Permanent Highlighting Color Nodes by Contribution

TESTING LITERATU OVERVIEW SYSTEM

This

test coverage test-su test generati race testing stematic testing fa mbolic execution regression testing

TeLO-S D3 visualization of testing publications in a node-link force- directed graph

Cypher Query Input and Conguration Selecting sepecic nodes from the Neo4J graph data base and manipulating the layout and color-coding Contribution Plot Immediate assessement

  • f proportions of

contribution representatives Node analysis Additional information on a selected node concerning his references

slide-41
SLIDE 41

Findings Findings

slide-42
SLIDE 42
slide-43
SLIDE 43
slide-44
SLIDE 44
slide-45
SLIDE 45
slide-46
SLIDE 46
slide-47
SLIDE 47
slide-48
SLIDE 48
slide-49
SLIDE 49
slide-50
SLIDE 50

Patterns Patterns

slide-51
SLIDE 51

Vanishing Point Pattern Vanishing Point Pattern

slide-52
SLIDE 52

Outsider Pattern Outsider Pattern

Loose nodes in a subgraph without any connection to other queried nodes Nodes might imply a connection to other unqueried research elds Misclassications or special cases

slide-53
SLIDE 53

Familiar Foreigner Pattern Familiar Foreigner Pattern

slide-54
SLIDE 54

Chain Pattern Chain Pattern

slide-55
SLIDE 55

Conclusion Conclusion

slide-56
SLIDE 56

Most evaluations conducted similarly Choice of benchmark varies signicantly Availability as a major reproducibility issue Solution: Dedicated sub-check systems (possibly provided by conferences) Mutation scores and coverage metrics widely used Findings of closely related papers rarely mentioned Bibliographic networks benet from sub-check system nodes and dierent relation types Comparability improves continuous improvement of research Comparing evaluations unfortunately very uncommon, yet benecial

slide-57
SLIDE 57

Future Work Future Work

slide-58
SLIDE 58

Adding referencing patterns to the visualization Classiers for testing paper classication Multiple renement cycles of the data set using relevant citations Implementation of author nodes, citation scores and bibliographic coupling Hierarchical edge bundling regarding relevancy, geography or popularity Generalization for other research topics aside from software testing

slide-59
SLIDE 59

Thank you for your attention. Thank you for your attention.