Using Synthetic Test Suites to Empirically Compare Search-Based and - - PowerPoint PPT Presentation

▶
using synthetic test suites to empirically compare search
SMART_READER_LITE
LIVE PREVIEW

Using Synthetic Test Suites to Empirically Compare Search-Based and - - PowerPoint PPT Presentation

Oct 23, 2022 34 likes •670 views

Using Synthetic Test Suites to Empirically Compare Search-Based and Greedy Prioritizers Zachary Williams Gregory M. Kapfhammer Department of Computer Science Allegheny College http://www.cs.allegheny.edu/ Genetic and Evolutionary Computation

slide-1
SLIDE 1

Using Synthetic Test Suites to Empirically Compare Search-Based and Greedy Prioritizers Zachary Williams

Gregory M. Kapfhammer Department of Computer Science Allegheny College

http://www.cs.allegheny.edu/

Genetic and Evolutionary Computation Conference Late Breaking Abstract Workshop July 2010

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 1 / 11

slide-2
SLIDE 2

Important Contributions

Synthetic Test Suites Detailed Empirical Study

Use synthetic test suites to empirically evaluate the efficiency and effectiveness of search-based and greedy prioritizers

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 2 / 11

slide-3
SLIDE 3

Important Contributions

Synthetic Test Suites Detailed Empirical Study

Use synthetic test suites to empirically evaluate the efficiency and effectiveness of search-based and greedy prioritizers

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 2 / 11

slide-4
SLIDE 4

Important Contributions

Synthetic Test Suites

Prioritization Technique Execution Time (sec) 20 40 60 80 100 GRD HC

0.5 1 1.5 200

20 40 60 80 100

0.5 1 1.5 200

20 40 60 80 100

0.5 1 1.5 200

Detailed Empirical Study

Use synthetic test suites to empirically evaluate the efficiency and effectiveness of search-based and greedy prioritizers

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 2 / 11

slide-5
SLIDE 5

Important Contributions

Synthetic Test Suites

Prioritization Technique Execution Time (sec) 20 40 60 80 100 GRD HC

0.5 1 1.5 200

20 40 60 80 100

0.5 1 1.5 200

20 40 60 80 100

0.5 1 1.5 200

Detailed Empirical Study

Use synthetic test suites to empirically evaluate the efficiency and effectiveness of search-based and greedy prioritizers

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 2 / 11

slide-6
SLIDE 6

Overview of Regression Testing

P Correct programing defect

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11

slide-7
SLIDE 7

Overview of Regression Testing

P T Correct programing defect

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11

slide-8
SLIDE 8

Overview of Regression Testing

P T Execute Test Suite Correct programing defect

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11

slide-9
SLIDE 9

Overview of Regression Testing

P T Execute Test Suite Report Correct programing defect

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11

slide-10
SLIDE 10

Overview of Regression Testing

P T Execute Test Suite P Correct programing defect

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11

slide-11
SLIDE 11

Overview of Regression Testing

P T Execute Test Suite P Report Correct programing defect

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11

slide-12
SLIDE 12

Overview of Regression Testing

P T Execute Test Suite P Add new functionallity

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11

slide-13
SLIDE 13

Overview of Regression Testing

P T Execute Test Suite P Report Add new functionallity

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11

slide-14
SLIDE 14

Overview of Regression Testing

P T Execute Test Suite P T P Modify test suite

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11

slide-15
SLIDE 15

Overview of Regression Testing

P T Execute Test Suite P T P Report Modify test suite

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11

slide-16
SLIDE 16

Overview of Regression Testing

P T Execute Test Suite P T P Report Complete retesting is often prohibitively expensive

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11

slide-17
SLIDE 17

Regression Test Suite Prioritization

R Requirements necessitate the coverage of the state and/or structure of a program under test

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

slide-18
SLIDE 18

Regression Test Suite Prioritization

R R3 R4 R2 R1 R5 Requirements necessitate the coverage of the state and/or structure of a program under test

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

slide-19
SLIDE 19

Regression Test Suite Prioritization

R R3 R4 R2 R1 R5 Requirements necessitate the coverage of the state and/or structure of a program under test

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

slide-20
SLIDE 20

Regression Test Suite Prioritization

R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 Each test covers specific requirements in a certain amount

  • f time and thus the ordering is critical

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

slide-21
SLIDE 21

Regression Test Suite Prioritization

R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 R3 R2 T2 R4 Each test covers specific requirements in a certain amount

  • f time and thus the ordering is critical

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

slide-22
SLIDE 22

Regression Test Suite Prioritization

R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 T1 T2 T3 T4 T5 T6 T7 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

slide-23
SLIDE 23

Regression Test Suite Prioritization

R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 T1 T2 T3 T4 T5 T6 T7 T2 T3 T5 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

slide-24
SLIDE 24

Regression Test Suite Prioritization

R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 T1 T2 T3 T4 T5 T6 T7 T2 T3 T5 T2 T3 T5 T1 T4 T6 T7 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

slide-25
SLIDE 25

Regression Test Suite Prioritization

R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 T1 T2 T3 T4 T5 T6 T7 T2 T3 T5 T2 T3 T5 T1 T4 T6 T7 T6 T7 T2 T3 T5 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

slide-26
SLIDE 26

Regression Test Suite Prioritization

R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 T1 T2 T3 T4 T5 T6 T7 T2 T3 T5 T2 T3 T5 T1 T4 T6 T7 T6 T7 T2 T3 T5 T6 T7 T1 T4 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

slide-27
SLIDE 27

Regression Test Suite Prioritization

R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 T1 T2 T3 T4 T5 T6 T7 T2 T3 T5 T2 T3 T5 T1 T4 T6 T7 T6 T7 T2 T3 T5 T6 T7 T1 T4 T6 T7 T1 T4 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

slide-28
SLIDE 28

Regression Test Suite Prioritization

R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 T1 T2 T3 T4 T5 T6 T7 T2 T3 T5 T2 T3 T5 T1 T4 T6 T7 T6 T7 T2 T3 T5 T6 T7 T1 T4 T6 T7 T1 T4 T1 T4 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

slide-29
SLIDE 29

Regression Test Suite Prioritization

R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 T1 T2 T3 T4 T5 T6 T7 T2 T3 T5 T2 T3 T5 T1 T4 T6 T7 T6 T7 T2 T3 T5 T6 T7 T1 T4 T6 T7 T1 T4 T1 T4 T1 T4 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

slide-30
SLIDE 30

Regression Test Suite Prioritization

R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 T1 T2 T3 T4 T5 T6 T7 T2 T3 T5 T2 T3 T5 T1 T4 T6 T7 T6 T7 T2 T3 T5 T6 T7 T1 T4 T6 T7 T1 T4 T1 T4 T1 T4 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

slide-31
SLIDE 31

Regression Test Suite Prioritization

R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 T1 T2 T3 T4 T5 T6 T7 T2 T3 T5 T2 T3 T5 T1 T4 T6 T7 T6 T7 T2 T3 T5 T6 T7 T1 T4 T6 T7 T1 T4 T1 T4 T1 T4 Testers can use greedy (Rothermel et al. TSE 2001) and search-based (Li et al. TSE 2007) methods to reorder suites

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

slide-32
SLIDE 32

Regression Test Suite Prioritization

R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 T1 T2 T3 T4 T5 T6 T7 T2 T3 T5 T2 T3 T5 T1 T4 T6 T7 T6 T7 T2 T3 T5 T6 T7 T1 T4 T6 T7 T1 T4 T1 T4 T1 T4 QUESTION: Which prioritization technique is the best?

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

slide-33
SLIDE 33

Existing Prioritization Techniques

Greedy approaches select the next best test case Hill climbers search the state space for improved orderings

Conventional wisdom dictates that greedy generally out performs hill climbing in terms of both efficiency and effectiveness

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 5 / 11

slide-34
SLIDE 34

Existing Prioritization Techniques

Greedy approaches select the next best test case Hill climbers search the state space for improved orderings

Conventional wisdom dictates that greedy generally out performs hill climbing in terms of both efficiency and effectiveness

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 5 / 11

slide-35
SLIDE 35

Existing Prioritization Techniques

Greedy approaches select the next best test case Hill climbers search the state space for improved orderings

Conventional wisdom dictates that greedy generally out performs hill climbing in terms of both efficiency and effectiveness

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 5 / 11

slide-36
SLIDE 36

Existing Prioritization Techniques

Greedy approaches select the next best test case Hill climbers search the state space for improved orderings

Conventional wisdom dictates that greedy generally out performs hill climbing in terms of both efficiency and effectiveness

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 5 / 11

slide-37
SLIDE 37

Conducting an Empirical Evaluation

Determine Problem

These highlighted tasks are manual, expensive, and prone to error

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 6 / 11

slide-38
SLIDE 38

Conducting an Empirical Evaluation

Determine Problem Download Case Studies

These highlighted tasks are manual, expensive, and prone to error

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 6 / 11

slide-39
SLIDE 39

Conducting an Empirical Evaluation

Determine Problem Download Case Studies

Configure Applica- tions and Test Suites

These highlighted tasks are manual, expensive, and prone to error

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 6 / 11

slide-40
SLIDE 40

Conducting an Empirical Evaluation

Determine Problem Download Case Studies

Configure Applica- tions and Test Suites

Produce Coverage Report

These highlighted tasks are manual, expensive, and prone to error

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 6 / 11

slide-41
SLIDE 41

Conducting an Empirical Evaluation

Determine Problem Download Case Studies

Configure Applica- tions and Test Suites

Produce Coverage Report Conduct Experiment

These highlighted tasks are manual, expensive, and prone to error

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 6 / 11

slide-42
SLIDE 42

Conducting an Empirical Evaluation

Determine Problem Download Case Studies

Configure Applica- tions and Test Suites

Produce Coverage Report Conduct Experiment Download Case Study

Configure Applica- tions and Test Suites

Produce Coverage Report

These highlighted tasks are manual, expensive, and prone to error

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 6 / 11

slide-43
SLIDE 43

Conducting an Empirical Evaluation

Determine Problem Download Case Studies

Configure Applica- tions and Test Suites

Produce Coverage Report Conduct Experiment

Generate Synthetic Test Suites

Synthetically generating a test suite is automated, effective, and efficient

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 6 / 11

slide-44
SLIDE 44

Conducting an Empirical Evaluation

Determine Problem Download Case Studies

Configure Applica- tions and Test Suites

Produce Coverage Report Conduct Experiment

Generate Synthetic Test Suites Generate Synthetic Test Suites

Synthetically generating a test suite is automated, effective, and efficient

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 6 / 11

slide-45
SLIDE 45

Generating Synthetic Test Suite

Tests

The total number of tests controls how many tests the suite will contain

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 7 / 11

slide-46
SLIDE 46

Generating Synthetic Test Suite

Tests Requirements

The total number of requirements governs how many requirements the test suite will cover

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 7 / 11

slide-47
SLIDE 47

Generating Synthetic Test Suite

Tests Requirements Coverage Points

The total number of coverage points controls how many unique test-requirement pairs the test suite will contain

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 7 / 11

slide-48
SLIDE 48

Generating Synthetic Test Suite

Tests Requirements Coverage Points Balancing Approach

The balancing configuration dictates how the coverage points will be distributed in the synthetic test suite

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 7 / 11

slide-49
SLIDE 49

Generating Synthetic Test Suite

Tests Requirements Coverage Points Balancing Approach

Synthetic Test Suite Generator Our empirical results show that synthetic generation takes less then 0.2 seconds for extremely large test suites

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 7 / 11

slide-50
SLIDE 50

Generating Synthetic Test Suite

Tests Requirements Coverage Points Balancing Approach

Synthetic Test Suite Generator

Synthetic Test Suite (Coverage Report)

Contains information concerning the requirements covered and the execution time of each test

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 7 / 11

slide-51
SLIDE 51

Empirical Results – Prioritizer Efficiency

Prioritization Technique Execution Time (sec) 20 40 60 80 100 GRD HC

0.5 1 1.5 200

GRD HC

0.5 1 1.5 200

GRD HC

0.5 1 1.5 200

HC demonstrated to be more efficient then GRD for large test suites

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 8 / 11

slide-52
SLIDE 52

Empirical Results – Prioritizer Efficiency

Prioritization Technique Execution Time (sec) 20 40 60 80 100 GRD HC

0.5 1 1.5 200

GRD HC

0.5 1 1.5 200

GRD HC

0.5 1 1.5 200

HC demonstrated to be more efficient then GRD for large test suites

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 8 / 11

slide-53
SLIDE 53

Empirical Results – Prioritizer Effectiveness

Prioritization Technique Coverage Effectiveness (CE) 0.80 0.85 0.90 0.95 1.00 GRD HC

1.5 200

0.1 0.25 0.5

As the amount of coverage points in the test suite increases the performance of HC becomes comparable to that of GRD

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 9 / 11

slide-54
SLIDE 54

Empirical Results – Prioritizer Effectiveness

Prioritization Technique Coverage Effectiveness (CE) 0.80 0.85 0.90 0.95 1.00 GRD HC

1.5 200

0.1 0.25 0.5

As the amount of coverage points in the test suite increases the performance of HC becomes comparable to that of GRD

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 9 / 11

slide-55
SLIDE 55

Empirical Results – Prioritizer Effectiveness

Prioritization Technique Coverage Effectiveness (CE) 0.80 0.85 0.90 0.95 1.00 GRD HC

0.5 1 1.5 2 50 100 200

GRD HC

0.5 1 1.5 2 50 100 200

GRD HC

0.5 1 1.5 2 50 100 200

GRD HC

0.5 1 1.5 2 50 100 200 0.5 1 1.5 2 50 100 200 0.5 1 1.5 2 50 100 200 0.5 1 1.5 2 50 100 200

0.80 0.85 0.90 0.95 1.00

0.5 1 1.5 2 50 100 200

0.80 0.85 0.90 0.95 1.00

0.5 1 1.5 2 50 100 200 0.5 1 1.5 2 50 100 200 0.5 1 1.5 2 50 100 200 0.5 1 1.5 2 50 100 200

0.1 0.25 0.5

The trend is evident over a wide range of experimental configurations

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 10 / 11

slide-56
SLIDE 56

Empirical Results – Prioritizer Effectiveness

Prioritization Technique Coverage Effectiveness (CE) 0.80 0.85 0.90 0.95 1.00 GRD HC

0.5 1 1.5 2 50 100 200

GRD HC

0.5 1 1.5 2 50 100 200

GRD HC

0.5 1 1.5 2 50 100 200

GRD HC

0.5 1 1.5 2 50 100 200 0.5 1 1.5 2 50 100 200 0.5 1 1.5 2 50 100 200 0.5 1 1.5 2 50 100 200

0.80 0.85 0.90 0.95 1.00

0.5 1 1.5 2 50 100 200

0.80 0.85 0.90 0.95 1.00

0.5 1 1.5 2 50 100 200 0.5 1 1.5 2 50 100 200 0.5 1 1.5 2 50 100 200 0.5 1 1.5 2 50 100 200

0.1 0.25 0.5

The trend is evident over a wide range of experimental configurations

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 10 / 11

slide-57
SLIDE 57

Conclusion and Future Work

Conclusion Synthetic test suite generation is efficient Enable the identification of fundemental trade-offs Future Work Apply our technique to genetic and other algorithms Implement and evaluate new and different synthetic generators http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 11 / 11

slide-58
SLIDE 58

Conclusion and Future Work

Conclusion Synthetic test suite generation is efficient Enable the identification of fundemental trade-offs Future Work Apply our technique to genetic and other algorithms Implement and evaluate new and different synthetic generators http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 11 / 11

slide-59
SLIDE 59

Conclusion and Future Work

Conclusion Synthetic test suite generation is efficient Enable the identification of fundemental trade-offs Future Work Apply our technique to genetic and other algorithms Implement and evaluate new and different synthetic generators http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 11 / 11

slide-60
SLIDE 60

Conclusion and Future Work

Conclusion Synthetic test suite generation is efficient Enable the identification of fundemental trade-offs Future Work Apply our technique to genetic and other algorithms Implement and evaluate new and different synthetic generators http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 11 / 11

slide-61
SLIDE 61

Conclusion and Future Work

Conclusion Synthetic test suite generation is efficient Enable the identification of fundemental trade-offs Future Work Apply our technique to genetic and other algorithms Implement and evaluate new and different synthetic generators http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 11 / 11

slide-62
SLIDE 62

Conclusion and Future Work

Conclusion Synthetic test suite generation is efficient Enable the identification of fundemental trade-offs Future Work Apply our technique to genetic and other algorithms Implement and evaluate new and different synthetic generators http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 11 / 11

slide-63
SLIDE 63

Conclusion and Future Work

Conclusion Synthetic test suite generation is efficient Enable the identification of fundemental trade-offs Future Work Apply our technique to genetic and other algorithms Implement and evaluate new and different synthetic generators http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/

Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 11 / 11