Using Synthetic Test Suites to Empirically Compare Search-Based and - PowerPoint PPT Presentation

Using Synthetic Test Suites to Empirically Compare Search-Based and Greedy Prioritizers Zachary Williams Gregory M. Kapfhammer Department of Computer Science Allegheny College http://www.cs.allegheny.edu/ Genetic and Evolutionary Computation Conference Late Breaking Abstract Workshop July 2010 Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 1 / 11

Important Contributions Synthetic Test Suites Detailed Empirical Study Use synthetic test suites to empirically evaluate the efficiency and effectiveness of search-based and greedy prioritizers Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 2 / 11

Important Contributions Synthetic Test Suites Detailed Empirical Study Use synthetic test suites to empirically evaluate the efficiency and effectiveness of search-based and greedy prioritizers Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 2 / 11

Important Contributions 200 0.5 1 1.5 100 80 60 40 20 200 Execution Time (sec) 0.5 1 1.5 100 80 60 40 20 200 0.5 1 1.5 100 80 60 40 20 GRD HC Prioritization Technique Synthetic Test Suites Detailed Empirical Study Use synthetic test suites to empirically evaluate the efficiency and effectiveness of search-based and greedy prioritizers Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 2 / 11

Important Contributions 200 0.5 1 1.5 100 80 60 40 20 200 Execution Time (sec) 0.5 1 1.5 100 80 60 40 20 200 0.5 1 1.5 100 80 60 40 20 GRD HC Prioritization Technique Synthetic Test Suites Detailed Empirical Study Use synthetic test suites to empirically evaluate the efficiency and effectiveness of search-based and greedy prioritizers Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 2 / 11

Overview of Regression Testing P Correct programing defect Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11

Overview of Regression Testing T P Correct programing defect Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11

Overview of Regression Testing T Execute Test Suite P Correct programing defect Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11

Overview of Regression Testing T Execute Report Test Suite P Correct programing defect Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11

Overview of Regression Testing T Execute Test Suite P P Correct programing defect Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11

Overview of Regression Testing T Execute Report Test Suite P P Correct programing defect Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11

Overview of Regression Testing T Execute Test Suite P P Add new functionallity Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11

Overview of Regression Testing T Execute Report Test Suite P P Add new functionallity Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11

Overview of Regression Testing T T Execute Test Suite P P P Modify test suite Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11

Overview of Regression Testing T T Execute Report Test Suite P P P Modify test suite Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11

Overview of Regression Testing T T Execute Report Test Suite P P P Complete retesting is often prohibitively expensive Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11

Regression Test Suite Prioritization R Requirements necessitate the coverage of the state and/or structure of a program under test Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

Regression Test Suite Prioritization R R2 R4 R3 R1 R5 Requirements necessitate the coverage of the state and/or structure of a program under test Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

Regression Test Suite Prioritization R R2 R4 R3 R1 R5 Requirements necessitate the coverage of the state and/or structure of a program under test Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

Regression Test Suite Prioritization R R2 R4 R3 R1 R5 T3 T5 T6 T1 T2 T4 T7 Each test covers specific requirements in a certain amount of time and thus the ordering is critical Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

Regression Test Suite Prioritization R R2 R2 R4 R4 R3 R3 R1 R5 T3 T5 T6 T1 T2 T2 T4 T7 Each test covers specific requirements in a certain amount of time and thus the ordering is critical Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

Regression Test Suite Prioritization R R2 R4 R3 R1 R5 T3 T5 T6 T1 T2 T4 T7 T1 T2 T3 T4 T5 T6 T7 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

Regression Test Suite Prioritization R R2 R4 R3 R1 R5 T3 T3 T5 T5 T6 T1 T2 T2 T4 T7 T1 T2 T3 T4 T5 T6 T7 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

Regression Test Suite Prioritization R R2 R4 R3 R1 R5 T3 T3 T5 T5 T6 T1 T2 T2 T4 T7 T1 T2 T2 T3 T3 T5 T4 T1 T4 T5 T6 T6 T7 T7 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

Regression Test Suite Prioritization R R2 R4 R3 R1 R5 T3 T3 T3 T5 T5 T5 T6 T6 T1 T2 T2 T2 T4 T7 T7 T2 T1 T2 T3 T3 T5 T4 T1 T5 T4 T6 T6 T7 T7 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

Regression Test Suite Prioritization R R2 R4 R3 R1 R5 T3 T3 T3 T5 T5 T5 T6 T6 T1 T2 T2 T2 T4 T7 T7 T2 T1 T3 T2 T5 T3 T4 T1 T6 T4 T7 T5 T1 T6 T6 T7 T7 T4 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

Regression Test Suite Prioritization R R2 R4 R3 R1 R5 T3 T3 T3 T5 T5 T5 T6 T6 T6 T1 T1 T2 T2 T2 T4 T4 T7 T7 T7 T2 T1 T3 T2 T5 T3 T4 T1 T6 T4 T7 T5 T6 T6 T1 T4 T7 T7 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

Regression Test Suite Prioritization R R2 R4 R3 R1 R5 T3 T3 T3 T5 T5 T5 T6 T6 T6 T1 T1 T2 T2 T2 T4 T4 T7 T7 T7 T2 T1 T3 T2 T5 T3 T1 T6 T4 T5 T4 T7 T1 T1 T6 T6 T4 T7 T4 T7 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

Regression Test Suite Prioritization R R2 R4 R3 R1 R5 T3 T3 T3 T5 T5 T5 T6 T6 T6 T1 T1 T1 T2 T2 T2 T4 T4 T4 T7 T7 T7 T2 T1 T3 T2 T3 T5 T6 T1 T4 T7 T4 T5 T1 T6 T6 T1 T7 T4 T7 T4 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

Regression Test Suite Prioritization R R2 R4 R3 R1 R5 T3 T3 T3 T5 T5 T5 T6 T6 T6 T1 T1 T1 T2 T2 T2 T4 T4 T4 T7 T7 T7 T2 T1 T3 T2 T3 T5 T6 T1 T4 T7 T4 T5 T1 T6 T6 T1 T7 T4 T7 T4 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

Regression Test Suite Prioritization R R2 R4 R3 R1 R5 T1 T1 T1 T2 T2 T2 T3 T3 T3 T4 T4 T4 T5 T5 T5 T6 T6 T6 T7 T7 T7 T1 T2 T2 T3 T3 T5 T6 T1 T4 T7 T4 T5 T1 T6 T6 T1 T7 T4 T7 T4 Testers can use greedy (Rothermel et al. TSE 2001) and search-based (Li et al. TSE 2007) methods to reorder suites Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11

Regression Test Suite Prioritization R R2 R4 R3 R1 R5 T1 T1 T1 T2 T2 T2 T3 T3 T3 T4 T4 T4 T5 T5 T5 T6 T6 T6 T7 T7 T7 T1 T2 T2 T3 T5 T3 T6 T1 T4 T5 T4 T7 T1 T1 T6 T6 T7 T4 T7 T4 QUESTION: Which prioritization technique is the best? Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11