Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary - - PowerPoint PPT Presentation

Introduction Resource-Constrained Testing Conclusions and Future Work

Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results Suvarshi Bhadra and Gregory M. Kapfhammer Department of Computer Science Allegheny College, Pennsylvania, USA http://www.cs.allegheny.edu/~gkapfham/

TAIC PART 2008, Fast Abstract Track, August 2008 Featuring images from www.pdclipart.org

1 / 10 Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results

Introduction Resource-Constrained Testing Conclusions and Future Work

Important Contributions

Technique Formulation

NCO JCO TCO 0.0 0.2 0.4 0.6 0.8 1.0 Overlap Metric Overlap Score

sss − covers

NCO JCO TCO 0.0 0.2 0.4 0.6 0.8 1.0 Overlap Metric Overlap Score

ssm − covers

NCO JCO TCO 0.0 0.2 0.4 0.6 0.8 1.0 Overlap Metric Overlap Score

ssl − covers

Empirical Results

Design, implement and empirically evaluate test suite prioritizers that leverage travelling salesperson problem (TSP) solvers to efficiently find cost-effective orderings

2 / 10 Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results

Introduction Resource-Constrained Testing Conclusions and Future Work

Regression Testing Techniques

Before After

Reduction Prunes the Test Suite

Before After

Prioritization Reorders the Tests It may expensive to run a test suite T = T1, . . . , Tn. Prioritization searches through the n! = n × n − 1 × . . . × 1 orderings for those that avoid costly database restarts, Web service calls, or memory interactions.

3 / 10 Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results

Introduction Resource-Constrained Testing Conclusions and Future Work

Prioritizing When Memory is Constrained

Frequent Memory Rewrites High Testing Costs

Frequent reads and writes to memory may increase execution time by as much as 600% when a Java application executes on a virtual machine with a small heap Solution: maximize memory reuse between test cases

4 / 10 Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results

Introduction Resource-Constrained Testing Conclusions and Future Work

The Impact of Test Ordering

m1 m2 m3 m4 m5 m6 Test 30 30 30 30 30 30 Size T1

• 90

T2

• 90

T3

• 90

T4

• 90

T5

• 60

T = T1, T2, T3, T4, T5 transfers 750 units to and from memory T ′ = T2, T4, T1, T3, T5 only loads and unloads 180 units

5 / 10 Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results

Introduction Resource-Constrained Testing Conclusions and Future Work

Test Prioritization: Steps One and Two

Program Instrumentation Adequacy Criterion Residual Test Requirements Cummulative Adequacy Calculator Instrumented Program Test Suite Execution Covered Requirements

Collect method invocation and size data using test coverage monitor and profiler

553 256 553 256 256 256 553 256 256 O 553 553 256 256 234 553 256 256 256 678 553 256 553 234 234 553 256 553 256

N0 N1 N2 N3 N4 N5

Formulate a complete graph using equations that estimate costs for all test pairs

6 / 10 Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results

Introduction Resource-Constrained Testing Conclusions and Future Work

Test Prioritization: Steps Three and Four

Use TSP solvers to identify a Hamiltonian path with low estimated costs

10 20 30 40 50 60 70 80 90 100 1.0 1.5 2.0 2.5 Size of the Test Suite (n) Execution Time (sec)

Evaluate the efficiency of the TSP solvers and the effectiveness of the test orderings

7 / 10 Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results

Introduction Resource-Constrained Testing Conclusions and Future Work

Empirical Results

200 400 600 800 1000 0.00 0.05 0.10 0.15 0.20

NN RNN NI FI CI AI

Time v No of Test Cases (n) (t)

Efficient Prioritizers

10 20 30 40 50 50 60 70 80 90 100 Score v No of Test Cases − NN (n) (score)

High Percentile Rankings

8 / 10 Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results

Introduction Resource-Constrained Testing Conclusions and Future Work

Avoiding Database Restarts

T0 T1 T2 T9 T13 T15 T5 T8 T12 T3 T7 T11 T14 T16 T10 T4 T6

Use prioritization to avoid costly database restarts

9 / 10 Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results

Introduction Resource-Constrained Testing Conclusions and Future Work

Concluding Remarks

Program

Analysis Technique

Test Suite Reduction Prioritization

Test Suite Prioritizers

sss − covers

ssm − covers

ssl − covers

Initial Empirical Evaluation Preliminary results with synthetic test suites indicate that it is possible to prioritize test suites with TSP solvers Use different methods for solving TSP instances (e.g., order-based genetic algorithms) and include real-world applications

http://www.cs.allegheny.edu/~gkapfham/research/

10 / 10 Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results