Pushing the Boundaries in Regression Testing Shin Yoo & Mark - - PowerPoint PPT Presentation

Pushing the Boundaries in Regression Testing

Shin Yoo & Mark Harman / Kingʼs College London Shmuel Ur / IBM Haifa Paolo Tonella & Angelo Susi / FBK

Regression Testing

Large Test Suites Limited Resource

Selection

Uses impact analysis to precisely identify the changed parts of the program and only test those parts Often not the answer:

• requires static analysis
• not enough reduction

Large Test Suites Limited Resource

Minimisation

Seeks to reduce the size of test suites while satisfying test adequacy goals

R1 R2 R3 R4 T1 T2 T3

✓ ✓ ✓ ✓

Large Test Suites

Prioritisation

Seeks to achieve test adequacy as much and as early as possible

25.00 50.00 75.00 100.00

A-B-C-D-E C-E-B-A-D

Limited Resource

Publication Trend

6 12 18 24 30 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

Relative Interest in Different Subjects Minimisation Selection Prioritisation Empirical/Comparative

Harrold et al.

• n minimisation

Rothermel et al.

• n prioritisation

Leung et al.

• n cost model

Fischer et al. 1977

• n selection

Potential Impact

20% 80%

Purely Academic Author from Industry

Summary of 157 papers on regression testing techniques from a recent survey

8% 92%

Toy Programs Industrial Scale Subjects

Research Output

• Pareto Efficient Multi-Objective Test Case Selection:
• S. Yoo & M. Harman, ISSTA 2007
• Mesuring and Improving Latency to Avoid Test Suite

Wear-Out: S. Yoo, M. Harman & S. Ur, SBST 2009

• Clustering Test Cases to Achieve Effective and

Scalable Prioritisation Incorporating Expert Knowledge: S. Yoo, M. Harman, P. Tonella & A. Susi, ISSTA 2009

Multi-Objectiveness : Problems

• “After performing minimisation, the test

suite is still too big. What can I actually do in the next 6 hours?”

• “I care not just code coverage, but

something else too. Can I also achieve X with the minimised suite?”

Test Case Prog Program Bl ram Blocks Blocks Time Test Case 1 2 3 4 5 6 7 8 9 10 Time T1 x x x x x x x x 4 T2 x x x x x x x x x 5 T3 x x x x 3 T4 x x x x x 3

Single Objective Choose test case with highest block per time ratio as the next one 1) T1 (ratio = 2.0) 2) T2 (ratio = 2 / 5 = 0.4) ∴ {T1, T2} (takes 9 hours) “But what if you have only 7 hours...?” Multi Objective

2 4 6 8 10

Execution Time

20 40 60 80 100

Coverage(%)

Additional Greedy Pareto Frontier

Latency : Problems

• “My regression testing seems to

depend on the same test cases all the time; is this okay?”

• “Code coverage is necessary but not

sufficient test adequacy; how do we make it safer?”

T1 T2 T3 T5 T4 T6 T8 T7

The Coverage Trap

Minimisation

T5 may detect an unknown fault. If the minimisation technique never picks T5, fault detection capability is compromised. We should consider multiple

• subsets. If necessary, we should

improve the remaining part of the test suite.

Minimisation

T9

100% 80% 100%

20 40 60 80 20 40 60 80 100 Times that reduction technique has been applied Average maximum statement coverage(%)

• flex

grep sed space gzip printtokens

Designing Test Suites

• 10

20 30 40 50 60 20 40 60 80 100

Times that reduction technique has been applied Average maximum branch coverage(%)

• ComplexBranch
• Hill Climbing

EDA Random Original

Expertise : Problems

• “I have seen the results of automated

test case prioritisation and I donʼt agree. This is the way it should be!”

• “We need to prioritise a specific set of

tests due to business priority.”

Interleaved Clusters Prioritisation

T1 T3 T2 T5 T6 T4

Cluster Intra-cluster Prioritisation Inter-cluster Prioritisation Interleaving Clusters

Experimental Setup

Human Pairwise Comparison

C1 C2 C13 C14

Structural Coverage

...

Prioritised Test Suite

Simple Agglomerative Hierarchical Clustering (k=14) Hamming distance between

• stmt. coverage as dissimilarity

metric A human user model with controlled error rate

Tolerance

• 0.05

0.2 0.35 0.5 0.65 0.8 0.95 0.95 0.96 0.97 0.98 0.99 1.00 APFD

gzip

This is what we initially expected to see. But...

Tolerance

• ●

• ●
• ●
• 0.05

• ●
• ●
• ●

• 0.05

0.2 0.35 0.5 0.65 0.8 0.95 0.86 0.88 0.90 0.92 0.94 APFD

schedule suite2

Some test suites are very resilient to errors

Boundaries & Open Questions

• Scalability: not only quantitative but also

qualitative scalability

• Complexity: set-up cost, oracle cost,

dependency

• Effectiveness: are the traditional metrics

good enough?

• Regression testing is hard - not a single

solution

• Multi-objective paradigm allows us to

formulate a complex problem

• Code coverage needs to be re-thought
• Humans are a vast pool of knowledge

Summaries