Predicting Fault Numbers via Testing Marc Roper Dept. Computer and - - PowerPoint PPT Presentation

▶

Nov 29, 2022 108 likes •270 views

Predicting Fault Numbers via Testing Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers

SLIDE 1

Predicting Fault Numbers via Testing

Marc Roper

Dept. Computer and Information Sciences

University of Strathclyde Glasgow, U.K.

Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 1 / 12

SLIDE 2

Motivation

Even after testing there are still going to be faults in software - how do we estimate how many are left? Useful for: Reliability estimation Decision making

Release? Test further?

Automated debugging etc... Aim of this talk is to look at the applicaiton of capture-recapture techniques to generate predictions of faults remaining after testing

Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 2 / 12

SLIDE 3

Capture-Recapture Techniques

Used by population ecologists to estimate number of particular species in an area

Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 3 / 12

SLIDE 4

Capture-Recapture Techniques

Used by population ecologists to estimate number of particular species in an area

Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 3 / 12

SLIDE 5

Capture-Recapture Techniques

Used by population ecologists to estimate number of particular species in an area Number of iterations is variable: animals may get caught 0 - many times Data fed into capture-recapture models to make predictions about total population in the area

Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 3 / 12

SLIDE 6

Mapping Capture-Recapture Techniques to Software Testing

Animal = Fault Capture Method = Testing Technique Trapping Occasion = Independent Application of a Testing Technique (just 2 considered) Number of models explored which incorporate different assumptions: Mt Probability of fault detection differs between techniques/testers Mh Probability of fault detection differs between faults Mth Combines the above so that both faults and techniques/testers become sources of variation

Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 4 / 12

SLIDE 7

Mapping Capture-Recapture Techniques to Software Testing

Various estimators for these models, e.g.

Lincoln-Petersen estimator for Mt

ˆ N = n1n2

m1

Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 5 / 12

SLIDE 8

Mapping Capture-Recapture Techniques to Software Testing

Various estimators for these models, e.g.

Lincoln-Petersen estimator for Mt

ˆ N = n1n2

m1

Five considered in total: LP Lincoln-Peterson estimator for Mt JK Jackknife estimator for Mh CMT Chao estimator for Mt CMH Chao estimator for Mh CMTH Chao estimator for Mth

Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 5 / 12

SLIDE 9

Experimental Evaluation

Data:

System Size (loc) Total Faults

No. Testers

Testing Strategy Strathclyde1 ˜200 8 47 Functional, Structural Strathclyde2 ˜200 9 47 Functional, Structural Strathclyde3 ˜200 8 47 Functional, Structural Myers 63 15 30∗ Functional, Structural Space 6218 38 30∗∗ Statement coverage + * split into 2 groups of 15 ** simulated, split into 3 groups of 10

Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 6 / 12

SLIDE 10

Experimental Evaluation

Data:

System Size (loc) Total Faults

No. Testers

Testing Strategy Strathclyde1 ˜200 8 47 Functional, Structural Strathclyde2 ˜200 9 47 Functional, Structural Strathclyde3 ˜200 8 47 Functional, Structural Myers 63 15 30∗ Functional, Structural Space 6218 38 30∗∗ Statement coverage + * split into 2 groups of 15 ** simulated, split into 3 groups of 10

Method: Each of the five estimators evaluated using fault data revealed by each tester Every possible pairwise combination considered

Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 6 / 12

SLIDE 11

Results (Strathclyde)

Values of Estimators for Program2 (N=9):

91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 N =

CMTHBTP2 CMTHSTP2 CMTHFTP2 CMHBTP2 CMHSTP2 CMHFTP2 CMTBTP2 CMTSTP2 CMTFTP2 JKBTP2 JKSTP2 JKFTP2 LPBTP2 LPSTP2 LPFTP2 20 18 16 14 12 10 8 6 4 2

29 62 48 65 36 40 56 62 58 62 29 48 56 62 58 64 50 54 68 51 34 20 58 62 56 45 17 51 79 2 34 20 30 74 8 5 28 63 47 59 38 60 62 58 56 88

Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 7 / 12

SLIDE 12

Results (Myers)

Values of Estimators for Myers Data (N=15)

95 95 95 95 95 95 95 95 95 95 95 95 95 95 95 N =

CMTHBT CMTHST CMTHFT CMHBT CMHST CMHFT CMTBT CMTST CMTFT JKBT JKST JKFT LPBT LPST LPFT 30 25 20 15 10 5

110 33 77 114 71 25 106 81 110 33 77 118 82 70 20 54 3 55

Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 8 / 12

SLIDE 13

Results (Space)

Values of Estimators for Space Data (N=38)

MS Min-statement Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 9 / 12

SLIDE 14

Impact of Coverage Data

Strathclyde data scaled according to coverage (N=9)

91 91 91 91 91 91 91 91 91 91 N =

CMTHBTP2 CMTHSTP2 CMHBTP2 CMHSTP2 CMTBTP2 CMTSTP2 JKBTP2 JKSTP2 LPBTP2 LPSTP2 20 18 16 14 12 10 8 6 4 2

48 62 56 62 58 58 20 34 28 47 59 58 56 29 40 34 20 30 47 28 59 58 56

Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 10 / 12

SLIDE 15

Impact of Coverage Data

SPACE data scaled according to coverage (N=38)

Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 11 / 12

SLIDE 16

Conclusions

Capture-recapture techniques show promise as predictors of faults numbers Performance of the models tends to vary between datasets Findings illustrate the estimators’ susceptibility to the data being used – in particular the pattern of overlap and distinctiveness between faults Diversity generated by different testers using different testing approaches are likely to generate the most accurate results However, too much diversity with respect to the overlap amongst faults may also lead to inaccurate estimates. Future work will look at more datasets and how random or evolved data can be used to improve the accuracy of estimates

Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 12 / 12