Parameter Tuning for Search-Based Test-Data Generation Revisited - - PowerPoint PPT Presentation

▶

parameter tuning for search based test data generation

Parameter Tuning for Search-Based Test-Data Generation Revisited - - PowerPoint PPT Presentation

Aug 10, 2023 146 likes •726 views

Parameter Tuning for Search-Based Test-Data Generation Revisited Support for Previous Results Anton Kotelyanskii Gregory M. Kapfhammer creative commons licensed ( BY-NC-ND ) ickr photo shared by sunface13 Software Testing Software

slide-1

SLIDE 1

Parameter Tuning for Search-Based Test-Data Generation Revisited

Support for Previous Results

Anton Kotelyanskii Gregory M. Kapfhammer

shared by creative commons licensed ( BY-NC-ND ) ickr photo sunface13

slide-2

SLIDE 2

Software Testing

slide-3

SLIDE 3

Software Testing

Test Suites

slide-4

SLIDE 4

Software Testing

Test Suites Automatic Generation

slide-5

SLIDE 5

Software Testing

Test Suites Automatic Generation Confronting Challenges

slide-6

SLIDE 6

Software Testing

Test Suites Automatic Generation Confronting Challenges Evaluation Strategies

slide-7

SLIDE 7

Empirical Studies

slide-8

SLIDE 8

Empirical Studies

Challenges

slide-9

SLIDE 9

Empirical Studies

Challenges Importance

slide-10

SLIDE 10

Empirical Studies

Challenges Importance Replication

slide-11

SLIDE 11

Empirical Studies

Challenges Importance Replication Rarity

slide-12

SLIDE 12

slide-13

SLIDE 13

EvoSuite

shared by creative commons licensed ( BY-SA ) ickr photo mcclanahoochie

slide-14

SLIDE 14

EvoSuite

Amazing test suite generator

shared by creative commons licensed ( BY-SA ) ickr photo mcclanahoochie

slide-15

SLIDE 15

EvoSuite

Amazing test suite generator Uses a genetic algorithm

shared by creative commons licensed ( BY-SA ) ickr photo mcclanahoochie

slide-16

SLIDE 16

EvoSuite

Amazing test suite generator Uses a genetic algorithm Input: A Java class

shared by creative commons licensed ( BY-SA ) ickr photo mcclanahoochie

slide-17

SLIDE 17

EvoSuite

Amazing test suite generator Uses a genetic algorithm Input: A Java class Output: A JUnit test suite

shared by creative commons licensed ( BY-SA ) ickr photo mcclanahoochie

slide-18

SLIDE 18

EvoSuite

Amazing test suite generator Uses a genetic algorithm Input: A Java class Output: A JUnit test suite

shared by

http://www.evosuite.org/

creative commons licensed ( BY-SA ) ickr photo mcclanahoochie

slide-19

SLIDE 19

Parameter Tuning

slide-20

SLIDE 20

Parameter Tuning

RSM: Response surface methodology

slide-21

SLIDE 21

Parameter Tuning

RSM: Response surface methodology SPOT: Sequential parameter optimization toolbox

slide-22

SLIDE 22

Parameter Tuning

RSM: Response surface methodology SPOT: Sequential parameter optimization toolbox Successfully applied to many diverse problems!

slide-23

SLIDE 23

Defaults or Tuned Values?

slide-24

SLIDE 24

slide-25

SLIDE 25

Experiment Design

shared by creative commons licensed ( BY-NC ) ickr photo Michael Kappel

slide-26

SLIDE 26

Experiment Design

Eight EvoSuite parameters

shared by creative commons licensed ( BY-NC ) ickr photo Michael Kappel

slide-27

SLIDE 27

Experiment Design

Eight EvoSuite parameters Ten projects from SF100

shared by creative commons licensed ( BY-NC ) ickr photo Michael Kappel

slide-28

SLIDE 28

Experiment Design

Eight EvoSuite parameters Ten projects from SF100 475 Java classes for subjects

shared by creative commons licensed ( BY-NC ) ickr photo Michael Kappel

slide-29

SLIDE 29

Experiment Design

Eight EvoSuite parameters Ten projects from SF100 475 Java classes for subjects 100 trials after parameter tuning

shared by creative commons licensed ( BY-NC ) ickr photo Michael Kappel

slide-30

SLIDE 30

Experiment Design

Eight EvoSuite parameters Ten projects from SF100 475 Java classes for subjects 100 trials after parameter tuning Aiming to improve statement coverage

shared by creative commons licensed ( BY-NC ) ickr photo Michael Kappel

slide-31

SLIDE 31

Parameters

Parameter Name Minimum Maximum Population Size 5 99 Chromosome Length 5 99 Rank Bias 1.01 1.99 Number of Mutations 1 10 Max Initial Test Count 1 10 Crossover Rate 0.01 0.99 Constant Pool Use Probability 0.01 0.99 Test Insertion Probability 0.01 0.99

slide-32

SLIDE 32

Experiments

slide-33

SLIDE 33

Experiments

184 days of computation time estimated

slide-34

SLIDE 34

Experiments

184 days of computation time estimated Cluster of 70 computers running for weeks

slide-35

SLIDE 35

Experiments

184 days of computation time estimated Cluster of 70 computers running for weeks Identied 139 "easy" and 21 "hard" classes

slide-36

SLIDE 36

Experiments

184 days of computation time estimated Cluster of 70 computers running for weeks Identied 139 "easy" and 21 "hard" classes Mann-Whitney U-test and

slide-37

SLIDE 37

Experiments

184 days of computation time estimated Cluster of 70 computers running for weeks Identied 139 "easy" and 21 "hard" classes Mann-Whitney U-test and Vargha-Delaney eect size

slide-38

SLIDE 38

Results

Category Eect Size p-value Results Across Trials and Classes 0.5029 0.1045 No "Easy" and "Hard" Classes 0.5048 0.0314

slide-39

SLIDE 39

Results

Using lower-is-better inverse statement coverage Category Eect Size p-value Results Across Trials and Classes 0.5029 0.1045 No "Easy" and "Hard" Classes 0.5048 0.0314

slide-40

SLIDE 40

Results

Using lower-is-better inverse statement coverage Eect size greater than 0.5 means that tuning is worse Category Eect Size p-value Results Across Trials and Classes 0.5029 0.1045 No "Easy" and "Hard" Classes 0.5048 0.0314

slide-41

SLIDE 41

Results

Using lower-is-better inverse statement coverage Eect size greater than 0.5 means that tuning is worse Testing shows we do not always reject the null hypothesis Category Eect Size p-value Results Across Trials and Classes 0.5029 0.1045 No "Easy" and "Hard" Classes 0.5048 0.0314

slide-42

SLIDE 42

Results

Using lower-is-better inverse statement coverage Eect size greater than 0.5 means that tuning is worse Testing shows we do not always reject the null hypothesis Additional empirical results in the QSIC 2014 paper! Category Eect Size p-value Results Across Trials and Classes 0.5029 0.1045 No "Easy" and "Hard" Classes 0.5048 0.0314

slide-43

SLIDE 43

slide-44

SLIDE 44

Discussion

shared by creative commons licensed ( BY ) photo Startup Stock Photos

slide-45

SLIDE 45

Discussion

Tuning improved scores for 11 classes

shared by creative commons licensed ( BY ) photo Startup Stock Photos

slide-46

SLIDE 46

Discussion

Tuning improved scores for 11 classes Otherwise, same as or worse than defaults

shared by creative commons licensed ( BY ) photo Startup Stock Photos

slide-47

SLIDE 47

Discussion

Tuning improved scores for 11 classes Otherwise, same as or worse than defaults A "soft oor" may exist for parameter tuning

shared by creative commons licensed ( BY ) photo Startup Stock Photos

slide-48

SLIDE 48

Discussion

Tuning improved scores for 11 classes Otherwise, same as or worse than defaults A "soft oor" may exist for parameter tuning Additional details in the QSIC 2014 paper!

shared by creative commons licensed ( BY ) photo Startup Stock Photos

slide-49

SLIDE 49

Practical Implications

slide-50

SLIDE 50

Practical Implications

Fundamental Challenges

slide-51

SLIDE 51

Practical Implications

Fundamental Challenges Tremendous Condence

slide-52

SLIDE 52

Practical Implications

Fundamental Challenges Tremendous Condence Great Opportunities

slide-53

SLIDE 53

slide-54

SLIDE 54

Important Contributions

shared by creative commons licensed ( BY-NC-ND ) ickr photo sunface13

slide-55

SLIDE 55

Important Contributions

Comprehensive Experiments

shared by creative commons licensed ( BY-NC-ND ) ickr photo sunface13

slide-56

SLIDE 56

Important Contributions

Comprehensive Experiments Conclusive Conrmation

shared by creative commons licensed ( BY-NC-ND ) ickr photo sunface13

slide-57

SLIDE 57

Important Contributions

Comprehensive Experiments Conclusive Conrmation For EvoSuite, Defaults = Tuned

shared by creative commons licensed ( BY-NC-ND ) ickr photo sunface13