EMPIRICALLY IDENTIFYING THE BEST GENETIC ALGORITHM FOR COVERING - - PowerPoint PPT Presentation

▶
empirically identifying the best genetic algorithm for
SMART_READER_LITE
LIVE PREVIEW

EMPIRICALLY IDENTIFYING THE BEST GENETIC ALGORITHM FOR COVERING - - PowerPoint PPT Presentation

Nov 11, 2022 715 likes •1.33k views

1 EMPIRICALLY IDENTIFYING THE BEST GENETIC ALGORITHM FOR COVERING ARRAY GENERATION Liang Yalan 1 , Changhai Nie 1 , Jonathan M. Kauffman 2 , Gregory M. Kapfhammer 2 , Hareton Leung 3 1 Department of Computer Science and Technology, Nanjing

slide-1
SLIDE 1

EMPIRICALLY IDENTIFYING THE BEST GENETIC ALGORITHM FOR COVERING ARRAY GENERATION

Liang Yalan1, Changhai Nie1, Jonathan M. Kauffman2, Gregory M. Kapfhammer2, Hareton Leung3

1Department of Computer Science and Technology, Nanjing University 2Department of Computer Science, Allegheny College 3Department of Computing, Hong Kong Polytechnic University

1

3rd International Symposium on Search Based Software Engineering Szeged, Hungary September 10-12, 2011

slide-2
SLIDE 2

Combinatorial Testing

2

Modern software systems are highly configurable and involve many interacting parameters Combinatorial testing is a widely used and practical technique for detecting failures caused by the parameter interactions One of the key challenges in combinatorial testing is covering array generation, which is an noteworthy area of research From Kuhn et al., 70% of failures can be detected by 2-way interactions of the software system’s parameters Covering arrays can save testing time while still detecting many important software faults

slide-3
SLIDE 3

Combinatorial Testing

3

Modern software systems are highly configurable and involve many interacting parameters Combinatorial testing is a widely used and practical technique for detecting failures caused by the parameter interactions One of the key challenges in combinatorial testing is covering array generation, which is an noteworthy area of research From Kuhn et al., 70% of failures can be detected by 2-way interactions of the software system’s parameters Covering arrays can save testing time while still detecting many important software faults

slide-4
SLIDE 4

Combinatorial Testing

4

Modern software systems are highly configurable and involve many interacting parameters Combinatorial testing is a widely used and practical technique for detecting failures caused by the parameter interactions One of the key challenges in combinatorial testing is covering array generation, which is an noteworthy area of research From Kuhn et al., 70% of failures can be detected by 2-way interactions of the software system’s parameters Covering arrays can save testing time while still detecting many important software faults

slide-5
SLIDE 5

Combinatorial Testing

5

Modern software systems are highly configurable and involve many interacting parameters Combinatorial testing is a widely used and practical technique for detecting failures caused by the parameter interactions One of the key challenges in combinatorial testing is covering array generation, which is an noteworthy area of research From Kuhn et al., 70% of failures can be detected by 2-way interactions of the software system’s parameters Covering arrays can save testing time while still detecting many important software faults

slide-6
SLIDE 6

Combinatorial Testing

6

Modern software systems are highly configurable and involve many interacting parameters Combinatorial testing is a widely used and practical technique for detecting failures caused by the parameter interactions One of the key challenges in combinatorial testing is covering array generation, which is an noteworthy area of research From Kuhn et al., 70% of failures can be detected by 2-way interactions of the software system’s parameters Covering arrays can save testing time while still detecting many important software faults

slide-7
SLIDE 7

2-way Covering Arrays

7

Suppose there are 4 parameters (pa1, pa2, pa3, and pa4) in a system under test (SUT), each with 3 values (0, 1, 2) If we want to cover all 54 pair-wise interactions between every 2 parameters in the SUT, then only 9 test cases are needed What is the most efficient and effective method for generating covering arrays?

slide-8
SLIDE 8

2-way Covering Arrays

8

Suppose there are 4 parameters (pa1, pa2, pa3, and pa4) in a system under test (SUT), each with 3 values (0, 1, 2) If we want to cover all 54 pair-wise interactions between every 2 parameters in the SUT, then only 9 test cases are needed What is the most efficient and effective method for generating covering arrays?

slide-9
SLIDE 9

Covering Array Generation

9

  • OFOT: One Factor One Time Method
  • AETG: Automatic Efficient Tests Generator

Mathematical and Greedy Methods

  • Particle swarm optimization
  • Simulated annealing
  • Ant colony optimization

Evolutionary Search Techniques

slide-10
SLIDE 10

Covering Array Generation

10

  • OFOT: One Factor One Time Method
  • AETG: Automatic Efficient Tests Generator

Mathematical and Greedy Methods

  • Particle swarm optimization
  • Simulated annealing
  • Ant colony optimization

Evolutionary Search Techniques

slide-11
SLIDE 11

Covering Array Generation

11

  • OFOT: One Factor One Time Method
  • AETG: Automatic Efficient Tests Generator

Mathematical and Greedy Methods

  • Particle swarm optimization
  • Simulated annealing
  • Ant colony optimization

Evolutionary Search Techniques

This paper studies and improves genetic algorithms for covering array generation

slide-12
SLIDE 12

Genetic Algorithm Phases

12

Initialize

  • Accept input

parameters

  • Generate

population

Fitness

  • Higher is

better

  • Pair-wise

interactions

Crossover

  • Create new

individuals

  • Combine two

parents

Mutation

  • Modify

individuals

  • Increase

diversity

Selection

  • Choose

individuals

  • Many

variants

1 2 3 4 5

Genetic algorithms solve complex problems

slide-13
SLIDE 13

Genetic Algorithm Phases

13

Initialize

  • Accept input

parameters

  • Generate

population

Fitness

  • Higher is

better

  • Pair-wise

interactions

Crossover

  • Create new

individuals

  • Combine two

parents

Mutation

  • Modify

individuals

  • Increase

diversity

Selection

  • Choose

individuals

  • Many

variants

1 2 3 4 5

Genetic algorithms are hard to configure

slide-14
SLIDE 14

Genetic Algorithm Parameters

14

Initialize

  • Accept input

parameters

  • Generate

population

Fitness

  • Higher is

better

  • Pair-wise

interactions

Crossover

  • Create new

individuals

  • Combine two

parents

Mutation

  • Modify

individuals

  • Increase

diversity

Selection

  • Choose

individuals

  • Many

variants

1 2 3 4 5

System under test (SUT) description (e.g., 313)

slide-15
SLIDE 15

Genetic Algorithm Parameters

15

Initialize

  • Accept input

parameters

  • Generate

population

Fitness

  • Higher is

better

  • Pair-wise

interactions

Crossover

  • Create new

individuals

  • Combine two

parents

Mutation

  • Modify

individuals

  • Increase

diversity

Selection

  • Choose

individuals

  • Many

variants

1 2 3 4 5

Number of uncovered pair-wise interactions

slide-16
SLIDE 16

Genetic Algorithm Parameters

16

Initialize

  • Accept input

parameters

  • Generate

population

Fitness

  • Higher is

better

  • Pair-wise

interactions

Crossover

  • Create new

individuals

  • Combine two

parents

Mutation

  • Modify

individuals

  • Increase

diversity

Selection

  • Choose

individuals

  • Many

variants

1 2 3 4 5

Pc controls the probability of crossover

slide-17
SLIDE 17

Genetic Algorithm Parameters

17

Initialize

  • Accept input

parameters

  • Generate

population

Fitness

  • Higher is

better

  • Pair-wise

interactions

Crossover

  • Create new

individuals

  • Combine two

parents

Mutation

  • Modify

individuals

  • Increase

diversity

Selection

  • Choose

individuals

  • Many

variants

1 2 3 4 5

If Pc is too high, then break good individuals

slide-18
SLIDE 18

Genetic Algorithm Parameters

18

Initialize

  • Accept input

parameters

  • Generate

population

Fitness

  • Higher is

better

  • Pair-wise

interactions

Crossover

  • Create new

individuals

  • Combine two

parents

Mutation

  • Modify

individuals

  • Increase

diversity

Selection

  • Choose

individuals

  • Many

variants

1 2 3 4 5

If Pc is too low, then miss good solutions

slide-19
SLIDE 19

Genetic Algorithm Parameters

19

Initialize

  • Accept input

parameters

  • Generate

population

Fitness

  • Higher is

better

  • Pair-wise

interactions

Crossover

  • Create new

individuals

  • Combine two

parents

Mutation

  • Modify

individuals

  • Increase

diversity

Selection

  • Choose

individuals

  • Many

variants

1 2 3 4 5

Pm controls the probability of mutation

slide-20
SLIDE 20

Genetic Algorithm Parameters

20

Initialize

  • Accept input

parameters

  • Generate

population

Fitness

  • Higher is

better

  • Pair-wise

interactions

Crossover

  • Create new

individuals

  • Combine two

parents

Mutation

  • Modify

individuals

  • Increase

diversity

Selection

  • Choose

individuals

  • Many

variants

1 2 3 4 5

If Pm is too small, then cannot escape minima

slide-21
SLIDE 21

Genetic Algorithm Parameters

21

Initialize

  • Accept input

parameters

  • Generate

population

Fitness

  • Higher is

better

  • Pair-wise

interactions

Crossover

  • Create new

individuals

  • Combine two

parents

Mutation

  • Modify

individuals

  • Increase

diversity

Selection

  • Choose

individuals

  • Many

variants

1 2 3 4 5

If Pm is too large, then degrade into random

slide-22
SLIDE 22

Genetic Algorithm Parameters

22

Initialize

  • Accept input

parameters

  • Generate

population

Fitness

  • Higher is

better

  • Pair-wise

interactions

Crossover

  • Create new

individuals

  • Combine two

parents

Mutation

  • Modify

individuals

  • Increase

diversity

Selection

  • Choose

individuals

  • Many

variants

1 2 3 4 5

Standard GA: Select the superior individuals

slide-23
SLIDE 23

Genetic Algorithm Parameters

23

Initialize

  • Accept input

parameters

  • Generate

population

Fitness

  • Higher is

better

  • Pair-wise

interactions

Crossover

  • Create new

individuals

  • Combine two

parents

Mutation

  • Modify

individuals

  • Increase

diversity

Selection

  • Choose

individuals

  • Many

variants

1 2 3 4 5

GA-: Select the inferior individuals

slide-24
SLIDE 24

Genetic Algorithm Parameters

24

Initialize

  • Accept input

parameters

  • Generate

population

Fitness

  • Higher is

better

  • Pair-wise

interactions

Crossover

  • Create new

individuals

  • Combine two

parents

Mutation

  • Modify

individuals

  • Increase

diversity

Selection

  • Choose

individuals

  • Many

variants

1 2 3 4 5

GAr: Randomly select the individuals

slide-25
SLIDE 25

Genetic Algorithm Parameters

25

Initialize

  • Accept input

parameters

  • Generate

population

Fitness

  • Higher is

better

  • Pair-wise

interactions

Crossover

  • Create new

individuals

  • Combine two

parents

Mutation

  • Modify

individuals

  • Increase

diversity

Selection

  • Choose

individuals

  • Many

variants

1 2 3 4 5

GA climb: Use elitism to keep best individual

slide-26
SLIDE 26

Genetic Algorithm Parameters

26

Initialize

  • Accept input

parameters

  • Generate

population

Fitness

  • Higher is

better

  • Pair-wise

interactions

Crossover

  • Create new

individuals

  • Combine two

parents

Mutation

  • Modify

individuals

  • Increase

diversity

Selection

  • Choose

individuals

  • Many

variants

1 2 3 4 5

GA, GA-, GAr, GA climb, GA- climb, GAr climb

slide-27
SLIDE 27

Experimental Design

27

Population Size Number of Generations Crossover Probability Mutation Probability

slide-28
SLIDE 28

Experimental Design

28

  • 100, 2100,
  • 4100, 6100

Population Size Number of Generations Crossover Probability Mutation Probability

slide-29
SLIDE 29

Experimental Design

29

Population Size

  • 100, 600, 1100

Number of Generations Crossover Probability Mutation Probability

slide-30
SLIDE 30

Experimental Design

30

Population Size Number of Generations

  • 0.2, 0.4, 0.6, 0.8, 1.0

Crossover Probability Mutation Probability

slide-31
SLIDE 31

Experimental Design

31

Population Size Number of Generations Crossover Probability

  • 0.2, 0.4, 0.6, 0.8, 1.0

Mutation Probability

slide-32
SLIDE 32

Experimental Design

32

Experiment Classes Pair-wise Base choice Hill climbing

Is there an improved configuration of genetic algorithm for a particular pair-wise SUT?

slide-33
SLIDE 33

Experimental Design

33

Experiment Classes Pair-wise Base choice Hill climbing

Is there a common improved configuration for all pair-wise SUTs?

slide-34
SLIDE 34

Experimental Design

34

Experiment Classes Pair-wise Base choice Hill climbing

Produce a 2-way covering array with 34 configurations and input into the next phases

slide-35
SLIDE 35

Experimental Design

35

Experiment Classes Pair-wise Base choice Hill climbing

Create configurations by changing the value

  • f one parameter and not modifying others
slide-36
SLIDE 36

Experimental Design

36

Experiment Classes Pair-wise Base choice Hill climbing

Iteratively refine the configurations in order to find the best one for each SUT

slide-37
SLIDE 37

37

For the chosen SUTs, there is no single genetic algorithm configuration that is the best

Experimental Results

slide-38
SLIDE 38

38

Different values for the effectiveness of the genetic algorithm (e.g., CA size)

Experimental Results

slide-39
SLIDE 39

39

Different values for the efficiency of the genetic algorithm (e.g., run time)

Experimental Results

slide-40
SLIDE 40

40

The VGAs of all the improved configurations all use a climbing genetic algorithm

Experimental Results

slide-41
SLIDE 41

41

GA- and GAr yield the best configuration for CA generation in 10 out of 15 SUTs

Experimental Results

slide-42
SLIDE 42

42

For all SUTs, a lengthier evolutionary process improves CA generation

Experimental Results

slide-43
SLIDE 43

43

In 13 out of 15 SUTS, creating fewer mutated individuals leads to better CAs

Experimental Results

slide-44
SLIDE 44

44

There is no common best value of Pc or m for the chosen SUTs

Experimental Results

slide-45
SLIDE 45

45

Please see the paper for additional insights concerning the experimental results

Experimental Results

slide-46
SLIDE 46

Conclusions

46

Initialize

  • Accept input

parameters

  • Generate

population

Fitness

  • Higher is

better

  • Pair-wise

interactions

Crossover

  • Create new

individuals

  • Combine two

parents

Mutation

  • Modify

individuals

  • Increase

diversity

Selection

  • Choose

individuals

  • Many

variants

1 2 3 4 5

Genetic algorithms for covering array generation

slide-47
SLIDE 47

Conclusions

47

Systematic study on the impact of GA parameters

Experiment Classes Pair-wise Base choice Hill climbing

slide-48
SLIDE 48

Conclusions

48

Fundamental insights into the genetic algorithm

Counter- intuitive selection strategies Elitist selection methods

Efficient and effective genetic algorithm for covering array generation

slide-49
SLIDE 49

Conclusions

49

Fundamental insights into the genetic algorithm

Counter- intuitive selection strategies Elitist selection methods

Efficient and effective genetic algorithm for covering array generation

slide-50
SLIDE 50

Conclusions

50

Fundamental insights into the genetic algorithm

Counter- intuitive selection strategies Elitist selection methods

Efficient and effective genetic algorithm for covering array generation

slide-51
SLIDE 51

Conclusions

51

Fundamental insights into the genetic algorithm

Counter- intuitive selection strategies Elitist selection methods

Efficient and effective genetic algorithm for covering array generation

slide-52
SLIDE 52

Conclusions

52

Fundamental insights into the genetic algorithm

Counter- intuitive selection strategies Elitist selection methods

Efficient and effective genetic algorithm for covering array generation

slide-53
SLIDE 53

Future Work

53

Improved Understanding of Parameters

Parallel Computing Discrete Parameters Continuous Parameters

slide-54
SLIDE 54

Future Work

54

Improved Understanding of Parameters

Parallel Computing Discrete Parameters Continuous Parameters

slide-55
SLIDE 55

Future Work

55

Improved Understanding of Parameters

Parallel Computing Discrete Parameters Continuous Parameters

slide-56
SLIDE 56

Future Work

56

Improved Understanding of Parameters

Parallel Computing Discrete Parameters Continuous Parameters

slide-57
SLIDE 57

Future Work

57

Efficient and Effective Genetic Algorithms

Parallel Computing Discrete Parameters Continuous Parameters

slide-58
SLIDE 58

Future Work

58

Enhanced Covering Array Generators

Parallel Computing Discrete Parameters Continuous Parameters

slide-59
SLIDE 59

Future Work

59

Better Tested and Higher Quality Software

Parallel Computing Discrete Parameters Continuous Parameters

slide-60
SLIDE 60

Thank you for your attention!

60

EMPIRICALLY IDENTIFYING THE BEST GENETIC ALGORITHM FOR COVERING ARRAY GENERATION

Liang Yalan, Changhai Nie, Jonathan M. Kauffman, Gregory M. Kapfhammer, Hareton Leung

QUESTIONS OR COMMENTS?

3rd International Symposium on Search Based Software Engineering Szeged, Hungary September 10-12, 2011