[PPT] - A Theoretical & Empirical Analysis of Evolutionary Testing and PowerPoint Presentation

SLIDE 1

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

A Theoretical & Empirical Analysis

f Evolutionary Testing and Hill Climbing

for Structural Test Data Generation

ISSTA July 2007 Mark Harman King’s College London Phil McMinn Sheffield University

Thursday, 10 December 2009

SLIDE 2

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

Which Search Technique: Global or Local?

Mark Harman King’s College London Phil McMinn Sheffield University

Thursday, 10 December 2009

SLIDE 3

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

Which Search Technique: Global or Local?

Mark Harman and Phil McMinn. A Theoretical and Empirical Analysis of Evolutionary Testing and Hill Climbing for Structural Test Data Generation ISSTA 2007. Mark Harman and Phil McMinn, A Theoretical and Empirical Study of Search Based Testing: Local, Global and Hybrid Search

TSE. To appear.

Thursday, 10 December 2009

SLIDE 4

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

Which Search Technique: Global or Local?

Mark Harman and Phil McMinn. A Theoretical and Empirical Analysis of Evolutionary Testing and Hill Climbing for Structural Test Data Generation ISSTA 2007. Mark Harman and Phil McMinn, A Theoretical and Empirical Study of Search Based Testing: Local, Global and Hybrid Search

TSE. To appear.

Thursday, 10 December 2009

SLIDE 5

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

Which Search Technique: Global or Local?

Mark Harman and Phil McMinn. A Theoretical and Empirical Analysis of Evolutionary Testing and Hill Climbing for Structural Test Data Generation ISSTA 2007. Mark Harman and Phil McMinn, A Theoretical and Empirical Study of Search Based Testing: Local, Global and Hybrid Search

TSE. To appear.

Author order is alphabetical

Thursday, 10 December 2009

SLIDE 6

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

6

Where is King’s College London?

UCL LSE King’s IC London Eye

Thursday, 10 December 2009

SLIDE 7

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

7

Where is Sheffield University?

London Sheffield

Thursday, 10 December 2009

SLIDE 8

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

8

No Full Monty Joke

Thursday, 10 December 2009

SLIDE 9

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

9

No Full Monty Joke

Sorry

Thursday, 10 December 2009

SLIDE 10

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

10

Overview

Search Based Testing Local: Hill Climbing using Alternating variable method

Global: Genetic Algorithms

Theoretical foundations Schemas

Royal Roads

Empirical study Implications

Thursday, 10 December 2009

SLIDE 11

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

11

What is SBT

In Search based testing we apply search techniques to search large input spaces, guided by a fitness function.

Thursday, 10 December 2009

SLIDE 12

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

12

What is SBT

In Search based testing we apply search techniques to search large input spaces, guided by a fitness function.

Genetic Algorithms, Hill climbing, Simulated Annealing, Random, Tabu Search, Estimation of Distribution Algorithms, Particle Swarm Optimization

Thursday, 10 December 2009

SLIDE 13

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

13

What is SBT

In Search based testing we apply search techniques to search large input spaces, guided by a fitness function.

Genetic Algorithms, Hill climbing, Simulated Annealing, Random, Tabu Search, Estimation of Distribution Algorithms, Particle Swarm Optimization

Thursday, 10 December 2009

SLIDE 14

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

14

What is SBT

In Search based testing we apply search techniques to search large input spaces, guided by a fitness function.

Genetic Algorithms, Hill climbing, Simulated Annealing, Random, Tabu Search, Estimation of Distribution Algorithms, Particle Swarm Optimization

Thursday, 10 December 2009

SLIDE 15

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

15

Structural Testing

Focus on branch testing

Most widely studied So ready for some more in depth analysis

Thursday, 10 December 2009

SLIDE 16

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

16

Other Search Based Testing Applications Temporal Wegener et al. Coverage Pargass & Harrold, Xanthakis et al., McMinn, Harman, Michael et al, Sthamer, Jones … Functional Wegener et al. Regression Rothermel et al., Woolcott et al., Yoo and Harman,… Interaction Cohen et al. Bryce, Colbourn Exception Tracey and Clark Stress Briand et al., Antoniol, Di Penta Robustness Shultz et al.

Thursday, 10 December 2009

SLIDE 17

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

17

Structural Testing

Focus on branch testing

Most widely studied So ready for some more in depth analysis Two algorithms:

Hill Climbing, using Korel’s alternating variable method Genetic Algorithms, using DaimlerChrysler approach

Thursday, 10 December 2009

SLIDE 18

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

18

Structural Testing

Focus on branch testing

Most widely studied So ready for some more in depth analysis Two algorithms:

Hill Climbing, using Korel’s alternating variable method Genetic Algorithms, using DaimlerChrysler approach … and Random Search

Thursday, 10 December 2009

SLIDE 19

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

19

Target Level 4 Level 3 Level 2 Level 1

Fitness = Approximation_Level + Local_Distance Evaluation of predicate in a branching condition if A = B Local_Distance = | A - B | Identify relevant branching statements using control dependence

Target Target Target

1. Approximation level

Fitness Computation

Thursday, 10 December 2009

SLIDE 20

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

20

The alternating variable method is hill climbing plus accelerated moves Near Neighbour? One small increase One small decrease Method: Cycle through input variables one at a time: probe moves move to near neighbour: If probing works, make accelerated pattern moves Until no improvement on any variable

Alternating Variable Method

For some input variable

Thursday, 10 December 2009

SLIDE 21

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

21

Goal-Oriented Approach:

Alternating Variable Method

Accelerated hill climb

Fitness Input variable value

Thursday, 10 December 2009

SLIDE 22

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

22

Hill Climbing  Steepest Descent

Thursday, 10 December 2009

SLIDE 23

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

23

Hill Climbing  Steepest Descent

Thursday, 10 December 2009

SLIDE 24

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

24

void example(int a, int b, int c) { if (a == 0) { ... } if (b == 0) { if (c == 0) { // target } } }

Random start: a=10 b=20 c=30 Case a :- Probe move has no efgect Case b :- Decrease probe improves So accelerate until b=0 Case c :-

Alternating Variable Method Example

Thursday, 10 December 2009

SLIDE 25

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

25

Evolutionary Algorithms

Selection Insertion Recombination Mutation Fitness evaluation

Test execution

End?

Thursday, 10 December 2009

SLIDE 26

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

26

Evolutionary Testing

Selection Insertion Recombination Mutation Fitness evaluation

Test cases

Test execution

End?

Thursday, 10 December 2009

SLIDE 27

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

27

Evolutionary Testing

Selection Insertion Recombination Mutation Fitness evaluation

Test cases

Test execution

End?

Execution

Thursday, 10 December 2009

SLIDE 28

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

28

Evolutionary Testing

Selection Insertion Recombination Mutation Fitness evaluation

Test cases

Test execution

End?

Execution Monitoring

Thursday, 10 December 2009

SLIDE 29

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

29

How mating makes life easier

Thursday, 10 December 2009

SLIDE 30

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

30

How mating makes life easier

Mating is really very much an analogy The important property is crossover

Thursday, 10 December 2009

SLIDE 31

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

31

How mating makes life easier

Thursday, 10 December 2009

SLIDE 32

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

32

a & b c & d e & f cost: 1 cost: 2 cost: 3 cost: 1 cost: 3 a & b c & d e & f cost: 1 cost: 2 cost: 3 cost: 1 cost: 3 a & b c & d e & f cost: 1 cost: 2 cost: 3 cost: 1 cost: 3 Cost: 4 a & b c & d e & f cost: 1 cost: 2 cost: 1 cost: 3 cost: 3 Cost: 2 cost: 1 cost: 3 a & b c & d e & f cost: 1 cost: 2 Cost: 3 cost: 3 a & b c & d e & f cost: 1 cost: 2 cost: 1 cost: 3 cost: 3 Cost: 8

How mating makes life easier

Thursday, 10 December 2009

SLIDE 33

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

33

a & b c & d e & f cost: 1 cost: 2 cost: 3 cost: 1 cost: 3 a & b c & d e & f cost: 1 cost: 2 cost: 3 cost: 1 cost: 3 a & b c & d e & f cost: 1 cost: 2 cost: 3 cost: 1 cost: 3 Cost: 4 a & b c & d e & f cost: 1 cost: 2 cost: 1 cost: 3 cost: 3 Cost: 2 cost: 1 cost: 3 a & b c & d e & f cost: 1 cost: 2 Cost: 3 cost: 3 a & b c & d e & f cost: 1 cost: 2 cost: 1 cost: 3 cost: 3 Cost: 8

How mating makes life easier

Thursday, 10 December 2009

SLIDE 34

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

34

Fitness Landscape

void MrNiceGuy(double a, double b) { if (a == b) { // target 1 double c = b + 1; if (c == 0) { // target 2 . . .

Thursday, 10 December 2009

SLIDE 35

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

35

Hitchcock Fitness Landscape

void alfred(double a, double b) { if (a == b) { // target 1 double c = b + 1; if (c == 0) { // target 2 . . .

Thursday, 10 December 2009

SLIDE 36

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

36

Hitchcock Fitness Landscape

Thursday, 10 December 2009

SLIDE 37

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

37

Hitchcock Fitness Landscape

Thursday, 10 December 2009

SLIDE 38

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

38

But …

When does it work Why does it work (when it does)? How does it compare to local search?

Thursday, 10 December 2009

SLIDE 39

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

39

Schemas

Pop[1] = 010111001010100110010010001 Pop[2] = 101001001010101011111000000 Pop[3] = 010100101010101000001010110 Pop[4] = 010111101010101111110101001 …

Thursday, 10 December 2009

SLIDE 40

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

40

Schemas

Pop[1] = 010111001010100110010010001 Pop[2] = 101001001010101011111000000 Pop[3] = 010100101010101000001010110 Pop[4] = 010111101010101111110101001 …

Thursday, 10 December 2009

SLIDE 41

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

41

Schemas

Pop[1] = 010111001010100110010010001 Pop[2] = 101001001010101011111000000 Pop[3] = 010100101010101000001010110 Pop[4] = 010111101010101111110101001 …

Thursday, 10 December 2009

SLIDE 42

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

42

Schemas

Pop[1] = 010111001010100110010010001 Pop[2] = 101001001010101011111000000 Pop[3] = 010100101010101000001010110 Pop[4] = 010111101010101111110101001 …

Thursday, 10 December 2009

SLIDE 43

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

43

Schemas

Thursday, 10 December 2009

SLIDE 44

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

44

Schemas

Thursday, 10 December 2009

SLIDE 45

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

45

The Royal Road

Thursday, 10 December 2009

SLIDE 46

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

46

The Royal Road

Thursday, 10 December 2009

SLIDE 47

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

47

The Royal Road

Thursday, 10 December 2009

SLIDE 48

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

48

The Royal Road

Thursday, 10 December 2009

SLIDE 49

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

49

The Royal Road

Thursday, 10 December 2009

SLIDE 50

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

50

The Royal Road

Thursday, 10 December 2009

SLIDE 51

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

51

S1: 1111**************************** S2: ****1111************************ S3: ********1111******************** S4: ************1111**************** S5: ****************1111************ S6: ********************1111******** S7: ************************1111**** S8: ****************************1111 S9: 11111111************************ S10: ********11111111**************** S11: ****************11111111******** S12: ************************11111111 S13: 1111111111111111**************** S14: ****************1111111111111111 S15: 11111111111111111111111111111111

The Royal Road

Thursday, 10 December 2009

SLIDE 52

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

52

S1: 1111**************************** S2: ****1111************************ S3: ********1111******************** S4: ************1111**************** S5: ****************1111************ S6: ********************1111******** S7: ************************1111**** S8: ****************************1111 S9: 11111111************************ S10: ********11111111**************** S11: ****************11111111******** S12: ************************11111111 S13: 1111111111111111**************** S14: ****************1111111111111111 S15: 11111111111111111111111111111111

The Royal Road

Thursday, 10 December 2009

SLIDE 53

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

53

S1: 1111**************************** S2: ****1111************************ S3: ********1111******************** S4: ************1111**************** S5: ****************1111************ S6: ********************1111******** S7: ************************1111**** S8: ****************************1111 S9: 11111111************************ S10: ********11111111**************** S11: ****************11111111******** S12: ************************11111111 S13: 1111111111111111**************** S14: ****************1111111111111111 S15: 11111111111111111111111111111111

The Royal Road

Thursday, 10 December 2009

SLIDE 54

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

54

S1: 1111**************************** S2: ****1111************************ S3: ********1111******************** S4: ************1111**************** S5: ****************1111************ S6: ********************1111******** S7: ************************1111**** S8: ****************************1111 S9: 11111111************************ S10: ********11111111**************** S11: ****************11111111******** S12: ************************11111111 S13: 1111111111111111**************** S14: ****************1111111111111111 S15: 11111111111111111111111111111111

The Royal Road

Thursday, 10 December 2009

SLIDE 55

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

55

S1: 1111**************************** S2: ****1111************************ S3: ********1111******************** S4: ************1111**************** S5: ****************1111************ S6: ********************1111******** S7: ************************1111**** S8: ****************************1111 S9: 11111111************************ S10: ********11111111**************** S11: ****************11111111******** S12: ************************11111111 S13: 1111111111111111**************** S14: ****************1111111111111111 S15: 11111111111111111111111111111111

The Royal Road

Thursday, 10 December 2009

SLIDE 56

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

56

S1: 1111**************************** S2: ****1111************************ S3: ********1111******************** S4: ************1111**************** S5: ****************1111************ S6: ********************1111******** S7: ************************1111**** S8: ****************************1111 S9: 11111111************************ S10: ********11111111**************** S11: ****************11111111******** S12: ************************11111111 S13: 1111111111111111**************** S14: ****************1111111111111111 S15: 11111111111111111111111111111111

The Royal Road

Thursday, 10 December 2009

SLIDE 57

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

57

Bibclean

pen source BibTeX pretty printer

Eurocheck

pen source € serial number validation

Gimp

pen source image manipulation

Spice analogue circuit simulator Tiff TIFF library for image manipulation Space ever heard of this one?

Subjects

Thursday, 10 December 2009

SLIDE 58

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

58

Fitness evaluations 100,000 Executions 30 Same seeds for statistical testing

Experimental set up

Thursday, 10 December 2009

SLIDE 59

59

Overall Results

Covered by all Uncovered or infeasible Covered by random only Covered by GA only Covered by hill climbing only Covered by GA and hill climbing

Thursday, 10 December 2009

SLIDE 60

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

60

Results

From 640 branches in the six subjects

10 branches for which Evolutionary Testing was successful but a simple Hill Climb search was not 5 branches for which Hill Climbing was successful but Evolutionary Testing was not 26 branches Comparability

Thursday, 10 December 2009

SLIDE 61

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

61

8 branches Evolutionary Testing succeeds where Hill Climbing fails: Branches of the bibclean test object String check for a valid ISBN/ISSN At least 10 digits

Royal Roads?

Thursday, 10 December 2009

SLIDE 62

62

25 50 75 100 c h e c k _ I S B N ( 2 3 F ) c h e c k _ I S B N ( 2 7 T ) c h e c k _ I S B N ( 2 9 F ) c h e c k _ I S B N ( 2 9 T ) c h e c k _ I S S N ( 2 3 F ) c h e c k _ I S S N ( 2 7 T ) c h e c k _ I S S N ( 2 9 F ) c h e c k _ I S S N ( 2 9 T ) Success Rate (%) Normal Parents Rnd 2nd Parent

Headless chicken test: success rate

Thursday, 10 December 2009

SLIDE 63

63

20,000 40,000 60,000 80,000 c h e c k _ I S B N ( 2 3 F ) c h e c k _ I S B N ( 2 7 T ) c h e c k _ I S B N ( 2 9 F ) c h e c k _ I S B N ( 2 9 T ) c h e c k _ I S S N ( 2 3 F ) c h e c k _ I S S N ( 2 7 T ) c h e c k _ I S S N ( 2 9 F ) c h e c k _ I S S N ( 2 9 T ) Average No. of Fitness Evaluations Normal Parents Rnd 2nd Parent

Headless chicken test: test effort

Thursday, 10 December 2009

SLIDE 64

64

Comparison Local vs Global

10 100 1,000 10,000 100,000

g i m p _ h w b _ t

_

r g b ( 3 T ) g i m p _ r g b _ t

_

h s v 4 ( 1 1 F ) g r a d i e n t _ c a l c _ c

n

i c a l _ a s y m _ f a c t

r

( 3 F ) s e q r

t

r g ( 1 7 T ) c l i p _ t

_

c i r c l e ( 1 F ) c l i p a r c ( 1 3 F ) c l i p a r c ( 2 4 T ) T I F F _ G e t S

u

r c e S a m p l e s ( 9 T ) T I F F _ S e t S a m p l e ( 7 T )

Av. No. of Fitness Evaluations

Evolutionary Testing Hill Climbing

Thursday, 10 December 2009

SLIDE 65

65

Comparison Local vs Global

10 100 1,000 10,000 100,000

g i m p _ h w b _ t

_

r g b ( 3 T ) g i m p _ r g b _ t

_

h s v 4 ( 1 1 F ) g r a d i e n t _ c a l c _ c

n

i c a l _ a s y m _ f a c t

r

( 3 F ) s e q r

t

r g ( 1 7 T ) c l i p _ t

_

c i r c l e ( 1 F ) c l i p a r c ( 1 3 F ) c l i p a r c ( 2 4 T ) T I F F _ G e t S

u

r c e S a m p l e s ( 9 T ) T I F F _ S e t S a m p l e ( 7 T )

Av. No. of Fitness Evaluations

Evolutionary Testing Hill Climbing

Thursday, 10 December 2009

SLIDE 66

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

66

HC outperforms GA

HC is fast, easy and effective In 24 of the 26 comparable cases it beats GA Average speed up is approximately a factor of 20 The results were statistically significant (paired t test)

Thursday, 10 December 2009

SLIDE 67

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

67

GA does perform well for Royal Road Functions … and this is because of the cross over operator But how many real programs have royal roads? For those which don’t HC is comfortably faster … by an order of magnitude … evolution strategies may outperform GA for RR Of course random covers most branches … but only the easy ones

Conclusions

Thursday, 10 December 2009

SLIDE 68

Mark Harman ISSTA: Empirical and Theoretical Search Based Testing

68

Future work

Memetic algorithms Evolution strategies Multi objective test data generation Study of SBT and DART Other GA theories

Thursday, 10 December 2009