Evolving Fault Localisation Shin Yoo, University College London, UK - - PowerPoint PPT Presentation

Human Competitive Award, GECCO 2012, 09 July 2012

Evolving Fault Localisation

Shin Yoo, University College London, UK

Spectra Based Fault Localisation

Program

Spectra Based Fault Localisation

Program Tests

Spectra Based Fault Localisation

Program Tests Spectrum

Spectra Based Fault Localisation

ef − ep ep + np + 1

Risk Evaluation Formula Program Tests Spectrum

Spectra Based Fault Localisation

ef − ep ep + np + 1

Risk Evaluation Formula Ranking Program Tests Spectrum

Evolving SBFL

ef − ep ep + np + 1

Risk Evaluation Formula Ranking Program Tests Spectrum

Evolving SBFL

ef − ep ep + np + 1

Risk Evaluation Formula Ranking

P T S P T S

Evolving SBFL

Ranking

P T S P T S

GP

Evolving SBFL

P T S P T S

GP

Fitness

Evolving SBFL

P T S P T S

GP

Fitness

Evolving SBFL

P T S P T S

GP

Fitness

e2

f(2ep + 2ef + 3np)

e2

f(e2 f + √np)

. . .

State of the Art

Over 30 formulæ in the literature, with various empirical studies with slightly different results

State of the Art

ef ef + nf + ep

ef ef + 2(nf + ep)

2ef 2ef + nf + ep

2ef ef + nf + ep

ef nf + ep

1 2( ef ef + nf + ef ef + ep )

ef ef + nf + ep + np

ef + np − nf − ep ef + nf + ep + np

ef + np ef + nf + ep + np

2(ef + np) 2(ef + np) + ep + nf

ef ef + np + 2(ep + nf)

ef + np nf + ep

ef ef +nf ep ep+np + ef ef +nf

Over 30 formulæ in the literature, with various empirical studies with slightly different results

State of the Art

Optimality Proof (Naish et al. 2011)

Op1 = ( −1 if nf > 0 np

• therwise

Op2 = ef − ep ep + np + 1

State of the Art

Optimality Proof (Naish et al. 2011) But the proof is against a specific model

·

if (t1()) s1(); /* S1 */ else s2(); /* S2 */ if (t2()) x = True; /* S3 */ else x = t3(); /* S4 - BUG */

Op1 = ( −1 if nf > 0 np

• therwise

Op2 = ef − ep ep + np + 1

State of the Art

Optimality Proof (Naish et al. 2011) But the proof is against a specific model

·

if (t1()) s1(); /* S1 */ else s2(); /* S2 */ if (t2()) x = True; /* S3 */ else x = t3(); /* S4 - BUG */

... not to mention hard.

Op1 = ( −1 if nf > 0 np

• therwise

Op2 = ef − ep ep + np + 1

Human Competitiveness

How many of 9 Existing Techniques can 30 GP runs match and/or outperform?

20% 13%

20% 23% 13% 10%

< 5 5 6 7 8 9

✤ 6 runs outperform 8 existing

techniques and match/outperform

• ne of the state of the art with

proof (Op1 and Op2).

✤ 16 runs outperform all 7 existing

techniques without proof.

Four Unix tools with 92 faults: 20 random faults for training, 72 for evaluation.

Human Competitiveness

✤ Per-fault view shows that

evolved techniques can

• utperform ones with
• ptimality proofs.
• 20

40 60 80 100 20 40 60 80 100

GP15 tarantula 61 1 10

• 20

40 60 80 100 20 40 60 80 100

GP15 tarantula 61 1 10

Human Competitiveness

✤ Per-fault view shows that

evolved techniques can

• utperform ones with
• ptimality proofs.
• 20

40 60 80 100 20 40 60 80 100

GP08

• p2

3 10 59

• 20

40 60 80 100 20 40 60 80 100

GP08

• p2

3 10 59

Human Competitiveness

✤ Per-fault view shows that

evolved techniques can

• utperform ones with
• ptimality proofs.
• 20

40 60 80 100 20 40 60 80 100

GP11

• p2

72

• 20

40 60 80 100 20 40 60 80 100

GP11

• p2

72

Future of Search-Based Software Engineering

From Solutions to Generic Problems... To Techniques and Strategies for Your Problems.

The most effective way to do it, is to do it.

✤ GP provides a structured,

automated way of doing iterative design.

✤ It can cope with a much diverse

spectra and other meta-data.

✤ GP can evolve to suit your

project.

Think Hard Write Formula Experiment

The most effective way to do it, is to do it.

✤ GP provides a structured,

automated way of doing iterative design.

✤ It can cope with a much diverse

spectra and other meta-data.

✤ GP can evolve to suit your

project. Human

Think Hard Write Formula Experiment

The most effective way to do it, is to do it.

✤ GP provides a structured,

automated way of doing iterative design.

✤ It can cope with a much diverse

spectra and other meta-data.

✤ GP can evolve to suit your

project.

Genetic Op. Evaluate Select

Human GP

Think Hard Write Formula Experiment

The most effective way to do it, is to do it.

✤ GP provides a structured,

automated way of doing iterative design.

✤ It can cope with a much diverse

spectra and other meta-data.

✤ GP can evolve to suit your

project.

Genetic Op. Evaluate Select

Human GP

Spectrum

Think Hard Write Formula Experiment

The most effective way to do it, is to do it.

✤ GP provides a structured,

automated way of doing iterative design.

✤ It can cope with a much diverse

spectra and other meta-data.

✤ GP can evolve to suit your

project.

Genetic Op. Evaluate Select

Human GP

Spectrum Spectrum

Think Hard Write Formula Experiment

The most effective way to do it, is to do it.

✤ GP provides a structured,

automated way of doing iterative design.

✤ It can cope with a much diverse

spectra and other meta-data.

✤ GP can evolve to suit your

project.

Genetic Op. Evaluate Select

Human GP

Spectrum Spectrum Change Hist. Dependency

Think Hard Write Formula Experiment

The most effective way to do it, is to do it.

✤ GP provides a structured,

automated way of doing iterative design.

✤ It can cope with a much diverse

spectra and other meta-data.

✤ GP can evolve to suit your

project.

Genetic Op. Evaluate Select

Human GP

Spectrum Spectrum Change Hist. Dependency Spectrum Change Hist. Dependency

Evolving SBFL

ef − ep ep + np + 1

Risk Evaluation Formula Ranking Program Tests Spectrum

Evolving SBFL

ef − ep ep + np + 1

Risk Evaluation Formula Ranking Program Tests Spectrum

Human Competitiveness

How many of 9 Existing Techniques can 30 GP runs match and/or outperform?

20% 13%

20% 23% 13% 10%

< 5 5 6 7 8 9

techniques and match/outperform

• ne of the state of the art with

proof (Op1 and Op2).

techniques without proof.

Evolving SBFL

ef − ep ep + np + 1

Risk Evaluation Formula Ranking Program Tests Spectrum

Human Competitiveness

How many of 9 Existing Techniques can 30 GP runs match and/or outperform?

20% 13%

20% 23% 13% 10%

< 5 5 6 7 8 9

techniques and match/outperform

• ne of the state of the art with

proof (Op1 and Op2).

techniques without proof.

Think Hard Write Formula Experiment

The most effective way to do it, is to do it.

automated way of doing iterative design.

spectra and other meta-data.

project.

Genetic Op. Evaluate Select

Human GP

Spectrum Spectrum Change Hist. Dependency Spectrum Change Hist. Dependency

Evolving SBFL

ef − ep ep + np + 1

Risk Evaluation Formula Ranking Program Tests Spectrum

Human Competitiveness

How many of 9 Existing Techniques can 30 GP runs match and/or outperform?

20% 13%

20% 23% 13% 10%

< 5 5 6 7 8 9

techniques and match/outperform

• ne of the state of the art with

proof (Op1 and Op2).

techniques without proof.

Think Hard Write Formula Experiment

The most effective way to do it, is to do it.

automated way of doing iterative design.

spectra and other meta-data.

project.

Genetic Op. Evaluate Select

Human GP

Spectrum Spectrum Change Hist. Dependency Spectrum Change Hist. Dependency

Detailed Statistics & Spectra Data http://www.cs.ucl.ac.uk/staff/s.yoo/ evolving-sbfl.html