Human Competitiveness of Genetic Programming for Spectrum Based - - PowerPoint PPT Presentation

Human Competitiveness of Genetic Programming for Spectrum Based Fault Localisation

Shin Yoo1, Xiaoyuan Xie2, Fei-Ching Kuo3, Tsong Yueh Chen3, Mark Harman4

HUMIES@GECCO 2017

1: KAIST, Republic of Korea 2: Wuhan University, China 3: Swinburne University of Technology, Australia 4: University College London/Facebook, UK

Automated Debugging

• Debugging is hard for

humans: we increasingly have to work on and with large code base written by others.

• Debugging is hard for

machines: automated repair techniques rely heavily on automated fault localisation.

Spectrum Based Fault Localisation

Program

Spectrum Based Fault Localisation

Program Tests

Spectrum Based Fault Localisation

Program Tests Spectrum

Spectrum Based Fault Localisation

ef − ep ep + np + 1

Risk Evaluation Formula Program Tests Spectrum

Spectrum Based Fault Localisation

ef − ep ep + np + 1

Risk Evaluation Formula Ranking Program Tests Spectrum

Spectrum Based Fault Localisation

ef − ep ep + np + 1

Risk Evaluation Formula Ranking Program Tests Spectrum Higher ranking = Fewer statements to check

Structural Test Test Test Spectrum Tarantula Rank Elements t1 t2 t3 ep ef np nf s1

• 1

2 0.00 9 s2

• 1

2 0.00 9 s3

• 1

2 0.00 9 s4

• 1

2 0.00 9 s5

• 1

2 0.00 9 s6

• 1

1 1 0.33 4 s7 (faulty)

• 2

1 1.00 1 s8

• 1

1 1 0.33 4 s9

• 1

2 0.50 2 Result P F F

Spectrum Based Fault Localisation

Structural Test Test Test Spectrum Tarantula Rank Elements t1 t2 t3 ep ef np nf s1

• 1

2 0.00 9 s2

• 1

2 0.00 9 s3

• 1

2 0.00 9 s4

• 1

2 0.00 9 s5

• 1

2 0.00 9 s6

• 1

1 1 0.33 4 s7 (faulty)

• 2

1 1.00 1 s8

• 1

1 1 0.33 4 s9

• 1

2 0.50 2 Result P F F

Spectrum Based Fault Localisation

Structural Test Test Test Spectrum Tarantula Rank Elements t1 t2 t3 ep ef np nf s1

• 1

2 0.00 9 s2

• 1

2 0.00 9 s3

• 1

2 0.00 9 s4

• 1

2 0.00 9 s5

• 1

2 0.00 9 s6

• 1

1 1 0.33 4 s7 (faulty)

• 2

1 1.00 1 s8

• 1

1 1 0.33 4 s9

• 1

2 0.50 2 Result P F F

Spectrum Based Fault Localisation

Structural Test Test Test Spectrum Tarantula Rank Elements t1 t2 t3 ep ef np nf s1

• 1

2 0.00 9 s2

• 1

2 0.00 9 s3

• 1

2 0.00 9 s4

• 1

2 0.00 9 s5

• 1

2 0.00 9 s6

• 1

1 1 0.33 4 s7 (faulty)

• 2

1 1.00 1 s8

• 1

1 1 0.33 4 s9

• 1

2 0.50 2 Result P F F

Spectrum Based Fault Localisation

Tarantula =

ef ef +nf ep ep+np + ef ef +nf

Structural Test Test Test Spectrum Tarantula Rank Elements t1 t2 t3 ep ef np nf s1

• 1

2 0.00 9 s2

• 1

2 0.00 9 s3

• 1

2 0.00 9 s4

• 1

2 0.00 9 s5

• 1

2 0.00 9 s6

• 1

1 1 0.33 4 s7 (faulty)

• 2

1 1.00 1 s8

• 1

1 1 0.33 4 s9

• 1

2 0.50 2 Result P F F

Spectrum Based Fault Localisation

Structural Test Test Test Spectrum Tarantula Rank Elements t1 t2 t3 ep ef np nf s1

• 1

2 0.00 9 s2

• 1

2 0.00 9 s3

• 1

2 0.00 9 s4

• 1

2 0.00 9 s5

• 1

2 0.00 9 s6

• 1

1 1 0.33 4 s7 (faulty)

• 2

1 1.00 1 s8

• 1

1 1 0.33 4 s9

• 1

2 0.50 2 Result P F F

Spectrum Based Fault Localisation

Structural Test Test Test Spectrum Tarantula Rank Elements t1 t2 t3 ep ef np nf s1

• 1

2 0.00 9 s2

• 1

2 0.00 9 s3

• 1

2 0.00 9 s4

• 1

2 0.00 9 s5

• 1

2 0.00 9 s6

• 1

1 1 0.33 4 s7 (faulty)

• 2

1 1.00 1 s8

• 1

1 1 0.33 4 s9

• 1

2 0.50 2 Result P F F

Spectrum Based Fault Localisation

(Empirical) State of the Art (circa 2012)

Over 30 formulæ in the literature, manually developed over a decade’s time: none guaranteed to perform best for all types of faults

(Empirical) State of the Art (circa 2012)

ef ef + nf + ep

ef ef + 2(nf + ep)

2ef 2ef + nf + ep

2ef ef + nf + ep

ef nf + ep

1 2( ef ef + nf + ef ef + ep )

ef ef + nf + ep + np

ef + np − nf − ep ef + nf + ep + np

ef + np ef + nf + ep + np

2(ef + np) 2(ef + np) + ep + nf

ef ef + np + 2(ep + nf)

ef + np nf + ep

ef ef +nf ep ep+np + ef ef +nf

Over 30 formulæ in the literature, manually developed over a decade’s time: none guaranteed to perform best for all types of faults

Evolving Formulæ

ef − ep ep + np + 1

Risk Evaluation Formula Ranking Program Tests Spectrum

Evolving Formulæ

ef − ep ep + np + 1

Risk Evaluation Formula Ranking

P S P S Training Data

Evolving Formulæ

Ranking

P S P S

GP

Training Data

Evolving Formulæ

P S P S

GP

Fitness (minimise) Training Data

Evolving Formulæ

P S P S

GP

Fitness (minimise)

e2

f(2ep + 2ef + 3np)

e2

f(e2 f + √np)

. . .

Training Data

Our Claims

Our Claims

GP evolved SBFL formulas are provably better than many human designs.

Our Claims

GP evolved SBFL formulas are provably better than many human designs. We proved that no human can surpass what GP evolved, ever.

Our Claims

GP evolved SBFL formulas are provably better than many human designs. We proved that no human can surpass what GP evolved, ever. GP has transformed the future research

• n fault localisation.

Crash Course into Our Proof System

Formula X Formula Y

SX

B

SY

B

SX

F

SX

A

SY

A

SY

F

SY

B ⊆ SX B ∧ SX A ⊆ SY A

To show that Y dominates X, we show that: (assuming that we break ties in F sets consistently) Statement Ranking Equivalence is defined as:

X ↔ Y ⇐ ⇒ X → Y ∧ Y → X

Crash Course into Our Proof System

• Maximal Groups: a set of formulas that are equivalent to

each other, but are strictly better to some others

• Previous work theoretically proved the existence of maximal

groups with respect to the space of known formulas:

• ER1 (contains 2 manually designed formulas) and ER5

(contains 3 manually designed formulas)

GP’s Human Competitiveness

• GP expanded the known

maximal groups:

• GP added one additional

formula to ER1

• GP founded three new

maximal groups, each containing one GP-evolved formula

Name Formula expression ER1’ Naish1 ⇢ −1 if ef<F P − ep if ef=F Naish2 ef −

ep ep+np+1

GP13 ef(1 +

1 2ep+ef )

ER5 Wong1 ef Russel & Rao

ef ef +nf +ep+np

Binary ⇢ 0 if ef<F 1 if ef=F GP02 2(ef + √np) + √ep GP03 q |e2

f − √ep|

GP19 ef p |ep − ef + nf − np|

GP’s Human Competitiveness

• We have proved that there is

no greatest formula (i.e. one that outperforms all maximals):

• GP evolved the best possible

formula.

• No future human

endeavour can surpass GP’s results.

GP’s Influence on Future Research

• Manually designing SBFL

formulae is no longer productive.

• We need richer information than

program spectrum: GP can deal with increased complexity better than human.

• GP continues to produce

state-of-the-art localisation results, outperforming SVMs (ISSTA 2017).

Automated Debugging

• Debugging is hard for

• Debugging is hard for

Debugging is hard.

GP’s Human Competitiveness

• GP expanded the known

• GP added one additional

• GP founded three new

GP-evolved fault localisation techniques were provably better than over a decade’s manual work.

Automated Debugging

• Debugging is hard for

• Debugging is hard for

Debugging is hard.

GP’s Human Competitiveness

• GP expanded the known

• GP added one additional

• GP founded three new

GP-evolved fault localisation techniques were provably better than over a decade’s manual work.

GP’s Human Competitiveness

• We have proved that there is

• GP evolved the best

• No future human

We proved that no human can surpass GP results, ever.

Automated Debugging

• Debugging is hard for

• Debugging is hard for

Debugging is hard.

GP’s Human Competitiveness

• GP expanded the known

• GP added one additional

• GP founded three new

GP-evolved fault localisation techniques were provably better than over a decade’s manual work.

GP’s Human Competitiveness

• We have proved that there is

• GP evolved the best

• No future human

We proved that no human can surpass GP results, ever.

GP’s Influence on Future Research

• Manually designing SBFL

• We need richer information

• GP continues to produce

• utperformed SVMs (ISSTA

GP continues to have strong influence

• n future research.

Automated Debugging

• Debugging is hard for

• Debugging is hard for

Debugging is hard.