FaultTracer: A Change Impact and Regression Fault Analysis Tool for - - PowerPoint PPT Presentation

▶
faulttracer a change impact and regression fault analysis
SMART_READER_LITE
LIVE PREVIEW

FaultTracer: A Change Impact and Regression Fault Analysis Tool for - - PowerPoint PPT Presentation

May 14, 2023 173 likes •389 views

FaultTracer: A Change Impact and Regression Fault Analysis Tool for Evolving Java Programs Lingming Zhang , Miryung Kim, Sarfraz Khurshid University of Texas at Austin zhanglm@utexas.edu FSE Formal Research Demo Track Nov 14, 2012 1 Scenario

slide-1
SLIDE 1

FaultTracer: A Change Impact and Regression Fault Analysis Tool for Evolving Java Programs

Lingming Zhang, Miryung Kim, Sarfraz Khurshid University of Texas at Austin

zhanglm@utexas.edu

FSE Formal Research Demo Track Nov 14, 2012

1

slide-2
SLIDE 2

Scenario

2 FaultTracer: A Change Impact and Regression Fault Analysis Tool for Evolving Java Programs

Old Version P Regression tests T Test

slide-3
SLIDE 3

Scenario

3 FaultTracer: A Change Impact and Regression Fault Analysis Tool for Evolving Java Programs

Old Version P New Version P’ Regression tests T

Evolve

Test

slide-4
SLIDE 4

Scenario

4 FaultTracer: A Change Impact and Regression Fault Analysis Tool for Evolving Java Programs

Old Version P New Version P’ Regression tests T

Evolve

Test

?

  • Q1: How to efficiently run the regression tests?
  • Which tests are relevant to program edits?
  • Q2: How to effectively localize faults when tests fail?
  • Which program edits are relevant to test failures?
slide-5
SLIDE 5
  • Chianti change impact analysis is effective at finding

suspicious edits but does not rank these edits. [Ren’04, Ren’06]

  • Spectrum-based fault localization ranks potential

faulty code fragments but does not focus on changes. [Jones’02, Abreu’07, Yu’08, Santelices’09, Parnin’11]

  • Our insight is to combine change-impact analysis and

spectrum-based fault localization [ICSM11].

  • Identify suspicious edits based on extended call

graphs.

  • Rank suspicious edits using dynamic program

spectrum information.

Motivation

5 FaultTracer: A Change Impact and Regression Fault Analysis Tool for Evolving Java Programs

slide-6
SLIDE 6

FaultTracer overview

6 FaultTracer: A Change Impact and Regression Fault Analysis Tool for Evolving Java Programs

Old program version:

P

New program version:

P’

Program edits: ∆

Detect changes and dependences

1

slide-7
SLIDE 7

FaultTracer overview

7 FaultTracer: A Change Impact and Regression Fault Analysis Tool for Evolving Java Programs

Old program version:

P

New program version:

P’

Program edits: ∆

Detect changes and dependences

Original test suite:

T

Affected tests:

T’

Select affected tests based on ECG analysis

1 2

slide-8
SLIDE 8

FaultTracer overview

8 FaultTracer: A Change Impact and Regression Fault Analysis Tool for Evolving Java Programs

Old program version:

P

New program version:

P’

Program edits: ∆

Detect changes and dependences

Original test suite:

T

Affected tests:

T’

Select affected tests based on ECG analysis

Failed test:

t

Subset of suspicious edits: Determine suspicious edits based on ECG analysis

1 2 3

slide-9
SLIDE 9

FaultTracer overview

9 FaultTracer: A Change Impact and Regression Fault Analysis Tool for Evolving Java Programs

Old program version:

P

New program version:

P’

Program edits: ∆

Detect changes and dependences

Original test suite:

T

Affected tests:

T’

Select affected tests based on ECG analysis

Failed test:

t

Subset of suspicious edits: Determine suspicious edits based on ECG analysis Rank suspicious edits based on program spectrum information Ranked list of suspicious edits:

1 2 3 4

slide-10
SLIDE 10

Extended call graph (ECG)

10 FaultTracer: A Change Impact and Regression Fault Analysis Tool for Evolving Java Programs

test1

A.Clinit() A.bar()

test2

C.C() C.foo() B.B()

<C, C.foo()>

test1

A.Clinit() A.bar()

test2

C.C() C.foo() B.B() A.f2

<SFW, A.f2>

B.f1

<FR, C.f1> <C, C.foo()>

Traditional Call Graphs used by Chianti [Ren’04] Extended Call Graphs used by FaultTracer

slide-11
SLIDE 11
  • Dependence inference illustration
  • For every AM change, if a method called by the added

method is new and all methods overridden by it are also new, the caller should be dependent on the added callee.

Change types Description

CM Change method AM Add method DM Delete method AF Add field DF Delete field CFI Change instance field CSFI Change static field LCm Method look-up change LCf Field look-up change

Step 1: Detect atomic changes & dependences

FaultTracer: A Change Impact and Regression Fault Analysis Tool for Evolving Java Programs 11

slide-12
SLIDE 12
  • FaultTracer directly matches all non-look-up changes

with ECGs of the old version to select affected tests.

  • Existing technique needs to transform field

changes into constructor change first.

  • FaultTracer identifies tests that are influenced by

method or field look-up changes as affected tests.

  • Existing technique does not handle field look-up

change.

Step 2: Select tests based on ECG analysis

FaultTracer: A Change Impact and Regression Fault Analysis Tool for Evolving Java Programs 12

slide-13
SLIDE 13
  • FaultTracer directly identifies all non-look-up changes
  • n ECGs of the new version as suspicious edits.
  • Existing technique needs to select
  • the changes covered by affected tests.
  • the changes that these covered changes

transitively depend on.

  • FaultTracer identifies method or field level edits that

caused look-up changes on ECGs as suspicious edits.

  • Existing technique cannot find field level edits that

caused field look-up changes.

Step 3: Identify suspicious edits based on ECG analysis

FaultTracer: A Change Impact and Regression Fault Analysis Tool for Evolving Java Programs 13

slide-14
SLIDE 14
  • Relation between suspicious edits and tests
  • Suspicious score calculation

Step 4: Localize failure-inducing program edits using test spectra

FaultTracer: A Change Impact and Regression Fault Analysis Tool for Evolving Java Programs 14 Edits Test1 Test2 test3 test4 Edit1 Edit2 Edit3 Edit4 Result Pass Pass Pass Fail Edits Tarantula [Jones’02] SBI [Yu’08] Jaccard [Abreu’07] Ochiai [Abreu’07] Edit1 0.00 0.00 0.00 0.00 Edit2 0.75 0.50 0.50 0.71 Edit3 0.75 0.50 0.50 0.71 Edit4 1.00 1.00 1.00 1.00

slide-15
SLIDE 15

Demo: Step 1.

  • Detect changes and dependences

15 FaultTracer: A Change Impact and Regression Fault Analysis Tool for Evolving Java Programs

Old program version:

P

New program version:

P’

Program edits: ∆

Detect changes and dependences

Original test suite:

T

Affected tests:

T’

Select affected tests based on ECG analysis

Failed test:

t

Subset of suspicious edits: Determine suspicious edits based on ECG analysis Rank suspicious edits based on program spectrum information Ranked list of suspicious edits:

1 2 3 4

slide-16
SLIDE 16

Demo: Step 2.

  • Select affected tests

16 FaultTracer: A Change Impact and Regression Fault Analysis Tool for Evolving Java Programs

Old program version:

P

New program version:

P’

Program edits: ∆

Detect changes and dependences

Original test suite:

T

Affected tests:

T’

Select affected tests based on ECG analysis

Failed test:

t

Subset of suspicious edits: Determine suspicious edits based on ECG analysis Rank suspicious edits based on program spectrum information Ranked list of suspicious edits:

1 2 3 4

slide-17
SLIDE 17

Demo: Step 3.

  • Determine suspicious edits

17 FaultTracer: A Change Impact and Regression Fault Analysis Tool for Evolving Java Programs

Old program version:

P

New program version:

P’

Program edits: ∆

Detect changes and dependences

Original test suite:

T

Affected tests:

T’

Select affected tests based on ECG analysis

Failed test:

t

Subset of suspicious edits: Determine suspicious edits based on ECG analysis Rank suspicious edits based on program spectrum information Ranked list of suspicious edits:

1 2 3 4

slide-18
SLIDE 18

Demo: Step 4.

  • Rank suspicious edits

18 FaultTracer: A Change Impact and Regression Fault Analysis Tool for Evolving Java Programs

Old program version:

P

New program version:

P’

Program edits: ∆

Detect changes and dependences

Original test suite:

T

Affected tests:

T’

Select affected tests based on ECG analysis

Failed test:

t

Subset of suspicious edits: Determine suspicious edits based on ECG analysis Rank suspicious edits based on program spectrum information Ranked list of suspicious edits:

1 2 3 4

slide-19
SLIDE 19
  • We have successfully applied FaultTracer to real-

world Java programs ranging 1.83~80.44 KLoC

  • jtopas, xml-security, jmeter, and ant.
  • Runtime overhead by Chianti and FaultTracer in

collecting call graph information (ms).

Real-world Applications

FaultTracer: A Change Impact and Regression Fault Analysis Tool for Evolving Java Programs 19

50000 100000 150000 200000 250000

jtopas0.0 jtopas1.0 jtopas2.0 jtopas3.0 xmlsec0.0 xmlsec1.0 xmlsec2.0 xmlsec3.0 jmeter0.0 jmeter1.0 jmeter2.0 jmeter3.0 jmeter4.0 jmeter5.0 ant0.0 ant1.0 ant2.0 ant3.0 ant4.0 ant5.0 ant6.0 ant7.0 ant8.0

slide-20
SLIDE 20
  • FaultTracer combines a Chianti-style change impact

analysis with spectrum-based fault localization.

  • FaultTracer improves a Chianti-style change impact

analysis based on extended call graph analysis.

  • Experimental results show that FaultTracer [ICSM11]
  • outperforms Chianti in determining affecting

changes by 20%.

  • outperforms existing technique for localizing failure-

inducing program edits by 50%.

  • FaultTracer Eclipse plug-in is available for public

download:

https://webspace.utexas.edu/lz3548/www/ftracer.html

Conclusion

FaultTracer: A Change Impact and Regression Fault Analysis Tool for Evolving Java Programs 20