Automatic Defect Detection Andrzej Wasylkowski Overview Automatic - - PowerPoint PPT Presentation

▶
automatic defect detection
SMART_READER_LITE
LIVE PREVIEW

Automatic Defect Detection Andrzej Wasylkowski Overview Automatic - - PowerPoint PPT Presentation

Mar 08, 2024 231 likes •685 views

Automatic Defect Detection Andrzej Wasylkowski Overview Automatic Defect Detection Horizontal Techniques Specification-checking Techniques Mining-based Techniques Mining Repositories Mining Traces Mining Source Code Mining

slide-1
SLIDE 1

Automatic Defect Detection

Andrzej Wasylkowski

slide-2
SLIDE 2

Overview

Automatic Defect Detection Specification-checking Techniques Mining-based Techniques Mining Repositories Mining Source Code Mining Traces “Horizontal” Techniques

slide-3
SLIDE 3

Mining Repositories (1): DynaMine

Benjamin Livshits, Thomas Zimmermann, “DynaMine: Finding common error patterns by mining software revision histories”, ESEC/FSE 2005

public void createPartControl(Composite parent) { ... // add listener for editor page activation getSite().getPage().addPartListener(partListener); } public void dispose() { ... getSite().getPage().removePartListener(partListener); }

slide-4
SLIDE 4

Mining Repositories (1): DynaMine

Benjamin Livshits, Thomas Zimmermann, “DynaMine: Finding common error patterns by mining software revision histories”, ESEC/FSE 2005

public void createPartControl(Composite parent) { ... // add listener for editor page activation getSite().getPage().addPartListener(partListener); } public void dispose() { ... getSite().getPage().removePartListener(partListener); }

slide-5
SLIDE 5

Mining Repositories (1): DynaMine

public void createPartControl(Composite parent) { ... // add listener for editor page activation getSite().getPage().addPartListener(partListener); } public void dispose() { ... getSite().getPage().removePartListener(partListener); }

co-added

Benjamin Livshits, Thomas Zimmermann, “DynaMine: Finding common error patterns by mining software revision histories”, ESEC/FSE 2005

slide-6
SLIDE 6

Mining Repositories (1): DynaMine

public void createPartControl(Composite parent) { ... // add listener for editor page activation getSite().getPage().addPartListener(partListener); } public void dispose() { ... getSite().getPage().removePartListener(partListener); }

co-added Co-changed items form usage patterns Patterns are validated at runtime

Benjamin Livshits, Thomas Zimmermann, “DynaMine: Finding common error patterns by mining software revision histories”, ESEC/FSE 2005

slide-7
SLIDE 7

Mining Repositories (1): DynaMine

Benjamin Livshits, Thomas Zimmermann, “DynaMine: Finding common error patterns by mining software revision histories”, ESEC/FSE 2005

  • Evaluated on Eclipse and jEdit
  • Preprocessing CVS history takes a few days
  • Mining for co-changed items takes minutes
  • Found 56 previously unknown patterns
  • Found 263 pattern violations
slide-8
SLIDE 8

Mining Repositories (2): BugMem

Sunghun Kim, Kai Pan, E. James Whitehead, Jr., “Memories of Bug Fixes”, FSE 2006

public void computePackageFragmentRoots(...) { ... if (requiredProjectRsc.exists() && requiredProjectRsc.isOpen()) { ... } public void computePackageFragmentRoots(...) { ... if (JavaProject.hasJavaNature(requiredProjectRsc)) ... }

Fix

slide-9
SLIDE 9

Mining Repositories (2): BugMem

Sunghun Kim, Kai Pan, E. James Whitehead, Jr., “Memories of Bug Fixes”, FSE 2006

public void computePackageFragmentRoots(...) { ... if (requiredProjectRsc.exists() && requiredProjectRsc.isOpen()) { ... } public void computePackageFragmentRoots(...) { ... if (JavaProject.hasJavaNature(requiredProjectRsc)) ... }

Fix

slide-10
SLIDE 10

Mining Repositories (2): BugMem

Sunghun Kim, Kai Pan, E. James Whitehead, Jr., “Memories of Bug Fixes”, FSE 2006

public void computePackageFragmentRoots(...) { ... if (requiredProjectRsc.exists() && requiredProjectRsc.isOpen()) { ... } public void computePackageFragmentRoots(...) { ... if (JavaProject.hasJavaNature(requiredProjectRsc)) ... }

Fix Previous bugs form bug patterns Previous fixes form fix patterns

slide-11
SLIDE 11

Mining Repositories (2): BugMem

Sunghun Kim, Kai Pan, E. James Whitehead, Jr., “Memories of Bug Fixes”, FSE 2006

  • Evaluated on 5 open source projects:

ArgoUML, Columba, Eclipse, jEdit, Scarab

  • 19.3%–40.3% of bugs appear in the memories
  • 7.9%–15.5% of bug and fix pairs appear

in the memories

slide-12
SLIDE 12

Mining Traces (1): AMPLE

Valentin Dallmeier, Christian Lindig, Andreas Zeller, “Lightweight defect localization for Java”, ECOOP 2005

slide-13
SLIDE 13

Mining Traces (1): AMPLE

Valentin Dallmeier, Christian Lindig, Andreas Zeller, “Lightweight defect localization for Java”, ECOOP 2005

aStack aList aStack: List.add List.add, aClient Stack.push Stack.top List.remove Stack.pop List.remove List.get List.get,

slide-14
SLIDE 14

abcbacbba

Mining Traces (1): AMPLE

Valentin Dallmeier, Christian Lindig, Andreas Zeller, “Lightweight defect localization for Java”, ECOOP 2005

cc

  • bject

Trace

  • bject

Sequence Set

ab bc cb ba ac bb

class

ab bc cb ba ac bb cc

Sequence Set

slide-15
SLIDE 15

abcbacbba

Mining Traces (1): AMPLE

Valentin Dallmeier, Christian Lindig, Andreas Zeller, “Lightweight defect localization for Java”, ECOOP 2005

cc

  • bject

Trace

  • bject

Sequence Set

ab bc cb ba ac bb

class

ab bc cb ba ac bb cc

Sequence Set

Differences in sequence sets between passing and failing runs point to defects

slide-16
SLIDE 16

Mining Traces (1): AMPLE

Valentin Dallmeier, Christian Lindig, Andreas Zeller, “Lightweight defect localization for Java”, ECOOP 2005

  • Evaluated on NanoXML (33 seeded defects)
  • Identifies the defective class in 36% of cases
  • On average, a programmer has to investigate

10% of all classes to find the defective one

slide-17
SLIDE 17

Mining Traces (2): Perracotta

Jinlin Yang, David Evans, Deepali Bhardwaj, Thirumalesh Bhat, Manuvir Das, “Perracotta: Mining Temporal API Rules from Imperfect Traces”, ICSE 2006

Trace

ENTRY: Producer.main ENTRY: Buffer.add EXIT: Buffer.add ENTRY: Buffer.add EXIT: Buffer.add ENTRY: Buffer.stop EXIT: Buffer.stop EXIT: Producer.main

Temporal property patterns

[-P]*;(P;[-S]*;S;[-P]*)* ...

Properties instantiations

P=ENTRY: Buffer.add S=ENTRY: Buffer.stop P=ENTRY: Buffer.add S=EXIT: Producer.main ...

slide-18
SLIDE 18

Mining Traces (2): Perracotta

Jinlin Yang, David Evans, Deepali Bhardwaj, Thirumalesh Bhat, Manuvir Das, “Perracotta: Mining Temporal API Rules from Imperfect Traces”, ICSE 2006

Trace

ENTRY: Producer.main ENTRY: Buffer.add EXIT: Buffer.add ENTRY: Buffer.add EXIT: Buffer.add ENTRY: Buffer.stop EXIT: Buffer.stop EXIT: Producer.main

Temporal property patterns

[-P]*;(P;[-S]*;S;[-P]*)* ...

Properties instantiations

P=ENTRY: Buffer.add S=ENTRY: Buffer.stop P=ENTRY: Buffer.add S=EXIT: Producer.main ...

Properties instantiations form API rules

slide-19
SLIDE 19

Mining Traces (2): Perracotta

Jinlin Yang, David Evans, Deepali Bhardwaj, Thirumalesh Bhat, Manuvir Das, “Perracotta: Mining Temporal API Rules from Imperfect Traces”, ICSE 2006

  • Evaluated on Daisy, JBoss and Windows kernel
  • Analysis of traces takes several minutes
  • Found a defect in the NTFS file system
slide-20
SLIDE 20

Mining Source Code (1): PR-Miner

Zhenmin Li, Yuanyuan Zhou, “PR-Miner: Automatically extracting implicit programming rules and detecting violations in large software code”, ESEC/FSE 2005

static void getRelationDescription (...) { HeapTuple relTup; ... relTup = SearchSysCache (...); if (!HeapTupleIsValid (relTup)) elog (...); relForm = ...; ... ReleaseSysCache (relTup); }

Function’s source code

slide-21
SLIDE 21

Mining Source Code (1): PR-Miner

Zhenmin Li, Yuanyuan Zhou, “PR-Miner: Automatically extracting implicit programming rules and detecting violations in large software code”, ESEC/FSE 2005

static void getRelationDescription (...) { HeapTuple relTup; ... relTup = SearchSysCache (...); if (!HeapTupleIsValid (relTup)) elog (...); relForm = ...; ... ReleaseSysCache (relTup); }

Function’s source code Itemset

T: HeapTuple F: SearchSysCache F: HeapTupleIsValid F: elog T: Form_pg_class F: ReleaseSysCache ...

slide-22
SLIDE 22

Mining Source Code (1): PR-Miner

Zhenmin Li, Yuanyuan Zhou, “PR-Miner: Automatically extracting implicit programming rules and detecting violations in large software code”, ESEC/FSE 2005

Itemset #1

T: HeapTuple F: SearchSysCache F: HeapTupleIsValid F: elog T: Form_pg_class F: ReleaseSysCache ...

Itemset #2

T: StringInfoData F: getObjectClass T: HeapTuple F: SearchSysCache F: NameStr T: Relation F: ReleaseSysCache ...

Itemset #3

T: Form_pg_class F: SearchSysCache F: elog F: RelationIsVisible F: ReleaseSysCache ...

slide-23
SLIDE 23

Mining Source Code (1): PR-Miner

Zhenmin Li, Yuanyuan Zhou, “PR-Miner: Automatically extracting implicit programming rules and detecting violations in large software code”, ESEC/FSE 2005

Itemset #1

T: HeapTuple F: SearchSysCache F: HeapTupleIsValid F: elog T: Form_pg_class F: ReleaseSysCache ...

Itemset #2

T: StringInfoData F: getObjectClass T: HeapTuple F: SearchSysCache F: NameStr T: Relation F: ReleaseSysCache ...

Itemset #3

T: Form_pg_class F: SearchSysCache F: elog F: RelationIsVisible F: ReleaseSysCache ...

slide-24
SLIDE 24

Mining Source Code (1): PR-Miner

Zhenmin Li, Yuanyuan Zhou, “PR-Miner: Automatically extracting implicit programming rules and detecting violations in large software code”, ESEC/FSE 2005

Itemset #1

T: HeapTuple F: SearchSysCache F: HeapTupleIsValid F: elog T: Form_pg_class F: ReleaseSysCache ...

Itemset #2

T: StringInfoData F: getObjectClass T: HeapTuple F: SearchSysCache F: NameStr T: Relation F: ReleaseSysCache ...

Itemset #3

T: Form_pg_class F: SearchSysCache F: elog F: RelationIsVisible F: ReleaseSysCache ...

Frequent itemsets form programming rules Missing items point to defects

slide-25
SLIDE 25

Mining Source Code (1): PR-Miner

Zhenmin Li, Yuanyuan Zhou, “PR-Miner: Automatically extracting implicit programming rules and detecting violations in large software code”, ESEC/FSE 2005

  • Evaluated on Linux, PostgreSQL and Apache
  • Finds rules and violations within 2 minutes
  • Found 32,283 rules for Linux
  • Found 23 bugs in 180 violations
slide-26
SLIDE 26

Program JADET

Modeling objects’ behavior using finite state automata

a b c d ! a a b c d e f ! a

Mining Source Code (2): JADET

slide-27
SLIDE 27

The mining process:

  • verview

Program Method model #1 Method model #2 Method model #n Object usage model #1 Object usage model #2 Object usage model #3 Object usage model #4 Object usage model #m

slide-28
SLIDE 28

Building a method model

public Stack cS () { Random r = new Random (); int n = r.nextInt (); Stack s = new Stack (); int i = 0; while (i < n) { s.push (rand (r)); i++; } s.push (-1); return s; }

Random r = new Random (); int n = r.nextInt (); s = new Stack (); int i = 0; i < n i < n i+ + ; s.push (-1); s.push (rand (r));

slide-29
SLIDE 29

Random r = new Random (); int n = r.nextInt (); s = new Stack (); int i = 0; i < n i < n i+ + ; s.push (-1); s.push (rand (r));

Creating a usage model

slide-30
SLIDE 30

Creating a usage model

s = new Stack (); s.push (-1); s.push (rand (r));

slide-31
SLIDE 31

Creating a usage model

s.< init> () s.push (...) s.push (...)

slide-32
SLIDE 32

Creating a usage model

Random r = new Random (); int n = r.nextInt (); s = new Stack (); int i = 0; i < n i < n i+ + ; s.push (-1); s.push (rand (r));

slide-33
SLIDE 33

Creating a usage model

Random r = new Random (); int n = r.nextInt (); s.push (rand (r));

slide-34
SLIDE 34

Creating a usage model

r.< init> () r.nextInt () rand (r)

slide-35
SLIDE 35

Time & Models

Program Classes Methods # Models Time ACT–RBOT 0.8.2 2,492 26,501 187,137 9:03 AspectJ 1.5.3 2,957 36,044 253,084 13:19 Azureus 2.5.0.0 3,585 22,359 157,313 7:28 Columba 1.2 1,165 6,894 54,371 2:06 MusiComp 1.0 beta 347 2,078 17,321 1:02

slide-36
SLIDE 36

StringTokenizer st

in AdviceSignatureImpl.toAdviceName()

st.< init> () st.hasMoreT

  • kens ()

st.nextT

  • ken ()
slide-37
SLIDE 37

StringTokenizer st

in Factory.makeConstructorSig()

st.< init> () st.countT

  • kens ()

st.nextT

  • ken ()
slide-38
SLIDE 38

Class c

in AjTypeSystem.getAjType()

Map.containsKey (c) AjT ypeImpl.< init> (c) Map.get (c) Map.put (c, ...) AjT ypeImpl.< init> (c) Map.put (c, ...)

slide-39
SLIDE 39

Temporal properties

<init> ⋏ nextInt <init> ⋏ rand nextInt ⋏ rand rand ⋏ rand

r.< init> () r.nextInt () rand (r)

slide-40
SLIDE 40

Programming patterns

in Dump.println()

hasNext ⋏ hasNext hasNext ⋏ next getProperty ⋏ getProperty <init> ⋏ append …

slide-41
SLIDE 41

Programming patterns

Method

hasNext < next hasNext < hasNext Dump.println

✔ ✔

ClassPathManager.<init>

✔ ✔

OrderedLock.updateCO

✘ ✘

BcelObjectType.getA

✔ ✔

DeclareParents.verifyNIAP

✔ ✘

Missing properties point to anomalies

slide-42
SLIDE 42

The anomalous method

private boolean verifyNIAP (…) { … Iterator iter = …; while (iter.hasNext()) { … = iter.next(); … return verifyNIAP (…); } return true; }

↯

slide-43
SLIDE 43

Time & Anomalies

Program # Models # Anomalies Time ACT–RBOT 0.8.2 187,137 192 0:53 AspectJ 1.5.3 253,084 276 3:47 Azureus 2.5.0.0 157,313 365 0:50 Columba 1.2 54,371 64 0:11 MusiComp 1.0 beta 17,321 3 0:04

Top 10 contained 5 previously unknown defects

slide-44
SLIDE 44