Automatic Generation of Oracles for Exceptional Behaviors a c f - - PowerPoint PPT Presentation

▶
automatic generation of oracles for exceptional behaviors
SMART_READER_LITE
LIVE PREVIEW

Automatic Generation of Oracles for Exceptional Behaviors a c f - - PowerPoint PPT Presentation

Mar 24, 2024 302 likes •1.13k views

Automatic Generation of Oracles for Exceptional Behaviors a c f i t t r A * o m C p * l e t n t e A e * t s * E i W s n e C o A l l C D T * o * S c e u s S u m E e e I R n * o t e v t

slide-1
SLIDE 1

Automatic Generation of Oracles for Exceptional Behaviors

Alberto Goffi — alberto.goffi@usi.ch
 USI Università della Svizzera italiana (Lugano, Switzerland) Alessandra Gorla
 IMDEA Software Institute (Madrid, Spain) Michael D. Ernst
 University of Washington (Seattle, Washington, USA) Mauro Pezzè
 USI Università della Svizzera italiana (Lugano, Switzerland)

C
  • n
s i s t e n t * C
  • m
p l e t e * W e l l D
  • c
u m e n t e d * E a s y t
  • R
e u s e *

*

E v a l u a t e d

* I S S T A *

A r t i f a c t

* A E C

slide-2
SLIDE 2

The Oracle Problem

Automatically generated test cases need oracles Formal specifications are often missing Developers write Javadoc comments

Automatically generate oracles
 from Javadoc comments

slide-3
SLIDE 3

The Oracle Problem

Automatically generated test cases need oracles Developers do not write formal specifications Developers write Javadoc comments

Automatically generate oracles
 from Javadoc comments

slide-4
SLIDE 4

The Oracle Problem

Automatically generated test cases need oracles Developers do not write formal specifications Developers do write Javadoc comments

Automatically generate oracles
 from Javadoc comments

slide-5
SLIDE 5

The Oracle Problem

Automatically generated test cases need oracles Developers do not write formal specifications Developers do write Javadoc comments

Automatically generate oracles
 from Javadoc comments

slide-6
SLIDE 6

Testers under-test exceptions

20 40 60 80 100 cli collections compress dbcp email io lang math net pool primitives guava average % covered statements % covered throw statements

slide-7
SLIDE 7

Testers under-test exceptions

20 40 60 80 100 cli collections compress dbcp email io lang math net pool primitives guava average % covered statements % covered throw statements

slide-8
SLIDE 8

20 40 60 80 100 cli collections compress dbcp email io lang math net pool primitives guava average % covered statements % covered throw statements

Testers under-test exceptions

slide-9
SLIDE 9

20 40 60 80 100 cli collections compress dbcp email io lang math net pool primitives guava average % covered statements % covered throw statements

  • 25%

Testers under-test exceptions

slide-10
SLIDE 10

…and automatic testing does not help

public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) {…} public Object next() {…} … }

slide-11
SLIDE 11

…and automatic testing does not help

public void test { FilterIterator i = new FilterIterator(null, null); i.next(); } public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) {…} public Object next() {…} … }

slide-12
SLIDE 12

What does an exception mean?

public void test { FilterIterator i = new FilterIterator(null, null); i.next(); } public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) {…} public Object next() {…} … }

NullPointerException

slide-13
SLIDE 13
  • 1. Expected behavior
  • 2. Illegal input
  • 3. Implementation bug

public void test { FilterIterator i = new FilterIterator(null, null); i.next(); } public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) {…} public Object next() {…} … }

NullPointerException

What does an exception mean?

?

slide-14
SLIDE 14

public void test { FilterIterator i = new FilterIterator(null, null); i.next(); } public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) {…} /** * @throws NullPointerException if either * the iterator or predicate are null */ public Object next() {…} }

NullPointerException

  • 1. Expected behavior
  • 2. Illegal input
  • 3. Implementation bug

What does an exception mean?

?

slide-15
SLIDE 15

public void test { FilterIterator i = new FilterIterator(null, null); i.next(); }

NullPointerException

public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) {…} /** * @throws NullPointerException if either * the iterator or predicate are null */ public Object next() {…} }

What does an exception mean?

  • 1. Expected behavior
  • 2. Illegal input
  • 3. Implementation bug
slide-16
SLIDE 16
  • 1. Expected behavior
  • 2. Illegal input
  • 3. Implementation bug

public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) {…} /** * @throws NullPointerException if either * the iterator or predicate are null */ public Object next() {…} } public void test { FilterIterator i = new FilterIterator(null, null); i.next(); }

F a l s e A l a r m

NullPointerException

What does an exception mean?

slide-17
SLIDE 17

Consider another implementation

public void test { FilterIterator i = new FilterIterator(null, null); i.next(); } public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) {…} /** * @throws NullPointerException if either * the iterator or predicate are null */ public Object next() {…} }

No Exceptions

  • 1. Expected behavior
  • 2. Illegal input
  • 3. Implementation bug?
slide-18
SLIDE 18

public void test { FilterIterator i = new FilterIterator(null, null); i.next(); }

  • 1. Expected behavior
  • 2. Illegal input
  • 3. Implementation bug

No Exceptions

public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) {…} /** * @throws NullPointerException if either * the iterator or predicate are null */ public Object next() {…} }

Consider another implementation

slide-19
SLIDE 19
  • 1. Expected behavior
  • 2. Illegal input
  • 3. Implementation bug

public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) {…} /** * @throws NullPointerException if either * the iterator or predicate are null */ public Object next() {…} } public void test { FilterIterator i = new FilterIterator(null, null); i.next(); }

No Exceptions

Consider another implementation

M i s s e d A l a r m

slide-20
SLIDE 20

public void test { FilterIterator i = new FilterIterator(null, null); i.next(); }

Test

slide-21
SLIDE 21

/** * @throws NullPointerException if either * the iterator or predicate are null */ public Object next() {…}

Test Documentation

public void test { FilterIterator i = new FilterIterator(null, null); i.next(); }

slide-22
SLIDE 22

Test Documentation

+

public void test { FilterIterator i = new FilterIterator(null, null); i.next(); }

/** * @throws NullPointerException if either * the iterator or predicate are null */ public Object next() {…}

slide-23
SLIDE 23

Test Documentation

+

Test + Oracle

public void test { FilterIterator i = new FilterIterator(null, null); if (i.getIterator() == null || i.getPredicate() == null) { try { i.next(); fail(“Expected NPE not thrown”); } catch(NullPointerException e) { // Expected exception } } else { i.next(); } }

public void test { FilterIterator i = new FilterIterator(null, null); i.next(); }

/** * @throws NullPointerException if either * the iterator or predicate are null */ public Object next() {…}

slide-24
SLIDE 24

Test Documentation

+

Test + Oracle

public void test { FilterIterator i = new FilterIterator(null, null); i.next(); }

/** * @throws NullPointerException if either * the iterator or predicate are null */ public Object next() {…} public void test { FilterIterator i = new FilterIterator(null, null); if (i.getIterator() == null || i.getPredicate() == null) { try { i.next(); fail(“Expected NPE not thrown”); } catch(NullPointerException e) { // Expected exception } } else { i.next(); } }

slide-25
SLIDE 25

Test Documentation

+

Test + Oracle

public void test { FilterIterator i = new FilterIterator(null, null); i.next(); }

/** * @throws NullPointerException if either * the iterator or predicate are null */ public Object next() {…} public void test { FilterIterator i = new FilterIterator(null, null); if (i.getIterator() == null || i.getPredicate() == null) { try { i.next(); fail(“Expected NPE not thrown”); } catch(NullPointerException e) { // Expected exception } } else { i.next(); } }

slide-26
SLIDE 26

Toradocu

Source

Javadoc
 Extractor Condition
 Translator Oracle
 Generator

Enhanced Tests Tests

slide-27
SLIDE 27

Source

Javadoc
 Extractor Condition
 Translator Oracle
 Generator

Enhanced Tests

slide-28
SLIDE 28

Source

Condition
 Translator Oracle
 Generator

Enhanced Tests

Javadoc
 Extractor

Source

  • 1. Method Signature
  • 2. Expected Exceptions
  • 3. Conditions (Text)
slide-29
SLIDE 29

Source

Condition
 Translator Oracle
 Generator

Enhanced Tests

Javadoc
 Extractor

/** * @throws NullPointerException * if either the iterator * or predicate are null */ public Object next() {…}

  • 1. FilterIterator.next()
  • 2. NullPointerException
  • 3. “if either the iterator

  • r predicate are null”
  • 1. Method Signature
  • 2. Expected Exceptions
  • 3. Conditions (Text)

Source

slide-30
SLIDE 30

Condition
 Translator

Source

Oracle
 Generator

Enhanced Tests

Javadoc
 Extractor

  • 1. Method Signature
  • 2. Expected Exceptions
  • 3. Conditions (Code)
  • 1. Method Signature
  • 2. Expected Exceptions
  • 3. Conditions (Text)
slide-31
SLIDE 31

Condition
 Translator

Source

Oracle
 Generator

Enhanced Tests

Javadoc
 Extractor

  • 1. FilterIterator.next()
  • 2. NullPointerException
  • 3. “if either the iterator or

predicate are null”

  • 1. FilterIterator.next()
  • 2. NullPointerException
  • 3. getIterator() == null ||


getPredicate() == null

  • 1. Method Signature
  • 2. Expected Exceptions
  • 3. Conditions (Code)
  • 1. Method Signature
  • 2. Expected Exceptions
  • 3. Conditions (Text)
slide-32
SLIDE 32

Condition
 Translator

Source Enhanced Tests

Javadoc
 Extractor Oracle
 Generator

Tests

  • 1. FilterIterator.next()
  • 2. NullPointerException
  • 3. getIterator() == null ||


getPredicate() == null

slide-33
SLIDE 33

Condition
 Translator

Source Enhanced Tests

Javadoc
 Extractor Oracle
 Generator

Test

+ =

Test + Oracle

Tests

public void test { FilterIterator i = new FilterIterator(null, null); if (i.getIterator()==null || i.getPredicate()==null) { try { i.next(); fail(“Expected NPE not thrown”); } catch(NullPointerException e) { // Expected exception } } else { i.next(); } } public void test { FilterIterator i = new FilterIterator(null, null); i.next(); }

  • 1. FilterIterator.next()
  • 2. NullPointerException
  • 3. getIterator() == null || getPredicate() == null

Oracle

slide-34
SLIDE 34

Source

Oracle
 Generator

Enhanced Tests

Javadoc
 Extractor Condition
 Translator Java boolean condition Natural Language Condition

slide-35
SLIDE 35

Source

Oracle
 Generator

Enhanced Tests

Javadoc
 Extractor Condition
 Translator array.length == 0 “if the array is empty”

slide-36
SLIDE 36

Source

Oracle
 Generator

Enhanced Tests

Javadoc
 Extractor Condition
 Translator foo > 0 “if foo is positive”

slide-37
SLIDE 37

Condition Translator

Subject Matcher NL Parser Predicate
 Matcher

Java Condition Natural Language Condition

slide-38
SLIDE 38

Subject Matcher Predicate
 Matcher NL Parser “if either the iterator or predicate are null”

slide-39
SLIDE 39

Subject Matcher Predicate
 Matcher NL Parser “if either the iterator or predicate are null”

Proposition Subject Predicate 1 “iterator” “are null” 2 “predicate” “are null”

slide-40
SLIDE 40

“if either the iterator or predicate are null” Subject Matcher Predicate
 Matcher NL Parser

Proposition Subject Predicate 1 “iterator” “are null” 2 “predicate” “are null”

slide-41
SLIDE 41

“if either the iterator or predicate are null” Subject Matcher Predicate
 Matcher NL Parser

slide-42
SLIDE 42

“if either the iterator or predicate are null” Subject Matcher Predicate
 Matcher NL Parser

Proposition Subject Predicate 1 “iterator” “are null” 2 “predicate” “are null”

“or” “predicate”,
 “are null” “iterator”,
 “are null”

slide-43
SLIDE 43

Predicate
 Matcher NL Parser

Proposition Subject Predicate 1 “iterator” “are null” 2 “predicate” “are null”

Subject Matcher

slide-44
SLIDE 44

Predicate
 Matcher NL Parser Subject Matcher

Subject: “iterator”

slide-45
SLIDE 45

Predicate
 Matcher NL Parser Subject Matcher Candidates Formal Parameters Class Name Non-void Nullary Methods Fields

Subject: “iterator”

slide-46
SLIDE 46

Predicate
 Matcher NL Parser Subject Matcher

public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) {…} public Iterator getIterator() {…} public Predicate getPredicate() {…} public Object next() {…} … }

Subject: “iterator”

Candidate Distance

FilterIterator getIterator getPredicate next

…

Candidates Formal Parameters Class Name Non-void Nullary Methods Fields

slide-47
SLIDE 47

Predicate
 Matcher NL Parser Subject Matcher

Subject: “iterator”

Candidate Distance

FilterIterator

6

getIterator

3

getPredicate

9

next

6 …

Candidates Formal Parameters Class Name Non-void Nullary Methods Fields

slide-48
SLIDE 48

Predicate
 Matcher NL Parser Subject Matcher

Subject: “iterator” => getIterator()

Candidate Distance

FilterIterator

6

getIterator

3

getPredicate

9

next

6 …

Candidates Formal Parameters Class Name Non-void Nullary Methods Fields

slide-49
SLIDE 49

Predicate
 Matcher NL Parser Subject Matcher

Proposition Subject Predicate 1 FilterIterator.getIterator() “are null” 2 FilterIterator.getPredicate() “are null” Proposition Subject Predicate 1 “iterator” “are null” 2 “predicate” “are null”

slide-50
SLIDE 50

Subject Matcher NL Parser Predicate
 Matcher

Proposition Subject Predicate 1 FilterIterator.getIterator() “are null” 2 FilterIterator.getPredicate() “are null”

slide-51
SLIDE 51

Subject Matcher NL Parser Predicate
 Matcher

Proposition Subject Predicate 1 FilterIterator.getIterator() “are null” 2 FilterIterator.getPredicate() “are null” Proposition Subject Predicate 1 FilterIterator.getIterator() “==null” 2 FilterIterator.getPredicate() “==null”

slide-52
SLIDE 52

Subject Matcher NL Parser Predicate
 Matcher

/** * @throws IllegalArgumentException if employees is empty */ public void sortByAge(List employees) {…}

Proposition Subject Predicate 1 employees “is empty”

slide-53
SLIDE 53

Subject Matcher NL Parser Predicate
 Matcher

Candidate Distance

clear

7

isEmpty

1

iterator

6

size

7 …

Proposition Subject Predicate 1 employees “is empty”

/** * @throws IllegalArgumentException if employees is empty */ public void sortByAge(List employees) {…}

slide-54
SLIDE 54

Subject Matcher NL Parser Predicate
 Matcher

Proposition Subject Predicate 1 employees “is empty” Condition Translation 1 employees.isEmpty()

/** * @throws IllegalArgumentException if employees is empty */ public void sortByAge(List employees) {…}

slide-55
SLIDE 55

Subject Matcher NL Parser Predicate
 Matcher “if either the iterator or predicate are null”

Condition Translation 1 FilterIterator.getIterator()==null 2 FilterIterator.getPredicate()==null

slide-56
SLIDE 56

“if either the iterator or predicate are null”

Condition Translation 1 (FilterIterator.getIterator()==null)==true 2 (FilterIterator.getPredicate()==null)==true

Positive conditions

slide-57
SLIDE 57

“if either the iterator or predicate are null”

Condition Translation 1 (FilterIterator.getIterator()==null)==true 2 (FilterIterator.getPredicate()==null)==true

Positive conditions Negative conditions

Condition Translation 1 (FilterIterator.getIterator()==null)==false 2 (FilterIterator.getPredicate()==null)==false

“if either the iterator or predicate aren’t null”

slide-58
SLIDE 58

Evaluation

slide-59
SLIDE 59

From NL to Java Conditions

0% 25% 50% 75% 100%

ArrayListMultimap AtomicDoubleArray ConcurrentHashMultiset Doubles Floats MoreObjects Shorts Strings Verify FilterIterator FixedOrderComparator ListPopulation PredicatedMap Correct Partially Correct Wrong Missing

slide-60
SLIDE 60

From NL to Java Conditions

0% 25% 50% 75% 100%

ArrayListMultimap AtomicDoubleArray ConcurrentHashMultiset Doubles Floats MoreObjects Shorts Strings Verify FilterIterator FixedOrderComparator ListPopulation PredicatedMap Correct Partially Correct Wrong Missing

slide-61
SLIDE 61

0% 25% 50% 75% 100%

ArrayListMultimap AtomicDoubleArray ConcurrentHashMultiset Doubles Floats MoreObjects Shorts Strings Verify FilterIterator FixedOrderComparator ListPopulation PredicatedMap Correct Partially Correct Wrong Missing

From NL to Java Conditions

Precision: 92% Recall: 73%

slide-62
SLIDE 62

False Alarms — Google Guava

4 9 13 18 22 ArrayListMultimap AtomicDoubleArray ConcurrentHashMultiset Doubles Floats MoreObjects Shorts Strings Verify Average Implicit oracle Toradocu (EvoSuite Test Cases)

slide-63
SLIDE 63

4 9 13 18 22 ArrayListMultimap AtomicDoubleArray ConcurrentHashMultiset Doubles Floats MoreObjects Shorts Strings Verify Average Implicit oracle Toradocu

  • 36%

(EvoSuite Test Cases)

False Alarms — Google Guava

slide-64
SLIDE 64

True Alarms — Apache Commons

Implicit Toradocu 0% 20% 40% 60% 80% 100% Pass False alarms True alarms

slide-65
SLIDE 65

True Alarms — Apache Commons

Implicit Toradocu 0% 20% 40% 60% 80% 100% Pass False alarms True alarms

slide-66
SLIDE 66

Regression Toradocu 0% 20% 40% 60% 80% 100%

True Alarms — Apache Commons

Implicit Toradocu 0% 20% 40% 60% 80% 100% Pass False alarms True alarms

slide-67
SLIDE 67

Regression Toradocu 0% 20% 40% 60% 80% 100%

True Alarms — Apache Commons

Implicit Toradocu 0% 20% 40% 60% 80% 100% Pass False alarms True alarms

slide-68
SLIDE 68

Regression Toradocu 0% 20% 40% 60% 80% 100% Heuristics Toradocu 0% 20% 40% 60% 80% 100%

True Alarms — Apache Commons

Implicit Toradocu 0% 20% 40% 60% 80% 100% Pass False alarms True alarms

slide-69
SLIDE 69

Regression Toradocu 0% 20% 40% 60% 80% 100% Heuristics Toradocu 0% 20% 40% 60% 80% 100%

True Alarms — Apache Commons

Implicit Toradocu 0% 20% 40% 60% 80% 100% Pass False alarms True alarms

slide-70
SLIDE 70

What’s next?

Oracles for exceptions on arrays/containers elements @throws IllegalArgumentException if the array contains null input values @param foo must not be null void sort(@NotNull int[] a) return values @return the size of the array …more

slide-71
SLIDE 71

What’s next?

Oracles for exceptions on arrays/containers elements @throws IllegalArgumentException if the array contains null input values @param foo must not be null void sort(@NotNull int[] a) return values @return the size of the array …more

slide-72
SLIDE 72

20 40 60 80 100 cli collections compress dbcp email io lang math net pool primitives guava average % covered statements % covered throw statements

  • 25%

Testers under-test exceptions

slide-73
SLIDE 73

Toradocu

Source

Javadoc
 Extractor Condition
 Translator Oracle
 Generator

Enhanced Tests Tests

20 40 60 80 100 cli collections compress dbcp email io lang math net pool primitives guava average % covered statements % covered throw statements

  • 25%

Testers under-test exceptions

slide-74
SLIDE 74

Toradocu

Source

Javadoc
 Extractor Condition
 Translator Oracle
 Generator

Enhanced Tests Tests

20 40 60 80 100 cli collections compress dbcp email io lang math net pool primitives guava average % covered statements % covered throw statements

  • 25%

Testers under-test exceptions

0% 25% 50% 75% 100%

ArrayListMultimap AtomicDoubleArray ConcurrentHashMultiset Doubles Floats MoreObjects Shorts Strings Verify FilterIterator FixedOrderComparator ListPopulation PredicatedMap Correct Partially Correct Wrong Missing

From NL to Java Conditions

Precision: 92% Recall: 73%

slide-75
SLIDE 75

Toradocu

Source

Javadoc
 Extractor Condition
 Translator Oracle
 Generator

Enhanced Tests Tests

20 40 60 80 100 cli collections compress dbcp email io lang math net pool primitives guava average % covered statements % covered throw statements

  • 25%

Testers under-test exceptions

0% 25% 50% 75% 100%

ArrayListMultimap AtomicDoubleArray ConcurrentHashMultiset Doubles Floats MoreObjects Shorts Strings Verify FilterIterator FixedOrderComparator ListPopulation PredicatedMap Correct Partially Correct Wrong Missing

From NL to Java Conditions

Precision: 92% Recall: 73%

4 9 13 18 22 ArrayListMultimap AtomicDoubleArray ConcurrentHashMultiset Doubles Floats MoreObjects Shorts Strings Verify Average Implicit oracle Toradocu

  • 36%

(EvoSuite Test Cases)

False Alarms — Google Guava

slide-76
SLIDE 76

Toradocu

Source

Javadoc
 Extractor Condition
 Translator Oracle
 Generator

Enhanced Tests Tests

0% 25% 50% 75% 100%

ArrayListMultimap AtomicDoubleArray ConcurrentHashMultiset Doubles Floats MoreObjects Shorts Strings Verify FilterIterator FixedOrderComparator ListPopulation PredicatedMap Correct Partially Correct Wrong Missing

From NL to Java Conditions

Precision: 92% Recall: 73%

4 9 13 18 22 ArrayListMultimap AtomicDoubleArray ConcurrentHashMultiset Doubles Floats MoreObjects Shorts Strings Verify Average Implicit oracle Toradocu

  • 36%

(EvoSuite Test Cases)

False Alarms — Google Guava

20 40 60 80 100 cli collections compress dbcp email io lang math net pool primitives guava average % covered statements % covered throw statements

  • 25%

Testers under-test exceptions

github.com/albertogoffi/toradocu alberto.goffi@usi.ch

slide-77
SLIDE 77
slide-78
SLIDE 78

Related Work

Heuristics JCrasher, Crash’n’Check (Csallner, and Smaragdakis. ICSE ’05) Randoop (Pacheco, Lahiri, Ernst, and Ball. ICSE ’07) Specifications ASTOOT (Doong, and Frankl. TOSEM ’94) Models, contracts, … Properties Cross-checking oracles (Carzaniga, Goffi, Gorla, Mattavelli, and Pezzè. ICSE ’14) Metamorphic testing (Chen, Kuo, Tse, and Zhou. STEP ’13) Symmetric testing (Gotlieb. ISSRE ’03) Natural Language Documentation @tComment (Tan, Marinov, Tan, and Leavens. ICST ’12) aComment (Tan, Zhou, and Padioleau. ICSE ’11) iComment (Tan, Yuan, Krishna, and Zhou. SOSP ’07)

slide-79
SLIDE 79

When does Toradocu fail?

Conditions on the elements in a container 


  • if collection or any of its elements is null

  • if any element in the arrays is null

  • if notes has fewer than 2 elements

Specific checks


  • if the runtime type of the specified array is not a supertype
  • f the runtime type of every element in this collection

  • if x > 0

Missing predicate translation (No output)


  • if the charset is not valid

  • if closed
slide-80
SLIDE 80

Examples

@throws NullPointerException if the predicate or either closure is null … ifClosure(Predicate predicate, Closure trueClosure, Closure falseClosure) {…} ==> predicate == null || trueClosure == null || falseClosure == null @throws UnsupportedOperationException if the comparator is locked … checkLocked() {…} ==> target.isLocked()