Automatic Generation of Oracles for Exceptional Behaviors a c f - PowerPoint PPT Presentation

Automatic Generation of Oracles for Exceptional Behaviors a c f i t t r A * o m C p * l e t n t e A e * t s * E i W s n e C o A l l C D T * o * S c e u s S u m E e e I R n * o t e v t d y s a * E a d l e u t a Alberto Go ffi — alberto.go ffi @usi.ch 
 USI Università della Svizzera italiana (Lugano, Switzerland) Alessandra Gorla 
 IMDEA Software Institute (Madrid, Spain) Michael D. Ernst 
 University of Washington (Seattle, Washington, USA) Mauro Pezzè 
 USI Università della Svizzera italiana (Lugano, Switzerland)

The Oracle Problem Automatically generated test cases need oracles Formal specifications are often missing Developers write Javadoc comments Automatically generate oracles 
 from Javadoc comments

The Oracle Problem Automatically generated test cases need oracles Developers do not write formal specifications Developers write Javadoc comments Automatically generate oracles 
 from Javadoc comments

The Oracle Problem Automatically generated test cases need oracles Developers do not write formal specifications Developers do write Javadoc comments Automatically generate oracles 
 from Javadoc comments

The Oracle Problem Automatically generated test cases need oracles Developers do not write formal specifications Developers do write Javadoc comments Automatically generate oracles 
 from Javadoc comments

Testers under-test exceptions % covered statements % covered throw statements 100 80 60 40 20 0 cli collections compress dbcp email io lang math net pool primitives guava average

Testers under-test exceptions % covered statements % covered throw statements 100 80 60 40 20 0 cli collections compress dbcp email io lang math net pool primitives guava average

Testers under-test exceptions % covered statements % covered throw statements 100 80 60 40 20 0 cli collections compress dbcp email io lang math net pool primitives guava average

Testers under-test exceptions % covered statements % covered throw statements 100 -25% 80 60 40 20 0 cli collections compress dbcp email io lang math net pool primitives guava average

…and automatic testing does not help public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) {…} public Object next() {…} … }

…and automatic testing does not help public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) {…} public Object next() {…} … } public void test { FilterIterator i = new FilterIterator( null , null ); i.next(); }

What does an exception mean? public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) {…} public Object next() {…} … } public void test { FilterIterator i = new FilterIterator( null , null ); i.next(); } NullPointerException

What does an exception mean? public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) {…} public Object next() {…} … } public void test { FilterIterator i = new FilterIterator( null , null ); i.next(); } ? 1. Expected behavior NullPointerException 2. Illegal input 3. Implementation bug

What does an exception mean? public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) {…} /** * @throws NullPointerException if either * the iterator or predicate are null */ public Object next() {…} } public void test { FilterIterator i = new FilterIterator( null , null ); i.next(); } ? 1. Expected behavior NullPointerException 2. Illegal input 3. Implementation bug

What does an exception mean? public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) {…} /** * @throws NullPointerException if either * the iterator or predicate are null */ public Object next() {…} } public void test { FilterIterator i = new FilterIterator( null , null ); i.next(); } 1. Expected behavior NullPointerException 2. Illegal input 3. Implementation bug

What does an exception mean? public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) {…} /** * @throws NullPointerException if either * the iterator or predicate are null */ public Object next() {…} } F a l s public void test { e A FilterIterator i = new FilterIterator( null , null ); l a i.next(); r m } 1. Expected behavior NullPointerException 2. Illegal input 3. Implementation bug

Consider another implementation public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) {…} /** * @throws NullPointerException if either * the iterator or predicate are null */ public Object next() {…} } public void test { FilterIterator i = new FilterIterator( null , null ); i.next(); } 3. Implementation bug ? 1. Expected behavior No Exceptions 2. Illegal input

Consider another implementation public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) {…} /** * @throws NullPointerException if either * the iterator or predicate are null */ public Object next() {…} } public void test { FilterIterator i = new FilterIterator( null , null ); i.next(); } 1. Expected behavior No Exceptions 2. Illegal input 3. Implementation bug

Consider another implementation public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) {…} /** * @throws NullPointerException if either * the iterator or predicate are null */ public Object next() {…} } M i s s e public void test { d FilterIterator i = new FilterIterator( null , null ); A l i.next(); a r } m 1. Expected behavior No Exceptions 2. Illegal input 3. Implementation bug

public void test { FilterIterator i = Test new FilterIterator( null , null ); i.next(); }

public void test { FilterIterator i = Test new FilterIterator( null , null ); i.next(); } /** * @throws NullPointerException if either Documentation * the iterator or predicate are null */ public Object next() {…}

public void test { FilterIterator i = Test new FilterIterator( null , null ); i.next(); + } /** * @throws NullPointerException if either Documentation * the iterator or predicate are null */ public Object next() {…}

public void test { FilterIterator i = Test new FilterIterator( null , null ); i.next(); + } /** * @throws NullPointerException if either Documentation * the iterator or predicate are null */ public Object next() {…} public void test { FilterIterator i = new FilterIterator( null , null ); if (i.getIterator() == null || i.getPredicate() == null ) { try { i.next(); Test + Oracle fail(“Expected NPE not thrown”); } catch (NullPointerException e) { // Expected exception } } else { i.next(); } }

public void test { FilterIterator i = Test new FilterIterator( null , null ); i.next(); + } /** * @throws NullPointerException if either Documentation * the iterator or predicate are null */ public Object next() {…} public void test { FilterIterator i = new FilterIterator( null , null ); if (i.getIterator() == null || i.getPredicate() == null ) { try { i.next(); Test + Oracle fail(“Expected NPE not thrown”); } catch (NullPointerException e) { // Expected exception } } else { i.next(); } }

public void test { FilterIterator i = Test new FilterIterator( null , null ); i.next(); + } /** * @throws NullPointerException if either Documentation * the iterator or predicate are null */ public Object next() {…} public void test { FilterIterator i = new FilterIterator( null , null ); if (i.getIterator() == null || i.getPredicate() == null ) { try { i.next(); Test + Oracle fail(“Expected NPE not thrown”); } catch (NullPointerException e) { // Expected exception } } else { i.next(); } }

Toradocu Javadoc 
 Condition 
 Oracle 
 Enhanced Source Extractor Translator Generator Tests Tests

Javadoc 
 Condition 
 Oracle 
 Enhanced Source Extractor Translator Generator Tests

Javadoc 
 Condition 
 Oracle 
 Enhanced Source Extractor Translator Generator Tests 1. Method Signature 2. Expected Exceptions Source 3. Conditions (Text)

Javadoc 
 Condition 
 Oracle 
 Enhanced Source Extractor Translator Generator Tests 1. Method Signature 2. Expected Exceptions Source 3. Conditions (Text) /** 1. FilterIterator.next() * @throws NullPointerException 2. NullPointerException * if either the iterator * or predicate are null 3. “if either the iterator 
 */ or predicate are null” public Object next() {…}

Javadoc 
 Condition 
 Oracle 
 Enhanced Source Extractor Translator Generator Tests 1. Method Signature 1. Method Signature 2. Expected Exceptions 2. Expected Exceptions 3. Conditions (Text) 3. Conditions (Code)

Javadoc 
 Condition 
 Oracle 
 Enhanced Source Extractor Translator Generator Tests 1. Method Signature 1. Method Signature 2. Expected Exceptions 2. Expected Exceptions 3. Conditions (Text) 3. Conditions (Code) 1. FilterIterator.next() 1. FilterIterator.next() 2. NullPointerException 2. NullPointerException 3. “if either the iterator or 3. getIterator() == null || 
 predicate are null” getPredicate() == null