Test Generation with JML Part I JMLUnitNG Wojciech Mostowski, - - PowerPoint PPT Presentation

test generation with jml
SMART_READER_LITE
LIVE PREVIEW

Test Generation with JML Part I JMLUnitNG Wojciech Mostowski, - - PowerPoint PPT Presentation

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References Test Generation with JML Part I JMLUnitNG Wojciech Mostowski, Gabriele Paganelli http://wwwhome.ewi.utwente.nl/~mostowskiwi/


slide-1
SLIDE 1

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Test Generation with JML

Part I – JMLUnitNG Wojciech Mostowski, Gabriele Paganelli

http://wwwhome.ewi.utwente.nl/~mostowskiwi/ http://www.cse.chalmers.se/~gabpag/

University of T wente Chalmers University of Technology

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-2
SLIDE 2

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Overview

Part I

◮ JML as test oracle

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-3
SLIDE 3

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Overview

Part I

◮ JML as test oracle ◮ Test data generation

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-4
SLIDE 4

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Overview

Part I

◮ JML as test oracle ◮ Test data generation ◮ JMLTestNG

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-5
SLIDE 5

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Overview

Part I

◮ JML as test oracle ◮ Test data generation ◮ JMLTestNG ◮ Good specifications for testing

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-6
SLIDE 6

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Overview

Part I

◮ JML as test oracle ◮ Test data generation ◮ JMLTestNG ◮ Good specifications for testing

Part II

◮ Provide even better test data with KeY

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-7
SLIDE 7

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

The Basic Idea

Use JML specs to check I/O behaviour of methods

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-8
SLIDE 8

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

The Basic Idea

Use JML specs to check I/O behaviour of methods

◮ Take the input test data, evaluate the precondition

◮ if true run the method with input data ◮ if false skip – meaningless test

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-9
SLIDE 9

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

The Basic Idea

Use JML specs to check I/O behaviour of methods

◮ Take the input test data, evaluate the precondition

◮ if true run the method with input data ◮ if false skip – meaningless test

◮ After the execution of the method evaluate the

postcondition

◮ if true – test passed ◮ if false – test fail, quote the values of the input data

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-10
SLIDE 10

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Running Example

/*@ public normal_behavior requires param >= 0; ensures \result >= 10; also public normal_behavior requires param < 0; ensures \result < -10; @*/ public int makeHole(int param) { if(param >= 0) { return param + 10; }else{ return param - 10; } }

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-11
SLIDE 11

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Running Example

/*@ public normal_behavior requires param >= 0; ensures \result >= 10; also public normal_behavior requires param < 0; ensures \result < -10; @*/ public int makeHole(int param) { if(param >= 0) { return param + 10; }else{ return param - 10; } } ◮ Take param==0, first

precondition true, the first spec defines a meaningful test

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-12
SLIDE 12

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Running Example

/*@ public normal_behavior requires param >= 0; ensures \result >= 10; also public normal_behavior requires param < 0; ensures \result < -10; @*/ public int makeHole(int param) { if(param >= 0) { return param + 10; }else{ return param - 10; } } ◮ Take param==0, first

precondition true, the first spec defines a meaningful test

◮ The second does not, skip

further checks

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-13
SLIDE 13

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Running Example

/*@ public normal_behavior requires param >= 0; ensures \result >= 10; also public normal_behavior requires param < 0; ensures \result < -10; @*/ public int makeHole(int param) { if(param >= 0) { return param + 10; }else{ return param - 10; } } ◮ Take param==0, first

precondition true, the first spec defines a meaningful test

◮ The second does not, skip

further checks

◮ Execute method, check

the postcondition(s) (true), test pass

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-14
SLIDE 14

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Running Example

/*@ public normal_behavior requires param >= 0; ensures \result >= 10; also public normal_behavior requires param < 0; ensures \result < -10; @*/ public int makeHole(int param) { if(param >= 0) { return param + 10; }else{ return param - 10; } } ◮ Take param==0, first

precondition true, the first spec defines a meaningful test

◮ The second does not, skip

further checks

◮ Execute method, check

the postcondition(s) (true), test pass

◮ Test with other inputs. . .

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-15
SLIDE 15

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Running Example

/*@ public normal_behavior requires param >= 0; ensures \result >= 10; also public normal_behavior requires param < 0; ensures \result < -10; @*/ public int makeHole(int param) { if(param >= 0) { return param + 10; }else{ return param - 10; } } ◮ Take param==0, first

precondition true, the first spec defines a meaningful test

◮ The second does not, skip

further checks

◮ Execute method, check

the postcondition(s) (true), test pass

◮ Test with other inputs. . . ◮ What inputs?!

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-16
SLIDE 16

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Test Data Generation

◮ Primitive types:

◮ Choose characteristic and borderline values

e.g. Integer.MIN_VALUE, -1, 0, 1, Integer.MAX_VALUE

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-17
SLIDE 17

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Test Data Generation

◮ Primitive types:

◮ Choose characteristic and borderline values

e.g. Integer.MIN_VALUE, -1, 0, 1, Integer.MAX_VALUE

◮ Provide manual input: -872463, 123316

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-18
SLIDE 18

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Test Data Generation

◮ Primitive types:

◮ Choose characteristic and borderline values

e.g. Integer.MIN_VALUE, -1, 0, 1, Integer.MAX_VALUE

◮ Provide manual input: -872463, 123316 ◮ Still may not be enough – KeY can do better

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-19
SLIDE 19

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Test Data Generation

◮ Primitive types:

◮ Choose characteristic and borderline values

e.g. Integer.MIN_VALUE, -1, 0, 1, Integer.MAX_VALUE

◮ Provide manual input: -872463, 123316 ◮ Still may not be enough – KeY can do better

◮ Reference types:

◮ Objects created with default constructors

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-20
SLIDE 20

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Test Data Generation

◮ Primitive types:

◮ Choose characteristic and borderline values

e.g. Integer.MIN_VALUE, -1, 0, 1, Integer.MAX_VALUE

◮ Provide manual input: -872463, 123316 ◮ Still may not be enough – KeY can do better

◮ Reference types:

◮ Objects created with default constructors ◮ Manual object factories. . .

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-21
SLIDE 21

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Test Data Generation

◮ Primitive types:

◮ Choose characteristic and borderline values

e.g. Integer.MIN_VALUE, -1, 0, 1, Integer.MAX_VALUE

◮ Provide manual input: -872463, 123316 ◮ Still may not be enough – KeY can do better

◮ Reference types:

◮ Objects created with default constructors ◮ Manual object factories. . .

We are getting close to regular, laborous test case writing

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-22
SLIDE 22

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Test Data Generation

◮ Primitive types:

◮ Choose characteristic and borderline values

e.g. Integer.MIN_VALUE, -1, 0, 1, Integer.MAX_VALUE

◮ Provide manual input: -872463, 123316 ◮ Still may not be enough – KeY can do better

◮ Reference types:

◮ Objects created with default constructors ◮ Manual object factories. . .

We are getting close to regular, laborous test case writing

◮ Use reflection to create objects with arbitrary constructors

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-23
SLIDE 23

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Test Data Generation

◮ Primitive types:

◮ Choose characteristic and borderline values

e.g. Integer.MIN_VALUE, -1, 0, 1, Integer.MAX_VALUE

◮ Provide manual input: -872463, 123316 ◮ Still may not be enough – KeY can do better

◮ Reference types:

◮ Objects created with default constructors ◮ Manual object factories. . .

We are getting close to regular, laborous test case writing

◮ Use reflection to create objects with arbitrary constructors ◮ Create “empty” objects (Objenesis library)

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-24
SLIDE 24

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Test Data Generation

◮ Primitive types:

◮ Choose characteristic and borderline values

e.g. Integer.MIN_VALUE, -1, 0, 1, Integer.MAX_VALUE

◮ Provide manual input: -872463, 123316 ◮ Still may not be enough – KeY can do better

◮ Reference types:

◮ Objects created with default constructors ◮ Manual object factories. . .

We are getting close to regular, laborous test case writing

◮ Use reflection to create objects with arbitrary constructors ◮ Create “empty” objects (Objenesis library)

◮ Regardless of the method, created objects are only

useful, when they satisfy some (JML) condtion

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-25
SLIDE 25

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Test Data Generation

◮ Primitive types:

◮ Choose characteristic and borderline values

e.g. Integer.MIN_VALUE, -1, 0, 1, Integer.MAX_VALUE

◮ Provide manual input: -872463, 123316 ◮ Still may not be enough – KeY can do better

◮ Reference types:

◮ Objects created with default constructors ◮ Manual object factories. . .

We are getting close to regular, laborous test case writing

◮ Use reflection to create objects with arbitrary constructors ◮ Create “empty” objects (Objenesis library)

◮ Regardless of the method, created objects are only

useful, when they satisfy some (JML) condtion

◮ Collect objects after successful tests of constructors

JMLUnitNG

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-26
SLIDE 26

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

JMLUnit New Generation

◮ Comprehensive JML based testing framework by Daniel

Zimmerman and Joe Kiniry

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-27
SLIDE 27

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

JMLUnit New Generation

◮ Comprehensive JML based testing framework by Daniel

Zimmerman and Joe Kiniry

◮ Core test generator:

◮ Collect classes and methods with JML specifications

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-28
SLIDE 28

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

JMLUnit New Generation

◮ Comprehensive JML based testing framework by Daniel

Zimmerman and Joe Kiniry

◮ Core test generator:

◮ Collect classes and methods with JML specifications ◮ Data generators with templates for manual input

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-29
SLIDE 29

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

JMLUnit New Generation

◮ Comprehensive JML based testing framework by Daniel

Zimmerman and Joe Kiniry

◮ Core test generator:

◮ Collect classes and methods with JML specifications ◮ Data generators with templates for manual input ◮ Create testing structure for everything

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-30
SLIDE 30

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

JMLUnit New Generation

◮ Comprehensive JML based testing framework by Daniel

Zimmerman and Joe Kiniry

◮ Core test generator:

◮ Collect classes and methods with JML specifications ◮ Data generators with templates for manual input ◮ Create testing structure for everything

◮ Runtime Assertion Checker (RAC) compiler:

◮ Embed JML checks into the compiled Java code ◮ Report results of evaluating JML expressions to the testing

framework

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-31
SLIDE 31

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

JMLUnit New Generation

◮ Comprehensive JML based testing framework by Daniel

Zimmerman and Joe Kiniry

◮ Core test generator:

◮ Collect classes and methods with JML specifications ◮ Data generators with templates for manual input ◮ Create testing structure for everything

◮ Runtime Assertion Checker (RAC) compiler:

◮ Embed JML checks into the compiled Java code ◮ Report results of evaluating JML expressions to the testing

framework

◮ Result: a standalone test suite based on the TestNG

engine – http://testng.org

◮ Very efficient (compared to the older, non-NG version)

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-32
SLIDE 32

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Some Technicalities – Process

  • 1. Annotate your code with JML specs

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-33
SLIDE 33

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Some Technicalities – Process

  • 1. Annotate your code with JML specs
  • 2. Run JMLUnitNG to generate tests

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-34
SLIDE 34

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Some Technicalities – Process

  • 1. Annotate your code with JML specs
  • 2. Run JMLUnitNG to generate tests
  • 3. (Fill in test data generators)

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-35
SLIDE 35

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Some Technicalities – Process

  • 1. Annotate your code with JML specs
  • 2. Run JMLUnitNG to generate tests
  • 3. (Fill in test data generators)
  • 4. Compile the SUT with a JML-RAC enabled Java compiler

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-36
SLIDE 36

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Some Technicalities – Process

  • 1. Annotate your code with JML specs
  • 2. Run JMLUnitNG to generate tests
  • 3. (Fill in test data generators)
  • 4. Compile the SUT with a JML-RAC enabled Java compiler
  • 5. Compile the test suite with regular Java compiler

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-37
SLIDE 37

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Some Technicalities – Process

  • 1. Annotate your code with JML specs
  • 2. Run JMLUnitNG to generate tests
  • 3. (Fill in test data generators)
  • 4. Compile the SUT with a JML-RAC enabled Java compiler
  • 5. Compile the test suite with regular Java compiler
  • 6. Run the test suite

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-38
SLIDE 38

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Some Technicalities – Libraries

◮ jmlunitng.jar (steps 2, 5 & 6): test generator & runner

and libraries it needs (OpenJML, antlr, TestNG, etc.)

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-39
SLIDE 39

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Some Technicalities – Libraries

◮ jmlunitng.jar (steps 2, 5 & 6): test generator & runner

and libraries it needs (OpenJML, antlr, TestNG, etc.)

◮ jml4c.jar (step 4), jml4rt.jar (step 6): one of the two

possible JML RAC compilers

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-40
SLIDE 40

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Some Technicalities – Libraries

◮ jmlunitng.jar (steps 2, 5 & 6): test generator & runner

and libraries it needs (OpenJML, antlr, TestNG, etc.)

◮ jml4c.jar (step 4), jml4rt.jar (step 6): one of the two

possible JML RAC compilers (The other has serious limitations)

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-41
SLIDE 41

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Some Technicalities – Libraries

◮ jmlunitng.jar (steps 2, 5 & 6): test generator & runner

and libraries it needs (OpenJML, antlr, TestNG, etc.)

◮ jml4c.jar (step 4), jml4rt.jar (step 6): one of the two

possible JML RAC compilers (The other has serious limitations)

◮ openjmlboot.jar (step 2): might be needed in boot class

path depending on the JVM used for test generation

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-42
SLIDE 42

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Some Technicalities – Libraries

◮ jmlunitng.jar (steps 2, 5 & 6): test generator & runner

and libraries it needs (OpenJML, antlr, TestNG, etc.)

◮ jml4c.jar (step 4), jml4rt.jar (step 6): one of the two

possible JML RAC compilers (The other has serious limitations)

◮ openjmlboot.jar (step 2): might be needed in boot class

path depending on the JVM used for test generation

◮ For Macs authors recommend OpenJDK ver. 7

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-43
SLIDE 43

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Some Technicalities – Libraries

◮ jmlunitng.jar (steps 2, 5 & 6): test generator & runner

and libraries it needs (OpenJML, antlr, TestNG, etc.)

◮ jml4c.jar (step 4), jml4rt.jar (step 6): one of the two

possible JML RAC compilers (The other has serious limitations)

◮ openjmlboot.jar (step 2): might be needed in boot class

path depending on the JVM used for test generation

◮ For Macs authors recommend OpenJDK ver. 7 ◮ I used OpenJDK ver. 6 on Ubuntu

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-44
SLIDE 44

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Example Again – Demo

/*@ public normal_behavior requires param >= 0; ensures \result >= 10; also public normal_behavior requires param < 0; ensures \result < -10; @*/ public int makeHole(int param) { if(param >= 0) { return param + 10; }else{ return param - 10; } }

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-45
SLIDE 45

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Specifications for Testing

Observation I

Input data that does not satisfy the precondition is considered meaningless.

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-46
SLIDE 46

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Specifications for Testing

Observation I

Input data that does not satisfy the precondition is considered

  • meaningless. The test is skipped!

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-47
SLIDE 47

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Specifications for Testing

Observation I

Input data that does not satisfy the precondition is considered

  • meaningless. The test is skipped!

Consequence

Try to have the overall precondition wide, ideally the disjunction (n number of spec cases): pre1 || pre2 || . . . || pren should be equivalent to true.

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-48
SLIDE 48

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Specifications for Testing

Observation II

The more non-overlapping specification cases the better – test data partitioning.

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-49
SLIDE 49

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Specifications for Testing

Observation II

The more non-overlapping specification cases the better – test data partitioning.

Consequence

Make sure that there is input test data for every specification case!

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-50
SLIDE 50

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Specifications for Testing

Observation II

The more non-overlapping specification cases the better – test data partitioning.

Consequence

Make sure that there is input test data for every specification case!

Observation IIa

The tested code may contain more information for data partitioning (e.g. if branches), KeY can help here, see Part II

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-51
SLIDE 51

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Specifications for Testing

Observation III

Postconditions contain what is actually checked, the oracle.

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-52
SLIDE 52

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Specifications for Testing

Observation III

Postconditions contain what is actually checked, the oracle.

Consequence

Given the information in the precondition and values that can be accessed, check what you can.

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-53
SLIDE 53

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Specifications for Testing

Observation III

Postconditions contain what is actually checked, the oracle.

Consequence

Given the information in the precondition and values that can be accessed, check what you can. Simple true is not a good postcondition, trivial test case. . .

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-54
SLIDE 54

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Specifications for Testing

Observation III

Postconditions contain what is actually checked, the oracle.

Consequence

Given the information in the precondition and values that can be accessed, check what you can. Simple true is not a good postcondition, trivial test case. . . Well, it still specifies absence of exceptions.

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-55
SLIDE 55

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Specifications for Testing

Observation III

Postconditions contain what is actually checked, the oracle.

Consequence

Given the information in the precondition and values that can be accessed, check what you can. Simple true is not a good postcondition, trivial test case. . . Well, it still specifies absence of exceptions.

Observation IIIa

Not all JML expressions are RAC checkable, e.g. some quantifications. Such specifications are quitely accepted as true!

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-56
SLIDE 56

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Specifications for Testing

Observation IV

Some JML tools do not deal well with multiple spec cases.

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-57
SLIDE 57

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Specifications for Testing

Observation IV

Some JML tools do not deal well with multiple spec cases.

Solution

It is always possible to combine multiple spec cases into one:

public normal_behavior requires pre1; ensures post1; also public behavior requires pre2; ensures post2; signals (Exception e) expost2;

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-58
SLIDE 58

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Specifications for Testing

Observation IV

Some JML tools do not deal well with multiple spec cases.

Solution

It is always possible to combine multiple spec cases into one:

public normal_behavior requires pre1; ensures post1; also public behavior requires pre2; ensures post2; signals (Exception e) expost2; public behavior requires pre1 || pre2; ensures \old(pre1) ==> post1; ensures \old(pre2) ==> post2; signals (Exception e) (\old(pre2) ==> expost2) && !\old(pre1);

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-59
SLIDE 59

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Specifications for Testing

Observation IV

Some JML tools do not deal well with multiple spec cases.

Solution

It is always possible to combine multiple spec cases into one:

public normal_behavior requires pre1; ensures post1; also public behavior requires pre2; ensures post2; signals (Exception e) expost2; public behavior requires pre1 || pre2; ensures \old(pre1) ==> post1; ensures \old(pre2) ==> post2; signals (Exception e) (\old(pre2) ==> expost2) && !\old(pre1);

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-60
SLIDE 60

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Examples to Play With

◮ Binary search algorithm:

◮ Specify two cases, element found and not found, hint:

requires array != null; requires Array is sorted; requires (\forall int i; i>=0 && i<array.length; array[i] != target); ensures \result == -1;

◮ Create and run the tests, find the bug in the code ◮ File BinSearch.java, then

$ make TESTCLASS=BinSearch testgen Fill in test data $ make TESTCLASS=BinSearch run

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-61
SLIDE 61

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Examples to Play With

◮ Binary search algorithm:

◮ Specify two cases, element found and not found, hint:

requires array != null; requires Array is sorted; requires (\forall int i; i>=0 && i<array.length; array[i] != target); ensures \result == -1;

◮ Create and run the tests, find the bug in the code ◮ File BinSearch.java, then

$ make TESTCLASS=BinSearch testgen Fill in test data $ make TESTCLASS=BinSearch run ◮ Integer division

◮ A specification that cannot be RAC checked ◮ File Divide.java

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-62
SLIDE 62

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Examples to Play With

◮ Binary search algorithm:

◮ Specify two cases, element found and not found, hint:

requires array != null; requires Array is sorted; requires (\forall int i; i>=0 && i<array.length; array[i] != target); ensures \result == -1;

◮ Create and run the tests, find the bug in the code ◮ File BinSearch.java, then

$ make TESTCLASS=BinSearch testgen Fill in test data $ make TESTCLASS=BinSearch run ◮ Integer division

◮ A specification that cannot be RAC checked ◮ File Divide.java

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-63
SLIDE 63

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

Reference Material – My Makefile

OPENJMLBOOT=lib/openjmlboot.jar JMLUNITNG=lib/jmlunitng.jar JML4C=lib/jml4c.jar JML4RT=lib/jml4rt.jar TESTCLASS?=Example all: testgen run clean: rm -rf tests test-output find src/ -name "*.class" -exec rm \{\} \; testgen: clean mkdir -p tests java -Xbootclasspath/p:$(OPENJMLBOOT) -jar $(JMLUNITNG) --dest tests --reflection src/ compile: java -jar $(JML4C) -cp $(JML4C):$(JMLUNITNG) src/ javac -cp $(JML4RT):$(JMLUNITNG):src/ ‘find tests/ -name "*.java"‘ run: compile java -cp $(JMLUNITNG):$(JML4RT):./tests/:./src/ org.charter.jmlunitng.$(TESTCLASS)_JML_Test report: compile java -cp $(JMLUNITNG):$(JML4RT):./tests/:./src/ com.beust.testng.TestNG test_$(TESTCLASS).xml Wojciech Mostowski & Gabriele Paganelli Test Generation with JML

slide-64
SLIDE 64

Overview Basics Test Data JMLUnitNG Demo Good Specifications Examples References

References

Yoonsik Cheon and Gary Leavens. A simple and practical approach to unit testing: The JML and JUnit way. ECOOP’02 Proceedings, Springer LNCS 2374. Daniel M. Zimmerman and Rinkesh Nagmoti. JMLUnit: The Next Generation, FoVeOOS 2010 Conference Proceedings, Springer LNCS 6528.

Software & Libraries

◮ OpenJML http://jmlspecs.sourceforge.net/openjml.tar.gz ◮ JML4C http://www.cs.utep.edu/cheon/download/jml4c/download.php ◮ JMLUnitNG http://formalmethods.insttech.washington.edu/software/ jmlunitng/

Wojciech Mostowski & Gabriele Paganelli Test Generation with JML