Construction Matt Staats KAIST Shin Hong, Moonzoo Kim, Gregory - - PowerPoint PPT Presentation

1

Supporting Test Oracle Construction

Matt Staats KAIST

Shin Hong, Moonzoo Kim, Gregg Rothermel KAIST / University of Nebraska-Lincoln Gregory Gay, Mats Heimdahl University of Minnesota Twin-Cities

2

Program

Talented Developers?

?

Test Input Program

Talented Developers?

Execute over

Test Input Program

Talented Developers?

Executed over

Program Behavior

?

Test Input Program

Talented Developers?

Executed over

Test Oracle

Check behavior

Program Behavior

?

Test Input Program

Talented Developers?

Executed over

Test Oracle

Check behavior

Program Behavior

Yes/No

The Testing Problem

Program

Talented Developers?

3

The Automated Testing Problem

Test Input Program

Talented Developers?

Executed over

Test Oracle Test Input Program

Talented Developers?

Generate from Executed over

Test Oracle

4

Test Input Generation Tools

Symbolic Execution Symbolic Java Finder KLEE Random Testing Randoop Adaptive Random Testing Naïve Random Testing Concolic Execution CUTE jCUTE DART SCORE SAGE Genetic Algorithm- Based Testing Testful AUSTIN Iguana

What now?

Program

Test Inputs

? ? ? ? ?

The Automated Testing Problem

• Huge numbers of tools for this!

Test Input Generation Tools

Symbolic Execution Symbolic Java Finder KLEE Random Testing Randoop Adaptive Random Testing Naïve Random Testing Concolic Execution CUTE jCUTE DART SCORE SAGE Genetic Algorithm- Based Testing Testful AUSTIN Iguana

What now?

Problems

More Work

? ? ?

5

The Automated Testing Problem

Test Input Program

Talented Developers?

Generate from Executed over

Test Oracle

Generate from?

Test Input Program

Talented Developers?

Generate from Executed over

Test Oracle

Generate from?

T h i s i s k i n d a w e i r d . Test Input Program

Talented Developers?

Generate from Executed over

Test Oracle

Fix Generate from? Must check result!

D i f f e r e n t r e p r e s e n t a t i

• n

, e a s i e r t

• f

i n d m i s t a k e s ? O r h a r d e r ?

Test Input Program

Talented Developers?

Generate from Executed over

Test Oracle Goal: get developer to state what the program should do (but correctly this time)

Sameish

6

Two Big Research Questions

• What types of test oracles can developers

easily/correctly understand and build?

– What tasks are people actually good at? – What should be trying to deliver?

• How can we help construct such test oracles?

– Techniques, algorithms, tools, etc. – Empirical studies (with users, necessarily) – Industrial case studies

7

Automatic Invariant Generation

• Idea: automatically generate

invariants from the system

• User then (necessarily)

evaluates result

• Remaining invariants represent

test oracle

• Several approaches, varying

result

• Daikon, AutoInfer, Xie/Notkin work

Automated Property / Behavior Extraction Program Program Behavior

User

Accept as correct Reject as incorrect

8

Automatic Invariant Generation

• Unclear how effective users

are at classifying results

– Problems if poor – Little evidence in favor of use

• Study: Daikon dynamic

invariant generator

– 2 case studies, approx. 30 students total – 3 programs – Each student classified an invariant as true, false, or unknown (unclassified)

Randoop (Test Input Generation) + Daikon Program Program + Program Invariants

User

Correct Invariants Incorrect Invariants

9

Automatic Invariant Generation

KAIST User Study Classification Results

Our classification User classification

10

Automatic Invariant Generation

• Questions:

– Why does this occur? – Impact of this on actual testing process?

• Answers:

– Why? Not really sure – Impact? No idea at all (but we guess negative)

To be presented at ISSTA 2012 Coauthors: Shin Hong, Moonzoo Kim, Gregg Rothermel

11

Test Oracle Generation Support

• Uncomfortable with complete automation for oracles

– Evidence is suspect – Requires change in user behavior

• As an alternative to complete construction, we thought we could

support users in making oracles

• Select oracle data: part of system oracle defined over
• User still has to define oracle

Test Input Generation Tools

Symbolic Execution Symbolic Java Finder KLEE Random Testing Randoop Adaptive Random Testing Naïve Random Testing Concolic Execution CUTE jCUTE DART SCORE SAGE Genetic Algorithm- Based Testing Testful AUSTIN Iguana

Must evaluate test results

Program

Test Inputs Oracle

Oracle Data Selection

Test Inputs + Oracle Data

12

Test Oracle Generation Support

• Mutation testing was used to determine where and when we

can detect changes

• Result is that for a set of test inputs, person has a list of useful

variables

– Note: domain is critical avionics, so problems of heap, etc. go away

• Goal: do better than other methods of selecting oracle data

Program Mutants Test Inputs (Generated Externally) Variable Effectiveness Ranking Oracle Data

Generate Measure Variable Effectiveness Run Against

Expected Value Test Oracle

Tester Specifies expected value

13

Test Oracle Generation Support

Latctl Person DWM_2 DWM_1

To be presented at ICSE 2012 Coauthors: Gregory Gay, Mats Heimdahl

14

Questions

15

Future Work

Automatically extract test oracle

Expected Value: “Zuel”

Program: Gatekeeper Input: “Keymaster”