from Structured Natural Language Specifications - - PowerPoint PPT Presentation

▶
from structured natural language specifications
SMART_READER_LITE
LIVE PREVIEW

from Structured Natural Language Specifications - - PowerPoint PPT Presentation

Oct 19, 2022 6 likes •651 views

Automatically Generating Precise Oracles from Structured Natural Language Specifications http://swami.cs.umass.edu Manish Motwani Yuriy Brun The Test Oracle Problem Software Under Input Actual Output Test The Test Oracle Problem Software

slide-1
SLIDE 1

Automatically Generating Precise Oracles from Structured Natural Language Specifications

Manish Motwani Yuriy Brun

http://swami.cs.umass.edu

slide-2
SLIDE 2

The Test Oracle Problem

Input Software Under Test Actual Output

slide-3
SLIDE 3

The Test Oracle Problem

Input Software Under Test Actual Output Test Oracle (Expected Output) Correct Incorrect

slide-4
SLIDE 4

The Test Oracle Problem

Input Software Under Test Actual Output Test Oracle (Expected Output) Correct Incorrect

Easy to generate

slide-5
SLIDE 5

The Test Oracle Problem

Input Software Under Test Actual Output Test Oracle (Expected Output) Correct Incorrect

Easy to generate Hard to generate

slide-6
SLIDE 6

Our Solution - Swami

Structured Informal Specification

slide-7
SLIDE 7

Our Solution - Swami

Swami

/*TEST TEMPLATE WITH ORACLE*/ function test_array_len( len ){ if ( ToUint32(len)!=len) { try{ var output = new Array ( len ); return; }catch(e){ assert.strictEqual(true, (e instanceof RangeError)); return; } } } /*TEST INPUTS*/ test_array_len(1.1825863363010669e+308); test_array_len(null); test_array_len(-747); test_array_len(368); …

Structured Informal Specification Executable Test

http://swami.cs.umass.edu

Test oracle Test inputs

slide-8
SLIDE 8

Why JavaScript specifications?

slide-9
SLIDE 9

Why JavaScript specifications?

Does not get deprecated

slide-10
SLIDE 10

Why JavaScript specifications?

Does not get deprecated Less ambiguous

slide-11
SLIDE 11

Why JavaScript specifications?

Does not get deprecated Less ambiguous Multiple real-world projects adhere to the spec

slide-12
SLIDE 12

Swami-generated tests are precise to the specification

50,086

Number of Tests (total 83,000)

Innocuous tests (60.4%)

slide-13
SLIDE 13

Swami-generated tests are precise to the specification

50,086

Number of Tests (total 83,000)

Innocuous tests Good tests (60.4%) 32,379 (39.0%)

slide-14
SLIDE 14

Swami-generated tests are precise to the specification

50,086 32,379 535

Number of Tests (total 83,000)

(0.6%) Innocuous tests Good tests (60.4%) 32,379 (39.0%) Bad tests

slide-15
SLIDE 15

Swami-generated tests are precise to the specification

50,086 32,379 535

Number of Tests (total 83,000)

(0.6%) Innocuous tests Good tests (60.4%) 32,379 (39.0%) Bad tests

Of the non-innocuous tests, 98.4% are Good and only 1.6% are Bad

slide-16
SLIDE 16

Swami covers more code and identifies features and bugs missed by developer-written tests

Missing Features / Bugs

  • 15 missing features in Rhino
  • 1 unknown bug in Rhino and Node.js
  • 18 semantic disambiguities in

JavaScript specification

slide-17
SLIDE 17

Swami covers more code and identifies features and bugs missed by developer-written tests

Missing Features / Bugs

  • 15 missing features in Rhino
  • 1 unknown bug in Rhino and Node.js
  • 18 semantic disambiguities in

JavaScript specification

line coverage branch coverage

Code Coverage Ratio

Developer Developer+Swami 15.2% 19.3%

slide-18
SLIDE 18

Swami generates fewer false alarms and covers code missed by EvoSuite

line coverage

Code Coverage Ratio

EvoSuite EvoSuite+Swami 19.5% bad tests

Number of False Alarms

EvoSuite Swami 73.9%

slide-19
SLIDE 19

Swami identifies the specifications that encode testable behavior precisely

0.00% 20.00% 40.00% 60.00% 80.00% 100.00%

performance using rule-based approach

precision recall 0.00% 20.00% 40.00% 60.00% 80.00% 100.00%

performance using IR-based approach

precision recall

slide-20
SLIDE 20

Why is it hard to derive oracles from informal specifications?

slide-21
SLIDE 21

Why is it hard to derive oracles from informal specifications?

Encode testable behavior

slide-22
SLIDE 22

Encode testable behavior Abstract Operations

Why is it hard to derive oracles from informal specifications?

slide-23
SLIDE 23

Implicit Operations Encode testable behavior Abstract Operations

Why is it hard to derive oracles from informal specifications?

slide-24
SLIDE 24

Implicit Operations Oracles embedded in Conditionals Encode testable behavior Abstract Operations

Why is it hard to derive oracles from informal specifications?

slide-25
SLIDE 25

Implicit Operations Encode testable behavior Abstract Operations Oracles embedded in Conditionals Assignments using local variables

Why is it hard to derive oracles from informal specifications?

slide-26
SLIDE 26

Implicit Operations Ambiguous and Deprecated Encode testable behavior Abstract Operations Oracles embedded in Conditionals Assignments using local variables

Why is it hard to derive oracles from informal specifications?

slide-27
SLIDE 27

Related work: What can the state-of-the-art tools do?

  • EvoSuite1, Randoop2
  • Cannot derive oracles from natural language specifications
  • Generated tests cannot identify missing features
  • Jdoctor3 , Toradocu4, @tComment5
  • Closely tied to JavaDoc (use tags, e.g., @params, @throws)

and Randoop, hence may not generalize

  • 1. Fraser et al. TSE 2013,
  • 2. Pacheco et al. ICSE 2007, 3. Blasi et al. ISSTA 2018, 4. Goffi et al. ISSTA 2016 , 5. Tan et al. ICST 2012

Implicit Operations Ambiguous and Deprecated Encode testable behavior Abstract Operations Oracles embedded in Conditionals Assignments using local variables

slide-28
SLIDE 28

Related work: What can the state-of-the-art tools do?

  • EvoSuite1, Randoop2
  • Cannot derive oracles from natural language specifications
  • Generated tests cannot identify missing features
  • Jdoctor3 , Toradocu4, @tComment5
  • Closely tied to JavaDoc (use tags, e.g., @params, @throws)

and Randoop, hence may not generalize State-of-the-art tools are not capable of deriving test oracles from informal specifications that exists independent of the source code.

  • 1. Fraser et al. TSE 2013,
  • 2. Pacheco et al. ICSE 2007, 3. Blasi et al. ISSTA 2018, 4. Goffi et al. ISSTA 2016 , 5. Tan et al. ICST 2012

Implicit Operations Ambiguous and Deprecated Encode testable behavior Abstract Operations Oracles embedded in Conditionals Assignments using local variables

slide-29
SLIDE 29

What kind of oracles exist in informal specifications?

Vague oracles for common inputs Concrete oracles for uncommon inputs

slide-30
SLIDE 30

What kind of oracles exist in informal specifications?

Vague oracles for common inputs Concrete oracles for uncommon inputs

Informal specifications typically contain oracles for Exceptions and Boundary conditions.

slide-31
SLIDE 31

Is it useful to generate tests only for Exceptions and Boundary conditions?

Source: Goffi, Alberto, et al. “Automatic generation of oracles for exceptional behaviors.” ISSTA, 2016.

  • 10 popular, well-tested open source libraries
  • The coverage of throw statements is usually significantly lower than
  • verall coverage, in two cases below 50%
slide-32
SLIDE 32

Is it useful to generate tests only for Exceptions and Boundary conditions?

Source: Goffi, Alberto, et al. “Automatic generation of oracles for exceptional behaviors.” ISSTA, 2016.

  • 10 popular, well-tested open source libraries
  • The coverage of throw statements is usually significantly lower than
  • verall coverage, in two cases below 50%

Exceptions are under-tested by the developers

slide-33
SLIDE 33

Goal of this work

Automatically generate executable tests (inputs with oracles) for Exceptions and Boundary conditions from structured informal specifications

Implicit Operations Ambiguous and deprecated encode testable behavior Abstract Operations Oracles embedded in Conditionals Assignments using local variables

Structured Informal specification Executable Test

Test inputs Test oracles

slide-34
SLIDE 34

Swami

Automatically generate executable tests (inputs with oracles) for Exceptions and Boundary conditions from structured informal specifications

Structured Informal specification Executable Test

Test inputs Test oracles Implicit Operations Ambiguous and deprecated encode testable behavior Abstract Operations Oracles embedded in Conditionals Assignments using local variables

slide-35
SLIDE 35

Step1: Identify specifications which encode testable behavior

Rule-based approach

Specification Document Relevant Specifications

Rules are regular expressions composed of POS tags, keywords, and wild card characters

slide-36
SLIDE 36

Step1: Identify specifications which encode testable behavior

Rule-based approach

Specification Document Relevant Specifications

Heading RE: [CD new* NN LRB NN.* RRB] Body RE: [If .* return .*] [if .* throw .* exception]

slide-37
SLIDE 37

Step1: Identify specifications which encode testable behavior

Rule-based approach Information Retrieval-based approach

Source code

when the format of specification document is unknown

Specification Document Relevant Specifications

OKAPI model

slide-38
SLIDE 38

Example specification encoding testable behavior

Relevant Specifications

Header RE: CD new* NN LRB NN.* RRB

slide-39
SLIDE 39

Example specification encoding testable behavior

Relevant Specifications

Header RE: CD new* NN LRB NN.* RRB Body RE: If .* return .* Body RE: If .* throw .* exception

slide-40
SLIDE 40

Step2: Extract method signature from specification heading and initialize Test Template

slide-41
SLIDE 41

Step2: Extract method signature from specification heading and initialize Test Template

function test_string_prototype_startswith(thisObj,searchString,position) {} function test_< method name >(thisObj,<[ method args ]>) {}

Initialized Test Template

slide-42
SLIDE 42

Step2: Extract method signature from specification heading and initialize Test Template

function test_string_prototype_startswith(thisObj,searchString,position) {} function test_< method name >(thisObj,<[ method args ]>) {}

new String(thisObj).startsWith(searchString, position); Method invocation code Initialized Test Template

slide-43
SLIDE 43

Step3: Identify and parse Assignments to store the local variables and their values

slide-44
SLIDE 44

Step3: Identify and parse Assignments to store the local variables and their values

Variable Value O RequireObjectCoercible(this value) S ToString(O) isRegExp IsRegExp(searchString) searchStr ToString(searchString) pos ToInteger(position) len length of S start min(max(pos,0),len) searchLength length of searchStr

slide-45
SLIDE 45

Step4: Identify and parse Conditionals to populate the conditional templates

slide-46
SLIDE 46

Step4: Identify and parse Conditionals to populate the conditional templates

if (<condition>) { var output = <method invocation>; <test constructor>(output, <expected output>); return; }

Boundary Condition Exception

if (<condition>) { try { var output = <method invocation>; return; }catch(e){ <test constructor>(true, (e instance of <expected error>)); return; } }

Exception

slide-47
SLIDE 47

Step4: Identify and parse Conditionals to populate the conditional templates

Boundary condition oracle Exception oracle

if (<condition>) { var output = <method invocation>; <test constructor>(output, <expected output>); return; } if (<condition>) { try { var output = <method invocation>; return; }catch(e){ <test constructor>(true, (e instance of <expected error>)); return; } }

Boundary Condition Exception

slide-48
SLIDE 48

Step4: Identify and parse Conditionals to populate the conditional templates

if (isRegExp is true){ try{ var output = new String(thisObj).startsWith(searchString, position); return; }catch(e){ assert.StrictEqual(true,(e instanceof TypeError)); return; } } if (<condition>) { try { var output = <method invocation>; return; }catch(e){ <test constructor>(true, (e instance of <expected error>)); return; } }

Exception

Exception oracle

slide-49
SLIDE 49

Step4: Identify and parse Conditionals to populate the conditional templates

if (isRegExp is true){ try{ var output = new String(thisObj).startsWith(searchString, position); return; }catch(e){ assert.StrictEqual(true,(e instanceof TypeError)); return; } } if (<condition>) { try { var output = <method invocation>; return; }catch(e){ <test constructor>(true, (e instance of <expected error>)); return; } }

Exception

Exception oracle

slide-50
SLIDE 50

Step4: Identify and parse Conditionals to populate the conditional templates

Exception oracle

if (isRegExp is true){ try{ var output = new String(thisObj).startsWith(searchString, position); return; }catch(e){ assert.StrictEqual(true,(e instanceof TypeError)); return; } } if (<condition>) { try { var output = <method invocation>; return; }catch(e){ <test constructor>(true, (e instance of <expected error>)); return; } }

Exception

From step2

slide-51
SLIDE 51

Step4: Identify and parse Conditionals to populate the conditional templates

Exception oracle

if (isRegExp is true){ try{ var output = new String(thisObj).startsWith(searchString, position); return; }catch(e){ assert.StrictEqual(true,(e instanceof TypeError)); return; } } if (<condition>) { try { var output = <method invocation>; return; }catch(e){ <test constructor>(true, (e instance of <expected error>)); return; } }

Exception

Input by developer From step2

slide-52
SLIDE 52

Step4: Identify and parse Conditionals to populate the conditional templates

Boundary condition oracle

if (searchLength+start is greater than len){ var output = new String(thisObj).startsWith(searchString, position); assert.strictEqual(output, false); return; } if (<condition>) { var output = <method invocation>; <test constructor>(output, <expected output>); return; }

Boundary Condition

slide-53
SLIDE 53

Step5: Recursively substitute local variables and implicit operations

if (searchLength+start is greater than len){ var output = new String(thisObj).startsWith(searchString, position); assert.strictEqual(output, false); return; } if (isRegExp is true){ try{ var output = new String(thisObj).startsWith(searchString, position); return; }catch(e){ assert.StrictEqual(true,(e instanceof TypeError)); return; } } Variable Value O RequireObjectCoercible(this value) S ToString(O) isRegExp IsRegExp(searchString) searchStr ToString(searchString) pos ToInteger(position) len length of S start min(max(pos,0),len) searchLength length of searchStr

Method Arguments: thisObj searchString position

slide-54
SLIDE 54

if (ToString(searchString).length + Math.min(Math.max(ToInteger(position), 0), ToString(RequireObjectCoercible(thisObj)).length) > ToString(RequireObjectCoercible(thisObj)).length){ var output = new String(thisObj).startsWith(searchString, position); assert.strictEqual(output, false); return; } if (IsRegExp(searchString) === true){ try{ var output = new String(thisObj).startsWith(searchString, position); return; }catch(e){ assert.StrictEqual(true,(e instanceof TypeError)); return; } }

Step5: Recursively substitute local variables and implicit operations

Variable Value O RequireObjectCoercible(this value) S ToString(O) isRegExp IsRegExp(searchString) searchStr ToString(searchString) pos ToInteger(position) len length of S start min(max(pos,0),len) searchLength length of searchStr

Method Arguments: thisObj searchString position

slide-55
SLIDE 55

Step6: Add conditionals to the initialized test template and check if it compiles

function test_string_prototype_startswith(thisObj,searchString,position) { } if (IsRegExp(searchString) === true){ try{ var output = new String(thisObj).startsWith(searchString, position); return; }catch(e){ assert.StrictEqual(true,(e instanceof TypeError)); return; } }

slide-56
SLIDE 56

function test_string_prototype_startswith(thisObj,searchString,position) { } if (IsRegExp(searchString) === true){ try{ var output = new String(thisObj).startsWith(searchString, position); return; }catch(e){ assert.StrictEqual(true,(e instanceof TypeError)); return; } }

Implement Abstract Operations (100 lines JS code)

function IsRegExp(argument){ return (argument instanceof RegExp); } …

slide-57
SLIDE 57

function test_string_prototype_startswith(thisObj,searchString,position) { } if (ToString(searchString).length + Math.min(Math.max(ToInteger(position), 0), ToString(RequireObjectCoercible(thisObj)).length) > ToString(RequireObjectCoercible(thisObj)).length){ var output = new String(thisObj).startsWith(searchString, position); assert.strictEqual(output, false); return; } if (IsRegExp(searchString) === true){ try{ var output = new String(thisObj).startsWith(searchString, position); return; }catch(e){ assert.StrictEqual(true,(e instanceof TypeError)); return; } } function IsRegExp(argument){ return (argument instanceof RegExp); } …

Abstract Operations Test Template encoding Oracles

Implement Abstract Operations (100 lines JS code)

slide-58
SLIDE 58

Step7: Instantiating test template by generating test inputs using random input generation

  • Total number of inputs: 3
  • Heuristic: String method => thisObj should be a

valid string

  • Number of test inputs to be generated: 1000

test_string_prototype_startswith("Y3I9", "E0RS6GU078", 894); test_string_prototype_startswith("T82LL6", 572, false); test_string_prototype_startswith("XU6W0", "J3A", Infinity); test_string_prototype_startswith("W5E74X0R", null, NaN); ...

function test_string_prototype_startswith(thisObj,searchString,position) { } if (ToString(searchString).length + Math.min(Math.max(ToInteger(position), 0), ToString(RequireObjectCoercible(thisObj)).length) > ToString(RequireObjectCoercible(thisObj)).length){ var output = new String(thisObj).startsWith(searchString, position); assert.strictEqual(output, false); return; } if (IsRegExp(searchString) === true){ try{ var output = new String(thisObj).startsWith(searchString, position); return; }catch(e){ assert.StrictEqual(true,(e instanceof TypeError)); return; } } function IsRegExp(argument){ return (argument instanceof RegExp); } …

Abstract Operations Test Template encoding Oracles

slide-59
SLIDE 59
slide-60
SLIDE 60

Swami

slide-61
SLIDE 61

function test_string_prototype_startswith(thisObj,searchString,position) { } if (ToString(searchString).length + Math.min(Math.max(ToInteger(position), 0), ToString(RequireObjectCoercible(thisObj)).length) > ToString(RequireObjectCoercible(thisObj)).length){ var output = new String(thisObj).startsWith(searchString, position); assert.strictEqual(output, false); return; } if (IsRegExp(searchString) === true){ try{ var output = new String(thisObj).startsWith(searchString, position); return; }catch(e){ assert.StrictEqual(true,(e instanceof TypeError)); return; } } function IsRegExp(argument){ return (argument instanceof RegExp); }

Abstract Operations

test_string_prototype_startswith("Y3I9", "E0RS6GU078", 894); test_string_prototype_startswith("T82LL6", 572, false); test_string_prototype_startswith("XU6W0", "J3A", Infinity); test_string_prototype_startswith("W5E74X0R", null, NaN); ...

Test Template encoding Oracles Test Inputs

Executable Test with Oracles

Swami

slide-62
SLIDE 62

Contributions

http://swami.cs.umass.edu

slide-63
SLIDE 63

Contributions

http://swami.cs.umass.edu

slide-64
SLIDE 64

Contributions

http://swami.cs.umass.edu

http://people.cs.umass.edu/~mmotwani