Automatic Generation of Program Specifications Jeremy Nimmer MIT - - PowerPoint PPT Presentation

▶
automatic generation of program specifications
SMART_READER_LITE
LIVE PREVIEW

Automatic Generation of Program Specifications Jeremy Nimmer MIT - - PowerPoint PPT Presentation

Apr 05, 2023 188 likes •490 views

Automatic Generation of Program Specifications Jeremy Nimmer MIT Lab for Computer Science http://pag.lcs.mit.edu/ Joint work with Michael Ernst Jeremy Nimmer, page 1 Synopsis Specifications are useful for many tasks Use of specifications

slide-1
SLIDE 1

Jeremy Nimmer, page 1

Automatic Generation

  • f Program Specifications

Jeremy Nimmer

MIT Lab for Computer Science http://pag.lcs.mit.edu/ Joint work with Michael Ernst

slide-2
SLIDE 2

Jeremy Nimmer, page 2

Synopsis

Specifications are useful for many tasks

  • Use of specifications has practical difficulties

Dynamic analysis can capture specifications

  • Recover from existing code
  • Infer from traces
  • Results are accurate (90%+)
  • Specification matches implementation
slide-3
SLIDE 3

Jeremy Nimmer, page 3

Outline

  • Motivation
  • Approach: Generate and check specifications
  • Evaluation: Accuracy experiment
  • Conclusion
slide-4
SLIDE 4

Jeremy Nimmer, page 4

Advantages of specifications

  • Describe behavior precisely
  • Permit reasoning using summaries
  • Can be verified automatically
slide-5
SLIDE 5

Jeremy Nimmer, page 5

Problems with specifications

  • Describe behavior precisely
  • Tedious and difficult to write and maintain
  • Permit reasoning using summaries
  • Must be accurate if used in lieu of code
  • Can be verified automatically
  • Verification may require uninteresting annotations
slide-6
SLIDE 6

Jeremy Nimmer, page 6

Solution

Automatically generate and check specifications from the code

Specification Code

Checker Generator

myStack.push(elt); Q.E.D. myStack.isEmpty() = false

Proof

slide-7
SLIDE 7

Jeremy Nimmer, page 7

Solution scope

  • Generate and check “complete” specifications
  • Very difficult
  • Generate and check partial specifications
  • Nullness, types, bounds, modification targets, ...
  • Need not operate in isolation
  • User might have some interaction
  • Goal: decrease overall effort
slide-8
SLIDE 8

Jeremy Nimmer, page 8

Outline

  • Motivation
  • Approach: Generate and check specifications
  • Evaluation: Accuracy experiment
  • Conclusion
slide-9
SLIDE 9

Jeremy Nimmer, page 9

Previous approaches

Generation:

  • By hand
  • Static analysis

Checking

  • By hand
  • Non-executable models

Specification Code

Checker Generator

myStack.push(elt); Q.E.D. myStack.isEmpty() = false

Proof

slide-10
SLIDE 10

Jeremy Nimmer, page 10

Our approach

  • Dynamic detection proposes likely properties
  • Static checking verifies properties
  • Combining the techniques overcomes the

weaknesses of each

  • Ease annotation
  • Guarantee soundness

Specification Code

Checker Generator

myStack.push(elt); Q.E.D. myStack.isEmpty() = false

Proof

slide-11
SLIDE 11

Jeremy Nimmer, page 11

Daikon: Dynamic invariant detection

Look for patterns in values the program computes:

  • Instrument the program to write data trace files
  • Run the program on a test suite
  • Invariant detector reads data traces, generates

potential invariants, and checks them

Invariants Instrumented program Original program Test suite

Run Instrument

Data trace database

Detect invariants

slide-12
SLIDE 12

Jeremy Nimmer, page 12

ESC/Java: Invariant checking

  • ESC/Java: Extended Static Checker for Java
  • Lightweight technology: intermediate between

type-checker and theorem-prover; unsound

  • Intended to detect array bounds and null

dereference errors, and annotation violations

/*@ requires x != null */ /*@ ensures this.a[this.top] == x */ void push(Object x);

  • Modular: checks, and relies on, specifications
slide-13
SLIDE 13

Jeremy Nimmer, page 13

Integration approach

Run Daikon over target program Insert results into program as annotations Run ESC/Java on the annotated program All steps are automatic.

Specification Code

ESC/Java Daikon

myStack.push(elt); Q.E.D. myStack.isEmpty() = false

Proof

slide-14
SLIDE 14

Jeremy Nimmer, page 14

/*@ invariant theArray != null; invariant \typeof(theArray) == \type(Object[]); invariant topOfStack >= -1; invariant topOfStack < theArray.length; invariant theArray[0..topOfStack] != null; invariant theArray[topOfStack+1..] == null; */

public class StackAr { Object[] theArray; int topOfStack;

...

Stack object invariants

A Y U O E I theArray topOfStack

invariant theArray != null; invariant \typeof(theArray) == \type(Object[]); invariant topOfStack >= -1; invariant topOfStack < theArray.length; invariant theArray[0..topOfStack] != null; invariant theArray[topOfStack+1..] == null;

slide-15
SLIDE 15

Jeremy Nimmer, page 15

Stack push method

/*@ requires x != null; requires topOfStack < theArray.length - 1; modifies topOfStack, theArray[*]; ensures topOfStack == \old(topOfStack) + 1; ensures x == theArray[topOfStack]; ensures theArray[0..\old(topOfStack)]; == \old(theArray[0..topOfStack]); */

A W Y U O E I theArray topOfStack

public void push( Object x ) { ... }

/*@ requires x != null; requires topOfStack < theArray.length - 1; modifies topOfStack, theArray[*]; ensures topOfStack == \old(topOfStack) + 1; ensures x == theArray[topOfStack]; ensures theArray[0..\old(topOfStack)]; == \old(theArray[0..topOfStack]); */

slide-16
SLIDE 16

Jeremy Nimmer, page 16

Stack summary

  • Reveal properties of the implementation

(e.g., garbage collection of popped elements)

  • No runtime errors if callers satisfy preconditions
  • Implementation meets generated specification
  • ESC/Java verified all 25 Daikon invariants
slide-17
SLIDE 17

Jeremy Nimmer, page 17

Outline

  • Motivation
  • Approach: Generate and check specifications
  • Evaluation: Accuracy experiment
  • Conclusion
slide-18
SLIDE 18

Jeremy Nimmer, page 18

Accuracy experiment

  • Dynamic generation is potentially unsound
  • How accurate are its results in practice?
  • Combining static and dynamic analyses

should produce benefits

  • But perhaps their domains are too dissimilar?
slide-19
SLIDE 19

Jeremy Nimmer, page 19

Programs studied

  • 11 programs from libraries, assignments, texts
  • Total 2449 NCNB LOC in 273 methods
  • Test suites
  • Used program’s test suite if provided (9 did)
  • If just example calls, spent <30 min. enhancing
  • ~70% statement coverage
slide-20
SLIDE 20

Jeremy Nimmer, page 20

invariant theArray != null; invariant topOfStack >= -1; invariant topOfStack < theArray.length; invariant theArray[0..length-1] == null; invariant theArray[0..topOfStack] != null; invariant theArray[topOfStack+1..] == null;

Accuracy measurement

  • Standard measures from info ret [Sal68, vR79]
  • Precision (correctness) : 3 / 4 = 75%
  • Recall (completeness) : 3 / 5 = 60%
  • Compare generated specification to a

verifiable specification

slide-21
SLIDE 21

Jeremy Nimmer, page 21

Experiment results

  • Daikon reported 554 invariants
  • Precision: 96% of reported invariants verified
  • Recall: 91% of necessary invariants were reported
slide-22
SLIDE 22

Jeremy Nimmer, page 22

Causes of inaccuracy

  • Limits on tool grammars
  • Daikon: May not propose relevant property
  • ESC: May not allow statement of relevant property
  • Incompleteness in ESC/Java
  • Always need programmer judgment
  • Insufficient test suite
  • Shows up as overly-strong specification
  • Verification failure highlights problem; helpful in fixing
  • System tests fared better than unit tests
slide-23
SLIDE 23

Jeremy Nimmer, page 23

Experiment conclusions

  • Our dynamic analysis is accurate
  • Recovered partial specification
  • Even with limited test suites
  • Enabled verifying lack of runtime exceptions
  • Specification matches the code
  • Results should scale
  • Larger programs dominate results
  • Approach is class- and method-centric
slide-24
SLIDE 24

Jeremy Nimmer, page 24

Value to programmers

Generated specifications are accurate

  • Are the specifications useful?
  • How much does accuracy matter?
  • How does Daikon compare with other

annotation assistants? Answers at FSE'02

slide-25
SLIDE 25

Jeremy Nimmer, page 25

Outline

  • Motivation
  • Approach: Generate and check specifications
  • Evaluation: Accuracy experiment
  • Conclusion
slide-26
SLIDE 26

Jeremy Nimmer, page 26

Conclusion

  • Specifications via dynamic analysis
  • Accurately produced from limited test suites
  • Automatically verifiable (minor edits)
  • Specification characterizes the code
  • Unsound techniques useful in program

development

slide-27
SLIDE 27

Jeremy Nimmer, page 27

Questions?

slide-28
SLIDE 28

Jeremy Nimmer, page 28

Formal specifications

  • Precise, mathematical desc. of behavior [LG01]
  • (Another type of spec: requirements documents)
  • Standard definition; novel use
  • Generated after implementation
  • Still useful to produce [PC86]
  • Many specifications for a program
  • Depends on task
  • e.g. runtime performance
slide-29
SLIDE 29

Jeremy Nimmer, page 29

Effect of bugs

  • Case 1: Bug is exercised by test suite
  • Falsifies one or more invariants
  • Weaker specification
  • May cause verification to fail
  • Case 2: Bug is not exercised by test suite
  • Not reflected in specification
  • Code and specification disagree
  • Verifier points out inconsistency