[PPT] - Automating Regression Verification Dennis Felsing , Sarah Grebing, PowerPoint Presentation

SLIDE 1

Automating Regression Verification

Dennis Felsing, Sarah Grebing, Vladimir Klebanov, Mattias Ulbrich, Philipp R¨ ummer 2014-07-23

1 / 16

SLIDE 2

Introduction

How to prevent regressions in software development?

2 / 16

SLIDE 3

Introduction

How to prevent regressions in software development?

Formal Verification

Formally prove correctness of software ⇒ Requires formal specification

Regression Testing

Discover new bugs by testing for them ⇒ Requires test cases

2 / 16

SLIDE 4

Introduction

How to prevent regressions in software development?

Formal Verification

Formally prove correctness of software ⇒ Requires formal specification

Regression Testing

Discover new bugs by testing for them ⇒ Requires test cases

Regression Verification

Formally prove there are no new bugs

2 / 16

SLIDE 5

Regression Verification

Formally prove there are no new bugs

Goal: Proving the equivalence of two closely related programs
No formal specification or test cases required
Instead use old program version as reference
Tools for proving function equivalence in a simple

programming language using SMT solvers

3 / 16

SLIDE 6

Overview

1 Overapproximation using Uninterpreted Functions 2 Approximation using Uninterpreted Predicates 3 Results and Future Work

4 / 16

SLIDE 7

Overview

1 Overapproximation using Uninterpreted Functions 2 Approximation using Uninterpreted Predicates 3 Results and Future Work

5 / 16

SLIDE 8

Function Equivalence

Existing approach by Strichman & Godlin

Function f (val n1; ret r1) Function f without recursion Static Single Assignment Sf Function g (val n2; ret r2) Function g without recursion Static Single Assignment Sg (n1 = n2 ∧ Sf ∧ Sg) → r1 = r2 Valid / Invalid Equivalent? Uninterpreted Function U for recursive calls in both f and g SMT Solver

6 / 16

SLIDE 9

Our Contribution: Extensions

Function f (val n1; ret r1) Sf Function g (val n2; ret r2) Sg (n1 = n2 ∧ Sf ∧ Sg) → r1 = r2 Valid / Invalid Single Static Assignment Form Equivalent? SMT Solver

7 / 16

SLIDE 10

Our Contribution: Extensions

Function f (val n1; ret r1) Sf Function g (val n2; ret r2) Sg (n1 = n2 ∧ Sf ∧ Sg) → r1 = r2 Valid / Invalid Equivalent! Single Static Assignment Form Equivalent? SMT Solver

7 / 16

SLIDE 11

Our Contribution: Extensions

Function f (val n1; ret r1) Sf Function g (val n2; ret r2) Sg (n1 = n2 ∧ Sf ∧ Sg) → r1 = r2 Valid / Invalid Equivalent! Counterexample: n = 0: r1 = −1 r2 = −3 Single Static Assignment Form Equivalent? SMT Solver

7 / 16

SLIDE 12

Our Contribution: Extensions

Function f (val n1; ret r1) Sf Function g (val n2; ret r2) Sg (n1 = n2 ∧ Sf ∧ Sg) → r1 = r2 Valid / Invalid Equivalent! Counterexample: n = 0: r1 = −1 r2 = −3 f (0) = g(0) = 0 Single Static Assignment Form Equivalent? SMT Solver Execute

7 / 16

SLIDE 13

Our Contribution: Extensions

Function f (val n1; ret r1) Sf Function g (val n2; ret r2) Sg (n1 = n2 ∧ Sf ∧ Sg∧ U(0) = 0 ) → r1 = r2 Valid / Invalid Equivalent! Counterexample: n = 0: r1 = −1 r2 = −3 f (0) = g(0) = 0 Single Static Assignment Form Equivalent? rerun SMT Solver Execute Add

7 / 16

SLIDE 14

Overapproximation using uninterpreted functions

Approach

Run the programs with input gathered from counterexamples
Detect whether CE is spurious or not
If spurious: Add additional constraints to the uninterpreted

function ⇒ Is a simple form of Counter Example Guided Abstraction Refinement (CEGAR)

Successful when

Finite number of constraints on the uninterpreted function

imply equivalence

These are often the “base cases” of recursive implementations

8 / 16

SLIDE 15

Overview

1 Overapproximation using Uninterpreted Functions 2 Approximation using Uninterpreted Predicates 3 Results and Future Work

9 / 16

SLIDE 16

Approximation using Uninterpreted Predicates

First approach (just shown)

Overapproximate recursion by uninterpreted Function U:

∀U.constraints(U) ∧ Sf ∧ Sg ∧ ... → r1 = r2

New approach

Infer a predicate C which couples recursive calls:

∃C.(C(...) ∧ ... → r1 = r2) ∧ “C couples f and g”

Use state-of-the-art SMT solvers (Eldarica, Z3) to

automatically find such a C or prove that is does not exist ⇒ Example will show loops with coupling loop invariants

10 / 16

SLIDE 17

Automatic Invariant Inference

int f1 ( int n) { int r = 0; i f (n == 0) return 1; while (n > 0) { n /= 10; r++; } return r ; }

11 / 16

SLIDE 18

Automatic Invariant Inference

int f1 ( int n) { int r = 0; i f (n == 0) return 1; while (n > 0) { n /= 10; r++; } return r ; } int f2 ( int n) { int r = 1; while ( true ) { i f (n < 10) return r ; i f (n < 100) return r +1; i f (n < 1000) return r +2; i f (n < 10000) return r +3; n /= 10000; r += 4; } }

11 / 16

SLIDE 19

Automatic Invariant Inference

Loop synchronisation

f1 f2

To show: Equal input

gives equal output

12 / 16

SLIDE 20

Automatic Invariant Inference

Loop synchronisation

f1 f2

= =

To show: Equal input

gives equal output

12 / 16

SLIDE 21

Automatic Invariant Inference

Loop synchronisation

f1 f2

= =

To show: Equal input

gives equal output

Loops are synchronised

12 / 16

SLIDE 22

Automatic Invariant Inference

Loop synchronisation

f1 f2

= = =

To show: Equal input

gives equal output

Loops are synchronised

12 / 16

SLIDE 23

Automatic Invariant Inference

Loop synchronisation

f1 f2

= = =

To show: Equal input

gives equal output

Loops are synchronised
... at least loosely

synchronised

12 / 16

SLIDE 24

Automatic Invariant Inference

Loop synchronisation

f1 f2

= =

C

To show: Equal input

gives equal output

Loops are synchronised
... at least loosely

synchronised

12 / 16

SLIDE 25

Automatic Invariant Inference

Loop synchronisation

f1 f2

= =

C

To show: Equal input

gives equal output

Loops are synchronised
... at least loosely

synchronised ⇒ Use C as loop invariant for both programs. (→coupling invariant)

12 / 16

SLIDE 26

Automatic Invariant Inference

Loop synchronisation

f1 f2

= =

C

To show: Equal input

gives equal output

Loops are synchronised
... at least loosely

synchronised ⇒ Use C as loop invariant for both programs. (→coupling invariant)

12 / 16

SLIDE 27

Automatic Invariant Inference

Loop synchronisation

f1 f2

=

C

To show: Equal input

gives equal output

Loops are synchronised
... at least loosely

synchronised ⇒ Use C as loop invariant for both programs. (→coupling invariant)

12 / 16

SLIDE 28

Automatic Invariant Inference

Loop synchronisation

f1 f2

=

C C

To show: Equal input

gives equal output

Loops are synchronised
... at least loosely

synchronised ⇒ Use C as loop invariant for both programs. (→coupling invariant)

12 / 16

SLIDE 29

Automatic Invariant Inference

Loop synchronisation

f1 f2

=

C C C

=

To show: Equal input

gives equal output

Loops are synchronised
... at least loosely

synchronised ⇒ Use C as loop invariant for both programs. (→coupling invariant)

12 / 16

SLIDE 30

Automatic Invariant Inference

Loop synchronisation

f1 f2

=

C C C

=

To show: Equal input

gives equal output

Loops are synchronised
... at least loosely

synchronised ⇒ Use C as loop invariant for both programs. (→coupling invariant)

Automatic Regression Verification:

Do not specify C but infer it automatically.

12 / 16

SLIDE 31

Automatic Invariant Inference

Three cases to consider:

1 Initially coupling loop invariant C holds 2 After both loop steps (or one if other finished), C holds 3 After both loops finished, C implies equality of results

13 / 16

SLIDE 32

Automatic Invariant Inference

Three cases to consider:

1 Initially coupling loop invariant C holds 2 After both loop steps (or one if other finished), C holds 3 After both loops finished, C implies equality of results

Automatically inferred coupling loop invariant: (Using Eldarica)

(n1 > 0 → (n1 = n2 ∧ r1 + 1 = r2)) ∧(n2 ≤ 0 → return2 = r1) ∧n1 ≥ n2

13 / 16

SLIDE 33

Automatic Invariant Inference

Three cases to consider:

1 Initially coupling loop invariant C holds 2 After both loop steps (or one if other finished), C holds 3 After both loops finished, C implies equality of results

Automatically inferred coupling loop invariant: (Using Eldarica)

(n1 > 0 → (n1 = n2 ∧ r1 + 1 = r2)) ∧(n2 ≤ 0 → return2 = r1) ∧n1 ≥ n2

Compare to loop invariant: n = n0

10r

Coupling invariant is not trivial, but linear and inferable!

13 / 16

SLIDE 34

Overview

1 Overapproximation using Uninterpreted Functions 2 Approximation using Uninterpreted Predicates 3 Results and Future Work

14 / 16

SLIDE 35

Evaluation and Results

Approaches implemented for a subset of C: simplRV, Rˆ eve Usable with webinterface: http://formal.iti.kit.edu/ improve/deduktionstreffen2014/

Rˆ eve evaluation (uninterpreted predicates)

32 short benchmarks of integer programs (10-50 lines)
Collected from literature
Good performance on most equivalent programs
Finds counterexample for non-equivalent programs as well

15 / 16

SLIDE 36

Conclusion

Regression Verification

Initial approach limited to strongly coupled recursions
r user feedback
Automatic Invariant Inference: More powerful, using recent

techniques in SMT solvers like Eldarica and Z3

Future Work

More examples (larger)
Support arrays, heaps, objects

16 / 16