Invariants and State in Testing and Formal Methods Dick Hamlet - - PowerPoint PPT Presentation

Invariants and State in Testing and Formal Methods

Dick Hamlet Portland State University

Supported by NSF CCR-0112654 and SFI E.T.S. Walton Fellowship

1/10

The Simplest Context

Meaning of a program with persistent state:

• input domain

(think: STDIN)

• utput domain

(think: STDOUT)

• state space

(think: permanent R/W file)

2/10

The Simplest Context

Meaning of a program with persistent state:

• input domain

(think: STDIN)

• utput domain

(think: STDOUT)

• state space

(think: permanent R/W file)

• ✁

2/10

State is Anomalous

On the one hand... On the other hand...

3/10

State is Anomalous

On the one hand... On the other hand... States are ‘inputs’ that influence program be- havior

3/10

State is Anomalous

On the one hand... On the other hand... States are ‘inputs’ that influence program be- havior States are ‘outputs’ that only the program creates

3/10

State is Anomalous

On the one hand... On the other hand... States are ‘inputs’ that influence program be- havior States are ‘outputs’ that only the program creates

(bottom line)

A state variable is not independent – sample at your own risk!

3/10

Testing Viewpoint

Stateless case: Black-box program

• .

Specification function

• .

Test point

• ✁

fails if

• ✞

• ✞

. Operational profile: Usage P .d.f. on .

4/10

Testing Viewpoint

Stateless case: Black-box program

• .

Specification function

• .

Test point

• ✁

fails if

• ✞

• ✞

. Operational profile: Usage P .d.f. on . Persistent state: Replace by sequences

• ✂
• ✁✄✂

.

• . (Sequence profile)

4/10

Testing Viewpoint

Stateless case: Black-box program

• .

Specification function

• .

Test point

• ✁

fails if

• ✞

• ✞

. Operational profile: Usage P .d.f. on . Persistent state: Replace by sequences

• ✂
• ✁✄✂

.

• . (Sequence profile)

State is only implicit — tester may sample ...(?)

4/10

Proving Viewpoint

Specification is a first-order formula in values

• f program variables

. Type, Symbol Evaluation Variables (

• ✝
• riginal)

Pre-cond before

Post-cond after

Assertion any

Invariant

before/after

5/10

Proving Viewpoint

Specification is a first-order formula in values

• f program variables

. Type, Symbol Evaluation Variables (

• ✝
• riginal)

Pre-cond before

Post-cond after

Assertion any

Invariant

before/after

State variable

is explicit – specification is state-prescriptive...(?)

5/10

Invariants in Proofs

Room for confusion – First-order formulas include implicit evaluation times; Hoare logic hides quantification. For example, correctness of program :

• ✄

6/10

Invariants in Proofs

Room for confusion – First-order formulas include implicit evaluation times; Hoare logic hides quantification. For example, correctness of program :

• ✄

Invariant role filter out

• impossible states.

Pre-condition role filter out inputs humans agree not to use.

• ✄

6/10

Testing with Invariants

Stateless testing of to approximate proof: Sample , and for each

such that

, run and check

. (TestEra)

7/10

Testing with Invariants

Stateless testing of to approximate proof: Sample , and for each

such that

, run and check

. (TestEra) With state it’s more complicated. First try: Sample

. For each

such that

• ✂

, run and check

.

7/10

Testing with Invariants

Stateless testing of to approximate proof: Sample , and for each

such that

, run and check

. (TestEra) With state it’s more complicated.

• First try: Sample

. For each

such that

• ✂

, run and check

. Better: Sample

• , say

, such that

• ☎

. Sample

. If

, run

• n the sequence,
• btaining state sequence

and check

• ✂

.

7/10

Proof-, Testing-like Formulas

Let be a logical formula (invariant, post-condition, etc.) applied to a program.

8/10

Proof-, Testing-like Formulas

Let be a logical formula (invariant, post-condition, etc.) applied to a program. is Proof-like: No test case can falsify . is Testing-like: There is a low probability that test-case sequences drawn according to a given operational profile will falsify . Since profiles are arbitrary human specifications, proof-like and testing-like can be very different.

8/10

Proof-, Testing-like Formulas

Let be a logical formula (invariant, post-condition, etc.) applied to a program. is Proof-like: No test case can falsify . is Testing-like: There is a low probability that test-case sequences drawn according to a given operational profile will falsify . Since profiles are arbitrary human specifications, proof-like and testing-like can be very different. itself can be proof- or testing-like if it is obtained using all possibilities, or only those from a profile.

8/10

Daikon, TestEra, Etc.

Daikon TestEra Generates possible pre- and post-conditions from given testset. Generates bounded exhaustive testset (BET) from given pre-condition; checks given post-condition.

9/10

Daikon, TestEra, Etc.

Daikon TestEra Generates possible pre- and post-conditions from given testset. Generates bounded exhaustive testset (BET) from given pre-condition; checks given post-condition. +invariant +profile

9/10

Daikon, TestEra, Etc.

Daikon TestEra Generates possible pre- and post-conditions from given testset. Generates bounded exhaustive testset (BET) from given pre-condition; checks given post-condition. +invariant +profile

• ✁

From invariant and profile, generate BET; check invariant as post-condition. Use BET to generate possible post-condition.

9/10

Summary

• Testing needs to recognize state and

invariants

• Sample state with care!
• Drive sampling with invariants

10/10

Summary

• Testing needs to recognize state and

invariants

• Sample state with care!
• Drive sampling with invariants
• Invariants are inherently prescriptive

10/10

Summary

• Testing needs to recognize state and

invariants

• Sample state with care!
• Drive sampling with invariants
• Invariants are inherently prescriptive
• Operational profiles define ‘usage invariants’

10/10

Summary

• Testing needs to recognize state and

invariants

• Sample state with care!
• Drive sampling with invariants
• Invariants are inherently prescriptive
• Operational profiles define ‘usage invariants’
• Tools using first-order formulas with tests

need specification-based invariants

10/10

Summary

• Testing needs to recognize state and

invariants

• Sample state with care!
• Drive sampling with invariants
• Invariants are inherently prescriptive
• Operational profiles define ‘usage invariants’
• Tools using first-order formulas with tests

need specification-based invariants

10/10