Testing Programs Programs Testing with Symmetry Symmetry with - - PowerPoint PPT Presentation

▶
testing programs programs testing with symmetry symmetry
SMART_READER_LITE
LIVE PREVIEW

Testing Programs Programs Testing with Symmetry Symmetry with - - PowerPoint PPT Presentation

Dec 06, 2022 363 likes •596 views

Testing Programs Programs Testing with Symmetry Symmetry with and why why not not Java Java Card Card applets applets and and APIs APIs ? ? and Arnaud Gotlieb Gotlieb Arnaud IRISA / INRIA IRISA / INRIA Rennes, FRANCE Rennes,

slide-1
SLIDE 1

Cassis Cassis

  • - Mar 2004

Mar 2004 1 1

Testing Testing Programs Programs with with Symmetry Symmetry

and and why why not not Java Java Card Card applets applets and and APIs APIs ? ?

Arnaud Arnaud Gotlieb Gotlieb IRISA / INRIA IRISA / INRIA Rennes, FRANCE Rennes, FRANCE

slide-2
SLIDE 2

2 2 Cassis Cassis

  • - Mar 2004

Mar 2004

Outline Outline

Motivations Motivations A A Generalized Generalized definition definition of

  • f symmetry

symmetry relation relation Symmetric Symmetric Testing Testing First First experimental experimental results results Related Related and and further further works works

slide-3
SLIDE 3

3 3 Cassis Cassis

  • - Mar 2004

Mar 2004

A A diagrammatic diagrammatic view view of

  • f Program

Program Testing Testing

A sequential program

computing Input test data Outputs checking verdict (pass, fail, ?) Oracle

slide-4
SLIDE 4

4 4 Cassis Cassis

  • - Mar 2004

Mar 2004

Non Non-

  • testable

testable programs programs [ [Weyuker Weyuker 82] 82]

  • No (

No (complete complete and and correct) oracle correct) oracle available available Because Because No No formal formal model model available available Only Only informal informal and and partial partial specifications specifications Expected Expected results results too too difficult difficult to to compute compute by hand by hand … … Typical Typical examples examples: : APIs APIs, , third third-

  • party

party libraries libraries ( (no no source code) source code) COTS ( COTS (no no source code) source code) complex complex mathematical mathematical functions functions

slide-5
SLIDE 5

5 5 Cassis Cassis

  • - Mar 2004

Mar 2004

Testing Testing with with symmetry symmetry : a : a very very first first example example

P : a P : a program program that that implements implements the the gcd gcd of

  • f 2

2 integers integers Problem Problem: P(1309, 693) = ? : P(1309, 693) = ?

Hence Hence, if P(1309, 693) , if P(1309, 693) ≠ ≠ P(693, 1309) P(693, 1309) then then verdict = verdict = fail fail Symmetry Symmetry relation: relation: ∀ ∀u, u, ∀ ∀v, v, gcd gcd(u,v) = (u,v) = gcd gcd(v,u) (v,u)

slide-6
SLIDE 6

6 6 Cassis Cassis

  • - Mar 2004

Mar 2004

Outline Outline

Motivations Motivations A A Generalized Generalized definition definition of

  • f symmetry

symmetry relation relation Symmetric Symmetric Testing Testing First First experimental experimental results results Related Related and and further further works works

slide-7
SLIDE 7

7 7 Cassis Cassis

  • - Mar 2004

Mar 2004

Background on Group Background on Group Theory Theory

Group (E, Group (E,o)

  • ) iff

iff ∃ ∃neutral, neutral, ∀ ∀x x ∃ ∃inverse(x inverse(x), o associative ), o associative Symmetric Group Symmetric Group S Sn

n : set of permutations over {1,..,n}

: set of permutations over {1,..,n} if x = (x if x = (x1

1,..,x

,..,xn

n)

) θ θ.x denotes (x .x denotes (xθ

θ(1) (1), ..,

, .., x xθ

θ(n (n) ))

) S Sn

n can be generated by

can be generated by τ τ = (12) = (12) and and σ σ = (12..k) = (12..k) Group homomorphism Group homomorphism from from S Sk

k to

to S Sl

l

ϕ ϕ : : S Sk

k

S Sl

l

such as such as ϕ ϕ( (θ θ o

  • θ

θ’ ’) = ) = ϕ ϕ( (θ θ) o ) o ϕ ϕ( (θ θ’ ’) )

slide-8
SLIDE 8

8 8 Cassis Cassis

  • - Mar 2004

Mar 2004

Symmetry Symmetry relation relation [

[Gotlieb Gotlieb ISSRE 03] ISSRE 03]

Program Program p : p : D D1

1 x

x ...

... x

x D

Dk

k

  • D

D1

1’

’ x

x ...

... x

x D

Dl

l’

’

ψ ψk

k,l ,l is

is a a symmetry symmetry relation relation for p for p over

  • ver D

D1

1 x

x ...

... x

x D

Dk

k iff

iff : : 1) 1) ∀ ∀θ θ ∈ ∈ S Sk

k,

, ∃η ∃η ∈ ∈ S Sl

l,

, such such as as ∀ ∀x x p( p( θ θ.x .x ) = ) = η η.p .p( x ) ( x )

2) ψk,l :

: S Sk

k

S Sl

l

is is a group a group homomorphism homomorphism θ θ η η Ex: Ex: gcd gcd satisfies satisfies a a ψ

ψ2,1

2,1 symmetry

symmetry relation relation over

  • ver ℕ

ℕ x

x ℕ

ℕ

slide-9
SLIDE 9

9 9 Cassis Cassis

  • - Mar 2004

Mar 2004

Symmetry Symmetry relation : relation : examples examples

Symm Symm relation relation

Per. Per.

  • utp
  • utp

Perm. Perm. inputs inputs

Methods Methods from from java.util.Collections

java.util.Collections

(12 symmetric methods over 19 distincts methods)

ψ ψ|A|,|A|

|A|,|A|

A A A A

void void sort sort(List A) (List A)

ψ ψ|A|,|B|

|A|,|B|

B B A A

void void copy copy(List B, (List B, List List A) A)

ψ ψ1,n

1,n

Ret Ret O O

List List nCopies nCopies( (int int n, n,Object Object O) O)

ψ ψ|A|,1

|A|,1

Ret Ret A A

Object Object max max(Collection A) (Collection A)

ψ ψ|A|,|A|

|A|,|A|

A A A A

boolean boolean replaceAll replaceAll(List A, (List A, Object Object oldVal

  • ldVal,

,Object Object newVal newVal) )

slide-10
SLIDE 10

1 1

Cassis Cassis

  • - Mar 2004

Mar 2004

Finding Finding symmetry symmetry violations violations

The symmetry relation has to be given by the tester: in extension { ( θ, η) }∀θ∈Sk If p(θ.x) ≠ η.p(x) for any x ∈ D1 x .. x Dk then verdict = fail Any test data generator can be employed (random, pair-wise, boundary-value, …) But, how to find all the symmetry violations ?

slide-11
SLIDE 11

11 11 Cassis Cassis

  • - Mar 2004

Mar 2004

Outline Outline

Motivations Motivations A A Generalized Generalized definition definition of

  • f Symmetry

Symmetry relation relation Symmetric Symmetric Testing Testing First First experimental experimental results results Related Related and and further further works works

slide-12
SLIDE 12

12 12 Cassis Cassis

  • - Mar 2004

Mar 2004

Local exhaustive Local exhaustive testing testing [Wood,Miller,

[Wood,Miller,Noonan Noonan 92] 92]

tuned tuned for for testing testing symmetry symmetry relations relations

  • Tries exhaustively all the input values

into a restricted finite domain D of D1 x .. x Dk in Symmetric Test., a Cartesian Product iterator

Ex: {a,b} x {c,d,e} gives (a,c),(a,d),(a,e),(b,c),(b,d),(b,e)

  • Proves that p(θ.x) = η.p(x) holds

∀x ∈ D when both the executions of p(θ.x) and p(x) terminate

slide-13
SLIDE 13

13 13 Cassis Cassis

  • - Mar 2004

Mar 2004

Comparison Comparison checks checks

x x

p p o

  • θ

θ (x

x) ≠ ≠ ψ

ψk

k,l ,l(θ

θ) ) o

  • p

p (x)

(x) p(x) p(x) θ θ.x x ∀ ∀x x ∈ ∈ D D, , ∀ ∀θ θ ∈ ∈ S Sk

k, , ST

ST checks checks: :

p p

θ θ

η = ψ

ψk

k,l ,l (θ

θ) )

  • but

but there are k! permutations in there are k! permutations in S Sk

k

  • needs to know

needs to know ψ ψk

k,l ,l(θ

θ) ) for all θ θ ∈ ∈ S Sk

k

slide-14
SLIDE 14

14 14 Cassis Cassis

  • - Mar 2004

Mar 2004

Checking Checking only

  • nly two

two permutations: permutations:

Symmetric Testing requires only to check τ = (12) and σ = (12..k) Proposition : p o τ = ψk,l (τ) o p ∀θ ∈ Sk, p o θ = ψk,l (θ) o p ⇔ p o σ = ψk,l (σ) o p Sketch of proof: (⇒) trivial (⇐) p o θ = p o (τ o σ…) = ψk,l(τ) o p o (σ…) = (ψk,l(τ) o ψk,l(σ) …) o p = ψk,l (θ) o p (because ψk,l is an homomorphism)

slide-15
SLIDE 15

15 15 Cassis Cassis

  • - Mar 2004

Mar 2004

A A semi semi-

  • correct

correct procedure procedure for ST for ST

In: In: program program p p, , finite finite domain domain D D, , ψ ψk

k,l ,l(

(τ τ), ), ψ ψk

k,l ,l(

(σ σ) ) Out: Out: a a symmetry symmetry violation or a violation or a proof proof that that ψ ψk

k,l ,l holds

holds over

  • ver D

D while while( D ( D ≠ ≠ ∅ ∅ ) ) pick pick up up x in D x in D and and D := D D := D \ \ {x} {x} let let r:= p(x), r:= p(x), r rτ

τ := p(

:= p(τ τ.x .x), ), r rσ

σ := p(

:= p(σ σ.x .x) ) if if( ( r rτ

τ ≠

≠ ψ ψk

k,l ,l(

(τ τ) . ) . r) r) then return violation then return violation ( (x,r x,r, ,r rτ

τ)

) if if( ( r rσ

σ ≠

≠ ψ ψk

k,l ,l(

(σ σ) . ) . r) r) then return then return violation violation(x,r (x,r, ,r rσ

σ)

) return return( (« «Q.E.D. Q.E.D.» ») )

slide-16
SLIDE 16

16 16 Cassis Cassis

  • - Mar 2004

Mar 2004

Limitations Limitations of

  • f Symmetric

Symmetric Testing Testing

Terminaison Terminaison not not guaranteed guaranteed, but , but # # comparison comparison checks checks is is O(d) in place O(d) in place of

  • f O(k! . d)

O(k! . d) where where d = # test data d = # test data Impossible to Impossible to know know which which inputs inputs among among x, x,τ τ.x .x, ,σ σ.x .x is is responsible responsible of

  • f the

the symmetry symmetry violation violation Incorrect versions Incorrect versions of

  • f p

p may may be be symmetric symmetric too too ! ! But, But, No oracle No oracle is is required required, ST , ST is is fully fully automatic automatic

slide-17
SLIDE 17

17 17 Cassis Cassis

  • - Mar 2004

Mar 2004

Outline Outline

Motivations Motivations A A Generalized Generalized definition definition of

  • f symmetry

symmetry relation relation Symmetric Symmetric Testing Testing First First experimental experimental results results Related Related and and further further works works

slide-18
SLIDE 18

18 18 Cassis Cassis

  • - Mar 2004

Mar 2004

Experiments Experiments on

  • n testing

testing Java Java methods methods

Symmetric Symmetric Testing Testing implemented implemented with with the the Java unit Java unit testing testing tool tool : : Roast Roast [ [Daley Daley, , Hoffman Hoffman, , Strooper Strooper 2002] 2002] performed performed on

  • n programs

programs where where faults faults were were injected injected by by mutation (37 mutants mutation (37 mutants manually manually created created) )

Integer Integer GetMid GetMid( (Integer Integer i, i,Integer Integer j, j,Integer Integer k) k) int int Trityp Trityp( (Integer Integer x, x,Integer Integer y, y,Integer Integer z) z) Vector Vector min_nb min_nb( (Vector Vector V, V, int int n) n)

performed on methods from java.lang.Collections

java.lang.Collections

void void sort sort(List A), (List A), void void copy copy(List A, (List A, List List B), B), bool bool replaceAll replaceAll(List A, (List A, Object Object oldVal

  • ldVal,

, newVal newVal) )

slide-19
SLIDE 19

19 19 Cassis Cassis

  • - Mar 2004

Mar 2004

Experimental Experimental results results

≈ ≈2* 2*10 106

6

≈ ≈10 106

6

≈ ≈10 106

6

Size Size of

  • f D

D #test data #test data 53.3 sec 53.3 sec 2/2 2/2 ψ ψ|A|,

|A|,Ret Ret

sort sort ° ° min_nb min_nb 9.6 sec 9.6 sec 23/33 23/33 ψ ψ3,1

3,1

trityp trityp 9.4 sec 9.4 sec 2/2 2/2 ψ ψ3,1

3,1

GetMid GetMid time time used used to to prove prove symmetry symmetry Mutation Mutation score score symmetry symmetry programs programs

(CPU time on 1.8GHz Pentium 4 with Sun Standard 1.4.1 JVM)

4606.6 sec 4606.6 sec ≈ ≈25 * 25 * 10 106

6

  • ψ

ψ|A|,|A|

|A|,|A|

replaceAll replaceAll 13.7 sec 13.7 sec ≈ ≈1/2 * 1/2 * 10 106

6

  • ψ

ψ|A|,|B|

|A|,|B|

copy copy 13.1 sec 13.1 sec ≈ ≈1/2 * 1/2 * 10 106

6

  • ψ

ψ|A|,|A|

|A|,|A|

sort sort

Programs extracted from java.lang.Collections

slide-20
SLIDE 20

20 20 Cassis Cassis

  • - Mar 2004

Mar 2004

Outline Outline

Motivations Motivations A A Generalized Generalized definition definition of

  • f symmetry

symmetry relation relation Symmetric Symmetric Testing Testing First First experimental experimental results results Related Related and and further further works works

slide-21
SLIDE 21

21 21 Cassis Cassis

  • - Mar 2004

Mar 2004

Related Related work work

Data Data Diversity Diversity

[ [Ammann Ammann, , Knight Knight TComp’88] TComp’88]

Symmetry Symmetry and and Model Model Ckecking Ckecking [Emerson,

[Emerson, Sistla Sistla CAV CAV’ 93] ’ 93] [ [Ip Ip, , Dill Dill CHDL’93] CHDL’93] Symmetry Symmetry is is used used to prune to prune the the exploration exploration of

  • f the

the states states space space

Metamorphic Metamorphic Testing Testing

[ [Chen Chen, ,Tse Tse, ,Zhou Zhou COMPSAC’01] COMPSAC’01]

r(x r(x1

1,..,

,..,x xn

n)

) ⇒ ⇒ r rf

f(p(x

(p(x1

1),..,p(

),..,p(x xn

n))

))

slide-22
SLIDE 22

22 22 Cassis Cassis

  • - Mar 2004

Mar 2004

Further Further works works

Reaching Reaching the the minimum minimum number number of

  • f comparison

comparison checks checks by by finding finding ad ad-

  • hoc

hoc order

  • rder of
  • f n_tuples

n_tuples generation generation Expressing Expressing symmetry symmetry relations in OCL (or in JML) relations in OCL (or in JML) as as postconditions postconditions requires requires to to define define Symmetric Symmetric Group classes Group classes Testing Testing Java Java Card Card applets applets and and APIs APIs, , where where non non-

  • trivial

trivial symmetric symmetric relations relations may may exist exist: :

Ex: Ex: abstract short abstract short javacard.security.Checksum. javacard.security.Checksum. doFinal doFinal(byte (byte inBuff inBuff[], ...) [], ...) which is based on CRC algorithms

slide-23
SLIDE 23

23 23 Cassis Cassis

  • - Mar 2004

Mar 2004

An An example example with with inheritance inheritance

abstract class Animal { abstract int m(); } class B extends A { int m() {return 0;}; } class C extends A { int m() {return 1;}; } class Use { int p(A a) { return a.m(); }; }

p : {b,c}2 {0,1}2 where b is identified to (b,c) and c is identified to (c,b) p has to satisfy a ψ2,2 symmetry relation because p(τ τ.(b,c) .(b,c)) = τ τ.p .p((b,c)) ((b,c)) and and p(τ τ.(c,b) .(c,b)) = τ τ.p .p((c,b)) ((c,b)) Practically Practically, to , to check check whether whether p(b) = 1 p(b) = 1 – – p(a) for p(a) for example example