[PPT] - U s i n g S i m u l a t e d E x e c u t i o n PowerPoint Presentation

SLIDE 1

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 1

U s i n g S i m u l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h m s

"How to help a theorem prover with execution data"

T

h

N e Wi n , M i c h a e l E r n s t , S t e p h e n G a r l a n d , D i l s u n K i r l i K a y n a r , N a n c y L y n c h

SLIDE 2

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 2

G

a

l : m a k e t h e

r

e m p r

v

e r s e a s i e r t

u

s e

Wh

y d

w

e w a n t t

u

s e a p r

v

e r ?

– T

v

e r i f y g e n e r a l , i n fj n i t e s t a t e s y s t e m s

Wh

a t ' s h a r d a b

u

t u s i n g a p r

v

e r ?

– T

h e y g e t s t u c k a n d n e e d h u m a n i n p u t

SLIDE 3

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 3

Wh a t k i n d

f

h u m a n i n p u t ?

Theorem prover Program to be verified Verified proof

SLIDE 4

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 4

Wh a t k i n d

f

h u m a n i n p u t ?

Theorem prover Program to be verified Verified proof

Tactics

Proof structure Case analysis Which facts to use

SLIDE 5

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 5

Wh a t k i n d

f

h u m a n i n p u t ?

Theorem prover Program to be verified Verified proof

Tactics Lemmas

Human insight and intuition

n invariants of reachable states

Proof structure Case analysis Which facts to use

SLIDE 6

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 6

T r a d i t i

n

a l a p p r

a

c h e s

Theorem prover Program to be verified Verified proof

Lemmas ??? Tactics Improved tactics to prover

Human insight and intuition

n invariants of reachable states

SLIDE 7

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 7

U s i n g e x e c u t i

n

d a t a t

h

e l p p r

v

e r s

P

r

g

r a m s a r e

f

t e n t e s t e d b e f

r

e v e r i fj c a t i

n

– T

e s t i n g s h

w

s e r r

r

s q u i c k l y

– V

e r i fj c a t i

n

i s e x p e n s i v e i n h u m a n t i m e

E

x e c u t i

n

d a t a i s n

r

m a l l y t h r

w

n a w a y

– Wh

a t i n f

r

m a t i

n

c a n b e k e p t f

r

p r

f

s ?

SLIDE 8

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 8

G e n e r a t i n g t a c t i c s

Theorem prover Program annotated for testing by execution Verified proof Proof structure and case analysis Translator from test cases to prover language

Lemmas Tactics

SLIDE 9

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 9

G e n e r a t i n g l e m m a s

Theorem prover Verified proof Proof structure and case analysis Translator from test cases to prover language Conjectured invariants Daikon runtime analysis tool generalizes over executions

Lemmas Tactics

Program annotated for testing by execution

SLIDE 10

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 1

O u t l i n e

M
t

i v a t i

n

: e x e c u t i

n
a

s s i s t e d t h e

r

e m p r

v

e r s

F
r

m a l m

d

e l : I O a u t

m

a t

n
C

a s e s t u d y : L a m p

r

t ' s P a x

s

p r

t
c
l
L

e m m a s : c

n

j e c t u r e d i n v a r i a n t s

T

a c t i c s : p r

f
u

t l i n e

C
n

c l u s i

n

SLIDE 11

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 1 1

F

r

m a l m

d

e l : I O a u t

m

a t

n
M
d

e l f

r

d i s t r i b u t e d s y s t e m s

– L

a b e l e d ( i n fj n i t e , n

n

d e t e r m i n i s t i c ) s t a t e m a c h i n e

– F

i r s t

r

d e r l

g

i c t

d

e fj n e t r a n s i t i

n

s

[Lynch/Tuttle 89]

SLIDE 12

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 1 2

F

r

m a l m

d

e l : I O a u t

m

a t

n
M
d

e l f

r

d i s t r i b u t e d s y s t e m s

– L

a b e l e d ( i n fj n i t e , n

n

d e t e r m i n i s t i c ) s t a t e m a c h i n e

– F

i r s t

r

d e r l

g

i c t

d

e fj n e t r a n s i t i

n

s

M

u l t i p l e l e v e l s

f

a b s t r a c t i

n

– A

b s t r a c t s p e c i fj c a t i

n

a u t

m

a t

n

– L

a y e r e d i m p l e m e n t a t i

n

a u t

m

a t a

Abstract Implementa tion Specifica tion Concrete Implementa tion [Lynch/Tuttle 89]

SLIDE 13

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 1 3

V e r i fj c a t i

n

m e t h

d

s

S

i m u l a t i

n

r e l a t i

n

s f

r

r e fj n e m e n t

Abstract Implementa tion Specifica tion Concrete Implementa tion Simulation relation Simulation relation

SLIDE 14

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 1 4

V e r i fj c a t i

n

m e t h

d

s

S

i m u l a t i

n

r e l a t i

n

s f

r

r e fj n e m e n t

I

n v a r i a n t a s s e r t i

n

s f

r

i m p l e m e n t a t i

n

s

Abstract Implementa tion Specifica tion Concrete Implementa tion Invariants Invariants Simulation relation Simulation relation

SLIDE 15

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 1 5

I O A l a n g u a g e a n d t

l

s

I

O A i n t e r p r e t e r

– A

l l

w

s s i m u l a t e d e x e c u t i

n
f
n

e a u t

m

a t

n

,

r
f

a p a i r f

r

r e fj n e m e n t

– U

s e r

s

p e c i fj e d s c h e d u l i n g t

r

e s

l

v e n

n

d e t e r m i n i s m

I

O A t r a n s l a t

r

s t

p

r

v

i n g l a n g u a g e s

– T

h e L a r c h P r

v

e r

– I

s a b e l l e / H O L

SLIDE 16

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 1 6

O u t l i n e

M
t

i v a t i

n

: e x e c u t i

n
a

s s i s t e d t h e

r

e m p r

v

e r s

I

O a u t

m

a t

n

m

d

e l

C

a s e s t u d y : L a m p

r

t ' s P a x

s

p r

t
c
l
L

e m m a s : c

n

j e c t u r e d i n v a r i a n t s

T

a c t i c s : p r

f
u

t l i n e

C
n

c l u s i

n

SLIDE 17

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 1 7

P a x

s

i n I O A

S

p e c i fj c a t i

n

f

r

c

n

s e n s u s

Globalized implementation using

ballots and quorums

Global1 Consens us Concrete Paxos

SLIDE 18

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 1 8

S p e c i fj c a t i

n

f

r

c

n

s e n s u s

automaton Consensus % Inputs and outputs are externally visible. signature input init (i:Node, v:Value) input fail (i:Node)

utput decide (i:Node, v:Value)

internal chooseVal (v:Value) states proposed, chosen : Set[Value] := {} ... transitions

internal chooseVal (v) pre v  proposed; chosen = {} eff chosen := {v} ...

Global1

Consens us

SLIDE 19

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 1 9

I m p l e m e n t a t i

n

b y G l

b

a l 1

Automaton Global1 signature input init (i:Node, v:Value) input fail (i:Node)

utput decide

(i:Node, v:Value) internal internalDecide (b:Ballot)... states succeeded, createdBallots : Set[Ballot] ... internal internalDecide(b:Ballot) pre % The ballot was created. b  createdBallots; % There was a quorum that voted on the ballot.  quorum : Set[Node] (quorum  wquorums  Global1

Consensus

SLIDE 20

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 2

G a m e p l a n f

r

p r

f

Global1 Consens us

S

h

w

t h a t G l

b

a l 1 i m p l e m e n t s C

n

s e n s u s

– S

i m u l a t i

n

r e l a t i

n

p r

f

Simulation relation

SLIDE 21

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 2 1

G a m e p l a n f

r

p r

f

Global1 Consens us

S

h

w

t h a t G l

b

a l 1 i m p l e m e n t s C

n

s e n s u s

– S

i m u l a t i

n

r e l a t i

n

p r

f
N

e e d i n v a r i a n t s

n

G l

b

a l 1

Invariants Simulation relation

SLIDE 22

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 2 2

O u t l i n e

M
t

i v a t i

n

: e x e c u t i

n
a

s s i s t e d t h e

r

e m p r

v

e r s

I

O a u t

m

a t

n

m

d

e l

C

a s e s t u d y : L a m p

r

t ' s P a x

s

p r

t
c
l
L

e m m a s : c

n

j e c t u r e d i n v a r i a n t s

T

a c t i c s : p r

f
u

t l i n e

C
n

c l u s i

n

SLIDE 23

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 2 3

U s e s

f

i n v a r i a n t s

L

e m m a s i n p r

f

s

– O

f s i m u l a t i

n

s r e l a t i

n

s

– O

f

t

h e r i n v a r i a n t s t a t e m e n t s

– O

f t e n n e e d e d b e c a u s e t h e i n d u c t i

n

h y p

t

h e s i s f

r

a p r

f

m u s t b e s t r

n

g e n

u

g h

Implementa tion Specifica tion Simulation relation Invariants

SLIDE 24

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 2 4

H

w

t

c
n

j e c t u r e i n v a r i a n t s

E

x e c u t e a u t

m

a t

n

u s i n g t e s t c a s e s

U

s e D a i k

n

t

l
n

e x e c u t i

n

d a t a

– A

n a l y z e s e x e c u t i

n

d a t a

– O

u t p u t s p r

p

e r t i e s t r u e f

r
b

s e r v e d e x e c u t i

n

s

– I

n v a r i a n t s i n fj r s t

r

d e r l

g

i c

IOA interpreter Daikon Scheduled automaton Execution data Conjectured invariants

SLIDE 25

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 2 5

I s s u e s w i t h c

n

j e c t u r e d i n v a r i a n t s

U

n s

u

n d

– S

t a t i s t i c a l a n a l y s i s r e d u c e s f a l s e p

s

i t i v e s

– U

s e p r

v

e r t

p

r

v

e c

n

j e c t u r e d i n v a r i a n t s

I

n c

m

p l e t e

– N

e c e s s a r y b e c a u s e s e a r c h s p a c e i s i n fj n i t e

N

e e d s t e s t c a s e s

– I

n p r a c t i c e , t e s t c a s e s e x i s t

– We

u s e r a n d

m

i z e d s c h e d u l i n g

– T

r i a l

a

n d

e

r r

r

e x e c u t i

n

u s u a l l y e n

u

g h

SLIDE 26

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 2 6

C

n

j e c t u r e d i n v a r i a n t s : e x a m p l e

val(nonNull)  proposed succeeded  createdBallots 0 = size(succeeded  dead) 0 = size(voted[aNode]  abstained[aNode])

P

a x

s

c a s e s t u d y

– F

u

n d 4

f

6 i n v a r i a n t s n e e d e d f

r

s i m u l a t i

n

r e l a t i

n

p r

f

SLIDE 27

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 2 7

Wh a t w a s n

t

f

u

n d

I

n v a r i a n t s w i t h

– E

x i s t e n t i a l q u a n t i fj e r s

I

f a b a l l

t

h a s s u c c e e d e d , a q u

r

u m v

t

e d f

r

i t

(b  succeeded   quorum : Set[Node] (quorum  wquorums   n : Node (n  quorum  b  voted[n])) (val[b] ~= nil  b' < b)  (val[b'] = val[b]  b' dead)

– T

m

a n y b

l

e a n c l a u s e s

I

f a b a l l

t

h a s n

n
n

i l v a l u e , i t i s t h e s a m e v a l u e a s a l l e a r l i e r n

n
d

e a d b a l l

t

s

SLIDE 28

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 2 8

O u t l i n e

M
t

i v a t i

n

: e x e c u t i

n
a

s s i s t e d t h e

r

e m p r

v

e r s

I

O a u t

m

a t

n

m

d

e l

C

a s e s t u d y : L a m p

r

t ' s P a x

s

p r

t
c
l
L

e m m a s : c

n

j e c t u r e d i n v a r i a n t s

T

a c t i c s : p r

f
u

t l i n e

C
n

c l u s i

n

SLIDE 29

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 2 9

G a m e p l a n f

r

p r

f

Global1 Consens us

S

h

w

t h a t G l

b

a l 1 i m p l e m e n t s C

n

s e n s u s

– S

i m u l a t i

n

r e l a t i

n

p r

f

Invariants Simulation relation

SLIDE 30

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 3

T

p

r

v

e a f

r

w a r d s i m u l a t i

n

r e l a t i

n
A implements B if there exists f such that f:

– I

s a r e l a t i

n
n

s t a t e s [ A ] a n d s t a t e s [ B ]

– S

a t i s fj e s a s t a r t c

n

d i t i

n

– S

a t i s fj e s a s t e p c

n

d i t i

n

A B

SLIDE 31

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 3 1

T

p

r

v

e a f

r

w a r d s i m u l a t i

n

r e l a t i

n

w

i t n e s s e x e c u t i

n

b b' a a' a

c t i

n

f f

p r e p

s

t s t e p c

n

d i t i

n

r e a c h a b l e s t a t e

b a f

s t a r t s t a r t c

n

d i t i

n

S p e c i fj c a t i

n

a u t

m

a t

n

B I m p l e m e n t a t i

n

a u t

m

a t

n

A s t a r t s t a t e

A

i m p l e m e n t s B i f t h e r e e x i s t s f s u c h t h a t : Red = proof obligation

SLIDE 32

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 3 2

F

r

w a r d s i m : i n t e r p r e t e r s u p p

r

t

P

a i r e d e x e c u t i

n

m

d

e

f

I O A i n t e r p r e t e r

– F

r

t e s t i n g f

r

w a r d s i m u l a t i

n

s

– U

s e r a n n

t

a t e s p r

g

r a m f

r

w i t n e s s e x e c u t i

n

s

M

e c h a n i c s

f

p a i r e d e x e c u t i

n

– E

x e c u t e i m p l e m e n t a t i

n

a u t

m

a t

n

– U

s e a n n

t

a t i

n

s t

d

r i v e e x e c u t i

n
f

s p e c i fj c a t i

n

a u t

m

a t

n

– C

h e c k t h a t f h

l

d s

SLIDE 33

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 3 3

A n n

t

a t i

n

e x a m p l e

for internal internalDecide (b) do if (b  Global1.succeeded) then ignore elseif (Global1.val[b] = nil) then ignore ... else fire internal chooseVal (Global1.val[b].val) fi

SLIDE 34

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 3 4

G e n e r a t i n g p r

v

e r t a c t i c s f r

m

t e s t s

T

r a n s l a t e t e s t i n g a n n

t

a t i

n

s i n t

p

r

f

s c r i p t s

– F

r

s t a r t c

n

d i t i

n
P

i c k w i t n e s s s t a r t s t a t e b

– F

r

s t e p c

n

d i t i

n
T

a c t i c : s t r u c t u r a l i n d u c t i

n
n

a c t i

n

d a t a t y p e

U

s e c

n

d i t i

n

a l s ( ' i f ' ) i n a n n

t

a t i

n

s t

p

e r f

r

m c a s e s p l i t s

P

i c k w i t n e s s e x e c u t i

n



SLIDE 35

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 3 5

F

r

w a r d s i m : s t e p e x a m p l e

% Annotation for internal internalDecide (b) do if (b  Global1.succeeded) then ignore elseif (Global1.val[b] = nil) then ignore ... else fire internal chooseVal (Global1.val[b].val) fi % Proof prove enabled(internalDecide(b)) =>...% Step condition resume by cases (b  Global1.succeeded) % case true resume by specializing  to []

SLIDE 36

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 3 6

O u t l i n e

M
t

i v a t i

n

: e x e c u t i

n
a

s s i s t e d t h e

r

e m p r

v

e r s

I

O a u t

m

a t

n

m

d

e l

C

a s e s t u d y : L a m p

r

t ' s P a x

s

p r

t
c
l
L

e m m a s : c

n

j e c t u r e d i n v a r i a n t s

T

a c t i c s : p r

f
u

t l i n e

C
n

c l u s i

n

SLIDE 37

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 3 7

D i s c u s s i

n
B

e t t e r t h e

r

e m p r

v

i n g e x p e r i e n c e

– L

e s s h u m a n e fg

r

t

– L

e t s d e s i g n e r s c

n

c e n t r a t e

n

h i g h

l

e v e l p r

f
D

e s i g n e r s h a v e c

n

c e p t

f

h i g h

l

e v e l p r

f

– T

h e

r

e m p r

v

e r s g e t s t u c k i n d e t a i l s

T

a c t i c s : p r

v

i d e p r

f

s t r u c t u r e ( 8 2 / 1 5 l i n e s )

– Wh

a t r e m a i n s i s r e p h r a s i n g

f

f a c t s

L

e m m a s : p r

v

i d e i n v a r i a n t s ( 4 / 6 )

– M

i s s i n g

n

e s s y n t a c t i c a l l y e v i d e n t i n p r

g

r a m c

d

e

SLIDE 38

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 3 8

R e s e a r c h d i r e c t i

n

s

B

e t t e r c

n

j e c t u r e d i n v a r i a n t s

– A

n a l y z e I O A c

d

e s t a t i c a l l y f

r

i n v a r i a n t t e m p l a t e s

F

i n d p r e d i c a t e s i n c

d

e , u s e a s l e f t s i d e

f

i m p l i c a t i

n

s

B

e t t e r p r

f

t a c t i c s , m

r

e a u t

m

a t i

n

– Wh

i c h l e m m a s a r e u s e d i n a l l I O A p r

f

s ?

– Wh

a t

r

d e r i n g

f

l e m m a s ?

E

. g . , “ a p p l y d e fj n i t i

n
f

a u t

m

a t

n

e fg e c t s

n

l y a f t e r i n d u c t i n g

n

t h e a c t i

n

t y p e ”

SLIDE 39

U s i n g S i mu l a t e d E x e c u t i

n

i n V e r i f y i n g D i s t r i b u t e d A l g

r

i t h ms V MC A I 2 3 3 9

C

n

c l u s i

n
T

h e

r

e m p r

v

e r s n e e d l e m m a s a n d t a c t i c s

– E

x e c u t i

n

d a t a c a n p r

v

i d e s

m

e

f

b

t

h

L

e m m a s

– G

e n e r a l i z e

v

e r e x e c u t i

n

d a t a

C
n

j e c t u r e d i n v a r i a n t s

T

a c t i c s

– A

n n

t

a t i

n

s f

r

p a i r e d t e s t i n g p r

v

i d e s

P

r

f
u

t l i n e

E

x i s t e n t i a l w i t n e s s e s

C
n

t r i b u t i

n

: e a s i e r t

u

s e t h e

r

e m p r

v