[PPT] - Observation and Control 1 Observ ation and Con trol for PowerPoint Presentation

SLIDE 1 Observation and Control 1 Observ ation and Con trol for Debugging Distributed Computations Vija y K. Garg Electrical and Computer Engineering Departmen t Univ ersit y

f

T exas at Austin, Austin, TX 78712 http://maple.ece .utex as.e du/~v ijay / c Vija y K. Ga rg

SLIDE 2 Observation and Control 2 Ackno wledgment s

Collab
rato

rs

n

va rious ideas C. M. Chase, E. F romentin, R. Kilgo re, R. Kuma r, J. R. Mitchell, V. V. Murt y , M. T. Raghunath, M. Ra ynal, A. T a rafda r, A. I. T

mlinson,

and B. W aldeck er c Vija y K. Ga rg

SLIDE 3 Observation and Control 3 Outline

f

the talk

Intro

duction :

ur

mo del

Observation:

Main ideas

Lack
f

sha red clo ck

Lack
f

sha red memo ry

Combinato

rial Explosion

Observation:

Algo rithms

W

CP algo rithm, Channel p redicates

Detecting

regula r exp ressions

Control
Dela

ying events:

ine
Dela

ying events:

nline
Controlling
rder:
ine
Controlling
rder:
nline

c Vija y K. Ga rg

SLIDE 4 Observation and Control 4 Cha racteristics

f

Distributed Systems

Lack
f

sha red clo ck

rder
f

events pa rtial

Lack
f

sha red memo ry

meaning
f

global state

need

messages fo r

bserving

"global state"

Multiple

p ro cesses

Combinato

rial explosion

non-determin

ism c Vija y K. Ga rg

SLIDE 5 Observation and Control 5 Mo del

f

a Distributed Program

0,1 pc,x 1,3 2,3 x := x+2 send(x) x := x-1 pc,y y := y+3 receive(y) y:=2*y 0,1 1,4 2,3 3,2 3,6 r[1] r[2]

messages:

asynchronous, reliable, no FIF O assumption

no

sha red clo ck

r

memo ry

lo

cal states

Lamp
rt's

causally p recede relation, concurrency relation c Vija y K. Ga rg

SLIDE 6 Observation and Control 6 Motivation fo r Observation De ar Watson, you se e but you do not

bserve...
Distributed

Debugging, T esting

stop

when the p redicate q is true

p

redicate q = (P1 is in critical section) and (P2 is in critical section).

Detect

if the p rogram violates any inva riant

F

ault-tolerance

Monito

ring while the p rogram is

p

erationa l

Distributed

Active Rules

On

global condition p, trigger rule a

General

pa radigm fo r

bserving

Distributed Algo rithms

T

ermination detection, deadlo ck detection, loss

f

tok en c Vija y K. Ga rg

SLIDE 7 Observation and Control 7 Lack

f

sha red clo ck

Problem:

dene truthness

f

the p redicate C S 1 ^ C S 2

based
n

real time

based
n

causalit y

Real-time

considered ha rmful in distributed system.

My

clo ck synchronizati

n

algo rithm achieves 10 ms

p

rograms should w

rk

indep ende nt

f

p ro cesso r sp eeds

Reject

linea r time, accept vecto r time

Lamp
rt

78, Fidge 89, Mattern 89

Simultanei

t y vs Concurrency c Vija y K. Ga rg

SLIDE 8 Observation and Control 8 Clo ck in a Distributed System

(0,0,0,2) (1,0,0,0) (0,0,1,0) P1 P2 P3 P4 (0,1,0,0) (0,2,0,0) (0,0,0,1) (2,1,0,0) (3,1,0,0) (0,0,2,1) (2,1,3,1) (2,3,3,1) (2,1,4,1)

Prop

ert y: s ! t i s:v < t:v . c Vija y K. Ga rg

SLIDE 9 Observation and Control 9 Lack

f

sha red state

C1

P3

C2

m1 m2 m3 P1 P2

consistent

global state

if

the receive

f

an event is reco rded, then send must b e reco rded c Vija y K. Ga rg

SLIDE 10 Observation and Control 10 Camera: Chandy and Lamp

rt's

Algo rithm

Algo

rithm to compute a snapshot

f

a computation: S

S
is

a p

ssible

global state in the computation

Stable

p redicate:

nce

true sta ys true

e.g.

terminati

n

detection, deadlo ck detection

T
monito

r stable p redicates: rep eatedly tak e the snapshots

Disadvantages
f

CL Algo rithm fo r p redicate detection

Not

useful fo r unstable p redicates

Do

es not return the rst cut

Ho

w

ften

should the snapshot b e tak en ?

Assumes

FIF O c Vija y K. Ga rg

SLIDE 11 Observation and Control 11 Unstable Predicates

t s0 s1 s2 s3 t0 t1 t2 t3 (0,0) 1 2 3 1 2 3 s

Multiple

timed executions consistent with

ne

run c Vija y K. Ga rg

SLIDE 12 Observation and Control 12 Tw

interp

retations

f

p redicates

Tw
mo

dalities: [Co

p

er and Ma rzullo 91], [Ga rg and W aldeck er 91]

P
ssibly:q

(also called w eak p redicates)

exists

a path from the initial state to the nal state along which q is true

n

some state

Denitely:q

(also called strong p redicates)

fo

r all paths from the initial state to the nal state q is true

n

some state c Vija y K. Ga rg

SLIDE 13 Observation and Control 13 Comm unicatio n Complexi t y

Consider

evaluation

f

the p redicate q (x 1 ; x 2 )

nly

P 1 kno ws all the values tak en b y x 1

nly

P 2 kno ws values tak en b y x 2

Is

q (x 1 ; x 2 ) true fo r some value

f

x 1 and x 2

Key

question: numb er

f

values that need to b e communi- cated

ne

value p er internal event,

r
ne

value p er external event c Vija y K. Ga rg

SLIDE 14 Observation and Control 14 Monotonicit y

Denition
Assume

x 1 tak es values from a totally

rdered

set

q

is monotone w.r.t. rst a rgument if 8a; b; x 2 : (a < b) ) (q (a; x 2 ) ) q (b; x 2 ))

Examples
q

= (x 1 > x 2 ): monotonic w.r.t x 1 and x 2

q

= l 1 ^ l 2 : monotonic

q

= (x 1 = x 2 ): not monotonic. c Vija y K. Ga rg

SLIDE 15 Observation and Control 15 Multiple Pro cesses

Intractabilit

y

f

the Global Predicate Detection Problem

Giv

en: an execution S

f

N p ro cesses, N va riables x 1 ; : : : ; x N , and a p redicate q dened

n

x.

Is

there a consistent cut G 2 S such that q (G) is true.

Theo

rem [Chase and Ga rg 95]: The p redicate detection p roblem is NP-Complete.

Pro
f:

By reduction from SA T ((x 1 _

x

2 _ x 3 ) ^ (

x

1 _ x 2 ) ^ : : :)

1 x1 x2 x3

c Vija y K. Ga rg

SLIDE 16 Observation and Control 16 Linea rit y

H G

F
rbidden

p redicate: fo rbidden (G,i) i

8H

: G

H

: (G[i] = H [i]) ) :q (H )

Predicate

q is linea r w.r.t. a computation S if

8G

: :q (G) ) 9i : fo rbidden (G; i)

Examples
l

1 ^ l 2 ^ ::: ^ l n

x

+ y

k

, x is non-increasin g

channel

is empt y c Vija y K. Ga rg

SLIDE 17 Observation and Control 17 Summa ry

f

Observation: Problems and Solutions

Cha

racteristic Problem Idea Bonus No sha red clo ck

rdering

events causalit y avoid race erro rs No sha red memo ry message/state change monotonicit y extremal functions multiple p ro cesses combinato ria l explosion linea rit y rst cut c Vija y K. Ga rg

SLIDE 18 Observation and Control 18 Co

p

er and Ma rzullo's Algo rithm

P
ssibly:p
construct

the lattice

f

global states, check each global state fo r truthness

f

p

Denitely:p
fo

r all paths from the initial state to the nal state p is true

n

some state

construct

the lattice

f

global states

remove

states satisfying p

Is

last state reachable from the initial state

Complexit

y: O (k n ) where

k

: Numb er

f

lo cal states p er p ro cess

n:

Numb er

f

p ro cesses c Vija y K. Ga rg

SLIDE 19 Observation and Control 19 W eak Conjunctive Predicates

W

CP

P
ssibly:l

1 ^ l 2 ^ : : : ^ l n

useful

fo r bad

r

undesirable p redicates

Example:

the classical mutual exclusion p roblem.

Example:

(John is sleeping) and (Ma ry is sleeping) and (Rob ert is sleeping)

detect

erro rs that ma y b e hidden in some run due to race conditions. c Vija y K. Ga rg

SLIDE 20 Observation and Control 20 Imp

rtance
f

W eak Conjunctive Predicates

Sucient

fo r detection

f

any b

lean

exp ression

which

can b e exp ressed as a disjunction

f

a small numb er

f

con- junctions.

Example

x; y and z a re in three dierent p ro cesses. Then, ev en(x) ^ ((y < 0) _ (z > 6))

(ev

en(x) ^ (y < 0))_ (ev en(x) ^ (z > 6))

the

global p redicate is satised b y

nly

a nite numb er

f

p

ssible

global states.

Example,

x and y a re in dierent p ro cesses.

(x

= y ) is not a lo c al p redicate c Vija y K. Ga rg

SLIDE 21 Observation and Control 21 Conditions fo r W eak Conjunctive Predicates

local predicate is false local predicate is true Predicate is true on this cut

P
ssibly

(l 1 ^ l 2 ^ : : : l n ) is true i there exist s i in P i such that l i is true in state s i , and s i and s j a re incompa rable fo r distinct i; j .

Key

p roblems and solutions

numb

er

f

states satisfying lo cal p redicates ma y b e la rge: Use monotonicit y (at most

ne

state p er message)

combinato

ria l explosion when combining them together: Use Linea rit y c Vija y K. Ga rg

SLIDE 22 Observation and Control 22 W eak Conjunctive Predicates: Centralized Algo rithm

Application Process 1 Application Application Process 2 Process n queue 1 queue n Process Checker Checker

Each

non-check er p ro cess maintains its lo cal v ector

send

to the check er p ro cess the vecto r clo ck whenever

lo

cal p redicate is true

at

most

nce

in each message interval.

Optimizati
n:

Sucient to send the vecto r

nce

after each message is sent c Vija y K. Ga rg

SLIDE 23 Observation and Control 23 Check er Pro cess

local predicate is false local predicate is true Predicate is true on this cut

Steps
Begin

with the initial global state

Eliminate

any state that happ ened b efo re any

ther

state along the current cut.

Predicate

true fo r the rst time

if

no states can b e eliminated.

Predicate

false

if

w e eliminate the nal state from any p ro cess c Vija y K. Ga rg

SLIDE 24 Observation and Control 24 Overhead: Non-check er p ro cesses

Space

complexit y

the

a rra y v ector : O (n).

message

complexit y

O

(m s ) where m s is the numb er

f

p rogram messages sent.

In

addition, p rogram messages have to include time vecto rs.

Time

complexit y

detection
f

lo cal p redicates

maintain

vecto r clo ck (O (n)=message ). c Vija y K. Ga rg

SLIDE 25 Observation and Control 25 Overhead: Check er p ro cesses

Space

complexit y

n

queues, each containing at most m vecto rs

Time

complexit y

The

algo rithm fo r check er requires at most O (n 2 m) compa risons.

Any

algo rithm which determines whether there exists a set

f

incompa rable vecto rs

f

size n in n chains

f

size at most m, mak es at least mn(n

1)=2

compa risons. [Ga rg and W aldeck er 94] c Vija y K. Ga rg

SLIDE 26 Observation and Control 26 Disadvantages

f

ab

ve

algo rithm

Centralized
Check

er p ro cess ma y b ecome a b

ttleneck
Space

requirements

Queues

at the check er p ro cess ma y gro w la rge

Message

complexit y

ma

y result in to

many

additional messages c Vija y K. Ga rg

SLIDE 27 Observation and Control 27 Other W CP algo rithms

tok

en based algo rithm [Ga rg and Chase 95]

eliminate

centraliz ed check er p ro cess

Completely

distributed algo rithm [Ga rg and Chase 95]

Uses

Scholten and Dijkstra's termination detection

Distributed

Oine-algo rithm [V enk atesan and Dathan 92]

assume

FIF O and

-line
Keeping

queues sho rter [Chiou and Ko rfhage 95]

eliminate

vecto rs that a re useless

Avoiding

control messages[Hurn, Mizuno et al 96]

piggyback

info/tok en with applicati

n

messages c Vija y K. Ga rg

SLIDE 28 Observation and Control 28 Channel Predicates: Observing hallw a ys

Many

p rop erties require channels

termination

detection: all p ro cesses a re idle and all channels a re empt y

A

channel p redicate: a b

lean

function

n

the state

f

the channel

uni-directi
na

l

memo

ryless. i.e. channel state = sequence

f

messages sent

set
f

messages received

Linea

rit y: Given any channel state in which the p redicate is false, then

cannot

b e made true b y sending mo re messages without receiving any messages,

r
cannot

b e made true b y receiving mo re messages without sending any messages. c Vija y K. Ga rg

SLIDE 29 Observation and Control 29 Linea r Channel Predicates

Empt

y channels

If

false, then it cannot b e made true b y sending mo re messages,

Channel

has exactly three red messages

If

less than three, then it cannot b e made true b y receiving mo re messages,

If

mo re than three, then it cannot b e made true b y sending mo re messages, c Vija y K. Ga rg

SLIDE 30 Observation and Control 30 Non-linea r Channel Predicates

Channel

has an

dd

numb er

f

messages

C[1] D[1] D[2] C[2]

Key

result: linea rit y = rst cut is w ell dened. c Vija y K. Ga rg

SLIDE 31 Observation and Control 31 Relational Predicates

k

tok ens co rresp

nding

to k resources in the system

x

i : numb er

f

tok en at P i

x

i < k : loss

f

tok ens

x

i > k : License violation p roblem

Predicate,

global function

9G

: consistent (G) : P s i 2G s i :x i < K

min

G : consistent (G) : P s i 2G s i :x i

Ideas:
max-o

w technique: [Chase and Ga rg 95]

Matrix

clo cks: detect p redicate

f

the fo rm x 1 + x 2 < k [T

mlinson

and Ga rg 93]

Use

Dilw

rth's

theo rem: [T

mlinson

and Ga rg 96] c Vija y K. Ga rg

SLIDE 32 Observation and Control 32 Other Algo rithm s

Conjunction
f

global p redicates

Example:(x

1 = x 2 ) ^ (x 3 > x 4 ) Stoller and Scneider 95, Ga rg and Mitchell 96

Notion
f

xed set [Stoller and Scneider 95]

set
f

va riables such that

n

xing them w e get a W CP

x

x 1 = 4 and x 4 = 6, w e get (4 = x 2 ) ^ (x 3 > 6)

evaluate

all W CP

btained

b y using all values

f

xed-set.

Denitely

T rue p redicates

strong

conjunctive p redicates [Ga rg and W aldeck er 93] c Vija y K. Ga rg

SLIDE 33 Observation and Control 33 Causal Predicates

Predicate

based

n

control

w
useful

fo r exp ressing and

bserving

the

w
f

info rmation.

Ea

rly w

rk
sequence
f

lo cal p redicates [Miller and Choi 88]

l

1 ! l 2 ! : : : l m .

regula

r exp ression

f

lo cal p redicates [F romentin, Ra ynal, Ga rg, T

mlinson

94] c Vija y K. Ga rg

SLIDE 34 Observation and Control 34 Detection

f

Regula r Exp ression

Example
f

a regula r exp ression ?

a

+ cb

c
a

regula r exp ression is true in a run i there exists a path in the run (p

set)

which matches the exp ression

Complexit

y

f

p roblem

Many

states

Many

paths p er state

Many

strings p er path c Vija y K. Ga rg

SLIDE 35 Observation and Control 35 Algo rithm

Regula

r exp ression: a + cb

c
convert

it to non-deterministic nite state machine (fsm)

simulate

it during the execution (piggybackin g state

f

the fsm)

k

eep z[1..m] with each p ro cess

z[i]

= 1 i there exists a causal path that tak es the fsm to state i.

initial state q3 q2 q1 c b a c

Dene
ne

bit fo r each state z 1 := init z 2 := (c ^

z

1 ) _ (b ^

z

2 ) z 3 := (a ^

z

1 ) _ (c ^

z

2 ) c Vija y K. Ga rg

SLIDE 36 Observation and Control 36 Other App roaches

D

A G patterns

f

lo cal p redicates [Ga rg, T

mlinson,

F romentin, Ra ynal 95]

A

tomic Sequences [Hurn, Plouzeau, Ra ynal 93]

l

i [r i ]l i+1

r

i do es not

ccur

b et w een l i and l i+1

Dynamic

Prop erties [Babaoglu and Ra ynal 95]

Generaliz

ati

n
f

atomic sequences

Event

No rmal F

rm

[Chiou and Ko rfhage 94]

sequences
f

conjunctive p redicates

Recursive

P

set

Logic [T

mlinson

and Ga rg 95]

Recursive

combination

f

sequencin g, conjunction, and linea r p redicates c Vija y K. Ga rg

SLIDE 37 Observation and Control 37 Motivation fo r Control Who c

ntr
ls

the p ast c

ntr
ls

the futur e, who c

ntr
ls

the pr esent c

ntr
ls

the p ast... Ge

r

ge Orwel l, Ninete en Eighty-F

ur.
maintain

global inva riants

r

p rop er

rder
f

events

Examples:

Distribute d Debugging

ensure

that busy 1 _ busy 2 is alw a ys true

ensure

that m 1 is delivered b efo re m 2

Resource

Allo cation

maintain

:C S 1 _ :C S 2

F

ault tolerance

On

fault, rollback and execute under control

Adaptive

p

licies
p

ro cedure A (B) b etter under light (heavy) load c Vija y K. Ga rg

SLIDE 38 Observation and Control 38 Mo dels fo r Control

Is

the future kno wn ?

Y

es:

ine

control

applications

in distributed debugging, recovery , fault tolerance..

No:
nline

control

applications:

global synchronization, resource allo cation

Dela

ying events vs Changing

rder
f

events

sup

erviso r simply adds dela y b et w een events

sup

erviso r changes

rder
f

events c Vija y K. Ga rg

SLIDE 39 Observation and Control 39 Dela ying events: Oine control

P1 P0

Maintain

at least

ne
f

the p ro cess is not red

Can

add additional a rro ws in the diagram

the

control relation should not interfere with existing causalit y relation

therwise,

the system deadlo cks c Vija y K. Ga rg

SLIDE 40 Observation and Control 40 Dela ying events: Oine control

P0 P1 P0 P1

Problem:
Instance:

Given a computation and a b

lean

exp ression q

f

lo cal p redicates

Question:

Is there a non-interfer in g control relation that maintains q

This

p roblem is NP-complete [T a rafda r and Ga rg 97] c Vija y K. Ga rg

SLIDE 41 Observation and Control 41 Dela ying events: disjunctive p redicates

P1 P0

Ecient

algo rithm fo r disjunctive p redicates

Example:

at least

ne
f

the philosopher do es not have a fo rk

Result:

a control strategy exists i there is no set

f
verlapping

false intervals

v

er l ap(I 1 ; I 2 ) = (I 1 :l

!

I 2 :hi) ^ (I 2 :l

!

I 1 :hi)

Result:

There exists an O (n 2 m) algo rithm to determine the strategy

n

= numb er

f

p ro cesses

m

= numb er

f

states p er p ro cess c Vija y K. Ga rg

SLIDE 42 Observation and Control 42 Dela ying events: Online control

Only

the past is kno wn

deadlo

ck is imp

ssible

to avoid

P1 P0

Assume:

a p ro cess cannot blo ck when its lo cal p redicate is false

maintaini

ng l 1 _ l 2 _ ::: _ l n is equivalent to n

1

mutual exclusion p roblem

in

CS = lo cal p redicate false

i.e.,

all n p ro cesses cannot b e in the CS

can

b e solved using tok en which is a liabilit y rather than p rivilege c Vija y K. Ga rg

SLIDE 43 Observation and Control 43 Controlling

rder:

Oine control

Problem:

Given a computation enfo rce an

rder
f

messages in a rep eated run

Same
rder
Repla

y

f

distributed execution (distributed debugging)

need

to sto re messages

r

message

rder
Dierent
rder
T

esting

f

a distributed p rogram [Kilgo re, Chase 97]

Recovery
f

a distributed p rogram

can

change the

rder
f

t w

indep

end en t messages

the

computation ma y change after rst reo rder c Vija y K. Ga rg

SLIDE 44 Observation and Control 44 Controlling

rder:

Online control

Simple

example: FIF O

rdering
f

messages

External

events:

invo

cation

f

a message

send
f

a message

receive
f

a message

delivery
f

a message

constraints
n

sup erviso r

invo

cation and receive events a re uncontrolla bl e

liveness

requiremen t

if
nly

events p

ssible

a re send and delivery then at least

ne

must b e enabled. c Vija y K. Ga rg

SLIDE 45 Observation and Control 45 Limitati

ns
f

Online Sup ervision

Sp

ecication: set

f

computation p

ssible

with a xed set

f

messages

Question:

Is there a control strategy to meet the sp ecication ?

Assumption:

Sup erviso r can send control messages and tag user messages

Control

p

ssible

i sp ecs include all synchronously

rder

computations [Murt y and Ga rg 97]

Assumption:

Sup erviso r can

nly

tag user messages

control

p

ssible

i sp ecs include all causally

rdered

computations [Murt y and Ga rg 97] c Vija y K. Ga rg

SLIDE 46 Observation and Control 46 Online sup ervision: Algo rithms

F
rbidden

p redicate [Murt y and Ga rg 97]

sub-structure

that is not allo w ed in the computation

Example

1: Causal

rdering
9x;

y : (x:s ! y :s) ^ (y :r ! x:r )

Example

2: Lo cal fo rw a rd ush channels

(pr
cess(x:s)

= pr

cess(y

:s)) ^ (pr

cess(x:r

) = pr

cess(y

:r ) ^ (col

r

(x) = r ed) ^ (x:s ! y :s) ^ (y :r ! x:r )

There

exists an algo rithm with

input:

a fo rbidden p redicate

utput:

either "not p

ssible",
r

a p roto col to meet sp ecs c Vija y K. Ga rg

SLIDE 47 Observation and Control 47 Applications to Distributed Debugging

Additional

command

do

action when c

ndition
Also

assume run and r erun

Conditions
b
lean

p redicate

n

the global state

requirement
f

(semi)-linea rit y

regula

r exp ression

Actions
stop

pids

p

rint exp ressions

maintain

b

lean

p redicate

maintain
rder-exp

ression c Vija y K. Ga rg

SLIDE 48 Observation and Control 48 Summa ry

Observation
Use

causalit y instead

f

time to dene "and"

Use

monotonicit y to reduce communicati

n

complexit y

Global
bservation

is quite ecient fo r many p ractical cases

linea

rit y fo r b

lean

p redicates

regula

r exp ressions

f

lo cal p redicates

Control
desirable

fo r many applicati

ns
ine

vs

nline

has implicati

n

s

n

limitations

dela

y vs change

f
rder

mo del c Vija y K. Ga rg

SLIDE 49 Observation and Control 49 F uture W

rk
Predicate

detection under fault y environment

p

ro cesses, channels

r

messages ma y fail

messages

from dierent inca rnati

ns
Mo

re complex mo del

f

control

plant

va riables vs control va riables

unobservable

events, uncontrollab l e events c Vija y K. Ga rg