[PPT] - Quantitative Analysis of Randomized Distributed Systems and PowerPoint Presentation

SLIDE 1

Quantitative Analysis of Randomized Distributed Systems and Probabilistic Automata Christel Baier Technische Universit¨ at Dresden joint work with Nathalie Bertrand Frank Ciesinski Marcus Gr¨

ßer

1 / 124

SLIDE 2

Probability elsewhere

int-01

randomized algorithms

[Rabin 1960]

breaking symmetry, fingerprints, input sampling, . . . . . . . . .

stochastic control theory

[Bellman 1957]

perations research
performance modeling

[Markov, Erlang, Kolm., ∼ ∼ ∼ 1900]

emphasis on steady-state and transient measures

biological systems, resilient systems, security protocols

. . . . . . . . .

2 / 124

SLIDE 3

Probability elsewhere

int-01

randomized algorithms

[Rabin 1960]

breaking symmetry, fingerprints, input sampling, . . . . . . . . . models: discrete-time Markov chains Markov decision processes

stochastic control theory

[Bellman 1957]

perations research

models: Markov decision processes

performance modeling

[Markov, Erlang, Kolm., ∼ ∼ ∼ 1900]

emphasis on steady-state and transient measures models: continuous-time Markov chains

biological systems, resilient systems, security protocols

. . . . . . . . .

3 / 124

SLIDE 4

Model checking

int-02

requirements (safety, liveness) specification, e.g., temporal formula Φ Φ Φ reactive system abstract model M M M model checking “does M | = Φ M | = Φ M | = Φ hold ?” no yes

4 / 124

SLIDE 5

Probabilistic model checking

int-03

quantitative requirements specification, e.g., temporal formula Φ Φ Φ probabilistic reactive system probabilistic model M M M probabilistic model checking “does M | = Φ M | = Φ M | = Φ hold ?” probability for “bad behaviors” is < 10−6 < 10−6 < 10−6 probability for “good behaviors” is 1 1 1 expected costs for ....

5 / 124

SLIDE 6

Probabilistic model checking

int-03

quantitative requirements linear temporal formula Φ Φ Φ (path event) probabilistic reactive system Markov decision process M M M probabilistic model checking quantitative analysis of M M M against Φ Φ Φ probability for “bad behaviors” is < 10−6 < 10−6 < 10−6 probability for “good behaviors” is 1 1 1

6 / 124

SLIDE 7

Outline

verview
Markov decision processes (MDP) and

quantitative analysis against path events

partial order reduction for MDP
partially-oberservable MDP
conclusions

7 / 124

SLIDE 8

Markov decision process (MDP)

mdp-01

perational model with nondeterminism and probabilism

8 / 124

SLIDE 9

Markov decision process (MDP)

mdp-01

perational model with nondeterminism and probabilism
modeling randomized distributed systems

by interleaving s s s

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2

process 1 1 1 tosses a coin process 2 2 2 tosses a coin process 1 1 1 tosses a coin process 2 2 2 tosses a coin

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2

9 / 124

SLIDE 10

Markov decision process (MDP)

mdp-01

perational model with nondeterminism and probabilism
modeling randomized distributed systems

by interleaving

nondeterminism useful for abstraction, underspec.,

modeling interactions with an unkown environment s s s

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2

process 1 1 1 tosses a coin process 2 2 2 tosses a coin process 1 1 1 tosses a coin process 2 2 2 tosses a coin

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2

10 / 124

SLIDE 11

Markov decision process (MDP)

mdp-02-r

M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .)

11 / 124

SLIDE 12

Markov decision process (MDP)

mdp-02-r

M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .)

finite state space S

S S

12 / 124

SLIDE 13

Markov decision process (MDP)

mdp-02-r

M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .)

finite state space S

S S

Act

Act Act finite set of actions

13 / 124

SLIDE 14

Markov decision process (MDP)

mdp-02-r

M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .)

finite state space S

S S

Act

Act Act finite set of actions

P : S × Act × S → [0, 1]

P : S × Act × S → [0, 1] P : S × Act × S → [0, 1] s.t.

s′∈S

P(s, α, s′)

s′∈S

P(s, α, s′)

s′∈S

P(s, α, s′) = 1 = 1 = 1 s s s α α α β β β

1 4 1 4 1 4 3 4 3 4 3 4 1 2 1 2 1 2 1 6 1 6 1 6 1 3 1 3 1 3

nondeterministic choice probabilistic choice

14 / 124

SLIDE 15

Markov decision process (MDP)

mdp-02-r

M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .)

finite state space S

S S

Act

Act Act finite set of actions

P : S × Act × S → [0, 1]

P : S × Act × S → [0, 1] P : S × Act × S → [0, 1] s.t. ∀s ∈ S ∀s ∈ S ∀s ∈ S ∀α ∈ Act. . . ∀α ∈ Act. . . ∀α ∈ Act. . .

s′∈S

P(s, α, s′)

s′∈S

P(s, α, s′)

s′∈S

P(s, α, s′) ∈ {0, 1} ∈ {0, 1} ∈ {0, 1} s s s α α α β β β

1 4 1 4 1 4 3 4 3 4 3 4 1 2 1 2 1 2 1 6 1 6 1 6 1 3 1 3 1 3

nondeterministic choice probabilistic choice ր ր ր α / ∈ Act(s) α / ∈ Act(s) α / ∈ Act(s) տ տ տ α ∈ Act(s) α ∈ Act(s) α ∈ Act(s) Act(s) = Act(s) = Act(s) = set of actions that are enabled in state s s s

15 / 124

SLIDE 16

Markov decision process (MDP)

mdp-02-r

M = (S, Act, P, s0, AP, L, rew, . . .) M = (S, Act, P, s0, AP, L, rew, . . .) M = (S, Act, P, s0, AP, L, rew, . . .)

finite state space S

S S

Act

Act Act finite set of actions

P : S × Act × S → [0, 1]

P : S × Act × S → [0, 1] P : S × Act × S → [0, 1] s.t. ∀s ∈ S ∀s ∈ S ∀s ∈ S ∀α ∈ Act. . . ∀α ∈ Act. . . ∀α ∈ Act. . .

s′∈S

P(s, α, s′)

s′∈S

P(s, α, s′)

s′∈S

P(s, α, s′) ∈ {0, 1} ∈ {0, 1} ∈ {0, 1}

s0

s0 s0 initial state

AP

AP AP set of atomic propositions

labeling L : S → 2AP

L : S → 2AP L : S → 2AP

reward function rew : S × Act → R

rew : S × Act → R rew : S × Act → R ր ր ր α / ∈ Act(s) α / ∈ Act(s) α / ∈ Act(s) տ տ տ α ∈ Act(s) α ∈ Act(s) α ∈ Act(s)

16 / 124

SLIDE 17

Randomized mutual exclusion protocol

mdp-05

2

2 2 concurrent processes P1 P1 P1, P2 P2 P2 with 3 3 3 phases: ni ni ni noncritical actions of process Pi Pi Pi wi wi wi waiting phase of process Pi Pi Pi ci ci ci critical section of process Pi Pi Pi

competition of both processes are waiting

17 / 124

SLIDE 18

Randomized mutual exclusion protocol

mdp-05

2

2 2 concurrent processes P1 P1 P1, P2 P2 P2 with 3 3 3 phases: ni ni ni noncritical actions of process Pi Pi Pi wi wi wi waiting phase of process Pi Pi Pi ci ci ci critical section of process Pi Pi Pi

competition of both processes are waiting
resolved by a randomized arbiter who tosses a coin

18 / 124

SLIDE 19

Randomized mutual exclusion protocol

mdp-05

n1n2 n1n2 n1n2 w1n2 w1n2 w1n2 n1w2 n1w2 n1w2 w1w2 w1w2 w1w2 c1n2 c1n2 c1n2 n1c2 n1c2 n1c2 c1w2 c1w2 c1w2 w1c2 w1c2 w1c2 MDP

r e q u e s t

2

r e q u e s t

2

r e q u e s t

2

release1 release1 release1 e n t e r

1

e n t e r

1

e n t e r

1

request1 request1 request1 request2 request2 request2 request2 request2 request2 request1 request1 request1 e n t e r

2

e n t e r

2

e n t e r

2

r e q u e s t

1

r e q u e s t

1

r e q u e s t

1

release2 release2 release2 toss a toss a toss a coin coin coin r e l e a s e2 r e l e a s e2 r e l e a s e2 r e l e a s e1 r e l e a s e1 r e l e a s e1

1 2 1 2 1 2 1 2 1 2 1 2

interleaving of the request operations
competition if both processes are waiting
randomized arbiter tosses a coin if both are waiting

19 / 124

SLIDE 20

Randomized mutual exclusion protocol

mdp-05

n1n2 n1n2 n1n2 w1n2 w1n2 w1n2 n1w2 n1w2 n1w2 w1w2 w1w2 w1w2 c1n2 c1n2 c1n2 n1c2 n1c2 n1c2 c1w2 c1w2 c1w2 w1c2 w1c2 w1c2 n1n2 n1n2 n1n2 w1n2 w1n2 w1n2 n1w2 n1w2 n1w2 w1w2 w1w2 w1w2 MDP

r e q u e s t

2

r e q u e s t

2

r e q u e s t

2

release1 release1 release1 e n t e r

1

e n t e r

1

e n t e r

1

request1 request1 request1 request2 request2 request2 request2 request2 request2 request1 request1 request1 e n t e r

2

e n t e r

2

e n t e r

2

r e q u e s t

1

r e q u e s t

1

r e q u e s t

1

release2 release2 release2 toss a toss a toss a coin coin coin r e l e a s e2 r e l e a s e2 r e l e a s e2 r e l e a s e1 r e l e a s e1 r e l e a s e1

1 2 1 2 1 2 1 2 1 2 1 2

interleaving of the request operations
competition if both processes are waiting
randomized arbiter tosses a coin if both are waiting

20 / 124

SLIDE 21

Randomized mutual exclusion protocol

mdp-05

n1n2 n1n2 n1n2 w1n2 w1n2 w1n2 n1w2 n1w2 n1w2 w1w2 w1w2 w1w2 c1n2 c1n2 c1n2 n1c2 n1c2 n1c2 c1w2 c1w2 c1w2 w1c2 w1c2 w1c2 w1w2 w1w2 w1w2 MDP

r e q u e s t

2

r e q u e s t

2

r e q u e s t

2

release1 release1 release1 e n t e r

1

e n t e r

1

e n t e r

1

request1 request1 request1 request2 request2 request2 request2 request2 request2 request1 request1 request1 e n t e r

2

e n t e r

2

e n t e r

2

r e q u e s t

1

r e q u e s t

1

r e q u e s t

1

release2 release2 release2 toss a toss a toss a coin coin coin r e l e a s e2 r e l e a s e2 r e l e a s e2 r e l e a s e1 r e l e a s e1 r e l e a s e1

1 2 1 2 1 2 1 2 1 2 1 2

interleaving of the request operations
competition if both processes are waiting
randomized arbiter tosses a coin if both are waiting

21 / 124

SLIDE 22

Randomized mutual exclusion protocol

mdp-05

n1n2 n1n2 n1n2 w1n2 w1n2 w1n2 n1w2 n1w2 n1w2 w1w2 w1w2 w1w2 c1n2 c1n2 c1n2 n1c2 n1c2 n1c2 c1w2 c1w2 c1w2 w1c2 w1c2 w1c2 w1w2 w1w2 w1w2 c1w2 c1w2 c1w2 w1c2 w1c2 w1c2 MDP

r e q u e s t

2

r e q u e s t

2

r e q u e s t

2

release1 release1 release1 e n t e r

1

e n t e r

1

e n t e r

1

request1 request1 request1 request2 request2 request2 request2 request2 request2 request1 request1 request1 e n t e r

2

e n t e r

2

e n t e r

2

r e q u e s t

1

r e q u e s t

1

r e q u e s t

1

release2 release2 release2 toss a toss a toss a coin coin coin r e l e a s e2 r e l e a s e2 r e l e a s e2 r e l e a s e1 r e l e a s e1 r e l e a s e1

1 2 1 2 1 2 1 2 1 2 1 2

interleaving of the request operations
competition if both processes are waiting
randomized arbiter tosses a coin if both are waiting

22 / 124

SLIDE 23

Reasoning about probabilities in MDP

mdp-10

requires resolving the nondeterminism by schedulers

23 / 124

SLIDE 24

Reasoning about probabilities in MDP

mdp-10

requires resolving the nondeterminism by schedulers
a scheduler is a function D : S+ −

→ Act D : S+ − → Act D : S+ − → Act s.t. action D (s0 . . . sn) D (s0 . . . sn) D (s0 . . . sn) is enabled in state sn sn sn

24 / 124

SLIDE 25

Reasoning about probabilities in MDP

mdp-10

requires resolving the nondeterminism by schedulers
a scheduler is a function D : S+ −

→ Act D : S+ − → Act D : S+ − → Act s.t. action D (s0 . . . sn) D (s0 . . . sn) D (s0 . . . sn) is enabled in state sn sn sn

each scheduler induces an infinite Markov chain

MDP β β β γ γ γ α α α

1 3 1 3 1 3 2 3 2 3 2 3

σ σ σ δ δ δ . . . . . . . . . . . . . . . . . . . . . . . . . . .

2 3 2 3 2 3 1 3

α

1 3

α

1 3

α δ δ δ β β β γ γ γ σ σ σ α 2

3

α 2

3

α 2

3 1 3 1 3 1 3

σ σ σ

25 / 124

SLIDE 26

Reasoning about probabilities in MDP

mdp-10

requires resolving the nondeterminism by schedulers
a scheduler is a function D : S+ −

→ Act D : S+ − → Act D : S+ − → Act s.t. action D (s0 . . . sn) D (s0 . . . sn) D (s0 . . . sn) is enabled in state sn sn sn

each scheduler induces an infinite Markov chain




yields a notion of probability measure PrD PrD PrD

n measurable sets of infinite paths

26 / 124

SLIDE 27

Reasoning about probabilities in MDP

mdp-10

requires resolving the nondeterminism by schedulers
a scheduler is a function D : S+ −

→ Act D : S+ − → Act D : S+ − → Act s.t. action D (s0 . . . sn) D (s0 . . . sn) D (s0 . . . sn) is enabled in state sn sn sn

each scheduler induces an infinite Markov chain




yields a notion of probability measure PrD PrD PrD

n measurable sets of infinite paths

typical task: given a measurable path event E E E, ∗ ∗ ∗ check whether E E E holds almost surely, i.e., PrD(E) = 1 PrD(E) = 1 PrD(E) = 1 for all schedulers D D D

27 / 124

SLIDE 28

Reasoning about probabilities in MDP

mdp-10

requires resolving the nondeterminism by schedulers
a scheduler is a function D : S+ −

→ Act D : S+ − → Act D : S+ − → Act s.t. action D (s0 . . . sn) D (s0 . . . sn) D (s0 . . . sn) is enabled in state sn sn sn

each scheduler induces an infinite Markov chain




yields a notion of probability measure PrD PrD PrD

n measurable sets of infinite paths

typical task: given a measurable path event E E E, ∗ ∗ ∗ check whether E E E holds almost surely ∗ ∗ ∗ compute the worst-case probability for E E E, i.e., sup

D

PrD(E) sup

D

PrD(E) sup

D

PrD(E)

r

inf

D

PrD(E) inf

D

PrD(E) inf

D

PrD(E)

28 / 124

SLIDE 29

Quantitative analysis of MDP

mdp-15

given: MDP M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .) with initial state s0 s0 s0 ω ω ω-regular path event E E E, e.g., given by an LTL formula task: compute PrM

max(s0, E) = sup D

PrD(s0, E) PrM

max(s0, E) = sup D

PrD(s0, E) PrM

max(s0, E) = sup D

PrD(s0, E)

29 / 124

SLIDE 30

Quantitative analysis of MDP

mdp-15

given: MDP M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .) with initial state s0 s0 s0 ω ω ω-regular path event E E E, e.g., given by an LTL formula task: compute PrM

max(s0, E) = sup D

PrD(s0, E) PrM

max(s0, E) = sup D

PrD(s0, E) PrM

max(s0, E) = sup D

PrD(s0, E) method: compute xs = PrM

max(s, E)

xs = PrM

max(s, E)

xs = PrM

max(s, E) for all s ∈ S

s ∈ S s ∈ S via graph analysis and linear program

[Vardi/Wolper’86] [Courcoubetis/Yannakakis’88] [Bianco/de Alfaro’95] [Baier/Kwiatkowska’98]

30 / 124

SLIDE 31

probabilistic system “bad behaviors”

31 / 124

SLIDE 32

probabilistic system “bad behaviors” MDP M M M

32 / 124

SLIDE 33

probabilistic system “bad behaviors” MDP M M M LTL formula ϕ ϕ ϕ deterministic automaton A A A

33 / 124

SLIDE 34

probabilistic system “bad behaviors” MDP M M M LTL formula ϕ ϕ ϕ deterministic automaton A A A quantitative analysis in the product-MDP M × A M × A M × A PrM

max(s, ϕ)

PrM

max(s, ϕ)

PrM

max(s, ϕ) =

= = PrM×A

max

s, inits,

PrM×A

max

s, inits,

PrM×A

max

s, inits, acceptance
cond. of A

A A

34 / 124

SLIDE 35

probabilistic system “bad behaviors” MDP M M M LTL formula ϕ ϕ ϕ deterministic automaton A A A quantitative analysis in the product-MDP M × A M × A M × A PrM

max(s, ϕ)

PrM

max(s, ϕ)

PrM

max(s, ϕ) =

= = PrM×A

max

s, inits,

PrM×A

max

s, inits,

PrM×A

max

s, inits, ♦accEC

♦accEC ♦accEC

maximal probabilility

for reaching an accepting end component

35 / 124

SLIDE 36

probabilistic system “bad behaviors” MDP M M M LTL formula ϕ ϕ ϕ deterministic automaton A A A probabilistic reachability analysis in the product-MDP M × A M × A M × A linear program PrM

max(s, ϕ)

PrM

max(s, ϕ)

PrM

max(s, ϕ) =

= = PrM×A

max

s, inits,

PrM×A

max

s, inits,

PrM×A

max

s, inits, ♦accEC

♦accEC ♦accEC

maximal probabilility

for reaching an accepting end component

36 / 124

SLIDE 37

probabilistic system “bad behaviors” MDP M M M LTL formula ϕ ϕ ϕ deterministic automaton A A A probabilistic reachability analysis in the product-MDP M × A M × A M × A linear program PrM

max(s, ϕ)

PrM

max(s, ϕ)

PrM

max(s, ϕ) =

= = PrM×A

max

s, inits,

PrM×A

max

s, inits,

PrM×A

max

s, inits, ♦accEC

♦accEC ♦accEC

polynomial

in |M| · |A| |M| · |A| |M| · |A|

37 / 124

SLIDE 38

2exp probabilistic system “bad behaviors” MDP M M M LTL formula ϕ ϕ ϕ deterministic automaton A A A probabilistic reachability analysis in the product-MDP M × A M × A M × A linear program PrM

max(s, ϕ)

PrM

max(s, ϕ)

PrM

max(s, ϕ) =

= = PrM×A

max

s, inits,

PrM×A

max

s, inits,

PrM×A

max

s, inits, ♦accEC

♦accEC ♦accEC

polynomial

in |M| · |A| |M| · |A| |M| · |A|

38 / 124

SLIDE 39

state explosion problem probabilistic system “bad behaviors” MDP M M M LTL formula ϕ ϕ ϕ deterministic automaton A A A probabilistic reachability analysis in the product-MDP M × A M × A M × A linear program PrM

max(s, ϕ)

PrM

max(s, ϕ)

PrM

max(s, ϕ) =

= = PrM×A

max

s, inits,

PrM×A

max

s, inits,

PrM×A

max

s, inits, ♦accEC

♦accEC ♦accEC

polynomial

in |M| · |A| |M| · |A| |M| · |A|

39 / 124

SLIDE 40

Advanced techniques for PMC

por-01-cs

symbolic model checking with variants of BDDs

e.g., in PRISM

[Kwiatkowska/Norman/Parker]

state aggregation with bisimulation

e.g., in MRMC

[Katoen et al]

abstraction-refinement

e.g., in RAPTURE [d’Argenio/Jeannet/Jensen/Larsen] PASS

[Hermanns/Wachter/Zhang]

partial order reduction

e.g., in LiQuor

[Baier/Ciesinski/Gr¨

ßer]

40 / 124

SLIDE 41

Advanced techniques for PMC

por-01-cs

symbolic model checking with variants of BDDs

e.g., in PRISM

[Kwiatkowska/Norman/Parker]

randomized distributed algorithms, communication and multimedia protocols, power management, security, . . . . . . . . .

state aggregation with bisimulation

e.g., in MRMC

[Katoen et al]

abstraction-refinement

e.g., in RAPTURE [d’Argenio/Jeannet/Jensen/Larsen] PASS

[Hermanns/Wachter/Zhang]

partial order reduction

e.g., in LiQuor

[Baier/Ciesinski/Gr¨

ßer]

41 / 124

SLIDE 42

Advanced techniques for PMC

por-01-cs

symbolic model checking with variants of BDDs

e.g., in PRISM

[Kwiatkowska/Norman/Parker]

randomized distributed algorithms, communication and multimedia protocols, power management, security, . . . . . . . . .

state aggregation with bisimulation

e.g., in MRMC

[Katoen et al]

abstraction-refinement

e.g., in RAPTURE [d’Argenio/Jeannet/Jensen/Larsen] PASS

[Hermanns/Wachter/Zhang]

partial order reduction

e.g., in LiQuor

[Baier/Ciesinski/Gr¨

ßer]

42 / 124

SLIDE 43

Partial order reduction

por-02

technique for reducing the state space of concurrent systems

[Godefroid,Peled,Valmari, ca. 1990]

attempts to analyze a sub-system by identifying

“redundant interleavings”

explores representatives of paths that agree up to

the order of independent actions

43 / 124

SLIDE 44

Partial order reduction

por-02

technique for reducing the state space of concurrent systems

[Godefroid,Peled,Valmari, ca. 1990]

attempts to analyze a sub-system by identifying

“redundant interleavings”

explores representatives of paths that agree up to

the order of independent actions e.g., x := x+y x := x+y x := x+y

action α

α α

z := z+3

z := z+3 z := z+3

action β

β β has the same effect as α; β α; β α; β or β; α β; α β; α

44 / 124

SLIDE 45

Partial order reduction

por-02

technique for reducing the state space of concurrent systems

[Godefroid,Peled,Valmari, ca. 1990]

attempts to analyze a sub-system by identifying

“redundant interleavings”

explores representatives of paths that agree up to

the order of independent actions DFS-based on-the-fly generation of a reduced system for each expanded state s s s

choose an appropriate subset Ample(s)

Ample(s) Ample(s) of Act(s) Act(s) Act(s)

expand only the α

α α-successors of s s s for α ∈ Ample(s) α ∈ Ample(s) α ∈ Ample(s) (but ignore the actions in Act(s) \ Ample(s) Act(s) \ Ample(s) Act(s) \ Ample(s))

45 / 124

SLIDE 46

Ample-set method

[Peled 1993]

por-03

given: processes Pi Pi Pi of a parallel system P1. . .Pn

P1. . .Pn
P1. . .Pn

with transition system T = (S, Act, →, . . .) T = (S, Act, →, . . .) T = (S, Act, →, . . .) task:

n-the-fly generation of a sub-system Tr

Tr Tr s.t. (A1) stutter condition . . . . . . . . . (A2) dependency condition . . . . . . . . . (A3) cycle condition . . . . . . . . .

46 / 124

SLIDE 47

Ample-set method

[Peled 1993]

por-03

given: processes Pi Pi Pi of a parallel system P1. . .Pn

P1. . .Pn
P1. . .Pn

with transition system T = (S, Act, →, . . .) T = (S, Act, →, . . .) T = (S, Act, →, . . .) task:

n-the-fly generation of a sub-system Tr

Tr Tr s.t. (A1) stutter condition (A2) dependency condition (A3) cycle condition

π πr

π πr π πr by permutations of independent actions Each path π π π in T T T is represented by an “equivalent” path πr πr πr in Tr Tr Tr

47 / 124

SLIDE 48

Ample-set method

[Peled 1993]

por-03

given: processes Pi Pi Pi of a parallel system P1. . .Pn

P1. . .Pn
P1. . .Pn

with transition system T = (S, Act, →, . . .) T = (S, Act, →, . . .) T = (S, Act, →, . . .) task:

n-the-fly generation of a sub-system Tr

Tr Tr s.t. (A1) stutter condition (A2) dependency condition (A3) cycle condition

π πr

π πr π πr by permutations of independent actions Each path π π π in T T T is represented by an “equivalent” path πr πr πr in Tr Tr Tr

T

T T and Tr Tr Tr satisfy the same stutter-invariant events, e.g., next-free LTL formulas

48 / 124

SLIDE 49

Ample-set method for MDP

por-04

given: processes Pi Pi Pi of a probabilistic system P1. . .Pn

P1. . .Pn
P1. . .Pn

with MDP-semantics M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .) task:

n-the-fly generation of a sub-MDP Mr

Mr Mr s.t. Mr Mr Mr and M M M have the same extremal probabilities for stutter-invariant events

49 / 124

SLIDE 50

Ample-set method for MDP

por-04

given: processes Pi Pi Pi of a probabilistic system P1. . .Pn

P1. . .Pn
P1. . .Pn

with MDP-semantics M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .) task:

n-the-fly generation of a sub-MDP Mr

Mr Mr s.t. For all schedulers D D D for M M M there is a scheduler Dr Dr Dr for Mr Mr Mr s.t. for all measurable, stutter-invariant events E E E: PrD

M(E) = PrDr Mr(E)

PrD

M(E) = PrDr Mr(E)

PrD

M(E) = PrDr Mr(E)

Mr

Mr Mr and M M M have the same extremal probabilities for stutter-invariant events

50 / 124

SLIDE 51

Example: ample set method

por-08-new

s s s α α α β β β γ γ γ β β β γ γ γ α α α α α α δ δ δ δ δ δ

riginal system T

T T α α α independent from β β β and γ γ γ

51 / 124

SLIDE 52

Example: ample set method

por-08-new

s s s α α α β β β γ γ γ β β β γ γ γ α α α α α α δ δ δ δ δ δ β β β γ γ γ δ δ δ δ δ δ P1 P2 P1 P2 P1 P2 α α α

riginal system T

T T α α α independent from β β β and γ γ γ action α α α: x := 1 x := 1 x := 1 action δ δ δ: IF x > 0 x > 0 x > 0 THEN y := 1 y := 1 y := 1 FI

52 / 124

SLIDE 53

Example: ample set method

por-08-new

s s s α α α β β β γ γ γ β β β γ γ γ α α α α α α δ δ δ δ δ δ s s s β β β γ γ γ α α α α α α δ δ δ δ δ δ

riginal system T

T T reduced system Tr Tr Tr (A1)-(A3) are fulfilled α α α independent from β β β and γ γ γ

53 / 124

SLIDE 54

Example: ample set method fails for MDP

por-08-new

s s s

1 2 1 2 1 2

α α α

1 2 1 2 1 2

β β β γ γ γ β β β γ γ γ γ γ γ β β β α α α

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2

α α α δ δ δ δ δ δ s s s β β β γ γ γ α α α

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2

α α α δ δ δ δ δ δ

riginal MDP M

M M reduced MDP Mr Mr Mr (A1)-(A3) are fulfilled α α α independent from β β β and γ γ γ

54 / 124

SLIDE 55

Example: ample set method fails for MDP

por-08-new

s s s

1 2 1 2 1 2

α α α

1 2 1 2 1 2

β β β γ γ γ β β β γ γ γ γ γ γ β β β α α α

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2

α α α δ δ δ δ δ δ s s s β β β γ γ γ α α α

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2

α α α δ δ δ δ δ δ

riginal MDP M

M M reduced MDP Mr Mr Mr PrM

max(s, ♦green) = 1

PrM

max(s, ♦green) = 1

PrM

max(s, ♦green) = 1

♦ ♦ ♦ “eventually”

55 / 124

SLIDE 56

Example: ample set method fails for MDP

por-08-new

s s s

1 2 1 2 1 2

α α α

1 2 1 2 1 2

β β β γ γ γ β β β γ γ γ γ γ γ β β β α α α

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2

α α α δ δ δ δ δ δ s s s β β β γ γ γ α α α

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2

α α α δ δ δ δ δ δ

riginal MDP M

M M reduced MDP Mr Mr Mr PrM

max(s, ♦green) = 1

PrM

max(s, ♦green) = 1

PrM

max(s, ♦green) = 1 > 1 2 = PrMr max(s, ♦green)

>

1 2 = PrMr max(s, ♦green)

>

1 2 = PrMr max(s, ♦green)

56 / 124

SLIDE 57

Partial order reduction for MDP

por-09

extend Peled’s conditions (A1)-(A3) for the ample-sets (A1) stutter condition . . . . . . . . . (A2) dependency condition . . . . . . . . . (A3) cycle condition . . . . . . . . . (A4) probabilistic condition If there is a path s

β1

− →

β2

− → . . .

βn

− →

α

− → s

β1

− →

β2

− → . . .

βn

− →

α

− → s

β1

− →

β2

− → . . .

βn

− →

α

− → in M M M s.t. β1, . . ., βn, α / ∈ Ample(s) β1, . . . , βn, α / ∈ Ample(s) β1, . . . , βn, α / ∈ Ample(s) and α α α is probabilistic then |Ample(s)| = 1 |Ample(s)| = 1 |Ample(s)| = 1.

57 / 124

SLIDE 58

Partial order reduction for MDP

por-09

extend Peled’s conditions (A1)-(A3) for the ample-sets (A1) stutter condition . . . . . . . . . (A2) dependency condition . . . . . . . . . (A3) cycle condition . . . . . . . . . (A4) probabilistic condition If there is a path s

β1

− →

β2

− → . . .

βn

− →

α

− → s

β1

− →

β2

− → . . .

βn

− →

α

− → s

β1

− →

β2

− → . . .

βn

− →

α

− → in M M M s.t. β1, . . ., βn, α / ∈ Ample(s) β1, . . . , βn, α / ∈ Ample(s) β1, . . . , βn, α / ∈ Ample(s) and α α α is probabilistic then |Ample(s)| = 1 |Ample(s)| = 1 |Ample(s)| = 1. If (A1)-(A4) hold then M M M and Mr Mr Mr have the same extremal probabilities for all stutter-invariant properties.

58 / 124

SLIDE 59

Probabilistic model checking

por-ifm-32

quantitative requirements LTL\

\ \ formula ϕ

ϕ ϕ (path event) probabilistic system Markov decision process M M M quantitative analysis

f M

M M against ϕ ϕ ϕ maximal/minimal probability for ϕ ϕ ϕ

59 / 124

SLIDE 60

Probabilistic model checking, e.g., LiQuor

por-ifm-32

quantitative requirements LTL\

\ \ formula ϕ

ϕ ϕ (path event) modeling language

P1. . .Pn
P1. . .Pn
P1. . .Pn

reduced MDP Mr Mr Mr partial order reduction quantitative analysis

f Mr

Mr Mr against ϕ ϕ ϕ maximal/minimal probability for ϕ ϕ ϕ

60 / 124

SLIDE 61

Probabilistic model checking, e.g., LiQuor

por-ifm-32a

quantitative requirements LTL\

\ \ formula ϕ

ϕ ϕ (path event) modeling language

P1. . .Pn
P1. . .Pn
P1. . .Pn

reduced MDP Mr Mr Mr partial order reduction quantitative analysis

f Mr

Mr Mr against ϕ ϕ ϕ maximal/minimal probability for ϕ ϕ ϕ worst-case analysis

61 / 124

SLIDE 62

Example: extremal scheduler for MDP

por-08-copy

s s s

1 2 1 2 1 2

α α α

1 2 1 2 1 2

β β β γ γ γ β β β γ γ γ γ γ γ β β β α α α

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2

α α α δ δ δ δ δ δ

riginal MDP M

M M α α α independent from β β β and γ γ γ

62 / 124

SLIDE 63

Example: extremal scheduler for MDP

por-08-copy

s s s

1 2 1 2 1 2

α α α

1 2 1 2 1 2

β β β γ γ γ β β β γ γ γ γ γ γ β β β α α α

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2

α α α δ δ δ δ δ δ β β β γ γ γ δ δ δ δ δ δ P1 P2 P1 P2 P1 P2 α α α

1 2 1 2 1 2 1 2 1 2 1 2

riginal MDP M

M M α α α independent from β β β and γ γ γ action α α α: x := random(0, 1) x := random(0, 1) x := random(0, 1) action δ δ δ: IF x > 0 x > 0 x > 0 THEN y := 1 y := 1 y := 1 FI

63 / 124

SLIDE 64

Example: extremal scheduler for MDP

por-08-copy

s s s

1 2 1 2 1 2

α α α

1 2 1 2 1 2

β β β γ γ γ β β β γ γ γ γ γ γ β β β α α α

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2

α α α δ δ δ δ δ δ β β β γ γ γ δ δ δ δ δ δ P1 P2 P1 P2 P1 P2 α α α

1 2 1 2 1 2 1 2 1 2 1 2

scheduler chooses action β β β or γ γ γ of process P1 P1 P1, depending on P2 P2 P2’s internal probabilistic choice

64 / 124

SLIDE 65

Outline

verview-pomdp
Markov decision processes (MDP) and

quantitative analysis against path events

partial order reduction for MDP
partially-oberservable MDP

← − ← − ← −

conclusions

65 / 124

SLIDE 66

Monty-Hall problem

pomdp-01

3 3 3 doors initially closed candidate show master

66 / 124

SLIDE 67

Monty-Hall problem

pomdp-01

no prize prize no prize 3 3 3 doors initially closed candidate show master

67 / 124

SLIDE 68

Monty-Hall problem

pomdp-01

no prize prize no prize 3 3 3 doors initially closed candidate show master 1. candidate chooses one of the doors

68 / 124

SLIDE 69

Monty-Hall problem

pomdp-01

no prize prize no prize 3 3 3 doors initially closed candidate show master 1. candidate chooses one of the doors 2. show master opens a non-chosen, non-winning door

69 / 124

SLIDE 70

Monty-Hall problem

pomdp-01

no prize prize no prize 3 3 3 doors initially closed candidate show master 1. candidate chooses one of the doors 2. show master opens a non-chosen, non-winning door 3. candidate has the choice:

keep the choice
r
switch to the other (still closed) door

70 / 124

SLIDE 71

Monty-Hall problem

pomdp-01

no prize 100.000 100.000 100.000 Euro no prize 3 3 3 doors initially closed candidate show master 1. candidate chooses one of the doors 2. show master opens a non-chosen, non-winning door 3. candidate has the choice:

keep the choice
r
switch to the other (still closed) door

4. show master opens all doors

71 / 124

SLIDE 72

Monty-Hall problem

pomdp-01

no prize 100.000 100.000 100.000 Euro no prize 3 3 3 doors initially closed candidate show master

ptimal strategy for the candidate:

initial choice of the door: arbitrary revision of the initial choice (switch) probability for getting the prize: 2

3 2 3 2 3

72 / 124

SLIDE 73

MDP for the Monty-Hall problem

pomdp-02 73 / 124

SLIDE 74

MDP for the Monty-Hall problem

pomdp-02

no prize prize no prize 3 3 3 doors initially closed candidate’s actions

1. choose one door
3. keep or switch ?

show master’s actions

2. opens a non-chosen,

non-winning door

4. opens all doors

74 / 124

SLIDE 75

MDP for the Monty-Hall problem

pomdp-02

no prize prize no prize 3 3 3 doors initially closed candidate’s actions

1. choose one door
3. keep or switch ?

show master’s actions

2. opens a non-chosen,

non-winning door

4. opens all doors

✟✟✟✟✟✟✟✟✟✟✟✟ ✟ ❍❍❍❍❍❍❍❍❍❍❍❍ ❍ ✟✟✟✟✟✟✟✟✟✟✟✟ ✟ ❍❍❍❍❍❍❍❍❍❍❍❍ ❍ ✟✟✟✟✟✟✟✟✟✟✟✟ ✟ ❍❍❍❍❍❍❍❍❍❍❍❍ ❍

start door1

1 1

door2

2 2

door3

3 3

lost won

1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3

keep switch keep switch keep switch

75 / 124

SLIDE 76

MDP for the Monty-Hall problem

pomdp-02

no prize prize no prize 3 3 3 doors initially closed candidate’s actions

1. choose one door
3. keep or switch ?

Prmax(start, ♦won) , ♦won) , ♦won) = 1 Prmax(start, ♦won) , ♦won) , ♦won) = 1 Prmax(start, ♦won) , ♦won) , ♦won) = 1

ptimal scheduler requires

complete information

n the states

start door1

1 1

door2

2 2

door3

3 3

lost won

1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3

won switch keep switch

76 / 124

SLIDE 77

MDP for the Monty-Hall problem

pomdp-02

no prize prize no prize 3 3 3 doors initially closed candidate’s actions

1. choose one door
3. keep or switch ?

cannot be distinguished by the candidate start door1

1 1

door2

2 2

door3

3 3

lost won

1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3

won switch keep switch keep switch keep

77 / 124

SLIDE 78

MDP for the Monty-Hall problem

pomdp-02

no prize prize no prize 3 3 3 doors initially closed candidate’s actions

1. choose one door
3. keep or switch ?
bservation-based strategy:

choose action switch in state doori

i i

start door1

1 1

door2

2 2

door3

3 3

lost won

1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3

won switch switch switch

78 / 124

SLIDE 79

MDP for the Monty-Hall problem

pomdp-02

no prize prize no prize 3 3 3 doors initially closed candidate’s actions

1. choose one door
3. keep or switch ?
bservation-based strategy:

choose action switch in state doori

i i

probability for ♦won ♦won ♦won: 2

3 2 3 2 3

start door1

1 1

door2

2 2

door3

3 3

lost won won

1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3

switch switch switch

79 / 124

SLIDE 80

Partially-observable Markov decision process

pomdp-05

A partially-observable MDP (POMDP for short) is an MDP M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .) together with an equivalence relation ∼ ∼ ∼ on S S S

80 / 124

SLIDE 81

Partially-observable Markov decision process

pomdp-05

A partially-observable MDP (POMDP for short) is an MDP M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .) together with an equivalence relation ∼ ∼ ∼ on S S S





if s1 ∼ s2 s1 ∼ s2 s1 ∼ s2 then s1, s2 s1, s2 s1, s2 cannot be distinguished from outside (or by the scheduler)

bservables: equivalence classes of states

81 / 124

SLIDE 82

Partially-observable Markov decision process

pomdp-05

A partially-observable MDP (POMDP for short) is an MDP M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .) together with an equivalence relation ∼ ∼ ∼ on S S S





if s1 ∼ s2 s1 ∼ s2 s1 ∼ s2 then s1, s2 s1, s2 s1, s2 cannot be distinguished from outside (or by the scheduler)

bservables: equivalence classes of states
bservation-based scheduler:

scheduler D : S+ → Act D : S+ → Act D : S+ → Act such that for all π1, π2 ∈ S+ π1, π2 ∈ S+ π1, π2 ∈ S+: D(π1) = D(π2) D(π1) = D(π2) D(π1) = D(π2) if obs(π1) = obs(π2)

bs(π1) = obs(π2)
bs(π1) = obs(π2)

where obs(s0 s1 . . . sn) = [s0] [s1] . . . [sn]

bs(s0 s1 . . . sn) = [s0] [s1] . . . [sn]
bs(s0 s1 . . . sn) = [s0] [s1] . . . [sn]

82 / 124

SLIDE 83

Extreme cases of POMDP

pomdp-11

extreme cases of an POMDP:

s1 ∼ s2

s1 ∼ s2 s1 ∼ s2 iff s1 = s2 s1 = s2 s1 = s2

s1 ∼ s2

s1 ∼ s2 s1 ∼ s2 for all s1 s1 s1, s2 s2 s2

83 / 124

SLIDE 84

Extreme cases of POMDP

pomdp-11

extreme cases of an POMDP:

s1 ∼ s2

s1 ∼ s2 s1 ∼ s2 iff s1 = s2 s1 = s2 s1 = s2 ← − ← − ← − standard MDP

s1 ∼ s2

s1 ∼ s2 s1 ∼ s2 for all s1 s1 s1, s2 s2 s2

84 / 124

SLIDE 85

Probabilistic automata are special POMDP

pomdp-11

extreme cases of an POMDP:

s1 ∼ s2

s1 ∼ s2 s1 ∼ s2 iff s1 = s2 s1 = s2 s1 = s2 ← − ← − ← − standard MDP

s1 ∼ s2

s1 ∼ s2 s1 ∼ s2 for all s1 s1 s1, s2 s2 s2 ← − ← − ← − probabilistic automata note that for totally non-observable POMDP:

bservation-based

scheduler

=
=
=

function D : N → Act D : N → Act D : N → Act

=
=
= infinite word
ver Act

Act Act

85 / 124

SLIDE 86

Undecidability results for POMDP

pomdp-11

extreme cases of an POMDP:

s1 ∼ s2

s1 ∼ s2 s1 ∼ s2 iff s1 = s2 s1 = s2 s1 = s2 ← − ← − ← − standard MDP

s1 ∼ s2

s1 ∼ s2 s1 ∼ s2 for all s1 s1 s1, s2 s2 s2 ← − ← − ← − probabilistic automata note that for totally non-observable POMDP:

bservation-based

scheduler

=
=
=

function D : N → Act D : N → Act D : N → Act

=
=
= infinite word
ver Act

Act Act undecidability results for PFA carry over to POMDP maximum probabilistic reachability problem “does Probs

max(♦F) > p

Probs

max(♦F) > p

Probs

max(♦F) > p hold ?”

=
=
=

non-emptiness problem for PFA

86 / 124

SLIDE 87

Undecidability results for POMDP

pomdp-30-new

The model checking problem for POMDP and

quantitative properties is undecidable, e.g., probabilistic reachability properties.

87 / 124

SLIDE 88

Undecidability results for POMDP

pomdp-30-new

The model checking problem for POMDP and

quantitative properties is undecidable, e.g., probabilistic reachability properties.

There is no even no approximation algorithm for

reachability objectives.

[Paz’71], [Madani/Hanks/Condon’99], [Giro/d’Argenio’07]

88 / 124

SLIDE 89

Undecidability results for POMDP

pomdp-30-new

The model checking problem for POMDP and

quantitative properties is undecidable, e.g., probabilistic reachability properties. ♦ ♦ ♦ =

=
= “infinitely often”
There is no even no approximation algorithm for

reachability objectives.

The model checking problem for POMDP and several

qualitative properties is undecidable, e.g., repeated reachability with positive probability “does Probs

max(♦F)

Probs

max(♦F)

Probs

max(♦F) > 0

> 0 > 0 hold ?”

89 / 124

SLIDE 90

Undecidability results for POMDP

pomdp-30-new

The model checking problem for POMDP and

quantitative properties is undecidable, e.g., probabilistic reachability properties.

There is no even no approximation algorithm for

reachability objectives.

The model checking problem for POMDP and several

qualitative properties is undecidable, e.g., repeated reachability with positive probability “does Probs

max(♦F)

Probs

max(♦F)

Probs

max(♦F) > 0

> 0 > 0 hold ?” Many interesting verification problems for distributed probabilistic multi-agent systems are undecidable.

90 / 124

SLIDE 91

Undecidability results for POMDP

pomdp-30-new

The model checking problem for POMDP and

quantitative properties is undecidable, e.g., probabilistic reachability properties.

There is no even no approximation algorithm for

reachability objectives.

The model checking problem for POMDP and several

qualitative properties is undecidable, e.g., repeated reachability with positive probability “does Probs

max(♦F)

Probs

max(♦F)

Probs

max(♦F) > 0

> 0 > 0 hold ?” ... already holds for totally non-observable POMDP

probabilistic B¨

uchi automata

91 / 124

SLIDE 92

Probabilistic B¨ uchi automaton (PBA)

pba-01

P = (Q, Σ, δ, µ, F) P = (Q, Σ, δ, µ, F) P = (Q, Σ, δ, µ, F)

Q

Q Q finite state space

Σ

Σ Σ alphabet

δ : Q × Σ × Q → [0, 1]

δ : Q × Σ × Q → [0, 1] δ : Q × Σ × Q → [0, 1] s.t. for all q ∈ Q q ∈ Q q ∈ Q, a ∈ Σ a ∈ Σ a ∈ Σ:

p∈Q

δ(q, a, p) ∈ {0, 1}

p∈Q

δ(q, a, p) ∈ {0, 1}

p∈Q

δ(q, a, p) ∈ {0, 1}

initial distribution µ

µ µ

set of final states F ⊆ Q

F ⊆ Q F ⊆ Q

92 / 124

SLIDE 93

Probabilistic B¨ uchi automaton (PBA)

pba-01

POMDP where Σ = Act Σ = Act Σ = Act and ∼ ∼ ∼ =

=
= Q × Q

Q × Q Q × Q P = (Q, Σ, δ, µ, F) P = (Q, Σ, δ, µ, F) P = (Q, Σ, δ, µ, F) ← − ← − ← −

Q

Q Q finite state space

Σ

Σ Σ alphabet

δ : Q × Σ × Q → [0, 1]

δ : Q × Σ × Q → [0, 1] δ : Q × Σ × Q → [0, 1] s.t. for all q ∈ Q q ∈ Q q ∈ Q, a ∈ Σ a ∈ Σ a ∈ Σ:

p∈Q

δ(q, a, p) ∈ {0, 1}

p∈Q

δ(q, a, p) ∈ {0, 1}

p∈Q

δ(q, a, p) ∈ {0, 1}

initial distribution µ

µ µ

set of final states F ⊆ Q

F ⊆ Q F ⊆ Q

93 / 124

SLIDE 94

Probabilistic B¨ uchi automaton (PBA)

pba-01

POMDP where Σ = Act Σ = Act Σ = Act and ∼ ∼ ∼ =

=
= Q × Q

Q × Q Q × Q P = (Q, Σ, δ, µ, F) P = (Q, Σ, δ, µ, F) P = (Q, Σ, δ, µ, F) ← − ← − ← −

Q

Q Q finite state space

Σ

Σ Σ alphabet

δ : Q × Σ × Q → [0, 1]

δ : Q × Σ × Q → [0, 1] δ : Q × Σ × Q → [0, 1] s.t. for all q ∈ Q q ∈ Q q ∈ Q, a ∈ Σ a ∈ Σ a ∈ Σ:

p∈Q

δ(q, a, p) ∈ {0, 1}

p∈Q

δ(q, a, p) ∈ {0, 1}

p∈Q

δ(q, a, p) ∈ {0, 1}

initial distribution µ

µ µ

set of final states F ⊆ Q

F ⊆ Q F ⊆ Q For each infinite word x ∈ Σω x ∈ Σω x ∈ Σω: Pr(x) = Pr(x) = Pr(x) = probability for the accepting runs for x x x ↑ ↑ ↑ accepting run: visits F F F infinitely often

94 / 124

SLIDE 95

Probabilistic B¨ uchi automaton (PBA)

pba-01

POMDP where Σ = Act Σ = Act Σ = Act and ∼ ∼ ∼ =

=
= Q × Q

Q × Q Q × Q P = (Q, Σ, δ, µ, F) P = (Q, Σ, δ, µ, F) P = (Q, Σ, δ, µ, F) ← − ← − ← −

Q

Q Q finite state space

Σ

Σ Σ alphabet

δ : Q × Σ × Q → [0, 1]

δ : Q × Σ × Q → [0, 1] δ : Q × Σ × Q → [0, 1] s.t. for all q ∈ Q q ∈ Q q ∈ Q, a ∈ Σ a ∈ Σ a ∈ Σ:

p∈Q

δ(q, a, p) ∈ {0, 1}

p∈Q

δ(q, a, p) ∈ {0, 1}

p∈Q

δ(q, a, p) ∈ {0, 1}

initial distribution µ

µ µ

set of final states F ⊆ Q

F ⊆ Q F ⊆ Q For each infinite word x ∈ Σω x ∈ Σω x ∈ Σω: Pr(x) = Pr(x) = Pr(x) = probability for the accepting runs for x x x ↑ ↑ ↑ probability measure in the infinite Markov chain induced by x x x viewed as a scheduler

95 / 124

SLIDE 96

Accepted language of a PBA

pba-03

P = (Q, Σ, δ, µ, F) P = (Q, Σ, δ, µ, F) P = (Q, Σ, δ, µ, F)

Q

Q Q finite state space, Σ Σ Σ alphabet

δ : Q × Σ × Q → [0, 1]

δ : Q × Σ × Q → [0, 1] δ : Q × Σ × Q → [0, 1] s.t. . . . . . . . . .

initial distribution µ

µ µ

set of final states F ⊆ Q

F ⊆ Q F ⊆ Q three types of accepted language: L>0(P) L>0(P) L>0(P) = = =

x ∈ Σω : Pr(x) > 0
x ∈ Σω : Pr(x) > 0
x ∈ Σω : Pr(x) > 0
probable semantics

L=1(P) L=1(P) L=1(P) = = =

x ∈ Σω : Pr(x) = 1
x ∈ Σω : Pr(x) = 1
x ∈ Σω : Pr(x) = 1
almost-sure sem.

L>λ(P) L>λ(P) L>λ(P) = = =

x ∈ Σω : Pr(x) > λ
x ∈ Σω : Pr(x) > λ
x ∈ Σω : Pr(x) > λ
threshold semantics

where 0 < λ < 1 0 < λ < 1 0 < λ < 1

96 / 124

SLIDE 97

Example for PBA

pba-05

1 1 1 2 2 2 a a a, 1

2 1 2 1 2

b b b a a a a a a, 1

2 1 2 1 2

initial state (probability 1) final state nonfinal state

97 / 124

SLIDE 98

Example for PBA

pba-05

1 1 1 2 2 2 a a a, 1

2 1 2 1 2

b b b a a a a a a, 1

2 1 2 1 2

accepted language: L>0(P) L>0(P) L>0(P) = = = (a + b)∗aω (a + b)∗aω (a + b)∗aω initial state (probability 1) final state nonfinal state

98 / 124

SLIDE 99

Example for PBA

pba-05

1 1 1 2 2 2 a a a, 1

2 1 2 1 2

b b b a a a a a a, 1

2 1 2 1 2

accepted language: L>0(P) L>0(P) L>0(P) = = = (a + b)∗aω (a + b)∗aω (a + b)∗aω L=1(P) L=1(P) L=1(P) = = = b∗aω b∗aω b∗aω initial state (probability 1) final state nonfinal state

99 / 124

SLIDE 100

Example for PBA

pba-05

1 1 1 2 2 2 a a a, 1

2 1 2 1 2

b b b a a a a a a, 1

2 1 2 1 2

accepted language: L>0(P) L>0(P) L>0(P) = = = (a + b)∗aω (a + b)∗aω (a + b)∗aω L=1(P) L=1(P) L=1(P) = = = b∗aω b∗aω b∗aω Thus: PBA>0

>0 >0 are strictly more expressive than DBA

100 / 124

SLIDE 101

Example for PBA

pba-05

1 1 1 2 2 2 a a a, 1

2 1 2 1 2

b b b a a a a a a, 1

2 1 2 1 2

accepted language: L>0(P) L>0(P) L>0(P) = = = (a + b)∗aω (a + b)∗aω (a + b)∗aω L=1(P) L=1(P) L=1(P) = = = b∗aω b∗aω b∗aω Thus: PBA>0

>0 >0 are strictly more expressive than DBA

3 3 3 2 2 2 1 1 1 a, 1

2

a, 1

2

a, 1

2

a, 1

2

a, 1

2

a, 1

2

b b b c c c b b b

101 / 124

SLIDE 102

Example for PBA

pba-05

1 1 1 2 2 2 a a a, 1

2 1 2 1 2

b b b a a a a a a, 1

2 1 2 1 2

accepted language: L>0(P) L>0(P) L>0(P) = = = (a + b)∗aω (a + b)∗aω (a + b)∗aω L=1(P) L=1(P) L=1(P) = = = b∗aω b∗aω b∗aω Thus: PBA>0

>0 >0 are strictly more expressive than DBA

3 3 3 2 2 2 1 1 1 a, 1

2

a, 1

2

a, 1

2

a, 1

2

a, 1

2

a, 1

2

b b b c c c b b b NBA accepts ((ac)∗ab)ω ((ac)∗ab)ω ((ac)∗ab)ω

102 / 124

SLIDE 103

Example for PBA

pba-05

1 1 1 2 2 2 a a a, 1

2 1 2 1 2

b b b a a a a a a, 1

2 1 2 1 2

accepted language: L>0(P) L>0(P) L>0(P) = = = (a + b)∗aω (a + b)∗aω (a + b)∗aω L=1(P) L=1(P) L=1(P) = = = b∗aω b∗aω b∗aω Thus: PBA>0

>0 >0 are strictly more expressive than DBA

3 3 3 2 2 2 1 1 1 a, 1

2

a, 1

2

a, 1

2

a, 1

2

a, 1

2

a, 1

2

b b b c c c b b b accepted language: L>0(P) L>0(P) L>0(P) = = = (ab + ac)∗(ab)ω (ab + ac)∗(ab)ω (ab + ac)∗(ab)ω but NBA accepts ((ac)∗ab)ω ((ac)∗ab)ω ((ac)∗ab)ω

103 / 124

SLIDE 104

Example for PBA

pba-05

1 1 1 2 2 2 a a a, 1

2 1 2 1 2

b b b a a a a a a, 1

2 1 2 1 2

accepted language: L>0(P) L>0(P) L>0(P) = = = (a + b)∗aω (a + b)∗aω (a + b)∗aω L=1(P) L=1(P) L=1(P) = = = b∗aω b∗aω b∗aω Thus: PBA>0

>0 >0 are strictly more expressive than DBA

3 3 3 2 2 2 1 1 1 a, 1

2

a, 1

2

a, 1

2

a, 1

2

a, 1

2

a, 1

2

b b b c c c b b b accepted language: L>0(P) L>0(P) L>0(P) = = = (ab + ac)∗(ab)ω (ab + ac)∗(ab)ω (ab + ac)∗(ab)ω L=1(P) L=1(P) L=1(P) = = = (ab)ω (ab)ω (ab)ω but NBA accepts ((ac)∗ab)ω ((ac)∗ab)ω ((ac)∗ab)ω

104 / 124

SLIDE 105

Expressiveness of PBA with probable semantics

pba-10a

PBA>0

>0 >0 are strictly more expressive than NBA

105 / 124

SLIDE 106

Expressiveness of PBA with probable semantics

pba-10a

PBA>0

>0 >0 are strictly more expressive than NBA

from NBA to PBA: via deterministic-in-limit NBA

106 / 124

SLIDE 107

Expressiveness of PBA with probable semantics

pba-10a

PBA>0

>0 >0 are strictly more expressive than NBA

from NBA to PBA: via deterministic-in-limit NBA
PBA can accept non-ω

ω ω-regular languages 2 2 2 1 1 1 a a a,1

2 1 2 1 2

a a a a a a, 1

2 1 2 1 2

b b b

107 / 124

SLIDE 108

Expressiveness of PBA with probable semantics

pba-10a

PBA>0

>0 >0 are strictly more expressive than NBA

from NBA to PBA: via deterministic-in-limit NBA
PBA can accept non-ω

ω ω-regular languages 2 2 2 1 1 1 a a a,1

2 1 2 1 2

a a a a a a, 1

2 1 2 1 2

b b b accepted language (probable semantics): L>0(P) =

ak1bak2bak3b. . . |

L>0(P) =

ak1bak2bak3b. . . |

L>0(P) =

ak1bak2bak3b. . . |

. . . . . . . . .

108 / 124

SLIDE 109

Expressiveness of PBA with probable semantics

pba-10a

PBA>0

>0 >0 are strictly more expressive than NBA

from NBA to PBA: via deterministic-in-limit NBA
PBA can accept non-ω

ω ω-regular languages 2 2 2 1 1 1 a a a,1

2 1 2 1 2

a a a a a a, 1

2 1 2 1 2

b b b accepted language (probable semantics): L>0(P) =

ak1bak2bak3b. . . |

L>0(P) =

ak1bak2bak3b. . . |

L>0(P) =

ak1bak2bak3b. . . |

∞

i=1
1 −

1

2

ki > 0

∞

i=1
1 −

1

2

ki > 0

∞

i=1
1 −

1

2

ki > 0

109 / 124

SLIDE 110

Expressiveness of PBA

pba-15

PBA>0

>0 >0

DBA

110 / 124

SLIDE 111

Expressiveness of PBA

pba-15

PBA>0

>0 >0

DBA NBA

111 / 124

SLIDE 112

Expressiveness of PBA

pba-15

PBA>0

>0 >0

DBA NBA PBA with thresholds

112 / 124

SLIDE 113

Expressiveness of PBA

pba-15

PBA>0

>0 >0

DBA NBA PBA with thresholds PBA=1

=1 =1

almost-sure semantics

113 / 124

SLIDE 114

Expressiveness of PBA

pba-15

PBA>0

>0 >0

DBA NBA PBA with thresholds PBA=1

=1 =1

almost-sure semantics (a + b)∗aω (a + b)∗aω (a + b)∗aω

114 / 124

SLIDE 115

Expressiveness of PBA

pba-15

PBA>0

>0 >0

DBA NBA PBA with thresholds PBA=1

=1 =1

almost-sure semantics (a + b)∗aω (a + b)∗aω (a + b)∗aω

ak1bak2b. . . :

∞

i=1

(1 − (1

2)ki) > 0

ak1bak2b. . . :

∞

i=1

(1 − (1

2)ki) > 0

ak1bak2b. . . :

∞

i=1

(1 − (1

2)ki) > 0

115 / 124

SLIDE 116

Expressiveness of PBA

pba-15

PBA>0

>0 >0

DBA NBA PBA with thresholds PBA=1

=1 =1

almost-sure semantics (a + b)∗aω (a + b)∗aω (a + b)∗aω

ak1bak2b. . . :

∞

i=1

(1 − (1

2)ki) > 0

ak1bak2b. . . :

∞

i=1

(1 − (1

2)ki) > 0

ak1bak2b. . . :

∞

i=1

(1 − (1

2)ki) > 0

ak1bak2b. . . :

∞

i=1

(1 − (1

2)ki) = 0

ak1bak2b. . . :

∞

i=1

(1 − (1

2)ki) = 0

ak1bak2b. . . :

∞

i=1

(1 − (1

2)ki) = 0

116 / 124

SLIDE 117

Expressiveness of PBA

pba-15

PBA>0

>0 >0

DBA NBA PBA with thresholds PBA=1

=1 =1

almost-sure semantics (a + b)∗aω (a + b)∗aω (a + b)∗aω emptiness problem: undecidable for PBA>0

>0 >0

decidable for PBA=1

=1 =1

117 / 124

SLIDE 118

Outline

verview-conc
Markov decision processes (MDP) and

quantitative analysis against path events

partial order reduction for MDP
partially-oberservable MDP
conclusions

← − ← − ← −

118 / 124

SLIDE 119

Conclusion

conc

worst/best-case analysis of MDP solvable by

∗ ∗ ∗ numerical methods for solving linear programs ∗ ∗ ∗ known techniques for non-probabilistic systems graph algorithms, LTL-2-AUT translators, . . . . . . . . . techniques to combat the state explosion problem (such as partial order reduction)

119 / 124

SLIDE 120

Conclusion

conc

worst/best-case analysis of MDP solvable by

∗ ∗ ∗ numerical methods for solving linear programs ∗ ∗ ∗ known techniques for non-probabilistic systems graph algorithms, LTL-2-AUT translators, . . . . . . . . . techniques to combat the state explosion problem (such as partial order reduction) but: strongly simplified definition of schedulers





assumption “full knowledge of the history” is inadequate, e.g., for agents of distributed systems

120 / 124

SLIDE 121

Conclusion

conc

worst/best-case analysis of MDP solvable by

∗ ∗ ∗ numerical methods for solving linear programs ∗ ∗ ∗ known techniques for non-probabilistic systems

more realistic model: partially-observable MDP

and multi-agents variants with distributed schedulers

121 / 124

SLIDE 122

Conclusion

conc

worst/best-case analysis of MDP solvable by

∗ ∗ ∗ numerical methods for solving linear programs ∗ ∗ ∗ known techniques for non-probabilistic systems

more realistic model: partially-observable MDP

and multi-agents variants with distributed schedulers − − − many algorithms for “finite-horizon properties” − − − few decidability results for qualitative properties − − − undecidability for quantitative properties and, e.g., repeated reachability with positive probability

122 / 124

SLIDE 123

Conclusion

conc

worst/best-case analysis of MDP solvable by

∗ ∗ ∗ numerical methods for solving linear programs ∗ ∗ ∗ known techniques for non-probabilistic systems

more realistic model: partially-observable MDP

and multi-agents variants with distributed schedulers − − − many algorithms for “finite-horizon properties” − − − few decidability results for qualitative properties − − − undecidability for quantitative properties and, e.g., repeated reachability with positive probability











 proof via probabilistic language acceptors (PFA/PBA)

123 / 124

SLIDE 124

Conclusion

conc

worst/best-case analysis of MDP solvable by

∗ ∗ ∗ numerical methods for solving linear programs ∗ ∗ ∗ known techniques for non-probabilistic systems

more realistic model: partially-observable MDP

and multi-agents variants with distributed schedulers − − − many algorithms for “finite-horizon properties” − − − few decidability results for qualitative properties − − − undecidability for quantitative properties and, e.g., repeated reachability with positive probability

probabilistic B¨

uchi automata interesting in their own . . . . . . . . .

124 / 124