[PPT] - On Model Checking Techniques for Randomized Distributed Systems PowerPoint Presentation

SLIDE 1

On Model Checking Techniques for Randomized Distributed Systems Christel Baier Technische Universit¨ at Dresden joint work with Nathalie Bertrand Frank Ciesinski Marcus Gr¨

ßer

1 / 161

SLIDE 2

Probability elsewhere

int-01

randomized algorithms

[Rabin 1960]

breaking symmetry, fingerprints, input sampling, . . . . . . . . .

stochastic control theory

[Bellman 1957]

perations research
performance modeling

[Markov, Erlang, Kolm., ∼ ∼ ∼ 1900]

emphasis on steady-state and transient measures

biological systems, resilient systems, security protocols

. . . . . . . . .

2 / 161

SLIDE 3

Probability elsewhere

int-01

randomized algorithms

[Rabin 1960]

breaking symmetry, fingerprints, input sampling, . . . . . . . . . models: discrete-time Markov chains Markov decision processes

stochastic control theory

[Bellman 1957]

perations research

models: Markov decision processes

performance modeling

[Markov, Erlang, Kolm., ∼ ∼ ∼ 1900]

emphasis on steady-state and transient measures models: continuous-time Markov chains

biological systems, resilient systems, security protocols

. . . . . . . . .

3 / 161

SLIDE 4

Model checking

[Clarke/Emerson, Queille/Sifakis]

mc

requirements (safety, liveness) specification, e.g., temporal formula Φ Φ Φ reactive system abstract model M M M model checking “does M | = Φ M | = Φ M | = Φ hold ?” no yes

4 / 161

SLIDE 5

Probabilistic model checking

int-03

quantitative requirements specification, e.g., temporal formula Φ Φ Φ probabilistic reactive system probabilistic model M M M probabilistic model checking “does M | = Φ M | = Φ M | = Φ hold ?” probability for “bad behaviors” is < 10−6 < 10−6 < 10−6 probability for “good behaviors” is 1 1 1 expected costs for ....

5 / 161

SLIDE 6

Probabilistic model checking

int-03

quantitative requirements linear temporal formula Φ Φ Φ (path event) probabilistic reactive system Markov decision process M M M probabilistic model checking quantitative analysis of M M M against Φ Φ Φ probability for “bad behaviors” is < 10−6 < 10−6 < 10−6 probability for “good behaviors” is 1 1 1

6 / 161

SLIDE 7

Outline

verview
Markov decision processes (MDP) and

quantitative analysis against path events

partial order reduction for MDP
partially-oberservable MDP
conclusions

7 / 161

SLIDE 8

Markov decision process (MDP)

mdp-01

perational model with nondeterminism and probabilism

8 / 161

SLIDE 9

Markov decision process (MDP)

mdp-01

perational model with nondeterminism and probabilism
modeling randomized distributed systems

by interleaving s s s

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2

process 1 1 1 tosses a coin process 2 2 2 tosses a coin process 1 1 1 tosses a coin process 2 2 2 tosses a coin

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2

9 / 161

SLIDE 10

Markov decision process (MDP)

mdp-01

perational model with nondeterminism and probabilism
modeling randomized distributed systems

by interleaving

nondeterminism useful for abstraction, underspec.,

modeling interactions with an unkown environment s s s

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2

process 1 1 1 tosses a coin process 2 2 2 tosses a coin process 1 1 1 tosses a coin process 2 2 2 tosses a coin

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2

10 / 161

SLIDE 11

Markov decision process (MDP)

mdp-02-r

M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .)

11 / 161

SLIDE 12

Markov decision process (MDP)

mdp-02-r

M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .)

finite state space S

S S

12 / 161

SLIDE 13

Markov decision process (MDP)

mdp-02-r

M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .)

finite state space S

S S

Act

Act Act finite set of actions

P : S × Act × S → [0, 1]

P : S × Act × S → [0, 1] P : S × Act × S → [0, 1]

13 / 161

SLIDE 14

Markov decision process (MDP)

mdp-02-r

M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .)

finite state space S

S S

Act

Act Act finite set of actions

P : S × Act × S → [0, 1]

P : S × Act × S → [0, 1] P : S × Act × S → [0, 1] s s s α α α β β β

1 4 1 4 1 4 3 4 3 4 3 4 1 2 1 2 1 2 1 6 1 6 1 6 1 3 1 3 1 3

nondeterministic choice probabilistic choice

14 / 161

SLIDE 15

Markov decision process (MDP)

mdp-02-r

M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .)

finite state space S

S S

Act

Act Act finite set of actions

P : S × Act × S → [0, 1]

P : S × Act × S → [0, 1] P : S × Act × S → [0, 1] s.t. ∀s ∈ S ∀s ∈ S ∀s ∈ S ∀α ∈ Act. . . ∀α ∈ Act. . . ∀α ∈ Act. . .

s′∈S

P(s, α, s′) ∈ {0, 1}

s′∈S

P(s, α, s′) ∈ {0, 1}

s′∈S

P(s, α, s′) ∈ {0, 1} s s s α α α β β β

1 4 1 4 1 4 3 4 3 4 3 4 1 2 1 2 1 2 1 6 1 6 1 6 1 3 1 3 1 3

nondeterministic choice probabilistic choice ր ր ր α / ∈ Act(s) α / ∈ Act(s) α / ∈ Act(s) տ տ տ α ∈ Act(s) α ∈ Act(s) α ∈ Act(s) Act(s) = Act(s) = Act(s) = set of actions that are enabled in state s s s

15 / 161

SLIDE 16

Markov decision process (MDP)

mdp-02-r

M = (S, Act, P, s0, AP, L, rew, . . .) M = (S, Act, P, s0, AP, L, rew, . . .) M = (S, Act, P, s0, AP, L, rew, . . .)

finite state space S

S S

Act

Act Act finite set of actions

P : S × Act × S → [0, 1]

P : S × Act × S → [0, 1] P : S × Act × S → [0, 1] s.t. ∀s ∈ S ∀s ∈ S ∀s ∈ S ∀α ∈ Act. . . ∀α ∈ Act. . . ∀α ∈ Act. . .

s′∈S

P(s, α, s′) ∈ {0, 1}

s′∈S

P(s, α, s′) ∈ {0, 1}

s′∈S

P(s, α, s′) ∈ {0, 1}

s0

s0 s0 initial state

AP

AP AP set of atomic propositions

labeling L : S → 2AP

L : S → 2AP L : S → 2AP

reward function rew : S × Act → R

rew : S × Act → R rew : S × Act → R ր ր ր α / ∈ Act(s) α / ∈ Act(s) α / ∈ Act(s) տ տ տ α ∈ Act(s) α ∈ Act(s) α ∈ Act(s)

16 / 161

SLIDE 17

Randomized mutual exclusion protocol

mdp-05

2

2 2 concurrent processes P1 P1 P1, P2 P2 P2 with 3 3 3 phases: ni ni ni noncritical actions of process Pi Pi Pi wi wi wi waiting phase of process Pi Pi Pi ci ci ci critical section of process Pi Pi Pi

competition of both processes are waiting

17 / 161

SLIDE 18

Randomized mutual exclusion protocol

mdp-05

2

2 2 concurrent processes P1 P1 P1, P2 P2 P2 with 3 3 3 phases: ni ni ni noncritical actions of process Pi Pi Pi wi wi wi waiting phase of process Pi Pi Pi ci ci ci critical section of process Pi Pi Pi

competition of both processes are waiting
resolved by a randomized arbiter who tosses a coin

18 / 161

SLIDE 19

Randomized mutual exclusion protocol

mdp-05

interleaving of the request operations
competition if both processes are waiting
randomized arbiter tosses a coin if both are waiting

n1n2 n1n2 n1n2 w1n2 w1n2 w1n2 n1w2 n1w2 n1w2 w1w2 w1w2 w1w2 c1n2 c1n2 c1n2 n1c2 n1c2 n1c2 c1w2 c1w2 c1w2 w1c2 w1c2 w1c2 MDP

r e q u e s t

2

r e q u e s t

2

r e q u e s t

2

release1 release1 release1 e n t e r

1

e n t e r

1

e n t e r

1

request1 request1 request1 request2 request2 request2 request2 request2 request2 request1 request1 request1 e n t e r

2

e n t e r

2

e n t e r

2

r e q u e s t

1

r e q u e s t

1

r e q u e s t

1

release2 release2 release2 coin coin coin r e l e a s e2 r e l e a s e2 r e l e a s e2 r e l e a s e1 r e l e a s e1 r e l e a s e1

1 2 1 2 1 2 1 2 1 2 1 2

19 / 161

SLIDE 20

Randomized mutual exclusion protocol

mdp-05

interleaving of the request operations
competition if both processes are waiting
randomized arbiter tosses a coin if both are waiting

n1n2 n1n2 n1n2 w1n2 w1n2 w1n2 n1w2 n1w2 n1w2 w1w2 w1w2 w1w2 c1n2 c1n2 c1n2 n1c2 n1c2 n1c2 c1w2 c1w2 c1w2 w1c2 w1c2 w1c2 n1n2 n1n2 n1n2 w1n2 w1n2 w1n2 n1w2 n1w2 n1w2 w1w2 w1w2 w1w2 MDP

r e q u e s t

2

r e q u e s t

2

r e q u e s t

2

release1 release1 release1 e n t e r

1

e n t e r

1

e n t e r

1

request1 request1 request1 request2 request2 request2 request2 request2 request2 request1 request1 request1 e n t e r

2

e n t e r

2

e n t e r

2

r e q u e s t

1

r e q u e s t

1

r e q u e s t

1

release2 release2 release2 coin coin coin r e l e a s e2 r e l e a s e2 r e l e a s e2 r e l e a s e1 r e l e a s e1 r e l e a s e1

1 2 1 2 1 2 1 2 1 2 1 2

20 / 161

SLIDE 21

Randomized mutual exclusion protocol

mdp-05

interleaving of the request operations
competition if both processes are waiting
randomized arbiter tosses a coin if both are waiting

n1n2 n1n2 n1n2 w1n2 w1n2 w1n2 n1w2 n1w2 n1w2 w1w2 w1w2 w1w2 c1n2 c1n2 c1n2 n1c2 n1c2 n1c2 c1w2 c1w2 c1w2 w1c2 w1c2 w1c2 w1w2 w1w2 w1w2 MDP

r e q u e s t

2

r e q u e s t

2

r e q u e s t

2

release1 release1 release1 e n t e r

1

e n t e r

1

e n t e r

1

request1 request1 request1 request2 request2 request2 request2 request2 request2 request1 request1 request1 e n t e r

2

e n t e r

2

e n t e r

2

r e q u e s t

1

r e q u e s t

1

r e q u e s t

1

release2 release2 release2 coin coin coin r e l e a s e2 r e l e a s e2 r e l e a s e2 r e l e a s e1 r e l e a s e1 r e l e a s e1

1 2 1 2 1 2 1 2 1 2 1 2

21 / 161

SLIDE 22

Randomized mutual exclusion protocol

mdp-05

interleaving of the request operations
competition if both processes are waiting
randomized arbiter tosses a coin if both are waiting

n1n2 n1n2 n1n2 w1n2 w1n2 w1n2 n1w2 n1w2 n1w2 w1w2 w1w2 w1w2 c1n2 c1n2 c1n2 n1c2 n1c2 n1c2 c1w2 c1w2 c1w2 w1c2 w1c2 w1c2 w1w2 w1w2 w1w2 c1w2 c1w2 c1w2 w1c2 w1c2 w1c2 MDP

r e q u e s t

2

r e q u e s t

2

r e q u e s t

2

release1 release1 release1 e n t e r

1

e n t e r

1

e n t e r

1

request1 request1 request1 request2 request2 request2 request2 request2 request2 request1 request1 request1 e n t e r

2

e n t e r

2

e n t e r

2

r e q u e s t

1

r e q u e s t

1

r e q u e s t

1

release2 release2 release2 toss a toss a toss a coin coin coin r e l e a s e2 r e l e a s e2 r e l e a s e2 r e l e a s e1 r e l e a s e1 r e l e a s e1

1 2 1 2 1 2 1 2 1 2 1 2

22 / 161

SLIDE 23

Reasoning about probabilities in MDP

mdp-10

requires resolving the nondeterminism by schedulers

23 / 161

SLIDE 24

Reasoning about probabilities in MDP

mdp-10

requires resolving the nondeterminism by schedulers
a scheduler is a function D : S∗ −

→ Act D : S∗ − → Act D : S∗ − → Act s.t. action D (s0 . . . sn) D (s0 . . . sn) D (s0 . . . sn) is enabled in state sn sn sn

24 / 161

SLIDE 25

Reasoning about probabilities in MDP

mdp-10

requires resolving the nondeterminism by schedulers
a scheduler is a function D : S∗ −

→ Act D : S∗ − → Act D : S∗ − → Act s.t. action D (s0 . . . sn) D (s0 . . . sn) D (s0 . . . sn) is enabled in state sn sn sn

each scheduler induces an infinite Markov chain

MDP β β β γ γ γ α α α

1 3 1 3 1 3 2 3 2 3 2 3

σ σ σ δ δ δ . . . . . . . . . . . . . . . . . . . . . . . . . . .

2 3 2 3 2 3 1 3

α

1 3

α

1 3

α δ δ δ β β β γ γ γ σ σ σ α 2

3

α 2

3

α 2

3 1 3 1 3 1 3

σ σ σ

25 / 161

SLIDE 26

Reasoning about probabilities in MDP

mdp-10

requires resolving the nondeterminism by schedulers
a scheduler is a function D : S∗ −

→ Act D : S∗ − → Act D : S∗ − → Act s.t. action D (s0 . . . sn) D (s0 . . . sn) D (s0 . . . sn) is enabled in state sn sn sn

each scheduler induces an infinite Markov chain




yields a notion of probability measure PrD PrD PrD

n measurable sets of infinite paths

26 / 161

SLIDE 27

Reasoning about probabilities in MDP

mdp-10

requires resolving the nondeterminism by schedulers
a scheduler is a function D : S∗ −

→ Act D : S∗ − → Act D : S∗ − → Act s.t. action D (s0 . . . sn) D (s0 . . . sn) D (s0 . . . sn) is enabled in state sn sn sn

each scheduler induces an infinite Markov chain




yields a notion of probability measure PrD PrD PrD

n measurable sets of infinite paths

typical task: given a measurable path event E E E, ∗ ∗ ∗ check whether E E E holds almost surely, i.e., PrD(E) = 1 PrD(E) = 1 PrD(E) = 1 for all schedulers D D D

27 / 161

SLIDE 28

Reasoning about probabilities in MDP

mdp-10

requires resolving the nondeterminism by schedulers
a scheduler is a function D : S∗ −

→ Act D : S∗ − → Act D : S∗ − → Act s.t. action D (s0 . . . sn) D (s0 . . . sn) D (s0 . . . sn) is enabled in state sn sn sn

each scheduler induces an infinite Markov chain




yields a notion of probability measure PrD PrD PrD

n measurable sets of infinite paths

typical task: given a measurable path event E E E, ∗ ∗ ∗ check whether E E E holds almost surely ∗ ∗ ∗ compute the worst-case probability for E E E, i.e., sup

D

PrD(E) sup

D

PrD(E) sup

D

PrD(E)

r

inf

D

PrD(E) inf

D

PrD(E) inf

D

PrD(E)

28 / 161

SLIDE 29

Quantitative analysis of MDP

mdp-15

given: MDP M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .) with initial state s0 s0 s0 ω ω ω-regular path event E E E, e.g., given by an LTL formula task: compute PrM

max(s0, E) = sup D

PrD(s0, E) PrM

max(s0, E) = sup D

PrD(s0, E) PrM

max(s0, E) = sup D

PrD(s0, E)

29 / 161

SLIDE 30

Quantitative analysis of MDP

mdp-15

given: MDP M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .) with initial state s0 s0 s0 ω ω ω-regular path event E E E, e.g., given by an LTL formula task: compute PrM

max(s0, E) = sup D

PrD(s0, E) PrM

max(s0, E) = sup D

PrD(s0, E) PrM

max(s0, E) = sup D

PrD(s0, E) method: compute xs = PrM

max(s, E)

xs = PrM

max(s, E)

xs = PrM

max(s, E) for all s ∈ S

s ∈ S s ∈ S via graph analysis and linear program

[Vardi/Wolper’86] [Courcoubetis/Yannakakis’88] [Bianco/de Alfaro’95] [Baier/Kwiatkowska’98]

30 / 161

SLIDE 31

probabilistic system “bad behaviors”

31 / 161

SLIDE 32

probabilistic system “bad behaviors” MDP M M M

32 / 161

SLIDE 33

probabilistic system “bad behaviors” MDP M M M LTL formula ϕ ϕ ϕ deterministic automaton A A A

33 / 161

SLIDE 34

probabilistic system “bad behaviors” MDP M M M LTL formula ϕ ϕ ϕ deterministic automaton A A A quantitative analysis in the product-MDP M × A M × A M × A PrM

max(s, ϕ)

PrM

max(s, ϕ)

PrM

max(s, ϕ) =

= = PrM×A

max

s, inits,

PrM×A

max

s, inits,

PrM×A

max

s, inits, acceptance
cond. of A

A A

34 / 161

SLIDE 35

probabilistic system “bad behaviors” MDP M M M LTL formula ϕ ϕ ϕ deterministic automaton A A A quantitative analysis in the product-MDP M × A M × A M × A PrM

max(s, ϕ)

PrM

max(s, ϕ)

PrM

max(s, ϕ) =

= = PrM×A

max

s, inits,

PrM×A

max

s, inits,

PrM×A

max

s, inits, ♦accEC

♦accEC ♦accEC

maximal probabilility

for reaching an accepting end component

35 / 161

SLIDE 36

probabilistic system “bad behaviors” MDP M M M LTL formula ϕ ϕ ϕ deterministic automaton A A A probabilistic reachability analysis in the product-MDP M × A M × A M × A linear program PrM

max(s, ϕ)

PrM

max(s, ϕ)

PrM

max(s, ϕ) =

= = PrM×A

max

s, inits,

PrM×A

max

s, inits,

PrM×A

max

s, inits, ♦accEC

♦accEC ♦accEC

maximal probabilility

for reaching an accepting end component

36 / 161

SLIDE 37

probabilistic system “bad behaviors” MDP M M M LTL formula ϕ ϕ ϕ deterministic automaton A A A probabilistic reachability analysis in the product-MDP M × A M × A M × A linear program PrM

max(s, ϕ)

PrM

max(s, ϕ)

PrM

max(s, ϕ) =

= = PrM×A

max

s, inits,

PrM×A

max

s, inits,

PrM×A

max

s, inits, ♦accEC

♦accEC ♦accEC

polynomial

in |M| · |A| |M| · |A| |M| · |A|

37 / 161

SLIDE 38

2exp probabilistic system “bad behaviors” MDP M M M LTL formula ϕ ϕ ϕ deterministic automaton A A A probabilistic reachability analysis in the product-MDP M × A M × A M × A linear program PrM

max(s, ϕ)

PrM

max(s, ϕ)

PrM

max(s, ϕ) =

= = PrM×A

max

s, inits,

PrM×A

max

s, inits,

PrM×A

max

s, inits, ♦accEC

♦accEC ♦accEC

polynomial

in |M| · |A| |M| · |A| |M| · |A|

38 / 161

SLIDE 39

state explosion problem probabilistic system “bad behaviors” MDP M M M LTL formula ϕ ϕ ϕ deterministic automaton A A A probabilistic reachability analysis in the product-MDP M × A M × A M × A linear program PrM

max(s, ϕ)

PrM

max(s, ϕ)

PrM

max(s, ϕ) =

= = PrM×A

max

s, inits,

PrM×A

max

s, inits,

PrM×A

max

s, inits, ♦accEC

♦accEC ♦accEC

polynomial

in |M| · |A| |M| · |A| |M| · |A|

39 / 161

SLIDE 40

Advanced techniques for PMC

por-01-cmu

symbolic model checking with variants of BDDs

e.g., in PRISM [Kwiatkowska/Norman/Parker] ProbVerus [Hartonas-Garmhausen, Campos, Clarke]

state aggregation with bisimulation

e.g., in MRMC

[Katoen et al]

abstraction-refinement

e.g., in RAPTURE [d’Argenio/Jeannet/Jensen/Larsen] PASS

[Hermanns/Wachter/Zhang]

partial order reduction

e.g., in LiQuor

[Baier/Ciesinski/Gr¨

ßer]

40 / 161

SLIDE 41

Advanced techniques for PMC

por-01-cmu

symbolic model checking with variants of BDDs

e.g., in PRISM [Kwiatkowska/Norman/Parker] ProbVerus [Hartonas-Garmhausen, Campos, Clarke] randomized distributed algorithms, communication and multimedia protocols, power management, security, . . . . . . . . .

state aggregation with bisimulation

e.g., in MRMC

[Katoen et al]

abstraction-refinement

e.g., in RAPTURE [d’Argenio/Jeannet/Jensen/Larsen] PASS

[Hermanns/Wachter/Zhang]

partial order reduction

e.g., in LiQuor

[Baier/Ciesinski/Gr¨

ßer]

41 / 161

SLIDE 42

Advanced techniques for PMC

por-01-cmu

symbolic model checking with variants of BDDs

e.g., in PRISM [Kwiatkowska/Norman/Parker] ProbVerus [Hartonas-Garmhausen, Campos, Clarke] randomized distributed algorithms, communication and multimedia protocols, power management, security, . . . . . . . . .

state aggregation with bisimulation

e.g., in MRMC

[Katoen et al]

abstraction-refinement

e.g., in RAPTURE [d’Argenio/Jeannet/Jensen/Larsen] PASS

[Hermanns/Wachter/Zhang]

partial order reduction

e.g., in LiQuor

[Baier/Ciesinski/Gr¨

ßer]

42 / 161

SLIDE 43

Partial order reduction

por-02

technique for reducing the state space of concurrent systems

[Godefroid,Peled,Valmari, ca. 1990]

attempts to analyze a sub-system by identifying

“redundant interleavings”

explores representatives of paths that agree up to

the order of independent actions

43 / 161

SLIDE 44

Partial order reduction

por-02

technique for reducing the state space of concurrent systems

[Godefroid,Peled,Valmari, ca. 1990]

attempts to analyze a sub-system by identifying

“redundant interleavings”

explores representatives of paths that agree up to

the order of independent actions e.g., x := x+y x := x+y x := x+y

action α

α α

z := z+3

z := z+3 z := z+3

action β

β β has the same effect as α; β α; β α; β or β; α β; α β; α

44 / 161

SLIDE 45

Partial order reduction

por-02

technique for reducing the state space of concurrent systems

[Godefroid,Peled,Valmari, ca. 1990]

attempts to analyze a sub-system by identifying

“redundant interleavings”

explores representatives of paths that agree up to

the order of independent actions DFS-based on-the-fly generation of a reduced system for each expanded state s s s

choose an appropriate subset Ample(s)

Ample(s) Ample(s) of Act(s) Act(s) Act(s)

expand only the α

α α-successors of s s s for α ∈ Ample(s) α ∈ Ample(s) α ∈ Ample(s) (but ignore the actions in Act(s) \ Ample(s) Act(s) \ Ample(s) Act(s) \ Ample(s))

45 / 161

SLIDE 46

Partial order reduction

por-02a

concurrent execution

f processes P1

P1 P1, P2 P2 P2

no communication
no competition

transition system for P1P2 P1P2 P1P2 where P1 = α; β; γ P1 = α; β; γ P1 = α; β; γ P2 = λ; µ; ν P2 = λ; µ; ν P2 = λ; µ; ν α α α λ λ λ β β β λ λ λ α α α µ µ µ γ γ γ λ λ λ β β β µ µ µ α α α ν ν ν λ λ λ γ γ γ µ µ µ β β β ν ν ν α α α µ µ µ γ γ γ ν ν ν β β β ν ν ν γ γ γ

46 / 161

SLIDE 47

Partial order reduction

por-02a

concurrent execution

f processes P1

P1 P1, P2 P2 P2

no communication
no competition

transition system for P1P2 P1P2 P1P2 where P1 = α; β; γ P1 = α; β; γ P1 = α; β; γ P2 = λ; µ; ν P2 = λ; µ; ν P2 = λ; µ; ν idea: explore just 1 1 1 path as representative for all paths α α α β β β λ λ λ µ µ µ ν ν ν γ γ γ

47 / 161

SLIDE 48

Ample-set method

[Peled 1993]

por-03

given: processes Pi Pi Pi of a parallel system P1. . .Pn

P1. . .Pn
P1. . .Pn

with transition system T = (S, Act, →, . . .) T = (S, Act, →, . . .) T = (S, Act, →, . . .) task:

n-the-fly generation of a sub-system Tr

Tr Tr s.t. (A1) stutter condition . . . . . . . . . (A2) dependency condition . . . . . . . . . (A3) cycle condition . . . . . . . . .

48 / 161

SLIDE 49

Ample-set method

[Peled 1993]

por-03

given: processes Pi Pi Pi of a parallel system P1. . .Pn

P1. . .Pn
P1. . .Pn

with transition system T = (S, Act, →, . . .) T = (S, Act, →, . . .) T = (S, Act, →, . . .) task:

n-the-fly generation of a sub-system Tr

Tr Tr s.t. (A1) stutter condition (A2) dependency condition (A3) cycle condition

π πr

π πr π πr by permutations of independent actions Each path π π π in T T T is represented by an “equivalent” path πr πr πr in Tr Tr Tr

49 / 161

SLIDE 50

Ample-set method

[Peled 1993]

por-03

given: processes Pi Pi Pi of a parallel system P1. . .Pn

P1. . .Pn
P1. . .Pn

with transition system T = (S, Act, →, . . .) T = (S, Act, →, . . .) T = (S, Act, →, . . .) task:

n-the-fly generation of a sub-system Tr

Tr Tr s.t. (A1) stutter condition (A2) dependency condition (A3) cycle condition

π πr

π πr π πr by permutations of independent actions Each path π π π in T T T is represented by an “equivalent” path πr πr πr in Tr Tr Tr

T

T T and Tr Tr Tr satisfy the same stutter-invariant events, e.g., next-free LTL formulas

50 / 161

SLIDE 51

Ample-set method for MDP

por-04

given: processes Pi Pi Pi of a probabilistic system P1. . .Pn

P1. . .Pn
P1. . .Pn

with MDP-semantics M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .) task:

n-the-fly generation of a sub-MDP Mr

Mr Mr s.t. Mr Mr Mr and M M M have the same extremal probabilities for stutter-invariant events

51 / 161

SLIDE 52

Ample-set method for MDP

por-04

given: processes Pi Pi Pi of a probabilistic system P1. . .Pn

P1. . .Pn
P1. . .Pn

with MDP-semantics M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .) task:

n-the-fly generation of a sub-MDP Mr

Mr Mr s.t. For all schedulers D D D for M M M there is a scheduler Dr Dr Dr for Mr Mr Mr s.t. for all measurable, stutter-invariant events E E E: PrD

M(E) = PrDr Mr(E)

PrD

M(E) = PrDr Mr(E)

PrD

M(E) = PrDr Mr(E)

Mr

Mr Mr and M M M have the same extremal probabilities for stutter-invariant events

52 / 161

SLIDE 53

Independence of actions

por-06 53 / 161

SLIDE 54

Independence of non-probabilistic actions

por-06

Actions α α α and β β β are called independent in a transition system T T T iff: whenever s

α

− → t s

α

− → t s

α

− → t and s

β

− → u s

β

− → u s

β

− → u then (1) α α α is enabled in u u u (2) β β β is enabled in t t t (3) if u

α

− → v u

α

− → v u

α

− → v and t

β

− → w t

β

− → w t

β

− → w then v = w v = w v = w s s s t t t u u u v v v α α α β β β α α α β β β

54 / 161

SLIDE 55

Independence of actions in an MDP

por-06

Let M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .) be a MDP and α, β ∈ Act α, β ∈ Act α, β ∈ Act. α α α and β β β are independent in M M M if for each state s s s s.t. α, β ∈ Act(s) α, β ∈ Act(s) α, β ∈ Act(s): (1) if P(s, α, t) > 0 P(s, α, t) > 0 P(s, α, t) > 0 then β ∈ Act(t) β ∈ Act(t) β ∈ Act(t) (2) if P(s, β, u) > 0 P(s, β, u) > 0 P(s, β, u) > 0 then α ∈ Act(u) α ∈ Act(u) α ∈ Act(u) (3) . . . . . . . . .

55 / 161

SLIDE 56

Independence of actions in an MDP

por-06

Let M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .) be a MDP and α, β ∈ Act α, β ∈ Act α, β ∈ Act. α α α and β β β are independent in M M M if for each state s s s s.t. α, β ∈ Act(s) α, β ∈ Act(s) α, β ∈ Act(s): (1) if P(s, α, t) > 0 P(s, α, t) > 0 P(s, α, t) > 0 then β ∈ Act(t) β ∈ Act(t) β ∈ Act(t) (2) if P(s, β, u) > 0 P(s, β, u) > 0 P(s, β, u) > 0 then α ∈ Act(u) α ∈ Act(u) α ∈ Act(u) (3) for all states w w w: P(s, αβ, w) = P(s, βα, w) P(s, αβ, w) = P(s, βα, w) P(s, αβ, w) = P(s, βα, w)

ր ր ր

t∈S

P(s, α, t) · P(t, β, w)

t∈S

P(s, α, t) · P(t, β, w)

t∈S

P(s, α, t) · P(t, β, w)

տ տ տ

u∈S

P(s, β, u) · P(u, α, w)

u∈S

P(s, β, u) · P(u, α, w)

u∈S

P(s, β, u) · P(u, α, w)

56 / 161

SLIDE 57

Example: ample set method

por-08

s s s α α α β β β γ γ γ β β β γ γ γ α α α α α α δ δ δ δ δ δ

riginal system T

T T α α α independent from β β β and γ γ γ

57 / 161

SLIDE 58

Example: ample set method

por-08

s s s α α α β β β γ γ γ β β β γ γ γ α α α α α α δ δ δ δ δ δ s s s β β β γ γ γ α α α α α α δ δ δ δ δ δ

riginal system T

T T reduced system Tr Tr Tr (A1)-(A3) are fulfilled α α α independent from β β β and γ γ γ

58 / 161

SLIDE 59

Example: ample set method fails for MDP

por-08

s s s

1 2 1 2 1 2

α α α

1 2 1 2 1 2

β β β γ γ γ β β β γ γ γ γ γ γ β β β α α α

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2

α α α δ δ δ δ δ δ s s s β β β γ γ γ α α α

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2

α α α δ δ δ δ δ δ

riginal MDP M

M M reduced MDP Mr Mr Mr (A1)-(A3) are fulfilled α α α independent from β β β and γ γ γ

59 / 161

SLIDE 60

Example: ample set method fails for MDP

por-08

s s s

1 2 1 2 1 2

α α α

1 2 1 2 1 2

β β β γ γ γ β β β γ γ γ γ γ γ β β β α α α

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2

α α α δ δ δ δ δ δ s s s β β β γ γ γ α α α

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2

α α α δ δ δ δ δ δ

riginal MDP M

M M reduced MDP Mr Mr Mr PrM

max(s, ♦green) = 1

PrM

max(s, ♦green) = 1

PrM

max(s, ♦green) = 1

♦ ♦ ♦ “eventually”

60 / 161

SLIDE 61

Example: ample set method fails for MDP

por-08

s s s

1 2 1 2 1 2

α α α

1 2 1 2 1 2

β β β γ γ γ β β β γ γ γ γ γ γ β β β α α α

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2

α α α δ δ δ δ δ δ s s s β β β γ γ γ α α α

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2

α α α δ δ δ δ δ δ

riginal MDP M

M M reduced MDP Mr Mr Mr PrM

max(s, ♦green) = 1

PrM

max(s, ♦green) = 1

PrM

max(s, ♦green) = 1 > 1 2 = PrMr max(s, ♦green)

>

1 2 = PrMr max(s, ♦green)

>

1 2 = PrMr max(s, ♦green)

61 / 161

SLIDE 62

Partial order reduction for MDP

por-09

extend Peled’s conditions (A1)-(A3) for the ample-sets (A1) stutter condition . . . . . . . . . (A2) dependency condition . . . . . . . . . (A3) cycle condition . . . . . . . . . (A4) probabilistic condition If there is a path s

β1

− →

β2

− → . . .

βn

− →

α

− → s

β1

− →

β2

− → . . .

βn

− →

α

− → s

β1

− →

β2

− → . . .

βn

− →

α

− → in M M M s.t. β1, . . ., βn, α / ∈ Ample(s) β1, . . . , βn, α / ∈ Ample(s) β1, . . . , βn, α / ∈ Ample(s) and α α α is probabilistic then |Ample(s)| = 1 |Ample(s)| = 1 |Ample(s)| = 1.

62 / 161

SLIDE 63

Partial order reduction for MDP

por-09

extend Peled’s conditions (A1)-(A3) for the ample-sets (A1) stutter condition . . . . . . . . . (A2) dependency condition . . . . . . . . . (A3) cycle condition . . . . . . . . . (A4) probabilistic condition If there is a path s

β1

− →

β2

− → . . .

βn

− →

α

− → s

β1

− →

β2

− → . . .

βn

− →

α

− → s

β1

− →

β2

− → . . .

βn

− →

α

− → in M M M s.t. β1, . . ., βn, α / ∈ Ample(s) β1, . . . , βn, α / ∈ Ample(s) β1, . . . , βn, α / ∈ Ample(s) and α α α is probabilistic then |Ample(s)| = 1 |Ample(s)| = 1 |Ample(s)| = 1. If (A1)-(A4) hold then M M M and Mr Mr Mr have the same extremal probabilities for all stutter-invariant properties.

63 / 161

SLIDE 64

Probabilistic model checking

por-ifm-32

quantitative requirements LTL\

\ \ formula ϕ

ϕ ϕ (path event) probabilistic system Markov decision process M M M quantitative analysis

f M

M M against ϕ ϕ ϕ maximal/minimal probability for ϕ ϕ ϕ

64 / 161

SLIDE 65

Probabilistic model checking, e.g., LiQuor

por-ifm-32

quantitative requirements LTL\

\ \ formula ϕ

ϕ ϕ (path event) modeling language

P1. . .Pn
P1. . .Pn
P1. . .Pn

reduced MDP Mr Mr Mr partial order reduction quantitative analysis

f Mr

Mr Mr against ϕ ϕ ϕ maximal/minimal probability for ϕ ϕ ϕ

65 / 161

SLIDE 66

Probabilistic model checking, e.g., LiQuor

por-ifm-32a

quantitative requirements LTL\

\ \ formula ϕ

ϕ ϕ (path event) modeling language

P1. . .Pn
P1. . .Pn
P1. . .Pn

reduced MDP Mr Mr Mr partial order reduction quantitative analysis

f Mr

Mr Mr against ϕ ϕ ϕ maximal/minimal probability for ϕ ϕ ϕ worst-case analysis

66 / 161

SLIDE 67

Outline

verview-pomdp
Markov decision processes (MDP) and

quantitative analysis against path events

partial order reduction for MDP
partially-oberservable MDP

← − ← − ← −

conclusions

67 / 161

SLIDE 68

Monty-Hall problem

pomdp-01

3 3 3 doors initially closed candidate show master

68 / 161

SLIDE 69

Monty-Hall problem

pomdp-01

no prize prize no prize 3 3 3 doors initially closed candidate show master

69 / 161

SLIDE 70

Monty-Hall problem

pomdp-01

no prize prize no prize 3 3 3 doors initially closed candidate show master 1. candidate chooses one of the doors

70 / 161

SLIDE 71

Monty-Hall problem

pomdp-01

no prize prize no prize 3 3 3 doors initially closed candidate show master 1. candidate chooses one of the doors 2. show master opens a non-chosen, non-winning door

71 / 161

SLIDE 72

Monty-Hall problem

pomdp-01

no prize prize no prize 3 3 3 doors initially closed candidate show master 1. candidate chooses one of the doors 2. show master opens a non-chosen, non-winning door 3. candidate has the choice:

keep the choice
r
switch to the other (still closed) door

72 / 161

SLIDE 73

Monty-Hall problem

pomdp-01

no prize 100.000 100.000 100.000 Euro no prize 3 3 3 doors initially closed candidate show master 1. candidate chooses one of the doors 2. show master opens a non-chosen, non-winning door 3. candidate has the choice:

keep the choice
r
switch to the other (still closed) door

4. show master opens all doors

73 / 161

SLIDE 74

Monty-Hall problem

pomdp-01

no prize 100.000 100.000 100.000 Euro no prize 3 3 3 doors initially closed candidate show master

ptimal strategy for the candidate:

initial choice of the door: arbitrary revision of the initial choice (switch) probability for getting the prize: 2

3 2 3 2 3

74 / 161

SLIDE 75

MDP for the Monty-Hall problem

pomdp-02 75 / 161

SLIDE 76

MDP for the Monty-Hall problem

pomdp-02

no prize prize no prize 3 3 3 doors initially closed candidate’s actions

1. choose one door
3. keep or switch ?

show master’s actions

2. opens a non-chosen,

non-winning door

4. opens all doors

76 / 161

SLIDE 77

MDP for the Monty-Hall problem

pomdp-02

no prize prize no prize 3 3 3 doors initially closed candidate’s actions

1. choose one door
3. keep or switch ?

show master’s actions

2. opens a non-chosen,

non-winning door

4. opens all doors

✟✟✟✟✟✟✟✟✟✟✟✟ ✟ ❍❍❍❍❍❍❍❍❍❍❍❍ ❍ ✟✟✟✟✟✟✟✟✟✟✟✟ ✟ ❍❍❍❍❍❍❍❍❍❍❍❍ ❍ ✟✟✟✟✟✟✟✟✟✟✟✟ ✟ ❍❍❍❍❍❍❍❍❍❍❍❍ ❍

start door1

1 1

door2

2 2

door3

3 3

lost won

1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3

keep switch keep switch keep switch

77 / 161

SLIDE 78

MDP for the Monty-Hall problem

pomdp-02

no prize prize no prize 3 3 3 doors initially closed candidate’s actions

1. choose one door
3. keep or switch ?

Prmax(start, ♦won) , ♦won) , ♦won) = 1 Prmax(start, ♦won) , ♦won) , ♦won) = 1 Prmax(start, ♦won) , ♦won) , ♦won) = 1

ptimal scheduler requires

complete information

n the states

start door1

1 1

door2

2 2

door3

3 3

lost won

1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3

won switch keep switch

78 / 161

SLIDE 79

MDP for the Monty-Hall problem

pomdp-02

no prize prize no prize 3 3 3 doors initially closed candidate’s actions

1. choose one door
3. keep or switch ?

cannot be distinguished by the candidate start door1

1 1

door2

2 2

door3

3 3

lost won

1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3

won switch keep switch keep switch keep

79 / 161

SLIDE 80

MDP for the Monty-Hall problem

pomdp-02

no prize prize no prize 3 3 3 doors initially closed candidate’s actions

1. choose one door
3. keep or switch ?
bservation-based strategy:

choose action switch in state doori

i i

start door1

1 1

door2

2 2

door3

3 3

lost won

1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3

won switch switch switch

80 / 161

SLIDE 81

MDP for the Monty-Hall problem

pomdp-02

no prize prize no prize 3 3 3 doors initially closed candidate’s actions

1. choose one door
3. keep or switch ?
bservation-based strategy:

choose action switch in state doori

i i

probability for ♦won ♦won ♦won: 2

3 2 3 2 3

start door1

1 1

door2

2 2

door3

3 3

lost won won

1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3

switch switch switch

81 / 161

SLIDE 82

Partially-observable Markov decision process

pomdp-05

A partially-observable MDP (POMDP for short) is an MDP M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .) together with an equivalence relation ∼ ∼ ∼ on S S S

82 / 161

SLIDE 83

Partially-observable Markov decision process

pomdp-05

A partially-observable MDP (POMDP for short) is an MDP M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .) together with an equivalence relation ∼ ∼ ∼ on S S S





if s1 ∼ s2 s1 ∼ s2 s1 ∼ s2 then s1, s2 s1, s2 s1, s2 cannot be distinguished from outside (or by the scheduler)

bservables: equivalence classes of states

83 / 161

SLIDE 84

Partially-observable Markov decision process

pomdp-05

A partially-observable MDP (POMDP for short) is an MDP M = (S, Act, P, . . .) M = (S, Act, P, . . .) M = (S, Act, P, . . .) together with an equivalence relation ∼ ∼ ∼ on S S S





if s1 ∼ s2 s1 ∼ s2 s1 ∼ s2 then s1, s2 s1, s2 s1, s2 cannot be distinguished from outside (or by the scheduler)

bservables: equivalence classes of states
bservation-based scheduler:

scheduler D : S∗ → Act D : S∗ → Act D : S∗ → Act such that for all π1, π2 ∈ S∗ π1, π2 ∈ S∗ π1, π2 ∈ S∗: D(π1) = D(π2) D(π1) = D(π2) D(π1) = D(π2) if obs(π1) = obs(π2)

bs(π1) = obs(π2)
bs(π1) = obs(π2)

where obs(s0 s1 . . . sn) = [s0] [s1] . . . [sn]

bs(s0 s1 . . . sn) = [s0] [s1] . . . [sn]
bs(s0 s1 . . . sn) = [s0] [s1] . . . [sn]

84 / 161

SLIDE 85

Extreme cases of POMDP

pomdp-11

extreme cases of an POMDP:

s1 ∼ s2

s1 ∼ s2 s1 ∼ s2 iff s1 = s2 s1 = s2 s1 = s2

s1 ∼ s2

s1 ∼ s2 s1 ∼ s2 for all s1 s1 s1, s2 s2 s2

85 / 161

SLIDE 86

Extreme cases of POMDP

pomdp-11

extreme cases of an POMDP:

s1 ∼ s2

s1 ∼ s2 s1 ∼ s2 iff s1 = s2 s1 = s2 s1 = s2 ← − ← − ← − standard MDP

s1 ∼ s2

s1 ∼ s2 s1 ∼ s2 for all s1 s1 s1, s2 s2 s2

86 / 161

SLIDE 87

Probabilistic automata are special POMDP

pomdp-11

extreme cases of an POMDP:

s1 ∼ s2

s1 ∼ s2 s1 ∼ s2 iff s1 = s2 s1 = s2 s1 = s2 ← − ← − ← − standard MDP

s1 ∼ s2

s1 ∼ s2 s1 ∼ s2 for all s1 s1 s1, s2 s2 s2 ← − ← − ← − probabilistic automata note that for totally non-observable POMDP:

bservation-based

scheduler

=
=
=

function D : N → Act D : N → Act D : N → Act

=
=
= infinite word
ver Act

Act Act

87 / 161

SLIDE 88

Undecidability results for POMDP

pomdp-11

extreme cases of an POMDP:

s1 ∼ s2

s1 ∼ s2 s1 ∼ s2 iff s1 = s2 s1 = s2 s1 = s2 ← − ← − ← − standard MDP

s1 ∼ s2

s1 ∼ s2 s1 ∼ s2 for all s1 s1 s1, s2 s2 s2 ← − ← − ← − probabilistic automata note that for totally non-observable POMDP:

bservation-based

scheduler

=
=
=

function D : N → Act D : N → Act D : N → Act

=
=
= infinite word
ver Act

Act Act undecidability results for PFA carry over to POMDP maximum probabilistic reachability problem “does Probs

max(♦F) > p

Probs

max(♦F) > p

Probs

max(♦F) > p hold ?”

=
=
=

non-emptiness problem for PFA

88 / 161

SLIDE 89

Undecidability results for POMDP

pomdp-30-new

The model checking problem for POMDP and

quantitative properties is undecidable, e.g., probabilistic reachability properties.

89 / 161

SLIDE 90

Undecidability results for POMDP

pomdp-30-new

The model checking problem for POMDP and

quantitative properties is undecidable, e.g., probabilistic reachability properties.

There is no even no approximation algorithm for

reachability objectives.

[Paz’71], [Madani/Hanks/Condon’99], [Giro/d’Argenio’07]

90 / 161

SLIDE 91

Undecidability results for POMDP

pomdp-30-new

The model checking problem for POMDP and

quantitative properties is undecidable, e.g., probabilistic reachability properties. ♦ ♦ ♦ =

=
= “infinitely often”
There is no even no approximation algorithm for

reachability objectives.

The model checking problem for POMDP and several

qualitative properties is undecidable, e.g., repeated reachability with positive probability “does Probs

max(♦F)

Probs

max(♦F)

Probs

max(♦F) > 0

> 0 > 0 hold ?”

91 / 161

SLIDE 92

Undecidability results for POMDP

pomdp-30-new

The model checking problem for POMDP and

quantitative properties is undecidable, e.g., probabilistic reachability properties.

There is no even no approximation algorithm for

reachability objectives.

The model checking problem for POMDP and several

qualitative properties is undecidable, e.g., repeated reachability with positive probability “does Probs

max(♦F)

Probs

max(♦F)

Probs

max(♦F) > 0

> 0 > 0 hold ?” Many interesting verification problems for distributed probabilistic multi-agent systems are undecidable.

92 / 161

SLIDE 93

Undecidability results for POMDP

pomdp-30-new

The model checking problem for POMDP and

quantitative properties is undecidable, e.g., probabilistic reachability properties.

There is no even no approximation algorithm for

reachability objectives.

The model checking problem for POMDP and several

qualitative properties is undecidable, e.g., repeated reachability with positive probability “does Probs

max(♦F)

Probs

max(♦F)

Probs

max(♦F) > 0

> 0 > 0 hold ?” ... already holds for totally non-observable POMDP

probabilistic B¨

uchi automata

93 / 161

SLIDE 94

Remind: LTL model checking for MDP

pomdp-50

2exp MDP M M M requirements LTL formula ϕ ϕ ϕ deterministic automaton A A A probabilistic reachability analysis in the product-MDP M × A M × A M × A

94 / 161

SLIDE 95

PA rather than DA ?

pomdp-50

? MDP M M M requirements LTL formula ϕ ϕ ϕ probabilistic automaton A A A probabilistic reachability analysis in the product-MDP M × A M × A M × A

95 / 161

SLIDE 96

PA rather than DA ?

pomdp-50

? MDP M M M requirements LTL formula ϕ ϕ ϕ probabilistic automaton A A A probabilistic reachability analysis in the product-MDP M × A M × A M × A impossible, due to undecidability results

96 / 161

SLIDE 97

Decidability results for POMDP

pomdp-15-fm 97 / 161

SLIDE 98

Decidability results for POMDP

pomdp-15-ifm

The model checking problem for POMDP and several qualitative properties is decidable, e.g.,

invariance with positive probability

“does Probs

max(F)

Probs

max(F)

Probs

max(F) > 0

> 0 > 0 hold ?”

almost-sure reachability

“does Probs

max(♦F)

Probs

max(♦F)

Probs

max(♦F) = 1

= 1 = 1 hold ?”

almost-sure repeated reachability

“does Probs

max(♦F)

Probs

max(♦F)

Probs

max(♦F) = 1

= 1 = 1 hold ?”

persistence with positive probability

“does Probs

max(♦F)

Probs

max(♦F)

Probs

max(♦F) > 0

> 0 > 0 hold ?”

98 / 161

SLIDE 99

Decidability results for POMDP

pomdp-15-ifm

The model checking problem for POMDP and several qualitative properties is decidable, e.g.,

invariance with positive probability

“does Probs

max(F)

Probs

max(F)

Probs

max(F) > 0

> 0 > 0 hold ?”

almost-sure reachability

“does Probs

max(♦F)

Probs

max(♦F)

Probs

max(♦F) = 1

= 1 = 1 hold ?”

almost-sure repeated reachability

“does Probs

max(♦F)

Probs

max(♦F)

Probs

max(♦F) = 1

= 1 = 1 hold ?”

persistence with positive probability

“does Probs

max(♦F)

Probs

max(♦F)

Probs

max(♦F) > 0

> 0 > 0 hold ?” algorithms use a certain powerset construction

99 / 161

SLIDE 100

Probabilistic Automata and Verification

verview-conc
Markov decision processes (MDP) and

quantitative analysis against path events

partial order reduction for MDP
partially-oberservable MDP
conclusions

← − ← − ← −

100 / 161

SLIDE 101

Conclusion

conc

worst/best-case analysis of MDP solvable by

∗ ∗ ∗ numerical methods for solving linear programs ∗ ∗ ∗ known techniques for non-probabilistic systems





graph algorithms, LTL-2-AUT translators, . . . . . . . . . techniques to combat the state explosion problem (such as partial order reduction)

101 / 161

SLIDE 102

Conclusion

conc

worst/best-case analysis of MDP solvable by

∗ ∗ ∗ numerical methods for solving linear programs ∗ ∗ ∗ known techniques for non-probabilistic systems





graph algorithms, LTL-2-AUT translators, . . . . . . . . . techniques to combat the state explosion problem (such as partial order reduction) but: strongly simplified definition of schedulers





assumption “full knowledge of the history” is inadequate, e.g., for agents of distributed systems

102 / 161

SLIDE 103

Conclusion

conc

worst/best-case analysis of MDP solvable by

∗ ∗ ∗ numerical methods for solving linear programs ∗ ∗ ∗ known techniques for non-probabilistic systems

more realistic model: partially-observable MDP

and multi-agents variants with distributed schedulers

103 / 161

SLIDE 104

Conclusion

conc

worst/best-case analysis of MDP solvable by

∗ ∗ ∗ numerical methods for solving linear programs ∗ ∗ ∗ known techniques for non-probabilistic systems

more realistic model: partially-observable MDP

and multi-agents variants with distributed schedulers − − − many algorithms for “finite-horizon properties” − − − few decidability results for qualitative properties − − − undecidability for quantitative properties and, e.g., repeated reachability with positive probability

104 / 161

SLIDE 105

Conclusion

conc

worst/best-case analysis of MDP solvable by

∗ ∗ ∗ numerical methods for solving linear programs ∗ ∗ ∗ known techniques for non-probabilistic systems

more realistic model: partially-observable MDP

and multi-agents variants with distributed schedulers − − − many algorithms for “finite-horizon properties” − − − few decidability results for qualitative properties − − − undecidability for quantitative properties and, e.g., repeated reachability with positive probability











 proof via probabilistic language acceptors (PFA/PBA)

105 / 161

SLIDE 106

Conclusion

conc

worst/best-case analysis of MDP solvable by

∗ ∗ ∗ numerical methods for solving linear programs ∗ ∗ ∗ known techniques for non-probabilistic systems

more realistic model: partially-observable MDP

and multi-agents variants with distributed schedulers − − − many algorithms for “finite-horizon properties” − − − few decidability results for qualitative properties − − − undecidability for quantitative properties and, e.g., repeated reachability with positive probability

probabilistic B¨

uchi automata interesting in their own . . . . . . . . .

106 / 161

SLIDE 107

Probabilistic B¨ uchi automaton (PBA)

pba-01

P = (Q, Σ, δ, µ, F) P = (Q, Σ, δ, µ, F) P = (Q, Σ, δ, µ, F)

Q

Q Q finite state space

Σ

Σ Σ alphabet

δ : Q × Σ × Q → [0, 1]

δ : Q × Σ × Q → [0, 1] δ : Q × Σ × Q → [0, 1] s.t. for all q ∈ Q q ∈ Q q ∈ Q, a ∈ Σ a ∈ Σ a ∈ Σ:

p∈Q

δ(q, a, p) ∈ {0, 1}

p∈Q

δ(q, a, p) ∈ {0, 1}

p∈Q

δ(q, a, p) ∈ {0, 1}

initial distribution µ

µ µ

set of final states F ⊆ Q

F ⊆ Q F ⊆ Q

107 / 161

SLIDE 108

Probabilistic B¨ uchi automaton (PBA)

pba-01

POMDP where Σ = Act Σ = Act Σ = Act and ∼ ∼ ∼ =

=
= Q × Q

Q × Q Q × Q P = (Q, Σ, δ, µ, F) P = (Q, Σ, δ, µ, F) P = (Q, Σ, δ, µ, F) ← − ← − ← −

Q

Q Q finite state space

Σ

Σ Σ alphabet

δ : Q × Σ × Q → [0, 1]

δ : Q × Σ × Q → [0, 1] δ : Q × Σ × Q → [0, 1] s.t. for all q ∈ Q q ∈ Q q ∈ Q, a ∈ Σ a ∈ Σ a ∈ Σ:

p∈Q

δ(q, a, p) ∈ {0, 1}

p∈Q

δ(q, a, p) ∈ {0, 1}

p∈Q

δ(q, a, p) ∈ {0, 1}

initial distribution µ

µ µ

set of final states F ⊆ Q

F ⊆ Q F ⊆ Q

108 / 161

SLIDE 109

Probabilistic B¨ uchi automaton (PBA)

pba-01

POMDP where Σ = Act Σ = Act Σ = Act and ∼ ∼ ∼ =

=
= Q × Q

Q × Q Q × Q P = (Q, Σ, δ, µ, F) P = (Q, Σ, δ, µ, F) P = (Q, Σ, δ, µ, F) ← − ← − ← −

Q

Q Q finite state space

Σ

Σ Σ alphabet

δ : Q × Σ × Q → [0, 1]

δ : Q × Σ × Q → [0, 1] δ : Q × Σ × Q → [0, 1] s.t. for all q ∈ Q q ∈ Q q ∈ Q, a ∈ Σ a ∈ Σ a ∈ Σ:

p∈Q

δ(q, a, p) ∈ {0, 1}

p∈Q

δ(q, a, p) ∈ {0, 1}

p∈Q

δ(q, a, p) ∈ {0, 1}

initial distribution µ

µ µ

set of final states F ⊆ Q

F ⊆ Q F ⊆ Q For each infinite word x ∈ Σω x ∈ Σω x ∈ Σω: Pr(x) = Pr(x) = Pr(x) = probability for the accepting runs for x x x ↑ ↑ ↑ accepting run: visits F F F infinitely often

109 / 161

SLIDE 110

Probabilistic B¨ uchi automaton (PBA)

pba-01

POMDP where Σ = Act Σ = Act Σ = Act and ∼ ∼ ∼ =

=
= Q × Q

Q × Q Q × Q P = (Q, Σ, δ, µ, F) P = (Q, Σ, δ, µ, F) P = (Q, Σ, δ, µ, F) ← − ← − ← −

Q

Q Q finite state space

Σ

Σ Σ alphabet

δ : Q × Σ × Q → [0, 1]

δ : Q × Σ × Q → [0, 1] δ : Q × Σ × Q → [0, 1] s.t. for all q ∈ Q q ∈ Q q ∈ Q, a ∈ Σ a ∈ Σ a ∈ Σ:

p∈Q

δ(q, a, p) ∈ {0, 1}

p∈Q

δ(q, a, p) ∈ {0, 1}

p∈Q

δ(q, a, p) ∈ {0, 1}

initial distribution µ

µ µ

set of final states F ⊆ Q

F ⊆ Q F ⊆ Q For each infinite word x ∈ Σω x ∈ Σω x ∈ Σω: Pr(x) = Pr(x) = Pr(x) = probability for the accepting runs for x x x ↑ ↑ ↑ probability measure in the infinite Markov chain induced by x x x viewed as a scheduler

110 / 161

SLIDE 111

Accepted language of a PBA

pba-03

P = (Q, Σ, δ, µ, F) P = (Q, Σ, δ, µ, F) P = (Q, Σ, δ, µ, F)

Q

Q Q finite state space, Σ Σ Σ alphabet

δ : Q × Σ × Q → [0, 1]

δ : Q × Σ × Q → [0, 1] δ : Q × Σ × Q → [0, 1] s.t. . . . . . . . . .

initial distribution µ

µ µ

set of final states F ⊆ Q

F ⊆ Q F ⊆ Q three types of accepted language: L>0(P) L>0(P) L>0(P) = = =

x ∈ Σω : Pr(x) > 0
x ∈ Σω : Pr(x) > 0
x ∈ Σω : Pr(x) > 0
probable semantics

L=1(P) L=1(P) L=1(P) = = =

x ∈ Σω : Pr(x) = 1
x ∈ Σω : Pr(x) = 1
x ∈ Σω : Pr(x) = 1
almost-sure sem.

L>λ(P) L>λ(P) L>λ(P) = = =

x ∈ Σω : Pr(x) > λ
x ∈ Σω : Pr(x) > λ
x ∈ Σω : Pr(x) > λ
threshold semantics

where 0 < λ < 1 0 < λ < 1 0 < λ < 1

111 / 161

SLIDE 112

Example for PBA

pba-5

1 1 1 2 2 2 a a a, 1

2 1 2 1 2

b b b a a a a a a, 1

2 1 2 1 2

112 / 161

SLIDE 113

Example for PBA

pba-5

1 1 1 2 2 2 a a a, 1

2 1 2 1 2

b b b a a a a a a, 1

2 1 2 1 2

accepted language: L>0(P) L>0(P) L>0(P) = = = (a + b)∗aω (a + b)∗aω (a + b)∗aω

113 / 161

SLIDE 114

Example for PBA

pba-5

1 1 1 2 2 2 a a a, 1

2 1 2 1 2

b b b a a a a a a, 1

2 1 2 1 2

accepted language: L>0(P) L>0(P) L>0(P) = = = (a + b)∗aω (a + b)∗aω (a + b)∗aω L=1(P) L=1(P) L=1(P) = = = b∗aω b∗aω b∗aω

114 / 161

SLIDE 115

Example for PBA

pba-5

1 1 1 2 2 2 a a a, 1

2 1 2 1 2

b b b a a a a a a, 1

2 1 2 1 2

accepted language: L>0(P) L>0(P) L>0(P) = = = (a + b)∗aω (a + b)∗aω (a + b)∗aω L=1(P) L=1(P) L=1(P) = = = b∗aω b∗aω b∗aω Thus: PBA>0

>0 >0 are strictly more expressive than DBA

115 / 161

SLIDE 116

Example for PBA

pba-5

1 1 1 2 2 2 a a a, 1

2 1 2 1 2

b b b a a a a a a, 1

2 1 2 1 2

accepted language: L>0(P) L>0(P) L>0(P) = = = (a + b)∗aω (a + b)∗aω (a + b)∗aω L=1(P) L=1(P) L=1(P) = = = b∗aω b∗aω b∗aω Thus: PBA>0

>0 >0 are strictly more expressive than DBA

3 3 3 2 2 2 1 1 1 a a a, 1

2 1 2 1 2

a a a, 1

2 1 2 1 2

b b b c c c b b b

116 / 161

SLIDE 117

Example for PBA

pba-5

1 1 1 2 2 2 a a a, 1

2 1 2 1 2

b b b a a a a a a, 1

2 1 2 1 2

accepted language: L>0(P) L>0(P) L>0(P) = = = (a + b)∗aω (a + b)∗aω (a + b)∗aω L=1(P) L=1(P) L=1(P) = = = b∗aω b∗aω b∗aω Thus: PBA>0

>0 >0 are strictly more expressive than DBA

3 3 3 2 2 2 1 1 1 a a a, 1

2 1 2 1 2

a a a, 1

2 1 2 1 2

b b b c c c b b b NBA accepts ((ac)∗ab)ω ((ac)∗ab)ω ((ac)∗ab)ω

117 / 161

SLIDE 118

Example for PBA

pba-5

1 1 1 2 2 2 a a a, 1

2 1 2 1 2

b b b a a a a a a, 1

2 1 2 1 2

accepted language: L>0(P) L>0(P) L>0(P) = = = (a + b)∗aω (a + b)∗aω (a + b)∗aω L=1(P) L=1(P) L=1(P) = = = b∗aω b∗aω b∗aω Thus: PBA>0

>0 >0 are strictly more expressive than DBA

3 3 3 2 2 2 1 1 1 a a a, 1

2 1 2 1 2

a a a, 1

2 1 2 1 2

b b b c c c b b b accepted language: L>0(P) L>0(P) L>0(P) = = = (ab + ac)∗(ab)ω (ab + ac)∗(ab)ω (ab + ac)∗(ab)ω but NBA accepts ((ac)∗ab)ω ((ac)∗ab)ω ((ac)∗ab)ω

118 / 161

SLIDE 119

Example for PBA

pba-5

1 1 1 2 2 2 a a a, 1

2 1 2 1 2

b b b a a a a a a, 1

2 1 2 1 2

accepted language: L>0(P) L>0(P) L>0(P) = = = (a + b)∗aω (a + b)∗aω (a + b)∗aω L=1(P) L=1(P) L=1(P) = = = b∗aω b∗aω b∗aω Thus: PBA>0

>0 >0 are strictly more expressive than DBA

3 3 3 2 2 2 1 1 1 a a a, 1

2 1 2 1 2

a a a, 1

2 1 2 1 2

b b b c c c b b b accepted language: L>0(P) L>0(P) L>0(P) = = = (ab + ac)∗(ab)ω (ab + ac)∗(ab)ω (ab + ac)∗(ab)ω L=1(P) L=1(P) L=1(P) = = = (ab)ω (ab)ω (ab)ω but NBA accepts ((ac)∗ab)ω ((ac)∗ab)ω ((ac)∗ab)ω

119 / 161

SLIDE 120

Expressiveness of PBA with probable semantics

pba-10

120 / 161

SLIDE 121

Expressiveness of PBA with probable semantics

pba-10

PBA>0

>0 >0 are strictly more expressive than NBA

121 / 161

SLIDE 122

Expressiveness of PBA with probable semantics

pba-10

PBA>0

>0 >0 are strictly more expressive than NBA

from NBA to PBA: NBA NBA deterministic in limit Courcoubetis/ Yannakakis

=
=
= PBA>0

>0 >0

122 / 161

SLIDE 123

Expressiveness of PBA with probable semantics

pba-10

PBA>0

>0 >0 are strictly more expressive than NBA

from NBA to PBA: NBA NBA deterministic in limit Courcoubetis/ Yannakakis

=
=
= PBA>0

>0 >0

d d d a a a a a a c c c b b b b b b d d d

123 / 161

SLIDE 124

Expressiveness of PBA with probable semantics

pba-10

PBA>0

>0 >0 are strictly more expressive than NBA

from NBA to PBA: NBA NBA deterministic in limit Courcoubetis/ Yannakakis

=
=
= PBA>0

>0 >0

d d d a a a a a a c c c b b b b b b d d d deterministic

124 / 161

SLIDE 125

Expressiveness of PBA with probable semantics

pba-10

PBA>0

>0 >0 are strictly more expressive than NBA

from NBA to PBA: NBA NBA deterministic in limit Courcoubetis/ Yannakakis

=
=
= PBA>0

>0 >0

d d d a a a a a a c c c b b b b b b d d d deterministic

=
=
=

d, 1

2

d, 1

2

d, 1

2

b b b d, 1

2

d, 1

2

d, 1

2

a, 1

2

a, 1

2

a, 1

2

a, 1

2

a, 1

2

a, 1

2

b b b c c c

125 / 161

SLIDE 126

PBA>0

>0 >0 are strictly more expressive than NBA

pba-11

from NBA to PBA:

via NBA that are deterministic in limit

PBA can accept non-ω

ω ω-regular languages 2 2 2 1 1 1 a a a,1

2 1 2 1 2

a a a a a a, 1

2 1 2 1 2

b b b

126 / 161

SLIDE 127

PBA>0

>0 >0 are strictly more expressive than NBA

pba-11

from NBA to PBA:

via NBA that are deterministic in limit

PBA can accept non-ω

ω ω-regular languages 2 2 2 1 1 1 a a a,1

2 1 2 1 2

a a a a a a, 1

2 1 2 1 2

b b b accepted language (probable semantics): L>0(P) =

ak1bak2bak3b. . . |

L>0(P) =

ak1bak2bak3b. . . |

L>0(P) =

ak1bak2bak3b. . . |

. . . . . . . . .

127 / 161

SLIDE 128

PBA>0

>0 >0 are strictly more expressive than NBA

pba-11

from NBA to PBA:

via NBA that are deterministic in limit

PBA can accept non-ω

ω ω-regular languages 2 2 2 1 1 1 a a a,1

2 1 2 1 2

a a a a a a, 1

2 1 2 1 2

b b b accepted language (probable semantics): L>0(P) =

ak1bak2bak3b. . . |

L>0(P) =

ak1bak2bak3b. . . |

L>0(P) =

ak1bak2bak3b. . . |

∞

i=1
1 −

1

2

ki > 0

∞

i=1
1 −

1

2

ki > 0

∞

i=1
1 −

1

2

ki > 0

128 / 161

SLIDE 129

Expressiveness of PBA

pba-15

PBA>0

>0 >0

DBA

129 / 161

SLIDE 130

Expressiveness of PBA

pba-15

PBA>0

>0 >0

DBA NBA

130 / 161

SLIDE 131

Expressiveness of PBA

pba-15

PBA>0

>0 >0

DBA NBA PBA with thresholds

131 / 161

SLIDE 132

Expressiveness of PBA

pba-15

PBA>0

>0 >0

DBA NBA PBA with thresholds PBA=1

=1 =1

almost-sure semantics

132 / 161

SLIDE 133

Expressiveness of PBA

pba-15

PBA>0

>0 >0

DBA NBA PBA with thresholds PBA=1

=1 =1

almost-sure semantics (a + b)∗aω (a + b)∗aω (a + b)∗aω

133 / 161

SLIDE 134

Expressiveness of PBA

pba-15

PBA>0

>0 >0

DBA NBA PBA with thresholds PBA=1

=1 =1

almost-sure semantics (a + b)∗aω (a + b)∗aω (a + b)∗aω

ak1bak2b. . . :

∞

i=1

(1 − (1

2)ki) > 0

ak1bak2b. . . :

∞

i=1

(1 − (1

2)ki) > 0

ak1bak2b. . . :

∞

i=1

(1 − (1

2)ki) > 0

134 / 161

SLIDE 135

Expressiveness of PBA

pba-15

PBA>0

>0 >0

DBA NBA PBA with thresholds PBA=1

=1 =1

almost-sure semantics (a + b)∗aω (a + b)∗aω (a + b)∗aω

ak1bak2b. . . :

∞

i=1

(1 − (1

2)ki) > 0

ak1bak2b. . . :

∞

i=1

(1 − (1

2)ki) > 0

ak1bak2b. . . :

∞

i=1

(1 − (1

2)ki) > 0

ak1bak2b. . . :

∞

i=1

(1 − (1

2)ki) = 0

ak1bak2b. . . :

∞

i=1

(1 − (1

2)ki) = 0

ak1bak2b. . . :

∞

i=1

(1 − (1

2)ki) = 0

135 / 161

SLIDE 136

Expressiveness of PBA

pba-15

PBA>0

>0 >0

DBA NBA PBA with thresholds PBA=1

=1 =1

almost-sure semantics (a + b)∗aω (a + b)∗aω (a + b)∗aω emptiness problem: undecidable for PBA>0

>0 >0

decidable for PBA=1

=1 =1

136 / 161

SLIDE 137

Decidability results for POMDP

pomdp-16

The model checking problem for POMDP and several qualitative properties is decidable:

almost-sure reachability

“does Probs

max(♦F)

Probs

max(♦F)

Probs

max(♦F) = 1

= 1 = 1 hold ?”

invariance with positive probability

“does Probs

max(F)

Probs

max(F)

Probs

max(F) > 0

> 0 > 0 hold ?”

almost-sure repeated reachability

“does Probs

max(♦F)

Probs

max(♦F)

Probs

max(♦F) = 1

= 1 = 1 hold ?”

persistence with positive probability

“does Probs

max(♦F)

Probs

max(♦F)

Probs

max(♦F) > 0

> 0 > 0 hold ?” algorithms use a certain powerset construction

137 / 161

SLIDE 138

Decidability results for POMDP

pomdp-16

The model checking problem for POMDP and several qualitative properties is decidable:

almost-sure reachability

“does Probs

max(♦F)

Probs

max(♦F)

Probs

max(♦F) = 1

= 1 = 1 hold ?”

invariance with positive probability

“does Probs

max(F)

Probs

max(F)

Probs

max(F) > 0

> 0 > 0 hold ?”

almost-sure repeated reachability

“does Probs

max(♦F)

Probs

max(♦F)

Probs

max(♦F) = 1

= 1 = 1 hold ?”

persistence with positive probability

“does Probs

max(♦F)

Probs

max(♦F)

Probs

max(♦F) > 0

> 0 > 0 hold ?” algorithms use a certain powerset construction

138 / 161

SLIDE 139

Almost-sure reachability/repeated reachability

pomdp-17

The almost-sure repeated reachability problem “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

is polynomially reducible to the almost-sure reachability problem “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

139 / 161

SLIDE 140

Almost-sure reachability/repeated reachability

pomdp-17

The almost-sure repeated reachability problem “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

is polynomially reducible to the almost-sure reachability problem “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

F F F POMDP M M M

bjective:

repeated reachability ♦F ♦F ♦F

140 / 161

SLIDE 141

Almost-sure reachability/repeated reachability

pomdp-17

The almost-sure repeated reachability problem “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

is polynomially reducible to the almost-sure reachability problem “does Probs

max(♦f ) = 1

Probs

max(♦f ) = 1

Probs

max(♦f ) = 1 hold ?”

F F F POMDP M M M

bjective:

repeated reachability ♦F ♦F ♦F f f f POMDP M′ M′ M′

bjective:

reachability ♦f ♦f ♦f

141 / 161

SLIDE 142

Almost-sure reachability/repeated reachability

pomdp-17

The almost-sure repeated reachability problem “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

is polynomially reducible to the almost-sure reachability problem “does Probs

max(♦f ) = 1

Probs

max(♦f ) = 1

Probs

max(♦f ) = 1 hold ?”

s ∈ F s ∈ F s ∈ F

1 3 1 3 1 3 2 3 2 3 2 3

POMDP M M M

bjective:

repeated reachability ♦F ♦F ♦F f f f POMDP M′ M′ M′

bjective:

reachability ♦f ♦f ♦f s ∈ F s ∈ F s ∈ F

1 6 1 6 1 6 1 3 1 3 1 3 1 2 1 2 1 2

142 / 161

SLIDE 143

Almost-sure reachability

pomdp-18

powerset construction for almost-sure reachability “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

143 / 161

SLIDE 144

Almost-sure reachability

pomdp-18

powerset construction for almost-sure reachability “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

POMDP M M M with equivalence ∼ ∼ ∼ − → − → − → MDP Pow(M) Pow(M) Pow(M) fully observable

144 / 161

SLIDE 145

Almost-sure reachability

pomdp-18

powerset construction for almost-sure reachability “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

POMDP M M M with equivalence ∼ ∼ ∼ − → − → − → MDP Pow(M) Pow(M) Pow(M) fully observable Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 in M

M M iff Prmax(♦F ′) = 1 Prmax(♦F ′) = 1 Prmax(♦F ′) = 1 in Pow(M) Pow(M) Pow(M)

145 / 161

SLIDE 146

Almost-sure reachability

pomdp-18

powerset construction for almost-sure reachability “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

POMDP M M M with equivalence ∼ ∼ ∼ − → − → − → MDP Pow(M) Pow(M) Pow(M) fully observable Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 in M

M M iff Prmax(♦F ′) = 1 Prmax(♦F ′) = 1 Prmax(♦F ′) = 1 in Pow(M) Pow(M) Pow(M) state s s s in M M M → → → states s, R s, R s, R where s ∈ R ⊆ [s] s ∈ R ⊆ [s] s ∈ R ⊆ [s] [s] = [s] = [s] = equivalence class of s s s w.r.t. ∼ ∼ ∼

146 / 161

SLIDE 147

Almost-sure reachability

pomdp-18

powerset construction for almost-sure reachability “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

POMDP M M M with equivalence ∼ ∼ ∼ − → − → − → MDP Pow(M) Pow(M) Pow(M) fully observable Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 in M

M M iff Prmax(♦f ) = 1 Prmax(♦f ) = 1 Prmax(♦f ) = 1 in Pow(M) Pow(M) Pow(M) state s s s in M M M → → → states s, R s, R s, R where s ∈ R ⊆ [s] s ∈ R ⊆ [s] s ∈ R ⊆ [s] fresh goal state f f f [s] = [s] = [s] = equivalence class of s s s w.r.t. ∼ ∼ ∼

147 / 161

SLIDE 148

Almost-sure reachability

pomdp-19

powerset construction for almost-sure reachability “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

state s s s in M M M action α α α

148 / 161

SLIDE 149

Almost-sure reachability

pomdp-19

powerset construction for almost-sure reachability “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

state s s s in M M M action α α α if Post(s, α) ∩ F = ∅ Post(s, α) ∩ F = ∅ Post(s, α) ∩ F = ∅

149 / 161

SLIDE 150

Almost-sure reachability

pomdp-19

powerset construction for almost-sure reachability “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

state s s s in M M M action α α α if Post(s, α) ∩ F = ∅ Post(s, α) ∩ F = ∅ Post(s, α) ∩ F = ∅ state s, R s, R s, R in Pow(M) Pow(M) Pow(M) where s ∈ R ⊆ [s] s ∈ R ⊆ [s] s ∈ R ⊆ [s] action α α α

150 / 161

SLIDE 151

Almost-sure reachability

pomdp-19

powerset construction for almost-sure reachability “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

state s s s in M M M action α α α t t t if Post(s, α) ∩ F = ∅ Post(s, α) ∩ F = ∅ Post(s, α) ∩ F = ∅ state s, R s, R s, R in Pow(M) Pow(M) Pow(M) where s ∈ R ⊆ [s] s ∈ R ⊆ [s] s ∈ R ⊆ [s] action α α α state t, . . .

t, . . .
t, . . .
t ∈ Post(s)

t ∈ Post(s) t ∈ Post(s)

151 / 161

SLIDE 152

Almost-sure reachability

pomdp-19

powerset construction for almost-sure reachability “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

state s s s in M M M action α α α t t t if Post(s, α) ∩ F = ∅ Post(s, α) ∩ F = ∅ Post(s, α) ∩ F = ∅ state s, R s, R s, R in Pow(M) Pow(M) Pow(M) where s ∈ R ⊆ [s] s ∈ R ⊆ [s] s ∈ R ⊆ [s] U = Post(R, α) U = Post(R, α) U = Post(R, α) action α α α state t, U ∩ [t] t, U ∩ [t] t, U ∩ [t] t ∈ Post(s) t ∈ Post(s) t ∈ Post(s)

152 / 161

SLIDE 153

Almost-sure reachability

pomdp-19

powerset construction for almost-sure reachability “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

state s s s in M M M action α α α t t t if Post(s, α) ∩ F = ∅ Post(s, α) ∩ F = ∅ Post(s, α) ∩ F = ∅ state s, R s, R s, R in Pow(M) Pow(M) Pow(M) action α α α state t, U ∩ [t] t, U ∩ [t] t, U ∩ [t] P(s, α, t) = P′(s, R, α, t, U ∩ [t]) P(s, α, t) = P′(s, R, α, t, U ∩ [t]) P(s, α, t) = P′(s, R, α, t, U ∩ [t])

153 / 161

SLIDE 154

Almost-sure reachability

pomdp-19

powerset construction for almost-sure reachability “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

state s s s in M M M action α α α v ∈ F v ∈ F v ∈ F if Post(s, α) ∩ F = ∅ Post(s, α) ∩ F = ∅ Post(s, α) ∩ F = ∅ where s ∈ R ⊆ [s] s ∈ R ⊆ [s] s ∈ R ⊆ [s] U = Post(R, α) U = Post(R, α) U = Post(R, α)

154 / 161

SLIDE 155

Almost-sure reachability

pomdp-19

powerset construction for almost-sure reachability “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

s s s ∼ s′ ∼ s′ ∼ s′ action α α α v ∈ F v ∈ F v ∈ F if Post(s, α) ∩ F = ∅ Post(s, α) ∩ F = ∅ Post(s, α) ∩ F = ∅ where s′ s′ s′, s ∈ R ⊆ [s] s ∈ R ⊆ [s] s ∈ R ⊆ [s] U = Post(R, α) U = Post(R, α) U = Post(R, α)

155 / 161

SLIDE 156

Almost-sure reachability

pomdp-19

powerset construction for almost-sure reachability “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

s s s ∼ s′ ∼ s′ ∼ s′ action α α α v ∈ F v ∈ F v ∈ F u ∈ F u ∈ F u ∈ F if Post(s, α) ∩ F = ∅ Post(s, α) ∩ F = ∅ Post(s, α) ∩ F = ∅ where s′ s′ s′, s ∈ R ⊆ [s] s ∈ R ⊆ [s] s ∈ R ⊆ [s] U = Post(R, α) U = Post(R, α) U = Post(R, α) state s, R s, R s, R

156 / 161

SLIDE 157

Almost-sure reachability

pomdp-19

powerset construction for almost-sure reachability “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

s s s ∼ s′ ∼ s′ ∼ s′ action α α α v ∈ F v ∈ F v ∈ F u ∈ F u ∈ F u ∈ F if Post(s, α) ∩ F = ∅ Post(s, α) ∩ F = ∅ Post(s, α) ∩ F = ∅ where s′ s′ s′, s ∈ R ⊆ [s] s ∈ R ⊆ [s] s ∈ R ⊆ [s] U = Post(R, α) U = Post(R, α) U = Post(R, α) state s, R s, R s, R t t t state t, U ∩ [t] t, U ∩ [t] t, U ∩ [t] where t ∈ U \ F t ∈ U \ F t ∈ U \ F

157 / 161

SLIDE 158

Almost-sure reachability

pomdp-19

powerset construction for almost-sure reachability “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

s s s ∼ s′ ∼ s′ ∼ s′ action α α α v ∈ F v ∈ F v ∈ F u ∈ F u ∈ F u ∈ F if Post(s, α) ∩ F = ∅ Post(s, α) ∩ F = ∅ Post(s, α) ∩ F = ∅ where s′ s′ s′, s ∈ R ⊆ [s] s ∈ R ⊆ [s] s ∈ R ⊆ [s] U = Post(R, α) U = Post(R, α) U = Post(R, α) state s, R s, R s, R t / ∈ F t / ∈ F t / ∈ F state t, U ∩ [t] t, U ∩ [t] t, U ∩ [t] where t ∈ U \ F t ∈ U \ F t ∈ U \ F

158 / 161

SLIDE 159

Almost-sure reachability

pomdp-19

powerset construction for almost-sure reachability “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

s s s ∼ s′ ∼ s′ ∼ s′ action α α α v ∈ F v ∈ F v ∈ F u ∈ F u ∈ F u ∈ F if Post(s, α) ∩ F = ∅ Post(s, α) ∩ F = ∅ Post(s, α) ∩ F = ∅ where s′ s′ s′, s ∈ R ⊆ [s] s ∈ R ⊆ [s] s ∈ R ⊆ [s] U = Post(R, α) U = Post(R, α) U = Post(R, α) state s, R s, R s, R f f f

bjective: ♦f

♦f ♦f

159 / 161

SLIDE 160

Almost-sure reachability

pomdp-19

powerset construction for almost-sure reachability “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

s s s ∼ s′ ∼ s′ ∼ s′ action α α α u ∈ F u ∈ F u ∈ F if Post(s, α) ∩ F = ∅ Post(s, α) ∩ F = ∅ Post(s, α) ∩ F = ∅ state s, R s, R s, R

1 2K 1 2K 1 2K 1 2K 1 2K 1 2K

f f f

1 2 1 2 1 2

P′(s, R, α, t, U ∩ [t]) =

1 2K

P′(s, R, α, t, U ∩ [t]) =

1 2K

P′(s, R, α, t, U ∩ [t]) =

1 2K

where K = |Post(R, α) \ F| K = |Post(R, α) \ F| K = |Post(R, α) \ F|

160 / 161

SLIDE 161

Almost-sure reachability

pomdp-19

powerset construction for almost-sure reachability “does Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1 hold ?”

s s s ∼ s′ ∼ s′ ∼ s′ action α α α u ∈ F u ∈ F u ∈ F state s, R s, R s, R

1 2K 1 2K 1 2K 1 2K 1 2K 1 2K

f f f

1 2 1 2 1 2

Probs

max(♦F) = 1

Probs

max(♦F) = 1

Probs

max(♦F) = 1

iff Prmax(♦f ) = 1 Prmax(♦f ) = 1 Prmax(♦f ) = 1 ↑ ↑ ↑ ↑ ↑ ↑

riginal

POMDP M M M fully-observable MDP Pow(M) Pow(M) Pow(M)

161 / 161