Randomized mutual exclusion protocol mdp-05 MDP n 1 n 2 n 1 n 2 n 1 n 2 request 2 request 2 request 2 request 1 request 1 request 1 release 2 release 2 release 2 release 1 release 1 release 1 w 1 n 2 w 1 n 2 w 1 n 2 n 1 w 2 n 1 w 2 n 1 w 2 request 2 request 2 request 2 request 1 request 1 request 1 e e e n n n r r r e e 1 1 1 e t t t t e t t e e n n n r r r e e e 2 2 2 w 1 w 2 w 1 w 2 w 1 w 2 w 1 w 2 w 1 w 2 w 1 w 2 r r r e e e e 1 e 1 e 1 l l l c 1 n 2 c 1 n 2 c 1 n 2 e s s s n 1 c 2 n 1 c 2 n 1 c 2 e e a a a a a a e e e s s s l l l e 2 e 2 e 2 1 1 1 1 1 1 e e e r r r toss a r r 2 2 2 toss a toss a 2 2 2 r e e e t t t q q q 1 1 1 s s s e u u u e e coin coin coin e e u u u e q q s s s q t t t e e e r r r 2 2 2 c 1 w 2 c 1 w 2 c 1 w 2 w 1 c 2 w 1 c 2 w 1 c 2 • interleaving of the request operations • competition if both processes are waiting • randomized arbiter tosses a coin if both are waiting 21 / 124
Randomized mutual exclusion protocol mdp-05 MDP n 1 n 2 n 1 n 2 n 1 n 2 request 2 request 2 request 2 request 1 request 1 request 1 release 2 release 2 release 2 release 1 release 1 release 1 w 1 n 2 w 1 n 2 w 1 n 2 n 1 w 2 n 1 w 2 n 1 w 2 request 2 request 2 request 2 request 1 request 1 request 1 e e e n n n r r r e e 1 1 1 e t t t t e t t e e n n n r r r e e e 2 2 2 w 1 w 2 w 1 w 2 w 1 w 2 w 1 w 2 w 1 w 2 w 1 w 2 r r r e e e e 1 e 1 e 1 l l l c 1 n 2 c 1 n 2 c 1 n 2 e s s s n 1 c 2 n 1 c 2 n 1 c 2 e e a a a a a a e e e s s s l l l e 2 e 2 e 2 1 1 1 1 1 1 e e e r r r toss a r r 2 2 2 toss a toss a 2 2 2 r e e e t t t q q q 1 1 1 s s s e u u u e e coin coin coin e e u u u e q q s s s q t t t e e e r r r 2 2 2 c 1 w 2 c 1 w 2 c 1 w 2 c 1 w 2 c 1 w 2 c 1 w 2 w 1 c 2 w 1 c 2 w 1 c 2 w 1 c 2 w 1 c 2 w 1 c 2 • interleaving of the request operations • competition if both processes are waiting • randomized arbiter tosses a coin if both are waiting 22 / 124
Reasoning about probabilities in MDP mdp-10 • requires resolving the nondeterminism by schedulers 23 / 124
Reasoning about probabilities in MDP mdp-10 • requires resolving the nondeterminism by schedulers a scheduler is a function D : S + − D : S + − D : S + − • → Act s.t. → Act → Act action D ( s 0 . . . s n ) D ( s 0 . . . s n ) D ( s 0 . . . s n ) is enabled in state s n s n s n 24 / 124
Reasoning about probabilities in MDP mdp-10 • requires resolving the nondeterminism by schedulers a scheduler is a function D : S + − D : S + − D : S + − • → Act → Act → Act s.t. action D ( s 0 . . . s n ) D ( s 0 . . . s n ) D ( s 0 . . . s n ) is enabled in state s n s n s n • each scheduler induces an infinite Markov chain β β β 1 2 2 2 1 1 α α α 3 3 3 3 3 3 σ σ σ α α α γ γ γ β 1 1 1 β β δ δ δ 2 2 2 3 3 3 3 3 3 δ δ δ γ γ γ σ σ σ MDP α 2 α 2 α 2 1 1 1 σ σ σ 3 3 3 3 3 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 / 124
Reasoning about probabilities in MDP mdp-10 • requires resolving the nondeterminism by schedulers a scheduler is a function D : S + − D : S + − D : S + − • → Act → Act → Act s.t. action D ( s 0 . . . s n ) D ( s 0 . . . s n ) D ( s 0 . . . s n ) is enabled in state s n s n s n • each scheduler induces an infinite Markov chain � � � yields a notion of probability measure Pr D Pr D Pr D on measurable sets of infinite paths 26 / 124
Reasoning about probabilities in MDP mdp-10 • requires resolving the nondeterminism by schedulers a scheduler is a function D : S + − D : S + − D : S + − • → Act → Act → Act s.t. action D ( s 0 . . . s n ) D ( s 0 . . . s n ) D ( s 0 . . . s n ) is enabled in state s n s n s n • each scheduler induces an infinite Markov chain � � � yields a notion of probability measure Pr D Pr D Pr D on measurable sets of infinite paths typical task: given a measurable path event E E , E ∗ ∗ ∗ check whether E E E holds almost surely, i.e., Pr D ( E ) = 1 Pr D ( E ) = 1 Pr D ( E ) = 1 for all schedulers D D D 27 / 124
Reasoning about probabilities in MDP mdp-10 • requires resolving the nondeterminism by schedulers a scheduler is a function D : S + − D : S + − D : S + − • → Act → Act → Act s.t. action D ( s 0 . . . s n ) D ( s 0 . . . s n ) D ( s 0 . . . s n ) is enabled in state s n s n s n • each scheduler induces an infinite Markov chain � � � yields a notion of probability measure Pr D Pr D Pr D on measurable sets of infinite paths typical task: given a measurable path event E E , E ∗ ∗ ∗ check whether E E E holds almost surely ∗ ∗ ∗ compute the worst-case probability for E E E , i.e., Pr D ( E ) Pr D ( E ) Pr D ( E ) Pr D ( E ) Pr D ( E ) Pr D ( E ) sup sup sup or inf inf inf D D D D D D 28 / 124
Quantitative analysis of MDP mdp-15 given: MDP M = ( S , Act , P , . . . ) M = ( S , Act , P , . . . ) M = ( S , Act , P , . . . ) with initial state s 0 s 0 s 0 ω -regular path event E ω ω E , E e.g., given by an LTL formula Pr M compute Pr M Pr M Pr D ( s 0 , E ) Pr D ( s 0 , E ) Pr D ( s 0 , E ) max ( s 0 , E ) = sup task: max ( s 0 , E ) = sup max ( s 0 , E ) = sup D D D 29 / 124
Quantitative analysis of MDP mdp-15 given: MDP M = ( S , Act , P , . . . ) M = ( S , Act , P , . . . ) M = ( S , Act , P , . . . ) with initial state s 0 s 0 s 0 ω ω ω -regular path event E E , E e.g., given by an LTL formula Pr M compute Pr M Pr M Pr D ( s 0 , E ) Pr D ( s 0 , E ) Pr D ( s 0 , E ) max ( s 0 , E ) = sup task: max ( s 0 , E ) = sup max ( s 0 , E ) = sup D D D x s = Pr M compute x s = Pr M x s = Pr M method: max ( s , E ) max ( s , E ) for all s ∈ S max ( s , E ) s ∈ S s ∈ S via graph analysis and linear program [Vardi/Wolper’86] [Courcoubetis/Yannakakis’88] [Bianco/de Alfaro’95] [Baier/Kwiatkowska’98] 30 / 124
probabilistic “bad behaviors” system 31 / 124
probabilistic “bad behaviors” system M MDP M M 32 / 124
probabilistic “bad behaviors” system LTL formula ϕ ϕ ϕ M MDP M M deterministic automaton A A A 33 / 124
probabilistic “bad behaviors” system LTL formula ϕ ϕ ϕ M MDP M M deterministic automaton A A A quantitative analysis in the product-MDP M × A M × A M × A � � � � � � � s , init s � , acceptance Pr M Pr M Pr M = Pr M×A Pr M×A Pr M×A max ( s , ϕ ) = � s , init s � , max ( s , ϕ ) max ( s , ϕ ) = � s , init s � , max max max cond. of A A A 34 / 124
probabilistic “bad behaviors” system LTL formula ϕ ϕ ϕ M MDP M M deterministic automaton A A A maximal probabilility for reaching an quantitative analysis accepting end in the product-MDP M × A M × A M × A component � � � � � � Pr M Pr M Pr M = Pr M×A Pr M×A Pr M×A max ( s , ϕ ) = � s , init s � , max ( s , ϕ ) max ( s , ϕ ) = � s , init s � , � s , init s � , ♦ accEC ♦ accEC ♦ accEC max max max 35 / 124
probabilistic “bad behaviors” system LTL formula ϕ ϕ ϕ M MDP M M deterministic automaton A A A maximal probabilility for reaching an probabilistic reachability analysis accepting end in the product-MDP M × A M × A M × A component linear program � � � � � � Pr M Pr M Pr M = Pr M×A Pr M×A Pr M×A max ( s , ϕ ) = � s , init s � , max ( s , ϕ ) max ( s , ϕ ) = � s , init s � , � s , init s � , ♦ accEC ♦ accEC ♦ accEC max max max 36 / 124
probabilistic “bad behaviors” system LTL formula ϕ ϕ ϕ M MDP M M deterministic automaton A A A probabilistic reachability analysis polynomial in the product-MDP M × A M × A M × A in |M| · |A| |M| · |A| |M| · |A| linear program � � � � � � Pr M Pr M Pr M = Pr M×A Pr M×A Pr M×A max ( s , ϕ ) = � s , init s � , max ( s , ϕ ) max ( s , ϕ ) = � s , init s � , � s , init s � , ♦ accEC ♦ accEC ♦ accEC max max max 37 / 124
probabilistic “bad behaviors” system LTL formula ϕ ϕ ϕ M MDP M M 2exp deterministic automaton A A A probabilistic reachability analysis polynomial in the product-MDP M × A M × A M × A in |M| · |A| |M| · |A| |M| · |A| linear program � � � � � � Pr M Pr M Pr M = Pr M×A Pr M×A Pr M×A max ( s , ϕ ) = � s , init s � , max ( s , ϕ ) max ( s , ϕ ) = � s , init s � , � s , init s � , ♦ accEC ♦ accEC ♦ accEC max max max 38 / 124
probabilistic “bad behaviors” system state explosion LTL formula ϕ ϕ ϕ problem M MDP M M deterministic automaton A A A probabilistic reachability analysis polynomial in the product-MDP M × A M × A M × A in |M| · |A| |M| · |A| |M| · |A| linear program � � � � � � Pr M Pr M Pr M = Pr M×A Pr M×A Pr M×A max ( s , ϕ ) = � s , init s � , max ( s , ϕ ) max ( s , ϕ ) = � s , init s � , � s , init s � , ♦ accEC ♦ accEC ♦ accEC max max max 39 / 124
Advanced techniques for PMC por-01-cs • • • symbolic model checking with variants of BDDs e.g., in PRISM [Kwiatkowska/Norman/Parker] • • • state aggregation with bisimulation e.g., in MRMC [Katoen et al] • • • abstraction-refinement e.g., in RAPTURE [d’Argenio/Jeannet/Jensen/Larsen] PASS [Hermanns/Wachter/Zhang] • • • partial order reduction e.g., in LiQuor [Baier/Ciesinski/Gr¨ oßer] 40 / 124
Advanced techniques for PMC por-01-cs • • • symbolic model checking with variants of BDDs e.g., in PRISM [Kwiatkowska/Norman/Parker] randomized distributed algorithms, communication and multimedia protocols, power management, security, . . . . . . . . . • • • state aggregation with bisimulation e.g., in MRMC [Katoen et al] • • • abstraction-refinement e.g., in RAPTURE [d’Argenio/Jeannet/Jensen/Larsen] PASS [Hermanns/Wachter/Zhang] • • • partial order reduction e.g., in LiQuor [Baier/Ciesinski/Gr¨ oßer] 41 / 124
Advanced techniques for PMC por-01-cs • • • symbolic model checking with variants of BDDs e.g., in PRISM [Kwiatkowska/Norman/Parker] randomized distributed algorithms, communication and multimedia protocols, power management, security, . . . . . . . . . • • • state aggregation with bisimulation e.g., in MRMC [Katoen et al] • • • abstraction-refinement e.g., in RAPTURE [d’Argenio/Jeannet/Jensen/Larsen] PASS [Hermanns/Wachter/Zhang] • • • partial order reduction e.g., in LiQuor [Baier/Ciesinski/Gr¨ oßer] 42 / 124
Partial order reduction por-02 technique for reducing the state space of concurrent systems [Godefroid,Peled,Valmari, ca. 1990] • attempts to analyze a sub-system by identifying “redundant interleavings” • explores representatives of paths that agree up to the order of independent actions 43 / 124
Partial order reduction por-02 technique for reducing the state space of concurrent systems [Godefroid,Peled,Valmari, ca. 1990] • attempts to analyze a sub-system by identifying “redundant interleavings” • explores representatives of paths that agree up to the order of independent actions x := x + y � z := z +3 e.g., x := x + y x := x + y � � z := z +3 z := z +3 � �� � � �� � action β β β action α α α α ; β β ; α has the same effect as α ; β α ; β or β ; α β ; α 44 / 124
Partial order reduction por-02 technique for reducing the state space of concurrent systems [Godefroid,Peled,Valmari, ca. 1990] • attempts to analyze a sub-system by identifying “redundant interleavings” • explores representatives of paths that agree up to the order of independent actions DFS-based on-the-fly generation of a reduced system for each expanded state s s s • choose an appropriate subset Ample ( s ) Ample ( s ) Ample ( s ) of Act ( s ) Act ( s ) Act ( s ) • expand only the α α α -successors of s s for α ∈ Ample ( s ) α ∈ Ample ( s ) α ∈ Ample ( s ) s (but ignore the actions in Act ( s ) \ Ample ( s ) Act ( s ) \ Ample ( s ) Act ( s ) \ Ample ( s )) 45 / 124
Ample-set method [Peled 1993] por-03 given: processes P i P i P i of a parallel system P 1 � . . . �P n P 1 � . . . �P n P 1 � . . . �P n with transition system T = ( S , Act , → , . . . ) T = ( S , Act , → , . . . ) T = ( S , Act , → , . . . ) task: on-the-fly generation of a sub-system T r T r T r s.t. . . . (A1) stutter condition . . . . . . . . . (A2) dependency condition . . . . . . . . . (A3) cycle condition . . . . . . 46 / 124
Ample-set method [Peled 1993] por-03 given: processes P i P i P i of a parallel system P 1 � . . . �P n P 1 � . . . �P n P 1 � . . . �P n with transition system T = ( S , Act , → , . . . ) T = ( S , Act , → , . . . ) T = ( S , Act , → , . . . ) task: on-the-fly generation of a sub-system T r T r T r s.t. � π � π r (A1) stutter condition π � π r π � π r (A2) dependency condition by permutations of independent actions (A3) cycle condition π T Each path π π in T T is represented by an “equivalent” π r T r path π r π r in T r T r 47 / 124
Ample-set method [Peled 1993] por-03 given: processes P i P i P i of a parallel system P 1 � . . . �P n P 1 � . . . �P n P 1 � . . . �P n with transition system T = ( S , Act , → , . . . ) T = ( S , Act , → , . . . ) T = ( S , Act , → , . . . ) task: on-the-fly generation of a sub-system T r T r T r s.t. � π � π r (A1) stutter condition π � π r π � π r (A2) dependency condition by permutations of independent actions (A3) cycle condition π T Each path π π in T T is represented by an “equivalent” π r T r path π r π r in T r T r � � � � � � T T r T T and T r T r satisfy the same stutter-invariant events, e.g., next-free LTL formulas 48 / 124
Ample-set method for MDP por-04 given: processes P i P i P i of a probabilistic system P 1 � . . . �P n P 1 � . . . �P n P 1 � . . . �P n with MDP-semantics M = ( S , Act , P , . . . ) M = ( S , Act , P , . . . ) M = ( S , Act , P , . . . ) task: on-the-fly generation of a sub-MDP M r M r M r s.t. M r M M r M r and M M have the same extremal probabilities for stutter-invariant events 49 / 124
Ample-set method for MDP por-04 given: processes P i P i P i of a probabilistic system P 1 � . . . �P n P 1 � . . . �P n P 1 � . . . �P n with MDP-semantics M = ( S , Act , P , . . . ) M = ( S , Act , P , . . . ) M = ( S , Act , P , . . . ) task: on-the-fly generation of a sub-MDP M r M r M r s.t. M For all schedulers D D D for M M there is a scheduler D r D r D r for M r M r s.t. for all measurable, stutter-invariant events E M r E : E Pr D M ( E ) = Pr D r M ( E ) = Pr D r M ( E ) = Pr D r Pr D Pr D M r ( E ) M r ( E ) M r ( E ) � � � � � � M r M M r M r and M M have the same extremal probabilities for stutter-invariant events 50 / 124
Example: ample set method por-08-new s s s β γ γ γ β β α α α α α α α α α γ γ γ β β β δ δ δ δ δ δ T original system T T α α α independent from β β β and γ γ γ 51 / 124
Example: ample set method por-08-new s s s P 1 � P 2 P 1 � P 2 P 1 � P 2 β γ γ γ β β α α α α α α α α α α α α γ γ γ γ γ γ β β β β β β δ δ δ δ δ δ δ δ δ δ δ δ T original system T T action α α α : x := 1 x := 1 x := 1 α α α independent action δ δ δ : from β β and γ β γ γ IF x > 0 x > 0 x > 0 THEN y := 1 y := 1 y := 1 FI 52 / 124
Example: ample set method por-08-new s s s s s s β γ γ γ β β γ γ γ β β β α α α α α α α α α γ γ γ β β β α α α α α α δ δ δ δ δ δ δ δ δ δ δ δ T T r original system T T reduced system T r T r (A1)-(A3) are fulfilled α α α independent from β β β and γ γ γ 53 / 124
Example: ample set method fails for MDP por-08-new s s s s s s β γ γ γ β β γ γ γ β β β 1 1 1 1 1 1 2 2 2 2 2 2 α α α 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 γ γ γ β β β α α α α α α α α α α α α γ γ γ β β β δ δ δ δ δ δ δ δ δ δ δ δ M M r original MDP M M reduced MDP M r M r (A1)-(A3) are fulfilled α α α independent from β β β and γ γ γ 54 / 124
Example: ample set method fails for MDP por-08-new s s s s s s β γ γ γ β β γ γ γ β β β 1 1 1 1 1 1 2 2 2 2 2 2 α α α 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 γ γ γ β β β α α α α α α α α α α α α γ γ γ β β β δ δ δ δ δ δ δ δ δ δ δ δ M M r original MDP M M reduced MDP M r M r Pr M Pr M Pr M max ( s , ♦ green ) = 1 ♦ ♦ ♦ “eventually” max ( s , ♦ green ) = 1 max ( s , ♦ green ) = 1 55 / 124
Example: ample set method fails for MDP por-08-new s s s s s s β γ γ γ β β γ γ γ β β β 1 1 1 1 1 1 2 2 2 2 2 2 α α α 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 γ γ γ β β β α α α α α α α α α α α α γ γ γ β β β δ δ δ δ δ δ δ δ δ δ δ δ M M r original MDP M M reduced MDP M r M r Pr M 2 = Pr M r Pr M Pr M 2 = Pr M r 2 = Pr M r 1 1 1 max ( s , ♦ green ) = 1 > max ( s , ♦ green ) max ( s , ♦ green ) = 1 max ( s , ♦ green ) = 1 > > max ( s , ♦ green ) max ( s , ♦ green ) 56 / 124
Partial order reduction for MDP por-09 extend Peled’s conditions (A1)-(A3) for the ample-sets (A1) stutter condition . . . . . . . . . (A2) dependency condition . . . . . . . . . . . . (A3) cycle condition . . . . . . (A4) probabilistic condition β 1 β 1 β 1 β 2 β 2 β 2 β n β n β n α α α If there is a path s s − − − → → → − − − → . . . → . . . → . . . − − − → → → − − − → → → in M M M s.t. s β 1 , . . . , β n , α / ∈ Ample ( s ) α β 1 , . . ., β n , α / β 1 , . . . , β n , α / ∈ Ample ( s ) ∈ Ample ( s ) and α α is probabilistic then | Ample ( s ) | = 1 | Ample ( s ) | = 1 | Ample ( s ) | = 1. 57 / 124
Partial order reduction for MDP por-09 extend Peled’s conditions (A1)-(A3) for the ample-sets (A1) stutter condition . . . . . . . . . (A2) dependency condition . . . . . . . . . . . . (A3) cycle condition . . . . . . (A4) probabilistic condition β 1 β 1 β 1 β 2 β 2 β 2 β n β n β n α α α If there is a path s s − − − → → → − − − → . . . → . . . → . . . − − − → → → − − − → → → in M M s.t. M s β 1 , . . . , β n , α / ∈ Ample ( s ) α β 1 , . . ., β n , α / β 1 , . . . , β n , α / ∈ Ample ( s ) ∈ Ample ( s ) and α α is probabilistic then | Ample ( s ) | = 1 | Ample ( s ) | = 1 | Ample ( s ) | = 1. M M r If (A1)-(A4) hold then M M and M r M r have the same extremal probabilities for all stutter-invariant properties. 58 / 124
Probabilistic model checking por-ifm-32 probabilistic quantitative system requirements Markov decision LTL \� \� formula ϕ ϕ ϕ \� M process M M (path event) quantitative analysis of M M M against ϕ ϕ ϕ maximal/minimal probability for ϕ ϕ ϕ 59 / 124
Probabilistic model checking, e.g., LiQuor por-ifm-32 modeling language quantitative P 1 � . . . �P n P 1 � . . . �P n P 1 � . . . �P n requirements partial order reduction reduced LTL \� \� formula ϕ ϕ ϕ \� M r MDP M r M r (path event) quantitative analysis of M r M r M r against ϕ ϕ ϕ maximal/minimal probability for ϕ ϕ ϕ 60 / 124
Probabilistic model checking, e.g., LiQuor por-ifm-32a modeling language quantitative P 1 � . . . �P n P 1 � . . . �P n P 1 � . . . �P n requirements partial order reduction reduced LTL \� \� formula ϕ ϕ ϕ \� M r MDP M r M r (path event) quantitative analysis of M r M r M r against ϕ ϕ ϕ � worst-case maximal/minimal probability for ϕ ϕ ϕ analysis 61 / 124
Example: extremal scheduler for MDP por-08-copy s s s β γ γ γ β β 1 1 1 1 1 1 2 2 2 2 2 2 α α α 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 γ γ γ β β β α α α α α α γ γ γ β β β δ δ δ δ δ δ M original MDP M M α α α independent from β β β and γ γ γ 62 / 124
Example: extremal scheduler for MDP por-08-copy s s s P 1 � P 2 P 1 � P 2 P 1 � P 2 β γ γ γ β β 1 1 1 1 1 1 2 2 2 2 2 2 α α α 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 γ γ γ β β β γ γ γ β β β α α α α α α α α α γ γ γ β β β δ δ δ δ δ δ δ δ δ δ δ δ M original MDP M M α x := random (0 , 1) action α α : x := random (0 , 1) x := random (0 , 1) α α α independent action δ δ δ : from β β and γ β γ γ IF x > 0 x > 0 x > 0 THEN y := 1 y := 1 y := 1 FI 63 / 124
Example: extremal scheduler for MDP por-08-copy s s s P 1 � P 2 P 1 � P 2 P 1 � P 2 β γ γ γ β β 1 1 1 1 1 1 2 2 2 2 2 2 α α α 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 γ γ γ β β β γ γ γ β β β α α α α α α α α α γ γ γ β β β δ δ δ δ δ δ δ δ δ δ δ δ β γ P 1 scheduler chooses action β β or γ γ of process P 1 P 1 , P 2 depending on P 2 P 2 ’s internal probabilistic choice 64 / 124
Outline overview-pomdp • Markov decision processes (MDP) and quantitative analysis against path events • partial order reduction for MDP • partially-oberservable MDP ← ← ← − − − • conclusions 65 / 124
Monty-Hall problem pomdp-01 3 3 3 doors initially closed show candidate master 66 / 124
Monty-Hall problem pomdp-01 3 3 3 doors no prize prize no prize initially closed show candidate master 67 / 124
Monty-Hall problem pomdp-01 3 3 3 doors no prize prize no prize initially closed show candidate master 1. candidate chooses one of the doors 68 / 124
Monty-Hall problem pomdp-01 3 3 3 doors no prize prize no prize initially closed show candidate master 1. candidate chooses one of the doors 2. show master opens a non-chosen, non-winning door 69 / 124
Monty-Hall problem pomdp-01 3 3 3 doors no prize prize no prize initially closed show candidate master 1. candidate chooses one of the doors 2. show master opens a non-chosen, non-winning door 3. candidate has the choice: • keep the choice or • switch to the other (still closed) door 70 / 124
Monty-Hall problem pomdp-01 3 3 3 doors 100 . 000 100 . 000 100 . 000 no prize no prize Euro initially closed show candidate master 1. candidate chooses one of the doors 2. show master opens a non-chosen, non-winning door 3. candidate has the choice: • keep the choice or • switch to the other (still closed) door 4. show master opens all doors 71 / 124
Monty-Hall problem pomdp-01 3 3 3 doors 100 . 000 100 . 000 100 . 000 no prize no prize Euro initially closed show candidate master optimal strategy for the candidate: initial choice of the door: arbitrary revision of the initial choice (switch) 2 probability for getting the prize: 2 2 3 3 3 72 / 124
MDP for the Monty-Hall problem pomdp-02 73 / 124
MDP for the Monty-Hall problem pomdp-02 3 3 3 doors no prize prize no prize initially closed show master’s actions candidate’s actions 2. opens a non-chosen, 1. choose one door non-winning door 3. keep or switch ? 4. opens all doors 74 / 124
MDP for the Monty-Hall problem pomdp-02 3 3 3 doors no prize prize no prize initially closed ❍❍❍❍❍❍❍❍❍❍❍❍ ❍❍❍❍❍❍❍❍❍❍❍❍ ❍❍❍❍❍❍❍❍❍❍❍❍ ✟ ✟ ✟ show master’s actions candidate’s actions ✟✟✟✟✟✟✟✟✟✟✟✟ ✟✟✟✟✟✟✟✟✟✟✟✟ ✟✟✟✟✟✟✟✟✟✟✟✟ 2. opens a non-chosen, 1. choose one door non-winning door 3. keep or switch ? 4. opens all doors ❍ ❍ ❍ start 1 1 1 1 1 1 1 1 1 3 3 3 3 3 3 3 3 3 door 1 door 2 door 3 1 1 2 2 3 3 keep switch keep switch keep switch won lost 75 / 124
MDP for the Monty-Hall problem pomdp-02 3 3 3 doors no prize prize no prize initially closed candidate’s actions Pr max (start , ♦ won ) , ♦ won ) , ♦ won ) = 1 , ♦ won ) , ♦ won ) Pr max (start , ♦ won ) Pr max (start , ♦ won ) , ♦ won ) = 1 , ♦ won ) = 1 1. choose one door 3. keep or switch ? optimal scheduler requires start complete information 1 1 1 1 1 1 on the states 1 1 1 3 3 3 3 3 3 3 3 3 door 1 door 2 door 3 1 1 2 2 3 3 keep switch switch won won lost 76 / 124
MDP for the Monty-Hall problem pomdp-02 3 3 3 doors no prize prize no prize initially closed candidate’s actions 1. choose one door cannot be distinguished 3. keep or switch ? by the candidate start 1 1 1 1 1 1 1 1 1 3 3 3 3 3 3 3 3 3 door 1 door 2 door 3 1 1 2 2 3 3 keep switch keep switch keep switch won won lost 77 / 124
MDP for the Monty-Hall problem pomdp-02 3 3 3 doors no prize prize no prize initially closed candidate’s actions observation-based strategy: 1. choose one door choose action switch in state door i 3. keep or switch ? i i start 1 1 1 1 1 1 1 1 1 3 3 3 3 3 3 3 3 3 door 1 door 2 door 3 1 1 2 2 3 3 switch switch switch won won lost 78 / 124
MDP for the Monty-Hall problem pomdp-02 3 3 3 doors no prize prize no prize initially closed candidate’s actions observation-based strategy: 1. choose one door choose action switch in state door i 3. keep or switch ? i i ♦ won : 2 2 2 probability for ♦ won ♦ won start 3 3 3 1 1 1 1 1 1 1 1 1 3 3 3 3 3 3 3 3 3 door 1 door 2 door 3 1 1 2 2 3 3 switch switch switch won won lost 79 / 124
Partially-observable Markov decision process pomdp-05 A partially-observable MDP (POMDP for short) is an MDP M = ( S , Act , P , . . . ) M = ( S , Act , P , . . . ) M = ( S , Act , P , . . . ) together with an equivalence relation ∼ ∼ ∼ on S S S 80 / 124
Partially-observable Markov decision process pomdp-05 A partially-observable MDP (POMDP for short) is an MDP M = ( S , Act , P , . . . ) M = ( S , Act , P , . . . ) M = ( S , Act , P , . . . ) together with an equivalence relation ∼ ∼ ∼ on S S S � � � if s 1 ∼ s 2 s 1 ∼ s 2 s 1 ∼ s 2 then s 1 , s 2 s 1 , s 2 s 1 , s 2 cannot be distinguished from outside (or by the scheduler) observables: equivalence classes of states 81 / 124
Partially-observable Markov decision process pomdp-05 A partially-observable MDP (POMDP for short) is an MDP M = ( S , Act , P , . . . ) M = ( S , Act , P , . . . ) M = ( S , Act , P , . . . ) together with an equivalence relation ∼ ∼ ∼ on S S S � � � if s 1 ∼ s 2 s 1 ∼ s 2 then s 1 , s 2 s 1 ∼ s 2 s 1 , s 2 s 1 , s 2 cannot be distinguished from outside (or by the scheduler) observables: equivalence classes of states observation-based scheduler: scheduler D : S + → Act D : S + → Act D : S + → Act such that for all π 1 , π 2 ∈ S + π 1 , π 2 ∈ S + : π 1 , π 2 ∈ S + D ( π 1 ) = D ( π 2 ) obs ( π 1 ) = obs ( π 2 ) D ( π 1 ) = D ( π 2 ) D ( π 1 ) = D ( π 2 ) if obs ( π 1 ) = obs ( π 2 ) obs ( π 1 ) = obs ( π 2 ) where obs ( s 0 s 1 . . . s n ) = [ s 0 ] [ s 1 ] . . . [ s n ] obs ( s 0 s 1 . . . s n ) = [ s 0 ] [ s 1 ] . . . [ s n ] obs ( s 0 s 1 . . . s n ) = [ s 0 ] [ s 1 ] . . . [ s n ] 82 / 124
Extreme cases of POMDP pomdp-11 extreme cases of an POMDP: • s 1 ∼ s 2 s 1 ∼ s 2 s 1 ∼ s 2 iff s 1 = s 2 s 1 = s 2 s 1 = s 2 • s 1 ∼ s 2 s 1 ∼ s 2 s 1 ∼ s 2 for all s 1 s 1 s 1 , s 2 s 2 s 2 83 / 124
Extreme cases of POMDP pomdp-11 extreme cases of an POMDP: • s 1 ∼ s 2 s 1 ∼ s 2 s 1 ∼ s 2 iff s 1 = s 2 s 1 = s 2 s 1 = s 2 ← ← ← − − − standard MDP • s 1 ∼ s 2 s 1 ∼ s 2 s 1 ∼ s 2 for all s 1 s 1 s 1 , s 2 s 2 s 2 84 / 124
Probabilistic automata are special POMDP pomdp-11 extreme cases of an POMDP: • s 1 ∼ s 2 s 1 ∼ s 2 s 1 ∼ s 2 iff s 1 = s 2 s 1 = s 2 s 1 = s 2 ← ← ← − − − standard MDP • s 1 ∼ s 2 s 1 ∼ s 2 s 1 ∼ s 2 for all s 1 s 1 s 1 , s 2 s 2 ← ← ← − − − probabilistic automata s 2 note that for totally non-observable POMDP: observation-based function = infinite word = = = = = � � � � � � scheduler D : N → Act D : N → Act D : N → Act over Act Act Act 85 / 124
Undecidability results for POMDP pomdp-11 extreme cases of an POMDP: • s 1 ∼ s 2 s 1 ∼ s 2 s 1 ∼ s 2 iff s 1 = s 2 s 1 = s 2 s 1 = s 2 ← ← ← − − − standard MDP • s 1 ∼ s 2 s 1 ∼ s 2 s 1 ∼ s 2 for all s 1 s 1 s 1 , s 2 s 2 ← ← ← − − − probabilistic automata s 2 note that for totally non-observable POMDP: observation-based function = infinite word = = = = = � � � � � � scheduler D : N → Act D : N → Act D : N → Act over Act Act Act undecidability results for PFA carry over to POMDP maximum probabilistic non-emptiness reachability problem problem for = = = � � � Pr obs “does Pr obs Pr obs PFA max ( ♦ F ) > p hold ? ” max ( ♦ F ) > p max ( ♦ F ) > p 86 / 124
Undecidability results for POMDP pomdp-30-new • The model checking problem for POMDP and quantitative properties is undecidable, e.g., probabilistic reachability properties. 87 / 124
Undecidability results for POMDP pomdp-30-new • The model checking problem for POMDP and quantitative properties is undecidable, e.g., probabilistic reachability properties. • There is no even no approximation algorithm for reachability objectives. [Paz’71], [Madani/Hanks/Condon’99], [Giro/d’Argenio’07] 88 / 124
Undecidability results for POMDP pomdp-30-new • The model checking problem for POMDP and quantitative properties is undecidable, e.g., probabilistic reachability properties. • There is no even no approximation algorithm for reachability objectives. • The model checking problem for POMDP and several qualitative properties is undecidable, e.g., repeated reachability with positive probability “does Pr obs Pr obs Pr obs max ( �♦ F ) max ( �♦ F ) max ( �♦ F ) > 0 > 0 > 0 hold ? ” = �♦ �♦ �♦ � = = “infinitely often” � � 89 / 124
Undecidability results for POMDP pomdp-30-new • The model checking problem for POMDP and quantitative properties is undecidable, e.g., probabilistic reachability properties. • There is no even no approximation algorithm for reachability objectives. • The model checking problem for POMDP and several qualitative properties is undecidable, e.g., repeated reachability with positive probability “does Pr obs Pr obs Pr obs max ( �♦ F ) max ( �♦ F ) max ( �♦ F ) > 0 > 0 > 0 hold ? ” Many interesting verification problems for distributed probabilistic multi-agent systems are undecidable. 90 / 124
Undecidability results for POMDP pomdp-30-new • The model checking problem for POMDP and quantitative properties is undecidable, e.g., probabilistic reachability properties. • There is no even no approximation algorithm for reachability objectives. • The model checking problem for POMDP and several qualitative properties is undecidable, e.g., repeated reachability with positive probability “does Pr obs Pr obs Pr obs max ( �♦ F ) max ( �♦ F ) max ( �♦ F ) > 0 > 0 > 0 hold ? ” ... already holds for totally non-observable POMDP � �� � probabilistic B¨ uchi automata 91 / 124
Probabilistic B¨ uchi automaton (PBA) pba-01 P = ( Q , Σ , δ, µ, F ) P = ( Q , Σ , δ, µ, F ) P = ( Q , Σ , δ, µ, F ) • Q Q finite state space Q • Σ Σ Σ alphabet • δ : Q × Σ × Q → [0 , 1] δ : Q × Σ × Q → [0 , 1] δ : Q × Σ × Q → [0 , 1] s.t. for all q ∈ Q q ∈ Q q ∈ Q , a ∈ Σ a ∈ Σ: a ∈ Σ � � � δ ( q , a , p ) ∈ { 0 , 1 } δ ( q , a , p ) ∈ { 0 , 1 } δ ( q , a , p ) ∈ { 0 , 1 } • initial distribution µ µ µ p ∈ Q p ∈ Q p ∈ Q • set of final states F ⊆ Q F ⊆ Q F ⊆ Q 92 / 124
Probabilistic B¨ uchi automaton (PBA) pba-01 P = ( Q , Σ , δ, µ, F ) ← − POMDP where Σ = Act Σ = Act Σ = Act P = ( Q , Σ , δ, µ, F ) P = ( Q , Σ , δ, µ, F ) ← ← − − and ∼ ∼ � ∼ = = = Q × Q Q × Q Q × Q � � • Q Q Q finite state space • Σ Σ Σ alphabet • δ : Q × Σ × Q → [0 , 1] δ : Q × Σ × Q → [0 , 1] δ : Q × Σ × Q → [0 , 1] s.t. for all q ∈ Q q ∈ Q q ∈ Q , a ∈ Σ a ∈ Σ a ∈ Σ: � � � δ ( q , a , p ) ∈ { 0 , 1 } δ ( q , a , p ) ∈ { 0 , 1 } δ ( q , a , p ) ∈ { 0 , 1 } • initial distribution µ µ µ p ∈ Q p ∈ Q p ∈ Q • set of final states F ⊆ Q F ⊆ Q F ⊆ Q 93 / 124
Probabilistic B¨ uchi automaton (PBA) pba-01 P = ( Q , Σ , δ, µ, F ) ← − POMDP where Σ = Act Σ = Act Σ = Act P = ( Q , Σ , δ, µ, F ) P = ( Q , Σ , δ, µ, F ) ← ← − − and ∼ ∼ � ∼ = = = Q × Q Q × Q Q × Q � � • Q Q Q finite state space • Σ Σ Σ alphabet • δ : Q × Σ × Q → [0 , 1] δ : Q × Σ × Q → [0 , 1] δ : Q × Σ × Q → [0 , 1] s.t. for all q ∈ Q q ∈ Q q ∈ Q , a ∈ Σ a ∈ Σ a ∈ Σ: � � � δ ( q , a , p ) ∈ { 0 , 1 } δ ( q , a , p ) ∈ { 0 , 1 } δ ( q , a , p ) ∈ { 0 , 1 } • initial distribution µ µ µ p ∈ Q p ∈ Q p ∈ Q • set of final states F ⊆ Q F ⊆ Q F ⊆ Q For each infinite word x ∈ Σ ω x ∈ Σ ω x ∈ Σ ω : Pr( x ) = Pr( x ) = Pr( x ) = probability for the accepting runs for x x x ↑ ↑ ↑ accepting run: visits F F F infinitely often 94 / 124
Probabilistic B¨ uchi automaton (PBA) pba-01 P = ( Q , Σ , δ, µ, F ) ← − POMDP where Σ = Act Σ = Act Σ = Act P = ( Q , Σ , δ, µ, F ) P = ( Q , Σ , δ, µ, F ) ← ← − − and ∼ ∼ � ∼ = = = Q × Q Q × Q Q × Q � � • Q Q Q finite state space • Σ Σ Σ alphabet • δ : Q × Σ × Q → [0 , 1] δ : Q × Σ × Q → [0 , 1] δ : Q × Σ × Q → [0 , 1] s.t. for all q ∈ Q q ∈ Q q ∈ Q , a ∈ Σ a ∈ Σ a ∈ Σ: � � � δ ( q , a , p ) ∈ { 0 , 1 } δ ( q , a , p ) ∈ { 0 , 1 } δ ( q , a , p ) ∈ { 0 , 1 } • initial distribution µ µ µ p ∈ Q p ∈ Q p ∈ Q • set of final states F ⊆ Q F ⊆ Q F ⊆ Q For each infinite word x ∈ Σ ω x ∈ Σ ω x ∈ Σ ω : Pr( x ) = Pr( x ) = Pr( x ) = probability for the accepting runs for x x x ↑ ↑ ↑ probability measure in the infinite Markov chain induced by x x x viewed as a scheduler 95 / 124
Accepted language of a PBA pba-03 P = ( Q , Σ , δ, µ, F ) P = ( Q , Σ , δ, µ, F ) P = ( Q , Σ , δ, µ, F ) Σ • Q Q Q finite state space, Σ Σ alphabet δ : Q × Σ × Q → [0 , 1] . . . • δ : Q × Σ × Q → [0 , 1] δ : Q × Σ × Q → [0 , 1] s.t. . . . . . . µ • initial distribution µ µ • set of final states F ⊆ Q F ⊆ Q F ⊆ Q three types of accepted language: � � � � � � x ∈ Σ ω : Pr( x ) > 0 x ∈ Σ ω : Pr( x ) > 0 x ∈ Σ ω : Pr( x ) > 0 L > 0 ( P ) L > 0 ( P ) L > 0 ( P ) = = = probable semantics � � � � � � x ∈ Σ ω : Pr( x ) = 1 x ∈ Σ ω : Pr( x ) = 1 x ∈ Σ ω : Pr( x ) = 1 L =1 ( P ) L =1 ( P ) L =1 ( P ) = = = almost-sure sem. � � � � � � x ∈ Σ ω : Pr( x ) > λ x ∈ Σ ω : Pr( x ) > λ x ∈ Σ ω : Pr( x ) > λ L >λ ( P ) L >λ ( P ) L >λ ( P ) = = = threshold semantics where 0 < λ < 1 0 < λ < 1 0 < λ < 1 96 / 124
Example for PBA pba-05 a , 1 1 1 a a 2 2 2 1 2 1 1 2 2 b b b 1 a a , 1 1 a a a a 2 2 2 initial state (probability 1) final state nonfinal state 97 / 124
Example for PBA pba-05 accepted language: a , 1 1 1 a a 2 2 2 1 2 1 1 2 2 ( a + b ) ∗ a ω L > 0 ( P ) L > 0 ( P ) = L > 0 ( P ) = ( a + b ) ∗ a ω ( a + b ) ∗ a ω = b b b 1 a a , 1 1 a a a a 2 2 2 initial state (probability 1) final state nonfinal state 98 / 124
Example for PBA pba-05 accepted language: a , 1 1 1 a a 2 2 2 1 2 1 1 2 2 ( a + b ) ∗ a ω L > 0 ( P ) L > 0 ( P ) L > 0 ( P ) = = ( a + b ) ∗ a ω ( a + b ) ∗ a ω = b b b 1 a a , 1 1 a a a a L =1 ( P ) = b ∗ a ω b ∗ a ω b ∗ a ω L =1 ( P ) L =1 ( P ) = = 2 2 2 initial state (probability 1) final state nonfinal state 99 / 124
Example for PBA pba-05 accepted language: a , 1 1 1 a a 2 2 2 1 2 1 1 2 2 ( a + b ) ∗ a ω L > 0 ( P ) L > 0 ( P ) = L > 0 ( P ) = ( a + b ) ∗ a ω ( a + b ) ∗ a ω = b b b 1 a a , 1 1 a a a a L =1 ( P ) = b ∗ a ω b ∗ a ω b ∗ a ω L =1 ( P ) L =1 ( P ) = = 2 2 2 > 0 are strictly more expressive than DBA PBA > 0 > 0 Thus: 100 / 124
Recommend
More recommend
