Logics for Markov Decision Processes Pedro S anchez Terraf Joint - PowerPoint PPT Presentation

Introduction Labelled Markov Processes (LMP) and its Non Deterministic version Results Future Work Logics for Markov Decision Processes Pedro S´ anchez Terraf Joint work with P .R. D’Argenio and N. Wolovick SLALM, UniAndes, 04 / 06 / 2012 unc u buntu

Introduction Labelled Markov Processes (LMP) and its Non Deterministic version Results Future Work Contents Introduction 1 Labelled Transition Systems (LTS) Modal Logics Labelled Markov Processes (LMP) and its Non Deterministic 2 version Analytic Spaces and Unique Structure Proving Completeness 3 Results Logics for non-deterministic processes Some counterexamples Future Work 4 unc u buntu

Introduction Labelled Markov Processes (LMP) and its Non Deterministic version Results Future Work Labelled Transition Systems (LTS) A toy model Labelled Transition Systems (LTS) � S , L , T � such that T a : S → Pow ( S ) for each a ∈ L . unc u buntu

Introduction Labelled Markov Processes (LMP) and its Non Deterministic version Results Future Work Labelled Transition Systems (LTS) A toy model Labelled Transition Systems (LTS) � S , L , T � such that T a : S → Pow ( S ) for each a ∈ L . Zig-zag morphism A surjective f : S → S ′ such that for all a ∈ L and every s ∈ S , Pow ( f ) ◦ T a = T ′ a ◦ f . lts02.jpg unc u buntu

Introduction Labelled Markov Processes (LMP) and its Non Deterministic version Results Future Work Labelled Transition Systems (LTS) A toy model Labelled Transition Systems (LTS) � S , L , T � such that T a : S → Pow ( S ) for each a ∈ L . Zig-zag morphism A surjective f : S → S ′ such that for all a ∈ L and every s ∈ S , Pow ( f ) ◦ T a = T ′ a ◦ f . We say that s simulates t because lts02.jpg s can perform every “sequence of actions” that t can. unc u buntu

Introduction Labelled Markov Processes (LMP) and its Non Deterministic version Results Future Work Labelled Transition Systems (LTS) Simulation and Bisimulation on LTS Simulation a → t 2 then there is s 2 such It is a relation R such that if s 1 Rt 1 and t 1 a that s 1 → s 2 and s 2 Rt 2 . In that case we say that s 1 simulates s 2 . unc u buntu

Introduction Labelled Markov Processes (LMP) and its Non Deterministic version Results Future Work Labelled Transition Systems (LTS) Simulation and Bisimulation on LTS Simulation a → t 2 then there is s 2 such It is a relation R such that if s 1 Rt 1 and t 1 a → s 2 and s 2 Rt 2 . In that case we say that s 1 simulates s 2 . that s 1 Bisimulation It is a symmetric simulation. We’ll say that s 1 is bisimilar to t 1 if there exists a bisimulation R such that s 1 Rt 1 . Note: Bisimulation is finer than “double simulation”. That’s to say, if s 1 is bisimilar to t 1 , then s 1 lts12.jpg simulates t 1 and t 1 simulates s 1 , but not conversely . unc u buntu

� Introduction Labelled Markov Processes (LMP) and its Non Deterministic version Results Future Work Labelled Transition Systems (LTS) Coalgebraic presentation of processes and bisimulation One categorical counterpart of a relation is a span of morphisms Bisimilarity (span) S � � � ���� f g � � S 1 S 2 unc u buntu

� � Introduction Labelled Markov Processes (LMP) and its Non Deterministic version Results Future Work Labelled Transition Systems (LTS) Coalgebraic presentation of processes and bisimulation One categorical counterpart of a relation is a span of morphisms Bisimilarity (span) Behavioral equivalence (cospan) S 1 S 2 S � � � � ���� f g � � ���� � � � � S 1 S 2 T There is a correspondence between cospans and logics unc u buntu

� � Introduction Labelled Markov Processes (LMP) and its Non Deterministic version Results Future Work Labelled Transition Systems (LTS) Coalgebraic presentation of processes and bisimulation One categorical counterpart of a relation is a span of morphisms Bisimilarity (span) Behavioral equivalence (cospan) S 1 S 2 S � � � � ���� f g � � ���� � � � � S 1 S 2 T There is a correspondence between cospans and logics Semipullbacks A category has semipullbacks if every cospan can be completed to a commutative diagram with a span. unc u buntu

� � Introduction Labelled Markov Processes (LMP) and its Non Deterministic version Results Future Work Labelled Transition Systems (LTS) Coalgebraic presentation of processes and bisimulation One categorical counterpart of a relation is a span of morphisms Bisimilarity (span) Behavioral equivalence (cospan) S 1 S 2 S � � � � ���� f g � � ���� � � � � S 1 S 2 T There is a correspondence between cospans and logics Semipullbacks A category has semipullbacks if every cospan can be completed to a commutative diagram with a span. It is the Amalgamation Property in the opposite category. unc u buntu

Introduction Labelled Markov Processes (LMP) and its Non Deterministic version Results Future Work Modal Logics Logics for Bisimulation Hennessy-Milner Logic (HML) � ϕ ≡ ⊤ | ¬ ϕ | ϕ i | � a � ψ i unc u buntu

Introduction Labelled Markov Processes (LMP) and its Non Deterministic version Results Future Work Modal Logics Simulation and Bisimulation on LTS Simulation a → t 2 then there is s 2 such It is a relation R such that if s 1 Rt 1 and t 1 a → s 2 and s 2 Rt 2 . In that case we say that s 1 simulates s 2 . that s 1 Bisimulation It is a symmetric simulation. We’ll say that s 1 is bisimilar to t 1 if there exists a bisimulation R such that s 1 Rt 1 . “ t 1 can make an a -transition after which a c -transition is not possible”. lts12.jpg t 1 | = � a �¬� c �⊤ unc u buntu s 1 �| = � a �¬� c �⊤

Introduction Labelled Markov Processes (LMP) and its Non Deterministic version Results Future Work Modal Logics Logics for Bisimulation Hennessy-Milner Logic (HML) � ϕ ≡ ⊤ | ¬ ϕ | ϕ i | � a � ψ i Logical Characterization of Bisimulation Two states in a LTS are bisimilar iff they satisfy the same HML formulas. unc u buntu

Introduction Labelled Markov Processes (LMP) and its Non Deterministic version Results Future Work Labelled Markov Processes (LMP) and Non Determinism LMP (Desharnais et al.) � S , S , L , t � such that t a ( s ) ∈ P ( S ) for each s ∈ S and a ∈ L , where � S , S � is a measurable space; P ( S ) is the space of (sub)probability measures over � S , S � ; t a : S → P ( S ) is measurable. unc u buntu

Introduction Labelled Markov Processes (LMP) and its Non Deterministic version Results Future Work Labelled Markov Processes (LMP) and Non Determinism LMP (Desharnais et al.) � S , S , L , t � such that t a ( s ) ∈ P ( S ) for each s ∈ S and a ∈ L , where � S , S � is a measurable space; P ( S ) is the space of (sub)probability measures over � S , S � ; t a : S → P ( S ) is measurable. NLMP (D’Argenio and Wolovick) � S , S , L , T � such that T a ( s ) ⊆ P ( S ) para each s ∈ S y a ∈ L , where: � S , S � , P ( S ) as before; For each s , T a ( s ) is measurable. I.e., T a : S → P ( S ) . T a : S → P ( S ) is a measurable map. unc u buntu

Introduction Labelled Markov Processes (LMP) and its Non Deterministic version Results Future Work Analytic Spaces and Unique Structure A pinch of Descriptive Set Theory: Analytic Spaces Definition An analytic topological space is the continuous image of a Borel set (v.g., of reals). unc u buntu

Introduction Labelled Markov Processes (LMP) and its Non Deterministic version Results Future Work Analytic Spaces and Unique Structure A pinch of Descriptive Set Theory: Analytic Spaces Definition An analytic topological space is the continuous image of a Borel set (v.g., of reals). An measurable space is analytic if it is isomorphic to � A , B ( A ) � for some analytic topological space A . unc u buntu

Introduction Labelled Markov Processes (LMP) and its Non Deterministic version Results Future Work Analytic Spaces and Unique Structure A pinch of Descriptive Set Theory: Analytic Spaces Definition An analytic topological space is the continuous image of a Borel set (v.g., of reals). An measurable space is analytic if it is isomorphic to � A , B ( A ) � for some analytic topological space A . Examples The convex hull of a Borel set in R n ; The relation of isomorphism between countable structures. unc u buntu

Introduction Labelled Markov Processes (LMP) and its Non Deterministic version Results Future Work Analytic Spaces and Unique Structure A pinch of Descriptive Set Theory: Analytic Spaces Definition An analytic topological space is the continuous image of a Borel set (v.g., of reals). An measurable space is analytic if it is isomorphic to � A , B ( A ) � for some analytic topological space A . Examples The convex hull of a Borel set in R n ; The relation of isomorphism between countable structures. Unique Structure Theorem If a sub- σ -algebra S ⊆ B ( A ) is countably generated and separates points, then it is B ( A ) . unc u buntu

Introduction Labelled Markov Processes (LMP) and its Non Deterministic version Results Future Work Proving Completeness Logics for bisimulation on LMP HML q (Larsen and Skou, Danos et al. ) ϕ ≡ ⊤ | ϕ 1 ∧ ϕ 2 | � a � q ϕ , q ∈ Q unc u buntu