PAC Statistical Model Checking for Markov Decision Processes and - PowerPoint PPT Presentation

PAC Statistical Model Checking for Markov Decision Processes and Stochastic Games 1 Pranav Ashok, Jan Kˇ ret´ ınsk´ y, Maximilian Weininger Technical University of Munich Highlights of Logic, Automata and Games Warsaw, Poland September 19, 2019 1 based on paper presented at CAV 2019

Stochastic Game Reachability 0 . 2 a b c 0 . 8 Objective player: maximize P(F ) player: minimize P(F ) Reachability in limited information stochastic games 2/6

This work: Black-box (limited information setting) Unknown successor distribution Problem statement Compute V ( s ) = max σ min τ P σ,τ ( F ) = min τ max σ P σ,τ ( F ) s s with guarantees Reachability in limited information stochastic games 3/6

Background ◮ Seminal paper on Stochastic Games [ Condon 90 ] quadratic programming, strategy iteration, value iteration Reachability in limited information stochastic games 4/6

Background ◮ Seminal paper on Stochastic Games [ Condon 90 ] quadratic programming, strategy iteration, value iteration ◮ Algos not directly applicable on general SG ◮ First practical algorithm for general SG giving guarantees [ Kelmendi et. al. 2018 ] Reachability in limited information stochastic games 4/6

Background ◮ Seminal paper on Stochastic Games [ Condon 90 ] quadratic programming, strategy iteration, value iteration ◮ Algos not directly applicable on general SG ◮ First practical algorithm for general SG giving guarantees [ Kelmendi et. al. 2018 ] ◮ This work: first algorithm for limited information SG Reachability in limited information stochastic games 4/6

The Algorithm Similar to Kelmendi et. al. 2018 while U − L is large 1. Simulate and estimate 2. Back-propagate Reachability in limited information stochastic games 5/6

The Algorithm Similar to Kelmendi et. al. 2018 while U − L is large 1. Simulate and estimate 2. Back-propagate The how ◮ Simulation finds important parts of state space Reachability in limited information stochastic games 5/6

The Algorithm Similar to Kelmendi et. al. 2018 while U − L is large 1. Simulate and estimate 2. Back-propagate The how ◮ Simulation finds important parts of state space ◮ Simulation computes Hoeffding confidence intervals ball around estimate such that real prob. falls in the ball with high confidence Reachability in limited information stochastic games 5/6

The Algorithm Similar to Kelmendi et. al. 2018 while U − L is large 1. Simulate and estimate 2. Back-propagate The how ◮ Simulation finds important parts of state space ◮ Simulation computes Hoeffding confidence intervals ball around estimate such that real prob. falls in the ball with high confidence ◮ Information conservatively back-propagated Reachability in limited information stochastic games 5/6

The Algorithm Similar to Kelmendi et. al. 2018 while U − L is large 1. Simulate and estimate 2. Back-propagate The how ◮ Simulation finds important parts of state space ◮ Simulation computes Hoeffding confidence intervals ball around estimate such that real prob. falls in the ball with high confidence ◮ Information conservatively back-propagated ◮ Other tricks to ensure fixpoint convergence Reachability in limited information stochastic games 5/6

Conclusion ◮ Algorithm for reachability in limited information MDP/SG result ∈ [0 . 6 − ǫ, 0 . 6 + ǫ ] with prob of going wrong 10 − 8 ◮ Implemented and benchmarked in PRISM Model Checker ◮ First algorithm to do so for SG ◮ First practical algorithm for MDPs Reachability in limited information stochastic games 6/6

PAC Statistical Model Checking for Markov Decision Processes and - PowerPoint PPT Presentation

PAC Statistical Model Checking for Markov Decision Processes and Stochastic Games 1 Pranav Ashok, Jan K ret nsk y, Maximilian Weininger Technical University of Munich Highlights of Logic, Automata and Games Warsaw, Poland September

Statistical Statistical Statistical Model Statistical Model Model Checking Model Checking

Statistical Model Checking for Markov Decision Processes David Henriques Joint work with Jo

Model checking quantum Markov chains Yuan Feng, Nengkun Yu, and Mingsheng Ying University of

Model checking perhaps the most important part of applied statistical modelling Simon Wood

Model Checking Continuous-Time Markov Chains Joost-Pieter Katoen Software Modeling and

Model Checking Continuous-Time Markov Chains Joost-Pieter Katoen Software Modeling and

Hidden Markov Models Pratik Lahiri Introduction A hidden Markov model (HMM) is a

Statistical Model Checking and Rare Events Paolo Zuliani Joint work with Edmund M. Clarke

time Markov Models Mohmmadsadegh Mohagheghi And Behrang Chaboki Vali-e-Asr University of

Statistical Model Checking of Simulink Models Simulink Models Ed Edmund M. Clarke d M Cl k

Probabilistic Model Checking Probabilistic Model Checking Marta Kwiatkowska Kwiatkowska Marta

Hoare Logic and Model Checking Model Checking Lecture 11: Model checking for Computation Tree

Limit-Deterministic Bchi Automata for Probabilistic Model Checking Jan Ketnsk Javier

Limit-Deterministic Bchi Automata for Probabilistic Model Checking Javier Esparza Jan K

Games for discrete-time Markov chain and their application to verification Shota Nakagawa The

Model Checking Continuous-Time Markov Chains Joost-Pieter Katoen Software Modeling and

L ECTURE 15: Regrade requests: L EARNING T HEORY Send us email, and come and see me next

SecLabel: Enhancing RISC-V Platform Security with Labelled Architecture Zhenyu Ning 1,2 , Yinqian

Protocol for Carrying Authentication for Network Access (PANA) (draft-ietf-pana-pana-00.txt)

Albany Medical Center Hospital PPS PAC Meeting July 18, 2016 Meeting Attendance Please email

Meeting January 10, 2018 2 Agenda Networking 1. CCB Update: Where Weve Been and Where

Computational Learning Theory Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch.

La th eorie PAC-Bayes en apprentissage supervis e Pr esentation au LRI de luniversit

Money 101 2019 PAC Conference March 30, 2019 Background Owner of Green Mountain Financial