time Markov Models Mohmmadsadegh Mohagheghi And Behrang Chaboki - - PowerPoint PPT Presentation

Dirac-based Reduction Techniques for Quantitative Analysis of Discrete- time Markov Models

Mohmmadsadegh Mohagheghi And Behrang Chaboki Vali-e-Asr University of Rafsanjan

Probabilistic Model Checking

Probabilistic Model Checking

• A Markov Decision Process (MDP) is a tuple

M = (S, s0, Act, P, R) where:

• S is a set of states,
• s0 is the initial state,
• Act is a finite set of actions,
• P is a probabilistic transition function,
• R is a reward function.

Probabilistic Model Checking

• A policy is used to resolve non-deterministic

choices of an MDP.

• A policy 𝜌: 𝑇 → 𝐵𝑑𝑢 selects one action for

each state s.

• For an MDP M, every possible policy 𝜌 induces

a quotient DTMC.

Numeric Computations

• Reachability Probabilities: The (maximal or

minimal) probability of finally reaching one of the goal states [For MDPs]

• Expected Rewards: The (maximal or minimal)

expectation of accumulated rewards until reaching a goal state

Numeric Computations

• Extremal Reachablity Probabilities

& Expected Rewards

Solving a Linear Program (Exact Solutions) Using Iterative Methods (In practice)

Jacobi Iterative Method (DTMCs)

• Starting from an initial vector

𝑊 of value-states, in each iteration and every state 𝑡 ∈ 𝑇 update 𝑊𝑡 as:

 



α) Post(s, s' s' s

).V s' α, P(s, V

 

 

α) Post(s, s' s' s

).V s' α, P(s, R(s) V

Value Iteration (MDPs)

• Starting from an initial vector

𝑊of value-states, in each iteration and every state 𝑡 ∈ 𝑇 update 𝑊𝑡 as:

) ).V s' α, P(s, ( Max V

α) Post(s, s' s' Act(s) α s

 



 ) ).V s' α, P(s, (R(s) Max V

α) Post(s, s' s' Act(s) α s

 



 

Value Iteration (MDPs)

• Starting from an initial vector

𝑊 of value-states, in each iteration and every state s of S update 𝑊𝑡 as:

• Terminate criterion:

for some tiny ε (10^-6)

) ).V s' α, P(s, (R(s) Max V

α) Post(s, s' s' Act(s) α s

 



 

 



) V

• (V

Max

• ld

s s S s

Policy Iteration

Select a policy 𝜌 repeat Compute the values of the induced DTMC Update 𝜌 Until no change in policies

Dirac-based Reduction Technique

• Idea: If 𝑄(𝑡, 𝑡′) = 1 in a DTMC, reachability

probability of s and 𝑡′ are equal.

Dirac-based Reduction Technique

• Dirac transitions are used to classify 𝑇.
• The states of each class are connected with

Dirac transitions and have the same reachability probabilities.

• Apply iterative commutations on the reduced

DTMC

Dirac-based Reduction Technique

• DTMC reduction can be used for policy

iteration

• Time complexity: Linear in the size of DTMCs

Dirac-based Reduction Technique

• Expected rewards: If 𝑄(𝑡, 𝑡′) = 1 then 𝑊

𝑡′ =

𝑊

𝑡 + 𝑆(𝑡)

• State-rewards should be modified for reduced

DTMCs

Experimental Results

• We implemented Dirac-based methods in

PRISM.

• Available in:

https://github.com/sadeghrk/prism/tree/Dira cBased-Improving

Experimental Results

Questions?