Progress on Parameter Synthesis for Markov Models Joost-Pieter - - PowerPoint PPT Presentation

▶
progress on parameter synthesis for markov models
SMART_READER_LITE
LIVE PREVIEW

Progress on Parameter Synthesis for Markov Models Joost-Pieter - - PowerPoint PPT Presentation

Dec 27, 2023 173 likes •1.21k views

Progress on Parameter Synthesis for Markov Models Joost-Pieter Katoen Joint with: Christian Dehnert, Nils Jansen, Sebastian Junges, Tim Quatman, Erika brahm, Harold Bruintjes, Florian Corzilius, Ufuk Topcu, Murat Cubutceke, Ivan Papusha,

slide-1
SLIDE 1

Progress on Parameter Synthesis for Markov Models

Joost-Pieter Katoen

Joint with: Christian Dehnert, Nils Jansen, Sebastian Junges, Tim Quatman, Erika Ábrahám, Harold Bruintjes, Florian Corzilius, Ufuk Topcu, Murat Cubutceke, Ivan Papusha, Hasan Poonawala, Matthias Volk

SYNCOP +PV’17 Workshop@Uppsala

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 1/51

slide-2
SLIDE 2

Randomised Algorithms: Simulating a Die

[Knuth & Yao, 1976]

Heads = “go left”; tails = “go right”. Does this model a six-sided die?

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 2/51

slide-3
SLIDE 3

Reachability Probabilities: Knuth-Yao’s Die

▸ Consider the event ◇4

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 3/51

slide-4
SLIDE 4

Reachability Probabilities: Knuth-Yao’s Die

▸ Consider the event ◇4 ▸ We obtain:

x1 = x2 = x3 = x5 = x6 = 0 and x4 = 1

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 3/51

slide-5
SLIDE 5

Reachability Probabilities: Knuth-Yao’s Die

▸ Consider the event ◇4 ▸ We obtain:

x1 = x2 = x3 = x5 = x6 = 0 and x4 = 1 xs1 = xs3 = xs4 = 0

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 3/51

slide-6
SLIDE 6

Reachability Probabilities: Knuth-Yao’s Die

▸ Consider the event ◇4 ▸ We obtain:

x1 = x2 = x3 = x5 = x6 = 0 and x4 = 1 xs1 = xs3 = xs4 = 0 xs0 = 1

2xs1 + 1 2xs2

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 3/51

slide-7
SLIDE 7

Reachability Probabilities: Knuth-Yao’s Die

▸ Consider the event ◇4 ▸ We obtain:

x1 = x2 = x3 = x5 = x6 = 0 and x4 = 1 xs1 = xs3 = xs4 = 0 xs0 = 1

2xs1 + 1 2xs2

xs2 = 1

2xs5 + 1 2xs6

xs5 = 1

2x5 + 1 2x4

xs6 = 1

2xs2 + 1 2x6

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 3/51

slide-8
SLIDE 8

Reachability Probabilities: Knuth-Yao’s Die

▸ Consider the event ◇4 ▸ We obtain:

x1 = x2 = x3 = x5 = x6 = 0 and x4 = 1 xs1 = xs3 = xs4 = 0 xs0 = 1

2xs1 + 1 2xs2

xs2 = 1

2xs5 + 1 2xs6

xs5 = 1

2x5 + 1 2x4

xs6 = 1

2xs2 + 1 2x6

▸ Gaussian elimination yields:

xs5 = 1

2, xs2 = 1 3, xs6 = 1 6, and xs0 = 1 6

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 3/51

slide-9
SLIDE 9

Correctness

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 4/51

slide-10
SLIDE 10

Reachability Probabilities are Pivotal

▸ Repeated reachability Pr(s ⊧ ◻◇G):

Probability to reach a terminal SCC containing a G-state

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 5/51

slide-11
SLIDE 11

Reachability Probabilities are Pivotal

▸ Repeated reachability Pr(s ⊧ ◻◇G):

Probability to reach a terminal SCC containing a G-state

▸ Probabilistic CTL model checking

Recursive descent on parse tree using reach-probabilities at nodes

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 5/51

slide-12
SLIDE 12

Reachability Probabilities are Pivotal

▸ Repeated reachability Pr(s ⊧ ◻◇G):

Probability to reach a terminal SCC containing a G-state

▸ Probabilistic CTL model checking

Recursive descent on parse tree using reach-probabilities at nodes

▸ LTL formulas Pr(s ⊧ ϕ):

  • 1. Transform ϕ into a deterministic (Rabin) automaton
  • 2. Take the product of the Markov chain and the automaton
  • 3. Determine the probability to reach an accepting terminal SCC

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 5/51

slide-13
SLIDE 13

Reliability Engineering

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 6/51

slide-14
SLIDE 14

Fault Tree Analysis

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 7/51

slide-15
SLIDE 15

Reliability: Dynamic Fault Trees

[Dugan et al., 1992]

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 8/51

slide-16
SLIDE 16

Dynamic Fault Trees

Markov chain process for a DFT

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 9/51

slide-17
SLIDE 17

Dynamic Fault Trees

Markov chain process for a DFT

(D)FTs: one of —if not the— most prominent models for risk analysis Aims: quantify system reliability and availability, MTTF, ... ...

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 9/51

slide-18
SLIDE 18

Probabilistic Model Checkers

▸ PRISM1

[Kwiatkowska, Parker et al.]

▸ MRMC

[Katoen et al.]

▸ iscasMC

[Zhang et al.]

▸ iBioSim

[Myers et al.]

▸ GreatSPN

[Franceschinis et al.]

▸ SMART

[Ciardo et al.]

▸ MarCie

[Heiner et al.]

▸ PAT

[Song Dong et al.]

▸ storm

[Dehnert, Katoen et al.]

▸ ... ...

Statistical model checkers: Ymer, Vesta, UppAal, APMC, PlasmaLab, ... ...

1Recipient HVC Award 2016. Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 10/51

slide-19
SLIDE 19

The Need for Parameter Synthesis

Fact: Probabilistic model checking is applicable to various areas, e.g.:

▸ fault trees ▸ randomised algorithms ▸ systems biology

Limitation: Probabilities need to be known a priori. Is this a valid assumption? How sensitive are results when transition probabilities fluctuate? Goal: Treat parametric models, synthesise “safe” parameter values

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 11/51

slide-20
SLIDE 20

Biased Knuth-Yao’s Die

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 12/51

slide-21
SLIDE 21

Biased Knuth-Yao’s Die

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 13/51

slide-22
SLIDE 22

Parametric Knuth-Yao’s Die

For which 1/10 ⩽ p ⩽ 9/10 and 2/5 ⩽ q ⩽ 3/5 does Pr(◇2) ⩾

3/20 hold?

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 14/51

slide-23
SLIDE 23

Conditional Probabilities

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 15/51

slide-24
SLIDE 24

Parameter Synthesis

Inputs:

  • 1. a (finite) parametric Markov model
  • 2. a property (e.g., reachability, expected reward, conditional reachability)
  • 3. a threshold

Output: For which parameter values does the pMC satisfy the property with the given threshold?

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 16/51

slide-25
SLIDE 25

Parameter Synthesis

Inputs:

  • 1. a (finite) parametric Markov model
  • 2. a property (e.g., reachability, expected reward, conditional reachability)
  • 3. a threshold

Output: For which parameter values does the pMC satisfy the property with the given threshold? Problem instances:

▸ What is the maximal tolerable message loss ensuring delivery ⩾ 98%? ▸ ... the tolerable failure rate in a DFT ensuring MTTF ⩾ 3 hours? ▸ ... ...

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 16/51

slide-26
SLIDE 26

Recall Dynamic Fault Trees

Markov chain process for a DFT

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 17/51

slide-27
SLIDE 27

Parametric Fault Trees

1 2 3 4 5 1 2 3 4 5

β MTTF

Sample parametric DFT and its MTTF

MTTF = 200x2 + 20x + 201 x⋅(20x + 201) for (α,β,γ,d) = (10,x,0.1,0.5)

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 18/51

slide-28
SLIDE 28

Parameter Synthesis

Aim: partition the parameter space into safe and unsafe regions

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 19/51

slide-29
SLIDE 29

Parameter Synthesis

Aim: partition the parameter space into safe and unsafe regions

▸ Region = half-space defined by linear inequalities over the parameters ▸ A region R for threshold ⩽ β is safe if no MC with v ∈ R exceeds β ▸ A region R for threshold ⩽ β is unsafe if no MC with v ∈ R is at most β

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 19/51

slide-30
SLIDE 30

Parameter Synthesis

Aim: partition the parameter space into safe and unsafe regions

▸ Region = half-space defined by linear inequalities over the parameters ▸ A region R for threshold ⩽ β is safe if no MC with v ∈ R exceeds β ▸ A region R for threshold ⩽ β is unsafe if no MC with v ∈ R is at most β

We present two approaches:

  • 1. An exact procedure.

How? Using SMT techniques

  • 2. An approximate technique.

How? Using parameter lifting

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 19/51

slide-31
SLIDE 31

Computing Rational Functions

[Daws, 2004]

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 20/51

slide-32
SLIDE 32

Computing Rational Functions

[Daws, 2004]

Pr (s0 ⊧ ◇(1 or 3)) ⩽ 1/3 iff p⋅q⋅ 1−p

1−p⋅q + p2⋅ 1−q 1−p⋅q ⩽ 1/3

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 20/51

slide-33
SLIDE 33

Computing Rational Functions

[Daws, 2004]

Pr (s0 ⊧ ◇(1 or 3)) ⩽ 1/3 iff p⋅q⋅ 1−p

1−p⋅q + p2⋅ 1−q 1−p⋅q ⩽ 1/3

This may yield large high-degree rational functions.

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 20/51

slide-34
SLIDE 34

Resulting Rational Functions

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 21/51

slide-35
SLIDE 35

Zooming In

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 22/51

slide-36
SLIDE 36

Zooming In

41 states, 138 transitions, 2 parameters: numerator = 48 terms, denominator = product of 48 (linear) polynomials

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 22/51

slide-37
SLIDE 37

Zooming In

41 states, 138 transitions, 2 parameters: numerator = 48 terms, denominator = product of 48 (linear) polynomials ⇒ Use bisimulation, SCC-decomposition and efficient gcd-computation

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 22/51

slide-38
SLIDE 38

Hierarchical SCC Decomposition

[Jansen et al., 2014]

1 2 3 4 5 6 7 8 9 1 1 0.4 0.2 0.4 0.8 0.2 1 q 1 − q 0.8 0.2 0.5 0.3 p 1 − p 0.2 S2.1 S2 S1 S

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 23/51

slide-39
SLIDE 39

Hierarchical SCC Decomposition

[Jansen et al., 2014]

1 2 3 4 5 6 7 8 9 1 1 0.4 0.2 0.4 0.8 0.2 1 q 1 − q 0.8 0.2 0.5 0.3 p 1 − p 0.2 S2.1

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 23/51

slide-40
SLIDE 40

Hierarchical SCC Decomposition

[Jansen et al., 2014]

1 2 3 4 5 6 S2.1 9 1 1 0.4 0.2 0.4 0.8 0.2 1 q 1 − q 0.8 0.2p 0.5p 0.3p 1 − p 0.2

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 23/51

slide-41
SLIDE 41

Hierarchical SCC Decomposition

[Jansen et al., 2014]

1 2 3 4 5 6 S2.1 9 1 1 0.4 0.2 0.4 0.8 0.2 1 q 1 − q 0.2 0.8

0.2p 1−0.3p 0.5p 1−0.3p 1−p 1−0.3p

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 23/51

slide-42
SLIDE 42

Hierarchical SCC Decomposition

[Jansen et al., 2014]

1 2 3 4 5 6 S2.1 9 1 1 0.4 0.2 0.4 0.8 0.2 1 q 1 − q 0.2 0.8

0.2p 1−0.3p 0.5p 1−0.3p 1−p 1−0.3p

S2

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 23/51

slide-43
SLIDE 43

Hierarchical SCC Decomposition

[Jansen et al., 2014]

1 2 3 4 5 S2 9 1 1 0.4 0.2 0.4 0.8 0.2 1 q 1 − q

0.2−0.06p 1−0.7p 0.16p 1−0.7p 0.8−0.8p 1−0.7p

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 23/51

slide-44
SLIDE 44

Hierarchical SCC Decomposition

[Jansen et al., 2014]

1 2 3 4 5 S2 9 1 1 0.4 0.2 0.4 0.8 0.2 1 q 1 − q

0.2−0.06p 1−0.7p 0.16p 1−0.7p 0.8−0.8p 1−0.7p

S1

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 23/51

slide-45
SLIDE 45

Hierarchical SCC Decomposition

[Jansen et al., 2014]

1 S12 S13 5 S2 9 1 1 0.4 0.2 0.4

0.2−0.06p 1−0.7p 0.16p 1−0.7p 0.8−0.8p 1−0.7p 0.8−0.8q 1−0.8q 0.2 1−0.8q 1−q 1−0.8q 0.2q 1−0.8q

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 23/51

slide-46
SLIDE 46

Hierarchical SCC Decomposition

[Jansen et al., 2014]

1 S12 S13 5 S2 9 1 1 0.4 0.2 0.4

0.2−0.06p 1−0.7p 0.16p 1−0.7p 0.8−0.8p 1−0.7p 0.8−0.8q 1−0.8q 0.2 1−0.8q 1−q 1−0.8q 0.2q 1−0.8q

S

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 23/51

slide-47
SLIDE 47

Hierarchical SCC Decomposition

[Jansen et al., 2014]

S

5 9 1 1

−0.2872p−0.52q+0.3192pq+0.52 −0.6712p−0.744q+0.5432pq+0.904 −0.384p−0.224q+0.224pq+0.384 −0.6712p−0.744q+0.5432pq+0.904

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 23/51

slide-48
SLIDE 48

Hierarchical SCC Decomposition

[Jansen et al., 2014]

S

5 9 1 1

−0.2872p−0.52q+0.3192pq+0.52 −0.6712p−0.744q+0.5432pq+0.904 −0.384p−0.224q+0.224pq+0.384 −0.6712p−0.744q+0.5432pq+0.904

For which (combinations of) values for p and q is the probability of reaching5smaller than c ∈ [0, 1]? ⇒ Evaluate rational function.

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 23/51

slide-49
SLIDE 49

Exploiting SMT

Goal: partition parameter space in regions R that are either safe or unsafe Idea: generate region candidates R and ask SMT solver2 for counterexample

2Over non-linear real arithmetic using Z3 or SMT-RAT. Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 24/51

slide-50
SLIDE 50

Exploiting SMT

Goal: partition parameter space in regions R that are either safe or unsafe Idea: generate region candidates R and ask SMT solver2 for counterexample

2Over non-linear real arithmetic using Z3 or SMT-RAT. Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 24/51

slide-51
SLIDE 51

CEGAR-Like Parameter Synthesis

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 25/51

slide-52
SLIDE 52

CEGAR-Like Parameter Synthesis

For which 1/10 ⩽ p ⩽ 9/10 and 2/5 ⩽ q ⩽ 3/5 does Pr(◇2) ⩾

3/20 hold?

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 26/51

slide-53
SLIDE 53

CEGAR-Like Parameter Synthesis

For which 1/10 ⩽ p ⩽ 9/10 and 2/5 ⩽ q ⩽ 3/5 does Pr(◇2) ⩾

3/20 hold?

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 26/51

slide-54
SLIDE 54

CEGAR-Like Parameter Synthesis

For which 1/10 ⩽ p ⩽ 9/10 and 2/5 ⩽ q ⩽ 3/5 does Pr(◇2) ⩾

3/20 hold?

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 26/51

slide-55
SLIDE 55

CEGAR-Like Parameter Synthesis

For which 1/10 ⩽ p ⩽ 9/10 and 2/5 ⩽ q ⩽ 3/5 does Pr(◇2) ⩾

3/20 hold?

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 26/51

slide-56
SLIDE 56

Experimental Results

[Dehnert et al., 2015] competitors

▸ PARAM [Hahn et al., 2010] ▸ PRISM [Parker et al., 2011]

models

▸ Bounded retransmission protocol ▸ NAND multiplexing ▸ Zeroconf, Crowds protocol ▸ 104 to 7.5 ⋅ 106 states

experiments:

▸ best set-up for each tool ▸ log-scale x- and y-axis

runner-up in the CAV 2015 artefact evaluation

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 27/51

slide-57
SLIDE 57

Experimental Results

[Dehnert et al., 2015] competitors

▸ PARAM [Hahn et al., 2010] ▸ PRISM [Parker et al., 2011] ▸ prototype [Baier et al., 2014]

models

▸ Bounded retransmission protocol ▸ NAND multiplexing ▸ Zeroconf, Crowds protocol ▸ 104 to 7.5 ⋅ 106 states

experiments:

▸ best set-up for each tool ▸ log-scale x- and y-axis

runner-up in the CAV 2015 artefact evaluation

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 27/51

slide-58
SLIDE 58

Parameter Synthesis using SMT

Pros:

▸ Exact results: rational function is an exact symbolic object ▸ Drastic improvements over existing tools

PARAM and PRISM

▸ User-friendly representation

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 28/51

slide-59
SLIDE 59

Parameter Synthesis using SMT

Pros:

▸ Exact results: rational function is an exact symbolic object ▸ Drastic improvements over existing tools

PARAM and PRISM

▸ User-friendly representation

Cons:

▸ Rational function requires many gcd-computations

> 4 parameters?

▸ SMT performance unpredictable

heuristics hard

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 28/51

slide-60
SLIDE 60

Parameter Synthesis using SMT

Pros:

▸ Exact results: rational function is an exact symbolic object ▸ Drastic improvements over existing tools

PARAM and PRISM

▸ User-friendly representation

Cons:

▸ Rational function requires many gcd-computations

> 4 parameters?

▸ SMT performance unpredictable

heuristics hard

Can we do better by sacrificing exactness? Yes.

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 28/51

slide-61
SLIDE 61

Approximate Parameter Synthesis

[Quatmann et al,, 2016]

Let transition probabilities be linear in each variable. That is, transition functions f are multi-affine multivariate polynomials of form: f = ∑ai ⋅ (∏

x∈V

x) with ai ∈ Q Examples: 3x⋅y + 4y⋅z, 1 − x, x⋅y⋅z etc.

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 29/51

slide-62
SLIDE 62

Approximate Parameter Synthesis

[Quatmann et al,, 2016]

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 29/51

slide-63
SLIDE 63

Approximate Parameter Synthesis

[Quatmann et al,, 2016]

Two-phase approach: first remove dependencies, then substitute extremal values

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 29/51

slide-64
SLIDE 64

Approximate Parameter Synthesis

[Quatmann et al,, 2016]

Two-phase approach: first remove dependencies, then substitute extremal values Also applicable to parametric MDPs.

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 29/51

slide-65
SLIDE 65

Phase 1: Relaxation

Parameter dependencies are removed; Pr(◇2) = (1 − z) ⋅

1−q 1−p⋅q

⇒ each state is equipped with its own parameter

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 30/51

slide-66
SLIDE 66

Phase 1: Relaxation

Correctness:

▸ Relaxed regions contain more valuations than original regions

⇒ Relaxation yields over-approximations ⇒ Relaxation preserves upper-bounds on reachability probs

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 31/51

slide-67
SLIDE 67

Phase 1: Relaxation

Correctness:

▸ Relaxed regions contain more valuations than original regions

⇒ Relaxation yields over-approximations ⇒ Relaxation preserves upper-bounds on reachability probs Complexity of parameter synthesis :

▸ Relaxation increases the number of parameters ▸ Extremal values of the state parameters attain maximal probabilities

⇒ Valuations for maximal probabilities are easier to find

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 31/51

slide-68
SLIDE 68

Phase 2: Substitution

Local parameters per state ⇒ extremal values at states suffice

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 32/51

slide-69
SLIDE 69

Phase 2: Substitution

Local parameters per state ⇒ extremal values at states suffice

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 33/51

slide-70
SLIDE 70

Phase 2: Substitution

This results in a Markov decision process. Its extremal reachability probabilities provide bounds for parametric MC.

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 34/51

slide-71
SLIDE 71

Parameter Synthesis

Until ≈ 95% of the parameter space is covered

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 35/51

slide-72
SLIDE 72

Parameter Synthesis

Until 95% of the parameter space is covered

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 36/51

slide-73
SLIDE 73

Coverage

ϕ n # states # trans % p t safe unsafe neither unkn pMC brp E 2 20 744 27 651 48% 51 14.9% 79.2% 5.8% 0.2% E 4 20 744 27 651 48% 71 7.5% 51.0% 40.6% 0.8% crowds P 2 104 512 246 082 19% 44 54.4% 41.1% 4.2% 0.3% nand P 2 35 112 52 647 47% 21 21.4% 68.5% 6.9% 3.2% pMDP brp P 2 40 721 55 143 50% 153 6.6% 90.4% 3.0% 0.0% cons P 4 22 656 75 232 41% 357 2.6% 87.0% 10.4% 0.0% sav P 4 379 1 127 50% 2 44.0% 15.4% 35.4% 5.3% zconf P 2 88 858 203 550 40% 186 16.6% 77.3% 5.6% 0.5%

Parameter space R = [10−5,1−10−5]n until 95% coverage for n parameters for 625 equally-sized regions without region refinement

single core, 2.0 GHz, 30GB RAM, TO = one hour

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 37/51

slide-74
SLIDE 74

Parametric Markov Chain Benchmarks

PLA PRISM benchmark instance ϕ #pars #states #trans #regions direct bisim best brp (256,5) P 2 19 720 26 627 37 6 14 TO (4096,5) P 2 315 400 425 987 13 233 TO TO (256,5) E 2 20 744 27 651 195 8 15 TO (4096,5) E 2 331 784 442 371 195 502 417 TO (16,5) E 4 1 304 1 731 1 251 220 2 764 1 597 TO (32,5) E 4 2 600 3 459 1 031 893 TO 2 722 TO (256,5) E 4 20 744 27 651 – TO TO TO crowds (10,5) P 2 104 512 246 082 123 17 6 2038 (15,7) P 2 8 364 409 25 108 729 116 1 880 518 TO (20,7) P 2 45 421 597 164 432 797 119 TO 2 935 TO nand (10,5) P 2 35 112 52 647 469 22 30 TO (25,5) P 2 865 592 1 347 047 360 735 2 061 TO

coverage of 95%; refinement into four equally-sized regions SMT approach needs >one hour on all instances.

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 38/51

slide-75
SLIDE 75

Parametric MDP Benchmarks

PLA PRISM benchmark instance ϕ #pars #states #trans #regions direct bisim best brp (256,5) P 2 40 721 55 143 37 35 3 359 TO (4096,5) P 2 647 441 876 903 13 3 424 TO TO consensus (2,2) P 2 272 492 119 < 1 < 1 31 (2,32) P 2 4 112 7 692 108 113 141 TO (4,2) P 4 22 656 75 232 6 125 1 866 2 022 TO (4,4) P 4 43 136 144 352 – TO TO TO sav (6,2,2) P 2 379 1 127 162 < 1 < 1 TO (100,10,10) P 2 1 307 395 6 474 535 37 1 612 TO TO (6,2,2) P 4 379 1 127 621 175 944 917 TO (10,3,3) P 4 1 850 6 561 TO TO TO zeroconf (2) P 2 88 858 203 550 186 86 1 295 TO (5) P 2 494 930 1 133 781 403 2 400 TO TO

coverage of 95%

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 39/51

slide-76
SLIDE 76

Summary So Far

SMT-based approach:

▸ Exact ▸ Requires rational functions ▸ Fickle SMT performance ▸ ≈ 106 states, 2 parameters ▸ Restricted to Markov chains ▸ CEGAR-like refinement

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 40/51

slide-77
SLIDE 77

Summary So Far

SMT-based approach:

▸ Exact ▸ Requires rational functions ▸ Fickle SMT performance ▸ ≈ 106 states, 2 parameters ▸ Restricted to Markov chains ▸ CEGAR-like refinement

Parameter lifting approach:

▸ Approximative ▸ Off-the-shelf model checking ▸ No SMT, no rational functions ▸ ≈ 107 states, 4–5 parameters ▸ Applicable to MDPs and games ▸ CEGAR-like refinement

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 40/51

slide-78
SLIDE 78

Multiple Objectives

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 41/51

slide-79
SLIDE 79

Multiple Objectives

Inputs:

  • 1. a (finite) parametric MDP M over V = {x1,... ,xn }

with signomial parameter functions c ⋅ xa1

1 ⋅ ... ⋅ xan n for c ∈ R

  • 2. multiple objectives ϕ1,... ,ϕm (reachability, expected reward)
  • 3. objective function f over V :

N

∑

k=1

ck ⋅ xa1k

1

⋅ ... ⋅ xank

n

for ck ∈ R

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 41/51

slide-80
SLIDE 80

Multiple Objectives

Inputs:

  • 1. a (finite) parametric MDP M over V = {x1,... ,xn }

with signomial parameter functions c ⋅ xa1

1 ⋅ ... ⋅ xan n for c ∈ R

  • 2. multiple objectives ϕ1,... ,ϕm (reachability, expected reward)
  • 3. objective function f over V :

N

∑

k=1

ck ⋅ xa1k

1

⋅ ... ⋅ xank

n

for ck ∈ R Output: A (randomised) policy σ and valuation u such that: Mσ[u] ⊧ ϕ1 ∧ ... ∧ ϕm ÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜ

“feasibility”

and the objective f is minimised ÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜ

“optimality”

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 41/51

slide-81
SLIDE 81

Multiple Objectives

Inputs:

  • 1. a (finite) parametric MDP M over V = {x1,... ,xn }

with signomial parameter functions c ⋅ xa1

1 ⋅ ... ⋅ xan n for c ∈ R

  • 2. multiple objectives ϕ1,... ,ϕm (reachability, expected reward)
  • 3. objective function f over V :

N

∑

k=1

ck ⋅ xa1k

1

⋅ ... ⋅ xank

n

for ck ∈ R Output: A (randomised) policy σ and valuation u such that: Mσ[u] ⊧ ϕ1 ∧ ... ∧ ϕm ÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜ

“feasibility”

and the objective f is minimised ÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜ

“optimality” multi-objective MDP: use LP [Etessami et al., 2008] multi-objective parametric MDP: use special type NLP [Cubuktepe et al., 2017]

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 41/51

slide-82
SLIDE 82

NLP for Two Objectives

Objectives: minimise f , reach T with probability ⩽ p, expected cost to reach G ⩽ c Subject to: psI ⩽ p reachability objective csI ⩽ c expected reward objective

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 42/51

slide-83
SLIDE 83

NLP for Two Objectives

Objectives: minimise f , reach T with probability ⩽ p, expected cost to reach G ⩽ c Subject to: psI ⩽ p reachability objective csI ⩽ c expected reward objective ∀s ∶ ∑

α∈Act(s)

σs,α = 1 randomised scheduler ∀s,α ∶ 0 ⩽ σs,α ⩽ 1

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 42/51

slide-84
SLIDE 84

NLP for Two Objectives

Objectives: minimise f , reach T with probability ⩽ p, expected cost to reach G ⩽ c Subject to: psI ⩽ p reachability objective csI ⩽ c expected reward objective ∀s ∶ ∑

α∈Act(s)

σs,α = 1 randomised scheduler ∀s,α ∶ 0 ⩽ σs,α ⩽ 1 ∀s,α ∶ ∑

t∈S

P(s,α,t) = 1 probabilistic choice ∀s,t,α ∶ 0 ⩽ P(s,α,t) ⩽ 1

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 42/51

slide-85
SLIDE 85

NLP for Two Objectives

Objectives: minimise f , reach T with probability ⩽ p, expected cost to reach G ⩽ c Subject to: psI ⩽ p reachability objective csI ⩽ c expected reward objective ∀s ∶ ∑

α∈Act(s)

σs,α = 1 randomised scheduler ∀s,α ∶ 0 ⩽ σs,α ⩽ 1 ∀s,α ∶ ∑

t∈S

P(s,α,t) = 1 probabilistic choice ∀s,t,α ∶ 0 ⩽ P(s,α,t) ⩽ 1 ∀s ∈ T ∶ ps = 1 reach prob of T ∀s / ∈ T ∶ ps = ∑

α∈Act(s)

σs,α ⋅ ∑

t∈S

P(s,α,t)⋅pt transition probabilities

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 42/51

slide-86
SLIDE 86

NLP for Two Objectives

Objectives: minimise f , reach T with probability ⩽ p, expected cost to reach G ⩽ c Subject to: psI ⩽ p reachability objective csI ⩽ c expected reward objective ∀s ∶ ∑

α∈Act(s)

σs,α = 1 randomised scheduler ∀s,α ∶ 0 ⩽ σs,α ⩽ 1 ∀s,α ∶ ∑

t∈S

P(s,α,t) = 1 probabilistic choice ∀s,t,α ∶ 0 ⩽ P(s,α,t) ⩽ 1 ∀s ∈ T ∶ ps = 1 reach prob of T ∀s / ∈ T ∶ ps = ∑

α∈Act(s)

σs,α ⋅ ∑

t∈S

P(s,α,t)⋅pt transition probabilities ∀s ∈ G ∶ cs = 0 expected cost of G ∀s / ∈ G ∶ cs = ∑

α∈Act(s)

σs,α ⋅ (c(s,α) + ∑

t∈S

P(s,α,t)⋅ct) expected costs

Theorem: This NLP is sound and complete. But solving NLPs is exponential.

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 42/51

slide-87
SLIDE 87

Can We Do Better?

Yes.

  • 1. Get a feasible solution in polynomial time3. How? Geometric programming.
  • 2. Get local optimum. How? Sequential convex programming.

Solutions are approximations that can be arbitrarily close.

3Approximation of arbitrarily precise results by interior point methods with barriers Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 43/51

slide-88
SLIDE 88

Geometric Programming

Objective: minimise f ∶∶

N

∑

k=1

ck ⋅ xa1k

1

⋅ ... ⋅ xank

n

for ck ∈ R⩾0 Subject to:

∀i ∈ [1..m] ∶ gi ⩽ 1

posynomial gi

∀j ∈ [1..ℓ] ∶ hj = 1

monomial hj

Division transformation: f ⩽ h if and only if f

h ⩽ 1

Relaxation: f = h implies f ⩽ h if and only if f

h ⩽ 1

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 44/51

slide-89
SLIDE 89

Convexification

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 45/51

slide-90
SLIDE 90

Lifting

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 46/51

slide-91
SLIDE 91

GP for Two Objectives

Objectives: reach T with probability ⩽ p, expected cost to reach G ⩽ c Subject to:

psI p

⩽ 1 reachability

csI c

⩽ 1 expected reward

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 47/51

slide-92
SLIDE 92

GP for Two Objectives

Objectives: reach T with probability ⩽ p, expected cost to reach G ⩽ c Subject to:

psI p

⩽ 1 reachability

csI c

⩽ 1 expected reward ∀s ∶ ∑

α∈Act(s)

σs,α ⩽ 1 randomised scheduler ∀s,α ∶ σs,α ⩽ 1

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 47/51

slide-93
SLIDE 93

GP for Two Objectives

Objectives: reach T with probability ⩽ p, expected cost to reach G ⩽ c Subject to:

psI p

⩽ 1 reachability

csI c

⩽ 1 expected reward ∀s ∶ ∑

α∈Act(s)

σs,α ⩽ 1 randomised scheduler ∀s,α ∶ σs,α ⩽ 1 ∀s,α ∶ ∑

t∈S

P(s,α,t) ⩽ 1 probabilistic choice ∀s,t,α ∶ P(s,α,t) ⩽ 1

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 47/51

slide-94
SLIDE 94

GP for Two Objectives

Objectives: reach T with probability ⩽ p, expected cost to reach G ⩽ c Subject to:

psI p

⩽ 1 reachability

csI c

⩽ 1 expected reward ∀s ∶ ∑

α∈Act(s)

σs,α ⩽ 1 randomised scheduler ∀s,α ∶ σs,α ⩽ 1 ∀s,α ∶ ∑

t∈S

P(s,α,t) ⩽ 1 probabilistic choice ∀s,t,α ∶ P(s,α,t) ⩽ 1 ∀s ∈ T ∶ ps = 1 reach prob of T ∀s / ∈ T ∶ ∑α σs,α ⋅ ∑t∈S P(s,α,t)⋅pt ps ⩽ 1 transition probabilities

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 47/51

slide-95
SLIDE 95

GP for Two Objectives

Objectives: reach T with probability ⩽ p, expected cost to reach G ⩽ c Subject to:

psI p

⩽ 1 reachability

csI c

⩽ 1 expected reward ∀s ∶ ∑

α∈Act(s)

σs,α ⩽ 1 randomised scheduler ∀s,α ∶ σs,α ⩽ 1 ∀s,α ∶ ∑

t∈S

P(s,α,t) ⩽ 1 probabilistic choice ∀s,t,α ∶ P(s,α,t) ⩽ 1 ∀s ∈ T ∶ ps = 1 reach prob of T ∀s / ∈ T ∶ ∑α σs,α ⋅ ∑t∈S P(s,α,t)⋅pt ps ⩽ 1 transition probabilities ∀s / ∈ G ∶ ∑α σs,α ⋅ (c(s,α) + ∑t∈S P(s,α,t)⋅ct) cs ⩽ 1 expected costs

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 47/51

slide-96
SLIDE 96

Correctness

Use the objective function F now4 Minimise ∑

p∈V

1 p + ∑

p∈L

1 p + ∑

s,α

1 σs,α yields that all variables p, p and σs,α are maximised.

Theorem: The GP with objective function F yields a feasible solution. Solving this GP can be done in polynomial time.

4Note: the original objective function f is dropped. Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 48/51

slide-97
SLIDE 97

Experimental Results

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 49/51

slide-98
SLIDE 98

Experimental Results

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 50/51

slide-99
SLIDE 99

Epilogue

SMT-based approach:

▸ Exact ▸ Requires rational functions ▸ Fickle SMT performance ▸ ≈ 106 states, 2 parameters ▸ Restricted to Markov chains ▸ CEGAR-like refinement

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 51/51

slide-100
SLIDE 100

Epilogue

SMT-based approach:

▸ Exact ▸ Requires rational functions ▸ Fickle SMT performance ▸ ≈ 106 states, 2 parameters ▸ Restricted to Markov chains ▸ CEGAR-like refinement

Parameter lifting approach:

▸ Approximative ▸ Off-the-shelf model checking ▸ No SMT, no rational functions ▸ ≈ 107 states, 4–5 parameters ▸ Applicable to MDPs and games ▸ CEGAR-like refinement

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 51/51

slide-101
SLIDE 101

Epilogue

SMT-based approach:

▸ Exact ▸ Requires rational functions ▸ Fickle SMT performance ▸ ≈ 106 states, 2 parameters ▸ Restricted to Markov chains ▸ CEGAR-like refinement

Geometric programming approach:

▸ Numerical approximation ▸ Multiple objectives ▸ ≈ 105 states, 10 parameters ▸ Applicable to MDPs ▸ Possibility of richer objectives

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 51/51

slide-102
SLIDE 102

Epilogue

SMT-based approach:

▸ Exact ▸ Requires rational functions ▸ Fickle SMT performance ▸ ≈ 106 states, 2 parameters ▸ Restricted to Markov chains ▸ CEGAR-like refinement

Geometric programming approach:

▸ Numerical approximation ▸ Multiple objectives ▸ ≈ 105 states, 10 parameters ▸ Applicable to MDPs ▸ Possibility of richer objectives

Significant progress in the last couple of years.

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 51/51

slide-103
SLIDE 103

Epilogue

SMT-based approach:

▸ Exact ▸ Requires rational functions ▸ Fickle SMT performance ▸ ≈ 106 states, 2 parameters ▸ Restricted to Markov chains ▸ CEGAR-like refinement

Geometric programming approach:

▸ Numerical approximation ▸ Multiple objectives ▸ ≈ 105 states, 10 parameters ▸ Applicable to MDPs ▸ Possibility of richer objectives

Significant progress in the last couple of years.

More info: QEST’14, CAV’15, ATVA’16, TACAS’17 and http://moves.rwth-aachen.de/research/tools/prophesy/

Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 51/51