Computing limit expectations of imprecise continuous-time Markov - - PowerPoint PPT Presentation

Computing limit expectations

• f imprecise continuous-time Markov chains

Alexander Erreygers Jasper De Bock WPMSIIP 2018

Ghent University, ELIS, SYSTeMS

Can we determine E∞( f ) without explicitly evaluating lim

t→+∞ EM0

( lim

n→+∞

( I + t

nQ

)n f ) ?

1

Continuous-time Markov chains

Basic set-up

Objective making inferences about the state Xt of some system Assumptions

• 1. state space is finite
• 2. time parameter is continuous
• 3. dynamics are non-deterministic, Markovian & homogeneous

t1 Xt1 tn Xtn t Xt t + ∆ Xt+∆ P(Xt+∆ = y | Xt1 = x1, . . . , Xtn = xn, Xt = x)

[Markov property] [homogeneity] 2

Basic set-up

Objective making inferences about the state Xt of some system Assumptions

• 1. state space is finite
• 2. time parameter is continuous
• 3. dynamics are non-deterministic, Markovian & homogeneous

t1 Xt1 tn Xtn t Xt t + ∆ Xt+∆ P(Xt+∆ = y | Xt1 = x1, . . . , Xtn = xn, Xt = x) = P(Xt+∆ = y | Xt = x)

[Markov property] [homogeneity] 2

Basic set-up

Objective making inferences about the state Xt of some system Assumptions

• 1. state space is finite
• 2. time parameter is continuous
• 3. dynamics are non-deterministic, Markovian & homogeneous

t1 Xt1 tn Xtn X0 ∆ X∆ P(Xt+∆ = y | Xt1 = x1, . . . , Xtn = xn, Xt = x) = P(Xt+∆ = y | Xt = x)

[Markov property]

= P(X∆ = y | X0 = x)

[homogeneity] 2

Characterisation

A homogeneous CTMC is fully characterised by

• 1. a (finite) state space X ;
• 2. an initial distribution π0;

[P(X0 = x) = π0(x)]

• 3. a transition rate matrix Q.

[nonnegative ofg-diagonal elements and zero row sums] 3

Marginal expectations

How do we compute E( f (Xt))?

• 1. solve the difgerential equation

d dτ Tτ f = QTτ f with initial condition T0 f = f

[Note: Tτ f : X → R]

i.e., evaluate

• 2. compute

E( f (Xt)) = Eπ0(Tt f )

4

Marginal expectations

How do we compute E( f (Xt))?

• 1. solve the difgerential equation

d dτ Tτ f = QTτ f with initial condition T0 f = f

[Note: Tτ f : X → R]

i.e., evaluate Tt f = etQ f := lim

n→+∞

( I + t nQ )n f

• 2. compute

E( f (Xt)) = Eπ0(Tt f )

4

Typical temporal behaviour of Tt f

t Tt f

5

Limit expectations

In many applications, one is interested in the limit expectation E∞( f ) := lim

t→+∞ E( f (Xt)) = lim t→+∞ Eπ0(Tt f ).

Definition (Ergodicity) The transition rate matrix Q is ergodic if, for all f : X → R, E∞( f ) does not depend on π0

• r equivalently,

lim

t→+∞ Tt f is a constant function. 6

Imprecise continuous-time Markov chains

Thomas Krak, Jasper De Bock, and Arno Siebes. “Imprecise continuous-time Markov chains”. In: International Journal of Approximate Reasoning 88 (2017), pp. 452–528 Jasper De Bock. “The Limit Behaviour of Imprecise Continuous-Time Markov Chains”. In: Journal of Nonlinear Science 27.1 (2017), pp. 159–196

7

Characterisation

An imprecise CTMC is characterised by

• 1. a (finite) state space X ;
• 2. an initial distribution π0;
• 3. a transition rate matrix Q.

Problem These sets do not characterise a single CTMC! Solution Consider the set of stochastic processes that is consistent with and : the set of consistent homogeneous CTMCs, the set of consistent CTMCs, the set of consistent stochastic processes.

8

Characterisation

An imprecise CTMC is characterised by

• 1. a (finite) state space X ;
• 2. a set of initial distributions M0;
• 3. a set of transition rate matrices Q.

Problem These sets do not characterise a single CTMC! Solution Consider the set of stochastic processes that is consistent with and : the set of consistent homogeneous CTMCs, the set of consistent CTMCs, the set of consistent stochastic processes.

8

Characterisation

An imprecise CTMC is characterised by

• 1. a (finite) state space X ;
• 2. a set of initial distributions M0;
• 3. a set of transition rate matrices Q.

Problem These sets do not characterise a single CTMC! Solution Consider the sets of stochastic processes that are consistent with M0 and Q: PHM

M0,Q the set of consistent homogeneous CTMCs,

the set of consistent CTMCs, the set of consistent stochastic processes.

8

Characterisation

An imprecise CTMC is characterised by

• 1. a (finite) state space X ;
• 2. a set of initial distributions M0;
• 3. a set of transition rate matrices Q.

Problem These sets do not characterise a single CTMC! Solution Consider the sets of stochastic processes that are consistent with M0 and Q: PHM

M0,Q the set of consistent homogeneous CTMCs,

PM

M0,Q the set of consistent CTMCs,

PM0,Q the set of consistent stochastic processes.

8

Lower envelopes

Krak et al. (2017) define the coherent lower expectations PHM

M0,Q lower envelope

− − − − − − − − → EHM

M0,Q

PM

M0,Q lower envelope

− − − − − − − − → EM

M0,Q

PM0,Q

lower envelope

− − − − − − − − → EM0,Q They also define a lower envelope of . The lower transition rate

• perator

is defined by

[superadditive, nonneg. hom., ~ zero row sums, ~ nonneg. ofg-diagonal elements] 9

Lower envelopes

Krak et al. (2017) define the coherent lower expectations PHM

M0,Q lower envelope

− − − − − − − − → EHM

M0,Q

PM

M0,Q lower envelope

− − − − − − − − → EM

M0,Q

PM0,Q

lower envelope

− − − − − − − − → EM0,Q They also define a lower envelope of Q. The lower transition rate

• perator Q: L (X ) → L (X ) is defined by

[Q f ](x) := inf{[Q f ](x): Q ∈ Q}.

[superadditive, nonneg. hom., ~ zero row sums, ~ nonneg. ofg-diagonal elements] 9

Marginal lower expectations

Observe that PM0,Q ⊇ PM

M0,Q ⊇ PHM M0,Q.

This implies that EM0,Q( f (Xt)) ≤ EM

M0,Q( f (Xt)) ≤ EHM M0,Q( f (Xt)).

Furthermore, Krak et al. (2017) show that [under some conditions on Q] EM0,Q( f (Xt)) = EM

M0,Q( f (Xt)) ≤ EHM M0,Q( f (Xt)). 10

Determining EM0,Q( f (Xt))

How do we compute EM0,Q( f (Xt))?

• 1. solve the difgerential equation

d dτ Tτ f = QTτ f with initial condition T0 f = f

[Note: Tτ f : X → R]

i.e., evaluate Tt f = lim

n→+∞

( I + t nQ )n f

• 2. compute

E( f (Xt)) = Eπ0(Tt f )

11

Determining EM0,Q( f (Xt))

How do we compute EM0,Q( f (Xt))?

• 1. solve the difgerential equation

d dτ Tτ f = QTτ f with initial condition T0 f = f

[Note: Tτ f : X → R]

i.e., evaluate Tt f = lim

n→+∞

( I + t nQ )n f

• 2. compute

E( f (Xt)) = Eπ0(Tt f )

Underline all the operators!

11

Determining EM0,Q( f (Xt))

How do we compute EM0,Q( f (Xt))?

• 1. solve the difgerential equation

d dτ Tτ f = QTτ f with initial condition T0 f = f

[Note: Tτ f : X → R]

i.e., evaluate Tt f = lim

n→+∞

( I + t nQ )n f.

• 2. compute

EM0,Q( f (Xt)) = EM0(Tt f )

11

Typical temporal behaviour of Tt f

t Tt f

12

Limit expectations

We now turn to the limit lower expectations E∞( f ) := lim

t→+∞ EM0(Tt f )

= lim

t→+∞ EM0,Q( f (Xt)) = lim t→+∞ EM M0,Q( f (Xt))

and EHM

∞ ( f ) := lim t→+∞ EHM M0,Q( f (Xt)).

Definition (De Bock, 2017) The lower transition rate operator is ergodic if, for all , does not depend on

• r equivalently,

is a constant function.

13

Limit expectations

We now turn to the limit lower expectation E∞( f ) := lim

t→+∞ EM0(Tt f ).

Definition (De Bock, 2017) The lower transition rate operator Q is ergodic if, for all f : X → R, E∞( f ) does not depend on M0

• r equivalently,

lim

t→+∞ Tt f is a constant function. 13

Can we determine E∞( f ) without explicitly evaluating lim

t→+∞ EM0

( lim

n→+∞

( I + t

nQ

)n f ) ?

13

A well-known result

Theorem If Q is an ergodic transition rate matrix, then for all δ > 0 with δ∥Q∥ < 2, E∞ is the unique (I + δQ)-invariant expectation

• perator:

E∞((I + δQ) f ) = E∞( f ) for all f ∈ L (X ) ⇔ E∞(Q f ) = 0 for all f ∈ L (X ). + simply solve the linear system of equations π∞Q = 0

14

A well-known result

Theorem If Q is an ergodic transition rate matrix, then for all δ > 0 with δ∥Q∥ < 2, E∞ is the unique (I + δQ)-invariant expectation

• perator:

E∞((I + δQ) f ) = E∞( f ) for all f ∈ L (X ) ⇔ E∞(Q f ) = 0 for all f ∈ L (X ). + simply solve the linear system of equations π∞Q = 0

Underline all the operators?

14

Explicitly determining E∞

Conjecture If Q is an ergodic lower transition rate operator, then

• 1. for all δ > 0 with δ∥Q∥ < 2, E∞ is the unique

(I + δQ)-invariant lower expectation operator: E∞((I + δQ) f ) = E∞( f ) for all f ∈ L (X );

• 2. E∞(Q f ) = 0 for all f ∈ L (X ).

¿ some alternative general upper bound ? ¿ effjcient way to solve this “set of equations” ?

15

Explicitly determining E∞

Conjecture If Q is an ergodic lower transition rate operator, then

• 1. for all δ > 0 with δ∥Q∥ < 2, E∞ is the unique

(I + δQ)-invariant lower expectation operator: E∞((I + δQ) f ) = E∞( f ) for all f ∈ L (X );

• 2. E∞(Q f ) = 0 for all f ∈ L (X ).

¿ some alternative general upper bound ? ¿ effjcient way to solve this “set of equations” ?

15

Explicitly determining E∞

Conjecture If Q is an ergodic lower transition rate operator, then

• 1. for all δ > 0 with δ <?, E∞ is the unique (I + δQ)-invariant

lower expectation operator: E∞((I + δQ) f ) = E∞( f ) for all f ∈ L (X );

• 2. E∞(Q f ) = 0 for all f ∈ L (X ).

¿ some alternative general upper bound ? ¿ effjcient way to solve this “set of equations” ?

15

Another well-known result

Theorem If Q is an ergodic transition rate matrix, then for all δ > 0 with δ∥Q∥ < 2, E∞( f ) = lim

m→+∞ min(I + δQ)m f.

+ works for any (suffjciently small) δ + non-decreasing in m

[Emp.: convergence is faster for larger δ]

+ easy to implement

16

Another well-known result

Theorem If Q is an ergodic transition rate matrix, then for all δ > 0 with δ∥Q∥ < 2, E∞( f ) = lim

m→+∞ min(I + δQ)m f.

+ works for any (suffjciently small) δ + non-decreasing in m

[Emp.: convergence is faster for larger δ]

+ easy to implement

Underline all the operators?

16

Iteratively determining E∞( f )

Conjecture If Q is an ergodic lower transition rate operator, then for all δ > 0 with δ∥Q∥ < 2, E∞( f ) = lim

m→+∞ min(I + δQ)m f.

• extra limit for

+ non-decreasing in + relatively easy to implement ¿ does the value of matter ?

17

Iteratively determining E∞( f )

Theorem If Q is an ergodic lower transition rate operator, then for all δ > 0 with δ∥Q∥ < 2, E∞( f ) = lim

δ→0+ lim m→+∞ min(I + δQ)m f.

• extra limit for δ

+ min(I + δQ)m f non-decreasing in m + relatively easy to implement ¿ does the value of δ matter ?

17

Can we determine EHM

∞ ( f )

without explicitly evaluating lim

t→+∞ inf{EP( f (Xt)): P ∈ PHM M0,Q}?

17

Iteratively computing a lower bound on EHM

∞ ( f )

Theorem If Q consists of only ergodic transition rate matrices, then for all n and δ > 0 with δ∥Q∥ < 2, min(I + δQ)n f ≤ EHM

∞ ( f ).

+ min(I + δQ)n f converges monotonously for n → +∞ ¿ behaviour in function of δ ?

18