Imprecise Markov chains by Jasper De Bock & Thomas Krak - - PowerPoint PPT Presentation

Imprecise Markov chains

by Jasper De Bock & Thomas Krak

A tutorial on

SMPS/BELIEF 2018

September 17

now :-)

by Jasper De Bock & Thomas Krak

September 17

Imprecise Markov chains

by Jasper De Bock & Thomas Krak

A tutorial on

SMPS/BELIEF 2018

now :-)

?

September 17

Imprecise Markov chains

by Jasper De Bock & Thomas Krak

A tutorial on

SMPS/BELIEF 2018

now :-)

?

(Walley 1991) (Augustin et al. 2014)

September 17

Imprecise Markov chains

by Jasper De Bock & Thomas Krak

A tutorial on

SMPS/BELIEF 2018

now :-)

?

P ∈ P

E(f) = inf

P ∈P E(f)

(

E(f) = sup

P ∈P

E(f) = −E(−f)

September 17

Imprecise Markov chains

by Jasper De Bock & Thomas Krak

A tutorial on

SMPS/BELIEF 2018

now :-)

September 17

Imprecise Markov chains

by Jasper De Bock & Thomas Krak

A tutorial on

SMPS/BELIEF 2018

now :-)

X0 Xt

t

• time stochastic process

X0 Xt

t

• time stochastic process

continuous discrete

n

• R≥0

N

X0

P(X0 = x0)

• time stochastic process

continuous discrete

n

• N

Xtn Xtn+1 Xtn−1 tn−1 tn

tn+1 Xt1 t1 R≥0 P(Xtn+1 = y |Xt1 = xt1, ..., Xtn−1 = xtn−1, Xtn = x)

P(X0 = x0)

continuous discrete

n

• N

X0 Xtn Xtn+1 Xtn−1 tn−1 tn tn+1

• time Markov chain

Xt1 t1 R≥0 P(Xtn+1 = y |Xt1 = xt1, ..., Xtn−1 = xtn−1, Xtn = x) = P(Xtn+1 = y |Xt1 = Xtn = x)

P(X0 = x0)

continuous discrete

n

• N

X0 Xtn Xtn+1 Xtn−1 Xt1 t1 tn−1 tn tn+1

• time Markov chain

homogeneous

• nly the time difference

matters!

∆ = tn+1 − tn

R≥0 P(Xtn+1 = y |Xt1 = xt1, ..., Xtn−1 = xtn−1, Xtn = x) = P(Xtn+1 = y |Xt1 = Xtn = x) = T∆(x, y)

P(X0 = x0)

continuous discrete

n

• N

X0 Xtn Xtn+1 Xtn−1 Xt1 t1 tn−1 tn tn+1

• time Markov chain

homogeneous

• R≥0
• nly the time difference

matters!

∆ = tn+1 − tn

P(Xtn+1 = y |Xt1 = xt1, ..., Xtn−1 = xtn−1, Xtn = x) = P(Xtn+1 = y |Xt1 = Xtn = x) = T∆(x, y)

P(X0 = x0)

continuous discrete

n

• N

X0 Xtn Xtn+1 Xtn−1 Xt1 t1 tn−1 tn tn+1

• time Markov chain

homogeneous

• = T∆(xtn, xtn+1)

that’s just a probability mass function

R≥0 π0(x0)

initial distribution

= T∆(xtn, xtn+1)

P(X0 = x0)

continuous discrete

n

• N

X0 Xtn Xtn+1 Xtn−1 Xt1 t1 tn−1 tn tn+1

• time Markov chain

homogeneous

• that’s just a probability

mass function

R≥0 π0(x0) = T ∆ T := T1

with

P

y T(x, y) = 1

(∀y) T(x, y) ≥ 0

initial distribution transition matrix

X0 Xtn Xtn+1 Xtn−1 Xt1 t1 tn−1 tn tn+1 = T∆(xtn, xtn+1) = T ∆ T := T1

with

P

y T(x, y) = 1

(∀y) T(x, y) ≥ 0 T =   0.6 0.3 0.1 0.2 0.3 0.5 0.4 0.1 0.5  

transition matrix

P(X0 = x0)

continuous discrete

n

• N

X0 Xtn Xtn+1 Xtn−1 Xt1 t1 tn−1 tn tn+1

• time Markov chain

homogeneous

• that’s just a probability

mass function

R≥0 π0(x0)

initial distribution

P(X0 = x0)

continuous discrete

n

• N

X0 Xtn Xtn+1 Xtn−1 Xt1 t1 tn−1 tn tn+1

• time Markov chain

homogeneous

• that’s just a probability

mass function

R≥0 π0(x0)

with transition rate matrix

T∆ = eQ∆ := lim

n→∞

⇣ I + t nQ ⌘n Q:= lim

∆→0

T∆ − I ∆ P

yQ(x, y) = 0

(8y 6= x) Q(x, y) 0

initial distribution

transition rate matrix

Q:= lim

∆→0

T∆ − I ∆ P

yQ(x, y) = 0

(8y 6= x) Q(x, y) 0

morous ickering

• nfusion

epression 3

1

3 4

2

2 2 1

continuous discrete

n

• N
• time Markov chain

homogeneous

• R≥0

π0(x0)

initial distribution transition rate matrix

• r

Q

transition matrix T

(

X0 Xt

t

X0 Xt

t

f(Xt) = = x = 0 I (Xt) = E(f(Xt)|X0 = x) = [Ttf](x) = X

y

Tt(x, y)f(y) = (P

y eQt(x, y)f(y)

P

y T t(x, y)f(y)

X0 Xt

t

f(Xt) = Iy(Xt) = ( 1 if Xt = y

• therwise

y = x = 0

E(f(Xt)|X0 = x) = [Ttf](x) = X

y

Tt(x, y)f(y) = (P

y eQt(x, y)f(y)

P

y T t(x, y)f(y)

P(Xt = y|X0 = x) = E(Iy(Xt)|X0 = x) = [TtIy](x)

X0 Xt

t

E(f(Xt)|X0 = x) = [Ttf](x) X (P

π∞(y):= lim

t→+∞ P(Xt = y|X0 = x)

P(Xt = y|X0 = x) = E(Iy(Xt)|X0 = x) = [TtIy](x) E∞(f):= lim

t→+∞ E(f(Xt)|X0 = x)

Reliability engineering (failure probabilities, …) Queuing theory (waiting in line …)

• optimising supermarket waiting times
• dimensioning of call centers
• airport security lines
• router queues on the internet

Chemical reactions (time-evolution …) … Pagerank

So how about imprecision?

So how about imprecision? What if we don’t know or M exactly?

Q T

Sets of transition (rate) matrices

Don’t know T (or Q) exactly But confident that T ∈ T for some set T of transition matrices (or that Q ∈ Q for some set Q of rate matrices) Induces imprecise Markov chain; set of processes compatible with T .

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Sets of transition (rate) matrices

Don’t know T (or Q) exactly But confident that T 2 T for some set T of transition matrices (or that Q 2 Q for some set Q of rate matrices) Induces imprecise Markov chain; set of processes compatible with T . Different versions: PHM

T : all homogeneous Markov chains with T 2 T

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Sets of transition (rate) matrices

Don’t know T (or Q) exactly But confident that T ∈ T for some set T of transition matrices (or that Q ∈ Q for some set Q of rate matrices) Induces imprecise Markov chain; set of processes compatible with T . Different versions: PHM

T : all homogeneous Markov chains with T ∈ T

PM

T : all (non-homogeneous) Markov chains with T (t) ∈ T

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Sets of transition (rate) matrices

Don’t know T (or Q) exactly But confident that T ∈ T for some set T of transition matrices (or that Q ∈ Q for some set Q of rate matrices) Induces imprecise Markov chain; set of processes compatible with T . Different versions: PHM

T : all homogeneous Markov chains with T ∈ T

PM

T : all (non-homogeneous) Markov chains with T (t) ∈ T

PT : all (non-Markov) processes with T (t,xu) ∈ T

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Sets of transition (rate) matrices

Don’t know T (or Q) exactly But confident that T ∈ T for some set T of transition matrices (or that Q ∈ Q for some set Q of rate matrices) Induces imprecise Markov chain; set of processes compatible with T . Different versions: PHM

T : all homogeneous Markov chains with T ∈ T

PM

T : all (non-homogeneous) Markov chains with T (t) ∈ T

PT : all (non-Markov) processes with T (t,xu) ∈ T Clearly PHM

T

⊆ PM

T ⊆ PT

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Lower expectations and lower probabilities

Given an imprecise Markov chain P∗

T , we are interested in

E∗

T

⇥ f (Xt)|X0 = x ⇤ = inf

P∈P∗

T

EP ⇥ f (Xt)|X0 = x ⇤ (And E

∗ T

⇥ f (Xt)|X0 = x ⇤ by conjugacy) Lower- (and upper) probabilities a special case: P∗

T

• Xt = y |X0 = x
• = inf

P∈P∗

T

P

• Xt = y |X0 = x
• = E∗

T

⇥ Iy(Xt)|X0 = x ⇤ Because different types are nested, ET ⇥ f (Xt)|X0 = x ⇤ ≤ EM

T

⇥ f (Xt)|X0 = x ⇤ ≤ EHM

T

⇥ f (Xt)|X0 = x ⇤

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Computing lower expectations, first try

Recall that for a homogeneous Markov chain P with transition matrix T, EP ⇥ f (X1)|X0 = x ⇤ = ⇥ Tf ⇤ (x). Now consider PHM

T . Then,

EHM

T

⇥ f (X1)|X0 = x ⇤ := inf

P∈PHM

T

EP ⇥ f (X1)|X0 = x ⇤ = inf

T∈T

⇥ Tf ⇤ (x) Linear optimisation problem with constraints given by T Relatively straightforward if T is “nice” Essentially solving a linear programming problem

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Computing lower expectations, first try

Recall that for a homogeneous Markov chain P with transition matrix T, EP ⇥ f (Xt)|X0 = x ⇤ = ⇥ T tf ⇤ (x). Now consider PHM

T . Then,

EHM

T

⇥ f (Xtn)|X0 = x ⇤ := inf

P∈PHM

T

EP ⇥ f (Xt)|X0 = x ⇤ = inf

T∈T

⇥ T tf ⇤ (x) Non-linear optimisation problem with constraints given by T Not straightforward even if T is “nice”

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Computing lower expectations, first try

Recall that for a homogeneous Markov chain P with transition matrix T, EP ⇥ f (Xt)|X0 = x ⇤ = ⇥ T tf ⇤ (x). Now consider PHM

T . Then,

EHM

T

⇥ f (Xtn)|X0 = x ⇤ := inf

P∈PHM

T

EP ⇥ f (Xt)|X0 = x ⇤ = inf

T∈T

⇥ T tf ⇤ (x) Non-linear optimisation problem with constraints given by T Not straightforward even if T is “nice” See e.g. (Kozine and Utkin, 2002) and (Campos et al., 2003) for analyses of this approach.

Jasper De Bock, Thomas Krak Imprecise Markov Chains

What about the non-Markov case?

PT : all (non-Markov) processes with T (t,xu) ∈ T How to interpret this? ⇒ Helps to draw a picture

Jasper De Bock, Thomas Krak Imprecise Markov Chains

What about the non-Markov case?

PT : all (non-Markov) processes with T (t,xu) ∈ T How to interpret this? ⇒ Helps to draw a picture Example with binary state space X = {a,b} Use event tree / probability tree Illustration of behaviour over time Need notation Tx :=

• T(x,·)
• T ∈ T

∀x ∈ X

Jasper De Bock, Thomas Krak Imprecise Markov Chains

What about the non-Markov case?

PT : all (non-Markov) processes with T (t,xu) ∈ T How to interpret this? ⇒ Helps to draw a picture Example with binary state space X = {a,b} Use event tree / probability tree Illustration of behaviour over time Need notation Tx :=

• T(x,·)
• T ∈ T

∀x ∈ X This setting explored by (De Cooman et al., 2009). Tree representation related to (Shafer and Vovk, 2001) game-theoretic probabil-

• ities. Connection to (Walley’s) imprecise probabilities in (De Cooman and Her-

mans, 2008).

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Visualising a stochastic process

time 1 2 / π0 a b P(X1 |X0 = a) aa ab ba bb P(X1 |X0 = b) P(X2 |X0 = a,X1 = b) P(X2 |X0 = b,X1 = b) P(X2 |X0 = a,X1 = a) P(X2 |X0 = b,X1 = a)

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Visualising a stochastic process in PT

time 1 2 / π0 a b P(X1 |X0 = a) ∈ Ta aa ab ba bb P(X1 |X0 = b) ∈ Tb P(X2 |X0 = a,X1 = b) ∈ Tb P(X2 |X0 = b,X1 = b) ∈ Tb P(X2 |X0 = a,X1 = a) ∈ Ta P(X2 |X0 = b,X1 = a) ∈ Ta

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Computations by iterated lower expectation

For the set PT , it can be shown that ET ⇥ f (Xt)

• X0 = x

⇤ = ET h ET ⇥ f (Xt)|X0 = x,Xs ⇤

• X0 = x

i ∀s ≤ t This is the law of iterated lower expectation. Provides backwards recursive scheme for computations. ⇒ Intuitive in the tree representation Example: compute ET ⇥ Ib(X2)

• X0 = a

⇤ = P

• X2 = b|X0 = a
• Ta :=

n T(a,·)

• T(a,a) ∈ [0.4,0.6]
• Tb :=

n T(b,·)

• T(b,a) ∈ [0.1,0.3]
• Jasper De Bock, Thomas Krak

Imprecise Markov Chains

Example ET ⇥ Ib(X2)

• X0 = a

⇤ base case

aa Ta aaa aab time 1 2

Example ET ⇥ Ib(X2)

• X0 = a

⇤ base case

aa Ta aaa aab time 1 2 Ib(a) = 0 Ib(b) = 1

Example ET ⇥ Ib(X2)

• X0 = a

⇤ base case

aa Ta aaa aab time 1 2 Ib(a) = 0 Ib(b) = 1 ET ⇥ Ib(X2)|X0 = a,X1 = a ⇤ ?

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Example ET ⇥ Ib(X2)

• X0 = a

⇤ base case

aa aaa aab time 1 2 Ib(a) = 0 Ib(b) = 1 ET ⇥ Ib(X2)|X0 = a,X1 = a ⇤ = inf

T(a,·)∈Ta

T(a,a)Ib(a)+T(a,b)Ib(b)

Example ET ⇥ Ib(X2)

• X0 = a

⇤ base case

aa aaa aab time 1 2 Ib(a) = 0 Ib(b) = 1 ET ⇥ Ib(X2)|X0 = a,X1 = a ⇤ = inf

T(a,·)∈Ta

T(a,a)Ib(a)+T(a,b)Ib(b) Ta =

• T(a,·)|T(a,a) ∈ [0.4,0.6]

= 0.4

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Recursive, local computations

time 1 2 / π0 a b Ta aa ab ba bb Tb ET ⇥ Ib(X2)|X0 = a,X1 = b ⇤ = 0.7 ET ⇥ Ib(X2)|X0 = a,X1 = b ⇤ = 0.7 ET ⇥ Ib(X2)|X0 = a,X1 = a ⇤ = 0.4 ET ⇥ Ib(X2)|X0 = a,X1 = a ⇤ = 0.4

Recursive, local computations

time 1 2 / π0 a b Ta aa ab ba bb Tb ET ⇥ Ib(X2)|X0 = a,X1 = b ⇤ = 0.7 ET ⇥ Ib(X2)|X0 = a,X1 = b ⇤ = 0.7 ET ⇥ Ib(X2)|X0 = a,X1 = a ⇤ = 0.4 ET ⇥ Ib(X2)|X0 = a,X1 = a ⇤ = 0.4 inf

T(a,·)∈Ta

T(a,a)×0.4+T(a,b)×0.7

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Recursive, local computations

time 1 2 / π0 a b ba bb Tb ET ⇥ Ib(X2)|X0 = a,X1 = b ⇤ = 0.7 ET ⇥ Ib(X2)|X0 = a,X1 = a ⇤ = 0.4 ET ⇥ Ib(X2)|X0 = a ⇤ = inf

T(a,·)∈Ta

T(a,a)×0.4+T(a,b)×0.7 = 0.52

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Local computations in operator form

Consider Tx, and define for any f : X → R, ⇥ Tf ⇤ (x) := inf

T(x,·)∈Tx ∑ y

T(x,y)f (y) Linear optimisation problem, and ⇥ Tf ⇤ (x) = inf

T∈T

⇥ Tf ⇤ (x) We call T the lower transition operator for T . We can write ET ⇥ f (Xt+1)|X0:t = x0:t ⇤ = ⇥ Tf ⇤ (xt)

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Local computations in operator form

Consider Tx, and define for any f : X → R, ⇥ Tf ⇤ (x) := inf

T(x,·)∈Tx ∑ y

T(x,y)f (y) Linear optimisation problem, and ⇥ Tf ⇤ (x) = inf

T∈T

⇥ Tf ⇤ (x) We call T the lower transition operator for T . We can write ET ⇥ f (Xt+1)|X0:t = x0:t ⇤ = ⇥ Tf ⇤ (xt) We find ET ⇥ f (Xt+1)|X0:t = x0:t ⇤ = ⇥ Tf ⇤ (xt) = ET ⇥ f (Xt+1)|Xt = xt ⇤ Lower envelope for imprecise Markov chain PT has “Markov” property But contains non-Markov models! Similarly the lower envelope is also homogeneous!

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Multiple time steps

By repeating the local computations, ET ⇥ f (X2)|X0 = x ⇤ = ⇥ T Tf ⇤ (x), if the set T has separately specified rows: 2 4 3 5, 2 4 3 5 ∈ T ⇒ 2 4 3 5 ∈ T (T is closed under recombination of rows)

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Multiple time steps

By repeating the local computations, ET ⇥ f (X2)|X0 = x ⇤ = ⇥ T Tf ⇤ (x), if the set T has separately specified rows: 2 4 3 5, 2 4 3 5 ∈ T ⇒ 2 4 3 5 ∈ T (T is closed under recombination of rows) By induction we get ET ⇥ f (Xt)|X0 = x ⇤ = ⇥ T tf ⇤ (x) Local, linear optimisations only Can be efficiently computed

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Multiple time steps

By repeating the local computations, ET ⇥ f (X2)|X0 = x ⇤ = ⇥ T Tf ⇤ (x), if the set T has separately specified rows: 2 4 3 5, 2 4 3 5 ∈ T ⇒ 2 4 3 5 ∈ T (T is closed under recombination of rows) By induction we get ET ⇥ f (Xt)|X0 = x ⇤ = ⇥ T tf ⇤ (x) Local, linear optimisations only Can be efficiently computed Imprecise Markov chain PT can be seen as credal network under epistemic irrele-

• vance. Gives a graphical model representation.

“Separately specified rows” is a well-known condition in that context.

Jasper De Bock, Thomas Krak Imprecise Markov Chains

That’s two extremes. What about the intermediate one?

So far ignored PM

T

Jasper De Bock, Thomas Krak Imprecise Markov Chains

That’s two extremes. What about the intermediate one?

So far ignored PM

T

Turns out that if T has separately specified rows, then EM

T

⇥ f (Xt)|X0 = x ⇤ = ⇥ T tf ⇤ (x) If follows that EM

T

⇥ f (Xt)|X0 = x ⇤ = ET ⇥ f (Xt)|X0 = x ⇤ Does not hold for functions on multiple time points Then only PT remains tractable

Jasper De Bock, Thomas Krak Imprecise Markov Chains

That’s two extremes. What about the intermediate one?

So far ignored PM

T

Turns out that if T has separately specified rows, then EM

T

⇥ f (Xt)|X0 = x ⇤ = ⇥ T tf ⇤ (x) If follows that EM

T

⇥ f (Xt)|X0 = x ⇤ = ET ⇥ f (Xt)|X0 = x ⇤ Does not hold for functions on multiple time points Then only PT remains tractable First pioneered by Hartfiel, Markov Set-Chains (Hartfiel, 1998) ⇒ No explicit connection to imprecise probabilities Exploration with imprecise probabilities by (ˇ Skulj, 2009)

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Limit behaviour?

Limit inference often of interest: E ⇥ f (X∞)|X0 = x ⇤ = lim

t→+∞E

⇥ f (Xt)|X0 = x ⇤ In imprecise setting, often exists: ET ⇥ f (X∞)|X0 = x ⇤ := lim

t→+∞

⇥ T tf ⇤ (x), and often independent of x. See e.g. (De Cooman et al., 2009) and (ˇ Skulj, 2009)

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Summary for imprecise Markov chains in discrete time

Parameterisation through set T of transition matrices. Can induce three different imprecise Markov chains: PHM

T : all homogeneous Markov chains compatible with T

PM

T : all (non-homogeneous) Markov chains compatible with T

PT : all (non-Markov) processes compatible with T For PHM

T , computations are difficult.

For PM

T and PT , computations using lower transition operator

EM

T

⇥ f (Xt)|X0 = x ⇤ = ET ⇥ f (Xt)|X0 = x ⇤ = ⇥ T tf ⇤ (x) The imprecise Markov chain PT satisfies an imprecise Markov property The limit limt→+∞ ⇥ T tf ⇤ (x) often exists, and often independent of x.

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Imprecise Continuous-Time Markov Chains

Going to go a bit faster with more intuition We use the same basic approach: Uncertain about Q, but consider a set Q Three imprecise (continuous-time) Markov chains, compatible with Q:

PHM

Q : all homogeneous Markov chains with Q 2 Q

PM

Q: all (non-homogeneous) Markov chains with Qt 2 Q

PQ: all (non-Markov) processes with Qt,xu 2 Q

Similar to discrete-time case, EHM

Q

⇥ f (Xt)|X0 = x ⇤ = inf

Q2Q

⇥ eQtf ⇤ (x) which is difficult due to nonlinearities in the optimisation.

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Imprecise Continuous-Time Markov Chains

Going to go a bit faster with more intuition We use the same basic approach: Uncertain about Q, but consider a set Q Three imprecise (continuous-time) Markov chains, compatible with Q:

PHM

Q : all homogeneous Markov chains with Q 2 Q

PM

Q: all (non-homogeneous) Markov chains with Qt 2 Q

PQ: all (non-Markov) processes with Qt,xu 2 Q

Similar to discrete-time case, EHM

Q

⇥ f (Xt)|X0 = x ⇤ = inf

Q2Q

⇥ eQtf ⇤ (x) which is difficult due to nonlinearities in the optimisation. See e.g. (Goldsztejn and Neumaier, 2014) and (Oppenheimer and Michel, 1988) for details on this homogeneous setting

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Non-homogeneous case in continuous-time

PM

Q: all (non-homogeneous) Markov chains with Qt 2 Q

How to interpret this? Homogeneous case, rate matrix is just a derivative, Q := lim

∆!0

T∆ I ∆ where T∆(x,y) := P(X∆ = y |X0 = x)

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Non-homogeneous case in continuous-time

PM

Q: all (non-homogeneous) Markov chains with Qt 2 Q

How to interpret this? Homogeneous case, rate matrix is just a derivative, Q := lim

∆!0

T∆ I ∆ where T∆(x,y) := P(X∆ = y |X0 = x) For non-homogeneous case we write T t+∆

t

(x,y) := P(Xt+∆ = y |Xt = x), which has a time-dependent derivative, Qt := lim

∆!0

T t+∆

t

T t

t

∆ = lim

∆!0

T t+∆

t

I ∆

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Non-homogeneous case in continuous-time

PM

Q: all (non-homogeneous) Markov chains with Qt 2 Q

How to interpret this? Homogeneous case, rate matrix is just a derivative, Q := lim

∆!0

T∆ I ∆ where T∆(x,y) := P(X∆ = y |X0 = x) For non-homogeneous case we write T t+∆

t

(x,y) := P(Xt+∆ = y |Xt = x), which has a time-dependent derivative, Qt := lim

∆!0

T t+∆

t

T t

t

∆ = lim

∆!0

T t+∆

t

I ∆ Setting explored by (Hartfiel, 1985) and (ˇ Skulj, 2015)

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Continuous-time local models

We have Qt = lim

∆!0

T t+∆

t

I ∆ and so for small ∆, T t+∆

t

⇡ I +∆Qt

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Continuous-time local models

We have Qt = lim

∆!0

T t+∆

t

I ∆ and so for small ∆, T t+∆

t

⇡ I +∆Qt Then we can write EM

Q

⇥ f (Xt+∆)|Xt = x ⇤ = inf

T t+∆

t

⇥ T t+∆

t

f ⇤ (x) ⇡ inf

Q2Q

⇥ (I +∆Q)f ⇤ (x)

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Continuous-time local models

We have Qt = lim

∆!0

T t+∆

t

I ∆ and so for small ∆, T t+∆

t

⇡ I +∆Qt Then we can write EM

Q

⇥ f (Xt+∆)|Xt = x ⇤ = inf

T t+∆

t

⇥ T t+∆

t

f ⇤ (x) ⇡ inf

Q2Q

⇥ (I +∆Q)f ⇤ (x) We get EM

Q

⇥ f (Xt+∆)|Xt = x ⇤ ⇡ ⇥ (I +∆Q)f ⇤ (x) ⇡ EM

Q

⇥ f (X∆)|X0 = x ⇤ where we have defined ⇥ Qf ⇤ (x) := inf

Q2Q

⇥ Qf ⇤ (x), Again homogeneous lower expectation!

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Arbitrary time points

If Q has separately specified rows, EM

Q

⇥ f (Xt)|X0 = x ⇤ ⇡ ⇥ (I + t/nQ)nf ⇤ (x) and in fact EM

Q

⇥ f (Xt)|X0 = x ⇤ = lim

n!+∞

⇥ (I + t/nQ)nf ⇤ (x) Allows practical computation Solve infQ2Q[Q·] multiple times Each is a linear optimisation problem

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Arbitrary time points

If Q has separately specified rows, EM

Q

⇥ f (Xt)|X0 = x ⇤ ⇡ ⇥ (I + t/nQ)nf ⇤ (x) and in fact EM

Q

⇥ f (Xt)|X0 = x ⇤ = lim

n!+∞

⇥ (I + t/nQ)nf ⇤ (x) Allows practical computation Solve infQ2Q[Q·] multiple times Each is a linear optimisation problem Better computational method in (Erreygers and De Bock, 2017)

Jasper De Bock, Thomas Krak Imprecise Markov Chains

The non-Markov case

For the set PQ, derivative becomes history dependent. Let xu = xu1,...,xun, 0  u1 < ··· < un < t. For all x,y 2 X , Qt,xu(x,y) := lim

∆!0

P(Xt+∆ = y |Xu = xu,Xt = x)I(x,y) ∆ This is becoming a bit unwieldy. . .

Jasper De Bock, Thomas Krak Imprecise Markov Chains

The non-Markov case

For the set PQ, derivative becomes history dependent. Let xu = xu1,...,xun, 0  u1 < ··· < un < t. For all x,y 2 X , Qt,xu(x,y) := lim

∆!0

P(Xt+∆ = y |Xu = xu,Xt = x)I(x,y) ∆ This is becoming a bit unwieldy. . . Turns out that EQ ⇥ f (Xs+t)|Xu = xu,Xs = x ⇤ = lim

n!+∞

⇥ (I + t/nQ)nf ⇤ (x)

Jasper De Bock, Thomas Krak Imprecise Markov Chains

The non-Markov case

For the set PQ, derivative becomes history dependent. Let xu = xu1,...,xun, 0  u1 < ··· < un < t. For all x,y 2 X , Qt,xu(x,y) := lim

∆!0

P(Xt+∆ = y |Xu = xu,Xt = x)I(x,y) ∆ This is becoming a bit unwieldy. . . Turns out that EQ ⇥ f (Xs+t)|Xu = xu,Xs = x ⇤ = lim

n!+∞

⇥ (I + t/nQ)nf ⇤ (x) = EM

Q

⇥ f (Xt)|X0 = x ⇤ Lower expectation for PQ has an imprecise Markov property! And is time-homogeneous! Not the same as PM

Q when f depends on multiple time points!

Then only PQ remains tractable.

Jasper De Bock, Thomas Krak Imprecise Markov Chains

The non-Markov case

For the set PQ, derivative becomes history dependent. Let xu = xu1,...,xun, 0  u1 < ··· < un < t. For all x,y 2 X , Qt,xu(x,y) := lim

∆!0

P(Xt+∆ = y |Xu = xu,Xt = x)I(x,y) ∆ This is becoming a bit unwieldy. . . Turns out that EQ ⇥ f (Xs+t)|Xu = xu,Xs = x ⇤ = lim

n!+∞

⇥ (I + t/nQ)nf ⇤ (x) = EM

Q

⇥ f (Xt)|X0 = x ⇤ Lower expectation for PQ has an imprecise Markov property! And is time-homogeneous! Not the same as PM

Q when f depends on multiple time points!

Then only PQ remains tractable.

Explored by (Krak et al., 2017)

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Continuous-time limit behaviour?

Limit inference often of interest: E ⇥ f (X∞)|X0 = x ⇤ = lim

t!+∞E

⇥ f (Xt)|X0 = x ⇤ In imprecise setting, limit always exists: EQ ⇥ f (X∞)|X0 = x ⇤ = lim

t!+∞EQ

⇥ f (Xt)|X0 = x ⇤ and often independent of x. See (De Bock, 2017)

Jasper De Bock, Thomas Krak Imprecise Markov Chains

Main take away points

If we do not know T or Q, we can consider sets T or Q Gives rise to three different imprecise models: Set of homogeneous Markov chains Set of non-homogeneous Markov chains Set of non-Markov processes For homogeneous Markov chains: Difficult to work with For non-homogeneous and non-Markov processes: Efficient computations using local models T or Q Have homogeneous lower expectations Have “Markov” lower expectations

Jasper De Bock, Thomas Krak Imprecise Markov Chains

That’s all fine and well, but what can you use it for?

Reliability engineering (failure probabilities, …) Queuing theory (waiting in line …)

• optimising supermarket waiting times
• dimensioning of call centers
• airport security lines
• router queues on the internet

Chemical reactions (time-evolution …) … Pagerank

Message passing in optical links

superchannels

m2 = m1 n2

type I messages require 1 channel type II messages require channels

n2 m1

channels We want to know the blocking probability of messages for a given policy, and optimise it

NOT 
 GOOD GOOD Amorous Bickering Confusion Depression

(Erreygers & De Bock 2018)

(Erreygers et al. 2018)

Partially specified and are allowed Time homogeneity can be relaxed The Markov assumption can be relaxed Advantages of imprecise Markov chains

• ver their precise counterpart

π0

Efficient computations remain possible State space explosion can be dealt with Q/T

All of this sounds too good to be true! What have you been hiding?

π0(x0)

initial distribution transition rate matrix

• r

Q

transition matrix T

(

X0 Xt

t

Can we learn these from data?

IDM (Walley 1996) (Quaeghebeur 2009) (Krak et al. 2018)

X0 Xt

t

What if the states can’t be observed directly?

(De Bock & De Cooman 2014) (Mauá et al. 2016) (Krak et al. 2017)

Ot O0

Can we still learn these?

not yet…

X0 Xt

t

How about more complicated inferences?

(Troffaes et al. 2015) (Lopatatzidis 2017) in some cases…

X0 Xt

t

Can we do infinite state spaces?

(Peng 2005)

• nly in

theory…

References (1)

• T. Augustin, F. P. A. Coolen, G. De Cooman, M.C.M. Troffaes.

Introduction to Imprecise Probabilities. Wiley, 2014.

• J. De Bock:

The limit behaviour of imprecise continuous-time Markov chains. J. Nonlinear Science. 27(1), 159–196 (2017)

• J. De Bock, G. de Cooman:

An efficient algorithm for estimating state sequences in imprecise hidden Markov models. Journal of Artificial Intelligence Research, 50: 189–233. 2014.

• M. Campos, G.P. Dimuro, A. da Rocha Costa, and V. Kreinovich: Computing 2-step

predictions for interval-valued finite stationary Markov chains. Technical report utep-cs- 03-20a, University of Texas at El Paso, 2003.

• G. de Cooman and F. Hermans: Imprecise probability trees: Bridging two theories of

imprecise probability. Artificial Intelligence, 172:1400–1427, 2008.

• G. de Cooman, F. Hermans, and E.Quaeghebeur: Imprecise Markov chains and their

limit behavior. Probability in the Engineering and Informational Sciences, 23:597–635, 2009

• A. Erreygers and J. De Bock:

Imprecise continuous-time Markov chains: Efficient computational methods with guaranteed error bounds. In: Proceedings of ISIPTA 2017,

• pp. 145–156, 2017

Jasper De Bock, Thomas Krak Imprecise Markov Chains

References (2)

• A. Erreygers and J. De Bock:

Inferences for Large- Scale Continuous-Time Markov Chains by Combining Lumping with Imprecision. Accepted for publication in Proceedings

• f SMPS, 2018.
• A. Erreygers, C. Rottondi, G. Verticale, J. De Bock:

Imprecise Markov Models for Scalable Robust Performance Evaluation of Flexi-Grid Spectrum Allocation Policies. Accepted for publication in IEEE Transactions on Communications. 2018.

• A. Goldsztejn and A. Neumaier: On the exponentiation of interval matrices, Reliab.
• Comput. 20:52–72, 2014
• D. Hartfiel: Markov Set-Chains. Springer, Berlin, 1998.
• D. Hartfiel: On the solutions to x0(t) = a(t)x(t) over all a(t), where p  a(t)  q.

Journal of Mathematical Analysis and Applications, 108:230–240, 1985.

• I. Kozine and L. Utkin:

Interval-valued finite Markov chains. Reliable Computing, 8:97–113, 2002.

• T. Krak, J. De Bock, A. Siebes: Imprecise continuous-time Markov chains.

Int. J.

• Approx. Reason. 88, 452–528, 2017
• T. Krak, J. De Bock, A. Siebes: Efficient computation of updated lower expectations

for imprecise continuous-time hidden Markov chains. PMLR: proceedings of machine learning research, 62 (proceedings of ISIPTA ’17), 193–204. 2017.

Jasper De Bock, Thomas Krak Imprecise Markov Chains

References (3)

• T. Krak, A. Erreygers, J. De Bock:

An Imprecise Probabilistic Estimator for the Transition Rate Matrix of a Continuous-Time Markov Chain. Accepted for publication in the Proceedings of SMPS 2018.

• S. Lopatatzidis:

Robust modelling and optimisation in stochastic processes using imprecise probabilities, with an application to queueing theory. PhD Thesis, Ghent

• University. 2017.
• D. D. Mau´

a, A. Antonucci, C. P. de Campos: Hidden Markov models with set-valued

• parameters. Neurocomputing, 180: 94–107. 2016.

E.P. Oppenheimer, A.N. Michel: Application of interval analysis techniques to linear systems.

• II. The interval matrix exponential function.

IEEE Trans. Circuits Syst. 35(10):1230–1242,1988

• S. Peng. Nonlinear expectations and nonlinear Markov chains. Chinese Ann. Math. Ser.

B, 26(2):159–184. 2005.

• E. Quaeghebeur: Learning from samples using coherent lower previsions. PhD Thesis,

Ghent University. 2009.

• G. Shafer and V. Vovk: Probability and Finance: It’s Only a Game! Wiley, New York,

2001

• D. ˇ

Skulj: Discrete time Markov chains with interval probabilities. International Journal Approximate Reasoning, 50:1314–1329, 2009.

Jasper De Bock, Thomas Krak Imprecise Markov Chains

References (4)

• D. ˇ

Skulj: Efficient computations of the bounds of continuous time imprecise Markov

• chains. Applied Mathematics and Computation, 250:165–180, 2015.

M.C.M. Troffaes, J. Gledhill, D. Skulj, S. Blake: Using imprecise continuous time Markov chains for assessing the reliability of power networks with common cause failure and non-immediate repair. Proceedings of ISIPTA ’15: 287–294, 2015.

• P. Walley: Inferences from multinomial data: learning about a bag of marbles. Journal
• f the Royal Statistical Society, Series B, 58:3–57. 1996.
• P. Walley: Statistical Reasoning with Imprecise Probabilities Chapman and Hall, 1991.

Jasper De Bock, Thomas Krak Imprecise Markov Chains

This work was partially supported by H2020-MSCA-ITN-2016 UTOPIAE, grant agreement 722734. http://twitter.com/utopiae network http://utopiae.eu

Jasper De Bock, Thomas Krak Imprecise Markov Chains