Continuous Time Markov Chain Birth and Death Process IE 502: - - PowerPoint PPT Presentation

Continuous Time Markov Chain Birth and Death Process

IE 502: Probabilistic Models Jayendran Venkateswaran IE & OR

IE502: Probabilistic Models IEOR @ IITBombay

Continuous Time Markov Chain

• {X(t), t ≥ 0} is a Continuous Time Markov Chain if,

for all s, t ≥ 0 and for all i, j in the state space, P{X(t+s) = j | X(s) = i, X(u) = x(u), 0 ≤ u < s} = P{X(t+s) = j | X(s) = i }

–X(t+s) is future state; X(s) is the current state; X(u) are past states –X(t) takes on discrete valued states, but change in state happens on continuous time! Markovian Property

IE502: Probabilistic Models IEOR @ IITBombay

Waiting time in a state

• Suppose a CTMC enters state i at some time.

How long will the process be in state i before it transitions into a different state?

That is, – After entering a state i, suppose the process has not left state i in the first 10 minutes. What is the probability the process will not leave i in the next 5 minutes? – Ti : Time in state i before process transitions

• In CTMC, the amount of time the process waits at

a state has an exponential distribution.

IE502: Probabilistic Models IEOR @ IITBombay

‘Hand on’ definition

• In a CTMC,
• 1. Ti, the amount of time process spends in each state i

before transitioning into a different state is exponentially distributed with rate νi.

• Ti ~ Expo(vi)
• 2. Has a discrete-time Markov chain with transition matrix

P (with no self-loops)

• Why no self loops?

IE502: Probabilistic Models IEOR @ IITBombay

Transition Probability Function

• In DTMC, given the one-step transition probabilities

pij we can find pij

(n) the probability of being in various

states after n timesteps.

• In CTMC, transition probability function denotes the

probability that the process presently in state i will be in state j a time t later. pi,j(t) = P{ X(t+s) = j | X(s) = i }

• Chapman-Kolmogorov Equations for CTMC

∑

∞ =

= + ) ( ) ( ) (

k kj ik ij

s p t p s t p

IE502: Probabilistic Models IEOR @ IITBombay

Birth and Death Process

• Let us identify by state X(t)=n the condition of the

system in which there are n objects. Given the system is in state n, new elements arrive at rate λn, and leave at rate μn.

– λn  Arrival rate or birth rate when process is in state n – μn  Departure rate or death rate when in state n

• In this CTMC, transitions can go from state n to

either state n+1 or state n-1

• State space representation

IE502: Probabilistic Models IEOR @ IITBombay

Birth and Death Process (contd..)

• Relation between birth and death rates (λn & μn)

state transition rates (vi) and transition probabilities (pi,j)

• Scenarios

– λn = 0 → pure death process → only decrement – μn = 0 → pure birth or Yule process → only increment – μn = 0 , λn = λ → Poisson process!

• Now, let’s looks at the transition probability function

IE502: Probabilistic Models IEOR @ IITBombay

Birth and Death Process

• 1. Probability that process will be in state j in the next

∆t (or h) time units

• Three ways this can happen:
• P{X(t + h) = j | X(t) = j - 1} =
• P{X(t + h) = j | X(t) = j + 1} =
• P{X(t + h) = j | X(t) = j} =
• 2. Now, probability that the process presently in state

i will be in state j a time t later, pi,j(t), is …

• pi,j(t) = P{X(t + s) = j | X(s) = i } = P{X(t) = j | X(0) = i }

IE502: Probabilistic Models IEOR @ IITBombay

Birth and Death Process

• 3. Next, combining equation set 1 and 2, and taking

the limit ∆t → 0, a set of linear differential equations can be derived for the transition probabilities

• Note: Above equation is a flow balance equation
• Variation of flow = inflow – outflow

) ( ) ( ) (

, 1 , 1 ,

t p t p t p

i i i

λ µ − = ′

) ( ) ( ) ( ) ( ) (

, 1 , 1 1 , 1 ,

t p t p t p t p

j i j j j i j j i j j i

µ λ µ λ + − + = ′

+ + − −

for , > j

IE502: Probabilistic Models IEOR @ IITBombay

Limiting Probabilities

• Probability that a CTMC will be in state j at time t

converges to a limiting value (Pj) which is independent of the initial state.

(Recall) Sufficient condition for limiting probabilities to exist: MC is irreducible, positive recurrent.

• Now, if steady state solution exists, it is

characterized by

) ( lim ) ( lim = = ′

∞ → ∞ →

t d t p d t p

ij t ij t

) ( lim t p P

ij t j ∞ →

=

IE502: Probabilistic Models IEOR @ IITBombay

Compute Limiting Probabilities

• 1. In steady state, rate of leaving = rate of entering

(to see this, apply limits to the differential eqns)

• 2. General Equilibrium Results:

– Compute Pn in terms of Pn-1 – Compute Pn in terms of P0

• 3. The Normalizing condition must hold
• 4. Solve for P0 and Pn

– What is the condition for the limiting probabilities to exist? 1 =

∑

∞ = n n

P