Discrete Time Markov Chains EECS 126 Fall 2019 October 15, 2019 1 - - PowerPoint PPT Presentation

Discrete Time Markov Chains

EECS 126 Fall 2019 October 15, 2019

1 / 24

Agenda

Announcements Introduction Recap of Discrete Time Markov Chains n-step Transition Probabilities Classification of States Recurrent and Transient States Decomposition of States General Decomposition of States Periodicity Stationary Distributions Definitions Balance Equations

2 / 24

Announcements

◮ Homework 7 due Tomorrow night (10/16)! ◮ Lab self-grades due on Monday night (10/21).

3 / 24

Recap of Discrete Time Markov Chains

Figure: Example of a Markov chain

◮ State changes at discrete times ◮ State Xn belongs to a finite set S (for now) ◮ Satisfies the Markov property for transitions from state i ∈ S to state j ∈ S P(Xn+1 = j | Xn = i, Xn−1 = xn−1 . . . X1 = x1) = P(Xn+1 = j | Xn = i) = pij Where, pij ≥ 0,

j pij = 1

◮ Time homogeneous: the evolution of the system or transition probabilities are time independent

4 / 24

Recap of Discrete Time Markov Chains

The probability transition matrix P contains all the information about transitions between different states P =      p11 p12 . . . p1n p21 p22 . . . p2n . . . . . . . . . . . . pn1 pn2 . . . pnn      Let π(n) =

• P(Xn = 1)

. . . P(Xn = m)

• then

π(n+1) = π(n)P ⇒ π(n) = π(0)Pn

5 / 24

Example

q1 q2 q3 a 1-a b 1-b c 1-c π0 =

• 0.3

0.3 0.4

• ◮ Write the probability transition matrix P

◮ What is P(X0 = q1, X1 = q2, X3 = q1)? ◮ What is P(X0 = q1, X1 = q1, X2 = q2, X3 = q3, X4 = q3)?

6 / 24

Answers

q1 q2 q3 a 1-a b 1-b c 1-c ◮ P =   a 1 − a b 1 − b c 1 − c   ◮ P(X0 = q1, X1 = q2, X3 = q1) = 0. You cannot go to q1 from q2. ◮ Use the Markov Property. P(X0 = q1, X1 = q1, X2 = q2, X3 = q3, X4 = q3) = P(X0 = q1) · P(X1 = q1 | X0 = q1) . . . P(X4 = q3 | X3 = q3) = 0.3 · a · (1 − a) · (1 − b) · (1 − c)

7 / 24

n-step Transition Probabilities

Let rij(n) = P(Xn = j | X0 = i) represent the probability that you are in state j exactly n steps after reaching state i. The value of rij(n) can be calculated recursively as rij(n) =

• k∈S

rik(n − 1)pkj Observe that rij(1) = pij. ⇒ rij(n) = Pn

i,j

the i, j entry of Pn.

8 / 24

Recurrent and Transient States

◮ Accessible: State j is accessible or reachable from state i if ∃n ∈ N such that rij(n) > 0. ◮ Recurrence: A state i is recurrent if ∀j reachable from i, i is reachable from j. That is if A(i) is the set of reachable states from i, then i is recurrent if ∀j ∈ A(i) ⇒ i ∈ A(j). ◮ Transient: A state i is transient if it is not recurrent. ◮ Classify the states in the below Markov chain as recurrent or transient. q1 q2 q3 a 1-a 0.5 0.5 0.3 0.7

9 / 24

Answer

q1 q2 q3 a 1-a 0.5 0.5 0.3 0.7 ◮ If a = 1, q1, q2, q3 are recurrent. ◮ If a < 1, q1 is transient and q2, q3 are recurrent.

10 / 24

Decomposition of States

◮ Recurrent Class: For any recurrent state i, all states A(i) (the set of states reachable from i) form a recurrent class. Any Markov chain can be decomposed into one ore more recurrent classes. ◮ A state in a recurrent class is not reachable from states in any

• ther recurrent class (try to prove this).

◮ Transient states are not reachable from a recurrent state. Moreover, from every transient state atleast one recurrent state is reachable. ◮ Find the recurrent classes in the following MC: q1 q2 q3 a 1-a 0.5 0.5 0.3 0.7

11 / 24

Answers

q1 q2 q3 a 1-a 0.5 0.5 0.3 0.7 ◮ If a = 1, {q1}, {q2, q3} form two recurrent classes. ◮ If a < 1, q1 is transient, and {q2, q3} form a recurrent class.

12 / 24

General Decomposition of States

A Markov chain is called irreducible if it only has one recurrent

• class. For any non-irreducible Markov chain, we can identify the

recurrent classes using the following process ◮ Create directed edges between any two nodes that have a non-zero transition probability between them. ◮ Find the strongly connected components of the graph. ◮ Use transitions between different strongly connected components to further topologically sort the graph. ◮ Each strongly connected component at the bottom of the topologically sorted structure forms a recurrent class. All

• ther nodes in this final structure are transient.

13 / 24

Example

q1 q2 q3 q4 q5 0.1 0.9 0.4 0.5 0.3 0.6 0.1 0.1 0.5 0.3 0.5 0.7

14 / 24

Periodicity

Consider an irreducible Markov chain. Define d(i) := g.c.d.{n ≥ 1|rii(n) > 0} ◮ Remember: rij(n) = P(Xt+n = j|Xt = i) ◮ ”All paths back to i take a multiple of d(i) steps” ◮ Fact: ∀i d(i) is the same. ◮ Fact: for Markov chains with more than one recurrent class, each class has a separate value for d ◮ We define a Markov chain as aperiodic if d(i) = 1 ∀i. ◮ Otherwise, we say it’s periodic with period d

15 / 24

Periodicity Examples

Are the following Markov chains aperiodic? 1 2 3 1 0.5 0.5 1 ◮ d = 2, so this is periodic. 1 2 3 1 0.5 0.5 0.99 0.01 ◮ d = 1, so this is aperiodic. ◮ Adding a self loop will make an irreducible Markov chain aperiodic!

16 / 24

Periodicity Examples (continued)

We won’t particularly worry about periodicity/aperiodicity for Markov chains with more than 1 recurrent class. 1 2 3 1 0.5 0.5 1

17 / 24

Stationary Distribution

If we choose the initial state of the Markov chain according to the distribution P(X0 = j) = π0(j) ∀j and this implies P(Xn = j) = π0(j) ∀j, n then we say that π0 is stationary. The balance equations are sufficient for stationarity: π0(j) =

m

• k=1

π0(k)pkj ∀j ◮ The balance equations can be written as π0 = π0P. In linear algebra terms, π0 is a left eigenvector of P that has corresponding eigenvalue λ = 1 ◮ In general, there can be multiple unique stationary distributions.

18 / 24

Stationary Distribution Example

1 2 0.25 0.75 0.25 0.75 Let’s try π0 = [1, 0]. π1(1) = P(X1 = 1|X0 = 1)π0(1) + P(X1 = 1|X0 = 2)π0(2) (1) = (0.25)(1) + (0.25)(0) (2) = 0.25 (3) Similarly, π1(2) = P(X1 = 2|X0 = 1)π0(1) + P(X1 = 2|X0 = 2)π0(2) (4) = (0.75)(1) + (0.75)(0) (5) = 0.75 (6) π1 = [0.25, 0.75] = π0, so π0 = [1, 0] is not stationary.

19 / 24

Stationary Distribution Example (continued)

1 2 0.25 0.75 0.25 0.75 Let’s solve for the stationary distribution. Let π0 = [x, 1 − x]. x = P(X1 = 1|X0 = 1)x + P(X1 = 1|X0 = 2)(1 − x) (7) = 0.25x + 0.25(1 − x) (8) 1 − x = P(X1 = 2|X0 = 1)x + P(X1 = 2|X0 = 2)(1 − x) (9) = 0.75x + 0.75(1 − x) (10) We see that 1 − x = 3x, so x = 0.25. Our stationary distribution is π0 = [0.25, 0.75]

20 / 24

Probability Flow Interpretation

i . . . π(1)p1i π(m)pmi . . . π(i)pi1 π(i)pim π(i)pii ◮ For any distribution, probability mass flows in and out of every state at each step. ◮ By subtracting π(i)pii from both sides of the balance equation, we have:

• j=i

π(j)pji

• flow in

= π(i)

• j=i

pij

• flow out

∀i

21 / 24

Example Revisited

1 2 1-a a b 1-b Let π0 = [x, 1 − x].

• j=i

π(j)pji = π(i)

• j=i

pij (11) Using this at state 2, xa = (1 − x)b (12) x = b a + b (13) 1 − x = a a + b (14)

22 / 24

The Big Theorem

◮ If a Markov chain is finite and irreducible, it has a unique invariant distribution π and π(i) is the long term fraction of time that X(n) is equal to i, almost surely. ◮ If the Markov chain is also aperiodic, then the distribution of X(n) converges to π.

23 / 24

References

Introduction to probability. DP Bertsekas, JN Tsitsiklis - 2002

24 / 24