Markov Chains Transition Matrix P = ( p ij ) where p ij = P ( j | i ) - - PowerPoint PPT Presentation

▶

Dec 09, 2023 304 likes •514 views

Markov Chains Transition Matrix P = ( p ij ) where p ij = P ( j | i ) . Defining Properties: 0 p ij 1 , j p ij = 1 . Regular P n for some n > 0 has all positive entries only. Main Property: An equilibrium/fixed vector exists and is

SLIDE 1

Markov Chains

Transition Matrix P = (pij) where pij = P(j | i). Defining Properties: 0 ≤ pij ≤ 1,

j pij = 1.

Regular Pn for some n > 0 has all positive entries only. Main Property: An equilibrium/fixed vector exists and is

unique. v · P = v where v is a probability row vector.

Absorbing There is an i such that pii = 1. If so, all OTHER entries in row i are 0; pij = 0 for j = i. Also there is a positive probability to go from any state to an absorbing state. MAY NOT BE AT THE FIRST STEP. Main Property: Long Term Trend.

Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 1 / 19

SLIDE 2

Examples

1 0.5 0.5

    0.4 0.3 0.3 0.3 0.7 0.2 0.2 0.3 0.3 0.6 0.4    

0.2 0.8 0.5 0.5

1 1

  0.2 0.8 0.8 0.2 0.8 0.2  

  1 1 1  

Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 2 / 19

SLIDE 3

Example Continued

For (5), P2 =   0.68 0.16 0.16 0.16 0.68 0.16 0.16 .16 0.68   . So the matrix is regular.

Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 3 / 19

SLIDE 4

Application

Example (Problem 25, Section 10.2)

The probability that a complex assembly line works correctly depends only

n whether the line worked the last time it was used. There is a 0.9

chance that the line will work correctly if it worked correctly the last time, and a 0.8 chance that it will work correctly if it failed last time. Set up a transition matrix, and find the long range probability that the line will work correctly.

Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 4 / 19

SLIDE 5

Answer.

The matrix is 0.9 0.1 0.8 0.2

. This is regular. We need to solve the system
x,

y

0.9, 0.1 0.8, 0.2

y

This is

0.9x + 0.8y,

0.1x + 0.2y

y

We can rewrite it as
−0.1x + 0.8y

= 0 0.1x − 0.8y = 0. The two equations are x = 8y. So the equilibrium vector is

1/9

In general, the coefficient matrix of the system is PT − I, and you set every equation equal to 0: [PT − I | 0]. Transpose means that you interchange rows and columns.

Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 5 / 19

SLIDE 6

Regular Markov Chains, Summary

1 The matrix P has positive entries only. 2 There is a unique equilibrium (probability) vector V such that

V · P = V .

3 limm→∞ Pm =

   V1 . . . Vn . . . . . . . . . V1 . . . Vn    , limm→∞ v · Pm = V .

4 To solve for V , you need to solve the system [Pt − In×n | 0]. There is

a unique solution satisfying 0 ≤ Vj ≤ 1 and V1 + · · · + Vn = 1.

Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 6 / 19

SLIDE 7

Previous Example

  −0.1 0.8 0.1 −0.8 1 1 1   →   1 −8 1 1 1   → −9 −1 1 1 1

1 1/9 1 1 1

→ 1 1/9 1 8/9

1 8/9 1 1/9

Dan Barbasch

Math 1105 Chapter 10, November 6 Week of November 6 7 / 19

SLIDE 8

Absorbing Markov Chains

1 The matrix P has some pii = 1. You rearrange the states so that the

matrix is P = I R Q

. For each state there must a positive

probability for it to go to an equilibrium state EVENTUALLY. This means each row of FR must have at least one nonzero entry.

2 limm→∞ Pm =

(I − Q)−1R

. The rows of (I − Q)−1R tell you

the probabilities of ending up in the state corresponding to the

column. The matrix F = (I − Q)−1 is called the fundamental matrix.

The number fij give the number of visits to state j before being absorbed, given that the current state is i. The explanation comes from the fact that (I − Q)−1 = I + Q + Q2 + · · · + Qm + . . . appears when you compute Pm.

Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 8 / 19

SLIDE 9

Gambler’s Ruin

Two players A and B, play a game. They toss a coin. If heads, then A pays B $1. If tails, B pays A $1. They start out with a total of $3. The game ends whenever on of the players is broke. Analyze the game.

Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 9 / 19

SLIDE 10

Examples

    1 1 0.4 0.6 0.2 0.4 0.4    

  1 0.4 0.6 0.6 0.4  

Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 10 / 19

SLIDE 11

Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 11 / 19

SLIDE 12

Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 12 / 19

SLIDE 13

Classification of States, Optional I

Question: How to tell if a Markov Chain is regular.

1 We say j is accessible from i if pm(i, j) > 0 for some m > 0. i, j

communicate, i ≃ j, if pn(i, j) > 0 and pm(j, i) > 0 for some m, n > 0.

2 i ≃ i 3 i ≃ j implies j ≃ i 4 i ≃ j and j ≃ k implies i ≃ j 5 The sets of equivalent states are called classes 6 A Markov Chain is irreducible if all states are in a single class. 7 Regular implies irreducible. 8 Absorbing states are single classes. 9 i is called transient if fi = P(i for some m | i) = 1. 10 i is called recurrent if fi = P(i for some m | i) < 1. Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 13 / 19

SLIDE 14

Classification of States, Optional II

To find the equivalence classes:

1 Draw the states 2 Forget the probabilities on the diagonal 3 Join two states by an edge if p(i, j) > 0 and p(j, i) > 0. 4 If only one is > 0, draw an arrow. Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 14 / 19

SLIDE 15

Genetics I

Genetics somethimes studied the case when offspring from the same parents are mated; two of these offspring are mated,and so on. Let A be the dominant gene and a the recessive one. The original offspring can carry genes AA, Aa = aA, aa. There are six ways in which two offspring can mate: State Mating 1 AA&AA 2 AA&Aa 3 AA&aa 4 Aa&Aa 5 Aa&aa 6 aa&aa

Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 15 / 19

SLIDE 16

Genetics II

Model the inheritance of the genes by a Markov chain. Suppose that the

ffspring are randomly mated with each other. Find the transition matrix.

P = 1 2 3 4 5 6 1 | 1 | 2 | 1/4 1/2 1/4 | 3 | 1 | 4 | 1/16 1/4 1/8 1/4 1/4 1/16 | 5 | 1/4 1/2 1/4 | 6 | 1 | The principle is that an offspring inherits one gene from the two of one parent with equal probability, and independently another gene from the

ther parent in the same way. The inheritances from the two parents are

independent.

Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 16 / 19

SLIDE 17

Genetics III

Aa&Aa =      AA 1/4 Aa 1/4 + 1/4 = 1/2 aa 1/4 AA&AA 1/16 = 1/4 · 1/4 AA&Aa 1/4 = 1/4 · 1/2 + 1/2 · 1/4 AA&aa 1/8 = 1/4 · 1/4 + 1/4 · 1/4 Aa&Aa 1/4 = 1/2 · 1/2 Aa&aa 1/8 = 1/2 · 1/4 + 1/4 · 1/2 aa&aa 1/16 = 1/4 · 1/4

Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 17 / 19

SLIDE 18

Questions

1 Identify the absorbing states. 2 Find the matrix Q. 3 Find the fundamental matrix F. 4 If two parents with genes Aa are mated, find the number of pairs of

ffspring that can be expected before the either the dominant or the

recessive gene no longer appears.

5 If two parents with the genes Aa are mated, find the probability that

the recessive gene will disappear.

Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 18 / 19

SLIDE 19

Answers

For item (4), you start with state (4), and ask how many times you pass through states (2), (3), (4), (5) before being absorbed; the sum of entries in the row for (4) in the matrix F. For item (5), you add the entries in the matrix FR in the row for (4). The matrix P is relabelled         1 1 1/4 1/2 1/4 1 1/16 1/16 1/4 1/8 1/4 1/4 1/4 1/4 1/2         So R =     1/4 1/16 1/16 1/4     and Q =     1/2 1/4 1 1/4 1/8 1/4 1/4 1/4 1/2    

Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 19 / 19