CMU-Q 15-381 Lecture 15: Predictions in Markov Chains Markov Decision Processes Teacher: Gianni A. Di Caro
M AKING P REDICTIONS : G ENERAL T WO - STATE MC ü 6 (7) = 6 (8) . 7 Probability distribution over the states after 9 steps, given initial distribution (7) = < = 7 = > : ; Absolute probability of state > at step 9 given by ü the initial distribution . = 1 − ( ( 0 < (, ) < 1 § How do we compute . 7 ? ) 1 − ) § … ! = # $ # % = 1 −( Eigenvector matrix 1 ) ) −( § … ! *$ = $ −1 1 +,- § Diagonalization: ! *$ .! = 1 1 − ( − ) = 0 $ 0 0 0 % = 1 Eigenvalue matrix 0 0 § Pre-multiplying both terms by ! and post-multiplying by ! *$ : . = !1! *$ § . % = (!1! *$ )(!1! *$ ) = (!1)(! *$ !)(1! *$ ) = (!1)4 % (1! *$ ) = !11! *$ % 1 % = 0 $ 0 = !1 % ! *$ , % 0 0 % 2
G ENERAL T WO - STATE MC ! = 1 − % % 0 < %, & < 1 & 1 − & * , * = / . 0 + = 1 −% & −% § ! * = +, * + -. + -. = . * , , 1 & −1 1 0 / 0 123 % −% & % 6 7 § ! * = ⋯ = . % + , 8 = 1 − % − & −& & & 123 123 § 8 * → 0 as : → ∞ & % § ! * → . % = < the matrix ! * in the limit of large : & 123 § Probability distribution over the states after : steps, given initial distribution = (?) : (?) ! * → A . (?) < = § = (*) = = (?) ! * = A . (?) (?) A 0 A 0 1 & % (?) + &A 0 (?) + %A 0 (?) = (?) = % + & & A . %A . % + & % + & (?) + A 0 (?) = 1 as : → ∞ , and given that A . 3
L IMITING DISTRIBUTION FOR GENERAL 2- STATE MC § State distribution over the states after ! steps, given the initial distribution " ($) : . / )→+ " ($) , ) = lim )→+ " ()) = lim / + . = " / + . à The chain has a limiting state probability distribution, denoted here as " § " is independent of " ($) § à " i s an Invariant limiting distribution of the chain: the limit exists and its invariant with respect to the initial distribution § The limiting distribution " is also a stationary distribution : if the chain starts (or arrives) in " as a state probability distribution, it stays in " (i.e., the distribution becomes stationary , it won’t change): ", = " 2 3 4 1 − α α h i h i h i β (1 − α )+ αβ αβ + α (1 − β ) β β α α 5 = = α + β α + β α + β α + β α + β α + β β 1 − β 4
L ONG - TERM BEHAVIOR : L IMITING DISTRIBUTIONS § For studying the long-term behavior of a generic MC with one-step transition matrix ! and " states, let’s consider the limit of the # -step conditional transition probabilities , denoted with $ : (() = lim (→* + ,- lim (→* 1 2 ( = 3 2 4 = 5) = $ ,- p ( n ) p ( n ) p ( n ) . . . Q 11 Q 12 . . . Q 1 m 11 12 1 m p ( n ) p ( n ) p ( n ) Q 21 Q 22 . . . Q 2 m . . . 21 22 2 m n →∞ T n = lim lim = . . . . ... ... . . . . n →∞ . . . . . . . . . . p ( n ) p ( n ) p ( n ) Q m 1 Q m 2 . . . Q mm . . . m 1 m 2 mm Let’s consider three different cases that can arise from the limit: 1) Limiting distribution exists 2) Limiting but no invariant distribution 3) No limiting (but possibly stationary) distribution 5
L IMITING DISTRIBUTION DOES EXIST (8) = lim 8→9 : &' lim 8→9 < = 8 = # = > = !) = % &' Limiting distribution: Let’s consider thet case when, for all !, #: 1. the limit reaches convergence values % &' o and for each # the value % &' is independent of initial the state ! , o 3 % ' = 1 → we can write % &' as % ' (i.e., % &' = % *' , ∀!, ,, # ∈ .), and ∑ '12 o p ( n ) p ( n ) p ( n ) . . . Q 1 Q 2 . . . Q m 11 12 1 m p ( n ) p ( n ) p ( n ) Q 1 Q 2 . . . Q m . . . 21 22 2 m n →∞ T n = lim lim = . . . . ... ... . . . . n →∞ . . . . . . . . . . p ( n ) p ( n ) p ( n ) Q 1 Q 2 . . . Q m . . . m 1 m 2 mm 6
L IMITING DISTRIBUTION IS INVARIANT 2 3 Q 1 Q 2 . . . Q m 6 7 Q 1 Q 2 . . . Q m 6 7 h i n →∞ p (0) T n = p (0) p (0) p (0) 6 7 lim . . . . . 6 7 m ... 1 2 . . 6 . . . . . 7 6 7 4 5 4 5 Q 1 Q 2 . . . Q m h h i i =1 p (0) i =1 p (0) i =1 p (0) P m P m P m ⇥ ⇤ Q 1 Q 2 . . . Q m = = = p Q 1 Q 2 . . . Q m i i i à The (unconditional) convergence values of the limits for the ! - step conditional transition probabilities define the limiting distribution of the chain, which is invariant with respect to the initial conditions § After the process has been in operation for some long duration, the probability of finding it in state " is # $ , irrespective of the starting state 7
L IMITING ⟹ S TATIONARY DISTRIBUTIONS From " ($) = " ($'() for ) → ∞ , also " ($) = " ($'() , à the limiting distribution § is the solution of the fixed point equation : ", = " à Because of the above equation, the limiting distribution is always also a stationary distribution : if the chain starts with or arrives to at any step ) to a probability state distribution equal to " , it doesn’t change it anymore § " = ", looks similar to an eigenvector equation: -. = /. , with eigenvalue / = 1 § By transposing the matrices and calling , as 1 : " 2 = ("1) 2 ⇒ 1 2 " 2 = " 2 , which is a “regular” eigenvector equation § à The transposed transition matrix 1 2 has eigenvectors with eigenvalue 1 that are stationary distributions expressed as column vectors. 8
E IGENVECTORS AND S TATIONARY DISTRIBUTION § Therefore, if the eigenvectors of the transposed transition matrix ! are known, then so are the stationary distributions of the Markov chain. This can save a lot of computations, avoiding to computing powers of ! ! § The stationary distribution is a left eigenvector (as opposed to the usual right eigenvectors) of the transition matrix, " = "! § Note: When there are multiple eigenvectors associated to an eigenvalue of value 1 , each such eigenvector gives rise to an associated stationary distribution. However, this can only occur when the Markov chain is reducible , i.e. has multiple communicating classes. 9
S TATIONARY D ISTRIBUTION ü Using ! = !# we can easily find the stationary distribution (assumed that there is one, and independently from the limiting distribution) either: by solving the linear equation ! = !# ü ü or by using the eigenvectors of the transposed transition matrix # $ § For instance, in the case of the general 2-state MC, let ! = % 1 − % and then we can solve the matrix equation and find the stationary matrix: 10
L IMITING BUT NO INVARIANT DISTRIBUTION 2. Limiting but no invariant distribution: Consider the case when for all !, #, the limit reaches convergence values $ %& and for each # the value $ %& is dependent of the initial the state ! , such that we cannot + $ %& = 1, ∀# must hold: write as before $ %& as $ & ; ∑ %)* p ( n ) p ( n ) p ( n ) . . . Q 11 Q 12 . . . Q 1 m 11 12 1 m p ( n ) p ( n ) p ( n ) Q 21 Q 22 . . . Q 2 m . . . 21 22 2 m n →∞ T n = lim lim = . . . . ... ... . . . . n →∞ . . . . . . . . . . p ( n ) p ( n ) p ( n ) Q m 1 Q m 2 . . . Q mm . . . m 1 m 2 mm 2 3 Q 11 Q 12 . . . Q 1 m 6 7 Q 21 Q 22 . . . Q 2 m 6 7 h i n →∞ p (0) T n = p (0) p (0) p (0) 6 7 lim . . . . . 6 ... 7 1 2 m . . 6 . . . . . 7 6 7 4 5 Q m 1 Q m 2 . . . Q mm à Each different initial distribution / (1) defines a possibly different limiting (stationary) distribution 11
L IMITING BUT NO INVARIANT DISTRIBUTION 2 3 Q 11 Q 12 . . . Q 1 m 6 7 Q 21 Q 22 . . . Q 2 m 6 7 h i n →∞ p (0) T n = p (0) p (0) p (0) 6 7 lim . . . . . 6 ... 7 1 2 m . . 6 . . . . . 7 6 7 4 5 Q m 1 Q m 2 . . . Q mm § Example: ! = 1 0 1 = % & , 2-state MC with 0 ≤ (, * ≤ 1 0 § ! + = ! for all , , such that a limiting distribution does exist but it always depends on - (/) 1 0 (/) (/) (/) (/) 1 = 1 2 1 2 1 & 1 & 0 12
N O L IMITING DISTRIBUTION 3. No Limiting distribution: The limit doesn’t reach a convergence value ! "# for all $, &. Therefore a limiting distribution as defined doesn’t exist. ( = 0 1 0 , in this case, ( ,- = 0 1 0 , ( ,-./ = 1 0 § 1 , 1 1 0 → the succession of ( ’s powers oscillates between the two matrices, § the MC is periodic of period 2 § However, a stationary distribution can still exist Limiting ⇒ Stationary, but the opposite doesn’t necessarily hold § 13
N O L IMITING , Y ES STATIONARY DISTRIBUTION ! = 0 1 0 , with, ! %& = 0 1 0 , ! %&() = 1 0 § 1 1 0 1 § The solution of the fixed point equation : 2 3 4 0 1 ⇥ ⇤ 5 = ⇥ ⇤ ⇥ ⇤ ⇥ ⇤ p T = p ⇒ a 1 − a a 1 − a 1 − a a a 1 − a = → 1 0 the resulting equation system: 1 − + = + + = 1 − + + = 0.5 satisfies the equations → / = 0.5 0.5 is a stationary distribution § This is intuitively expected since the oscillating behavior of the powers of ! that results in pairwise symmetric matrices, perfectly balances the probabilities of the two states of the chain. 14
Recommend
More recommend
