A Theorem Suppose we are dealing with a finite MC. Then: • There is at least one recurrent state. from i is finite. Proof: (first part) Consider a non-recurrent state. If we start at that timestep, there is a nonzero probability that we will never see it again. Then if we start from that state and do an infinite number of timesteps, the probability that we see that state infinitely many times is zero. Start anywhere on the MC and do an infinite number of timesteps. Since the MC is finite, some step must appear infinitely many times. So, that step must be recurrent. 6 • For any recurrent state i , the expected hitting time h i , i if we start
A Theorem Suppose we are dealing with a finite MC. Then: • There is at least one recurrent state. from i is finite. Proof: (first part) Consider a non-recurrent state. If we start at that timestep, there is a nonzero probability that we will never see it again. Then if we start from that state and do an infinite number of timesteps, the probability that we see that state infinitely many times is zero. Start anywhere on the MC and do an infinite number of timesteps. Since the MC is finite, some step must appear infinitely many times. So, that step must be recurrent. 6 • For any recurrent state i , the expected hitting time h i , i if we start
A Theorem Suppose we are dealing with a finite MC. Then: • There is at least one recurrent state. from i is finite. Proof: (first part) Consider a non-recurrent state. If we start at that timestep, there is a nonzero probability that we will never see it again. Then if we start from that state and do an infinite number of timesteps, the probability that we see that state infinitely many times is zero. Start anywhere on the MC and do an infinite number of timesteps. Since the MC is finite, some step must appear infinitely many times. So, that step must be recurrent. 6 • For any recurrent state i , the expected hitting time h i , i if we start
A Theorem Suppose we are dealing with a finite MC. Then: • There is at least one recurrent state. from i is finite. Proof: (first part) Consider a non-recurrent state. If we start at that timestep, there is a nonzero probability that we will never see it again. Then if we start from that state and do an infinite number of timesteps, the probability that we see that state infinitely many times is zero. Start anywhere on the MC and do an infinite number of timesteps. Since the MC is finite, some step must appear infinitely many times. So, that step must be recurrent. 6 • For any recurrent state i , the expected hitting time h i , i if we start
A Theorem Suppose we are dealing with a finite MC. Then: • There is at least one recurrent state. from i is finite. Proof: (first part) Consider a non-recurrent state. If we start at that timestep, there is a nonzero probability that we will never see it again. Then if we start from that state and do an infinite number of timesteps, the probability that we see that state infinitely many times is zero. Start anywhere on the MC and do an infinite number of timesteps. Since the MC is finite, some step must appear infinitely many times. So, that step must be recurrent. 6 • For any recurrent state i , the expected hitting time h i , i if we start
A Theorem Suppose we are dealing with a finite MC. Then: • There is at least one recurrent state. from i is finite. Proof: (first part) Consider a non-recurrent state. If we start at that timestep, there is a nonzero probability that we will never see it again. Then if we start from that state and do an infinite number of timesteps, the probability that we see that state infinitely many times is zero. Start anywhere on the MC and do an infinite number of timesteps. Since the MC is finite, some step must appear infinitely many times. So, that step must be recurrent. 6 • For any recurrent state i , the expected hitting time h i , i if we start
Aperiodicity j X t Opposite of periodic: aperiodic . A Markov chain is said to be periodic if any of its states is periodic. s . divides 0 unless j S Intuition: Suppose we’re in one of these states at some timestep. Pr X t j j 1 such that P s integer To capture this intuition: state j is periodic if there exists some Then we can never return to it an odd number of timesteps later. 7
Aperiodicity j X t Opposite of periodic: aperiodic . A Markov chain is said to be periodic if any of its states is periodic. s . divides 0 unless j S Intuition: Suppose we’re in one of these states at some timestep. Pr X t j j 1 such that P s integer To capture this intuition: state j is periodic if there exists some Then we can never return to it an odd number of timesteps later. 7
Aperiodicity Intuition: Suppose we’re in one of these states at some timestep. Then we can never return to it an odd number of timesteps later. To capture this intuition: state j is periodic if there exists some s . A Markov chain is said to be periodic if any of its states is periodic. Opposite of periodic: aperiodic . 7 integer ∆ > 1 such that P s j , j = Pr [ X t + S = j | X t = j ] = 0 unless ∆ divides
Aperiodicity Intuition: Suppose we’re in one of these states at some timestep. Then we can never return to it an odd number of timesteps later. To capture this intuition: state j is periodic if there exists some s . A Markov chain is said to be periodic if any of its states is periodic. Opposite of periodic: aperiodic . 7 integer ∆ > 1 such that P s j , j = Pr [ X t + S = j | X t = j ] = 0 unless ∆ divides
Aperiodicity Intuition: Suppose we’re in one of these states at some timestep. Then we can never return to it an odd number of timesteps later. To capture this intuition: state j is periodic if there exists some s . A Markov chain is said to be periodic if any of its states is periodic. Opposite of periodic: aperiodic . 7 integer ∆ > 1 such that P s j , j = Pr [ X t + S = j | X t = j ] = 0 unless ∆ divides
j j is nonzero is greater than 1On j j is zero. Aperiodicity of Irreducible Chains - Another Definition Definition: If d j 1 gcd = greatest common divisor. timesteps s that are not multiples of d j , P s If gcd of all the timesteps where P s Are the definitions the same? Yes. Otherwise, it is periodic with period d j . 1, the Markov chain is said to be aperiodic. Proof: See Lecture note 18. Theorem: Assume that the MC is irreducible. has the same value for all states i . 0 j j P s 0 g.c.d. s d j Then 8
j j is nonzero is greater than 1On j j is zero. Aperiodicity of Irreducible Chains - Another Definition Theorem: Assume that the MC is irreducible. Then has the same value for all states i . Proof: See Lecture note 18. Definition: If d j 1, the Markov chain is said to be aperiodic. Otherwise, it is periodic with period d j . Are the definitions the same? Yes. If gcd of all the timesteps where P s timesteps s that are not multiples of d j , P s 1 gcd = greatest common divisor. 8 d ( j ) := g.c.d. { s > 0 | P s j , j > 0 }
j j is nonzero is greater than 1On j j is zero. Aperiodicity of Irreducible Chains - Another Definition Otherwise, it is periodic with period d j . 1 gcd = greatest common divisor. timesteps s that are not multiples of d j , P s If gcd of all the timesteps where P s Are the definitions the same? Yes. 1, the Markov chain is said to be aperiodic. Theorem: Assume that the MC is irreducible. Then If d j Definition: Proof: See Lecture note 18. has the same value for all states i . 8 d ( j ) := g.c.d. { s > 0 | P s j , j > 0 }
j j is nonzero is greater than 1On j j is zero. Aperiodicity of Irreducible Chains - Another Definition Theorem: Assume that the MC is irreducible. Then has the same value for all states i . Proof: See Lecture note 18. Otherwise, it is periodic with period d j . Are the definitions the same? Yes. If gcd of all the timesteps where P s timesteps s that are not multiples of d j , P s 1 gcd = greatest common divisor. 8 d ( j ) := g.c.d. { s > 0 | P s j , j > 0 } Definition: If d ( j ) = 1, the Markov chain is said to be aperiodic.
j j is nonzero is greater than 1On j j is zero. Aperiodicity of Irreducible Chains - Another Definition Theorem: Assume that the MC is irreducible. Then has the same value for all states i . Proof: See Lecture note 18. Are the definitions the same? Yes. If gcd of all the timesteps where P s timesteps s that are not multiples of d j , P s 1 gcd = greatest common divisor. 8 d ( j ) := g.c.d. { s > 0 | P s j , j > 0 } Definition: If d ( j ) = 1, the Markov chain is said to be aperiodic. Otherwise, it is periodic with period d ( j ) .
j j is nonzero is greater than 1On j j is zero. Aperiodicity of Irreducible Chains - Another Definition Theorem: Assume that the MC is irreducible. Then has the same value for all states i . Proof: See Lecture note 18. Are the definitions the same? Yes. If gcd of all the timesteps where P s timesteps s that are not multiples of d j , P s 1 gcd = greatest common divisor. 8 d ( j ) := g.c.d. { s > 0 | P s j , j > 0 } Definition: If d ( j ) = 1, the Markov chain is said to be aperiodic. Otherwise, it is periodic with period d ( j ) .
j j is nonzero is greater than 1On j j is zero. Aperiodicity of Irreducible Chains - Another Definition Theorem: Assume that the MC is irreducible. Then has the same value for all states i . Proof: See Lecture note 18. Are the definitions the same? Yes. If gcd of all the timesteps where P s timesteps s that are not multiples of d j , P s 1 gcd = greatest common divisor. 8 d ( j ) := g.c.d. { s > 0 | P s j , j > 0 } Definition: If d ( j ) = 1, the Markov chain is said to be aperiodic. Otherwise, it is periodic with period d ( j ) .
j j is zero. Aperiodicity of Irreducible Chains - Another Definition Theorem: Assume that the MC is irreducible. Then has the same value for all states i . Proof: See Lecture note 18. Are the definitions the same? Yes. If gcd of all the timesteps where P s On timesteps s that are not multiples of d j , P s 1 gcd = greatest common divisor. 8 d ( j ) := g.c.d. { s > 0 | P s j , j > 0 } Definition: If d ( j ) = 1, the Markov chain is said to be aperiodic. Otherwise, it is periodic with period d ( j ) . j , j is nonzero is greater than 1
Aperiodicity of Irreducible Chains - Another Definition Theorem: Assume that the MC is irreducible. Then has the same value for all states i . Proof: See Lecture note 18. Are the definitions the same? Yes. If gcd of all the timesteps where P s 1 gcd = greatest common divisor. 8 d ( j ) := g.c.d. { s > 0 | P s j , j > 0 } Definition: If d ( j ) = 1, the Markov chain is said to be aperiodic. Otherwise, it is periodic with period d ( j ) . j , j is nonzero is greater than 1On timesteps s that are not multiples of d ( j ) , P s j , j is zero.
Ergodicity century German-speaker, Boltzmann knew far too chain is ergodic. Theorem: A finite, irreducible, aperiodic Markov from an Elementary Point of View by Shalizi, pg. 479) path.” The name stuck.” ( Advanced Data Analysis property”, from ergon “energy, work” and hodos “way, much ancient Greek, so he called this the “ergodic as a whole. Being a highly-educated nineteenth An aperiodic state that is recurrent is called long enough, would be representative of the system energy and let it run, any one trajectory, continued idea that if you took an isolated system at constant “Ludwig Boltzmann needed a word to express the all its states are ergodic. ergodic . A Markov chain is said to be ergodic if 9
Ergodicity century German-speaker, Boltzmann knew far too chain is ergodic. Theorem: A finite, irreducible, aperiodic Markov from an Elementary Point of View by Shalizi, pg. 479) path.” The name stuck.” ( Advanced Data Analysis property”, from ergon “energy, work” and hodos “way, much ancient Greek, so he called this the “ergodic as a whole. Being a highly-educated nineteenth An aperiodic state that is recurrent is called long enough, would be representative of the system energy and let it run, any one trajectory, continued idea that if you took an isolated system at constant “Ludwig Boltzmann needed a word to express the all its states are ergodic. ergodic . A Markov chain is said to be ergodic if 9
Ergodicity century German-speaker, Boltzmann knew far too chain is ergodic. Theorem: A finite, irreducible, aperiodic Markov from an Elementary Point of View by Shalizi, pg. 479) path.” The name stuck.” ( Advanced Data Analysis property”, from ergon “energy, work” and hodos “way, much ancient Greek, so he called this the “ergodic as a whole. Being a highly-educated nineteenth An aperiodic state that is recurrent is called long enough, would be representative of the system energy and let it run, any one trajectory, continued idea that if you took an isolated system at constant “Ludwig Boltzmann needed a word to express the all its states are ergodic. ergodic . A Markov chain is said to be ergodic if 9
Stationary and Limiting Distributions
Stationary Distributions: Motivation Consider the driving exam MC again. Once we pass the test (state 4), we’re done forever. We never leave state 4. If our distribution is 0 0 0 1 : distribution is unchanged over a timestep. 10
Stationary Distributions: Motivation Consider the driving exam MC again. Once we pass the test (state 4), we’re done forever. We never leave state 4. If our distribution is 0 0 0 1 : distribution is unchanged over a timestep. 10
Stationary Distributions: Motivation Consider the driving exam MC again. Once we pass the test (state 4), we’re done forever. We never leave state 4. timestep. 10 If our distribution is [ 0 0 0 1 ] : distribution is unchanged over a
Stationary Distributions: Motivation II Or how about the two-cycle? What if our distribution is 0 5 0 5 ? Does it change with timesteps? No! 11
Stationary Distributions: Motivation II Or how about the two-cycle? No! 11 What if our distribution is [ 0 . 5 0 . 5 ] ? Does it change with timesteps?
Stationary Distributions: Motivation II Or how about the two-cycle? No! 11 What if our distribution is [ 0 . 5 0 . 5 ] ? Does it change with timesteps?
Definition: Stationary Distribution stationary distribution (a.k.a. an invariant or equilibrium Basically: not affected by timesteps. If we have this distribution, we have it forever. Another way of looking at it: is an eigenvector of P : If we multiply by P , we get a multiple of (actually, itself). Consequence: stochastic matrix always has 1 as an eigenvalue! To find stationary distribution: solve P (”balance equations”) 12 A distribution π over states in a Markov chain is said to be a distribution ) if π = π P .
Definition: Stationary Distribution stationary distribution (a.k.a. an invariant or equilibrium Basically: not affected by timesteps. If we have this distribution, we have it forever. Another way of looking at it: is an eigenvector of P : If we multiply by P , we get a multiple of (actually, itself). Consequence: stochastic matrix always has 1 as an eigenvalue! To find stationary distribution: solve P (”balance equations”) 12 A distribution π over states in a Markov chain is said to be a distribution ) if π = π P .
Definition: Stationary Distribution stationary distribution (a.k.a. an invariant or equilibrium Basically: not affected by timesteps. If we have this distribution, we have it forever. : If we multiply by P , we get a multiple of (actually, itself). Consequence: stochastic matrix always has 1 as an eigenvalue! To find stationary distribution: solve P (”balance equations”) 12 A distribution π over states in a Markov chain is said to be a distribution ) if π = π P . Another way of looking at it: π is an eigenvector of P
Definition: Stationary Distribution stationary distribution (a.k.a. an invariant or equilibrium Basically: not affected by timesteps. If we have this distribution, we have it forever. Consequence: stochastic matrix always has 1 as an eigenvalue! To find stationary distribution: solve P (”balance equations”) 12 A distribution π over states in a Markov chain is said to be a distribution ) if π = π P . Another way of looking at it: π is an eigenvector of P : If we multiply π by P , we get a multiple of π (actually, π itself).
Definition: Stationary Distribution stationary distribution (a.k.a. an invariant or equilibrium Basically: not affected by timesteps. If we have this distribution, we have it forever. stochastic matrix always has 1 as an eigenvalue! To find stationary distribution: solve P (”balance equations”) 12 A distribution π over states in a Markov chain is said to be a distribution ) if π = π P . Another way of looking at it: π is an eigenvector of P : If we multiply π by P , we get a multiple of π (actually, π itself). Consequence:
Definition: Stationary Distribution stationary distribution (a.k.a. an invariant or equilibrium Basically: not affected by timesteps. If we have this distribution, we have it forever. stochastic matrix always has 1 as an eigenvalue! 12 A distribution π over states in a Markov chain is said to be a distribution ) if π = π P . Another way of looking at it: π is an eigenvector of P : If we multiply π by P , we get a multiple of π (actually, π itself). Consequence: To find stationary distribution: solve π P = π (”balance equations”)
1 and 2 1 1. 1 a 2 b These equations are redundant! Add equation equation: 1 2 Solves to: b b a b a a b 2 An Example P 1 a 2 b a 1 1 2 1 b 1 b a a 1 2 1 13
1 and 2 1 1. 1 a 2 b These equations are redundant! Add equation equation: 1 2 Solves to: b b a b a a b 2 An Example 1 a 2 b a 1 1 2 1 b 1 b a a 1 2 1 13 π P = π
1 and 2 1 2 b a a b a b Solves to: 1. 2 1 These equations are redundant! Add equation equation: 2 b 1 a An Example b 1 a 2 b a 1 1 b a 13 [ ] 1 − a π P = π ⇔ [ π 1 , π 2 ] = [ π 1 , π 2 ] 1 − b
2 1 An Example 1 a b a a b a b Solves to: 1. 2 1 These equations are redundant! Add equation equation: 2 b b 2 1 a b a 13 [ ] 1 − a π P = π ⇔ [ π 1 , π 2 ] = [ π 1 , π 2 ] 1 − b ⇔ π ( 1 )( 1 − a ) + π 2 b = π 1 and
An Example 2 b b a a b a b Solves to: 1. 2 1 These equations are redundant! Add equation equation: 1 a b a 13 [ ] 1 − a π P = π ⇔ [ π 1 , π 2 ] = [ π 1 , π 2 ] 1 − b ⇔ π ( 1 )( 1 − a ) + π 2 b = π 1 and π 1 a + π 2 ( 1 − b ) = π 2
An Example These equations are redundant! Add equation equation: b a a b a b Solves to: 1. 2 1 13 a b [ ] 1 − a π P = π ⇔ [ π 1 , π 2 ] = [ π 1 , π 2 ] 1 − b ⇔ π ( 1 )( 1 − a ) + π 2 b = π 1 and π 1 a + π 2 ( 1 − b ) = π 2 ⇔ π 1 a = π 2 b .
An Example These equations are redundant! b a a b a b Solves to: 1. 2 1 Add equation equation: 13 a b [ ] 1 − a π P = π ⇔ [ π 1 , π 2 ] = [ π 1 , π 2 ] 1 − b ⇔ π ( 1 )( 1 − a ) + π 2 b = π 1 and π 1 a + π 2 ( 1 − b ) = π 2 ⇔ π 1 a = π 2 b .
An Example These equations are redundant! Add equation equation: b a a b a b Solves to: 1. 2 1 13 a b [ ] 1 − a π P = π ⇔ [ π 1 , π 2 ] = [ π 1 , π 2 ] 1 − b ⇔ π ( 1 )( 1 − a ) + π 2 b = π 1 and π 1 a + π 2 ( 1 − b ) = π 2 ⇔ π 1 a = π 2 b .
An Example Solves to: b a a b a b 13 b a [ ] 1 − a π P = π ⇔ [ π 1 , π 2 ] = [ π 1 , π 2 ] 1 − b ⇔ π ( 1 )( 1 − a ) + π 2 b = π 1 and π 1 a + π 2 ( 1 − b ) = π 2 ⇔ π 1 a = π 2 b . These equations are redundant! Add equation equation: π 1 + π 2 = 1.
An Example Solves to: b a a b a b 13 b a [ ] 1 − a π P = π ⇔ [ π 1 , π 2 ] = [ π 1 , π 2 ] 1 − b ⇔ π ( 1 )( 1 − a ) + π 2 b = π 1 and π 1 a + π 2 ( 1 − b ) = π 2 ⇔ π 1 a = π 2 b . These equations are redundant! Add equation equation: π 1 + π 2 = 1.
An Example b a b Solves to: 13 a [ ] 1 − a π P = π ⇔ [ π 1 , π 2 ] = [ π 1 , π 2 ] 1 − b ⇔ π ( 1 )( 1 − a ) + π 2 b = π 1 and π 1 a + π 2 ( 1 − b ) = π 2 ⇔ π 1 a = π 2 b . These equations are redundant! Add equation equation: π 1 + π 2 = 1. π = [ a + b , a + b ] .
1 and X 0 for all n . Hence, Pr X n Another Example i n i Pr X 0 i since X n Every distribution is invariant for this Markov chain. This is obvious, 2 2 1 P So: 2 1 1 0 0 1 2 1 14
1 and X 0 for all n . Hence, Pr X n Another Example i n i Pr X 0 i since X n Every distribution is invariant for this Markov chain. This is obvious, 2 2 1 So: 2 1 1 0 0 1 2 1 14 π P = π
X 0 for all n . Hence, Pr X n Another Example 2 i n i Pr X 0 i since X n Every distribution is invariant for this Markov chain. This is obvious, 2 14 So: 1 0 0 1 [ ] π P = π = [ π 1 , π 2 ] = [ π 1 , π 2 ] π 1 = π 1 and
X 0 for all n . Hence, Pr X n Another Example Every distribution is invariant for this Markov chain. This is obvious, i n i Pr X 0 i since X n So: 1 0 0 1 14 [ ] π P = π = [ π 1 , π 2 ] = [ π 1 , π 2 ] π 1 = π 1 and π 2 = π 2 .
X 0 for all n . Hence, Pr X n Another Example Every distribution is invariant for this Markov chain. i n i Pr X 0 i since X n This is obvious, 14 So: 1 0 0 1 [ ] π P = π = [ π 1 , π 2 ] = [ π 1 , π 2 ] π 1 = π 1 and π 2 = π 2 .
Another Example Every distribution is invariant for this Markov chain. This is obvious, i n i Pr X 0 i Hence, Pr X n So: 0 1 0 1 14 [ ] π P = π = [ π 1 , π 2 ] = [ π 1 , π 2 ] π 1 = π 1 and π 2 = π 2 . since X n = X 0 for all n .
Another Example 1 0 0 1 So: Every distribution is invariant for this Markov chain. This is obvious, 14 [ ] π P = π = [ π 1 , π 2 ] = [ π 1 , π 2 ] π 1 = π 1 and π 2 = π 2 . since X n = X 0 for all n . Hence, Pr [ X n = i ] = Pr [ X 0 = i ] , ∀ ( i , n ) .
j i exists and is independent of j . Main Theorem lim t expect you to know this). Proof: really long and messy, see note 18 or Ch. 7 of MU. (we won’t 1 h i i j i P t • i Suppose we are given a finite, irreducible, aperiodic Markov chain. P t • For all j , i , the limit lim t . • There is a unqiue stationary distribution Then: 15
j i exists and is independent of j . Main Theorem Suppose we are given a finite, irreducible, aperiodic Markov chain. Then: • For all j , i , the limit lim t P t • i lim t P t j i 1 h i i Proof: really long and messy, see note 18 or Ch. 7 of MU. (we won’t expect you to know this). 15 • There is a unqiue stationary distribution π .
Main Theorem Suppose we are given a finite, irreducible, aperiodic Markov chain. Then: • i lim t P t j i 1 h i i Proof: really long and messy, see note 18 or Ch. 7 of MU. (we won’t expect you to know this). 15 • There is a unqiue stationary distribution π . • For all j , i , the limit lim t →∞ P t j , i exists and is independent of j .
Main Theorem Suppose we are given a finite, irreducible, aperiodic Markov chain. Then: Proof: really long and messy, see note 18 or Ch. 7 of MU. (we won’t expect you to know this). 15 • There is a unqiue stationary distribution π . • For all j , i , the limit lim t →∞ P t j , i exists and is independent of j . • π i = lim t →∞ P t j , i = 1 / h i , i
Main Theorem Suppose we are given a finite, irreducible, aperiodic Markov chain. Then: Proof: really long and messy, see note 18 or Ch. 7 of MU. (we won’t expect you to know this). 15 • There is a unqiue stationary distribution π . • For all j , i , the limit lim t →∞ P t j , i exists and is independent of j . • π i = lim t →∞ P t j , i = 1 / h i , i
Connections between Linear Algebra and Markov Chains It turns out that the convergence of the limiting distribution to the stationary distribution corresponds to a nice result from linear algebra: if you multiply a random vector by a matrix a lot of times, the result will converge towards an eigenvector (specifically, one corresponding to the highest eigenvalue) w.h.p. here), i.e. one with a unique eigenvector (up to constant factors). (No, you do not need to know this for the midterms and the homeworks). 16 Perron-Frobenius: positive elements → single highest eigenvalue (1,
Connections between Linear Algebra and Markov Chains It turns out that the convergence of the limiting distribution to the stationary distribution corresponds to a nice result from linear algebra: if you multiply a random vector by a matrix a lot of times, the result will converge towards an eigenvector (specifically, one corresponding to the highest eigenvalue) w.h.p. here), i.e. one with a unique eigenvector (up to constant factors). (No, you do not need to know this for the midterms and the homeworks). 16 Perron-Frobenius: positive elements → single highest eigenvalue (1,
The Gambler’s Ruin Suppose you play a game with your friend. Flip a fair coin. Heads: you win a dollar. Tails: you lose a dollar. Repeat. You win when you get all your friend’s money. You lose when your friend gets all of yours. What is the probability that you win? If you and your friend have same amount of money: 1 2 by symmetry. What if you and your friend are willing to bet different amounts? 17
The Gambler’s Ruin Suppose you play a game with your friend. Flip a fair coin. Heads: you win a dollar. Tails: you lose a dollar. Repeat. You win when you get all your friend’s money. You lose when your friend gets all of yours. What is the probability that you win? If you and your friend have same amount of money: 1 2 by symmetry. What if you and your friend are willing to bet different amounts? 17
The Gambler’s Ruin Suppose you play a game with your friend. Flip a fair coin. Heads: you win a dollar. Tails: you lose a dollar. Repeat. You win when you get all your friend’s money. You lose when your friend gets all of yours. What is the probability that you win? If you and your friend have same amount of money: 1 2 by symmetry. What if you and your friend are willing to bet different amounts? 17
The Gambler’s Ruin Suppose you play a game with your friend. Flip a fair coin. Heads: you win a dollar. Tails: you lose a dollar. Repeat. You win when you get all your friend’s money. You lose when your friend gets all of yours. What is the probability that you win? What if you and your friend are willing to bet different amounts? 17 If you and your friend have same amount of money: 1 / 2 by symmetry.
The Gambler’s Ruin Suppose you play a game with your friend. Flip a fair coin. Heads: you win a dollar. Tails: you lose a dollar. Repeat. You win when you get all your friend’s money. You lose when your friend gets all of yours. What is the probability that you win? What if you and your friend are willing to bet different amounts? 17 If you and your friend have same amount of money: 1 / 2 by symmetry.
l 1 , l 2 are recurrent; all others are transient. What is the probability that you win (i.e. you hit state l 2 before l 1 )? i be the probability that you’re at state i after t timesteps. i for i 1 ? 0 (since they are transient absorbed into l 2 ). l 2 : probability that you win (state is P t lim t Want to find: q states). The Gambler’s Ruin II 1 l 2 l 1 P t What’s lim t Let P t States Markov chain. 18 Suppose you have l 1 dollars and your friend has l 2 . Express as above
Recommend
More recommend
