MATH 20: PROBABILITY Markov Chain Xingru Chen - PowerPoint PPT Presentation

MATH 20: PROBABILITY Markov Chain Xingru Chen xingru.chen.gr@dartmouth.edu XC 2020

Random Walk 4 1 3 5 2 A ra walk is a mathematical object, known as a stochastic or random process, random that describes a path that consists of a succession of random steps on some … mathematical space such as the integers. XC 2020

Markov Chain 2 4 1 5 3 XC 2020

Specifying a Markov Chain § We describe a Markov chain as follows: We have a set of states, 𝑇 = 𝑡 ! , 𝑡 " , ⋯ , 𝑡 # . § The process starts in one of 𝑡 " these states and moves successively from one state to 𝑡 $ another. Each move is called a step. 𝑡 ! 𝑡 % 𝑡 # XC 2020

§ If the chain is currently in state 𝑡 ! , then it moves to state 𝑡 " at the 𝑞 !" next step with a probability 𝑡 " 𝑞 $" denoted by 𝑞 !" . 𝑡 $ § The probability 𝑞 !" does not 𝑞 "& 𝑞 &" depend upon which states the 𝑡 ! chain was in before the current state. § These probabilities are called 𝑡 % transition probabilities. 𝑞 %& 𝑡 # XC 2020

§ The process can remain in the state it is in, and this occurs with probability 𝑞 !! . 𝑡 " 𝑞 !! 𝑡 $ 𝑡 ! 𝑡 % 𝑞 %% 𝑡 # XC 2020

§ An initial probability distribution, de fi ned on 𝑇 , speci fi es the starting state. Usually this is done by specifying a particular state as the starting state. 𝑣 = 𝑣 # 𝑣 $ 𝑣 % 𝑣 & 𝑣 ' 𝑣 " 𝑣 $ 0 1 0 0 0 𝑡 $ 𝑣 ! 𝑡 & ) ( 𝑣 ! = 1 𝑡 # !(# 𝑣 % 𝑣 # 𝑡 ' 𝑡 % XC 2020

THE LAND OF OZ the Land of Oz is blessed by many things, but § not by good weather. They never have two nice days in a row. If they § have a nice day, they are just as likely to have snow as rain the next day. If they have snow or rain, they have an even § chance of having the same the next day. If there is change from snow or rain, only half § of the time is this a change to a nice day. XC 2020

They never have two nice days in a row. If they have a nice day, they are just as likely to § have snow as rain the next day. 1 1 2 2 XC 2020

If they have snow or rain, they have an even chance of having the same the next day. § 1 1 2 2 1 1 2 2 XC 2020

If there is change from snow or rain, only half of the time is this a change to a nice day. § 1 1 2 2 1 1 4 4 1 1 2 2 XC 2020

If there is change from snow or rain, only half of the time is this a change to a nice day. § 1 1 2 2 1 1 4 4 1 4 1 1 1 2 2 4 XC 2020

R N S ↗ R 1 1 § N 2 2 S 1 1 4 4 1 4 1 1 1 2 2 4 XC 2020

R N S ↗ ) ) ) R 1 1 § * + + N 2 2 S 1 1 4 4 1 4 1 1 1 2 2 4 XC 2020

R N S ↗ ) ) ) R * + + 1 1 § N ) ) 0 2 2 * * S 1 1 4 4 1 4 1 1 1 2 2 4 XC 2020

R N S ) ) ) ↗ * + + R 1 1 § ) ) N 0 2 2 * * S ) ) ) + + * 1 1 4 4 1 4 1 1 1 2 2 4 XC 2020

§ States: § 𝑡 ! : rain § 𝑡 " : nice 1 1 § 𝑡 & : snow 2 2 ! ! ! " $ $ ! ! 1 1 § 𝑄 = 0 " " 4 4 ! ! ! $ $ " 1 4 1 1 1 2 2 4 XC 2020

Transition Matrix § The entries in the fi rst row of the matrix 𝑄 in the example represent the probabilities for the various kinds of weather following a rainy day. § Similarly, the entries in the second and third rows represent the probabilities for the various kinds of weather following nice and snowy days, respectively. § Such a square array is called the matrix of transition probabilities, or the transition matrix. 1 1 2 2 1 1 1 1 1 2 4 4 4 4 1 1 𝑄 = 0 1 2 2 4 1 1 1 4 4 2 1 1 1 2 2 4 XC 2020

1 1 2 2 1 1 1 § States: 1 1 2 4 4 4 4 § 𝑡 ! : rain 1 1 𝑄 = 0 § 𝑡 " : nice 1 2 2 4 § 𝑡 & : snow 1 1 1 4 4 2 1 1 1 2 2 4 the probability that, given the chain is in state 𝑗 today, it ？ will be in state 𝑘 tomorrow XC 2020

1 1 2 2 1 1 1 § States: 1 1 2 4 4 4 4 § 𝑡 ! : rain 1 1 𝑄 = 0 § 𝑡 " : nice 1 2 2 4 § 𝑡 & : snow 1 1 1 4 4 2 1 1 1 2 2 4 the probability that, given the chain is in (!) = 𝑞 '( ？ state 𝑗 today, it will be in state 𝑘 𝑞 '( tomorrow the probability that, given the chain is in (") = ⋯ ？ state 𝑗 today, it will be in state 𝑘 the day 𝑞 '( after tomorrow XC 2020

1 1 2 2 1 1 1 § States: 1 1 2 4 4 4 4 § 𝑡 ! : rain 1 1 𝑄 = 0 § 𝑡 " : nice 1 2 2 4 § 𝑡 & : snow 1 1 1 4 4 2 1 1 1 2 2 4 ？ … (") = ⋯ 𝑞 !& Day 0 Day 1 Day 2 … XC 2020

1 1 2 2 1 1 1 § States: 1 1 2 4 4 § 𝑡 # : rain 1 1 4 4 𝑄 = 0 § 𝑡 $ : nice 2 2 1 § 𝑡 % : snow 1 1 1 4 4 4 2 1 1 1 2 2 4 𝑞 ## 𝑞 #% 𝑞 #$ 𝑞 $% … (") 𝑞 !& = 𝑞 !! 𝑞 !& + 𝑞 !" 𝑞 "& + 𝑞 !& 𝑞 && 𝑞 #% 𝑞 %% Day 0 Day 1 Day 2 … XC 2020

1 1 1 ($) = 𝑞 ## 𝑞 #% + 𝑞 #$ 𝑞 $% + 𝑞 #% 𝑞 %% 𝑞 #% § States: 2 4 4 𝑞 ## 𝑞 #$ 𝑞 #% § 𝑡 # : rain 1 1 𝑞 $# 𝑞 $$ 𝑞 $% 𝑄 = = 0 § 𝑡 $ : nice 2 2 𝑞 %# 𝑞 %$ 𝑞 %% § 𝑡 % : snow 1 1 1 𝑞 ## 𝑞 #$ 𝑞 #% 𝑞 #% 4 4 2 𝑞 $% 𝑞 %% 𝑞 !! 𝑞 !# 𝑞 ## 𝑞 #$ 𝑞 #% 𝑞 ## 𝑞 #$ 𝑞 #% 𝑞 !" 𝑞 "# 𝑄 $ = 𝑞 $# 𝑞 $$ 𝑞 $% 𝑞 $# 𝑞 $$ 𝑞 $% … 2 𝑞 %# 𝑞 %$ 𝑞 %% 𝑞 %# 𝑞 %$ 𝑞 %% ($) 𝑞 !# 𝑞 ## 𝑞 #% = … Day 0 Day 1 Day 2 XC 2020

𝑞 !! 𝑞 !# ($) = 𝑞 ## 𝑞 #% + 𝑞 #$ 𝑞 $% + 𝑞 #% 𝑞 %% 𝑞 !" 𝑞 "# 𝑞 #% … % = ( 𝑞 #, 𝑞 ,% ,(# 𝑞 !# 𝑞 ## … Day 0 Day 1 Day 2 𝑞 !! 𝑞 !# 𝑞 !" 𝑞 "# ($) = 𝑞 ## 𝑞 #% + 𝑞 #$ 𝑞 $% + ⋯ + 𝑞 #- 𝑞 -% 𝑞 #% … - … 𝑞 !⋯ 𝑞 ⋯# = ( 𝑞 #, 𝑞 ,% ,(# 𝑞 !& 𝑞 &# … Day 0 Day 1 Day 2 XC 2020

𝑞 !! 𝑞 !# 𝑞 ## 𝑞 #$ 𝑞 #% 𝑞 ## 𝑞 #$ 𝑞 #% 𝑄 $ = 𝑞 $# 𝑞 $$ 𝑞 $% 𝑞 $# 𝑞 $$ 𝑞 $% 𝑞 !" 𝑞 "# 2 … 𝑞 %# 𝑞 %$ 𝑞 %% 𝑞 %# 𝑞 %$ 𝑞 %% ($) 𝑞 #% 𝑞 !# 𝑞 ## = … Day 0 Day 1 Day 2 𝑞 ## 𝑞 #$ 𝑞 #% ) 𝑄 ) = 𝑞 $# 𝑞 $$ 𝑞 $% ？ … … ？ 𝑞 %# 𝑞 %$ 𝑞 %% ()) 𝑞 #% Day 0 Day 1 Day 2 Day … Day n … = XC 2020

Transition Matrix § Let 𝑄 be the transition matrix of a Markov chain. 𝑄 + gives § The 𝑗𝑘 th entry 𝑞 '( of the matrix the probability that the Markov chain, starting in state 𝑡 ' , will be in state 𝑡 ( after 𝑜 steps. 𝑞 ## 𝑞 #$ 𝑞 #% ) 𝑄 ) = 𝑞 $# 𝑞 $$ 𝑞 $% ？ … … ？ 𝑞 %# 𝑞 %$ 𝑞 %% ()) 𝑞 #% Day 0 Day 1 Day 2 Day … Day n … = XC 2020

𝑞 !# 𝑣 ! § Starting states: the probability that § § rain: 𝑣 # 𝑞 "# the chain is in state § nice: 𝑣 $ … 𝑣 " 𝑡 ( after 𝑜 steps: § snow: 𝑣 % 𝑙 = 3 § 𝑣 # 𝑞 ## 𝑜 = 1 § 𝑣 = 𝑣 ! 𝑣 " 𝑣 & Day 0 Day 1 … (#) = 𝑣 # 𝑞 #% + 𝑣 $ 𝑞 $% + 𝑣 % 𝑞 %% 𝑣 % Transition matrix: § 𝑞 !! 𝑞 !" 𝑞 !# 𝑞 !" 𝑞 !# 𝑞 "! 𝑞 "" 𝑞 "# 𝑄 = 𝑞 ## 𝑞 #$ 𝑞 #% 𝑞 #! 𝑞 #" 𝑞 ## 𝑞 "! 𝑞 #! 𝑣 # 𝑣 $ 𝑣 % 1 1 1 𝑞 $# 𝑞 $$ 𝑞 $% 𝑞 %# 𝑞 %$ 𝑞 %% 2 4 4 𝑞 "# 1 1 = 0 2 2 1 1 1 𝑞 #" (#) = 𝑣𝑄 𝑣 (#) = 𝑣 # (#) (#) 𝑣 $ 𝑣 % 4 4 2 𝑞 "" 𝑞 ## XC 2020

𝑞 !# (#) = 𝑣 # 𝑞 #% + 𝑣 $ 𝑞 $% + 𝑣 % 𝑞 %% 𝑣 ! 𝑣 % the probability § 𝑞 "# that the chain is … 𝑣 " in state 𝑡 ( after (#) = 𝑣 # 𝑞 #, + 𝑣 $ 𝑞 $, + 𝑣 % 𝑞 %, 𝑣 , 𝑜 steps: 𝑣 # 𝑞 ## 𝑜 = 1 § (#) = 𝑣𝑄 𝑣 (#) = 𝑣 # (#) (#) 𝑣 $ 𝑣 % Day 0 Day 1 … # (") = 𝑞 !! 𝑞 !# + 𝑞 !" 𝑞 "# + 𝑞 !# 𝑞 ## = . 𝑞 !! 𝑞 !# 𝑞 !( 𝑞 (# 𝑞 !# 𝑣 ! (+! 𝑞 !" the probability § (") = 𝑣 ! 𝑞 !# (") + 𝑣 " 𝑞 "# (") + 𝑣 # 𝑞 ## 𝑞 "# (") 𝑣 # that the chain is … 𝑣 " in state 𝑡 ( after 𝑞 !# 𝑜 steps: (") = 𝑣 ! 𝑞 !( (") + 𝑣 " 𝑞 "( (") + 𝑣 # 𝑞 #( (") 𝑣 ( 𝑣 # 𝑞 ## 𝑜 = 2 § (") = 𝑣𝑄 " 𝑣 (") = 𝑣 ! (") (") 𝑣 " 𝑣 # … Day 0 Day 1 Day 2 XC 2020