[PPT] - MATH 20: PROBABILITY Markov Chain Xingru Chen PowerPoint Presentation

SLIDE 1

MATH 20: PROBABILITY

Markov Chain Xingru Chen xingru.chen.gr@dartmouth.edu

XC 2020

SLIDE 2

Random Walk

1 2 3 4 5

… A ra random walk is a mathematical

bject,

known as a stochastic

r random

process, that describes a path that consists

f

a succession

f random steps
n

some mathematical space such as the integers.

XC 2020

SLIDE 3

Markov Chain

1 2 3 4 5

XC 2020

SLIDE 4

Specifying a Markov Chain

§ We describe a Markov chain as follows: We have a set

f

states, 𝑇 = 𝑡!, 𝑡", ⋯ , 𝑡# . § The process starts in

ne
f

these states and moves successively from

ne

state to

another. Each

move is called a step.

𝑡! 𝑡" 𝑡# 𝑡$ 𝑡%

XC 2020

SLIDE 5

§ If the chain is currently in state 𝑡!, then it moves to state 𝑡

" at

the next step with a probability denoted by 𝑞!". § The probability 𝑞!" does not depend upon which states the chain was in before the current state. § These probabilities are called transition probabilities.

𝑡! 𝑡" 𝑡# 𝑡$ 𝑡%

𝑞!" 𝑞$" 𝑞%& 𝑞&" 𝑞"&

XC 2020

SLIDE 6

§ The process can remain in the state it is in, and this

ccurs

with probability 𝑞!!.

𝑡! 𝑡" 𝑡# 𝑡$ 𝑡%

𝑞!! 𝑞%%

XC 2020

SLIDE 7

§ An initial probability distribution, defined

n

𝑇, specifies the starting

state. Usually

this is done by specifying a particular state as the starting state.

𝑣!

𝑡#

𝑣"

𝑡$

𝑣#

𝑡%

𝑣$

𝑡&

𝑣%

𝑡' 𝑣 = 𝑣# 𝑣$ 𝑣% 𝑣& 𝑣' 1 (

!(# )

𝑣! = 1

XC 2020

SLIDE 8

THE LAND OF OZ

§ the Land

f

Oz is blessed by many things, but not by good weather. § They never have two nice days in a

row. If

they have a nice day, they are just as likely to have snow as rain the next day. § If they have snow

r

rain, they have an even chance

f

having the same the next day. § If there is change from snow

r

rain,

nly

half

f

the time is this a change to a nice day.

XC 2020

SLIDE 9

§ They never have two nice days in a

row. If

they have a nice day, they are just as likely to have snow as rain the next day. 1 2 1 2

XC 2020

SLIDE 10

§ If they have snow

r

rain, they have an even chance

f

having the same the next day. 1 2 1 2 1 2 1 2

XC 2020

SLIDE 11

§ If there is change from snow

r

rain,

nly

half

f

the time is this a change to a nice day. 1 2 1 2 1 2 1 2 1 4 1 4

XC 2020

SLIDE 12

§ If there is change from snow

r

rain,

nly

half

f

the time is this a change to a nice day. 1 2 1 2 1 2 1 2 1 4 1 4 1 4 1 4

XC 2020

SLIDE 13

1 2 1 2 1 2 1 2 1 4 1 4 1 4 1 4

§ ↗ R N S R N S

XC 2020

SLIDE 14

1 2 1 2 1 2 1 2 1 4 1 4 1 4 1 4

§ ↗ R N S R N S

) * ) + ) +

XC 2020

SLIDE 15

1 2 1 2 1 2 1 2 1 4 1 4 1 4 1 4

§ ↗ R N S R N S

) * ) + ) + ) * ) *

XC 2020

SLIDE 16

1 2 1 2 1 2 1 2 1 4 1 4 1 4 1 4

§ ↗ R N S R N S

) * ) + ) + ) * ) * ) + ) + ) *

XC 2020

SLIDE 17

1 2 1 2 1 2 1 2 1 4 1 4 1 4 1 4 § States: § 𝑡!: rain § 𝑡": nice § 𝑡&: snow § 𝑄 =

! " ! $ ! $ ! " ! " ! $ ! $ ! "

XC 2020

SLIDE 18

Transition Matrix

§ The entries in the first row

f

the matrix 𝑄 in the example represent the probabilities for the various kinds

f

weather following a rainy day. § Similarly, the entries in the second and third rows represent the probabilities for the various kinds

f

weather following nice and snowy days, respectively. § Such a square array is called the matrix

f

transition probabilities,

r

the transition matrix.

𝑄 = 1 2 1 4 1 4 1 2 1 2 1 4 1 4 1 2

1 2 1 2 1 2 1 2 1 4 1 4 1 4 1 4

XC 2020

SLIDE 19

𝑄 = 1 2 1 4 1 4 1 2 1 2 1 4 1 4 1 2

？

the probability that, given the chain is in state 𝑗 today, it will be in state 𝑘 tomorrow

§ States: § 𝑡!: rain § 𝑡": nice § 𝑡&: snow

1 2 1 2 1 2 1 2 1 4 1 4 1 4 1 4

XC 2020

SLIDE 20

𝑄 = 1 2 1 4 1 4 1 2 1 2 1 4 1 4 1 2

？

the probability that, given the chain is in state 𝑗 today, it will be in state 𝑘 tomorrow

§ States: § 𝑡!: rain § 𝑡": nice § 𝑡&: snow 𝑞'(

(!) = 𝑞'(

？

the probability that, given the chain is in state 𝑗 today, it will be in state 𝑘 the day after tomorrow

𝑞'(

(") = ⋯

1 2 1 2 1 2 1 2 1 4 1 4 1 4 1 4

XC 2020

SLIDE 21

𝑄 = 1 2 1 4 1 4 1 2 1 2 1 4 1 4 1 2 § States: § 𝑡!: rain § 𝑡": nice § 𝑡&: snow

Day Day 1 Day 2 …

？ …

𝑞!&

(") = ⋯

1 2 1 2 1 2 1 2 1 4 1 4 1 4 1 4

XC 2020

SLIDE 22

1 2 1 2 1 2 1 2 1 4 1 4 1 4 1 4

𝑄 = 1 2 1 4 1 4 1 2 1 2 1 4 1 4 1 2 § States: § 𝑡#: rain § 𝑡$: nice § 𝑡%: snow Day Day 1 Day 2 …

…

𝑞!&

(")

= 𝑞!!𝑞!& + 𝑞!"𝑞"& + 𝑞!&𝑞&&

𝑞## 𝑞#% 𝑞#$ 𝑞$% 𝑞#% 𝑞%%

XC 2020

SLIDE 23

𝑄 = 𝑞## 𝑞#$ 𝑞#% 𝑞$# 𝑞$$ 𝑞$% 𝑞%# 𝑞%$ 𝑞%% = 1 2 1 4 1 4 1 2 1 2 1 4 1 4 1 2 § States: § 𝑡#: rain § 𝑡$: nice § 𝑡%: snow 𝑞#%

($) = 𝑞##𝑞#% + 𝑞#$𝑞$% + 𝑞#%𝑞%%

Day Day 1 Day 2

… … 𝑞!! 𝑞!# 𝑞!" 𝑞"# 𝑞!# 𝑞##

𝑞## 𝑞#$ 𝑞#% 𝑞#% 𝑞$% 𝑞%% 𝑄$ = 𝑞## 𝑞#$ 𝑞#% 𝑞$# 𝑞$$ 𝑞$% 𝑞%# 𝑞%$ 𝑞%% 2 𝑞## 𝑞#$ 𝑞#% 𝑞$# 𝑞$$ 𝑞$% 𝑞%# 𝑞%$ 𝑞%% = 𝑞#%

($)

XC 2020

SLIDE 24

𝑞#%

($) = 𝑞##𝑞#% + 𝑞#$𝑞$% + 𝑞#%𝑞%%

= (

,(# %

𝑞#,𝑞,%

Day Day 1 Day 2

… … 𝑞!! 𝑞!# 𝑞!" 𝑞"# 𝑞!# 𝑞##

Day Day 1 Day 2

… … 𝑞!! 𝑞!# 𝑞!" 𝑞"# 𝑞!& 𝑞&# … 𝑞!⋯ 𝑞⋯#

𝑞#%

($) = 𝑞##𝑞#% + 𝑞#$𝑞$% + ⋯ + 𝑞#-𝑞-%

= (

,(#

𝑞#,𝑞,%

XC 2020

SLIDE 25

Day Day 1 Day 2 Day … Day n … ？ … ？ …

Day Day 1 Day 2

… … 𝑞!! 𝑞!# 𝑞!" 𝑞"# 𝑞!# 𝑞##

𝑄$ = 𝑞## 𝑞#$ 𝑞#% 𝑞$# 𝑞$$ 𝑞$% 𝑞%# 𝑞%$ 𝑞%% 2 𝑞## 𝑞#$ 𝑞#% 𝑞$# 𝑞$$ 𝑞$% 𝑞%# 𝑞%$ 𝑞%% = 𝑞#%

($)

𝑄) = 𝑞## 𝑞#$ 𝑞#% 𝑞$# 𝑞$$ 𝑞$% 𝑞%# 𝑞%$ 𝑞%%

)

= 𝑞#%

())

XC 2020

SLIDE 26

Transition Matrix

§ Let 𝑄 be the transition matrix

f

a Markov chain. § The 𝑗𝑘th entry 𝑞'( of the matrix 𝑄+ gives the probability that the Markov chain, starting in state 𝑡', will be in state 𝑡

( after

𝑜 steps.

Day Day 1 Day 2 Day … Day n … ？ … ？ …

𝑄) = 𝑞## 𝑞#$ 𝑞#% 𝑞$# 𝑞$$ 𝑞$% 𝑞%# 𝑞%$ 𝑞%%

)

= 𝑞#%

())

XC 2020

SLIDE 27

§ Starting states: § rain: 𝑣# § nice: 𝑣$ § snow: 𝑣%

𝑣 = 𝑣! 𝑣" 𝑣&

§ Transition matrix: 𝑄 = 𝑞!! 𝑞!" 𝑞!# 𝑞"! 𝑞"" 𝑞"# 𝑞#! 𝑞#" 𝑞## = 1 2 1 4 1 4 1 2 1 2 1 4 1 4 1 2

𝑞!" 𝑞!# 𝑞## 𝑞"" 𝑞"! 𝑞#! 𝑞"# 𝑞#"

𝑣%

(#) = 𝑣#𝑞#% + 𝑣$𝑞$% + 𝑣%𝑞%%

𝑣# 𝑣$ 𝑣% 𝑞## 𝑞#$ 𝑞#% 𝑞$# 𝑞$$ 𝑞$% 𝑞%# 𝑞%$ 𝑞%% 𝑣(#) = 𝑣#

(#)

𝑣$

(#)

𝑣%

(#) = 𝑣𝑄

𝑞!# 𝑞"# 𝑞##

Day Day 1 …

… 𝑣! 𝑣" 𝑣# § the probability that the chain is in state 𝑡( after 𝑜 steps: § 𝑙 = 3 § 𝑜 = 1

XC 2020

SLIDE 28

𝑣,

(#) = 𝑣#𝑞#, + 𝑣$𝑞$, + 𝑣%𝑞%,

𝑣(#) = 𝑣#

(#)

𝑣$

(#)

𝑣%

(#) = 𝑣𝑄

𝑞!# 𝑞"# 𝑞##

Day Day 1 …

… 𝑣! 𝑣" 𝑣# § the probability that the chain is in state 𝑡( after 𝑜 steps: § 𝑜 = 1

𝑣%

(#) = 𝑣#𝑞#% + 𝑣$𝑞$% + 𝑣%𝑞%%

§ the probability that the chain is in state 𝑡( after 𝑜 steps: § 𝑜 = 2

Day Day 1 Day 2

… 𝑣! … 𝑞!! 𝑞!# 𝑞!" 𝑞"# 𝑞!# 𝑞## 𝑣" 𝑣# 𝑞!#

(") = 𝑞!!𝑞!# + 𝑞!"𝑞"# + 𝑞!#𝑞## = . (+! #

𝑞!(𝑞(# 𝑣#

(") = 𝑣!𝑞!# (") + 𝑣"𝑞"# (") + 𝑣#𝑞## (")

𝑣(

(") = 𝑣!𝑞!( (") + 𝑣"𝑞"( (") + 𝑣#𝑞#( (")

𝑣(") = 𝑣!

(")

𝑣"

(")

𝑣#

(") = 𝑣𝑄"

XC 2020

SLIDE 29

Transition Matrix

§ Let 𝑄 be the transition matrix

f

a Markov chain, and let 𝑣 be the probability vector which represents the starting distribution. § Then the probability that the chain is in state 𝑡, after 𝑜 steps is the 𝑙th entry in the vector 𝑣(+) = 𝑣𝑄+.

Day Day 1 Day 2 Day … Day n … ？ … ？ …

𝑣()) = 𝑣𝑄) = 𝑣 𝑞## 𝑞#$ 𝑞#% 𝑞$# 𝑞$$ 𝑞$% 𝑞%# 𝑞%$ 𝑞%%

)

= 𝑣#

())

𝑣$

())

𝑣%

())

𝑣! 𝑣" 𝑣#

XC 2020

SLIDE 30

Open book Scope: Mostly Chapters 7, 8, 9, 10, and 11 § sum

f

random variables § LLN and CLT § generating functions § Markov chains Materials: sl slides, ho homework, qu quizzes, textbook Date & Time: 12:00 pm – 10: 00 pm, August 30 Office hours: August 27, 28 Homework due: 11:00 pm August 28

Au August 2020

Fi Final

26 26 27 27 28 28 29 29 30 30 31 31 01 01 02 02 03 03 04 04 05 05 06 06 07 07 08 08 09 09 10 10 11 11 12 12 13 13 14 14 15 15 16 16 17 17 18 18 19 19 20 20 21 21 22 22 23 23 24 24 25 25 26 26 28 28 27 27 30 30 05 05 31 31 01 01 02 02 04 04 03 03 29 29

Sun Mon Tue Wed Thu Fri Sat

XC 2020