# MATH 20: PROBABILITY Markov Chain Xingru Chen - PowerPoint PPT Presentation

## MATH 20: PROBABILITY Markov Chain Xingru Chen xingru.chen.gr@dartmouth.edu XC 2020 Random Walk 4 1 3 5 2 A ra walk is a mathematical object, known as a stochastic or random process, random that describes a path that

1. MATH 20: PROBABILITY Markov Chain Xingru Chen xingru.chen.gr@dartmouth.edu XC 2020

2. Random Walk 4 1 3 5 2 A ra walk is a mathematical object, known as a stochastic or random process, random that describes a path that consists of a succession of random steps on some โฆ mathematical space such as the integers. XC 2020

3. Markov Chain 2 4 1 5 3 XC 2020

4. Specifying a Markov Chain ยง We describe a Markov chain as follows: We have a set of states, ๐ = ๐ก ! , ๐ก " , โฏ , ๐ก # . ยง The process starts in one of ๐ก " these states and moves successively from one state to ๐ก \$ another. Each move is called a step. ๐ก ! ๐ก % ๐ก # XC 2020

5. ยง If the chain is currently in state ๐ก ! , then it moves to state ๐ก " at the ๐ !" next step with a probability ๐ก " ๐ \$" denoted by ๐ !" . ๐ก \$ ยง The probability ๐ !" does not ๐ "& ๐ &" depend upon which states the ๐ก ! chain was in before the current state. ยง These probabilities are called ๐ก % transition probabilities. ๐ %& ๐ก # XC 2020

6. ยง The process can remain in the state it is in, and this occurs with probability ๐ !! . ๐ก " ๐ !! ๐ก \$ ๐ก ! ๐ก % ๐ %% ๐ก # XC 2020

7. ยง An initial probability distribution, de fi ned on ๐ , speci fi es the starting state. Usually this is done by specifying a particular state as the starting state. ๐ฃ = ๐ฃ # ๐ฃ \$ ๐ฃ % ๐ฃ & ๐ฃ ' ๐ฃ " ๐ฃ \$ 0 1 0 0 0 ๐ก \$ ๐ฃ ! ๐ก & ) ( ๐ฃ ! = 1 ๐ก # !(# ๐ฃ % ๐ฃ # ๐ก ' ๐ก % XC 2020

8. THE LAND OF OZ the Land of Oz is blessed by many things, but ยง not by good weather. They never have two nice days in a row. If they ยง have a nice day, they are just as likely to have snow as rain the next day. If they have snow or rain, they have an even ยง chance of having the same the next day. If there is change from snow or rain, only half ยง of the time is this a change to a nice day. XC 2020

9. They never have two nice days in a row. If they have a nice day, they are just as likely to ยง have snow as rain the next day. 1 1 2 2 XC 2020

10. If they have snow or rain, they have an even chance of having the same the next day. ยง 1 1 2 2 1 1 2 2 XC 2020

11. If there is change from snow or rain, only half of the time is this a change to a nice day. ยง 1 1 2 2 1 1 4 4 1 1 2 2 XC 2020

12. If there is change from snow or rain, only half of the time is this a change to a nice day. ยง 1 1 2 2 1 1 4 4 1 4 1 1 1 2 2 4 XC 2020

13. R N S โ R 1 1 ยง N 2 2 S 1 1 4 4 1 4 1 1 1 2 2 4 XC 2020

14. R N S โ ) ) ) R 1 1 ยง * + + N 2 2 S 1 1 4 4 1 4 1 1 1 2 2 4 XC 2020

15. R N S โ ) ) ) R * + + 1 1 ยง N ) ) 0 2 2 * * S 1 1 4 4 1 4 1 1 1 2 2 4 XC 2020

16. R N S ) ) ) โ * + + R 1 1 ยง ) ) N 0 2 2 * * S ) ) ) + + * 1 1 4 4 1 4 1 1 1 2 2 4 XC 2020

17. ยง States: ยง ๐ก ! : rain ยง ๐ก " : nice 1 1 ยง ๐ก & : snow 2 2 ! ! ! " \$ \$ ! ! 1 1 ยง ๐ = 0 " " 4 4 ! ! ! \$ \$ " 1 4 1 1 1 2 2 4 XC 2020

18. Transition Matrix ยง The entries in the fi rst row of the matrix ๐ in the example represent the probabilities for the various kinds of weather following a rainy day. ยง Similarly, the entries in the second and third rows represent the probabilities for the various kinds of weather following nice and snowy days, respectively. ยง Such a square array is called the matrix of transition probabilities, or the transition matrix. 1 1 2 2 1 1 1 1 1 2 4 4 4 4 1 1 ๐ = 0 1 2 2 4 1 1 1 4 4 2 1 1 1 2 2 4 XC 2020

19. 1 1 2 2 1 1 1 ยง States: 1 1 2 4 4 4 4 ยง ๐ก ! : rain 1 1 ๐ = 0 ยง ๐ก " : nice 1 2 2 4 ยง ๐ก & : snow 1 1 1 4 4 2 1 1 1 2 2 4 the probability that, given the chain is in state ๐ today, it ๏ผ will be in state ๐ tomorrow XC 2020

20. 1 1 2 2 1 1 1 ยง States: 1 1 2 4 4 4 4 ยง ๐ก ! : rain 1 1 ๐ = 0 ยง ๐ก " : nice 1 2 2 4 ยง ๐ก & : snow 1 1 1 4 4 2 1 1 1 2 2 4 the probability that, given the chain is in (!) = ๐ '( ๏ผ state ๐ today, it will be in state ๐ ๐ '( tomorrow the probability that, given the chain is in (") = โฏ ๏ผ state ๐ today, it will be in state ๐ the day ๐ '( after tomorrow XC 2020

21. 1 1 2 2 1 1 1 ยง States: 1 1 2 4 4 4 4 ยง ๐ก ! : rain 1 1 ๐ = 0 ยง ๐ก " : nice 1 2 2 4 ยง ๐ก & : snow 1 1 1 4 4 2 1 1 1 2 2 4 ๏ผ โฆ (") = โฏ ๐ !& Day 0 Day 1 Day 2 โฆ XC 2020

22. 1 1 2 2 1 1 1 ยง States: 1 1 2 4 4 ยง ๐ก # : rain 1 1 4 4 ๐ = 0 ยง ๐ก \$ : nice 2 2 1 ยง ๐ก % : snow 1 1 1 4 4 4 2 1 1 1 2 2 4 ๐ ## ๐ #% ๐ #\$ ๐ \$% โฆ (") ๐ !& = ๐ !! ๐ !& + ๐ !" ๐ "& + ๐ !& ๐ && ๐ #% ๐ %% Day 0 Day 1 Day 2 โฆ XC 2020

23. 1 1 1 (\$) = ๐ ## ๐ #% + ๐ #\$ ๐ \$% + ๐ #% ๐ %% ๐ #% ยง States: 2 4 4 ๐ ## ๐ #\$ ๐ #% ยง ๐ก # : rain 1 1 ๐ \$# ๐ \$\$ ๐ \$% ๐ = = 0 ยง ๐ก \$ : nice 2 2 ๐ %# ๐ %\$ ๐ %% ยง ๐ก % : snow 1 1 1 ๐ ## ๐ #\$ ๐ #% ๐ #% 4 4 2 ๐ \$% ๐ %% ๐ !! ๐ !# ๐ ## ๐ #\$ ๐ #% ๐ ## ๐ #\$ ๐ #% ๐ !" ๐ "# ๐ \$ = ๐ \$# ๐ \$\$ ๐ \$% ๐ \$# ๐ \$\$ ๐ \$% โฆ 2 ๐ %# ๐ %\$ ๐ %% ๐ %# ๐ %\$ ๐ %% (\$) ๐ !# ๐ ## ๐ #% = โฆ Day 0 Day 1 Day 2 XC 2020

24. ๐ !! ๐ !# (\$) = ๐ ## ๐ #% + ๐ #\$ ๐ \$% + ๐ #% ๐ %% ๐ !" ๐ "# ๐ #% โฆ % = ( ๐ #, ๐ ,% ,(# ๐ !# ๐ ## โฆ Day 0 Day 1 Day 2 ๐ !! ๐ !# ๐ !" ๐ "# (\$) = ๐ ## ๐ #% + ๐ #\$ ๐ \$% + โฏ + ๐ #- ๐ -% ๐ #% โฆ - โฆ ๐ !โฏ ๐ โฏ# = ( ๐ #, ๐ ,% ,(# ๐ !& ๐ &# โฆ Day 0 Day 1 Day 2 XC 2020

25. ๐ !! ๐ !# ๐ ## ๐ #\$ ๐ #% ๐ ## ๐ #\$ ๐ #% ๐ \$ = ๐ \$# ๐ \$\$ ๐ \$% ๐ \$# ๐ \$\$ ๐ \$% ๐ !" ๐ "# 2 โฆ ๐ %# ๐ %\$ ๐ %% ๐ %# ๐ %\$ ๐ %% (\$) ๐ #% ๐ !# ๐ ## = โฆ Day 0 Day 1 Day 2 ๐ ## ๐ #\$ ๐ #% ) ๐ ) = ๐ \$# ๐ \$\$ ๐ \$% ๏ผ โฆ โฆ ๏ผ ๐ %# ๐ %\$ ๐ %% ()) ๐ #% Day 0 Day 1 Day 2 Day โฆ Day n โฆ = XC 2020

26. Transition Matrix ยง Let ๐ be the transition matrix of a Markov chain. ๐ + gives ยง The ๐๐ th entry ๐ '( of the matrix the probability that the Markov chain, starting in state ๐ก ' , will be in state ๐ก ( after ๐ steps. ๐ ## ๐ #\$ ๐ #% ) ๐ ) = ๐ \$# ๐ \$\$ ๐ \$% ๏ผ โฆ โฆ ๏ผ ๐ %# ๐ %\$ ๐ %% ()) ๐ #% Day 0 Day 1 Day 2 Day โฆ Day n โฆ = XC 2020

27. ๐ !# ๐ฃ ! ยง Starting states: the probability that ยง ยง rain: ๐ฃ # ๐ "# the chain is in state ยง nice: ๐ฃ \$ โฆ ๐ฃ " ๐ก ( after ๐ steps: ยง snow: ๐ฃ % ๐ = 3 ยง ๐ฃ # ๐ ## ๐ = 1 ยง ๐ฃ = ๐ฃ ! ๐ฃ " ๐ฃ & Day 0 Day 1 โฆ (#) = ๐ฃ # ๐ #% + ๐ฃ \$ ๐ \$% + ๐ฃ % ๐ %% ๐ฃ % Transition matrix: ยง ๐ !! ๐ !" ๐ !# ๐ !" ๐ !# ๐ "! ๐ "" ๐ "# ๐ = ๐ ## ๐ #\$ ๐ #% ๐ #! ๐ #" ๐ ## ๐ "! ๐ #! ๐ฃ # ๐ฃ \$ ๐ฃ % 1 1 1 ๐ \$# ๐ \$\$ ๐ \$% ๐ %# ๐ %\$ ๐ %% 2 4 4 ๐ "# 1 1 = 0 2 2 1 1 1 ๐ #" (#) = ๐ฃ๐ ๐ฃ (#) = ๐ฃ # (#) (#) ๐ฃ \$ ๐ฃ % 4 4 2 ๐ "" ๐ ## XC 2020

28. ๐ !# (#) = ๐ฃ # ๐ #% + ๐ฃ \$ ๐ \$% + ๐ฃ % ๐ %% ๐ฃ ! ๐ฃ % the probability ยง ๐ "# that the chain is โฆ ๐ฃ " in state ๐ก ( after (#) = ๐ฃ # ๐ #, + ๐ฃ \$ ๐ \$, + ๐ฃ % ๐ %, ๐ฃ , ๐ steps: ๐ฃ # ๐ ## ๐ = 1 ยง (#) = ๐ฃ๐ ๐ฃ (#) = ๐ฃ # (#) (#) ๐ฃ \$ ๐ฃ % Day 0 Day 1 โฆ # (") = ๐ !! ๐ !# + ๐ !" ๐ "# + ๐ !# ๐ ## = . ๐ !! ๐ !# ๐ !( ๐ (# ๐ !# ๐ฃ ! (+! ๐ !" the probability ยง (") = ๐ฃ ! ๐ !# (") + ๐ฃ " ๐ "# (") + ๐ฃ # ๐ ## ๐ "# (") ๐ฃ # that the chain is โฆ ๐ฃ " in state ๐ก ( after ๐ !# ๐ steps: (") = ๐ฃ ! ๐ !( (") + ๐ฃ " ๐ "( (") + ๐ฃ # ๐ #( (") ๐ฃ ( ๐ฃ # ๐ ## ๐ = 2 ยง (") = ๐ฃ๐ " ๐ฃ (") = ๐ฃ ! (") (") ๐ฃ " ๐ฃ # โฆ Day 0 Day 1 Day 2 XC 2020

Recommend

More recommend