10/2/2018 Department of Veterinary and Animal Sciences The Markov - PDF document

10/2/2018 Department of Veterinary and Animal Sciences The Markov property Anders Ringgaard Kristensen Department of Veterinary and Animal Sciences The Markov property – again! Let i n be the state at stage n The Markov property is satisfied if and only if • P( i n+ 1 | i n , i n- 1 , … , i 1 ) = P ( i n+ 1 | i n ) • In words: The distribution of the state at next stage depends only on the present state – previous states are not relevant. This property is crucial in Markov decision processes. Slide 2 Department of Veterinary and Animal Sciences Markov property: Example Litter size in sows: • Litter size in sows may be represented as a multi dimensional normal distribution from previous exercise. • We wish to predict litter size of parity n • How shall we define the state space in order to fulfill the Markov property? Slide 3 1

10/2/2018 Department of Veterinary and Animal Sciences Markovian prediction of litter size I Straight forward solution: • Define the state as i n = ( y 1 , y 2 , … , y n ) • Use the n+ 1 dimensional normal distribution of litter sizes to find the conditional distribution ( y n +1 | y 1 , y 2 , … , y n ) ~ N( ν 1– n , C 1– n ), where ν 1– n and C 1– n are determined as in the previous exercise (Advanced topics from statistics). • For a sow in parity 8 this means e.g. 15 8 = 2.5 x 10 9 state combinations. • Prohibitive Slide 4 Department of Veterinary and Animal Sciences Markovian prediction of litter size II Trick most often used in practice: • Only include the 2 – 3 most recent litter size results. • Regard ( y n -2 , y n -1 , y n , y n +1 )’ as a 4 dimensional normal distribution – or ( y n -1 , y n , y n +1 )’ as a 3 dimensional normal distribution. • Determine the conditional normal distribution ( y n +1 | y n -2 , y n -1 , y n ) ~ N( ν ( n -2)– n , C ( n -2)– n ) – or ( y n +1 | y n -1 , y n ) ~ N( ν ( n -1)– n , C ( n -1)– n ) Slide 5 Department of Veterinary and Animal Sciences Litter size – remember two most recent parities A valid (and soluble) Decision Graph NOT a Markov Decision Process Slide 6 2

10/2/2018 Department of Veterinary and Animal Sciences Trick – memory variable NOW it is a Markov Decision Process Slide 7 Department of Veterinary and Animal Sciences Markovian prediction of litter size III Motivation for trick: • We want the prediction to be as precise as possible. In other words, we wish to minimize the conditional variance. • The conditional variance is minimized by including all previous litter sizes in the prediction. • By including the most recent litter size, the variance is decreased considerably. • By including the two most recent litter sizes, the variance is further decreased (but less than first time). • Including the three most recent litter sizes will only slightly decrease the variance. Slide 8 Markovian prediction of litter size IV Conditional variance of litter size, parity 12 8,8 Conditional variance 8,6 8,4 8,2 8 7,8 7,6 7,4 0 1 2 3 4 5 6 7 8 9 10 11 Number of previous parities included Effect of including the m = 0, … , 11 most recent litter sizes in prediction of litter size of parity 12. 3

10/2/2018 Department of Veterinary and Animal Sciences Markovian prediction of litter size V Including ”memory variables” in the state space is the most commonly applied technique for (approximately) satisfying the Markov property. Always check the Markov property! Slide 10 4