10/2/2018 Department of Veterinary and Animal Sciences The Markov - - PDF document

10/2/2018 1

The Markov property

Anders Ringgaard Kristensen

Department of Veterinary and Animal Sciences

The Markov property – again! Let in be the state at stage n The Markov property is satisfied if and only if

• P(in+1| in, in-1, … , i1) = P (in+1| in)
• In words: The distribution of the state at next stage

depends only on the present state – previous states are not relevant.

This property is crucial in Markov decision processes.

Department of Veterinary and Animal Sciences Slide 2

Markov property: Example

Litter size in sows:

• Litter size in sows may be represented as a multi

dimensional normal distribution from previous exercise.

• We wish to predict litter size of parity n
• How shall we define the state space in order to fulfill the

Markov property?

Department of Veterinary and Animal Sciences Slide 3

10/2/2018 2

Markovian prediction of litter size I

Straight forward solution:

• Define the state as in = (y1, y2, … , yn)
• Use the n+1 dimensional normal distribution of

litter sizes to find the conditional distribution (yn+1 | y1, y2, … , yn) ~ N(ν1–n, C1–n), where ν1–n and C1–n are determined as in the previous exercise (Advanced topics from statistics).

• For a sow in parity 8 this means e.g. 158 = 2.5 x

109 state combinations.

• Prohibitive

Department of Veterinary and Animal Sciences Slide 4

Markovian prediction of litter size II

Trick most often used in practice:

• Only include the 2 – 3 most recent litter size

results.

• Regard (yn-2, yn-1, yn, yn+1)’ as a 4

dimensional normal distribution – or (yn-1, yn, yn+1)’ as a 3 dimensional normal distribution.

• Determine the conditional normal distribution

(yn+1 | yn-2, yn-1, yn) ~ N(ν(n-2)–n, C(n-2)–n) – or (yn+1 | yn-1, yn) ~ N(ν(n-1)–n, C(n-1)–n)

Department of Veterinary and Animal Sciences Slide 5

Litter size – remember two most recent parities

A valid (and soluble) Decision Graph NOT a Markov Decision Process

Department of Veterinary and Animal Sciences Slide 6

10/2/2018 3

Trick – memory variable

NOW it is a Markov Decision Process

Department of Veterinary and Animal Sciences Slide 7

Markovian prediction of litter size III

Motivation for trick:

• We want the prediction to be as precise as possible. In
• ther words, we wish to minimize the conditional

variance.

• The conditional variance is minimized by including all

previous litter sizes in the prediction.

• By including the most recent litter size, the variance is

decreased considerably.

• By including the two most recent litter sizes, the

variance is further decreased (but less than first time).

• Including the three most recent litter sizes will only

slightly decrease the variance.

Department of Veterinary and Animal Sciences Slide 8

Markovian prediction of litter size IV

Conditional variance of litter size, parity 12

7,4 7,6 7,8 8 8,2 8,4 8,6 8,8 1 2 3 4 5 6 7 8 9 10 11 Number of previous parities included Conditional variance

Effect of including the m = 0, … , 11 most recent litter sizes in prediction of litter size of parity 12.

10/2/2018 4

Markovian prediction of litter size V Including ”memory variables” in the state space is the most commonly applied technique for (approximately) satisfying the Markov property. Always check the Markov property!

Department of Veterinary and Animal Sciences Slide 10