Decision support methods revisited Anders Ringgaard Kristensen - - PowerPoint PPT Presentation

Decision support methods – revisited

Anders Ringgaard Kristensen

Markov decision processes – aka “Dynamic programming”

A true dynamic method – may run “for ever” Handles the combinatorial explosion in a very efficient way The most important limitations are:

• The Markov property
• Full observability of state space
• The curse of dimensionality

Dias 2 Advanced Quantitative Methods in Herd Management

What is this?

… …

Advanced Quantitative Methods in Herd Management Slide 3

What is this?

Advanced Quantitative Methods in Herd Management Slide 4

What is this?

Advanced Quantitative Methods in Herd Management Slide 5

What is this?

Advanced Quantitative Methods in Herd Management Slide 6

What is this?

Advanced Quantitative Methods in Herd Management Slide 7

The Markov property

Formal definition: Let it be the state at stage t. The Markov property is satisfied if and only if P(it+1| it, it-1, … , i1) = P (it+1| it) In words: The distribution of the state at next stage depends only on the present state – previous states are not relevant. This property is crucial in Markov decision processes.

Milk Milk Milk Milk

Red edges violate the Markov property If our biological knowledge implies that the red edges should be there, we need to take it into account by

• 1. Memory variables
• 2. Bayesian updating techniques

Dias 8 Advanced Quantitative Methods in Herd Management

The Markov property – how to compensate I

If the “biological truth” is as shown above, we may include memory variables in the state space. The trick has been used in numerous dairy cow (and sow) replacement studies.

Milk Milk Milk Milk Milk Milk Milk Milk Milk prev Milk prev Milk prev Milk prev

Dias 9 Advanced Quantitative Methods in Herd Management

What is this?

Advanced Quantitative Methods in Herd Management Slide 10

What is this

Advanced Quantitative Methods in Herd Management Slide 11

What is this?

Advanced Quantitative Methods in Herd Management Slide 12

What is this?

Our sheep litter size model from mandatory report Yn = µn + A + εn Does the model make sense?

Advanced Quantitative Methods in Herd Management Slide 13

The Markov property – how to compensate II

A more general approach is to introduce a latent (unobservable) variable interpreted as the milk production potential (MPP) of the cow. Each time a new milk yield observation is made, the MPP is re- estimated using Bayesian updating. The estimated milk production potential, eMPP, is included in the state space. It works because eMPP is observable! The trick is used in several newer dairy cow replacement studies.

Milk Milk Milk Milk eMPP eMPP eMPP eMPP

Dias 14 Advanced Quantitative Methods in Herd Management

Changing to an MDP

Figure 12.8 in textbook … Dynamic Linear Model:

• Yn = µn + An + εn
• An = An-1

Kalman filter:

• Updates estimate for An

each time a new litter size is observed.

Advanced Quantitative Methods in Herd Management Slide 15

The dynamic case – when time matters

Time t Time t+1 Time t+2 Time t+3

Mast Dec Util Age Milk Mast Dec Util Age Milk

Replace Treat Keep

Mast Dec Util Age Milk Mast Util Age Milk

Gross margin

Dias 16 Advanced Quantitative Methods in Herd Management

The curse of dimensionality – and how to avoid it

When several state variables (cow traits) are considered at a realistic number of levels, the state space grows to prohibitive dimensions. Dairy cow replacement models often have millions of state combinations. The solution is to decompose the state space according to time and build a hierarchical model. Has a tremendous effect on computational performance – even models with millions of state combinations can be solved. The technique has been used in numerous dairy cow replacement models. Implemented in

• The MLHMP software system (MLHMP) as used in this

course

• The MDP package for R
• SIMBA – The Israeli MLHMP project

Dias 17 Advanced Quantitative Methods in Herd Management

A hierarchical Markov decision process with three levels

BI ePP ePP ePP ePP eTP ePP eTP ePP

Cow level Lact. level Month/ Week/ Day

ePP eTP ePP eTP ePP eTP ePP eTP

BI: Breeding index ePP: Estimated Permanent Potential eTP: Estimated Temporary Potential 1st lactation 2nd lactation

Dec Util eTP

Other decisions and utilities are omitted for clarity Herd life of cow

Dias 18 Advanced Quantitative Methods in Herd Management

Markov decision processes – summary

In a Markov process we have:

• A structure
• Time as stages
• A state being observed at each stage
• Often defined by the values of several state

variables

• An action being taken when the state is known
• A numerical content
• Rewards depending on state and action
• (Outputs of various kinds …)
• Transition probabilities from state i to state j

depending on action

• The Markov property

Algorithms

• Value iteration: Finite time
• Policy iteration: Infinite time

Advanced Quantitative Methods in Herd Management Slide 19

Hierarchical Markov processes

In a hierarchical model, each level is modeled by separate Markov decision processes. The uppermost level is called the founder Lower levels are called children/child levels They all have the usual properties of an MDP:

• Structure (stages, states, actions)
• Numerical content (rewards, outputs, transition

probabilities)

But:

• The numerical content is only specified at the lowest

level

• Higher levels calculate their parameters from their

children

Advanced Quantitative Methods in Herd Management Slide 20

Hierarchical Markov processes

In a state of a process ρ at child level n we know:

• Stage and state of process ρ
• Stage, state and action of process at level n-1
• …
• State and action of founder

ρ

Sow Parity Phase

Advanced Quantitative Methods in Herd Management Slide 21