Monitoring and data filtering II. Correlated data Advanced Herd - - PowerPoint PPT Presentation

Monitoring and data filtering

• II. Correlated data

Advanced Herd Management Anders Ringgaard Kristensen

The framework – as last lecture

Implementation: Products Factors Farmer's utility function Goals Restraints Planning Adjustments Analysis Control Plan

Summary, previous lecture

Key figures k1, k2, … , kt are regarded as a time series. Basic model (textbook pools the two errors): The values k1, k2, … , kt were regarded as independent and identically distributed.

Summary, previous lecture

The distribution of kt is N(k, σ2) – if errors are normally distributed. The variance: σ2 = V(est) + V(eot) Under those assumptions various control charts were presented.

Relevant questions

Is it (always) reasonable to assume that the true underlying value k is constant over time:

Trends: k increases/decreases Seasonality: k changes with season

Is it (always) reasonable to assume that the sample errors es1, es2, … , est are independent?

Repeated measurements on same animal(s) Temporary environmental effects

Is it (always) reasonable to assume that the observation errors eo1, eo2, … , eot are independent?

Yes, very often, but it depends on the method of measurement.

Our main example – were we wrong?

Daily gain, slaughter pigs

600 650 700 750 800 850 900 950

• 2. kvartal 97
• 3. kvartal 97
• 4. kvartal 97
• 1. kvartal 98
• 2. kvartal 98
• 3. kvartal 98
• 4. kvartal 98
• 1. kvartal 99
• 2. kvartal 99
• 3. kvartal 99
• 4. kvartal 99
• 1. kvartal 00
• 2. kvartal 00
• 3. kvartal 00
• 4. kvartal 00
• 1. kvartal 01
• 2. kvartal 01

Period g

Were we wrong?

Is there a trend? Is there a seasonal pattern? Are the sample errors es1, es2, … , est correlated? If yes, positively or negatively? Are the observation errors eo1, eo2, … , eot correlated? If yes, positively or negatively?

Cor r el ati on 650 700 750 800 850 900 650 700 750 800 850 900 P r e v i o s

Check for autocorrelation

Present versus previous observation Not obvious!

Cor r el ati on 650 700 750 800 850 900 650 700 750 800 850 900 P r e v i o s

Check for autocorrelation

Present versus same quarter last year Seems clear! Short series

Sample autocorrelation

• 1
• 0,8
• 0,6
• 0,4
• 0,2

0,2 0,4 0,6 0,8 1 1 2 3 4 Lag

Clear negative autocorrelation for lag 2 Clear positive autocorrelation for lag 4 Low autocorrelation for lags 1 & 3

A model for the autocorrelation

Assume that we are dealing with a pure seasonal effect: kt = µ + ρ4 (kt-4 - µ) + εt Where µ = 762 g, ρ4 = 0.68 and the residual εt ∼ N(0, σ2(1-ρ4

2)) where σ = 7.4 g

Predicted value for kt+1: Forecast error:

Observed and predicted gain

Daily gain, slaughter pigs

600 650 700 750 800 850 900 950

• 2. kvartal 97
• 3. kvartal 97
• 4. kvartal 97
• 1. kvartal 98
• 2. kvartal 98
• 3. kvartal 98
• 4. kvartal 98
• 1. kvartal 99
• 2. kvartal 99
• 3. kvartal 99
• 4. kvartal 99
• 1. kvartal 00
• 2. kvartal 00
• 3. kvartal 00
• 4. kvartal 00
• 1. kvartal 01
• 2. kvartal 01

Period g Observed gain Predicted gain

Control chart – correlated data

Construct a model describing the correlation Use the model to predict next observation Calculate the forecast error (difference between the predicted and observed value) Calculate the standard deviation of the forecast error Create a usual control chart for the prediction error

Prediction error control chart, visual

Daily gain, slaughter pigs

• 100
• 80
• 60
• 40
• 20

20 40 60 80 100 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Period g Prediction error Upper control limit Lower control limit

Daily gain example, comments

Further options

Include information on kt-2 (and perhaps even kt-3 and kt-1) – 4th order autoregressive time series: no marked improvement. Systematic seasonal effect. Combine with what you know about animal production!

Daily gain, slaughter pigs

600 650 700 750 800 850 900 950

• 2. kvartal 97
• 3. kvartal 97
• 4. kvartal 97
• 1. kvartal 98
• 2. kvartal 98
• 3. kvartal 98
• 4. kvartal 98
• 1. kvartal 99
• 2. kvartal 99
• 3. kvartal 99
• 4. kvartal 99
• 1. kvartal 00
• 2. kvartal 00
• 3. kvartal 00
• 4. kvartal 00
• 1. kvartal 01
• 2. kvartal 01

Period g Observed gain Predicted gain

Dynamic linear models

West & Harrison, chapter 2. Bayesian framework Patterns Self-calibrating – learning pattern from data Only very simple versions in textbook A more advanced version is presented here

A simple DLM

Observation equation: kt = µt + et, et ∼ N(0, σ2) Like before: et = est + eos The symbol µt is the underlying true value at time t. System equation: µt = µt-1 + wt, wt ∼ N(0, σw2) The true value is not any longer assumed to be constant. A fair assumption in animal production! Basically, we wish to detect ”large” changes in µt

A DLM with a trend

Observation equation: kt = µt + et, et ∼ N(0, σ2) System equations: µt = µt-1 + βt-1 + w1t, w1t ∼ N(0, σ1w

2)

βt = βt-1 + w2t, w2t ∼ N(0, σ2w

2)

Matrix notation

Let Kt = (k1, … , kn)’ be a vector of key figures observed at time t. Let θt = (θ1, … , θm)’ be a vector of parameters describing the system at time t. Observation equation Kt = Ftθt + et System equation: θt = Gtθt-1 + wt

Updating equations

See the textbook for details! Each time an observation is made, the current estimates for the parameters are updating in a Bayesian framework.

Trend model in matrix notation

Seasonal pig gain model – 4 seasons

Seasonal pig gain DLM

Daily gain, slaughter pigs

600 650 700 750 800 850 900 950

• 2. kvartal 97
• 3. kvartal 97
• 4. kvartal 97
• 1. kvartal 98
• 2. kvartal 98
• 3. kvartal 98
• 4. kvartal 98
• 1. kvartal 99
• 2. kvartal 99
• 3. kvartal 99
• 4. kvartal 99
• 1. kvartal 00
• 2. kvartal 00
• 3. kvartal 00
• 4. kvartal 00
• 1. kvartal 01

Period g Observed gain Predicted gain

Control chart, forecast error

Note that the standard deviation of the forecast error is calculated by the model.

Daily gain, slaughter pigs

• 100
• 80
• 60
• 40
• 20

20 40 60 80 100 2 . k v a r t a l 9 7 3 . k v a r t a l 9 7 4 . k v a r t a l 9 7 1 . k v a r t a l 9 8 2 . k v a r t a l 9 8 3 . k v a r t a l 9 8 4 . k v a r t a l 9 8 1 . k v a r t a l 9 9 2 . k v a r t a l 9 9 3 . k v a r t a l 9 9 4 . k v a r t a l 9 9 1 . k v a r t a l 2 . k v a r t a l 3 . k v a r t a l 4 . k v a r t a l 1 . k v a r t a l 1 Period g Forecast error Lower limit Upper limit

Components – trend

Daily gain, slaughter pigs

600 650 700 750 800 850 900 950

• 2. kvartal 97
• 3. kvartal 97
• 4. kvartal 97
• 1. kvartal 98
• 2. kvartal 98
• 3. kvartal 98
• 4. kvartal 98
• 1. kvartal 99
• 2. kvartal 99
• 3. kvartal 99
• 4. kvartal 99
• 1. kvartal 00
• 2. kvartal 00
• 3. kvartal 00
• 4. kvartal 00
• 1. kvartal 01

Period g Observed gain Predicted gain Level Season 1 Season 2 Season 3 Season 4

Components – seasonal parts

Daily gain, slaughter pigs

• 100
• 80
• 60
• 40
• 20

20 40 60 80 100

• 2. kvartal 97
• 3. kvartal 97
• 4. kvartal 97
• 1. kvartal 98
• 2. kvartal 98
• 3. kvartal 98
• 4. kvartal 98
• 1. kvartal 99
• 2. kvartal 99
• 3. kvartal 99
• 4. kvartal 99
• 1. kvartal 00
• 2. kvartal 00
• 3. kvartal 00
• 4. kvartal 00
• 1. kvartal 01

Period g Season 1 Season 2 Season 3 Season 4

Seasonal effects

More sophisticated approaches exist ”Seasonal” may just as well be a diurnal pattern. Thomas Nejsum Madsen will present an approach based on sine functions.

DLM in production monitoring

Not necessarily as graphs – automatic alarms. Variance components often ”guestimated” Many handles to adjust – dangerous Always combine with your knowledge on animal production. Well suited for mandatory report