How can we best combine forecasts for added value? J. Broecker 1 , - - PowerPoint PPT Presentation

▶
how can we best combine forecasts for added value
SMART_READER_LITE
LIVE PREVIEW

How can we best combine forecasts for added value? J. Broecker 1 , - - PowerPoint PPT Presentation

Jul 15, 2023 259 likes •535 views

Thorpex Montreal 2004 How can we best combine forecasts for added value? J. Broecker 1 , L. Clarke 1 , D. Kilminster 2 and L.A. Smith 1 , 2 1. Department of Statistics, London School of Economics, UK 2. Pembroke College, Oxford, UK Thorpex

slide-1
SLIDE 1

Thorpex Montreal 2004

How can we best combine forecasts for added value?

  • J. Broecker1, L. Clarke1,
  • D. Kilminster2 and L.A. Smith1,2
  • 1. Department of Statistics, London School of Economics, UK
  • 2. Pembroke College, Oxford, UK
slide-2
SLIDE 2

Thorpex Montreal 2004

How can we best combine forecasts for added value? How might we use TIGGE?

  • J. Broecker1, L. Clarke1,
  • D. Kilminster2 and L.A. Smith1,2
  • 1. Department of Statistics, London School of Economics, UK
  • 2. Pembroke College, Oxford, UK
slide-3
SLIDE 3

Motivation

1 Motivation Overview Combining Evaluation Example

Forecast improvement can be achieved in two ways:

slide-4
SLIDE 4

Motivation

1 Motivation Overview Combining Evaluation Example

Forecast improvement can be achieved in two ways:

  • improving the models (strategic)
slide-5
SLIDE 5

Motivation

1 Motivation Overview Combining Evaluation Example

Forecast improvement can be achieved in two ways:

  • improving the models (strategic)
  • using the available information more

effectively (tactical)

slide-6
SLIDE 6

Motivation

1 Motivation Overview Combining Evaluation Example

Forecast improvement can be achieved in two ways:

  • improving the models (strategic)
  • using the available information more

effectively (tactical) THORPEX:

“THORPEX will develop, demonstrate and evaluate a multi-model, multi-analysis and multi-national ensemble prediction system, referred to as TIGGE.”

slide-7
SLIDE 7

Overview

2 Motivation Overview Combining Evaluation Example

  • Combining Simulations
  • Evaluation

– skill scores → Broecker – bootstrapping and meaningful skill comparison

  • Example - combining ECMWF and NCEP
slide-8
SLIDE 8

Inputs

3 Motivation Overview Combining Evaluation Example

Climatological Distribution

slide-9
SLIDE 9

Inputs

3 Motivation Overview Combining Evaluation Example

Dressed Point Forecast Climatological Distribution x

Roulston & Smith, Tellus 55 2003 Raftery et al. Univ. Washington Dept. of Stat. Tech. Report 440 2003

slide-10
SLIDE 10

Inputs

3 Motivation Overview Combining Evaluation Example

Dressed Point Forecast Ensemble Product Climatological Distribution x x x x x x x x x x x

Roulston & Smith, Tellus 55 2003 Raftery et al. Univ. Washington Dept. of Stat. Tech. Report 440 2003

slide-11
SLIDE 11

Inputs

3 Motivation Overview Combining Evaluation Example

Dressed Point Forecast Ensemble Product Combined Forecast Climatological Distribution x x x x x x x x x x x

Roulston & Smith, Tellus 55 2003 Raftery et al. Univ. Washington Dept. of Stat. Tech. Report 440 2003

slide-12
SLIDE 12

Combining: Skill Scores

4 Motivation Overview Combining Evaluation Example

The combination is based on the skill of the final forecast s = S(f, o) f forecast distribution

  • verifying observation

S skill score s skill of forecast f

slide-13
SLIDE 13

Combining

5 Motivation Overview Combining Evaluation Example

One combination method is to take a weighted sum of the component distributions

f =

  • αifi

Choose αi that maximises the skill score

s = max S(f, o)

  • ver a set of historical forecast-verification

pairs

Ignorance : s = − log p(o) Roulston & Smith, Monthly Weather Review 130

slide-14
SLIDE 14

Evaluation:Bootstrapping

6 Motivation Overview Combining Evaluation Example

Skill Model A Model B Model C

Out of sample, particular location, particular lead time, particular target

slide-15
SLIDE 15

Evaluation:Bootstrapping

6 Motivation Overview Combining Evaluation Example

Skill Model A Model B Model C

Out of sample, particular location, particular lead time, particular target

slide-16
SLIDE 16

The Right Comparison

7 Motivation Overview Combining Evaluation Example

We do not want to compare the uncertainty in the average performance of two models. We want the uncertainty in the comparative performance of the models to each other.

Bootstrap the difference: sA − sBBS Not the difference of the bootstraps:sABS − sBBS

slide-17
SLIDE 17

Comparative Skill

8 Motivation Overview Combining Evaluation Example

Comparitive Skill

A vs C A vs B B vs C

slide-18
SLIDE 18

Example

9 Motivation Overview Combining Evaluation Example Predicting temperature at Heathrow. Using

  • NCEP high resolution
  • NCEP ensemble
  • ECMWF high resolution
  • ECMWF ensemble

Evaluating using Ignorance - out of sample

slide-19
SLIDE 19

Example - one lead time

10 Motivation Overview Combining Evaluation Example

Comparitive Skill Lead Time (Days) IGN: smaller = good Combination vs NCEP Hi Res.

slide-20
SLIDE 20

Example - one lead time

10 Motivation Overview Combining Evaluation Example

Comparitive Skill Lead Time (Days) IGN: smaller = good Combination vs NCEP Ens. Combination vs NCEP Hi Res.

slide-21
SLIDE 21

Example - one lead time

10 Motivation Overview Combining Evaluation Example

Comparitive Skill Lead Time (Days) IGN: smaller = good Combination vs ECMWF Hi Res. Combination vs NCEP Ens. Combination vs NCEP Hi Res.

slide-22
SLIDE 22

Example - one lead time

10 Motivation Overview Combining Evaluation Example

Combination vs ECMWF Hi Res. Combination vs NCEP Ens. Combination vs NCEP Hi Res. Combination vs ECMWF Ens. Comparitive Skill Lead Time (Days) IGN: smaller = good

slide-23
SLIDE 23

Example - all lead times

11 Motivation Overview Combining Evaluation Example

Comparitive Skill Lead Time (Days) IGN: smaller = good Combination vs NCEP Hi Res.

slide-24
SLIDE 24

Example - all lead times

11 Motivation Overview Combining Evaluation Example

Comparitive Skill Lead Time (Days) IGN: smaller = good Combination vs NCEP Ens. Combination vs NCEP Hi Res.

slide-25
SLIDE 25

Example - all lead times

11 Motivation Overview Combining Evaluation Example

Comparitive Skill Lead Time (Days) IGN: smaller = good Combination vs ECMWF Hi Res. Combination vs NCEP Ens. Combination vs NCEP Hi Res.

slide-26
SLIDE 26

Example - all lead times

11 Motivation Overview Combining Evaluation Example

Comparitive Skill Lead Time (Days) IGN: smaller = good Combination vs ECMWF Hi Res. Combination vs NCEP Ens. Combination vs NCEP Hi Res. Combination vs ECMWF Ens.

slide-27
SLIDE 27

Summary

12 Motivation Overview Combining Evaluation Example

We have:

  • presented a user-orientated methodology for com-

bining simulations

  • whatever combination method, the evaluation must

be robust

  • dressing method, combination method and size of

forecast-verifi cation archive affects performance

  • potential relevance to TIGGE, provides a frame-

work for allowing users to extract the forecast in- formation most relevant to them