Decadal Prediction Drift as a Particular Challenge for Verification - - PowerPoint PPT Presentation

Mean temperature (NH+SH)/2 [K] 1960 1970 1980 1990 2000 2010 288 289 290

• Decadal Prediction

Drift as a Particular Challenge for Verification

Henning Rust

Institut für Meteorologie, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 1

Decadal climate prediction . . .

. . . provides information about the future evolution of the statistics of regional climate from the output of a numerical model that has been initialized with observations . . .

1

1cited from Meehl et al. [2014]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 2

Decadal climate prediction . . .

2

2taken from Boer et al. [2016]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 3

Initialization with “observations”

Hindcast set: initialize every year, 10-yr hindcast each

courtesy of Jens Grieger

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 4

Initialization with “observations”

Hindcast set: initialize every year, 10-yr hindcast each

Full-field

assimilate directly Annual mean global temperature,

taken from Smith et al. [2013]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 4

Initialization with “observations”

Hindcast set: initialize every year, 10-yr hindcast each

Full-field

assimilate directly

Anomaly

assimilate anomalies Annual mean global temperature,

taken from Smith et al. [2013]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 4

Verification of decadal prediction

Mean temperature (NH+SH)/2 [K] 1960 1970 1980 1990 2000 2010 288 289 290

• Drift

x x x x x x x x x x −0.2 0.0 0.2 bias(τ) 1 2 3 4 5 6 7 8 9 10

Drift adjustment Re-Calibration

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 5

A framework

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 6

A framework

Q1: Predictions more accurate due to initialization? Q2: Does ensemble spread appropriately represents uncertainty?

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 6

Verification: What can we expect?3 T emporal scales

annual/seasonal averages 1yr lead-year 1 4yrs lead-years 2-5, 6-9 8yrs lead-years 2-9 more are preferable

3Decadal prediction verification framework Goddard et al. [2013]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 7

Verification: What can we expect?3 T emporal scales

annual/seasonal averages 1yr lead-year 1 4yrs lead-years 2-5, 6-9 8yrs lead-years 2-9 more are preferable

Spatial scales

scale of reference or larger, e.g. Temp 5◦×5◦ Precip 2.5◦×2.5◦ depends on study

3Decadal prediction verification framework Goddard et al. [2013]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 7

Verifying ensemble predictions: Accuracy of mean Ensemble mean

Hjτ = 1 Ne

Ne

• i=1

Hijτ Hijτ ens. member i, initialization j, lead-year τ

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 8

Verifying ensemble predictions: Accuracy of mean Ensemble mean

Hjτ = 1 Ne

Ne

• i=1

Hijτ Hijτ ens. member i, initialization j, lead-year τ

Q1: More accurate due to initialization?

MSESS = 1 − MSEH MSER historicals (no initialization but forcing) as reference

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 8

Verifying ensemble predictions: Spread Ensemble Spread

σ2

Hj =

1 Ne − 1

Ne

• i=1
• Hijτ − Hjτ

2

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 9

Verifying ensemble predictions: Spread Ensemble Spread

σ2

Hj =

1 Ne − 1

Ne

• i=1
• Hijτ − Hjτ

2

Q2: Does ensemble spread appropriately represents uncertainty?

CRPS(N(ˆ Hj, σ2H), Oj) Gaussian hindcast distributiona conditional and unconditional bias adjusted (ˆ Hj)

aGneiting and Raftery [2007]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 9

Verifying ensemble predictions: Spread Ensemble Spread

σ2

Hj =

1 Ne − 1

Ne

• i=1
• Hijτ − Hjτ

2

Q2: Does ensemble spread appropriately represents uncertainty?

CRPSS = 1 − CRPSH CRPSR MSE as variance of ref. fore- cast, CRPSS=0 is optimal!

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 9

Verifying ensemble predictions: Spread Ensemble Spread

σ2

Hj =

1 Ne − 1

Ne

• i=1
• Hijτ − Hjτ

2

Q2: Does ensemble spread appropriately represents uncertainty?

CRPSS = 1 − CRPSH CRPSR MSE as variance of ref. fore- cast, CRPSS=0 is optimal!

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 9

Verifying ensemble predictions: Spread Ensemble Spread

σ2

Hj =

1 Ne − 1

Ne

• i=1
• Hijτ − Hjτ

2

Q2: Does ensemble spread appropriately represents uncertainty?

CRPSS = 1 − CRPSH CRPSR LESS = ln

• σ2H

MSE

• MSE as variance of ref. fore-

cast, CRPSS=0 is optimal! Additionally, logarithmic ensemble spread score

e.g. Kadow et al. [2014]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 9

Small ensembles and significance MiKlip ensembles

baseline0 3 baseline1 10 prototype 15 + 15 preop ≤ 15 Goddard et al. [2013] suggest a bootstrap for significance

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 10

Small ensembles and significance MiKlip ensembles

baseline0 3 baseline1 10 prototype 15 + 15 preop ≤ 15 Goddard et al. [2013] suggest a bootstrap for significance

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 10

Small ensembles and significance MiKlip ensembles

baseline0 3 baseline1 10 prototype 15 + 15 preop ≤ 15 Goddard et al. [2013] suggest a bootstrap for significance

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 10

Small ensembles and biased scores

4

❼ bias corrected scores Ferro et al. [2008] ❼ application with RPS Kruschke et al. [2015] ❼ implemented in R-package SpecsVerification (Stefan Siegert)

4taken from Müller et al. [2005]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 11

Mean temperature (NH+SH)/2 [K] 1960 1970 1980 1990 2000 2010 288 289 290

• Drift

“Bias” or mean difference

ME = 1 N

N

• j=1

Hj − 1 N

N

• j=1

Oj

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 13

“Bias” or mean difference

ME = 1 N

N

• j=1

Hj − 1 N

N

• j=1

Oj := b For decadal prediction and other cases b = b(τ, X) , τ: forecast lead-time; X and climate state X. Sytematic error we belief we can compensate a posteriori

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 13

Drift

change in bias with forecast lead-time τ D(τ, X) = ∂ ∂τ b(τ, X)

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 14

Drift

change in bias with forecast lead-time τ D(τ, X) = ∂ ∂τ b(τ, X)

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 14

Quantifying drift

D(τ, X) =

• (D(τ, X))2

=

• ∂

∂τ b(τ, X) 2

Igor Kröner Drift Quantification and Correction in Decadal Predictions of Climate Extremes Indices, in preparation

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 15

Examples from MiKlip: Drift in global mean temperature

Mean temperature (NH+SH)/2 [K] 1960 1970 1980 1990 2000 2010 288 289 290

• Mean temperature (NH+SH)/2 [K]

1960 1970 1980 1990 2000 2010 288 289 290

• 1960−INITIALISATIO

N S−2000 HadCrut 4 (absolute) Uninitialised runs/RCP4.5

full-field anomaly

courtesy of Igor Kröner

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 16

Examples from MiKlip: Drift in global mean temperature

lead years bias(τ) D 1 2 3 4 5 6 7 8 9 10 −1 −0.5 0.5 1 0.1 0.2 0.3 0.4 0.5

x x x x x x x x x x x + + + + + + + + + + + decs4e dffs4e

red anomaly initialisation (baseline1, decs4e) brown full-field initialisation (prototype, dffs4e)

courtesy of Igor Kröner

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 17

x x x x x x x x x x −0.2 0.0 0.2 bias(τ) 1 2 3 4 5 6 7 8 9 10

Drift Adjustment

Verification of forecasts with bias/drift

S(H(t, τ), o(t))

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 19

Verification of forecasts with bias/drift

S(H(t, τ), o(t))

How good is the forecast once we have com- pensated for systematic errors?

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 19

Verification of forecasts with bias/drift

S(ˆ H(t, τ), o(t)) ˆ H(t, τ) = H(t, τ) − ˆ b(t, τ)

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 19

Verification of forecasts with bias/drift

S(ˆ H(t, τ), o(t)) ˆ H(t, τ) = H(t, τ) − ˆ b(t, τ)

Recommendationa full-field and anomalies

assume b(τ, X(t)) = b(τ, t) ˆ bICPO(t, τ) = 1 #{ti\t}

• ti\t

(H(ti, τ) − o(ti)) ≈ ME(τ) yr of forecast to be verified left out (ICPO 2011) lead-years τ are treated individually

aBoer et al. [2016]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 19

Back to the drift . . .

lead years bias(τ) D 1 2 3 4 5 6 7 8 9 10 −1 −0.5 0.5 1 0.1 0.2 0.3 0.4 0.5

x x x x x x x x x x x + + + + + + + + + + + decs4e dffs4e

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 20

Back to the drift . . .

lead years bias(τ) D 1 2 3 4 5 6 7 8 9 10 −1 −0.5 0.5 1 0.1 0.2 0.3 0.4 0.5

x x x x x x x x x x x + + + + + + + + + + + decs4e dffs4e

Drift ist smooth in τ

b(τ, X(t)) = b(τ), smooth/parametric form in τ ˆ bGan(t, τ) = a0 + a1 τ + a2 τ2 + a3 τ3 third order polynomial in τ Gangstø et al. [2013] (exponential Pattantyús-Ábrahám et al. [2016])

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 20

Back to the drift . . .

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 21

Back to the drift . . . Drift might change with climate (t)5

5Kharin et al. [2012]Fuˇ

ckar et al. [2014]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 21

Back to the drift . . . Drift might change with climate (t)5

ˆ b(t, τ) = a0 + a1 τ + a2 τ2 + a3 τ3

5Kharin et al. [2012]Fuˇ

ckar et al. [2014]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 21

Back to the drift . . . Drift might change with climate (t)5

ˆ b(t, τ) = a0(t) + a1(t) τ + a2(t) τ2 + a3(t) τ3

5Kharin et al. [2012]Fuˇ

ckar et al. [2014]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 21

Back to the drift . . . Drift might change with climate (t)5

ˆ bKru(t, τ) = (b0 +b1 t)+(b2 +b3 t) τ +(b4 +b5 t) τ2 +(b6 +b7 t) τ3

Kruschke et al. [2015]

5Kharin et al. [2012]Fuˇ

ckar et al. [2014]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 21

Drift . . .

ˆ bKru(t, τ) = (b0 +b1 t)+(b2 +b3 t) τ +(b4 +b5 t) τ2 +(b6 +b7 t) τ3

x x x x x x x x x x −1.0 0.0 1.0 bias(τ) 1 2 3 4 5 6 7 8 9 10

tas, full-field initialisation, red (early init, 1960) to blue (late init, 2004)

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 22

Drift . . .

ˆ bKru(t, τ) = (b0 +b1 t)+(b2 +b3 t) τ +(b4 +b5 t) τ2 +(b6 +b7 t) τ3

x x x x x x x x x x −0.2 0.0 0.2 bias(τ) 1 2 3 4 5 6 7 8 9 10

tas, anomaly initialisation, red (early init, 1960) to blue (late init, 2004)

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 22

Drift . . .

ˆ bKru(t, τ) = (b0 +b1 t)+(b2 +b3 t) τ +(b4 +b5 t) τ2 +(b6 +b7 t) τ3

x x x x x x x x x x −0.2 0.0 0.2 bias(τ) 1 2 3 4 5 6 7 8 9 10

tas, anomaly initialisation, red (early init, 1960) to blue (late init, 2004) That seems complex! Does this help?

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 22

Parametric drift adjustment vs ICPO

tas, full-field, MSESS, polynomial vs ICPO, yr2-5

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 23

Re-calibration

Probabilistic forecast

all figures curtesy of Alexander Pasternack

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 25

Probabilistic forecast

all figures curtesy of Alexander Pasternack

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 25

Re-calibration method for decadal predictions

fi(t, τ) = μ(t, τ) + εi(t, τ)

6 μ(t, τ): ensemble mean, i = 1 . . . M member, t =init. year, τ = lead year

6e.g. Weigel et al. [2008]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 26

Re-calibration method for decadal predictions

fi(t, τ) = μ(t, τ) + εi(t, τ)

6 μ(t, τ): ensemble mean, i = 1 . . . M member, t =init. year, τ = lead year

Re-calibrated ensemble

f Cal

i

(t, τ) = μ(t, τ) + εi(t, τ)

6e.g. Weigel et al. [2008]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 26

Re-calibration method for decadal predictions

fi(t, τ) = μ(t, τ) + εi(t, τ)

6 μ(t, τ): ensemble mean, i = 1 . . . M member, t =init. year, τ = lead year

Re-calibrated ensemble

f Cal

i

(t, τ) = α(t, τ) + μ(t, τ) + εi(t, τ) 1) α: bias and drift,

6e.g. Weigel et al. [2008]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 26

Re-calibration method for decadal predictions

fi(t, τ) = μ(t, τ) + εi(t, τ)

6 μ(t, τ): ensemble mean, i = 1 . . . M member, t =init. year, τ = lead year

Re-calibrated ensemble

f Cal

i

(t, τ) = α(t, τ) + β(t, τ)μ(t, τ) + εi(t, τ) 1) α: bias and drift, 2) β: conditional bias,

6e.g. Weigel et al. [2008]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 26

Re-calibration method for decadal predictions

fi(t, τ) = μ(t, τ) + εi(t, τ)

6 μ(t, τ): ensemble mean, i = 1 . . . M member, t =init. year, τ = lead year

Re-calibrated ensemble

f Cal

i

(t, τ) = α(t, τ) + β(t, τ)μ(t, τ) + γ(t, τ)εi(t, τ) 1) α: bias and drift, 2) β: conditional bias, 3) γ: spread

6e.g. Weigel et al. [2008]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 26

Re-calibration method for decadal predictions

fi(t, τ) = μ(t, τ) + εi(t, τ)

6 μ(t, τ): ensemble mean, i = 1 . . . M member, t =init. year, τ = lead year

Re-calibrated ensemble

f Cal

i

(t, τ) = α(t, τ) + β(t, τ)μ(t, τ) + γ(t, τ)εi(t, τ) 1) α: bias and drift, 2) β: conditional bias, 3) γ: spread find α(t, τ), β(t, τ), γ(t, τ) such that ensemble is calibrated with maximum sharpness

6e.g. Weigel et al. [2008]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 26

A model for α, β, and γ

a first go a: α(t, τ) = (a0 + a1t) + (a2 + a3t)τ + (a4 + a5t)τ2 + (a6 + a7t)τ3 ❼ ❼

aPasternack et al. [2017]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 27

A model for α, β, and γ

a first go a: α(t, τ) = (a0 + a1t) + (a2 + a3t)τ + (a4 + a5t)τ2 + (a6 + a7t)τ3 β(t, τ) = (b0 + b1t) + (b2 + b3t)τ + (b4 + b5t)τ2 + (b6 + b7t)τ3 γ(t, τ) = (c0 + c1t) + (c2 + c3t)τ + (c4 + c5t)τ2 + (c6 + c7t)τ3 ❼ ❼

aPasternack et al. [2017]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 27

A model for α, β, and γ

a first go a: α(t, τ) = (a0 + a1t) + (a2 + a3t)τ + (a4 + a5t)τ2 + (a6 + a7t)τ3 β(t, τ) = (b0 + b1t) + (b2 + b3t)τ + (b4 + b5t)τ2 + (b6 + b7t)τ3 γ(t, τ) = (c0 + c1t) + (c2 + c3t)τ + (c4 + c5t)τ2 + (c6 + c7t)τ3

Find parameters . . .

. . . by minimizing scores: ❼ CRPS Gneiting et al. [2005]

details

❼ ignorance score

aPasternack et al. [2017]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 27

A model for α, β, and γ

a first go a: α(t, τ) = (a0 + a1t) + (a2 + a3t)τ + (a4 + a5t)τ2 + (a6 + a7t)τ3 β(t, τ) = (b0 + b1t) + (b2 + b3t)τ + (b4 + b5t)τ2 + (b6 + b7t)τ3 γ(t, τ) = (c0 + c1t) + (c2 + c3t)τ + (c4 + c5t)τ2 + (c6 + c7t)τ3

Find parameters . . .

. . . by minimizing scores: ❼ CRPS Gneiting et al. [2005]

details

❼ ignorance score

aPasternack et al. [2017]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 27

Re-calibration for MiKlip

tas, global mean, full-field

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 28

Re-calibration for MiKlip

tas, global mean, full-field

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 28

Re-calibration for MiKlip – Verification

• 0.02

0.04 0.06 0.08 0.1 0.2 0.4 0.6 0.8 1 1.5 2 1 2 3 4 5 6 7 8 9 10 lead year CRPS

• Raw model

Drift corrected model Re−calibrated model

CRPS

cross validation

• 1e−05

1e−04 0.001 0.002 0.004 0.01 0.02 0.04 0.1 0.5 1 2 1 2 3 4 5 6 7 8 9 10 lead year Reliability comp. of CRPS

• Raw model

Drift corrected model Re−calibrated model

Reliability

cross validation

• 0.05

0.1 0.15 0.2 0.25 0.3 1 2 3 4 5 6 7 8 9 10 lead year IQR

• Raw model

Drift corrected model Re−calibrated model

Sharpness

cross validation

• −1

−0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1 1 2 3 4 5 6 7 8 9 10 lead year CRPSS

• Raw model

Drift corrected model Re−calibrated model

CRPSS (ref. Obs.)

cross validation

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 29

Re-calibration for MiKlip – Verification

Re-calibrating the historical simulations (reference):

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 30

Re-calibration for MiKlip – Verification

Re-calibrating the historical simulations (reference):

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 30

Verifying grid-cells with CRPSS

mean surface temperature, lead-year 7-10, vs historical ICPO

• param. drift correction

plus cond. bias plus ensemble spread αICPO(t, τ) α(t, τ) β(t, τ) γ(t, τ)

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 31

Verifying grid-cells with CRPSS

mean surface temperature, lead-year 7-10, vs historical ICPO

• param. drift correction

plus cond. bias plus ensemble spread αICPO(t, τ) α(t, τ) β(t, τ) γ(t, τ)

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 31

Verifying grid-cells with CRPSS

mean surface temperature, lead-year 7-10, vs historical ICPO

• param. drift correction

plus cond. bias plus ensemble spread αICPO(t, τ) α(t, τ) β(t, τ) γ(t, τ)

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 31

Verifying grid-cells with CRPSS

mean surface temperature, lead-year 7-10, vs historical ICPO

• param. drift correction

plus cond. bias plus ensemble spread αICPO(t, τ) α(t, τ) β(t, τ) γ(t, τ)

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 31

Summary Verification of decadal predictions

❼ framework Goddard et al. [2013] ❼ ensemble mean accuracy: MSESS ❼ ensemble spread: CRPS based σ2

ens vs MSE

❼ consider: LESS ❼ (multi-)annual averages ❼ score corrections for small ensembles ❼ drift issue

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 32

Summary

x x x x x x x x x x −0.2 0.0 0.2 bias(τ) 1 2 3 4 5 6 7 8 9 10

Drift adjustment

❼ b = b(τ, X(t)) ❼ drift: full-field > anomaly initialisation ❼ parametric post-processing helps ❼ drift depending on climate not just on time Fuˇ ckar et al. [2014]

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 33

Summary Re-calibration

❼ f Cal

i

= α(t, τ) + β(t, τ)μ(t, τ) + γ(t, τ)εi(t, τ) ❼ parametrics form for α, β, γ ❼ minimize CRPS/IGN to estimate parameters ❼ improves calibration, does not reduce sharpness (leave-10-yrs out cross validation)

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 34

Open issues

❼ grid-cell wise ’climate trend’ estimation ❼ model selection ❼ parameter uncertainty

,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 35

References

• G. J. Boer et al. The decadal climate prediction

project (DCPP) contribution to CMIP6. Geosci. Model Dev., 9:3751–77, 2016.

• C. A. T. Ferro et al. On the effect of ensemble size on

the discrete and continuous ranked probability

• scores. Meteor. Appl., 15:19 – 24, 2008.
• N. S. Fuˇ

ckar et al. A posteriori adjustment of near-term climate predictions: Accounting for the drift dependence on the initial conditions.

• Geophys. Res. Lett., 41(14):5200–5207, 2014.
• R. Gangstø et al. Methodological aspects of the

validation of decadal predictions. Clim. Res., 55: 181–200, 2013.

• T. Gneiting and A. E. Raftery. Strictly proper scoring

rules, prediction, and estimation. J. Amer. Statist. Assoc., 102(477):359–378, 2007.

• T. Gneiting et al. Calibrated probabilistic forecasting

using ensemble model output statistics and minimum crps estimation. Month. Weather Rev., 133:1098–1118, 2005.

• L. Goddard et al. A verification framework for

interannual-to-decadal predictions experiments. Climate Dynamics, 40:245–272, 2013.

• C. Kadow et al. Evaluation of forecasts by accuracy

and spread in the miklip decadal climate prediction system. Met. Z., 01 2014.

• V. V. Kharin et al. Statistical adjustment of decadal

predictions in a changing climate. Geophys. Res. Lett., 39:L19705, 2012.

• T. Kruschke et al. Probabilistic evaluation of decadal

predictions for northern hemisphere winter

• storms. Meteorol. Z., 2015.
• G. A. Meehl et al. Decadal climate prediction: An

update from the trenches. Bull. Amer. Meteorol. Soc., 95(2):243–267, 2014.

• W. A. Müller et al. A debiased ranked probability skill

score to evaluate probabilistic ensemble forecasts with small ensemble sizes. J. Clim., 18(10): 1513–1523, 2005.

• A. Pasternack et al. Decadal forecast calibration – a

parametric strategy accounting for drift, conditional bias and ensemble spread. in preparation, 2017.

• M. Pattantyús-Ábrahám et al. Bias and drift of the

medium-range decadal climate prediction system (MiKlip) validated by european radiosonde data. Meteorologische Zeitschrift, pages 709–720, 2016.

• D. M. Smith, R. Eade, and H. Pohlmann. A comparison
• f full-field and anomaly initialization for seasonal

to decadal climate prediction. Climate Dynamics, 41(11):3325–3338, 2013.

• A. P

. Weigel, M. A. Liniger, and C. Appenzeller. Can multi-model combination really enhance the prediction skill of probabilistic ensemble forecasts? Quart. J. Royal Meteor. Soc., 134(630): 241–260, 2008. ,

• H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 36