Using the SAL technique for ensemble forecast verification Sabine - - PowerPoint PPT Presentation

▶
using the sal technique for ensemble forecast verification
SMART_READER_LITE
LIVE PREVIEW

Using the SAL technique for ensemble forecast verification Sabine - - PowerPoint PPT Presentation

Jan 30, 2024 144 likes •445 views

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions Using the SAL technique for ensemble forecast verification Sabine Radanovics Until March 2017: LSCE/ESTIMR, now: CNRM/GMME/MICADO May 10, 2017 1 / 19 Why

slide-1
SLIDE 1

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

Using the SAL technique for ensemble forecast verification

Sabine Radanovics

Until March 2017: LSCE/ESTIMR, now: CNRM/GMME/MICADO

May 10, 2017

1 / 19

slide-2
SLIDE 2

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

Why was spatial verification invented?

2 / 19

slide-3
SLIDE 3

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

Why was spatial verification invented?

2 / 19

Forecaster Paul

COSMO2 VERA CLEPS 44 46 48 50 2007062100 5 10 15 5 10 15 5 10 15 10 20 30 40 prec

slide-4
SLIDE 4

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

Why was spatial verification invented?

2 / 19

Forecaster Paul

COSMO2 VERA CLEPS 44 46 48 50 2007062100 5 10 15 5 10 15 5 10 15 10 20 30 40 prec

slide-5
SLIDE 5

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

Why was spatial verification invented?

3 / 19

Model developer Cate

COSMO2 VERA CLEPS 44 46 48 50 2007062100 5 10 15 5 10 15 5 10 15 10 20 30 40 prec

slide-6
SLIDE 6

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

Why was spatial verification invented?

3 / 19

Model developer Cate

COSMO2 VERA CLEPS 44 46 48 50 2007062100 5 10 15 5 10 15 5 10 15 10 20 30 40 prec

RMSE = 452.02 RMSE = 352.03

slide-7
SLIDE 7

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

Why was spatial verification invented?

3 / 19

Model developer Cate

COSMO2 VERA CLEPS 44 46 48 50 2007062100 5 10 15 5 10 15 5 10 15 10 20 30 40 prec

RMSE = 452.02 RMSE = 352.03 → Need to develop verification that gives credit to the attributes Paul appreciates

slide-8
SLIDE 8

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

SAL - Structure, Amplitude, Location

Amplitude component A

4 / 19

(Wernli et al., 2008)

  • bs

fc 4 8 12 16 4 8 12 16 4 8 12 16

x y

1 2 3 4 5 6 precip

S = 0 A = 0.667 L = 0

Doubled precipitation → relative Amplitude error A

slide-9
SLIDE 9

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

SAL - Structure, Amplitude, Location

Location component L

5 / 19

  • bs

fc 4 8 12 16 4 8 12 16 4 8 12 16

x y

1 2 3 precip

S = 0 A = 0 L = 0.442

Shifted feature → location error of center of mass relative to domain size L

slide-10
SLIDE 10

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

SAL - Structure, Amplitude, Location

Structure component S

6 / 19

  • bs

fc 4 8 12 16 4 8 12 16 4 8 12 16

x y

1 2 3 precip

S = 0.4 A = 0 L = 0

Flatten feature → structure error in terms of scaled feature volumes S

slide-11
SLIDE 11

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

SAL verification

7 / 19

COSMO2 VERA CLEPS

  • 44

46 48 50 2007062100 5 10 15 5 10 15 5 10 15 10 20 30 40 prec

RMSE = 452.02 S = -0.32 A = -0.22 L = 0.11 RMSE = 352.03 S = 0.55 A = -0.69 L = 0.38

slide-12
SLIDE 12

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

Questions

  • 1. Ensemble verification with SAL?
  • 2. What is the effect of observation uncertainty?

8 / 19

slide-13
SLIDE 13

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

Ensemble verification with SAL?

9 / 19

slide-14
SLIDE 14

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

Ensemble verification with SAL?

9 / 19

Option 1

Calculate the scores for every ensemble member combination → study spread of scores

slide-15
SLIDE 15

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

Ensemble verification with SAL?

9 / 19

Option 1

Calculate the scores for every ensemble member combination → study spread of scores

Option 2

Calculate one score for the whole ensemble → allows comparison with deterministic model

slide-16
SLIDE 16

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

Ensemble SAL

Standard SAL Ensemble SAL A =

rrmod−rrobs 0.5[rrmod+rrobs]

A =

rrmod−rrobs 0.5[rrmod+rrobs]

L1 = |x(rrmod)−x(rrobs)|

d

L1 = |x(rrmod)−x(rrobs)|

d

L2 = 2

  • |r(mod)

d

− r(obs)

d

|

  • L2 = 2 × CRPS
  • P

r(mod)

d

  • , P

r(obs)

d

  • r =
  • i rri|xi−x|
  • i rri

S =

V (mod)−V (obs) 0.5[V (mod)+V (obs)]

S =

V (mod)−V (obs) 0.5[V (mod)+V (obs)]

V =

  • i rri

rri rrmax i

  • i rri

rr domain average precipitation. x center of mass of the precipitation field d largest distance between two domain borders. xi the center of mass of the ith feature rri is the sum of precipitation over all grid cells in feature i rr max

i

maximum precipitation of the ith feature.

10 / 19

slide-17
SLIDE 17

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

MesoVICT project core case 20-22 June 2007

(Mesoscale Verification In Complex Terrain)

11 / 19

3-hourly precipitation Ensembles CLEPS ensemble forecast (16 members) (Marsigli et al., 2005) VERA analysis ensemble (50 members) [VERAens]

(Gorgas and Dorninger, 2012)

Deterministic COSMO2 forecasts

(Ament and Arpagaus, 2009)

VERA analysis

(Steinacker et al., 2000)

slide-18
SLIDE 18

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

MesoVICT project core case 20-22 June 2007

(Mesoscale Verification In Complex Terrain)

11 / 19

3-hourly precipitation Ensembles CLEPS ensemble forecast (16 members) (Marsigli et al., 2005) VERA analysis ensemble (50 members) [VERAens]

(Gorgas and Dorninger, 2012)

Deterministic COSMO2 forecasts

(Ament and Arpagaus, 2009)

VERA analysis

(Steinacker et al., 2000)

slide-19
SLIDE 19

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

CLEPS-VERAens 16×50 SAL scores

3-hourly precipitation, 20-22 June 2007 3h-24h lead times (depending on the hour of the day)

S A L −1 1 −1 1 0.00 0.25 0.50 0.75 1.00 Jun 20 00:00 Jun 20 12:00 Jun 21 00:00 Jun 21 12:00 Jun 22 00:00 Jun 22 12:00 Jun 23 00:00 Jun 20 00:00 Jun 20 12:00 Jun 21 00:00 Jun 21 12:00 Jun 22 00:00 Jun 22 12:00 Jun 23 00:00 Jun 20 00:00 Jun 20 12:00 Jun 21 00:00 Jun 21 12:00 Jun 22 00:00 Jun 22 12:00 Jun 23 00:00

large spreads in scores, especially S

12 / 19

slide-20
SLIDE 20

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

CLEPS-VERAens Ensemble SAL vs. median of 800 x standard SAL

S A L −1.0 −0.5 0.0 0.5 1 0.0 0.2 0.4 0.6 Jun 20 00:00 Jun 20 12:00 Jun 21 00:00 Jun 21 12:00 Jun 22 00:00 Jun 22 12:00 Jun 23 00:00 Jun 20 00:00 Jun 20 12:00 Jun 21 00:00 Jun 21 12:00 Jun 22 00:00 Jun 22 12:00 Jun 23 00:00 Jun 20 00:00 Jun 20 12:00 Jun 21 00:00 Jun 21 12:00 Jun 22 00:00 Jun 22 12:00 Jun 23 00:00

sal

Ensemble SAL Median of standard SAL

Similar for 2-sided scores A and S Ensemble L tends to be smaller than median of standard L

13 / 19

slide-21
SLIDE 21

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

Compare models - SAL diagrams

14 / 19

VERA VERA ensemble

  • −2

−1 1 2 −2 −1 1 2 COSMO2 CLEPS −2 −1 1 2 −2 −1 1 2

S A

0.1 0.2 0.3 0.4 0.5

L

slide-22
SLIDE 22

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

Compare models - SAL diagrams

14 / 19

VERA VERA ensemble

  • −2

−1 1 2 −2 −1 1 2 COSMO2 CLEPS −2 −1 1 2 −2 −1 1 2

S A

0.1 0.2 0.3 0.4 0.5

L

difference between VERA and VERAens bigger than difference between forecasts

slide-23
SLIDE 23

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

Compare models - SAL diagrams

14 / 19

VERA VERA ensemble

  • −2

−1 1 2 −2 −1 1 2 COSMO2 CLEPS −2 −1 1 2 −2 −1 1 2

S A

0.1 0.2 0.3 0.4 0.5

L

difference between VERA and VERAens bigger than difference between forecasts

Why?

slide-24
SLIDE 24

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

Compare models - SAL diagrams

14 / 19

VERA VERA ensemble

  • −2

−1 1 2 −2 −1 1 2 COSMO2 CLEPS −2 −1 1 2 −2 −1 1 2

S A

0.1 0.2 0.3 0.4 0.5

L

difference between VERA and VERAens bigger than difference between forecasts

Why?

Forecasts very similar due to short lead times VERAens very different from VERA

slide-25
SLIDE 25

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

VERAens vs. VERA

  • hourly

3−hourly −2 −1 1 2 −2 −1 1 2 −2 −1 1 2

S A

0.1 0.2 0.3 0.4

L

Total precipitation A systematically higher in VERAens S spans the whole range of values, majority negative S for hourly large values of L, same order of magnitude as forecasts

15 / 19

slide-26
SLIDE 26

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

What is known for VERAens

Gorgas and Dorninger (2012) found that

  • 1. the quality control procedure to remove unrealistic values like

from the perturbed analysis fields tends to remove more of the negative perturbations than from the positive ones → high bias in the VERA ensemble.

  • 2. the standard deviations are on average increased due to the

perturbations, but this varies strongly with the ensemble member

Increased standard deviation → smaller/more peaked objects → negative S dominates. What if the standard deviation due to the perturbations does not only vary with the ensemble member but also over different time steps? This would partly explain the wide range

  • f S values.

16 / 19

slide-27
SLIDE 27

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions

Conclusions

The behaviour of the ensemble SAL seems comparable to the standard SAL + less computation than 800 standard SAL – loose information on spread SAL is very sensitive to small differences in the analysis (or the forecasts) (see also Weniger and Friederichs (2016)) → may be useful to highlight subtle differences and as a rather qualitative diagnostic in model development → probably less useful for ranking models There might be an issue with a too large verification domain for L and S. → domain size sensitivity experiments needed, in particular with complex terrain

17 / 19

slide-28
SLIDE 28

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions 18 / 19

Thank you!

slide-29
SLIDE 29

References I

Ament, F. and Arpagaus, M. (2009). dphase cosmoch2: Cosmo model forecasts (2.2 km) run by meteoswiss for the map d-phase project. Gorgas, T. and Dorninger, M. (2012). Concepts for a pattern-oriented analysis ensemble based on observational uncertainties. Quarterly Journal of the Royal Meteorological Society, 138(664):769–784. Marsigli, C., Boccanera, F., Montani, A., and Paccagnella, T. (2005). The cosmo-leps mesoscale ensemble system: validation of the methodology and verification. Nonlinear Processes in Geophysics, 12(4):527–536. Steinacker, R., H¨ aberli, C., and P¨

  • ttschacher, W. (2000). A transparent method for

the analysis and quality evaluation of irregularly distributed and noisy observational

  • data. Monthly Weather Review, 128(7):2303–2316.

Weniger, M. and Friederichs, P. (2016). Using the sal technique for spatial verification

  • f cloud processes: A sensitivity analysis. Journal of Applied Meteorology and

Climatology, 55(9):2091–2108. Wernli, H., Paulat, M., Hagen, M., and Frei, C. (2008). SAL–a novel quality measure for the verification of quantitative precipitation forecasts. Monthly Weather Review, 136(11):4470–4487.

19 / 19