Evaluatjon of non-standard variables Barbara Brown (bgb@ucar.edu) - - PowerPoint PPT Presentation

Evaluatjon of “non-standard” variables

Barbara Brown (bgb@ucar.edu) Natjonal Center for Atmospheric Research (NCAR) Boulder, Colorado, USA Verifjcatjon Methods Tutorial Berlin, Germany

What makes variables “non-standard”

• Not focused on commonly measured weather variables

(e.g., T, Td, Wind speed, u, v, etc.)

• ???
• Perhaps…

– Not observed well or require special observatjons – Forecasts of things that are diffjcult to measure – Predictjons directly serve specifjc users

• Partjcular events are forecasted for partjcular decision-making

situatjons (e.g., C&V for determining if planes can land)

• The stakes can be high! (i.e., the decisions can have major safety

and/or economic impacts)

Topics

• Tropical cyclones
• Wildfjres and fjre weather
• Sea ice
• Aviatjon
• Resources

TROPICAL CYCLONE FORECAST VERIFICATION

TC Inigo 2003 TC Gillian 2013

What makes TC forecast verifjcatjon “special”?

• High impact weather and High

impact weather Forecasts

– TC weather impacts afgect large populatjons and have major economic impacts – TC weather forecasts impact disaster management decisions

• TC forecasts are given intense

atuentjon by the media and public – in the public “eye”, so to speak

• Observatjons are generally

inferred and limited

What is not difgerent?

• Informatjon is needed by

Managers, Forecasters, Model developers and End users

• Basic verifjcatjon methods are

applicable, in general (i.e., contjnuous, categorical, probabilistjc)

What atuributes of TC forecasts?

Deterministjc

• TC track

– Overall error – Cross-track error – Along-track error – Landfall tjming and locatjon

• Intensity

– maximum wind – central pressure – temporal trend (rapid intensifjcatjon)

• Wind fjeld
• Size / radii
• Precipitatjon
• Temporal consistency
• Storm surge
• Waves

What Atuributes of TC forecasts?

Ensemble

• Track distribution
• Strike probability
• Intensity distribution

– mean / median – spread – 90th percentile

• Prob (wind > threshold)
• Prob (precip > threshold
• Storm surge
• Landfall timing

What verifjcatjon methods are appropriate?

Since we are evaluatjng a variety of variables and atuributes... A variety of methods are used

• Categorical

Rapid intensifjcatjon / weakening

• Contjnuous

Intensity, track locatjon, wind, size, winds, precipitatjon, ...

• Probabilistjc / ensemble

Track and intensity, locatjon ellipses, exceedance probabilitjes, precipitatjon, winds, size, strike probability, ...

• Spatjal

Wind structure, precipitatjon, ...

What about observatjons?

Many hurricane observatjons are inferred...

As usual there is no such thing as “truth” – but maybe more so for tropical cyclones than other phenomena

Track and intensity

• Identjfjed in “Best track” - Subjectjve analysis

–Track: Latjtude, longitude –Intensity: Minimum sea level pressure, maximum 1-min surface wind speed

• Best track is an analysis of all of the latest informatjon about a storm in post-analysis

–Uses satellite and reconnaissance informatjon –Smoothed version of the track –Intensity ofuen subjectjvely inferred from fmight level winds or satellite informatjon (Dvorak technique)

Precipitatjon and wind fjelds

Over oceans limited to satellite based informatjon + data from reconnaissance

Forecast characteristjcs

• Forecast types

– Human-generated tracks and intensity – NWP Models: Cyclone tracks are analyzed from gridded model output using a “tracker” algorithm – Statjstjcal models: Especially useful for predictjng intensity

• Model interpolatjon

– Needed to adjust “late” models with current track informatjon

• Reference forecasts

– Statjstjcal forecast or climate/persistence

Quality of deterministjc TC Track forecasts

Example questjons:

• What are the track errors (along-track, cross-

track)?

• What are the intensity errors?
• Are temporal intensity trends correctly predicted?
• What is the error in tjming of landfall?
• What is the error in forecast maximum wind (rain)?

– Multj-day total precipitatjon

• Is the spatjal distributjon of wind (rain) correct?

Others?

Total, Along-, and Cross-Track Errors

Track error Cross-Track Error (Forecast too far to the right) Along-Track Error (Forecast too slow)

Actual Track

Forecast Actual Track

Courtesy, J. Franklin

Cross-track measures error in direction of movement Along-track measures error in speed of movement

T rack error is typically summarized as Average error (always positive)

Track error summary

Verifjcatjon methods for deterministjc TC forecasts

• Example: Along-track and cross-track errors

Courtesy James Franklin, NHC

• “Along-track” measures errors in “Speed”
• “Cross-track” measures errors in “Direction”

Intensity error is typically summarized as (1)mean error (bias) or (2) mean absolute error (always positive)

Intensity error

Alternative:

Examine distributions

• f errors

And difgerences in errors

Paired comparisons: Track and intensity % improvement and p-value

Target forecasts signifjcantly improve on standard of comparison for intensity forecasts

Paired comparisons: Track and intensity % improvement and p-value

Target forecasts signifjcantly reduce performance relative to standard of comparison for track forecasts and some intensity forecasts

Using a fmexible defjnitjon of Rapid Intensifjcatjon / Rapid Weakening events Standard Defjnitjon: NHC Defjnitjon 30m/s in 24 hours Stricter Defjnitjon: 30m/s in 12 hours “Fuzzy” Defjnitjon: Adjustable window to give credit even if there is a tjming error Categorical statjstjcs for RI/RW events can then be calculated: POD, FAR, CSI, etc.

Miss: Events in Fcst and Observed Track do not occur at same tjme Hit: Events in Fcst and Observed Track both fall into an 18 hour window

Evaluating features: Rapid intensifjcation and weakening (RI/RW)

Evaluatjng features: TC precipitatjon evaluatjon Storm-following masking with range rings

Accumulated storm precipitation distributions for Model, Satellite, and Radar by range ring Shifted forecast precipitation to account for track error, with range rings around the best track

WILDFIRES AND FIRE WEATHER

Fire weather verifjcatjon

• Wildfjre conditjons and

associated weather can be predicted by humans, spread simulators, or coupled weather-fjre models

• Variables of interest:

– Fire perimeter – Fire rate-of-spread – Underlying wind and other weather variables – Signifjcant fjre behavior (fmame length, pyrocumulus, etc.) Many complications with evaluation of these variables

Meetjng the users’ needs

• Focus on process to identjfy and

document stakeholders’ goals

• Difgerent users have difgerent

needs

– Management (Which model/simulator is best?) – Fire behavior analysts (How accurate are fjre predictjons?) – Simulator / Model developers (quantjfy uncertainty in weather inputs to identjfy simulator improvements needed)

Australia BOM process

Observatjon issues

• Fire perimeter

– Observed from the air? – Satellite? – Obs are infrequent at best…

• Only rare observatjons of

signifjcant phenomena (fmame height, heat release, pyrocumulus, etc.)

• Weather observatjons very

limited…

– Poor coverage in complex terrain

Verifjcatjon approaches

• Spatjal methods

– MODE? CRA?

• Contjngency table

statjstjcs (TS, Bias)

• Area measures

Issue: What about the impact of fjre suppression efgorts?

From Ebert presentation Monday

SEA ICE

The Challenges

• Arctjc sea ice is changing

dramatjcally and quickly

• Climate, seasonal, and other models

depend on good estjmates of sea ice extent – and other characteristjcs

• Many users interested in impacts of

changes in ice (shipping, mining, etc.)

• Observatjons are limited…

– Mainly satellite-based – Ice extent is best observed; other propertjes (thickness, concentratjon) more limited

Possible verifjcatjon methods

• Spatjal

– MODE – CRA – Image warping

• Distance metrics:

– Baddeley, Hausdorfg (see methods in R package) – See references by Gilleland and

• thers at

htups://ral.ucar.edu/projects/icp/

From Arbetter 2012

AVIATION WEATHER

Issues

• Main issue: Observatjons!!

– Limited in space and tjme – Biased in space and tjme, and by event (e.g., around airports, on fmight routes; where weather is good!)

Example: Icing PIREPs

From Brown et al. 1997 (Weather and Forecasting) Notable biases in location, time, intensity Potentially systematic in areas near airports?

EDR (turbulence) example: Automated

• bservatjons

Spatjal biases and highly skewed distributjon

• Diffjcult to tune

forecasts to predict “positjve” events

• Turbulence forecasts

may not be representatjve of areas where planes don’t fmy

From Sharman et al. 2014 (J. Appl. Climate and Weather)

TAIWIN: Terminal Area Icing Weather Informatjon for NextGen (TAIWIN)

• Goal: Improve NWP

forecasts of precip type (especially freezing rain/drizzle) to predict super-cooled liquid

• First step: Identjfy

appropriate

• bservatjons

– METARs – Radar/Satellite – Crowd-sourced (MPING)

None Rain Frz Rain Drizzle Snow

Courtesy J. Wolf

Perfect score

Overforecast Underforecast

Bias

C r i t i c a l S u c c e s s I n d e x

Rain Snow

Frz Rn

Ice pellets Credit: J. Wolff, NCAR

Observatjon implicatjons…

• Observatjon characteristjcs ofuen limit the kinds
• f verifjcatjon that can be done

– Ex: In-fmight icing, turbulence; gridded C&V

• Observatjon characteristjcs can bias the

verifjcatjon results

• Improvement of aviatjon weather observatjons

would greatly help improve development and evaluatjon of aviatjon weather forecasts

SUMMARY

Summary

• Standard verifjcatjon methods apply to most

variables

But: – Focus is needed on what aspects users care about

• Good news: Typically easier to understand who the users are

– Care needed to understand implicatjons of observatjon biases

• May limit what verifjcatjon approaches are reasonable to apply
• Observatjon issues – availability, uncertainty – are

even more important than for more standard variables!

RESOURCES

WMO Working Group on Forecast Verifjcatjon Research

Many resources on the WMO website:

htups:// www.wmo.int/pages/prog/arep/wwrp/ne w/Forecast_Verifjcatjon.html

• Guidance documents
• Links to past tutorials
• Informatjon about upcoming

meetjngs, etc.

Resources: Verifjcatjon methods and FAQ

• Website maintained by

WMO verifjcatjon working group (JWGFVR)

• Includes

– Issues – Methods (brief defjnitjons) – FAQs – Links and references

• Verifjcatjon discussion

group:

http://mail.rap.ucar.edu/ma ilman/listinfo/vx-discuss

http://www.cawcr.gov.au/projects/verifjcation/

Resources: Overview papers

• Casatj et al. 2008: Forecast verifjcatjon:

current status and future directjons.

Meteorological Applicatjons, 15: 3-18.

• Ebert et al. 2013: Progress and challenges in

forecast verifjcatjon

Meteorological Applicatjons, 20, 130-139.

Informatjon about spatjal methods

MesoVICT website:

• Includes

references and some software

• New results

will be provided as available

http://www.ral.ucar.edu/projects/icp/

Resources - Books

• Jollifge and Stephenson (2012):

Forecast Verifjcatjon: a practjtjoner’s guide, Wiley & Sons, 240 pp.

• Stanski, Burrows, Wilson (1989) Survey
• f Common Verifjcatjon Methods in

Meteorology (available at htup://www.cawcr.gov.au/projects/veri fjcatjon/)

• Wilks (2011): Statjstjcal Methods in

Atmospheric Science, Academic press. (Updated chapter on Forecast Verifjcatjon)

Sofuware: R verifjcatjon packages

• R is a fmexible, open source

statjstjcal computjng package

– Available from htups://www.r- project.org/ – Works on all platgorms

• “Verifjcatjon” and

“SpatjalVx” packages are available from the contributed packages list on the “CRAN” website: htup://cran.repo.bppt.go.id/ in Indonesia

• “Verifjcatjon” package

– Includes all standard verifjcatjon metrics for

• Contjngency tables
• Contjnuous variables
• Probability forecasts
• Ensemble forecasts

– Many graphical tools (e.g., aturibute diagrams)

• “SpatjalVx” package

– Includes many of the new spatjal verifjcatjon methods – New methods added as available

Sofuware: Model Evaluatjon Tools (MET)

• MET is a freely available

sofuware package

• Supported to the

community and well- documented

• Highly confjgurable and

fmexible

• Tutorials (on-line and in

person) are available

• Includes

– Traditjonal methods (contjngency table, contjnuous, probabilistjc) – Ensemble approaches – Spatjal methods – Package for Tropical Cyclones (MET-TC)

MET is available at: www.dtcenter.org/met/users