User-oriented evaluatjon of fjre spread predictjons Beth Ebert 1 , - - PowerPoint PPT Presentation

User-oriented evaluatjon of fjre spread predictjons

Beth Ebert1, Nathan Faggian2, Paul Fox-Hughes1, Chris Bridge2, Howard Jacobs2, Catherine Jolley2, Barb Brown3, Stuart Matuhews4, Greg Esnouf5

1 Research and Development Branch, Bureau of Meteorology, Australia 2 Weather Forecastjng Branch, Bureau of Meteorology, Australia 3 Research Applicatjons Laboratory, Natjonal Center for Atmospheric Research, USA 4 New South Wales Rural Fire Service, Australia 5 Australian Fire and Emergency Services Authoritjes Council, Australia

2

What is a fjre spread simulator?

Tool that models fjre characteristjcs (spatjally):

• Flame height
• Intensity
• Rate of spread
• Area of impact

A simulator is a collectjon of fjre behavior models that can be used to infer the fjre danger. Weather forecasts are one of the principal drivers of the simulators.

3

T=0 T=6h

4 Adapted from: Fire Behaviour Knowledge in Australia, Cruz et, al. 2014, Bushfjre CRC, Technical Report: EP145697

Cell based fjre spread models ‐ Geometric fjre spread models

Fire spread simulators in Australia

Which is best?

Bureau of Meteorology asked to run and evaluate these fjre spread simulators for a set of common cases from around Australia

5

Australis Phoenix Prometheus Spark

User focus of the evaluatjon

6

Consultatjon with end users (fjre agencies)

• Kick-ofg workshop, site visits,

consultatjons with simulator developers and fjre behavior analysts

• Understand how they use fjre spread

simulators

• Understand what "good quality" means

to them

Verifjcatjon planning template

7

What do users want to know?

• Management-level users:

– Which simulator is best? – Best for a partjcular case study?

• Fire behavior analysts (expert users):

– How accurately does this simulator predict fjre area, rate of spread, bearing? – How sensitjve is this simulator to variatjons in weather, fuel, ignitjon locatjon/tjme?

• Simulator developers:

– How can the uncertainty in weather inputs be quantjfjed to assist in the discriminatjon between model errors and input errors?

Data

10 case studies for this project:

• Fire boundaries (isochrones) from line

scans or reconstructjons

– Limited as agencies focus on protectjon

• f life and property

– Prefer cases without suppression

• Weather

– Offjcial weather forecast grids – Weather statjon observatjons

• Fuel layers from agencies
• Topography

9

Sample simulatjons

10

State Mine fjre, New South Wales, 16 October 2013

Spatjal verifjcatjon metrics

Summary metric

• Threat score

Diagnostjc metrics

• Bearing error
• Forward spread error
• Area error

11

Evaluatjon approach

For each simulator and all case studies:

• Baseline performance

– Simulate fjre spread using forecast weather in ignitjon grid cell(s)

• Sensitjvity studies

– Perturb input weather – Perturb fuel, ignitjon locatjon

• Relatjve and absolute performance

12 betuer poorer Gleckler et al. JGR 2008 CMIP3

Estjmatjng uncertainty in weather inputs

For each case:

• Verify 1-day weather forecasts at fjre

locatjon against observatjons averaged over three "nearest" AWS

• Bin each hour for all

days of month in which fjre occurred

• Use error PDF as template for

perturbing weather inputs

T RH

14

• bs

fcst

X X 

High level view - relatjve performance

• Management-level users

want to know: – Which simulator is best? – Best for a partjcular case study?

• Compare aggregate accuracy
• ver all perturbed inputs to

the whole populatjon (overall or for each case)

15

Dashboard

Deeper view – accuracy & sensitjvity

• Fire behaviour analysts (expert

users) want to know: – How accurately does this simulator predict fjre area, rate of spread, bearing? – How sensitjve is this simulator to variatjons in weather, fuel, ignitjon locatjon/tjme?

• Box size shows sensitjvity (how

does IQR compare to all IQRs?)

16 State Mine fjre, NSW, 16 October 2013

Modifjed Hinton diagram

Deeper view – accuracy & sensitjvity

• Fire behaviour analyst (expert

users) want to know: – How accurately does this simulator predict fjre area, rate of spread, bearing? – How sensitjve is this simulator to variatjons in weather, fuel, ignitjon locatjon/tjme?

• Pink = below median,

Green = above median

17

Categorical performance diagram

State Mine fjre, NSW, 16 October 2013 Constant TS

What did we learn?

• No single simulator stood out overall as being superior to the others and none

performed well in all circumstances. All simulators over-predicted some fjres and under-predicted others.

• Simulators (and fjres) are sensitjve to weather, partjcularly wind. This highlights the

value of an ensemble approach to the operatjonal use of fjre spread simulators.

• This evaluatjon framework will be a community tool for evaluatjng fjre spread

simulators, and has already prompted the community to make signifjcant improvements to their simulators.

• Need more cases, and standards for observing and reportjng fjre behavior.

18