Model evaluation based on a large observation data set Jean-Baptiste - - PowerPoint PPT Presentation

▶
model evaluation based on a large observation data set
SMART_READER_LITE
LIVE PREVIEW

Model evaluation based on a large observation data set Jean-Baptiste - - PowerPoint PPT Presentation

Nov 09, 2023 251 likes •766 views

Model evaluation based on a large observation data set Jean-Baptiste Filippi 1 Vivien Mallet 2 , 3 Bahaa Nader 1 1 SPE CNRS 2 INRIA 3 CEREA, joint laboratory ENPC - EDF R&D, Universit Paris-Est Numerical simulation of forest fires, Cargse,

slide-1
SLIDE 1

Model evaluation based on a large observation data set

Jean-Baptiste Filippi1 Vivien Mallet2,3 Bahaa Nader1

1SPE CNRS 2INRIA 3CEREA, joint laboratory ENPC - EDF R&D, Université Paris-Est

Numerical simulation of forest fires, Cargèse, May 2013

Filippi, Mallet, Nader Model evaluation May 2013 1 / 50

slide-2
SLIDE 2

Topic

Evaluation of the performance of a propagation model

Questions

1 How to rank models? Can we identify the “best” model out of a pool

  • f models?

2 How to evaluate the dynamics of the model when the observation is

the final burned surface?

3 Can we evaluate a model regardless of the quality of its inputs? 4 Can we carry out probabilistic forecasts? Filippi, Mallet, Nader Model evaluation May 2013 2 / 50

slide-3
SLIDE 3

Notation

Observation

Observation of final burned surface at time to

f

Observed burned surface: So(to

f )

Simulation

Final simulation time: ts

f

Simulated burned surface at time t: S(t)

Area

|S| is the area of the surface S Ω is the simulation domain

Scores

Classical scores compare So(to

f ) and S(ts f )

Filippi, Mallet, Nader Model evaluation May 2013 3 / 50

slide-4
SLIDE 4

Classical scores

Sørensen similarity index

2|So(to

f ) ∩ S(ts f )|

|So(to

f )| + |S(ts f )| ∈ [0, 1]

Jaccard similarity coefficient

|So(to

f ) ∩ S(ts f )|

|So(to

f ) ∪ S(ts f )| ∈ [0, 1]

Kappa coefficient

Pa − Pe 1 − Pe ≤ 1 where Pa = |So(to

f ) ∩ S(ts f )|

|Ω| + |Ω\(So(to

f ) ∪ S(ts f ))|

|Ω| Pe = |So(to

f )||S(ts f )|

|Ω|2 + |Ω\So(to

f )||Ω\S(ts f )|

|Ω|2

Filippi, Mallet, Nader Model evaluation May 2013 4 / 50

slide-5
SLIDE 5

Dynamic-aware scores

Arrival time agreement

Addition notation

Arrival time: earliest time at which the front is known to have reached some point; +∞ if the front never reaches the point Observed arrival time at X: T o(X), say T o(X) = to

f if X ∈ So(to f )

Simulated arrival time at X: T (X), here, T (X) ≤ ts

f if X ∈ S(ts f )

Arrival time agreement

1 − 1 |S(ts

f ) ∪ So(to f )| max(ts f , to f )

  • S(ts

f )∩So(to f )

max(T (X) − T o(X), 0)dX +

  • S(ts

f )\So(to f )

max(to

f − T (X), 0)dX

+

  • So(to

f )\S(ts f )

(ts

f − T o(X))dX

  • ∈ [0, 1]

Filippi, Mallet, Nader Model evaluation May 2013 5 / 50

slide-6
SLIDE 6

Dynamic-aware scores

Shape agreement

Shape agreement

1 − 1 ts

f

  • ]0,to

f ]

|S(t)\So(to

f )|

|S(t)| dt +

  • [to

f ,ts f [

|So(to

f )\S(t)|

|So(to

f )|

dt

  • ∈ [0, 1]

Filippi, Mallet, Nader Model evaluation May 2013 6 / 50

slide-7
SLIDE 7

Application of the scores to an idealized case

200 400 600 800 200 400 600 800 200 400 600 800 1000 1200 1400 1600 1800 2000 200 400 600 800 200 400 600 800 200 400 600 800 1000 1200 1400 1600 1800 2000

S = 0.981 J = 0.962 K = 0.977 ATA = 0.985 SA = 0.973

200 400 600 800 200 400 600 800 250 500 750 1000 1250 1500 1750 2000 2250

S = 0.981 J = 0.962 K = 0.977 ATA = 0.998 SA = 0.997

J.-B. Filippi, V. Mallet, and B. Nader (2013). “Representation and evaluation of wildfire propagation simulations”. In: Under review for Int. J. Wild. Fire Scoring methods in Python at http://sf.net/projects/pyfirescore/

Filippi, Mallet, Nader Model evaluation May 2013 7 / 50

slide-8
SLIDE 8

Simulation of 80 fire cases

Fire cases

Using observations from Prométhée, database of french fires Available data for a subset of the fires: date, ignition point, final contour Unfortunately, no data on firefights Considering 80 Corsican fires from 2003 to 2008

Filippi, Mallet, Nader Model evaluation May 2013 8 / 50

slide-9
SLIDE 9

80 fire cases [1/5]

2000400060008000 10000 12000 14000 16000 2000 4000 6000 8000 10000 12000 14000

Aullene [990 ha]

500 1000 1500 2000 2500 3000 500 1000 1500 2000 2500 3000 3500

Oletta [54 ha]

10002000300040005000600070008000 1000 2000 3000 4000 5000 6000 7000

Olmi Cappella [193 ha]

0 2000 4000 6000 8000 10000 12000 14000 16000 18000 2000 4000 6000 8000 10000 12000 14000

Pietracorbara [1083 ha]

2000 4000 6000 800010000 12000 14000 5000 10000 15000 20000 25000

Santo Pietro di Tenda [1310 ha]

5000 10000 15000 20000 25000 5000 10000 15000 20000 25000

Calenzana [1419 ha]

200 400 600 800 1000 1200 1400 500 1000 1500 2000

Calenzana [18 ha]

200 400 600 800 1000 1200 1400 500 1000 1500 2000 2500 3000 3500

Solaro [29 ha]

500 1000 1500 2000 2500 3000 3500 500 1000 1500 2000 2500

Volpajola [50 ha]

1000 2000 3000 4000 5000 6000 1000 2000 3000 4000 5000 6000 7000

Murzo [180 ha]

1000 2000 3000 4000 5000 1000 2000 3000 4000 5000

Ghisonaccia [95 ha]

500 1000 1500 2000 2500 3000 3500 500 1000 1500 2000 2500 3000 3500 4000 4500

Vivario [57 ha]

500 1000150020002500300035004000 500 1000 1500 2000 2500

Corscia [58 ha]

2000 4000 6000 8000 10000 12000 2000 4000 6000 8000 10000 12000 14000 16000

Biguglia [783 ha]

100 200 300 400 500 600 700 800 900 100 200 300 400 500 600

Casaglione [2 ha]

500 1000 1500 2000 2500 3000 1000 2000 3000 4000 5000

Aleria [62 ha]

Filippi, Mallet, Nader Model evaluation May 2013 9 / 50

slide-10
SLIDE 10

80 fire cases [2/5]

10002000300040005000600070008000 2000 4000 6000 8000 10000 12000

Sisco [441 ha]

200 400 600 8001000 1200 1400 1600 1800 500 1000 1500 2000 2500 3000 3500

Linguizzetta [34 ha]

500 1000 1500 2000 2500 3000 1000 2000 3000 4000 5000 6000

Aleria [80 ha]

500 1000 1500 2000 500 1000 1500 2000 2500

Prunelli di Fiumorbo [26 ha]

500 1000 1500 2000 2500 3000 1000 2000 3000 4000 5000 6000

Olmeta di Tuda [81 ha]

500 1000150020002500300035004000 1000 2000 3000 4000 5000

Calenzana [76 ha]

500 1000 1500 2000 2500 3000 1000 2000 3000 4000 5000 6000 7000 8000

Nocario [82 ha]

500 1000 1500 2000 2500 500 1000 1500 2000 2500 3000

Ventiseri [29 ha]

100 200 300 400 500 600 200 400 600 800 1000

Porto Vecchio [3 ha]

1000 2000 3000 4000 5000 6000 7000 1000 2000 3000 4000 5000 6000 7000 8000 9000

Afa [242 ha]

500 1000 1500 2000 2500 3000 3500 500 1000 1500 2000 2500

Loreto di Casinca [42 ha]

500 1000150020002500300035004000 500 1000 1500 2000 2500 3000

Calenzana [59 ha]

200 400 600 800 1000 1200 1400 500 1000 1500 2000 2500

Corbara [10 ha]

2000 4000 6000 8000 10000 2000 4000 6000 8000 10000 12000

Vero [535 ha]

200 400 600 800 1000120014001600 500 1000 1500 2000 2500 3000 3500 4000 4500

Canale di Verde [35 ha]

500 1000 1500 2000 2500 3000 200 400 600 800 1000 1200 1400 1600 1800

Altiani [24 ha]

Filippi, Mallet, Nader Model evaluation May 2013 10 / 50

slide-11
SLIDE 11

80 fire cases [3/5]

200 400 600 8001000 1200 1400 1600 1800 200 400 600 800 1000 1200 1400

Patrimonio [13 ha]

1000 2000 3000 4000 5000 6000 7000 500 1000 1500 2000 2500 3000 3500 4000

Pruno [116 ha]

5001000 1500 2000 2500 3000 3500 4000 4500 500 1000 1500 2000 2500 3000 3500 4000

Olcani [70 ha]

50 100 150 200 250 300 50 100 150 200 250

Sartene [0 ha]

500 1000 1500 2000 2500 3000 200 400 600 800 1000 1200 1400

Calenzana [16 ha]

100 200 300 400 500 600 700 200 400 600 800 1000

Propriano [3 ha]

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 1000 2000 3000 4000 5000 6000

Canari [95 ha]

200 400 600 800 1000 1200 1400 200 400 600 800 1000 1200 1400 1600 1800

Olmeta di Tuda [18 ha]

1000 2000 3000 4000 5000 6000 1000 2000 3000 4000 5000 6000 7000 8000

Calenzana [91 ha]

500 1000 1500 2000 2500 3000 3500 1000 2000 3000 4000 5000 6000

Calenzana [103 ha]

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 500 1000 1500 2000 2500 3000

Calenzana [143 ha]

200 400 600 800 1000 100 200 300 400 500 600 700

Piana [2 ha]

200 400 600 8001000 1200 1400 1600 1800 200 400 600 800 1000 1200 1400 1600

Ajaccio [14 ha]

50 100 150 200 250 300 50 100 150 200 250 300 350

Bastelicaccia [0 ha]

40 60 80 100 120 140 160 180 200 220 50 100 150 200 250 300

Propriano [1 ha]

0 2000 4000 6000 8000 10000 12000 14000 16000 18000 2000 4000 6000 8000 10000 12000 14000 16000 18000

Oletta [1126 ha]

Filippi, Mallet, Nader Model evaluation May 2013 11 / 50

slide-12
SLIDE 12

80 fire cases [4/5]

500 1000 1500 2000 2500 3000 500 1000 1500 2000 2500 3000

Soveria [56 ha]

5001000 1500 2000 2500 3000 3500 4000 4500 2000 4000 6000 8000 10000 12000 14000

Sisco [382 ha]

1000 2000 3000 4000 5000 1000 2000 3000 4000 5000 6000 7000

Coti Chiavari [185 ha]

1000 2000 3000 4000 5000 6000 1000 2000 3000 4000 5000 6000 7000 8000

Lumio [201 ha]

500 1000 1500 2000 2500 3000 500 1000 1500 2000 2500 3000 3500 4000

Santa Maria Poggio [59 ha]

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 2000 4000 6000 8000 10000 12000

Barbaggio [518 ha]

500 1000 1500 2000 2500 3000 3500 500 1000 1500 2000 2500 3000 3500

Rutali [60 ha]

1000 2000 3000 4000 5000 1000 2000 3000 4000 5000 6000 7000

Oletta [186 ha]

50 100 150 200 250 300 50 100 150 200 250 300 350

Propriano [0 ha]

2000 4000 6000 8000 10000 12000 1000 2000 3000 4000 5000 6000 7000 8000

Calenzana [465 ha]

500 1000150020002500300035004000 1000 2000 3000 4000 5000 6000

Calvi [79 ha]

200 400 600 800 1000 1200 200 400 600 800 1000 1200 1400 1600

San Giovanni di Moriani [10 ha]

500 1000 1500 2000 2500 3000 1000 2000 3000 4000 5000

Luri [51 ha]

200 400 600 8001000 1200 1400 1600 1800 500 1000 1500 2000 2500

Saint Florent [25 ha]

2000400060008000 10000 12000 14000 16000 2000 4000 6000 8000 10000 12000 14000 16000

Peri [750 ha]

500 1000 1500 2000 2500 3000 500 1000 1500 2000 2500 3000 3500 4000

Poggio d'Oletta [53 ha]

Filippi, Mallet, Nader Model evaluation May 2013 12 / 50

slide-13
SLIDE 13

80 fire cases [5/5]

500 1000 1500 2000 500 1000 1500 2000 2500 3000 3500

Borgo [41 ha]

5001000 1500 2000 2500 3000 3500 4000 4500 2000 4000 6000 8000 10000 12000 14000 16000

Sisco [357 ha]

500 1000 1500 2000 2500 3000 500 1000 1500 2000

Calenzana [27 ha]

2000 4000 6000 8000 10000 12000 2000 4000 6000 8000 10000 12000 14000 16000

Bonifacio [477 ha]

100 200 300 400 500 600 700 800 100 200 300 400 500 600 700 800

Propriano [4 ha]

2000400060008000 10000 12000 14000 16000 2000 4000 6000 8000 10000 12000 14000 16000

Tolla [942 ha]

500 1000 1500 2000 2500 500 1000 1500 2000 2500

Santa Lucia di Mercurio [19 ha]

40 60 80 100 120 140 160 180 200 50 100 150 200 250

Alata [0 ha]

5001000 1500 2000 2500 3000 3500 4000 4500 500 1000 1500 2000 2500 3000 3500

Omessa [116 ha]

10002000300040005000600070008000 1000 2000 3000 4000 5000 6000 7000 8000 9000

Ersa [157 ha]

100 200 300 400 500 600 700 800 900 500 1000 1500 2000 2500

Manso [11 ha]

200 400 600 800 1000 500 1000 1500 2000 2500

Olmeta di Tuda [13 ha]

500 1000150020002500300035004000 1000 2000 3000 4000 5000

Santo Pietro di Tenda [111 ha]

500 1000 1500 2000 2500 3000 3500 500 1000 1500 2000 2500

Calenzana [28 ha]

200 400 600 8001000 1200 1400 1600 1800 200 400 600 800 1000 1200 1400

Bisinchi [14 ha]

500 1000 1500 2000 2500 500 1000 1500 2000 2500 3000 3500

Montegrosso [44 ha]

Filippi, Mallet, Nader Model evaluation May 2013 13 / 50

slide-14
SLIDE 14

Simulation of 80 fire cases

Simulations

Land use cover: “inventaire forestier national” (IFN, from IGN) and global land cover (GLC) Elevation at 25 m resolution from IGN Wind velocity and direction

Taken at Ajaccio or Bastia meteorological station, or from ECMWF simulations And then computed by Windninja (US Forest Service and Colorado State University)

Running ForeFire (SPE), until the final burned area is attained Four models for the rate of spread

Balbi model (cf. his talk, tomorrow), “stationary” Balbi model, “non-stationary”, i.e., with front depth and dependence

  • n the front geometry

Rothermel model Simple rate of spread equal to 3% of wind velocity

Filippi, Mallet, Nader Model evaluation May 2013 14 / 50

slide-15
SLIDE 15

80 fire cases [1/5]

2000400060008000 10000 12000 14000 16000 2000 4000 6000 8000 10000 12000 14000

Aullene [990 ha]

500 1000 1500 2000 2500 3000 500 1000 1500 2000 2500 3000 3500

Oletta [54 ha]

10002000300040005000600070008000 1000 2000 3000 4000 5000 6000 7000

Olmi Cappella [193 ha]

0 2000 4000 6000 8000 10000 12000 14000 16000 18000 2000 4000 6000 8000 10000 12000 14000

Pietracorbara [1083 ha]

2000 4000 6000 800010000 12000 14000 5000 10000 15000 20000 25000

Santo Pietro di Tenda [1310 ha]

5000 10000 15000 20000 25000 5000 10000 15000 20000 25000

Calenzana [1419 ha]

200 400 600 800 1000 1200 1400 500 1000 1500 2000

Calenzana [18 ha]

200 400 600 800 1000 1200 1400 500 1000 1500 2000 2500 3000 3500

Solaro [29 ha]

500 1000 1500 2000 2500 3000 3500 500 1000 1500 2000 2500

Volpajola [50 ha]

1000 2000 3000 4000 5000 6000 1000 2000 3000 4000 5000 6000 7000

Murzo [180 ha]

1000 2000 3000 4000 5000 1000 2000 3000 4000 5000

Ghisonaccia [95 ha]

500 1000 1500 2000 2500 3000 3500 500 1000 1500 2000 2500 3000 3500 4000 4500

Vivario [57 ha]

500 1000150020002500300035004000 500 1000 1500 2000 2500

Corscia [58 ha]

2000 4000 6000 8000 10000 12000 2000 4000 6000 8000 10000 12000 14000 16000

Biguglia [783 ha]

100 200 300 400 500 600 700 800 900 100 200 300 400 500 600

Casaglione [2 ha]

500 1000 1500 2000 2500 3000 1000 2000 3000 4000 5000

Aleria [62 ha]

Filippi, Mallet, Nader Model evaluation May 2013 15 / 50

slide-16
SLIDE 16

80 fire cases [2/5]

10002000300040005000600070008000 2000 4000 6000 8000 10000 12000

Sisco [441 ha]

200 400 600 8001000 1200 1400 1600 1800 500 1000 1500 2000 2500 3000 3500

Linguizzetta [34 ha]

500 1000 1500 2000 2500 3000 1000 2000 3000 4000 5000 6000

Aleria [80 ha]

500 1000 1500 2000 500 1000 1500 2000 2500

Prunelli di Fiumorbo [26 ha]

500 1000 1500 2000 2500 3000 1000 2000 3000 4000 5000 6000

Olmeta di Tuda [81 ha]

500 1000150020002500300035004000 1000 2000 3000 4000 5000

Calenzana [76 ha]

500 1000 1500 2000 2500 3000 1000 2000 3000 4000 5000 6000 7000 8000

Nocario [82 ha]

500 1000 1500 2000 2500 500 1000 1500 2000 2500 3000

Ventiseri [29 ha]

100 200 300 400 500 600 200 400 600 800 1000

Porto Vecchio [3 ha]

1000 2000 3000 4000 5000 6000 7000 1000 2000 3000 4000 5000 6000 7000 8000 9000

Afa [242 ha]

500 1000 1500 2000 2500 3000 3500 500 1000 1500 2000 2500

Loreto di Casinca [42 ha]

500 1000150020002500300035004000 500 1000 1500 2000 2500 3000

Calenzana [59 ha]

200 400 600 800 1000 1200 1400 500 1000 1500 2000 2500

Corbara [10 ha]

2000 4000 6000 8000 10000 2000 4000 6000 8000 10000 12000

Vero [535 ha]

200 400 600 800 1000120014001600 500 1000 1500 2000 2500 3000 3500 4000 4500

Canale di Verde [35 ha]

500 1000 1500 2000 2500 3000 200 400 600 800 1000 1200 1400 1600 1800

Altiani [24 ha]

Filippi, Mallet, Nader Model evaluation May 2013 16 / 50

slide-17
SLIDE 17

80 fire cases [3/5]

200 400 600 8001000 1200 1400 1600 1800 200 400 600 800 1000 1200 1400

Patrimonio [13 ha]

1000 2000 3000 4000 5000 6000 7000 500 1000 1500 2000 2500 3000 3500 4000

Pruno [116 ha]

5001000 1500 2000 2500 3000 3500 4000 4500 500 1000 1500 2000 2500 3000 3500 4000

Olcani [70 ha]

50 100 150 200 250 300 50 100 150 200 250

Sartene [0 ha]

500 1000 1500 2000 2500 3000 200 400 600 800 1000 1200 1400

Calenzana [16 ha]

100 200 300 400 500 600 700 200 400 600 800 1000

Propriano [3 ha]

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 1000 2000 3000 4000 5000 6000

Canari [95 ha]

200 400 600 800 1000 1200 1400 200 400 600 800 1000 1200 1400 1600 1800

Olmeta di Tuda [18 ha]

1000 2000 3000 4000 5000 6000 1000 2000 3000 4000 5000 6000 7000 8000

Calenzana [91 ha]

500 1000 1500 2000 2500 3000 3500 1000 2000 3000 4000 5000 6000

Calenzana [103 ha]

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 500 1000 1500 2000 2500 3000

Calenzana [143 ha]

200 400 600 800 1000 100 200 300 400 500 600 700

Piana [2 ha]

200 400 600 8001000 1200 1400 1600 1800 200 400 600 800 1000 1200 1400 1600

Ajaccio [14 ha]

50 100 150 200 250 300 50 100 150 200 250 300 350

Bastelicaccia [0 ha]

40 60 80 100 120 140 160 180 200 220 50 100 150 200 250 300

Propriano [1 ha]

0 2000 4000 6000 8000 10000 12000 14000 16000 18000 2000 4000 6000 8000 10000 12000 14000 16000 18000

Oletta [1126 ha]

Filippi, Mallet, Nader Model evaluation May 2013 17 / 50

slide-18
SLIDE 18

80 fire cases [4/5]

500 1000 1500 2000 2500 3000 500 1000 1500 2000 2500 3000

Soveria [56 ha]

5001000 1500 2000 2500 3000 3500 4000 4500 2000 4000 6000 8000 10000 12000 14000

Sisco [382 ha]

1000 2000 3000 4000 5000 1000 2000 3000 4000 5000 6000 7000

Coti Chiavari [185 ha]

1000 2000 3000 4000 5000 6000 1000 2000 3000 4000 5000 6000 7000 8000

Lumio [201 ha]

500 1000 1500 2000 2500 3000 500 1000 1500 2000 2500 3000 3500 4000

Santa Maria Poggio [59 ha]

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 2000 4000 6000 8000 10000 12000

Barbaggio [518 ha]

500 1000 1500 2000 2500 3000 3500 500 1000 1500 2000 2500 3000 3500

Rutali [60 ha]

1000 2000 3000 4000 5000 1000 2000 3000 4000 5000 6000 7000

Oletta [186 ha]

50 100 150 200 250 300 50 100 150 200 250 300 350

Propriano [0 ha]

2000 4000 6000 8000 10000 12000 1000 2000 3000 4000 5000 6000 7000 8000

Calenzana [465 ha]

500 1000150020002500300035004000 1000 2000 3000 4000 5000 6000

Calvi [79 ha]

200 400 600 800 1000 1200 200 400 600 800 1000 1200 1400 1600

San Giovanni di Moriani [10 ha]

500 1000 1500 2000 2500 3000 1000 2000 3000 4000 5000

Luri [51 ha]

200 400 600 8001000 1200 1400 1600 1800 500 1000 1500 2000 2500

Saint Florent [25 ha]

2000400060008000 10000 12000 14000 16000 2000 4000 6000 8000 10000 12000 14000 16000

Peri [750 ha]

500 1000 1500 2000 2500 3000 500 1000 1500 2000 2500 3000 3500 4000

Poggio d'Oletta [53 ha]

Filippi, Mallet, Nader Model evaluation May 2013 18 / 50

slide-19
SLIDE 19

80 fire cases [5/5]

500 1000 1500 2000 500 1000 1500 2000 2500 3000 3500

Borgo [41 ha]

5001000 1500 2000 2500 3000 3500 4000 4500 2000 4000 6000 8000 10000 12000 14000 16000

Sisco [357 ha]

500 1000 1500 2000 2500 3000 500 1000 1500 2000

Calenzana [27 ha]

2000 4000 6000 8000 10000 12000 2000 4000 6000 8000 10000 12000 14000 16000

Bonifacio [477 ha]

100 200 300 400 500 600 700 800 100 200 300 400 500 600 700 800

Propriano [4 ha]

2000400060008000 10000 12000 14000 16000 2000 4000 6000 8000 10000 12000 14000 16000

Tolla [942 ha]

500 1000 1500 2000 2500 500 1000 1500 2000 2500

Santa Lucia di Mercurio [19 ha]

40 60 80 100 120 140 160 180 200 50 100 150 200 250

Alata [0 ha]

5001000 1500 2000 2500 3000 3500 4000 4500 500 1000 1500 2000 2500 3000 3500

Omessa [116 ha]

10002000300040005000600070008000 1000 2000 3000 4000 5000 6000 7000 8000 9000

Ersa [157 ha]

100 200 300 400 500 600 700 800 900 500 1000 1500 2000 2500

Manso [11 ha]

200 400 600 800 1000 500 1000 1500 2000 2500

Olmeta di Tuda [13 ha]

500 1000150020002500300035004000 1000 2000 3000 4000 5000

Santo Pietro di Tenda [111 ha]

500 1000 1500 2000 2500 3000 3500 500 1000 1500 2000 2500

Calenzana [28 ha]

200 400 600 8001000 1200 1400 1600 1800 200 400 600 800 1000 1200 1400

Bisinchi [14 ha]

500 1000 1500 2000 2500 500 1000 1500 2000 2500 3000 3500

Montegrosso [44 ha]

Filippi, Mallet, Nader Model evaluation May 2013 19 / 50

slide-20
SLIDE 20

Distribution of the Sørensen similarity indexes

Scores sorted independently for each model

10 20 30 40 50 60 70 80 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Score

Balbi non stationary Balbi stationary Rothermel 3-percent

Filippi, Mallet, Nader Model evaluation May 2013 20 / 50

slide-21
SLIDE 21

Distribution of the Jaccard similarity coefficients

Scores sorted independently for each model

10 20 30 40 50 60 70 80 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Score

Balbi non stationary Balbi stationary Rothermel 3-percent

Filippi, Mallet, Nader Model evaluation May 2013 21 / 50

slide-22
SLIDE 22

Distribution of the Kappa coefficients

Scores sorted independently for each model

10 20 30 40 50 60 70 80 0.2 0.0 0.2 0.4 0.6 0.8 1.0 Score

Balbi non stationary Balbi stationary Rothermel 3-percent

Filippi, Mallet, Nader Model evaluation May 2013 22 / 50

slide-23
SLIDE 23

Distribution of the shape agreements

Scores sorted independently for each model

10 20 30 40 50 60 70 80 0.0 0.2 0.4 0.6 0.8 1.0 Score

Balbi non stationary Balbi stationary Rothermel 3-percent

Filippi, Mallet, Nader Model evaluation May 2013 23 / 50

slide-24
SLIDE 24

Distribution of the Sørensen similarity indexes

Scores sorted according to the average score

10 20 30 40 50 60 70 80 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Score

Filippi, Mallet, Nader Model evaluation May 2013 24 / 50

slide-25
SLIDE 25

Distribution of the Jaccard similarity coefficients

Scores sorted according to the average score

10 20 30 40 50 60 70 80 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Score

Filippi, Mallet, Nader Model evaluation May 2013 25 / 50

slide-26
SLIDE 26

Distribution of the Kappa coefficients

Scores sorted according to the average score

10 20 30 40 50 60 70 80 0.2 0.0 0.2 0.4 0.6 0.8 1.0 Score

Filippi, Mallet, Nader Model evaluation May 2013 26 / 50

slide-27
SLIDE 27

Distribution of the shape agreements

Scores sorted according to the average score

10 20 30 40 50 60 70 80 0.0 0.2 0.4 0.6 0.8 1.0 Score

Filippi, Mallet, Nader Model evaluation May 2013 27 / 50

slide-28
SLIDE 28

Monte Carlo simulations

Pertubed parameters

Wind direction: additive perturbation, clipped normal distribution, with zero mean and standard deviation of 40◦ Wind velocity: log-normal distribution, assuming [ 1

2v, 2v] is a

0.95-confidence interval Final burned area: log-normal distribution, assuming [ 2

3v, 3 2v] is a

0.95-confidence interval Fuel load: log-normal distribution, assuming [ 1

1.1v, 1.1v] is a

0.95-confidence interval Moisture content: log-normal distribution, assuming [ 1

1.1v, 1.1v] is a

0.95-confidence interval

Filippi, Mallet, Nader Model evaluation May 2013 28 / 50

slide-29
SLIDE 29

Monte Carlo simulations

Monte Carlo experiment

Selecting the four models with equal probability Running on 75 cases (out of the 80 cases), with at least 1150 simulations per case About 89,000 simulations in total

Filippi, Mallet, Nader Model evaluation May 2013 29 / 50

slide-30
SLIDE 30

Distribution of the Sørensen similarity indexes

Scores sorted independently for each model

5000 10000 15000 20000 25000 0.0 0.2 0.4 0.6 0.8 1.0 Score

Balbi non stationary Balbi stationary Rothermel 3-percent

Filippi, Mallet, Nader Model evaluation May 2013 30 / 50

slide-31
SLIDE 31

Distribution of the Jaccard similarity coefficients

Scores sorted independently for each model

5000 10000 15000 20000 25000 0.0 0.2 0.4 0.6 0.8 1.0 Score

Balbi non stationary Balbi stationary Rothermel 3-percent

Filippi, Mallet, Nader Model evaluation May 2013 31 / 50

slide-32
SLIDE 32

Distribution of the Kappa coefficients

Scores sorted independently for each model

5000 10000 15000 20000 25000 0.2 0.0 0.2 0.4 0.6 0.8 1.0 Score

Balbi non stationary Balbi stationary Rothermel 3-percent

Filippi, Mallet, Nader Model evaluation May 2013 32 / 50

slide-33
SLIDE 33

Distribution of the shape agreements

Scores sorted independently for each model

5000 10000 15000 20000 25000 0.0 0.2 0.4 0.6 0.8 1.0 Score

Balbi non stationary Balbi stationary Rothermel 3-percent

Filippi, Mallet, Nader Model evaluation May 2013 33 / 50

slide-34
SLIDE 34

Distribution of the Sørensen similarity indexes

Scores of the best simulations, sorted independently for each model

10 20 30 40 50 60 70 80 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Score

Balbi non stationary Balbi stationary Rothermel 3-percent

Filippi, Mallet, Nader Model evaluation May 2013 34 / 50

slide-35
SLIDE 35

Distribution of the Jaccard similarity coefficients

Scores of the best simulations, sorted independently for each model

10 20 30 40 50 60 70 80 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Score

Balbi non stationary Balbi stationary Rothermel 3-percent

Filippi, Mallet, Nader Model evaluation May 2013 35 / 50

slide-36
SLIDE 36

Distribution of the Kappa coefficients

Scores of the best simulations, sorted independently for each model

10 20 30 40 50 60 70 80 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Score

Balbi non stationary Balbi stationary Rothermel 3-percent

Filippi, Mallet, Nader Model evaluation May 2013 36 / 50

slide-37
SLIDE 37

Distribution of the shape agreements

Scores of the best simulations, sorted independently for each model

10 20 30 40 50 60 70 80 0.0 0.2 0.4 0.6 0.8 1.0 Score

Balbi non stationary Balbi stationary Rothermel 3-percent

Filippi, Mallet, Nader Model evaluation May 2013 37 / 50

slide-38
SLIDE 38

Distribution of the Jaccard similarity coefficients

Against input parameters, for case “Patrimonio (2005-02-11)”

50 0 50 100150200250300350400 0.1 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 2 4 6 8 10 12 0.1 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 12 14 16 18 20 22 24 0.1 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.4 0.1 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.4 0.1 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

Filippi, Mallet, Nader Model evaluation May 2013 38 / 50

slide-39
SLIDE 39

Distribution of the shape agreements

Against input parameters, for case “Patrimonio (2005-02-11)”

50 0 50 100150200250300350400 0.5 0.6 0.7 0.8 0.9 1.0 2 4 6 8 10 12 0.5 0.6 0.7 0.8 0.9 1.0 12 14 16 18 20 22 24 0.5 0.6 0.7 0.8 0.9 1.0 0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.4 0.5 0.6 0.7 0.8 0.9 1.0 0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.4 0.5 0.6 0.7 0.8 0.9 1.0

Filippi, Mallet, Nader Model evaluation May 2013 39 / 50

slide-40
SLIDE 40

Distribution of the shape agreements

Against input parameters, for case “Olcani (2006-01-01)”

50 100 150 200 250 300 0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 60 70 80 90 100 110 120 130 140 0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.4 0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.4 0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2

Filippi, Mallet, Nader Model evaluation May 2013 40 / 50

slide-41
SLIDE 41

Distribution of the shape agreements

Against input parameters, for case “Sisco (2003-08-14)”

50 0 50 100150200250300350400 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.0 0.5 1.0 1.5 2.0 2.5 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 350 400 450 500 550 600 650 700 750 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.4 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.4 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

Filippi, Mallet, Nader Model evaluation May 2013 41 / 50

slide-42
SLIDE 42

75 fire cases [1/5]

2000 4000 6000 800010000 12000 14000 2000 4000 6000 8000 10000 12000 500 1000 1500 2000 2500 500 1000 1500 2000 2500 3000 1000 2000 3000 4000 5000 6000 7000 1000 2000 3000 4000 5000 6000 5000 10000 15000 20000 5000 10000 15000 20000 200400600800 1000 1200 500 1000 1500 200 400 600 800 1000 1200 500 1000 1500 2000 2500 3000 500 1000 1500 2000 2500 3000 500 1000 1500 2000 1000 2000 3000 4000 5000 1000 2000 3000 4000 5000 6000 1000 2000 3000 4000 1000 2000 3000 4000 5001000 1500 2000 2500 3000 500 1000 1500 2000 2500 3000 3500 4000 500 1000 1500 2000 2500 3000 3500 500 1000 1500 2000 2000 4000 6000 8000 10000 2000 4000 6000 8000 10000 12000 14000 100 200 300 400 500 600 700 800 100 200 300 400 500 500 1000 1500 2000 2500 1000 2000 3000 4000 200 400 600 800 1000 1200 1400 1600 500 1000 1500 2000 2500 3000 500 1000 1500 2000 2500 1000 2000 3000 4000 5000

Filippi, Mallet, Nader Model evaluation May 2013 42 / 50

slide-43
SLIDE 43

75 fire cases [2/5]

500 1000 1500 500 1000 1500 2000 500 1000 1500 2000 2500 1000 2000 3000 4000 5000 500 1000 1500 2000 2500 3000 3500 1000 2000 3000 4000 500 1000 1500 2000 2500 1000 2000 3000 4000 5000 6000 7000 500 1000 1500 2000 500 1000 1500 2000 2500 100 200 300 400 500 100 200 300 400 500 600 700 800 900 1000 2000 3000 4000 5000 6000 1000 2000 3000 4000 5000 6000 7000 8000 500 1000 1500 2000 2500 3000 3500 500 1000 1500 2000 2500 200 400 600 800 1000 1200 500 1000 1500 2000 500 1000 1500 2000 2500 3000 500 1000 1500 2000 200 400 600 800 1000 1200 1400 500 1000 1500 2000 2500 3000 3500 4000 500 1000 1500 2000 2500 200 400 600 800 1000 1200 1400 1600 200 400 600 800 1000 1200 1400 1600 200 400 600 800 1000 1200 1000 2000 3000 4000 5000 6000 500 1000 1500 2000 2500 3000 3500 500 1000150020002500300035004000 500 1000 1500 2000 2500 3000 3500 50 100 150 200 250 50 100 150 200

Filippi, Mallet, Nader Model evaluation May 2013 43 / 50

slide-44
SLIDE 44

75 fire cases [3/5]

500 1000 1500 2000 2500 200 400 600 800 1000 1200 100200300400500600 200 400 600 800 1000 2000 3000 4000 5000 6000 7000 8000 1000 2000 3000 4000 5000 200 400 600 8001000 1200 200 400 600 800 1000 1200 1400 1600 10002000300040005000 1000 2000 3000 4000 5000 6000 7000 500 1000 1500 2000 2500 3000 1000 2000 3000 4000 5000 1000 2000 3000 4000 5000 6000 7000 8000 500 1000 1500 2000 2500 200 400 600 800 100 200 300 400 500 600 200 400 600 800 1000120014001600 200 400 600 800 1000 1200 1400 50 100 150 200 250 50 100 150 200 250 300 60 80100 120 140 160 180 200 50 100 150 200 250 2000 4000 6000 8000 10000 12000 14000 16000 2000 4000 6000 8000 10000 12000 14000 16000 500 1000 1500 2000 2500 500 1000 1500 2000 2500 500 1000 1500 2000 2500 3000 3500 4000 2000 4000 6000 8000 10000 12000 1000 2000 3000 4000 1000 2000 3000 4000 5000 6000 10002000300040005000 1000 2000 3000 4000 5000 6000 7000

Filippi, Mallet, Nader Model evaluation May 2013 44 / 50

slide-45
SLIDE 45

75 fire cases [4/5]

500 1000150020002500 500 1000 1500 2000 2500 3000 3500 1000 2000 3000 4000 5000 6000 7000 8000 2000 4000 6000 8000 10000 500 1000 1500 2000 2500 3000 500 1000 1500 2000 2500 3000 10002000 30004000 1000 2000 3000 4000 5000 6000 50 100 150 200 250 50 100 150 200 250 300 2000 4000 6000 8000 10000 1000 2000 3000 4000 5000 6000 7000 500 1000 1500 2000 2500 3000 3500 1000 2000 3000 4000 5000 200 400 600 800 1000 200 400 600 800 1000 1200 1400 500 1000 1500 2000 2500 1000 2000 3000 4000 200 400 600 800 1000 1200 1400 1600 500 1000 1500 2000 2000400060008000 10000 12000 14000 2000 4000 6000 8000 10000 12000 14000 500 1000 1500 2000 2500 500 1000 1500 2000 2500 3000 3500 500 1000 1500 500 1000 1500 2000 2500 3000 500 1000 1500 2000 2500 3000 3500 4000 2000 4000 6000 8000 10000 12000 14000 500 1000 1500 2000 2500 500 1000 1500 100 200 300 400 500 600 700 100 200 300 400 500 600 700

Filippi, Mallet, Nader Model evaluation May 2013 45 / 50

slide-46
SLIDE 46

75 fire cases [5/5]

2000400060008000 10000 12000 14000 2000 4000 6000 8000 10000 12000 14000 500 1000 1500 2000 500 1000 1500 2000 60 80100 120 140 160 180 50 100 150 200 500 1000 1500 2000 2500 3000 3500 4000 500 1000 1500 2000 2500 3000 1000 2000 3000 4000 5000 6000 7000 1000 2000 3000 4000 5000 6000 7000 8000 100 200 300 400 500 600 700 800 500 1000 1500 2000 200 400 600 800 500 1000 1500 2000 5001000 1500 2000 2500 3000 3500 1000 2000 3000 4000 500 1000 1500 2000 2500 3000 500 1000 1500 2000 200 400 600 800 1000 1200 1400 1600 200 400 600 800 1000 1200 500 1000 1500 2000 500 1000 1500 2000 2500 3000

Filippi, Mallet, Nader Model evaluation May 2013 46 / 50

slide-47
SLIDE 47

Probability density

For case “Olcani (2006-01-01)”

500 1000150020002500300035004000 500 1000 1500 2000 2500 3000 3500

Filippi, Mallet, Nader Model evaluation May 2013 47 / 50

slide-48
SLIDE 48

Probability density

For case “Patrimonio (2005-02-11)”

200 400 600 800 1000 1200 1400 1600 200 400 600 800 1000 1200

Filippi, Mallet, Nader Model evaluation May 2013 48 / 50

slide-49
SLIDE 49

Reliability diagram

0.0 0.2 0.4 0.6 0.8 1.0 Computed probability 0.0 0.2 0.4 0.6 0.8 1.0 Observed probability

Filippi, Mallet, Nader Model evaluation May 2013 49 / 50

slide-50
SLIDE 50

Conclusions

Scoring methods

It is possible to take into account the model dynamics

Arrival time agreement and shape agreement

Evaluation on 80 fires

It seems possible to rank models, without any (over)tuning 3-percent rate of spread significantly worse

Monte Carlo simulations

May also be used to rank models Toward probabilistic forecasts and uncertainty estimation

Filippi, Mallet, Nader Model evaluation May 2013 50 / 50