Model evaluation based on a large observation data set Jean-Baptiste - PowerPoint PPT Presentation

Model evaluation based on a large observation data set Jean-Baptiste Filippi 1 Vivien Mallet 2 , 3 Bahaa Nader 1 1 SPE CNRS 2 INRIA 3 CEREA, joint laboratory ENPC - EDF R&D, Université Paris-Est Numerical simulation of forest fires, Cargèse, May 2013 Filippi, Mallet, Nader Model evaluation May 2013 1 / 50

Topic Evaluation of the performance of a propagation model Questions 1 How to rank models? Can we identify the “best” model out of a pool of models? 2 How to evaluate the dynamics of the model when the observation is the final burned surface? 3 Can we evaluate a model regardless of the quality of its inputs? 4 Can we carry out probabilistic forecasts? Filippi, Mallet, Nader Model evaluation May 2013 2 / 50

Notation Observation Simulation Observation of final burned Final simulation time: t s surface at time t o f f Simulated burned surface at Observed burned surface: time t : S ( t ) S o ( t o f ) Area |S| is the area of the surface S Ω is the simulation domain Scores Classical scores compare S o ( t o f ) and S ( t s f ) Filippi, Mallet, Nader Model evaluation May 2013 3 / 50

Classical scores Sørensen similarity index Jaccard similarity coefficient 2 |S o ( t o f ) ∩ S ( t s |S o ( t o f ) ∩ S ( t s f ) | f ) | f ) | ∈ [ 0 , 1 ] f ) | ∈ [ 0 , 1 ] |S o ( t o f ) | + |S ( t s |S o ( t o f ) ∪ S ( t s Kappa coefficient P a − P e ≤ 1 1 − P e where P a = |S o ( t o f ) ∩ S ( t s + | Ω \ ( S o ( t o f ) ∪ S ( t s f ) | f )) | | Ω | | Ω | P e = |S o ( t o f ) ||S ( t s + | Ω \S o ( t o f ) || Ω \S ( t s f ) | f ) | | Ω | 2 | Ω | 2 Filippi, Mallet, Nader Model evaluation May 2013 4 / 50

Dynamic-aware scores Arrival time agreement Addition notation Arrival time: earliest time at which the front is known to have reached some point; + ∞ if the front never reaches the point Observed arrival time at X : T o ( X ) , say T o ( X ) = t o f if X ∈ S o ( t o f ) Simulated arrival time at X : T ( X ) , here, T ( X ) ≤ t s f if X ∈ S ( t s f ) Arrival time agreement �� 1 max ( T ( X ) − T o ( X ) , 0 ) d X 1 − |S ( t s f ) ∪ S o ( t o f ) | max ( t s f , t o f ) S ( t s f ) ∩S o ( t o f ) � max ( t o + f − T ( X ) , 0 ) d X S ( t s f ) \S o ( t o f ) � � ( t s f − T o ( X )) d X + ∈ [ 0 , 1 ] S o ( t o f ) \S ( t s f ) Filippi, Mallet, Nader Model evaluation May 2013 5 / 50

Dynamic-aware scores Shape agreement Shape agreement �� � |S ( t ) \S o ( t o |S o ( t o 1 − 1 f ) | f ) \S ( t ) | � d t + d t ∈ [ 0 , 1 ] t s |S o ( t o |S ( t ) | f ) | ] 0 , t o [ t o f , t s f ] f [ f Filippi, Mallet, Nader Model evaluation May 2013 6 / 50

Application of the scores to an idealized case 0 2000 0 2000 0 2250 1800 1800 2000 200 1600 200 1600 200 1750 1400 1400 1500 400 1200 400 1200 400 1250 1000 1000 1000 600 800 600 800 600 750 600 600 500 800 400 800 400 800 200 200 250 0 0 0 0 200 400 600 800 0 200 400 600 800 0 200 400 600 800 S = 0.981 S = 0.981 J = 0.962 J = 0.962 K = 0.977 K = 0.977 ATA = 0.985 ATA = 0.998 SA = 0.973 SA = 0.997 J.-B. Filippi, V. Mallet, and B. Nader (2013). “Representation and evaluation of wildfire propagation simulations”. In: Under review for Int. J. Wild. Fire Scoring methods in Python at http://sf.net/projects/pyfirescore/ Filippi, Mallet, Nader Model evaluation May 2013 7 / 50

Simulation of 80 fire cases Fire cases Using observations from Prométhée, database of french fires Available data for a subset of the fires: date, ignition point, final contour Unfortunately, no data on firefights Considering 80 Corsican fires from 2003 to 2008 Filippi, Mallet, Nader Model evaluation May 2013 8 / 50

80 fire cases [1/5] Aullene [990 ha] Oletta [54 ha] Olmi Cappella [193 ha] Pietracorbara [1083 ha] 14000 3500 7000 14000 12000 3000 6000 12000 10000 2500 5000 10000 8000 2000 4000 8000 6000 1500 3000 6000 4000 1000 2000 4000 2000 500 1000 2000 0 0 0 0 0 2000400060008000 10000 12000 14000 16000 0 500 1000 1500 2000 2500 3000 0 10002000300040005000600070008000 0 2000 4000 6000 8000 10000 12000 14000 16000 18000 Santo Pietro di Tenda [1310 ha] Calenzana [1419 ha] Calenzana [18 ha] Solaro [29 ha] 25000 25000 2000 3500 3000 20000 20000 1500 2500 15000 15000 2000 1000 1500 10000 10000 1000 500 5000 5000 500 0 0 0 0 0 2000 4000 6000 800010000 12000 14000 0 5000 10000 15000 20000 25000 0 200 400 600 800 1000 1200 1400 0 200 400 600 800 1000 1200 1400 Volpajola [50 ha] Murzo [180 ha] Ghisonaccia [95 ha] Vivario [57 ha] 2500 7000 5000 4500 4000 6000 2000 4000 3500 5000 3000 1500 3000 4000 2500 2000 3000 1000 2000 1500 2000 1000 500 1000 1000 500 0 0 0 0 0 500 1000 1500 2000 2500 3000 3500 0 1000 2000 3000 4000 5000 6000 0 1000 2000 3000 4000 5000 0 500 1000 1500 2000 2500 3000 3500 Biguglia [783 ha] Casaglione [2 ha] Corscia [58 ha] Aleria [62 ha] 2500 16000 600 5000 14000 500 2000 4000 12000 400 10000 1500 3000 8000 300 1000 2000 6000 200 4000 500 1000 100 2000 0 0 0 0 0 500 1000150020002500300035004000 0 2000 4000 6000 8000 10000 12000 0 100 200 300 400 500 600 700 800 900 0 500 1000 1500 2000 2500 3000 Filippi, Mallet, Nader Model evaluation May 2013 9 / 50

80 fire cases [2/5] Sisco [441 ha] Linguizzetta [34 ha] Aleria [80 ha] Prunelli di Fiumorbo [26 ha] 12000 3500 6000 2500 3000 10000 5000 2000 2500 8000 4000 1500 2000 6000 3000 1500 1000 4000 2000 1000 500 2000 1000 500 0 0 0 0 0 10002000300040005000600070008000 0 200 400 600 8001000 1200 1400 1600 1800 0 500 1000 1500 2000 2500 3000 0 500 1000 1500 2000 Olmeta di Tuda [81 ha] Calenzana [76 ha] Nocario [82 ha] Ventiseri [29 ha] 6000 5000 8000 3000 7000 5000 2500 4000 6000 4000 2000 5000 3000 3000 4000 1500 2000 3000 2000 1000 2000 1000 1000 500 1000 0 0 0 0 0 500 1000 1500 2000 2500 3000 0 500 1000150020002500300035004000 0 500 1000 1500 2000 2500 3000 0 500 1000 1500 2000 2500 Porto Vecchio [3 ha] Afa [242 ha] Loreto di Casinca [42 ha] Calenzana [59 ha] 1000 9000 2500 3000 8000 2500 800 2000 7000 6000 2000 600 1500 5000 1500 4000 400 1000 3000 1000 2000 200 500 500 1000 0 0 0 0 0 100 200 300 400 500 600 0 1000 2000 3000 4000 5000 6000 7000 0 500 1000 1500 2000 2500 3000 3500 0 500 1000150020002500300035004000 Corbara [10 ha] Vero [535 ha] Canale di Verde [35 ha] Altiani [24 ha] 2500 12000 4500 1800 4000 1600 10000 2000 3500 1400 8000 3000 1200 1500 2500 1000 6000 2000 800 1000 4000 1500 600 1000 400 500 2000 500 200 0 0 0 0 0 200 400 600 800 1000 1200 1400 0 2000 4000 6000 8000 10000 0 200 400 600 800 1000120014001600 0 500 1000 1500 2000 2500 3000 Filippi, Mallet, Nader Model evaluation May 2013 10 / 50

80 fire cases [3/5] Patrimonio [13 ha] Pruno [116 ha] Olcani [70 ha] Sartene [0 ha] 1400 4000 4000 250 1200 3500 3500 200 3000 3000 1000 2500 2500 150 800 2000 2000 600 100 1500 1500 400 1000 1000 50 200 500 500 0 0 0 0 0 200 400 600 8001000 1200 1400 1600 1800 0 1000 2000 3000 4000 5000 6000 7000 0 5001000 1500 2000 2500 3000 3500 4000 4500 0 50 100 150 200 250 300 Propriano [3 ha] Calenzana [16 ha] Canari [95 ha] Olmeta di Tuda [18 ha] 1400 1000 6000 1800 1600 1200 5000 800 1400 1000 4000 1200 600 800 1000 3000 800 600 400 2000 600 400 400 200 1000 200 200 0 0 0 0 0 500 1000 1500 2000 2500 3000 0 100 200 300 400 500 600 700 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 0 200 400 600 800 1000 1200 1400 Calenzana [91 ha] Calenzana [103 ha] Calenzana [143 ha] Piana [2 ha] 8000 6000 3000 700 7000 600 5000 2500 6000 500 4000 2000 5000 400 4000 3000 1500 300 3000 2000 1000 200 2000 1000 500 100 1000 0 0 0 0 0 1000 2000 3000 4000 5000 6000 0 500 1000 1500 2000 2500 3000 3500 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 0 200 400 600 800 1000 Ajaccio [14 ha] Propriano [1 ha] Bastelicaccia [0 ha] Oletta [1126 ha] 1600 350 300 18000 1400 16000 300 250 14000 1200 250 200 12000 1000 200 10000 800 150 150 8000 600 100 6000 100 400 4000 50 50 200 2000 0 0 0 0 0 200 400 600 8001000 1200 1400 1600 1800 0 50 100 150 200 250 300 40 60 80 100 120 140 160 180 200 220 0 2000 4000 6000 8000 10000 12000 14000 16000 18000 Filippi, Mallet, Nader Model evaluation May 2013 11 / 50