CEN/TC Some open issues Ari Karppinen Athens 6/2017
How to use MQO in real life ? • Station representativeness • Demonstration of eqivalence • Minimum number of stations needed • Focus area vs. validation area
Example:Modelling system - FMI Dispersion models - Dispersion and effects Weather prediction models – urban, local models long-range, regional HIRLAM ECMWF (7.5) CAR-FMI, roadside SILAM LRT, meso-scale, UDM-FMI, urban radioactivity , pollen OSPM (NERI), street HILATAR canyon HARMONIE LRT, meso-scale LAPS;PALM BUOYANT, fires (2.5) ESCAPE, chemical accidents PALM, LESCAPE Aerosol process models: (DNS-code development) SALSA(UH,UKU,FMI) EXPAND (FMI, YTV) SILAM-APMs population exposure MPP-FMI, Meteorological pre-processing model
” I deal” situation Validation area = whole country Measurements used for assesment of the modelling system = all stations available (~150) Focus area = any area inside the country Number of stations in focus area irrelevant (?!)
Typical situtation • Different model for different regions • Different/completly separate models for e.g. urban vs. rurual areas Challenge : number of stations available for evaluation (taking into account the type of model) very limited => definition of minum number of stations needed for model evaluation Suggestions: ~5 , not based on any statistical jsutification , but more on experince/the real/existing status with measurements
Some additional comments Certain types of models are still ”automatically” left out from this type of evaluation (even without discussing the problems with long term time series /statistics): e.g. CFD/LES modelling of urban areas UNTIL the measurement networks are able to support these very high resolution models .. Or.. eg. Wind-tunnel validation would be ”accepted” as a feasible oprtion for Validating these
Data assimilation Models which take full use of the available measurement data will be of fastly growing importance everywhere <-> compare with opreative meteorological models : any ”reasonable” model uses heavily data -assimilation Models utilizing data assimilation or data fusion need to have clear guidance for proper evaluation: multitude of different assessment methods (leave n-out/bootstrapping etc..) Good: don’t need to invent anything really new Bad: not trivial to ”force” a single evaluationmethod , which would be the ¨optimal” choice for all different DA/DF -modelling system Additional concern: lot of modelling sytems use ”hidden” data -assimilation ..e.g. bias – correction - > should we jsut accept thtat this is ”feature” of the model, or should we evaluate teh mdoel without bias-correaction ?!
Operational modelling system For selected cities in Finland FMI-ENFUSER produces high resolution (15x15 m) AQ information - Includes AQ forecasts for the next 12-48 hours - Model data accessible through a Web Portal Information Calibration extraction update (every 10 minutes) (1-4 times per year) FMI-ENFUSER (runs every 1-6h) Model output Web Portal (figures, animations, gridded data, etc) NO2, PM2.5, PM10, O3, AQI Amazon Web Service, H erokuApp , Dropbox 8
Online /real-time evaluaton 29.6.2017 9
Some concerns about the delta-formulation: beta-parameter In order to assess and classify model quality • in a scientific way, a criterion based only on β or β90 does not seem sufficient. There are many situations where an • assessment using only β or β90 will lead to disputable or simply wrong classifications, compared to an assessment based on standard regression techniques. Requiring values for β or β 90 less than 2 in • the MQO will result in more possibly “wrong” assessments.
Some concerns about the delta-formulation : normality of residuals) In practically all cases the tail of the distribution • was “long” and sometimes also skewed. There is a larger possibility of outliers. o The usual assumption that 95% of residues is o between ± 2 times the standard deviation may not hold. Determining the meaning of this finding for the • MQO requires more tests and research …
Some concerns about the delta-formulation ( temporal and spatial shift ) PM10, Germany, ½year switch +12 h, switch NO2, Germany, ½year switch +12 h, switch O3, Germany, ½year switch +12 h, switch
Some concerns about the delta-formulation ( temporal and spatial shift) PM10, Germany, correct location vs. wrong location
Preliminary conclusions ( tempora l and spatial shift) • The shape of the diurnal/seasonal variation is crucial: different pollutants have completely different sensitivity to temporal shift : e.g O3 very sensitive whereas NO2/PM10 not for this specific tested shift • Additional systematic analysis is needed e. g.: • more drastic temporal shift e.g.randomly distribute the time dimension of the modeled concentration field • more drastic spatial shift e.g spatial averaging of the modeled concentration field • optimal stringency of MQI und MPI with respect to spatial and temporal shift should be clarified (long-term vs short-term, different pollutants)
Summary CEN/TC 264/WG 43 has still several interesting open issues • to discuss, study & finally close Reasonable amount of studies & work & discussions still • needed before we can propose the final suggestions for model evaluation standards Some of the open issued are highly relevant/tightly • connected to FAIRMODE/DELTA tool development
Recommend
More recommend
