CEN/TC Some open issues Ari Karppinen Athens 6/2017 How to use - - PowerPoint PPT Presentation

CEN/TC

Some open issues

Ari Karppinen

Athens 6/2017

How to use MQO in real life ?

• Station representativeness
• Demonstration of eqivalence
• Minimum number of stations needed
• Focus area vs. validation area

ECMWF HIRLAM (7.5) HARMONIE (2.5) LAPS;PALM

Weather prediction models Dispersion models - long-range, regional Dispersion and effects models – urban, local

PALM, LESCAPE (DNS-code development)

SILAM LRT, meso-scale, radioactivity , pollen

Example:Modelling system - FMI

OSPM (NERI), street canyon UDM-FMI, urban CAR-FMI, roadside

HILATAR LRT, meso-scale

Aerosol process models: SALSA(UH,UKU,FMI) SILAM-APMs MPP-FMI, Meteorological pre-processing model ESCAPE, chemical accidents BUOYANT, fires EXPAND (FMI, YTV) population exposure

”Ideal” situation

Validation area = whole country Measurements used for assesment of the modelling system = all stations available (~150) Focus area = any area inside the country Number of stations in focus area irrelevant (?!)

Typical situtation

• Different model for different regions
• Different/completly separate models for e.g. urban vs. rurual areas

Challenge : number of stations available for evaluation (taking into account the type of model) very limited => definition of minum number of stations needed for model evaluation Suggestions: ~5 , not based on any statistical jsutification , but more on experince/the real/existing status with measurements

Some additional comments

Certain types of models are still ”automatically” left out from this type of evaluation (even without discussing the problems with long term time series /statistics): e.g. CFD/LES modelling of urban areas UNTIL the measurement networks are able to support these very high resolution models .. Or.. eg. Wind-tunnel validation would be ”accepted” as a feasible oprtion for Validating these

Data assimilation

Models utilizing data assimilation or data fusion need to have clear guidance for proper evaluation: multitude of different assessment methods (leave n-out/bootstrapping etc..) Good: don’t need to invent anything really new Bad: not trivial to ”force” a single evaluationmethod , which would be the ¨optimal” choice for all different DA/DF-modelling system Models which take full use of the available measurement data will be

• f fastly growing importance everywhere

<-> compare with opreative meteorological models : any ”reasonable” model uses heavily data-assimilation Additional concern: lot of modelling sytems use ”hidden” data-assimilation ..e.g. bias –correction -> should we jsut accept thtat this is ”feature” of the model,

• r should we evaluate teh mdoel without bias-correaction ?!

Operational modelling system

8 FMI-ENFUSER

(runs every 1-6h)

Information extraction

(every 10 minutes)

Calibration update

(1-4 times per year)

Model output

(figures, animations, gridded data, etc)

Web Portal For selected cities in Finland FMI-ENFUSER produces high resolution (15x15 m) AQ information

• Includes AQ forecasts for the next 12-48 hours
• Model data accessible through a Web Portal

NO2, PM2.5, PM10, O3, AQI

Amazon Web Service, HerokuApp, Dropbox

Online /real-time evaluaton

29.6.2017

9

Some concerns about the delta-formulation: beta-parameter

• In order to assess and classify model quality

in a scientific way, a criterion based only on β or β90 does not seem sufficient.

• There are many situations where an

assessment using only β or β90 will lead to disputable or simply wrong classifications, compared to an assessment based on standard regression techniques.

• Requiring values for β or β90 less than 2 in

the MQO will result in more possibly “wrong” assessments.

Some concerns about the delta-formulation : normality of residuals)

• In practically all cases the tail of the distribution

was “long” and sometimes also skewed.

• There is a larger possibility of outliers.
• The usual assumption that 95% of residues is

between ±2 times the standard deviation may not hold.

• Determining the meaning of this finding for the

MQO requires more tests and research …

Some concerns about the delta-formulation ( temporal and spatial shift)

NO2, Germany, ½year switch +12 h, switch

PM10, Germany, ½year switch +12 h, switch O3, Germany, ½year switch +12 h, switch

Some concerns about the delta-formulation ( temporal and spatial shift)

PM10, Germany, correct location vs. wrong location

• The shape of the diurnal/seasonal variation is crucial: different

pollutants have completely different sensitivity to temporal shift : e.g O3 very sensitive whereas NO2/PM10 not for this specific tested shift

• Additional systematic analysis is needed e. g.:
• more drastic temporal shift e.g.randomly distribute the time

dimension of the modeled concentration field

• more drastic spatial shift e.g spatial averaging of the modeled

concentration field

• optimal stringency of MQI und MPI with respect to spatial and

temporal shift should be clarified (long-term vs short-term, different pollutants)

Preliminary conclusions ( temporal and spatial shift)

Summary

• CEN/TC 264/WG 43 has still several interesting open issues

to discuss, study & finally close

• Reasonable amount of studies & work & discussions still

needed before we can propose the final suggestions for model evaluation standards

• Some of the open issued are highly relevant/tightly

connected to FAIRMODE/DELTA tool development