assessment of the cost/benefit of meterological observations and - - PowerPoint PPT Presentation

On the use of data assimilation in the assessment of the cost/benefit of meterological observations and observing systems.

Lars Peter Riishojgaard

Director, Joint Center for Satellite Data Assimilation Chair, OPAG-IOS, WMO Commission for Basic Systems

6th WMO Data Assimilation Symposium

Overview

• Money and weather forecasting

– Enabling capabilities

• Data and weather forecasting

– Data requirements – Cost of meteorological observations

• Data impacts and the WMO RRR
• Can we put a monetary value on individual observing

systems and/or individual observations? (and should

we?) – Role of Data Assimilation community and WMO RRR

• Final remarks

WMO Data Assimilation Symposium, College Park Oct 7-11 2013 2

WMO Data Assimilation Symposium, College Park Oct 7-11 2013

Weather Prediction and the US Economy; A Macroscopic View

• Department of Commerce: “20% of overall US

economy is weather sensitive”: ~$3 trillion/year – Impact to air and surface transportation, agriculture, construction, energy production and distribution, etc.

• Assume that half of this is “forecast sensitive”:

$1.5 trillion/year

• Assume that the potential savings due to weather

forecasting amount to 5% of the “forecast sensitive total”: ~$75B/year

3

(discussed during CBS TECO in Windhoek, 2010)

WMO Data Assimilation Symposium, College Park Oct 7-11 2013

… a Macroscopic View … (II)

• “Perfect forecast” is an NWP run with useful skill at two

weeks!

• 0 h useful forecast range => $0 in savings
• 336 h useful forecast (two weeks maximum predictability)

range => $75B in savings

• Assume now that the savings are distributed linearly over the

achieved forecast range for the global NWP system:

– $75B/336h ~ $223B/hr

• This implies that the value to the United States economy of

weather observations, dissemination, forecast products and services is >$220M per hour of forecast range per year !

4

(discussed during TECO in Windhoek, 2010)

The global picture

• The amount of $75B/year is one estimate of the

magnitude of the total potential socioeconomic benefit

• f weather prediction activities to the US economy
• Scaling exercise, using World Bank (2011) numbers:
• Annual GDP of United States: ~$15T
• Annual GDP of all nations combined: ~$70T

– Assuming on average (i) equal sensitivity to weather, and (ii) equal potential benefits from ability to predict across all nations, we get an estimated $75B *($15T/$70T) = $350B as the total global potential benefit of weather prediction activities (indicating a likely range of $100B to $1T)

WMO Data Assimilation Symposium, College Park Oct 7-11 2013 5

Weather Prediction Enabling Capabilities

1. Observing Systems (GOS0 2. Dissemination Systems 3. Numerical Weather Prediction – Science (modeling, data assimilation) – High-end computing

WMO Data Assimilation Symposium, College Park Oct 7-11 2013 6

• 1, 2 and 3 are of a foundational nature
• Among the foundational capabilities, 1 represents

the single largest expenditure

WMO Data Assimilation Symposium, College Park Oct 7-11 2013

NWP requirements for upper- air data coverage

7

Hence the need for a global

• bserving system, irrespective
• f target location of forecast!

Estimating the total cost of running all components of the GOS

• Informal exercise launched by the WMO Commission

for Basic Systems in 2012

• Approach was (perhaps too?) simple:

– Survey a small, but representative number of WMO member states about their total annual investment in running, maintaining and updating the GOS – Use Cost-to-GDP ratios to extrapolate that to the rest of the WMO members – Add estimates for external (non-NMHS owned) contributions:

• Satellite systems
• Aircraft observations
• Third party networks
• Marine observations
• …

WMO Data Assimilation Symposium, College Park Oct 7-11 2013 8

NMHS input to survey

(anonymous)

Country GOS cost (K USD, 2011) Expenditure

• n GOS, GDP

fraction Comment 1 8330 1.3 x 10-5 Excludes radar data 2 39096 1.1 x 10-5 3 7793 1.6 x 10-3 (LDC) 4 14400 2.3 x 10-4 (Amortization unclear) … … … …

WMO Data Assimilation Symposium, College Park Oct 7-11 2013 9

What do we think we know about the global cost of acquiring the observations?

• Ratios are too disparate to be used for scaling, but

the running costs of the conventional parts of the GOS appear to be in the single-digit $B-range

• “Non-NMHS”-provided data (satellite, third-party

networks, marine, aircraft observations) add an estimated $5B total

• Total costs of all meteorological observations

add up to an estimated $5-10B per year

• That is a lot of money, and decision-makers are
• n the hunt for justification and objective metrics!

WMO Data Assimilation Symposium, College Park Oct 7-11 2013 10

What do NWP and data assimilation have to do with this?

 Objective, quantitative metrics:

 NWP poses a well-defined prediction

problem with a “right” answer

 (and an infinity of wrong ones)

 Well-defined measures for quality of output  Well-established methodologies for

assigning merit (or blame) to individual

• bserving systems

WMO Data Assimilation Symposium, College Park Oct 7- 11 2013

11

WMO Commission for Basic System, Rolling Review of Requirements



Under its Commission for Basic Systems WMO maintains a standing Expert Team (ET-EGOS) which is responsible for documenting

1.

Observational data requirements for all 12 WMO application areas (database)

2.

Capability of all relevant observing system (databases)

3.

Statements of Guidance, or gap analyses, matching 1. against 2 (a brief narrative for each application area)

4.

…



ET-EGOS is helped in its work by numerous other ET’s and Rapporteurs and by the WMO NWP Impact Workshops taking place once every four years

WMO Data Assimilation Symposium, College Park Oct 7- 11 2013

12

WMO Workshops on the Impact of Various Observing Systems on NWP

Five Workshops so far (the last two in close collaboration with THORPEX):

• 1st - Geneva, 1997
• 2nd – Toulouse, 2000
• 3rd – Alpbach, 2004
• 4th – Geneva, 2008
• Workshop Report available on

http://www.wmo.int/pages/prog/www/OSY/Reports/NWP-4_Geneva2008_index.html

• 5th – Sedona AZ, May 2012

http://www.wmo.int/pages/prog/www/OSY/Meetings/NWP5_Sedona2012/Final_Report.pdf

Workshops aim to bring together scientists from all major NWP centers to assess the contribution to forecast skill of various elements of the global

• bserving system; guidance to participants provided well in advance of

Workshop itself.

13 WMO Data Assimilation Symposium, College Park Oct 7-11 2013

OSE and FSO (I)

• OSEs (Observing System

Experiments) are based on data denial (or addition)

• Impact focuses on the

medium to long range

• Results show the impact of

withdrawing (or adding) certain data

• OSE results are absolute;

e.g. “observing system X extends the useful forecast range by N hours in the NH”

WMO Data Assimilation Symposium, College Park Oct 7-11 2013

14

Jung et al., WMO Impact Workshop in Sedona, May 2012

OSE and FSO (II)

• FSO (Forecast Sensitivity to Observations)

are based on the adjoint of the model/analysis system or an ensemble approach

• Approach focuses exclusively on the short

(quasi-linear) range

• Results show the impact of observations

in the presence of all other observations

• FSO measures of impact are relative

(e.g. often expressed in percentages that add up 100, even for poor forecasts or poor system performance)

WMO Data Assimilation Symposium, College Park Oct 7-11 2013

15

Gelaro et al, Fifth WMO Impact Workshop, Sedona 2012

4th WMO Impact Workshop, Alpbach 2008; impact summary slide

Overall impact (“marginal skill”) on global NWP

16 WMO Data Assimilation Symposium, College Park Oct 7-11 2013

Some Preliminary Conclusions from the Fifth WMO Impact Workshop in Sedona, May 2012

• Modern, 4-dimensional data assimilation methods (4D-VAR,

ENKF) have led to greatly improved use of data, especially of – Asynoptic data (e.g. aircraft, satellite observations) – Observations with complex relationships between measured and model variables (satellite radiances, GPSRO, radar,…)

• Broad consensus about highest-ranking contributors to forecast

skill, but not necessarily about their ranking order: – AMSU-A (microwave temperature sounder) – AIRS/IASI (hyper-spectral infrared sounders) – Radiosondes – Aircraft observations – Atmospheric motion vectors (feature tracking satellite winds)

WMO Data Assimilation Symposium, College Park Oct 7-11 2013 17

Some Preliminary Conclusions (II)

• Radio occultation data (GPSRO) also have substantial impact

but data volumes are currently declining as COSMIC is approaching the end of its lifetime

• There is now no single, dominating satellite sensor; several

sensors contribute to forecast skill in roughly equal measure

• The relative impacts of specific observation types depends on

which other observations are used and how – If certain data are withheld, other datatypes can in some contexts compensate for the lost skill

• However, the continued value of in situ data, and in particular of

wind measurements, was clearly demonstrated

• Regional data assimilation systems making progress in the use
• f radar and satellite observations

– Radiance assimilation still problematic

WMO Data Assimilation Symposium, College Park Oct 7-11 2013 18

Forecast impact experiment from Dec. 2010 to Jan. 2011

Impact Impact / Obs. number

WMO Workshop on the Impact of Various Observing Systems on NWP Sedona – 22-25 May 2012

Could we use this type of FSO information to rank observing systems by impact per dollar?

1 9 Of course we can! Simply divide the impact by the cost of running the system and come up with a third “impact per dollar” bar chart!

Obs impact and value, versus cost

• Armed with the following three pieces of information

– Cost of acquiring the observations (difficult but not impossible to acquire this information) – Overall economic benefits of products derived from

• bservations (can be WAG’ed or properly analyzed by trained

economists) – Individual contribution of observing systems to NWP skill (done through RRR, previous slide),

we could in fact take the reasoning on the previous slide one step further and provide cost/benefit numbers for individual components of the GOS; it is not an easy exercise, but probably doable

WMO Data Assimilation Symposium, College Park Oct 7-11 2013 20

Should we use DA methodology for Cost Benefit analysis for observations?

• Caveat!

– The respective contributions of the components of the GOS vary from center to center, even at comparable levels of skill – The contributions vary depending on which other GOS components are used in the experiments – OSE results are expensive to acquire, and often inconclusive – FSO impacts are relative; even if the overall forecast performance is poor, some observing systems may stand out due to their large share in a modest improvement – The sum of the contributions always add up to 100, down until the very last GOS component standing

• When are we done cutting?

WMO Data Assimilation Symposium, College Park Oct 7-11 2013 21

Final remarks

• The economic impact of weather is relatively well

understood

• In contrast, the economic impact of weather

prediction is generally not well studied and documented

• The cost/benefit of meteorological observations are a

subject of intense interest among program managers and decision makers

– The costs are incurred (and known) mostly at the regional levels, but the benefits are realized and assessed globally – Tendency to focus too much on NWP diagnostics due to their compelling nature

WMO Data Assimilation Symposium, College Park Oct 7-11 2013 22

Final remarks (II)

• The NWP and data assimilation community is being

dragged into this whether we like it or not

– Victim of our own success – Does our community need to collectively (e.g. through WMO/CAS, THORPEX) adopt a position on this issue?

• We need to remind people that while NWP is

important and can provide very compelling metrics, it is only one of many application areas supported by the GOS

• New, widely adopted community standard metrics

needed?

– Not much room for further improvement of the 500 hPa AC scores

WMO Data Assimilation Symposium, College Park Oct 7-11 2013 23