Near real time assessment of operational oceanography products: - - PowerPoint PPT Presentation

▶
near real time assessment of
SMART_READER_LITE
LIVE PREVIEW

Near real time assessment of operational oceanography products: - - PowerPoint PPT Presentation

May 03, 2023 250 likes •602 views

Near real time assessment of operational oceanography products: advances in the GOV community, overview of Class 4 intercomparison and multimodel assessment approaches Greg Smith Environment Canada F. Hernandez Mercator-Ocean M. Martin, A.

slide-1
SLIDE 1

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013 GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Near real time assessment of

  • perational oceanography products:

advances in the GOV community,

  • verview of Class 4 intercomparison

and multimodel assessment approaches

Greg Smith Environment Canada

  • F. Hernandez

Mercator-Ocean

  • M. Martin, A. Sellar

UK Metoffice With thanks to the many IV-TT contributors!

www.godae-oceanview.org

slide-2
SLIDE 2

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

  • Pursues activities developed during GODAE
  • Develop a framework for comparing outputs of the

various operational ocean forecasting systems (OOFS)

– leads to improvements to the systems and to the quality of products from those systems – provides a framework for scientific discussions on forecasting system performance assessment, on ocean analysis and validation from numerical simulations – offers a demonstration of the work in GODAE OceanView and has the potential to increase the visibility to the external community

The Intercomparison and Validation Task Team:

slide-3
SLIDE 3

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

The Intercomparison and Validation Task Team

Coordinates and promotes the development of scientific validation and intercomparison of operational oceanography systems

– Mainly for short term forecast assessment, but benefit for/from reanalysis assessment, and medium range forecast assessment (eg, seasonal forecast), – include the definition of metrics to assess the quality of analyses and forecasts (e.g. forecast skills) both for physical and biogeochemical parameters and the setting up of specific global and regional intercomparison experiments – metrics related to specific applications also considered – links with the OSE-TT – links with the JCOMM ET-OOFS team for operational implementation – cooperation with CLIVAR/GSOP for climate issues.

slide-4
SLIDE 4

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

  • Main outcomes from the IV-TT:

– Class 4 metrics, forecast skill monitored in near real time among 6 operational centres – Class 1 metrics, ensemble and multi-model approach – Participation to the GSOP Intercomparison project (CLIVAR/GOV initiative, presented by M. Balmaseda)

  • Advances in real time monitoring and validation

methodology implemented in operational

  • ceanography centres: highlights with some examples
  • How operational validation is provided to users in term
  • f Product Quality: some illustration and examples
  • Conclusions

Outline of this talk

slide-5
SLIDE 5

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Class 4 metrics

  • Create a real-time system for generating model-equivalents

to a common set of observations

  • Major effort made to ensure consistency in methods and

comparisons

  • Initial comparison:

– Jan-Jun 2013, 6 institutions participated

  • Systems evaluated using:

– SST from drifting buoys – Argo temperature and salinity profiles – Sea level anomaly

  • Statistics compiled for analyses as well as as a function of lead

time

slide-6
SLIDE 6

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

USGODAE – In-situ drifters

Courtesy Andy Ryan, UK Metoffice

slide-7
SLIDE 7

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

ARGO – Salinity profiles

slide-8
SLIDE 8

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Taylor diagram of SST around Australia

  • Taylor diagram

provides another perspective

See poster from Divakaran for details

slide-9
SLIDE 9

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Class4 intercomparison

  • Radar plots of RMS error computed for the period of 1st January to 30th September 2013, for SLA, SST and

in the layer 0-500m for temperature and salinity

  • Systems involved : PSY3 (Mercator OCEAN), FOAM (UK Met Office), GIOPS (CMC), RTOFS(NOAA/NCEP),

OceanMAPS (Bureau Of Meteorology)

  • For salinity PSY3 shows the best performance
  • For temperature, except OceanMAPS (lower resolution except in Australian region), four other systems

are very close

  • FOAM SST is the most accurate system with GIOPS approaching in terms of forecast

See poster from C. Regnier for details

slide-10
SLIDE 10

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Mapping of RMS error

  • We are working with the July-August-September 2013 quarter with the five systems
  • The ocean is divided in regular bins of 2°x2°
  • In the left we show the system for which the RMS error is the lowest
  • In the right the corresponding RMS value is displayed for each bin
  • Maps exhibit a regional distribution of the error
  • This kind of map is a way to compare system with a given criterion
  • Need to cross several criteria for robust conclusion

See poster from C. Regnier for details

slide-11
SLIDE 11

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Class 1 Multi-model and ensemble validations

slide-12
SLIDE 12

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Multimodel ensemble SST validation

  • Demonstrates value
  • f multimodel

ensemble

  • Results sensitive to

method used to calculate ensemble mean:

– simple – weighted – kmeans

See poster from T. Spindler for details

slide-13
SLIDE 13

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

CLASS1: drifter trajectory forecast

  • How can we best use

OOFS to determine the position of the AF447 crash?

  • Use 15m drifter

trajectories to estimate error from 6 systems

  • Demonstrates added

value of multimodel ensemble

7-day fcst 3-day 1-day 1-day forecast: in 50% of the forecasts, the distance between the drifter and the end point trajectory is less than 20km ~20 km

slide-14
SLIDE 14

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Evaluating oil-spill dispersion forecasting in the Northern Aegean Sea

See poster by S. Sofianos et al. for details Highlights the importance of ensemble methods!

slide-15
SLIDE 15

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

  • Sea ice forecast verification using

analyses is unreliable due to large analysis uncertainty

  • Patchy coverage, foot-print

issues, data reliability

  • Representation of leads
  • Only evaluate points where the

analysis changes by more than 10%

 Pro: Only includes points where we

have confidence in ice analysis

  • Focus on ice edge in particular

Con: Excludes areas of incorrect

model changes

  • E.g: coastal polynyas, false

alarms along the ice edge

Sea ice verification: Analysis tendencies

7day GIOPS forecast error for 2011-03-30 Van Woert et al. (2004) See poster from G. Smith for details

slide-16
SLIDE 16

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Sea ice verification : Analysis tendencies

  • Over a full annual cycle, this method provides a reasonably reliable idea
  • f forecast skill
  • But what about leads, coastal polynyas and false alarms in the marginal

ice zone?

GIOPS 7 day forecasts for 2011 GIOPS Analysis persistence Van Woert et al. (2004)

slide-17
SLIDE 17

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Contingency table analysis

  • Comparison with IMS Analyses:

– Interactive Multisensor Snow and Ice Mapping System (NOAA-NIC) – Daily Northern Hemisphere ice analyses on 4km grid (ice/water)

  • Evaluation Methodology:

– Calculate contingency table values using 0.4 ice concentration cutoff

IMS Ice IMS No ice Forecast Ice

Hit ice False Alarm

Forecast No ice

Miss Hit water

  • Proportion Correction Ice:

PCI = Hit ice / (Hit ice + Miss) [0,1]

  • Proportion Correction Water:

PCW = Hit water / (Hit water + False Alarm) [0,1]

slide-18
SLIDE 18

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Contingency table verification

  • Proportion Correct Ice

– Ability to forecast ice formation and advection – PCI= Hits ice / (Hits + Misses) – Shows skillful forecasts along most ice edges

  • Proportion Correct Water

– PCW= Hits water / (Hits water + False Alarms) – Shows important false alarm rate missing from analysis tendency verification ΔPCI ( Fcst – Pers) ΔPCW ( Fcst – Pers) GIOPS 7 day forecasts for 2011

Ice? YIMS NIMS YFcst

Hit ice False Alarm

NFcst

Miss Hit water

Skill Error See poster from G. Smith for details

slide-19
SLIDE 19

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

BAL MFC: Developing ice edge metrics for the Baltic Sea

GODAE OceanView Symposium – November 2013

  • bs2mod

distance mod2obs distance

Observed ice concentration

  • bs ice edge

model ice edge min distance for each edge point

N i i

D N RMSdist

1 2

1

mod

n n N

  • bs 

 

mod _ mod, 2 1 mod, 2 _ , 2 mod 1 , 2 mod

... ...

n

  • bs
  • bs
  • bs

n

  • bs
  • bs

i

d d d d D 

Metric must be calculated sub-regionally to avoid long overland distances

Ice edge RMS distance

Find minimum distance from modeled ice edge for each observed ice edge cell and vice versa Take the RMS of these distances Straightforward interpretation for the users

Courtesy of P. Lagemaa

slide-20
SLIDE 20

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Conclusions

  • Direct benefit from IV-TT activity:

– several OOFS have now implemented routine Class-4 and Class-1 delivery, allowing enhanced monitoring for these teams, and definition of new validation approaches – Operational oceanography validation experts

  • rganized similar to that in the atmospheric

community

slide-21
SLIDE 21

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Conclusions

  • Direct benefit from IV-TT activity:

– several OOFS have now implemented routine Class-4 and Class-1 delivery, allowing enhanced monitoring for these teams, and definition of new validation approaches – Operational oceanography validation experts

  • rganized similar to that in the atmospheric

community

slide-22
SLIDE 22

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Conclusions (con’t)

  • General overview:

– Validation techniques are implemented in most OOFS, global or regional – New metrics are defined to satisfy specific user needs and down stream applications

slide-23
SLIDE 23

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Thanks!

What is the role of GOV in improving Polar Predictions?

slide-24
SLIDE 24

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Sea Level Forecast Skill at Mercator Océan

Improvement of the forecast / persistence

Global: ¼° old ¼° new Zoom: 1/12° old 1/12° new

  • Implemented routinely, based on Class-4

metrics

  • Statistics published every 3 months
  • Used here to compare former and new
  • peratioanl system
slide-25
SLIDE 25

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Communicate Product Quality to users:

  • Regular reporting:

– QuOVaDis quarterly report by Mercator Océan – MyOcean2 Quality Information Docucments, complemented by quarterly Class-4 reporting – …..

  • Real time reporting

– Website

slide-26
SLIDE 26

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

« NARVAL »: regional validation webserver for the MyOcean2 IBI area

n

2 NARVAL modes

n

On-line “Real-time” mode: Val procedures launched after daily cycle

n

Delayed mode Component: Val procedures on specific metrics and statistics covering longer periods (i.e. monthly, seasonal and annual)

n

NARVAL updated to generate automatically Statistics for MyO Quarterly Validation Reports.

n

Intercompares with same metrics available MyO fcst in same area. eg, 3 in the western Med sea (MFS, Mercator, IBI)

Courtesy of Puertos del Estado and MyOcean2.

MyOcean2 V3 Acceptance & V4 Product Design Workshops – Madrid 28,29th January 2013

Partial snapshots of NARVAL Real-Time & Delayed-Modes web pages

slide-27
SLIDE 27

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Inter-comparison partners

Institution System Ocean Model Resolution UK Met Office FOAM NEMO ¼ degree Bureau of Meteorology OceanMAPS mom4p1 1 degree (1/10 around Australia) NOAA/NWS/NCEP/EMC/MMAB RTOFS HYCOM 1/12 degree Mercator-Ocean PSY3 NEMO ¼ degree Canadian Meteorological Centre CONCEPTS-GIOPS NEMO ¼ degree REMO HYCOM ATLe0.25 ¼ degree (Atlantic only)

  • Results available for all partners for January-June 2013
slide-28
SLIDE 28

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

ARGO – Temperature profiles

slide-29
SLIDE 29

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

ARGO – Temperature profiles

slide-30
SLIDE 30

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

ARGO – Salinity profiles

slide-31
SLIDE 31

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

CLASS 1: drifter trajectory forecast

 Scientific question: How to make the best trajectory forecast in the context of operational oceanography ?

6 products used:

  • GLORYS
  • PSY2
  • PSY3
  • NCOM
  • HYCOM
  • CLS
slide-32
SLIDE 32

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

CLASS 1: drifter trajectory forecast

slide-33
SLIDE 33

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

CLASS1: drifter trajectory forecast

7-day fcst 3-day 1-day 1-day forecast: in 50% of the forecasts, the distance between the drifter and the end point trajectory is less than 20km ~20 km

slide-34
SLIDE 34

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

CLASS1: drifter trajectory forecast

Main conclusions:

  • Trajectory forecasts is a nice way to evaluate OOFS skill
  • The added value of an ensemble of forecasts is shown
  • The ensemble filters out the error

Smoothing increases Error RMS (km)