ensemble verification old scores new perspectives
play

Ensemble verification: Old scores, new perspectives Sabrina Wahl, - PowerPoint PPT Presentation

Jan 18, 2023 •382 likes •600 views

Ensemble verification Quantile score and decomposition Generalization to CRPS Ensemble verification: Old scores, new perspectives Sabrina Wahl, Petra Friederichs, Jan Keller WMO Verification Workshop Berlin, May 2017 Sabrina Wahl Ensemble

  1. Ensemble verification Quantile score and decomposition Generalization to CRPS Ensemble verification: Old scores, new perspectives Sabrina Wahl, Petra Friederichs, Jan Keller WMO Verification Workshop Berlin, May 2017 Sabrina Wahl Ensemble verification

  2. Ensemble verification Quantile score and decomposition Generalization to CRPS equally probable simulations ensemble forecast of numerical model ➡ calibration: rank (pit) histogram, beta score ➡ discrimination: generalized discrimination score translation, interpretation, ➡ sharpness: prediction interval post-processing pdf, cdf, mean, sd, probabilistic forecast quantiles, probabilities,… xf M p(x |y ) i i ➡ proper scoring rules: 
 p(x ) f xi -1 -1 CRPS, Brier score, quantile score, 
 H G model space logarithmic score, MSE, MAE, … p(y ) N i p(y |x ) f f observation space ti tf time Fig. 1 from Stephenson et al. (2005) Sabrina Wahl Ensemble verification

  3. Ensemble verification Quantile score and decomposition Generalization to CRPS 35 ● 30 ● temperature (°C) ● 25 ● ● ● ● ● ● ● ● ● ● ● ● 20 ● ● ● ● ● ● ● ● ● 15 ● ● ● ● ● 10 ● ● 5 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 days ensemble forecast in terms of empirical distribution boxplot represents forecast distribution in terms of quantiles evaluation of ensemble members as quantiles Sabrina Wahl Ensemble verification

  4. Ensemble verification single quantile Quantile score and decomposition multiple quantiles Generalization to CRPS Verification-framework for quantiles score for quantile forecasts q τ when y is the event that materializes, with τ ∈ ( 0 , 1 ) the probability level � | y − q τ | τ if y ≥ q τ S Q ( q τ , y ) = ρ τ ( y − q τ ) = | y − q τ | ( 1 − τ ) if y < q τ empirical quantile score from a set of N forecast-observation pairs N QS ( τ ) = 1 � ρ τ ( y i − q τ, i ) N i = 1 decomposition of the quantile score (Bentzien and Friederichs, 2014) N QS ( τ ) = 1 � ρ τ ( y i − q τ, i ) = UNC ( τ ) − RES ( τ ) + REL ( τ ) N i = 1 Sabrina Wahl Ensemble verification

  5. Ensemble verification single quantile Quantile score and decomposition multiple quantiles Generalization to CRPS 50.00 ● ● ● Calibration : quantile 20.00 ● ● ● ● ● ● ● ● ● reliability diagram ● ● ● ● 10.00 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 5.00 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● forecast intervals I k ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● observation ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 2.00 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● y ( k ) ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● τ : conditional observed ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1.00 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● quantile in I k ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.50 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.20 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● discrete values y ( k ) τ , q ( k ) ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● with ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● τ ● ● ● ● ● ● 0.10 ● ● ● ● ● ● ● ● ● k = 1 , ..., K ≤ N ● ● ● ● ● ● ● ● ● ● ● 0.05 ● 0.05 0.20 0.50 2.00 5.00 20.00 quantile forecast Sabrina Wahl Ensemble verification

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Using the SAL technique for ensemble forecast verification Sabine Radanovics Until March 2017:

Using the SAL technique for ensemble forecast verification Sabine Radanovics Until March 2017:

Why spatial verification? SAL Ensemble SAL MesoVICT data Results Conclusions Using the SAL technique for ensemble forecast verification Sabine Radanovics Until March 2017: LSCE/ESTIMR, now: CNRM/GMME/MICADO May 10, 2017 1 / 19 Why

440 views • 29 slides
Chapter 5: z-Scores : Location of Scores Chapter 5: z-Scores : Location of Scores and Standardized

Chapter 5: z-Scores : Location of Scores Chapter 5: z-Scores : Location of Scores and Standardized

Chapter 5: z-Scores : Location of Scores Chapter 5: z-Scores : Location of Scores and Standardized Distributions I t Introduction to z-Scores d ti t S In the previous two chapters, we introduced th the concepts of the mean and the

845 views • 30 slides
Boosting (ensemble) Module 4 - Ensemble classifiers - Objectives module 4: boosting (ensemble

Boosting (ensemble) Module 4 - Ensemble classifiers - Objectives module 4: boosting (ensemble

Boosting (ensemble) Module 4 - Ensemble classifiers - Objectives module 4: boosting (ensemble models) LEARNING PERFORMANCE REPRESENTATION DATA PROBLEM RAW DATA CLUSTERING EVALUATION FEATURES UCI datasets unigrams 20newsgroups SUPERVISED

926 views • 34 slides
Applied to Clouds and Precipitation Categorical Scores Contingency Table D-A-CH Subregions SAL

Applied to Clouds and Precipitation Categorical Scores Contingency Table D-A-CH Subregions SAL

Spatial Verifications vs. Point Verifications Applied to Clouds and Precipitation Categorical Scores Contingency Table D-A-CH Subregions SAL 1.) Motivation 4.) SAL Verification 2.) Point Scores 5.) Case Applications 3.) VERA Analyses 6.)

223 views • 21 slides
1/12/2011 Chapter 5: z-Scores : Location of Scores and Standardized Distributions Introduction to

1/12/2011 Chapter 5: z-Scores : Location of Scores and Standardized Distributions Introduction to

1/12/2011 Chapter 5: z-Scores : Location of Scores and Standardized Distributions Introduction to z-Scores In the previous two chapters, we introduced the concepts of the mean and the standard deviation as methods for describing an entire

264 views • 10 slides
Why z-scores? Transforming scores in order to make comparisons, especially when using

Why z-scores? Transforming scores in order to make comparisons, especially when using

Why z-scores? Transforming scores in order to make comparisons, especially when using different scales Gives information about the relative standing of a score in relation to the characteristics of the sample or population Location

648 views • 19 slides
Convergence of ensemble Kalman filters in the large ensemble limit and infinite dimension Jan

Convergence of ensemble Kalman filters in the large ensemble limit and infinite dimension Jan

Convergence of ensemble Kalman filters in the large ensemble limit and infinite dimension Jan Mandel Department of Mathematical and Statistical Sciences University of Colorado Denver Based on joint work with Loren Cobb, Evan Kwiatkowski, and

583 views • 25 slides
Stochastic Physics Perturbations For Ensemble Forecast Yuejian Zhu Ensemble Team Environmental

Stochastic Physics Perturbations For Ensemble Forecast Yuejian Zhu Ensemble Team Environmental

Stochastic Physics Perturbations For Ensemble Forecast Yuejian Zhu Ensemble Team Environmental Modeling Center NCEP/NWS/NOAA Acknowledgements: Philip Pegion . , Walter Kolczynski, Dingchen Hou and Xiaqiong Zhou Special thanks to IITM and Dr.

1.33k views • 54 slides
Developing Scale Scores &amp; Cut Scores for On-Demand Assessments of Individual Standards Nathan

Developing Scale Scores &amp; Cut Scores for On-Demand Assessments of Individual Standards Nathan

Developing Scale Scores &amp; Cut Scores for On-Demand Assessments of Individual Standards Nathan Dadey 1 , Shuqin Tao 2 , and Leslie Keng 1 1 2 NCME - New York, NY April 16th, 2018 Context Much work has been done on improving a single

319 views • 29 slides
What is it? Instrument or ensemble? Lars Bo Andersen, Humans and IT research seminar, 13/5-2015

What is it? Instrument or ensemble? Lars Bo Andersen, Humans and IT research seminar, 13/5-2015

What is it? Instrument or ensemble? Lars Bo Andersen, Humans and IT research seminar, 13/5-2015 Instrument &lt;&gt; Ensemble Our instruments Concepts and methods Empirical stuff Ensembles From French ensemble &quot;all the parts of a thing

285 views • 12 slides
State Song &amp; Dance Ensemble LIETUVA proposal of cooperation Who are we? We are

State Song &amp; Dance Ensemble LIETUVA proposal of cooperation Who are we? We are

State Song &amp; Dance Ensemble LIETUVA proposal of cooperation Who are we? We are Lithuania! Ensemble Lietuva consisting of you , me and all of us together. True, authentic, with a long history of its own as an ensemble, genuine and

657 views • 16 slides
Ensemble Learning 4/10/17 Ensemble Learning Hypothesis Space: Supervised learning (data has

Ensemble Learning 4/10/17 Ensemble Learning Hypothesis Space: Supervised learning (data has

Ensemble Learning 4/10/17 Ensemble Learning Hypothesis Space: Supervised learning (data has labels) Classification (labels are discrete) Also regression, but the algorithms differ. The type of mapping that can be learned depends

377 views • 11 slides
A non-Gaussian analysis scheme using rank histograms for ensemble data assimilation Sammy Metref 1

A non-Gaussian analysis scheme using rank histograms for ensemble data assimilation Sammy Metref 1

Non-Gaussian Data Assimilation Multivariate Rank Histogram Filter A few results Outcomes and perspectives A non-Gaussian analysis scheme using rank histograms for ensemble data assimilation Sammy Metref 1 , E.Cosme 1 , C.Snyder 2 and P.Brasseur

430 views • 28 slides
Midterm Scores Min 1Q Median 3Q Max 23 58 68 83 100 The exam will be curved, with

Midterm Scores Min 1Q Median 3Q Max 23 58 68 83 100 The exam will be curved, with

Midterm Scores Min 1Q Median 3Q Max 23 58 68 83 100 The exam will be curved, with roughly 70100 being in the A/B range Scores less than 40 are a matter of concern Students who do substantially better on the final will have their

493 views • 21 slides
Announcements Monday, August 27 Make sure your poll scores are in the gradebook. (Poll scores

Announcements Monday, August 27 Make sure your poll scores are in the gradebook. (Poll scores

Announcements Monday, August 27 Make sure your poll scores are in the gradebook. (Poll scores dont count until this Wednesday, 8/29.) WeBWorK 2.1 due on Wednesday at 11:59pm. The first quiz is on Friday, during recitation. It covers

314 views • 18 slides
Parent Seminar Welcome! PSAT Scores SAT vs. ACT Next Steps Overview New PSAT Score Report

Parent Seminar Welcome! PSAT Scores SAT vs. ACT Next Steps Overview New PSAT Score Report

Parent Seminar Welcome! PSAT Scores SAT vs. ACT Next Steps Overview New PSAT Score Report TEST CHANGES Your Scores Your Scores Your Scores (cont.) National Merit Next Steps TEST CHANGES PSAT &amp; New SAT Scoring SAT: 400-1600

490 views • 23 slides
Higher Levels of Learning Diana Skrzydlo, Continuing Lecturer Outline Learning Taxonomies

Higher Levels of Learning Diana Skrzydlo, Continuing Lecturer Outline Learning Taxonomies

Designing Exams to Test Higher Levels of Learning Diana Skrzydlo, Continuing Lecturer Outline Learning Taxonomies Assessment is Curriculum Higher Level Questions Examples How you can use these ideas Learning Taxonomies

283 views • 17 slides
Application Roadmap New Markets Tax Credit Program PREPARED ON May 1st, 2018 The CDFI Fund is

Application Roadmap New Markets Tax Credit Program PREPARED ON May 1st, 2018 The CDFI Fund is

2018 NMTC Allocation Application Roadmap New Markets Tax Credit Program PREPARED ON May 1st, 2018 The CDFI Fund is an equal opportunity provider. Roadmap Overview: The purpose of this presentation is to direct Applicants to important

704 views • 38 slides
EAST BALT AT A GLANCE Leading foodservice-focused company that supplies baked goods to Quick

EAST BALT AT A GLANCE Leading foodservice-focused company that supplies baked goods to Quick

EAST BALT BAKERIES July 2017 ACQUISITION EAST BALT AT A GLANCE Leading foodservice-focused company that supplies baked goods to Quick Service Restaurants (QSR) across the world Operates 21 bakeries in 11 countries 2,200 associates Product

474 views • 11 slides
Think? Minjeong Kim Yujin Kwon Yongdae Kim 1 Stellar 2 Stellar $150 - Problem of

Think? Minjeong Kim Yujin Kwon Yongdae Kim 1 Stellar 2 Stellar $150 - Problem of

Is Stellar As Secure As You Think? Minjeong Kim Yujin Kwon Yongdae Kim 1 Stellar 2 Stellar $150 - Problem of central authority - Cross-border Payment is too slow and costly - Try to solve those problems with blockchain 3 Stellar

792 views • 46 slides
Getting Started at UFV! Overview Where do I start? Program Options Entrance

Getting Started at UFV! Overview Where do I start? Program Options Entrance

Getting Started at UFV! Overview Where do I start? Program Options Entrance Requirements Qualifying Studies &amp; Upgrading Applying to UFV Next Steps Transfer Credits UFV Supports &amp; Services Questions

378 views • 16 slides
Investor Presentation May/June 2017 Forward-Looking Statements Under the Private Securities

Investor Presentation May/June 2017 Forward-Looking Statements Under the Private Securities

Investor Presentation May/June 2017 Forward-Looking Statements Under the Private Securities Litigation Act of 1995 This document may contain or incorporate by reference forward-looking statements as defined under the federal securities laws

507 views • 31 slides
PERSONALITY INDEXES For Hiring, Team Building, and the Bottom Line Presentation by Deb Harris /

PERSONALITY INDEXES For Hiring, Team Building, and the Bottom Line Presentation by Deb Harris /

PERSONALITY INDEXES For Hiring, Team Building, and the Bottom Line Presentation by Deb Harris / TeamWorx, LLC Kolbe Concept Property of Kathy Kolbe and Kolbe Corp. INDEXES USED Have you used index at work? Did it have a lasting impact?

469 views • 32 slides
Cost Benefit Analysis ECN 240 CMD ECN 240 Cost Benefit Analysis Intro Cost Benefit Analysis

Cost Benefit Analysis ECN 240 CMD ECN 240 Cost Benefit Analysis Intro Cost Benefit Analysis

Intro Cost Benefit Analysis CEA QALYs Cost Benefit Analysis ECN 240 CMD ECN 240 Cost Benefit Analysis Intro Cost Benefit Analysis CEA QALYs What is economic evaluation? If the government is going to take on a project or intervene, we

593 views • 25 slides

More recommend