The challenge of bounded, non-Gaussian, non- linear and multi-scale - PowerPoint PPT Presentation

The challenge of bounded, non-Gaussian, non- linear and multi-scale variables Craig H. Bishop University of Melbourne, Parkville, Victoria, Australia

Some Motivation • Major source of model error in weather and climate prediction models is clouds and precipitation. • Until recently, observations of clouds and precipitation have been studiously avoided in DA – not surprising then that they are a big source of model error. • Clouds/precip DA problem cluttered with bounded, non-Gaussian, non- linear and mult-scale variables. 2

A cloud DA thought experiment Imagine an infinite number of replicate Earths -same solar/GHG forcing -each giving an independent realization of today’s climate -same observation types and locations but different random observation-errors 3

Visible Geostationary Imagery that we do not yet assimilate 4

The ideal objective of Data Assimilation Each point represents a (u, u 2 ) pair from a distinct replicate Earth Imagine an infinite number of replicate Earths -same solar forcing - each giving an independent realization of today’s prior -same observation types and locations but different random observation-errors 5

The ideal objective of Data Assimilation Each cross represents a (u, y) pair from a distinct replicate Earth Imagine an infinite number of replicate Earths -same solar forcing - each giving an independent realization of today’s prior -same observation types and locations but different random observation-errors Error standard deviation of unbiased observation of a bounded variable must tend to zero as truth tends to zero. Bishop (2019, Q. J. Roy. Met. Soc.) 6

The ideal objective of Data Assimilation On our Earth, the observation of u 2 gives y=1.71 Imagine an infinite number of replicate Earths -same solar forcing - each giving an independent realization of today’s prior -same observation types and locations but different random observation-errors 7

Ideal Data Assimilation (DA) in a simple model Prior and posterior pdf of u 2 are like gamma pdfs. Highly non-Gaussian. Ideal DA gives the posterior pdf of replicate Earths having the same y value as our Earth’s y value. Ideal posterior pdf is bi-modal. Bi-modality caused by non-linearity 8

Current DA: 4DVar-No-Outer-Loop (US Navy) and EnKF (DWD) EnKF & 4DvarNoOuterLoop (4DVarNOL) posterior pdf of u 2 is highly inaccurate. Also, analyzed u 2 values are not equal to the square of analyzed u values 4DVarNOL and EnKF estimate the posterior pdf using linear regression and perturbed observations ( ) ( ) ( ) ( ) ( ) ( ) - é ù 1 é ù a 2 2 2 é ù 2 = f + f fT f fT + e o - f u u covar u y covar y y y u ê ú ë û ê ú ë û ë û i i i i i i i i ( ) ( ) ( ) ( ) - 1 é ù 2 é ù a = f + f fT f fT + e o - f u u co var u y covar y y y u ë û ê ú ë û i i i i i i i i EnKF/4DVarNOL posterior pdf of u is very poor. Fails due to linear, Gaussian assumptions 9

Current DA: Incremental 4DVar (4DVar-with-outer-loop) 4DVar posterior pdf of u 2 is highly inaccurate. However, analyzed u 2 values are now equal to the square of analyzed u values 4DVar uses perturbed observations. Each posterior 4DVar posterior pdf of u is still member is a local extreme value poor. of a Gaussian approximation to the true posterior pdf. Fails due to Gaussians assumption and the presence of multiple extrema (non-linearity) 10

Overview of Challenges 1. Observations of bounded non-Gaussian variables Ø EnKF’s, 4DVar (with Gaussian assumptions) inaccurate 2. Assimilation of an on/off variable like rain when no rain is in the (ensemble) forecast Ø EnKFs, 4DVar and Particle Filters very unsatisfactory 3. Non-linear relationship between model variables and observed variables (cf above example and Leonhard Scheck talk) Ø EnKFs and VAR-with-no-outer-loop or grossly inaccurate TLM/adjoint are inaccurate. 4. Multi-scale error structures Ø Somewhat challenging for all methods 11

Overview of talk Improvements to GIGG-EnKF (Bishop, 2016, QJRMS.) 1. Background 2. GIG-EnKF solution to Bayes’ theorem for gamma prior and inverse- gamma-likelihood is now precise as K => infinity - previously just approximate. Significance: Rigorous basis for GIG for all moments 3. Test of standard GIG for tropical cyclone surface wind energy assimilation problem: Significance: Standard GIG better than EAKF/EnKF for this problem. 4. Local iterative regression to account for non-linearity in observation operator. Significance: Greatly reduces analysis error. 5. Rigorous approach for dealing with on-off variables (rain, cloud, fire, etc) with gamma based delta function. Significance: Justifies ignoring dry members when rain is observed. 6. Discussion of ways to include GIGG ideas/tools in other DA schemes. 12

Background: A typical EnKF serial observation assimilation scheme = for j 1: p ; % where is the number of observations p = a Step 1: Do univariate Gaussian assimilation of y to obtain y , i 1,2,..., K j ji Step 2: Find corresponding analysis ensemble for observations a nd model variables ( ) f f covar y , y ( ) k j = + - = = a f a f y y y y , for k 1,2,..., ; p i 1,2,..., K ( ) ki ki ji ji f var y j ( ) f f covar x , y ( ) µ j = + - µ = = a f a f x x y y , for 1,2,..., ; n i 1,2,..., K ( ) µ i µ i ji ji f var y j Step 3: Let the analysis ensemble be the prior ensemble for the next observation f = a = = y y , for k 1,2,..., ; p i 1,2,..., K ki ki f = a µ = = x x , for 1,2,..., ; n i 1,2,..., K µ µ i i end 13

Background: Gaussian pdfs versus bounded pdfs Haboob, Iraq. 27 April, 2005 Extratropical Cyclone Xynthia 2010, 63 fatalities, losses $2-4B, linked to H20 plume 14 14

Background: Gaussian anamorphosis problematic • Gaussian anamorphosis transforms a non-Gaussian variable to a Gaussian variable via a non-linear function; e.g. taking the log of a log- normal variable makes it Gaussian • In observation space, this is problematic because a non-linear transform of an error prone unbiased observation will create a biased observation of the non-linear function of the truth. • In model space, this is problematic because while a good DA scheme will deliver a posterior ensemble whose variance is equal to the mean square error (mse) of the mean in the transformed space, in general, analysis ensemble variance will not equal mse once the transformation is undone. The bar on the right shows that imperfect Gaussian anamorphosis via non- linear transform approach led to overdispersive ensemble in the case considered in Bishop (2019, Q.J. Roy. Met. Soc.) 15 15

Background: Gaussian anamorphosis problematic • Gaussian anamorphosis transforms a non-Gaussian variable to a Gaussian variable via a non-linear function; e.g. taking the log of a log- normal variable makes it Gaussian • In observation space, this is problematic because a non-linear transform of an error prone unbiased observation will create a biased observation of the non-linear function of the truth. • In model space, this is problematic because while a good DA scheme will deliver a posterior ensemble whose variance is equal to the mean square error (mse) of the mean in the transformed space, in general, analysis ensemble variance will not equal mse once the transformation is undone. Non-linear transformation of observations and/or model variables unsatisfactory The bar on the right shows that imperfect Gaussian anamorphosis via non- linear transform approach led to overdispersive ensemble in the case considered in Bishop (2019, Q.J. Roy. Met. Soc.) 16 16

A prior Gamma pdf gamma prior r prior ( ) c prior mean 17

Inverse-Gamma pdf of obs given truth y , the observed value gamma prior r prior ( ) c = L y ( _ possible c | y ), the inverse gamma pdf that gives the pdf of obs that could occur if the truth was equal to the vertical prior mean "dot-dash" line. 18

The likelihood function y , the observed value gamma prior r prior ( ) c L y c ( | ) with fixed at the observed value. y prior mean 19

The posterior pdf is then a gamma y , the observed value posterior mean r L y c ( | ) ( ) c gamma prior r = prior ( | ) c y (a gamma posterior) r prior ( ) c ¥ post ò r L y c ( | ) ( ) c dc prior 0 L y c ( | ) prior mean 20

Equation for posterior mean y , the observed value - é ù æ ö æ ö ( ) ( ) 1 1 1 1 1 ˆ ˆ ˆ ˆ = + + - + ç ÷ ê ç ÷ ú P P R 1 R ç ÷ ç ÷ r r r r y c c c ê ú è ø è ø ë û post prior prior ˆ ˆ where P and R are type 2 relative prior and observation r r ( ) var c c s 2 ˆ prior = = prior variances; e.g. P ( ) ( ) r 2 µ 2 + s 2 + c var c prior prior Posterior mean equation has Kalman like gain but everything else is inverted ! (See Bishop, 2016, QJRMS and Bishop, 2018 for details of GIG method) 21