Improving the conditioning of estimated covariance matrices Jemima - PowerPoint PPT Presentation

Improving the conditioning of estimated covariance matrices Jemima M. Tabeart Supervised by Sarah L. Dance, Nancy K. Nichols, Amos S. Lawless, Joanne A. Waller (University of Reading), David Simonin (MetOffice@Reading) Additional collaboration Stefano Migliorini and Fiona Smith (Met Office, Exeter) January 8, 2019 Jemima M. Tabeart (UoR, NCEO) Reconditioning covariance matrices January 8, 2019 1 / 15

Motivation Including correlated observation error allows us to maximise the information content of observations. But diagnosed correlated covariance matrices have caused problems with convergence of the data assimilation minimisation procedure. Need to treat diagnosed matrices (symmetry, positive definiteness). Jemima M. Tabeart (UoR, NCEO) Reconditioning covariance matrices January 8, 2019 2 / 15

What is reconditioning? Methods which can be applied to matrices to reduce their condition number, while retaining underlying matrix structure. Examples of methods: Thresholding Tapering General regularisation methods. Jemima M. Tabeart (UoR, NCEO) Reconditioning covariance matrices January 8, 2019 3 / 15

What is reconditioning? Methods which can be applied to matrices to reduce their condition number, while retaining underlying matrix structure. Examples of methods: Thresholding Tapering General regularisation methods. We will focus on two methods that are used in NWP. Both work by altering eigenvalues of the covariance matrix. Jemima M. Tabeart (UoR, NCEO) Reconditioning covariance matrices January 8, 2019 3 / 15

What is reconditioning? Methods which can be applied to matrices to reduce their condition number, while retaining underlying matrix structure. Examples of methods: Thresholding Tapering General regularisation methods. We will focus on two methods that are used in NWP. Both work by altering eigenvalues of the covariance matrix. Reminder: If S ∈ R p × p is a symmetric and positive definite matrix with eigenvalues λ 1 ( S ) ≥ . . . ≥ λ p ( S ) > 0 then we can write the condition number in the L 2 norm as κ ( S ) = λ 1 ( S ) λ p ( S ) . If S is singular, we take κ ( S ) = ∞ . Jemima M. Tabeart (UoR, NCEO) Reconditioning covariance matrices January 8, 2019 3 / 15

The ridge regression (RR) and minimum eigenvalue (ME) methods Both methods improve the condition number of a covariance matrix by altering their eigenvalues to yield a reconditioned matrix with a user-defined condition number κ max . Figure: Illustration of recond methods: original spectrum (black), and spectrum reconditioned via ME and RR Jemima M. Tabeart (UoR, NCEO) Reconditioning covariance matrices January 8, 2019 4 / 15

Ridge regression method Idea: Add a scalar multiple of identity to R to obtain reconditioned R RR with κ ( R RR ) = κ max . Setting δ Define δ = λ 1 ( R ) − λ p ( R ) κ max . κ max − 1 Set R RR = R + δ I Jemima M. Tabeart (UoR, NCEO) Reconditioning covariance matrices January 8, 2019 5 / 15

Ridge regression method Idea: Add a scalar multiple of identity to R to obtain reconditioned R RR with κ ( R RR ) = κ max . Setting δ Define δ = λ 1 ( R ) − λ p ( R ) κ max . κ max − 1 Set R RR = R + δ I Similar to Steinian linear shrinkage [Ledoit and Wolf, 2004] Used at the Met Office [Weston et al, 2014]. Jemima M. Tabeart (UoR, NCEO) Reconditioning covariance matrices January 8, 2019 5 / 15

Theory of the ridge regression method Effect of RR on standard deviations : Σ RR = ( Σ 2 + δ I p ) 1 / 2 . (1) i.e. variances are increased by the reconditioning constant, δ . Jemima M. Tabeart (UoR, NCEO) Reconditioning covariance matrices January 8, 2019 6 / 15

Theory of the ridge regression method Effect of RR on standard deviations : Σ RR = ( Σ 2 + δ I p ) 1 / 2 . (1) i.e. variances are increased by the reconditioning constant, δ . Effect of RR on correlations : For i � = j , | C RR ( i , j ) | < | C ( i , j ) | (2) i.e. the magnitude of all off-diagonal correlations is strictly decreased. Jemima M. Tabeart (UoR, NCEO) Reconditioning covariance matrices January 8, 2019 6 / 15

Minimum eigenvalue method Idea: Fix a threshold, T, below which all eigenvalues of the reconditioned matrix, R ME , are set equal to T to yield κ ( R ME ) = κ max . Setting T: Set λ 1 ( R ME ) = λ 1 ( R ) Define T = λ 1 ( R ) /κ max . Set the remaining eigenvalues of R ME via � λ k ( R ) if λ k ( R ) > T λ k ( R ME ) = if λ k ( R ) ≤ T . (3) T We define Γ ( k , k ) = max { T − λ i , 0 } . Jemima M. Tabeart (UoR, NCEO) Reconditioning covariance matrices January 8, 2019 7 / 15

Minimum eigenvalue method Idea: Fix a threshold, T, below which all eigenvalues of the reconditioned matrix, R ME , are set equal to T to yield κ ( R ME ) = κ max . Setting T: Set λ 1 ( R ME ) = λ 1 ( R ) Define T = λ 1 ( R ) /κ max . Set the remaining eigenvalues of R ME via � λ k ( R ) if λ k ( R ) > T λ k ( R ME ) = if λ k ( R ) ≤ T . (3) T We define Γ ( k , k ) = max { T − λ i , 0 } . A variant of this method is used at the European Centre for Medium-Range Weather Forecasts (ECMWF) [Bormann et al, 2016]. Jemima M. Tabeart (UoR, NCEO) Reconditioning covariance matrices January 8, 2019 7 / 15

Theory of the minimum eigenvalue method Effect of ME on standard deviations: � 1 / 2 p � Σ ( i , i ) 2 + � V R ( i , k ) 2 Γ ( k , k ) Σ ME ( i , i ) = (4) k =1 Jemima M. Tabeart (UoR, NCEO) Reconditioning covariance matrices January 8, 2019 8 / 15

Theory of the minimum eigenvalue method Effect of ME on standard deviations: � 1 / 2 p � Σ ( i , i ) 2 + � V R ( i , k ) 2 Γ ( k , k ) Σ ME ( i , i ) = (4) k =1 This can be bounded by � 1 / 2 . Σ ( i , i ) 2 + T − λ p ( R ) � Σ ( i , i ) ≤ Σ ME ( i , i ) ≤ (5) Jemima M. Tabeart (UoR, NCEO) Reconditioning covariance matrices January 8, 2019 8 / 15

Theory of the minimum eigenvalue method Effect of ME on standard deviations: � 1 / 2 p � Σ ( i , i ) 2 + � V R ( i , k ) 2 Γ ( k , k ) Σ ME ( i , i ) = (4) k =1 This can be bounded by � 1 / 2 . Σ ( i , i ) 2 + T − λ p ( R ) � Σ ( i , i ) ≤ Σ ME ( i , i ) ≤ (5) Effect of ME on correlations: It is not evident how correlation entries are altered in general. This is due to the fact that the spectrum of R is not altered uniformly by this method. Jemima M. Tabeart (UoR, NCEO) Reconditioning covariance matrices January 8, 2019 8 / 15

Comparison of both methods Both methods increase (or maintain) standard deviations Jemima M. Tabeart (UoR, NCEO) Reconditioning covariance matrices January 8, 2019 9 / 15

Comparison of both methods Both methods increase (or maintain) standard deviations We can show that T − λ p ( R ) < δ which yields: � 1 / 2 < ( Σ ( i , i ) 2 + δ ) 1 / 2 = Σ RR ( i , i ) Σ ( i , i ) 2 + T − λ p ( R ) � Σ ME ( i , i ) ≤ Therefore RR increases standard deviations more than ME Jemima M. Tabeart (UoR, NCEO) Reconditioning covariance matrices January 8, 2019 9 / 15

IASI matrix - framework Interchannel correlations for a covariance matrix of satellite observation errors The UK Met Office diagnosed a correlated observation error covariance matrix in 2011. This was extremely ill-conditioned and crashed the system when used directly. Recondition 137 channels. Original condition number: 27703 . Jemima M. Tabeart (UoR, NCEO) Reconditioning covariance matrices January 8, 2019 10 / 15

Diagnosed IASI correlation matrix Jemima M. Tabeart (UoR, NCEO) Reconditioning covariance matrices January 8, 2019 11 / 15

IASI - change to standard deviations Figure: Standard deviations Σ (solid), Σ RR and Σ ME for κ max = 100. Recall κ ( R ) = 27703 . 45 for the original IASI matrix. Jemima M. Tabeart (UoR, NCEO) Reconditioning covariance matrices January 8, 2019 12 / 15

IASI - change to correlations Figure: Difference in correlations (a) ( C − C RR ) ◦ sign ( C ), (b) ( C − C ME ) ◦ sign ( C ), and (c) ( C ME − C RR ) ◦ sign ( C ). The colorscale is the same for (a) and (b) but different for (c). Jemima M. Tabeart (UoR, NCEO) Reconditioning covariance matrices January 8, 2019 13 / 15

Ongoing work (see Poster Session 1-23) Focused on improving conditioning and the relationship between condition number and convergence of a conjugate gradient type minimisation. But by reconditioning we are altering the problem we are solving! Studying impact of reconditioning on an operational system - 1D-Var QC procedure used at the Met Office. If you want to find out more, poster with results ‘ The impact of using reconditioned correlated observation error covariance matrices in the Met Office 1D-Var system ’ Jemima M. Tabeart (UoR, NCEO) Reconditioning covariance matrices January 8, 2019 14 / 15

Conclusions Developed new theory showing how two reconditioning methods alter covariance matrices theoretically. The ridge regression method increases standard deviations more than the minimum eigenvalue method. The ridge regression method moves all correlations closer to zero, whereas the minimum eigenvalue method can increase correlations. The ridge regression method changes most correlation entries by a larger amount than the minimum eigenvalue method in numerical testing. Jemima M. Tabeart (UoR, NCEO) Reconditioning covariance matrices January 8, 2019 15 / 15