Efficient Determination of the Hyperparameter via L-curve in Large - PowerPoint PPT Presentation

Efficient Determination of the Hyperparameter via L-curve in Large Scale Least Squares and Total Least Squares Problems Jörg Lampe joerg.lampe@tuhh.de Hamburg University of Technology Institute of Numerical Simulation Joint work with Heinrich Voß 8th GAMM Workshop on Applied and Numerical Linear Algebra TUHH Jörg Lampe RTLS GAMM Workshop ’08 1 / 27

Outline Background 1 LS and TLS problems RLS and RTLS problems Solving RLS and RTLS problems 2 Solving RLS problem via QEP Solving RTLS problem via QEPs Determining the hyperparamter 3 L-curve Numerical Examples Conclusions 4 TUHH Jörg Lampe RTLS GAMM Workshop ’08 2 / 27

Background LS and TLS problems Outline Background 1 LS and TLS problems RLS and RTLS problems Solving RLS and RTLS problems 2 Solving RLS problem via QEP Solving RTLS problem via QEPs Determining the hyperparamter 3 L-curve Numerical Examples Conclusions 4 TUHH Jörg Lampe RTLS GAMM Workshop ’08 3 / 27

Background LS and TLS problems Least Squares Consider overdetermined linear system A ∈ R m × n , b ∈ R m , m ≥ n Ax ≈ b , with A and b contaminated by noise. Least Squares (LS) approach: � Ax − b � 2 = min ! or equivalently � ∆ b � 2 = min ! subject to Ax = b + ∆ b . Total Least Squares (TLS) approach is suitable: � [∆ A , ∆ b ] � 2 F = min ! subject to ( A + ∆ A ) x = b + ∆ b . TUHH Jörg Lampe RTLS GAMM Workshop ’08 4 / 27

Background LS and TLS problems Least Squares Consider overdetermined linear system A ∈ R m × n , b ∈ R m , m ≥ n Ax ≈ b , with A and b contaminated by noise. Least Squares (LS) approach: � Ax − b � 2 = min ! or equivalently � ∆ b � 2 = min ! subject to Ax = b + ∆ b . Total Least Squares (TLS) approach is suitable: � [∆ A , ∆ b ] � 2 F = min ! subject to ( A + ∆ A ) x = b + ∆ b . TUHH Jörg Lampe RTLS GAMM Workshop ’08 4 / 27

Background LS and TLS problems Total Least Squares With singular value decomposition of [ A , b ] [ A , b ] = U Σ V T , Σ = diag { σ 1 , . . . , σ n + 1 } and σ ′ 1 ≥ · · · ≥ σ ′ n singular values of A . Lemma (Golub, van Loan 1980) If σ n + 1 ([ A , b ]) < σ ′ n ( A ) holds, a unique TLS solution exist. Closed form solution of TLS problem: x TLS = ( A T A − σ 2 n + 1 I ) − 1 A T b . or equivalently � � 1 x TLS v n + 1 ( n + 1 ) v n + 1 . = − − 1 TUHH Jörg Lampe RTLS GAMM Workshop ’08 5 / 27

Background LS and TLS problems Total Least Squares With singular value decomposition of [ A , b ] [ A , b ] = U Σ V T , Σ = diag { σ 1 , . . . , σ n + 1 } and σ ′ 1 ≥ · · · ≥ σ ′ n singular values of A . Lemma (Golub, van Loan 1980) If σ n + 1 ([ A , b ]) < σ ′ n ( A ) holds, a unique TLS solution exist. Closed form solution of TLS problem: x TLS = ( A T A − σ 2 n + 1 I ) − 1 A T b . or equivalently � � 1 x TLS v n + 1 ( n + 1 ) v n + 1 . = − − 1 TUHH Jörg Lampe RTLS GAMM Workshop ’08 5 / 27

Background RLS and RTLS problems Outline Background 1 LS and TLS problems RLS and RTLS problems Solving RLS and RTLS problems 2 Solving RLS problem via QEP Solving RTLS problem via QEPs Determining the hyperparamter 3 L-curve Numerical Examples Conclusions 4 TUHH Jörg Lampe RTLS GAMM Workshop ’08 6 / 27

Background RLS and RTLS problems Regularized (Total) Least Squares For ill-conditioned LS and TLS problems regularization is necessary. Apply Tikhonov regularization with hyperparameter λ or adding quadratic constraint with hyperparameter δ . � [∆ b ] � 2 = min ! ( A ) x = b + ∆ b , � Lx � ≤ δ (RLS) subject to � [∆ A , ∆ b ] � 2 = min ! ( A + ∆ A ) x = b + ∆ b , � Lx � ≤ δ (RTLS) subject to F with δ > 0 and L ∈ R k × n , k ≤ n defining a seminorm on the solution. Lemma (Beck, Ben-Tal 2006) Let K be an orthonormal basis of ker ( L ) . If σ min ([ AK , b ]) < σ min ( AK ) holds, a solution of the RTLS problem exists. A solution of RLS problem always exists. Assume active quadratic constraint: δ < � Lx LS � and δ < � Lx TLS � respectively. Hence � Lx � ≤ δ replaced by � Lx � = δ . TUHH Jörg Lampe RTLS GAMM Workshop ’08 7 / 27

Background RLS and RTLS problems Regularized (Total) Least Squares For ill-conditioned LS and TLS problems regularization is necessary. Apply Tikhonov regularization with hyperparameter λ or adding quadratic constraint with hyperparameter δ . � [∆ b ] � 2 = min ! ( A ) x = b + ∆ b , � Lx � ≤ δ (RLS) subject to � [∆ A , ∆ b ] � 2 = min ! ( A + ∆ A ) x = b + ∆ b , � Lx � ≤ δ (RTLS) subject to F with δ > 0 and L ∈ R k × n , k ≤ n defining a seminorm on the solution. Lemma (Beck, Ben-Tal 2006) Let K be an orthonormal basis of ker ( L ) . If σ min ([ AK , b ]) < σ min ( AK ) holds, a solution of the RTLS problem exists. A solution of RLS problem always exists. Assume active quadratic constraint: δ < � Lx LS � and δ < � Lx TLS � respectively. Hence � Lx � ≤ δ replaced by � Lx � = δ . TUHH Jörg Lampe RTLS GAMM Workshop ’08 7 / 27

Solving RLS and RTLS problems Solving RLS problem via QEP Outline Background 1 LS and TLS problems RLS and RTLS problems Solving RLS and RTLS problems 2 Solving RLS problem via QEP Solving RTLS problem via QEPs Determining the hyperparamter 3 L-curve Numerical Examples Conclusions 4 TUHH Jörg Lampe RTLS GAMM Workshop ’08 8 / 27

Solving RLS and RTLS problems Solving RLS problem via QEP Regularized Least Squares How to solve a quadratically constrained Least Squares Problem? One possibility is by one QEP (another by LSTRS): Consider Lagrangian L ( x , µ ) = � Ax − b � 2 + µ ( � Lx � 2 − δ 2 ) with first-order optimality conditions 2 ( A T A + µ L T L ) x 2 A T b , = � Lx � 2 δ 2 . = Lemma (Gander 1981) Choose largest value of µ to obtain the RLS solution. Assume L is square and nonsingular: Substitute z := Lx , W := L − T A T AL − 1 and h := L − T A T b Wz + µ z = h , z T z δ 2 . = TUHH Jörg Lampe RTLS GAMM Workshop ’08 9 / 27

Solving RLS and RTLS problems Solving RLS problem via QEP Regularized Least Squares How to solve a quadratically constrained Least Squares Problem? One possibility is by one QEP (another by LSTRS): Consider Lagrangian L ( x , µ ) = � Ax − b � 2 + µ ( � Lx � 2 − δ 2 ) with first-order optimality conditions 2 ( A T A + µ L T L ) x 2 A T b , = � Lx � 2 δ 2 . = Lemma (Gander 1981) Choose largest value of µ to obtain the RLS solution. Assume L is square and nonsingular: Substitute z := Lx , W := L − T A T AL − 1 and h := L − T A T b Wz + µ z = h , z T z δ 2 . = TUHH Jörg Lampe RTLS GAMM Workshop ’08 9 / 27

Solving RLS and RTLS problems Solving RLS problem via QEP Connection to QEP Remark: If rank ( L ) = k < n , basis for range and kernel is needed. h T u = z T z = δ 2 h = δ − 2 hh T u Denoting u := ( W + µ I ) − 2 h ⇒ ⇒ ( W + µ I ) 2 u − δ − 2 hh T u = 0 . (Gander, Golub, von Matt 1989) µ, ˆ Reconstruct x RLS from rightmost eigenpair ( ˆ u ): u = δ 2 ˆ u Scale ˜ u , then it holds h T ˆ x RLS = L − T ( W + ˆ µ I )˜ u . The same idea can be carried over to RTLS problems In general there exist no closed form solution for x RTLS This leads to a converging sequence of QEPs TUHH Jörg Lampe RTLS GAMM Workshop ’08 10 / 27

Solving RLS and RTLS problems Solving RLS problem via QEP Connection to QEP Remark: If rank ( L ) = k < n , basis for range and kernel is needed. h T u = z T z = δ 2 h = δ − 2 hh T u Denoting u := ( W + µ I ) − 2 h ⇒ ⇒ ( W + µ I ) 2 u − δ − 2 hh T u = 0 . (Gander, Golub, von Matt 1989) µ, ˆ Reconstruct x RLS from rightmost eigenpair ( ˆ u ): u = δ 2 ˆ u Scale ˜ u , then it holds h T ˆ x RLS = L − T ( W + ˆ µ I )˜ u . The same idea can be carried over to RTLS problems In general there exist no closed form solution for x RTLS This leads to a converging sequence of QEPs TUHH Jörg Lampe RTLS GAMM Workshop ’08 10 / 27

Solving RLS and RTLS problems Solving RTLS problem via QEPs Outline Background 1 LS and TLS problems RLS and RTLS problems Solving RLS and RTLS problems 2 Solving RLS problem via QEP Solving RTLS problem via QEPs Determining the hyperparamter 3 L-curve Numerical Examples Conclusions 4 TUHH Jörg Lampe RTLS GAMM Workshop ’08 11 / 27

Solving RLS and RTLS problems Solving RTLS problem via QEPs Regularized Total Least Squares Assume x RTLS exists and constraint is active, then (RTLS) is equivalent to f ( x ) := � Ax − b � 2 � Lx � 2 = δ 2 . 1 + � x � 2 = min ! subject to Consider Lagrangian again L = � Ax − b � 2 1 + � x � 2 + µ ( � Lx � 2 − δ 2 ) , First-order optimality conditions are equivalent to ( A T A + λ I I + λ L L T L ) x A T b , = � Lx � 2 δ 2 µ ≥ 0 , = with µ = b T ( b − Ax ) + λ I λ I = −� Ax − b � 2 λ L = µ ( 1 + � x � 2 ) , 1 + � x � 2 , . δ 2 ( 1 + � x � 2 ) TUHH Jörg Lampe RTLS GAMM Workshop ’08 12 / 27