[PPT] - Efficient Determination of the Hyperparameter via L-curve in Large PowerPoint Presentation

SLIDE 1

Efficient Determination of the Hyperparameter via L-curve in Large Scale Least Squares and Total Least Squares Problems

Jörg Lampe

joerg.lampe@tuhh.de

Hamburg University of Technology Institute of Numerical Simulation

Joint work with Heinrich Voß

8th GAMM Workshop on Applied and Numerical Linear Algebra

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 1 / 27

SLIDE 2

Outline

1

Background LS and TLS problems RLS and RTLS problems

2

Solving RLS and RTLS problems Solving RLS problem via QEP Solving RTLS problem via QEPs

3

Determining the hyperparamter L-curve Numerical Examples

4

Conclusions

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 2 / 27

SLIDE 3

Background LS and TLS problems

Outline

1

Background LS and TLS problems RLS and RTLS problems

2

Solving RLS and RTLS problems Solving RLS problem via QEP Solving RTLS problem via QEPs

3

Determining the hyperparamter L-curve Numerical Examples

4

Conclusions

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 3 / 27

SLIDE 4

Background LS and TLS problems

Least Squares

Consider overdetermined linear system Ax ≈ b, A ∈ Rm×n, b ∈ Rm, m ≥ n with A and b contaminated by noise. Least Squares (LS) approach: Ax − b2 = min!

r equivalently

∆b2 = min! subject to Ax = b + ∆b. Total Least Squares (TLS) approach is suitable: [∆A, ∆b]2

F = min!

subject to (A + ∆A)x = b + ∆b.

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 4 / 27

SLIDE 5

Background LS and TLS problems

Least Squares

Consider overdetermined linear system Ax ≈ b, A ∈ Rm×n, b ∈ Rm, m ≥ n with A and b contaminated by noise. Least Squares (LS) approach: Ax − b2 = min!

r equivalently

∆b2 = min! subject to Ax = b + ∆b. Total Least Squares (TLS) approach is suitable: [∆A, ∆b]2

F = min!

subject to (A + ∆A)x = b + ∆b.

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 4 / 27

SLIDE 6

Background LS and TLS problems

Total Least Squares

With singular value decomposition of [A, b] [A, b] = UΣV T , Σ = diag{σ1, . . . , σn+1} and σ′

1 ≥ · · · ≥ σ′ n singular values of A.

Lemma (Golub, van Loan 1980) If σn+1([A, b]) < σ′

n(A) holds, a unique TLS solution exist.

Closed form solution of TLS problem: xTLS = (AT A − σ2

n+1I)−1AT b.

r equivalently
xTLS

−1

= −

1 v n+1(n + 1)v n+1.

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 5 / 27

SLIDE 7

Background LS and TLS problems

Total Least Squares

With singular value decomposition of [A, b] [A, b] = UΣV T , Σ = diag{σ1, . . . , σn+1} and σ′

1 ≥ · · · ≥ σ′ n singular values of A.

Lemma (Golub, van Loan 1980) If σn+1([A, b]) < σ′

n(A) holds, a unique TLS solution exist.

Closed form solution of TLS problem: xTLS = (AT A − σ2

n+1I)−1AT b.

r equivalently
xTLS

−1

= −

1 v n+1(n + 1)v n+1.

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 5 / 27

SLIDE 8

Background RLS and RTLS problems

Outline

1

Background LS and TLS problems RLS and RTLS problems

2

Solving RLS and RTLS problems Solving RLS problem via QEP Solving RTLS problem via QEPs

3

Determining the hyperparamter L-curve Numerical Examples

4

Conclusions

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 6 / 27

SLIDE 9

Background RLS and RTLS problems

Regularized (Total) Least Squares

For ill-conditioned LS and TLS problems regularization is necessary. Apply Tikhonov regularization with hyperparameter λ or adding quadratic constraint with hyperparameter δ. [∆b]2 = min! subject to (A )x = b + ∆b, Lx ≤ δ (RLS) [∆A, ∆b]2

F

= min! subject to (A + ∆A)x = b + ∆b, Lx ≤ δ (RTLS) with δ > 0 and L ∈ Rk×n, k ≤ n defining a seminorm on the solution. Lemma (Beck, Ben-Tal 2006) Let K be an orthonormal basis of ker(L). If σmin([AK, b]) < σmin(AK) holds, a solution of the RTLS problem exists. A solution of RLS problem always exists. Assume active quadratic constraint: δ < LxLS and δ < LxTLS respectively. Hence Lx ≤ δ replaced by Lx = δ.

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 7 / 27

SLIDE 10

Background RLS and RTLS problems

Regularized (Total) Least Squares

For ill-conditioned LS and TLS problems regularization is necessary. Apply Tikhonov regularization with hyperparameter λ or adding quadratic constraint with hyperparameter δ. [∆b]2 = min! subject to (A )x = b + ∆b, Lx ≤ δ (RLS) [∆A, ∆b]2

F

= min! subject to (A + ∆A)x = b + ∆b, Lx ≤ δ (RTLS) with δ > 0 and L ∈ Rk×n, k ≤ n defining a seminorm on the solution. Lemma (Beck, Ben-Tal 2006) Let K be an orthonormal basis of ker(L). If σmin([AK, b]) < σmin(AK) holds, a solution of the RTLS problem exists. A solution of RLS problem always exists. Assume active quadratic constraint: δ < LxLS and δ < LxTLS respectively. Hence Lx ≤ δ replaced by Lx = δ.

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 7 / 27

SLIDE 11

Solving RLS and RTLS problems Solving RLS problem via QEP

Outline

1

Background LS and TLS problems RLS and RTLS problems

2

Solving RLS and RTLS problems Solving RLS problem via QEP Solving RTLS problem via QEPs

3

Determining the hyperparamter L-curve Numerical Examples

4

Conclusions

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 8 / 27

SLIDE 12

Solving RLS and RTLS problems Solving RLS problem via QEP

Regularized Least Squares

How to solve a quadratically constrained Least Squares Problem? One possibility is by one QEP (another by LSTRS): Consider Lagrangian L(x, µ) = Ax − b2 + µ(Lx2 − δ2) with first-order optimality conditions 2(AT A + µLTL)x = 2AT b, Lx2 = δ2. Lemma (Gander 1981) Choose largest value of µ to obtain the RLS solution. Assume L is square and nonsingular: Substitute z := Lx, W := L−TAT AL−1 and h := L−T AT b Wz + µz = h, zTz = δ2.

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 9 / 27

SLIDE 13

Solving RLS and RTLS problems Solving RLS problem via QEP

Regularized Least Squares

How to solve a quadratically constrained Least Squares Problem? One possibility is by one QEP (another by LSTRS): Consider Lagrangian L(x, µ) = Ax − b2 + µ(Lx2 − δ2) with first-order optimality conditions 2(AT A + µLTL)x = 2AT b, Lx2 = δ2. Lemma (Gander 1981) Choose largest value of µ to obtain the RLS solution. Assume L is square and nonsingular: Substitute z := Lx, W := L−TAT AL−1 and h := L−T AT b Wz + µz = h, zTz = δ2.

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 9 / 27

SLIDE 14

Solving RLS and RTLS problems Solving RLS problem via QEP

Connection to QEP

Remark: If rank(L) = k < n, basis for range and kernel is needed. Denoting u := (W + µI)−2h ⇒ hT u = zTz = δ2 ⇒ h = δ−2hhT u (W + µI)2u − δ−2hhT u = 0. (Gander, Golub, von Matt 1989) Reconstruct xRLS from rightmost eigenpair (ˆ µ, ˆ u): Scale ˜ u = δ2

ˆ u hT ˆ u , then it holds

xRLS = L−T (W + ˆ µI)˜ u. The same idea can be carried over to RTLS problems In general there exist no closed form solution for xRTLS This leads to a converging sequence of QEPs

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 10 / 27

SLIDE 15

Solving RLS and RTLS problems Solving RLS problem via QEP

Connection to QEP

Remark: If rank(L) = k < n, basis for range and kernel is needed. Denoting u := (W + µI)−2h ⇒ hT u = zTz = δ2 ⇒ h = δ−2hhT u (W + µI)2u − δ−2hhT u = 0. (Gander, Golub, von Matt 1989) Reconstruct xRLS from rightmost eigenpair (ˆ µ, ˆ u): Scale ˜ u = δ2

ˆ u hT ˆ u , then it holds

xRLS = L−T (W + ˆ µI)˜ u. The same idea can be carried over to RTLS problems In general there exist no closed form solution for xRTLS This leads to a converging sequence of QEPs

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 10 / 27

SLIDE 16

Solving RLS and RTLS problems Solving RTLS problem via QEPs

Outline

1

Background LS and TLS problems RLS and RTLS problems

2

Solving RLS and RTLS problems Solving RLS problem via QEP Solving RTLS problem via QEPs

3

Determining the hyperparamter L-curve Numerical Examples

4

Conclusions

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 11 / 27

SLIDE 17

Solving RLS and RTLS problems Solving RTLS problem via QEPs

Regularized Total Least Squares

Assume xRTLS exists and constraint is active, then (RTLS) is equivalent to f(x) := Ax − b2 1 + x2 = min! subject to Lx2 = δ2. Consider Lagrangian again L = Ax − b2 1 + x2 + µ(Lx2 − δ2), First-order optimality conditions are equivalent to (AT A + λII + λLLT L)x = AT b, µ ≥ 0, Lx2 = δ2 with λI = −Ax − b2 1 + x2 , λL = µ(1 + x2), µ = bT (b − Ax) + λI δ2(1 + x2) .

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 12 / 27

SLIDE 18

Solving RLS and RTLS problems Solving RTLS problem via QEPs

Fixed-point Iteration for RTLS

Look at nonlinear first-order equation again

AT A + λI(x)I + λL(x)LT L
x = AT b.

What about the iteration with maximal real Lagrange parameter each step:

AT A − f(xk)I
xk+1 + λLT Lxk+1 = AT b, Lxk+12 = δ2

? Theorem (Lampe, Voß 2007) Any limit point of the sequence {xk} constructed by the fixed point algorithm above is a global minimizer of f(x) = Ax − b2 1 + x2 s.t. Lx2 = δ2. (Under mild condition concerning starting vector x0)

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 13 / 27

SLIDE 19

Solving RLS and RTLS problems Solving RTLS problem via QEPs

Fixed-point Iteration for RTLS

Look at nonlinear first-order equation again

AT A + λI(x)I + λL(x)LT L
x = AT b.

What about the iteration with maximal real Lagrange parameter each step:

AT A − f(xk)I
xk+1 + λLT Lxk+1 = AT b, Lxk+12 = δ2

? Theorem (Lampe, Voß 2007) Any limit point of the sequence {xk} constructed by the fixed point algorithm above is a global minimizer of f(x) = Ax − b2 1 + x2 s.t. Lx2 = δ2. (Under mild condition concerning starting vector x0)

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 13 / 27

SLIDE 20

Solving RLS and RTLS problems Solving RTLS problem via QEPs

Connection to QEPs

Assume L is square and nonsingular: Substitute z := Lxk+1, Wk := L−T(AT A − f(xk)I)L−1 and h := L−T AT b Wkz + λz = h, zTz = δ2. Denoting again u := (Wk + λI)−2h ⇒ hT u = zTz = δ2 ⇒ h = δ−2hhT u (Wk + λI)2u − δ−2hhT u = 0. (QEPs) Reconstruct xk+1 from rightmost eigenpair (ˆ λ, ˆ u): With ˜ u = δ2

ˆ u hT ˆ u it holds

xk+1 = L−T (Wk + ˆ λI)˜ u. Remark: If the matrix (Wk + ˆ λI) ≥ 0 is singular, the rightmost eigenpair might not contain the solution of the f.o.c. An additional linear system has to be solved to cope with nonunique solutions. This is not the generic case.

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 14 / 27

SLIDE 21

Solving RLS and RTLS problems Solving RTLS problem via QEPs

Connection to QEPs

Assume L is square and nonsingular: Substitute z := Lxk+1, Wk := L−T(AT A − f(xk)I)L−1 and h := L−T AT b Wkz + λz = h, zTz = δ2. Denoting again u := (Wk + λI)−2h ⇒ hT u = zTz = δ2 ⇒ h = δ−2hhT u (Wk + λI)2u − δ−2hhT u = 0. (QEPs) Reconstruct xk+1 from rightmost eigenpair (ˆ λ, ˆ u): With ˜ u = δ2

ˆ u hT ˆ u it holds

xk+1 = L−T (Wk + ˆ λI)˜ u. Remark: If the matrix (Wk + ˆ λI) ≥ 0 is singular, the rightmost eigenpair might not contain the solution of the f.o.c. An additional linear system has to be solved to cope with nonunique solutions. This is not the generic case.

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 14 / 27

SLIDE 22

Solving RLS and RTLS problems Solving RTLS problem via QEPs

Algorithm for solving RTLS

With Tk(λ) = (Wk + λI)2 − δ−2hhT Algorithm 1 RTLSQEP [Sima/van Huffel/Golub 2005]

1: Let x0 be an initial guess. Compute f(x0) = Ax0−b2

1+x02 .

2: Set k = 1 3: while Not converged do 4:

Solve Tk(λ)u = 0 for eigenpair (uk, λk) corresponding to the rightmost λk

5:

Scale ˜ u = δ2

uk hT uk

6:

Compute xk+1 = L−1(Wk + λkI)˜ u and f(xk+1) = Axk+1 − b2 1 + xk+12

7:

Set k = k + 1

8: end while

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 15 / 27

SLIDE 23

Solving RLS and RTLS problems Solving RTLS problem via QEPs

Solving the QEPs

RTLSQEP algorithm contains sequence of quadratic eigenproblems Tk(λ)u = (Wk + λI)2u − δ−2hhT u = 0, for k = 0, 1 . . . Sequence of QEPs converges → Use previously gained information Linearization to EVP of double size is not appropriate Second Order Krylov Methods use one starting vector each QEP , i.e. use solution vector of previous QEP (SOAR, Li/Ye) Can we use more information than only one vector? Yes, a method that can make use of all previous information by performing thick starts is the Nonlinear Arnoldi method

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 16 / 27

SLIDE 24

Solving RLS and RTLS problems Solving RTLS problem via QEPs

Solving the QEPs

RTLSQEP algorithm contains sequence of quadratic eigenproblems Tk(λ)u = (Wk + λI)2u − δ−2hhT u = 0, for k = 0, 1 . . . Sequence of QEPs converges → Use previously gained information Linearization to EVP of double size is not appropriate Second Order Krylov Methods use one starting vector each QEP , i.e. use solution vector of previous QEP (SOAR, Li/Ye) Can we use more information than only one vector? Yes, a method that can make use of all previous information by performing thick starts is the Nonlinear Arnoldi method

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 16 / 27

SLIDE 25

Solving RLS and RTLS problems Solving RTLS problem via QEPs

Nonlinear Arnoldi

With Tk(λ) = (Wk + λI)2 − δ−2hhT Algorithm 2 Nonlinear Arnoldi [Voß 2003]

1: Start with initial basis V, V T V = I 2: Determine preconditioner M ≈ T(σ)−1, σ close to wanted eigenvalue 3: Find rightmost eigenvalue λ of V T Tk(λ)Vy = 0 and corr. eigenvector y 4: Set u = Vy, r = Tk(λ)u 5: while r/u > ǫ do 6:

v = Mr

7:

v = v − VV T v

8:

˜ v = v/v, V = [V, ˜ v]

9:

Find rightmost eigenvalue λ of V T Tk(λ)Vy = 0 and corr. eigenvector y

10:

Set u = Vy, r = Tk(λ)u

11: end while

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 17 / 27

SLIDE 26

Solving RLS and RTLS problems Solving RTLS problem via QEPs

Comments on Nonlinear Arnoldi

Main advantage: When solving Tk(λ)u = 0 in step k, start with complete search space V from preceding steps No preconditioner is needed, i.e. M = I Projected problems can be updated very cheaply: V T Tk(λ)Vy =

(Wk + λI)V

T (Wk + λI)V

y − δ−2(V T h)(V T h)T y = 0

Only V, WkV and V T h are needed. A closer look shows Wk = L−T (AT A + f(xk)I)L−1 = L−T AT AL−1 + f(xk)L−T L−1 Main part of Wk is not changing within the sequence of QEPs Simply store L−T AT AL−1V, L−T L−1V, V T h and V for all QEPs! Just append one column every inner iteration step No matrix-matrix multiplication is performed

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 18 / 27

SLIDE 27

Determining the hyperparamter L-curve

Outline

1

Background LS and TLS problems RLS and RTLS problems

2

Solving RLS and RTLS problems Solving RLS problem via QEP Solving RTLS problem via QEPs

3

Determining the hyperparamter L-curve Numerical Examples

4

Conclusions

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 19 / 27

SLIDE 28

Determining the hyperparamter L-curve

L-curve

How to determine hyperparamter δ in constraint condition Lx ≤ δ ? Several methods are available: Discrepancy principle, Cross validation, Information Criteria, L-curve Idea of the L-curve: Developed to balance Axλ − b2 and Lxλ in Tikhonov Ax − b + λLx = minx! Can be extended to f(xδ) = Axδ − b2 1 + xδ2 and δ = Lxδ Choose set of δi, i = 1, . . . and solve one RLS/RTLS problem for each δi Advantage when using Nonlinear Arnoldi method: RLS : Search space V reused during sequence of QEPs RTLS: Search space V reused during sequence of sequence of QEPs If search space grows too large, include restart strategy

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 20 / 27

SLIDE 29

Determining the hyperparamter L-curve

L-curve

How to determine hyperparamter δ in constraint condition Lx ≤ δ ? Several methods are available: Discrepancy principle, Cross validation, Information Criteria, L-curve Idea of the L-curve: Developed to balance Axλ − b2 and Lxλ in Tikhonov Ax − b + λLx = minx! Can be extended to f(xδ) = Axδ − b2 1 + xδ2 and δ = Lxδ Choose set of δi, i = 1, . . . and solve one RLS/RTLS problem for each δi Advantage when using Nonlinear Arnoldi method: RLS : Search space V reused during sequence of QEPs RTLS: Search space V reused during sequence of sequence of QEPs If search space grows too large, include restart strategy

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 20 / 27

SLIDE 30

Determining the hyperparamter L-curve

L-curve

How to determine hyperparamter δ in constraint condition Lx ≤ δ ? Several methods are available: Discrepancy principle, Cross validation, Information Criteria, L-curve Idea of the L-curve: Developed to balance Axλ − b2 and Lxλ in Tikhonov Ax − b + λLx = minx! Can be extended to f(xδ) = Axδ − b2 1 + xδ2 and δ = Lxδ Choose set of δi, i = 1, . . . and solve one RLS/RTLS problem for each δi Advantage when using Nonlinear Arnoldi method: RLS : Search space V reused during sequence of QEPs RTLS: Search space V reused during sequence of sequence of QEPs If search space grows too large, include restart strategy

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 20 / 27

SLIDE 31

Determining the hyperparamter Numerical Examples

Outline

1

Background LS and TLS problems RLS and RTLS problems

2

Solving RLS and RTLS problems Solving RLS problem via QEP Solving RTLS problem via QEPs

3

Determining the hyperparamter L-curve Numerical Examples

4

Conclusions

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 21 / 27

SLIDE 32

Determining the hyperparamter Numerical Examples

Example phillips (2000), RLS

P .C. Hansen, discretized Fredholm integral equation of first-kind L is 1D discrete first order derivative operator Noise level 20% of average absolute value of [A, b] For L-curve: δi = δtrue · (0, 0001 . . . 100) with δtrue = Lxtrue

10 20 30 40 50 60 70 10

−16

10

−14

10

−12

10

−10

10

−8

10

−6

10

−4

10

−2

10 Convergence history: RLSQEP − Lexact inner iterations residual norm QEP residual

Figure: Convergence history of RLS-QEP for different δi

Search space build up during first QEP contains such good information that the following QEPs are solved in much less MatVecs

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 22 / 27

SLIDE 33

Determining the hyperparamter Numerical Examples

Example phillips (2000), RTLS

20 40 60 80 100 120 10

−18

10

−16

10

−14

10

−12

10

−10

10

−8

10

−6

10

−4

10

−2

Convergence history: RTLSQEP − Lexact inner iterations residual norm QEP residual

Figure: Convergence history of RTLS-QEP for different δi

Restart performed if dimension of search space > 45 Each RTLS problem is solved by very few QEPs Search space build up during first RTLS problem contains such good information that following problems are solved within less MatVecs

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 23 / 27

SLIDE 34

Determining the hyperparamter Numerical Examples

Example phillips (2000), RTLS

δi = δtrue · (0, 0001 . . . 100), δtrue = Lxtrue, i = 1, . . . 20

10

−8

10

−7

10

−6

10

−5

10

−4

10

−3

10

−2

10

−1

10

−4

10

−3

10

−2

10

−1

RTLSQEP − L−curve − Lexact || Lx || f(x)= || Ax − b ||2 / ( 1 + ||x||2 )

Figure: L-curve of RTLS-QEP

L-curve of RTLS looks similar to L-curve of RLS 120 inner iterations for 20 RTLS problems (→ 500 MatVecs) 70 inner iterations for 20 RLS problems (→ 280 MatVecs) Computation time 4, 2sec for RTLS, and 2, 4sec for RLS

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 24 / 27

SLIDE 35

Determining the hyperparamter Numerical Examples

Example deriv2 (2000), RTLS

δi = δtrue · (0, 0001 . . . 100), δtrue = Lxtrue, i = 1, . . . 20

10 20 30 40 50 60 70 80 90 10

−16

10

−14

10

−12

10

−10

10

−8

10

−6

10

−4

10

−2

Convergence history: RTLSQEP − Lexact inner iterations residual norm QEP residual

Figure: Convergence history of RTLS-QEP for different δi

90 inner iterations for 20 RTLS problems (→ 360 MatVecs) Computation time 3sec (resp. 2, 3sec for 20 RLS problems)

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 25 / 27

SLIDE 36

Determining the hyperparamter Numerical Examples

Example deriv2 (2000), RTLS

δi = δtrue · (0, 0001 . . . 100), δtrue = Lxtrue, i = 1, . . . 20 90 inner iterations for 20 RTLS problems (→ 360 MatVecs) Computation time 3sec (resp. 2, 3sec for 20 RLS problems)

10

−9

10

−8

10

−7

10

−6

10

−5

10

−4

10

−3

10

−2

10

−6.8

10

−6.7

10

−6.6

RTLSQEP − L−curve − Lexact || Lx || f(x)= || Ax − b ||2 / ( 1 + ||x||2 )

Figure: L-curve of RTLSQEP TUHH

Jörg Lampe RTLS GAMM Workshop ’08 25 / 27

SLIDE 37

Determining the hyperparamter Numerical Examples

Example deriv2 (2000), RLS/RTLS

L =    −1 1 ... ... −1 1   

r

˜ L =      ε −1 1 ... ... −1 1     

200 400 600 800 1000 1200 1400 1600 1800 2000 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 x 10

−3

exact RLS − QEP Lnoisy RTLS − QEP Lnoisy RLS − QEP Lexact RTLS − QEP Lexact

Figure: Solution curves for RLS and RTLS

Not much difference between RLS and RTLS approach Little difference between approaches with different regularization matrices L and ˜ L

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 26 / 27

SLIDE 38

Conclusions

RLS/RTLS problems can be solved efficiently by one/sequence of QEPs Determine δ via L-curve Improvements of RTLSQEP method: Global convergence to RTLS solution was proven Nonlinear Arnoldi uses all previous information Computational complexity O(n2), number of MatVecs smaller n Remark: For RTLS problems there is also an approach where instead of QEPs linear EVPs are solved [Renaut/Guo 2005]. When solving a sequence of EVPs via Nonlinear Arnoldi for several δi (→ L-curve), similar advantages due to reusing the search space V are observable. For RLS problems this can be recognized as well, when solving one problem by sequence of EVPs (→ LSTRS), but solving by Nonlinear Arnoldi again.

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 27 / 27

SLIDE 39

Conclusions

RLS/RTLS problems can be solved efficiently by one/sequence of QEPs Determine δ via L-curve Improvements of RTLSQEP method: Global convergence to RTLS solution was proven Nonlinear Arnoldi uses all previous information Computational complexity O(n2), number of MatVecs smaller n Remark: For RTLS problems there is also an approach where instead of QEPs linear EVPs are solved [Renaut/Guo 2005]. When solving a sequence of EVPs via Nonlinear Arnoldi for several δi (→ L-curve), similar advantages due to reusing the search space V are observable. For RLS problems this can be recognized as well, when solving one problem by sequence of EVPs (→ LSTRS), but solving by Nonlinear Arnoldi again.

TUHH

Jörg Lampe RTLS GAMM Workshop ’08 27 / 27