Two-Point Gradient Methods for Nonlinear Ill-Posed Problems Simon - - PowerPoint PPT Presentation

Johann Radon Institute for Computational and Applied Mathematics

Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Simon Hubmer and Ronny Ramlau

Johann Radon Institute for Computational and Applied Mathematics (RICAM) Austrian Academy of Sciences (ÖAW) Linz, Austria

Joint Fudan - RICAM Seminar, July 8, 2020

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Outline

1 Introduction 2 TPG Methods 3 Convergence Analysis 4 Numerical Examples 5 Recent Developments

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

The Problem

Hilbert spaces X and Y, with norms .. Operator F : X → Y, continuously Fréchet differentiable. Noisy data yδ ∈ Y and noise level δ ∈ R+. Problem

F(x) = y (δ)

The noisy data yδ satisfies

• y − yδ
• ≤ δ .

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Tikhonov Regularization

Required: Initial guess x0 and regularization parameter α. The method:

min

x

1 2

• F(x) − y δ

2 + α 2 x − x02

• Properties:

+ Weak conditions necessary for analysis. + Very versatile (different norms, regularization functionals). − Computation of the minimum ↔ HOW??

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Landweber Iteration

Required: Initial guess x0 and stopping criterion. The method:

x δ

k+1 = x δ k + F ′(x δ k)∗(y δ − F(x δ k))

Properties: + Easy to implement. − Strong conditions necessary for analysis. − Slow convergence, i.e., lots of iterations required.

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Second Order Methods

Levenberg-Marquardt method xδ

k+1 = xδ k + (F ′(xδ k)∗F ′(xδ k) + αkI)−1F ′(xδ k)∗(yδ − F(xδ k))

Iteratively regularized Gauss-Newton method xδ

k+1 = xδ k + (F ′(xδ k)∗F ′(xδ k) + αkI)−1(F ′(xδ k)∗(yδ − F(xδ k))

+αk(x0 − xδ

k))

Properties: + Require much less iterations. − Very strong conditions necessary for analysis. − Require inversion of (F ′(x)∗F ′(x) + αI) in every iteration step → difficult and takes time.

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Acceleration Techniques

Landweber Iteration with operator approximation: xδ

k+1 = xδ k + ˜

F ′(xδ

k)∗(yδ − ˜

F(xδ

k))

Landweber Iteration in Hilbert Scales: xδ

k+1 = xδ k + L−2sF ′(xδ k)∗(yδ − F(xδ k))

Landweber Iteration with intelligent stepsizes: xδ

k+1 = xδ k + αδ kF ′(xδ k)∗(yδ − F(xδ k))

Examples: Steepest Descent, Barzilai-Borwein, Neubauer.

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Connection: Residual Functional

Φ(x) = 1 2

• F(x) − yδ
• 2

Tikhonov = Minimize{ Φ(x) + Regularization(x) }. Landweber = Gradient Descent for Φ(x). Levenberg Marquardt = 2nd order descent for Φ(x). Iteratively regularized Gauss-Newton = 2nd order descent for Φ(x) + Tikhonov Type Stabilization.

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Nesterov Acceleration

General minimization problem min

x

{Φ(x)} Yurii Nesterov: Instead of using gradient descent: xk+1 = xk − ω∇Φ(xk) , use the following iteration:

zk = xk +

k−1 k+α−1(xk − xk−1) ,

xk+1 = zk − ω∇Φ(zk) .

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Motivating Picture

xk−1 xk ˜ xk+1 zk xk+1

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Motivating Picture

xk−1 xk ˜ xk+1 zk xk+1

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Motivating Picture

xk−1 xk ˜ xk+1 zk xk+1

˜ xk+1 = xk − ω∇Φ(xk)

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Motivating Picture

xk−1 xk ˜ xk+1 zk xk+1

zk = xk +

k−1 k+α−1(xk − xk−1)

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Motivating Picture

xk−1 xk ˜ xk+1 zk xk+1

zk = xk +

k−1 k+α−1(xk − xk−1)

xk+1 = zk − ω∇Φ(zk)

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

What’s so good about that?

Assume: Φ is convex. Gradient Descent:

• Φ(xk) − Φ(x†)
• = O(k−1)

Nesterov Acceleration:

• Φ(xk) − Φ(x†)
• = O(k−2)
• H. Attouch, J. Peypouquet, The rate of convergence of Nesterov’s accelerated

forward-backward method is actually o(k−2), SIAM Journal on Optimization

• Y. Nesterov, A method of solving a convex programming problem with

convergence rate O(1/k2), Soviet Mathematics Doklady, 27, 2, 1983

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Application to Nonlinear Ill-Posed Problems

For our problem, the method reads zδ

k = xδ k + k−1 k+α−1(xδ k − xδ k−1) ,

xδ

k+1 = zδ k + αδ k F ′(zδ k)∗(yδ − F(zδ k)) .

There is a generalization to deal with min{Φ(x) + Ψ(x)} , which reads zk = xk +

k−1 k+α−1(xk − xk−1) ,

xk+1 = proxΨ (zk − ω∇Φ(zk)) . = ⇒ Sparsity Constraints, Projections, etc.

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Application to Nonlinear Ill-Posed Problems

For our problem, the method reads zδ

k = xδ k + k−1 k+α−1(xδ k − xδ k−1) ,

xδ

k+1 = zδ k + αδ k F ′(zδ k)∗(yδ − F(zδ k)) .

There is a generalization to deal with min{Φ(x) + Ψ(x)} , which reads zk = xk +

k−1 k+α−1(xk − xk−1) ,

xk+1 = proxΨ (zk − ω∇Φ(zk)) . = ⇒ Sparsity Constraints, Projections, etc.

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Neubauer’s Linear Results

Assumptions: Linear operator F(x) = Tx, source condition x† ∈ R((T ∗T)µ), a priori stopping rule. If 0 ≤ µ ≤ 1

2, then

k(δ) = O(δ−

1 2µ+1 ) ,

• xδ

k(δ) − x†

• = o(δ

2µ 2µ+1 ) .

If µ > 1

2, then

k(δ) = O(δ−

2 2µ+3 ) ,

• xδ

k(δ) − x†

• = o(δ

2µ+1 2µ+3 ) .

Similar results also when using the discrepancy principle.

• A. Neubauer, On Nesterov acceleration for Landweber iteration of linear ill-posed

problems, J. Inv. Ill-Posed Problems, Vol. 25, No. 3, 2017

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Two-Point Gradient (TPG) Methods

How about general methods of the form zδ

k = xδ k + λδ k(xδ k − xδ k−1) ,

xδ

k+1 = zδ k + αδ kF ′(zδ k)∗(yδ − F(zδ k)) .

Question: Do they converge under standard assumptions? Yes for linear problems and λδ

k = k−1 k+α−1 ← Neubauer

Yes for λδ

k → 0 fast enough

Yes for some explicit choices of λδ

k

Yes for λδ

k defined via a backtracking search

Yes for λδ

k = k−1 k+α−1 and a locally convex residual functional

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Two-Point Gradient (TPG) Methods

How about general methods of the form zδ

k = xδ k + λδ k(xδ k − xδ k−1) ,

xδ

k+1 = zδ k + αδ kF ′(zδ k)∗(yδ − F(zδ k)) .

Question: Do they converge under standard assumptions? Yes for linear problems and λδ

k = k−1 k+α−1 ← Neubauer

Yes for λδ

k → 0 fast enough

Yes for some explicit choices of λδ

k

Yes for λδ

k defined via a backtracking search

Yes for λδ

k = k−1 k+α−1 and a locally convex residual functional

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Two-Point Gradient (TPG) Methods

How about general methods of the form zδ

k = xδ k + λδ k(xδ k − xδ k−1) ,

xδ

k+1 = zδ k + αδ kF ′(zδ k)∗(yδ − F(zδ k)) .

Question: Do they converge under standard assumptions? Yes for linear problems and λδ

k = k−1 k+α−1 ← Neubauer

Yes for λδ

k → 0 fast enough

Yes for some explicit choices of λδ

k

Yes for λδ

k defined via a backtracking search

Yes for λδ

k = k−1 k+α−1 and a locally convex residual functional

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Two-Point Gradient (TPG) Methods

How about general methods of the form zδ

k = xδ k + λδ k(xδ k − xδ k−1) ,

xδ

k+1 = zδ k + αδ kF ′(zδ k)∗(yδ − F(zδ k)) .

Question: Do they converge under standard assumptions? Yes for linear problems and λδ

k = k−1 k+α−1 ← Neubauer

Yes for λδ

k → 0 fast enough

Yes for some explicit choices of λδ

k

Yes for λδ

k defined via a backtracking search

Yes for λδ

k = k−1 k+α−1 and a locally convex residual functional

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Two-Point Gradient (TPG) Methods

How about general methods of the form zδ

k = xδ k + λδ k(xδ k − xδ k−1) ,

xδ

k+1 = zδ k + αδ kF ′(zδ k)∗(yδ − F(zδ k)) .

Question: Do they converge under standard assumptions? Yes for linear problems and λδ

k = k−1 k+α−1 ← Neubauer

Yes for λδ

k → 0 fast enough

Yes for some explicit choices of λδ

k

Yes for λδ

k defined via a backtracking search

Yes for λδ

k = k−1 k+α−1 and a locally convex residual functional

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Two-Point Gradient (TPG) Methods

How about general methods of the form zδ

k = xδ k + λδ k(xδ k − xδ k−1) ,

xδ

k+1 = zδ k + αδ kF ′(zδ k)∗(yδ − F(zδ k)) .

Question: Do they converge under standard assumptions? Yes for linear problems and λδ

k = k−1 k+α−1 ← Neubauer

Yes for λδ

k → 0 fast enough

Yes for some explicit choices of λδ

k

Yes for λδ

k defined via a backtracking search

Yes for λδ

k = k−1 k+α−1 and a locally convex residual functional

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Two-Point Gradient (TPG) Methods

How about general methods of the form zδ

k = xδ k + λδ k(xδ k − xδ k−1) ,

xδ

k+1 = PB

• zδ

k + αδ kF ′(zδ k)∗(yδ − F(zδ k))

• .

Question: Do they converge under standard assumptions? Yes for linear problems and λδ

k = k−1 k+α−1 ← Neubauer

Yes for λδ

k → 0 fast enough

Yes for some explicit choices of λδ

k

Yes for λδ

k defined via a backtracking search

Yes for λδ

k = k−1 k+α−1 and a locally convex residual functional

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Convergence Conditions I

Nonlinearity Condition

• F(x) − F(˜

x) − F ′(x)(x − ˜ x)

• ≤ η F(x) − F(˜

x) , x, ˜ x ∈ B4ρ(x0) ⊂ D(F) , η < 1 2 . Parameters 0 ≤ λδ

k ≤ 1 and stepsizes αδ k ≥ αmin > 0 satisfy

λδ

k(λδ k + 1)

• xδ

k − xδ k+1

• 2

−

• 1 + Ψ

µ

• αδ

k

• F(zδ

k) − yδ

• 2

+(αδ

k)2

• F ′(zδ

k)∗(F(zδ k) − yδ)

• 2

≤ 0 . Parameters λδ

k satisfy ∞

• k=0

λ0

k

• x0

k − x0 k−1

• < ∞ .

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Convergence Conditions I

Nonlinearity Condition

• F(x) − F(˜

x) − F ′(x)(x − ˜ x)

• ≤ η F(x) − F(˜

x) , x, ˜ x ∈ B4ρ(x0) ⊂ D(F) , η < 1 2 . Parameters 0 ≤ λδ

k ≤ 1 and stepsizes αδ k ≥ αmin > 0 satisfy

λδ

k(λδ k + 1)

• xδ

k − xδ k+1

• 2

−

• 1 + Ψ

µ

• αδ

k

• F(zδ

k) − yδ

• 2

+(αδ

k)2

• F ′(zδ

k)∗(F(zδ k) − yδ)

• 2

≤ 0 . Parameters λδ

k satisfy ∞

• k=0

λ0

k

• x0

k − x0 k−1

• < ∞ .

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Convergence Conditions I

Nonlinearity Condition

• F(x) − F(˜

x) − F ′(x)(x − ˜ x)

• ≤ η F(x) − F(˜

x) , x, ˜ x ∈ B4ρ(x0) ⊂ D(F) , η < 1 2 . Parameters 0 ≤ λδ

k ≤ 1 and stepsizes αδ k ≥ αmin > 0 satisfy

λδ

k(λδ k + 1)

• xδ

k − xδ k+1

• 2

−

• 1 + Ψ

µ

• αδ

k

• F(zδ

k) − yδ

• 2

+(αδ

k)2

• F ′(zδ

k)∗(F(zδ k) − yδ)

• 2

≤ 0 . Parameters λδ

k satisfy ∞

• k=0

λ0

k

• x0

k − x0 k−1

• < ∞ .

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Some Possible Choices

For the stepsizes αδ

k, one can use

a constant stepsize αδ

k = ω,

the steepest descent stepsize or the minimal error stepsize. The parameters λδ

k can be chosen

as any sequence decaying sufficiently fast, explicitly via λδ

k = min

• −1

2 +

• 1

4 + Ψ(τδ)2 µ¯ ω2 xδ

k − xδ k−1

• 2 , 1
• ,

via a backtracking algorithm.

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Some Possible Choices

For the stepsizes αδ

k, one can use

a constant stepsize αδ

k = ω,

the steepest descent stepsize or the minimal error stepsize. The parameters λδ

k can be chosen

as any sequence decaying sufficiently fast, explicitly via λδ

k = min

• −1

2 +

• 1

4 + Ψ(τδ)2 µ¯ ω2 xδ

k − xδ k−1

• 2 , 1
• ,

via a backtracking algorithm.

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Main Result I

Discrepancy Principle:

• yδ − F(zδ

k∗)

• ≤ τδ <
• yδ − F(zδ

k)

• ,

0 ≤ k < k∗ = k∗(δ, yδ) . Theorem Under the above assumptions, there holds lim

δ→0 zδ k∗(δ,yδ) = x∗ .

If additionally N(F ′(x†)) ⊂ N(F ′(x)), then we have lim

δ→0 zδ k∗(δ,yδ) = x† .

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Main Result I

Discrepancy Principle:

• yδ − F(zδ

k∗)

• ≤ τδ <
• yδ − F(zδ

k)

• ,

0 ≤ k < k∗ = k∗(δ, yδ) . Theorem Under the above assumptions, there holds lim

δ→0 zδ k∗(δ,yδ) = x∗ .

If additionally N(F ′(x†)) ⊂ N(F ′(x)), then we have lim

δ→0 zδ k∗(δ,yδ) = x† .

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Backtracking Algorithm

For many stepsizes αδ

k, the coupling condition above reduces to

λδ

k(λδ k + 1)

• xδ

k − xδ k−1

• 2

≤ Ψ µ αδ

k

• yδ − F(zδ

k)

• 2

. Idea: Given a summable sequence (qn)n, choose λδ

k via

λδ

k = min

• qnk
• xδ

k − xδ k−1

• , 1
• ,

where the subsequence (qnk)k is chosen such that the above inequality is satisfied. With this choice, one also has

∞

• k=0

λ0

k

• x0

k − x0 k−1

• < ∞ .

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Backtracking Algorithm

For many stepsizes αδ

k, the coupling condition above reduces to

λδ

k(λδ k + 1)

• xδ

k − xδ k−1

• 2

≤ Ψ µ αδ

k

• yδ − F(zδ

k)

• 2

. Idea: Given a summable sequence (qn)n, choose λδ

k via

λδ

k = min

• qnk
• xδ

k − xδ k−1

• , 1
• ,

where the subsequence (qnk)k is chosen such that the above inequality is satisfied. With this choice, one also has

∞

• k=0

λ0

k

• x0

k − x0 k−1

• < ∞ .

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Backtracking Algorithm

For many stepsizes αδ

k, the coupling condition above reduces to

λδ

k(λδ k + 1)

• xδ

k − xδ k−1

• 2

≤ Ψ µ αδ

k

• yδ − F(zδ

k)

• 2

. Idea: Given a summable sequence (qn)n, choose λδ

k via

λδ

k = min

• qnk
• xδ

k − xδ k−1

• , 1
• ,

where the subsequence (qnk)k is chosen such that the above inequality is satisfied. With this choice, one also has

∞

• k=0

λ0

k

• x0

k − x0 k−1

• < ∞ .

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Convergence Conditions II

Φδ(x) = 1 2

• F(x) − yδ
• 2

Φ0 is convex in B6ρ Φ0(λx1 + (1 − λ)x2) ≤ λΦ0(x1) + (1 − λ)Φ0(x2) Φ0 is Lipschitz continuous with constant L in B6ρ

• Φ0(x1) − Φ0(x2)
• ≤ L x1 − x2

α > 3 and the scaling satisfies 0 < ω ≤ 1/L

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Convergence Conditions II

Φδ(x) = 1 2

• F(x) − yδ
• 2

Φ0 is convex in B6ρ Φ0(λx1 + (1 − λ)x2) ≤ λΦ0(x1) + (1 − λ)Φ0(x2) Φ0 is Lipschitz continuous with constant L in B6ρ

• Φ0(x1) − Φ0(x2)
• ≤ L x1 − x2

α > 3 and the scaling satisfies 0 < ω ≤ 1/L

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Convergence Conditions II

Φδ(x) = 1 2

• F(x) − yδ
• 2

Φ0 is convex in B6ρ Φ0(λx1 + (1 − λ)x2) ≤ λΦ0(x1) + (1 − λ)Φ0(x2) Φ0 is Lipschitz continuous with constant L in B6ρ

• Φ0(x1) − Φ0(x2)
• ≤ L x1 − x2

α > 3 and the scaling satisfies 0 < ω ≤ 1/L

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Main Result II

Adapted Discrepancy Principle:

• yδ − F(xδ

k∗)

• 2

≤ 2(k + α − 1)2 k(α − 3) ∆(δ) + τ 2δ2 <

• yδ − F(xδ

k)

• 2

Theorem Under the above assumptions, one gets weak subsequential convergence of xδ

k∗(δ,yδ) to an element x∗ in the solution set

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Main Result II

Adapted Discrepancy Principle:

• yδ − F(xδ

k∗)

• 2

≤ 2(k + α − 1)2 k(α − 3) ∆(δ) + τ 2δ2 <

• yδ − F(xδ

k)

• 2

Theorem Under the above assumptions, one gets weak subsequential convergence of xδ

k∗(δ,yδ) to an element x∗ in the solution set

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Example Problem: Hammerstein

F(x)(s) = 1 k(s, t)φ(x(t)) dt x†(t) = 1 + 10−2(7 − 3t2 + 2t3) x0(t) = 1

Choice of λδ

k

k∗ Time λδ

k = 0

125 79 s Backtracking 41 26 s Explicit 35 22 s Nesterov 14 9 s

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Example Problem: Hammerstein

F(x)(s) = s x3(t) dt x†(t) = 1 + 10−2(7 − 3t2 + 2t3) x0(t) = 1

Choice of λδ

k

k∗ Time λδ

k = 0

125 79 s Backtracking 41 26 s Explicit 35 22 s Nesterov 14 9 s

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Example Problem: Hammerstein

F(x)(s) = s x3(t) dt x†(t) = 1 + 10−2(7 − 3t2 + 2t3) x0(t) = 1

Choice of λδ

k

k∗ Time λδ

k = 0

125 79 s Backtracking 41 26 s Explicit 35 22 s Nesterov 14 9 s

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Example Problem: SPECT

Figure: Activity function f∗ (left) and attenuation function µ∗ (right).

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Example Problem: SPECT

A(f , µ)(s, ω) =

• R

f (sω⊥ + tω) exp  −

∞

• t

µ(sω⊥ + rω) dr   dt Choice of λδ

k

k∗ Time λδ

k = 0

3433 489 s Backtracking 349 77 s Explicit 631 90 s Nesterov 205 30 s

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Example Problem: SPECT

A(f , µ)(s, ω) =

• R

f (sω⊥ + tω) exp  −

∞

• t

µ(sω⊥ + rω) dr   dt Choice of λδ

k

k∗ Time λδ

k = 0

3433 489 s Backtracking 349 77 s Explicit 631 90 s Nesterov 205 30 s

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Evolution of λδ

k and Residuals

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Example Problem: Autoconvolution

F(x)(s) = (x ∗ x)(s) =

1

• x(s − t)x(t) dt

x† = x0(t) =

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Example Problem: Autoconvolution

F(x)(s) = (x ∗ x)(s) =

1

• x(s − t)x(t) dt

x† = 10 + √ 2 sin(2πs) x0(t) = 10 + 27 28 √ 2 sin(2πs)

Choice of λδ

k

k∗ Time

• Rel. Error

λδ

k = 0

526 57 s 0.0244 % Nesterov 50 6 s 0.0271 %

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Example Problem: Autoconvolution

F(x)(s) = (x ∗ x)(s) =

1

• x(s − t)x(t) dt

x† = 10 + √ 2 sin(8πs) x0(t) = 10 + √ 2 sin(2πs)

Choice of λδ

k

k∗ Time

• Rel. Error

λδ

k = 0

10000 1067 s 9.57 % Nesterov 797 87 s 0.65 %

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Example Problem: Autoconvolution

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 8 8.5 9 9.5 10 10.5 11 11.5 12

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Some Recent Developments - 2018

• J. Wang, W. Wang, B. Han, Regularization of inverse problems by two-point

gradient methods with convex constraints ArXiv preprint, 2018.

• Y. Zhang, B. Hofmann, On the second-order asymptotical regularization of linear

ill-posed inverse problems Applicable Analysis, 2018.

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Some Recent Developments - 2019

• P. Pornsawad, N. Sapsakul, C. Böckmann A Modified Asymptotical

Regularization of Nonlinear Ill-Posed Problems Mathematics, 2019.

• G. Gao, B. Han, S. Tong A fast two-point gradient algorithm based on sequential

subspace optimization method for nonlinear ill-posed problems arXiv preprint, 2019.

• M. Zhong, W. Wang, Q. Jin, Regularization of inverse problems by two-point

gradient methods with convex constraints Numerische Mathematik, 2019.

• H. Long, B. Han., S. Tong, A new Kaczmarz-type method and its acceleration

for nonlinear ill-posed problems Inverse Problems, 2019.

• M. Zhong, W. Wang, A regularizing multilevel approach for nonlinear inverse

problems Applied Numerical Mathematics, 2019.

• S. Tong, B. Han, H. Long, R. Gu, An accelerated sequential subspace
• ptimization method based on homotopy perturbation iteration for nonlinear

ill-posed problems Inverse Problems, 2019.

• J. Wang, W. Wang, B. Han, An iterative regularization method with general

convex penalty for nonlinear inverse problems in Banach spaces Journal of Computational and Applied Mathematics, 2019.

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Some Recent Developments - 2020

• M. Zhong, W. Wang The two-point gradient methods for nonlinear inverse

problems based on Bregman projections Inverse Problems, 2020.

• M. Zhong, J. Liu Recovery of non-smooth radiative coefficient from nonlocal
• bservation by diffusion system Journal of Inverse and Ill-Posed Problems, 2020.
• G. Mittal, A. K. Giri A novel two-point gradient method for Regularization of

inverse problems in Banach spaces ArXiv Preprint, 2020.

• H. Long, B. Han, S. Tong A proximal regularized Gauss-Newton-Kaczmarz

method and its acceleration for nonlinear ill-posed problems Applied Numerical Mathematics, 2020.

• R. Gong, B. Hofmann, Y. Zhang A new class of accelerated regularization

methods, with application to bioluminescence tomography Inverse Problems, 2020.

• Y. Zhang, R. Gong, Second order asymptotical regularization methods for inverse

problems in partial differential equations Journal of Computational and Applied Mathematics, 2020.

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

Summary

Two-Point Gradient (TPG) methods zδ

k = xδ k + λδ k(xδ k − xδ k−1) ,

xδ

k+1 = zδ k + αδ kF ′(zδ k)∗(yδ − F(zδ k)) ,

converge under standard assumptions, are very easy to implement, require no more computation time than Landweber iteration, and lead to a considerable speed-up in practice.

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

References

• H. Attouch, J. Peypouquet, The rate of convergence of Nesterov’s accelerated

forward-backward method is actually o(k−2), SIAM Journal on Optimization

• S. Hubmer, R. Ramlau Convergence Analysis of a Two-Point Gradient Method

for Nonlinear Ill-Posed Problems, Inverse Problems, Vol. 33 No. 3, 2017

• S. Hubmer, R. Ramlau Nesterov’s Accelerated Gradient Method for Nonlinear

Ill-Posed Problems with a Locally Convex Residual Functional Accepted, 2018.

• Y. Nesterov, A method of solving a convex programming problem with

convergence rate O(1/k2), Soviet Mathematics Doklady, 27, 2, 1983

• A. Neubauer, On Nesterov acceleration for Landweber iteration of linear ill-posed

problems, J. Inv. Ill-Posed Problems, Vol. 25, No. 3, 2017

Thank you for your attention!

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems

Johann Radon Institute for Computational and Applied Mathematics

References

• H. Attouch, J. Peypouquet, The rate of convergence of Nesterov’s accelerated

forward-backward method is actually o(k−2), SIAM Journal on Optimization

• S. Hubmer, R. Ramlau Convergence Analysis of a Two-Point Gradient Method

for Nonlinear Ill-Posed Problems, Inverse Problems, Vol. 33 No. 3, 2017

• S. Hubmer, R. Ramlau Nesterov’s Accelerated Gradient Method for Nonlinear

Ill-Posed Problems with a Locally Convex Residual Functional Accepted, 2018.

• Y. Nesterov, A method of solving a convex programming problem with

convergence rate O(1/k2), Soviet Mathematics Doklady, 27, 2, 1983

• A. Neubauer, On Nesterov acceleration for Landweber iteration of linear ill-posed

problems, J. Inv. Ill-Posed Problems, Vol. 25, No. 3, 2017

Thank you for your attention!

www.ricam.oeaw.ac.at

• S. Hubmer, R. Ramlau, Two-Point Gradient Methods for Nonlinear Ill-Posed Problems