On iterative regularization methods for nonlinear inverse problems - - PowerPoint PPT Presentation

On iterative regularization methods for nonlinear inverse problems

Antonio Leitão

acgleitao@gmail.com

Department of Math., Federal Univ. of St. Catarina, Brazil

Workshop 2: Large Scale Inv. Probl. and Appl. in the Earth Sciences RICAM, Linz, October 24–28, 2011

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

Outline

1

Introduction

2

Preliminary results

3

iTK Method: Exact data case

4

iTK Method: Noisy data case

5

L-iTK method

Collaborators: A. DeCezaro (Rio Grande), J. Baumeister (Frankfurt)

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

Outline

1

Introduction

2

Preliminary results

3

iTK Method: Exact data case

4

iTK Method: Noisy data case

5

L-iTK method

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

Abstract

• We investigate iterated Tikhonov methods for obtaining stable solutions of

nonlinear systems of ill-posed operator equations.

• We show that the proposed method is a convergent regularization method.
• In the case of noisy data we propose a modification, where a sequence of

relaxation parameters is introduced and a different stopping rule is used.

• Convergence analysis for this method is also provided.

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

• The inverse problem: Determine an unknown physical quantity x ∈ X from

the set of data (y0,...,yN−1) ∈ Y N. (X, Y Hilbert; N ≥ 1)

• In practical situations, only approximate measured data is available

yδ

i − yi ≤ δi , i = 0,...,N − 1,

(1) (δi > 0 noise level; notation: δ := (δ0,...,δN−1).

• The finite set of data (y0,...,yN−1) is obtained by indirect measurements
• f the parameter:

Fi(x) = yi , i = 0,...,N − 1, (2) (Fi : Di ⊂ X → Y)

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

• Standard methods for the solution of system (2):

— Iterative type regularization methods [EngHanNeu96, KalNeuSch08]; — Tikhonov type regularization methods [EngHanNeu96, TikArs77] after rewriting (2) as a single equation F(x) = y, where F := (F0,...,FN−1) : N−1

i=0 Di → Y N ,

y := (y0,...,yN−1). (3)

• These methods become inefficient if N is large or the evaluations of Fi(x)

and F ′

i (x)∗ are expensive.

• In such a situation, methods which cyclically consider each equation in (2)

separately are much faster and are often the method of choice in practice.

• The starting point of our approach is the iterated Tikhonov method

[HanGro98] for solving linear ill-posed problems. xδ

k+1 ∈ argmin

• F x − yδ2 +αx − xδ

k 2

,

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

• The above definition corresponds to the iteration (linear case)

xδ

k+1 = xδ k −α−1F ∗(F xδ k+1 − yδ).

• In the nonlinear case we have

xδ

k+1 = xδ k −α−1F ∗(xδ k+1)(F(xδ k+1)− yδ).

• Why to choose such a difficult method to implement?

Remark

• Comparison with classical Tikhonov regularization: One has to solve

xδ

α ∈ argmin

• F(x)− yδ2 +αx − x02

,

several times, since the ’optimal parameter’ α in not known.

• Comparison with quasi-optimality rule: One has to solve

xδ

αk ∈ argmin

• F(x)− yδ2 +αkx − x02

,

where αk = cqk, for some q < 1 and k = 1,2,...; until

xδ

αk − xδ αk+1 → mink.

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

• Relation to the Landweber method: (linear case again)

Remark — Plain least squares: xδ ∈ argminF x − yδ2 , — Optimality conditions: normal equations F ∗F x = F ∗y. — Equivalently: 0 = −F ∗(F x − y). — Related fixed-point equation: x = x − F ∗(F x − y). — Euler methods: xx+1 = xk −α−1F ∗(F xk − y) (Landweber) xx+1 = xk −α−1F ∗(F xk+1 − y) (iterated Tikhonov)

α−1 = δt (descretization of the continuous dynamics).

jump to [HanGro98] !!!

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

• Once again: iterated Tikhonov method [HanGro98].

(linear case) xδ

k+1 ∈ argmin

• F x − yδ2 +αx − xδ

k 2

,

• The above definition corresponds to the iteration

xδ

k+1 = xδ k −α−1F ∗(F xδ k+1 − yδ).

• We propose a generalization for nonlinear systems (2):

The iterated Tikhonov-Kaczmarz method (ITK method). xδ

k+1 ∈ argmin

• F[k](x)− yδ

[k]2 +αx − xδ k 2

.

(4) — α > 0 is an appropriate chosen number, — [k] := (k mod N), — xδ

0 = x0 ∈ X is an initial guess (incorporates a priori information).

• From the iteration formula (4) follows

xδ

k+1 = xδ k −α−1F ′ [k](xδ k+1)∗(F[k](xδ k+1)− yδ [k]).

(5) (Relevant issues: Existence of xδ

k+1 ∈ X?

Uniqueness?)

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

• As usual for nonlinear Tikhonov type regularization, the global minimum for

the Tikhonov functionals in (4) need not be unique.

• For exact data we obtain the same convergence statements for any

possible sequence of iterates and we will accept any global solution.

• For noisy data, a (strong) semi-convergence result is obtained under a

smooth assumption on the functionals Fi (assumption (A4) below), which guarantees uniqueness of global minimizers in (4).

• A word of caution:

Some authors consider iterated Tikhonov regularization with the number of iterations n ∈ N being fixed [Engl, Scherzer, Neubauer]. In this case, α plays the role of the regularization parameter. (also called n-th iterated Tikhonov method)

• The ITK method consists in incorporating the Kaczmarz (ciclic) strategy in

the iterated Tikhonov method.

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

• Stop criteria:

— A group of N subsequent steps (starting at some multiple k of N) is called a cycle. — The iteration should be terminated when, for the first time, at least one of the residuals F[k](xδ

k+1)− yδ [k] drops below a specified threshold within a

cycle. — That is, we stop the iteration at kδ

∗ := min{lN ∈ N : Fi(xδ

lN+i+1)−yδ i ≤ τδi , for some 0 ≤ i ≤ N −1}, (6)

• For k = kδ

∗ we do not necessarily have Fi(xδ

kδ

∗ +i)− yδ

i ≤ τδi for all

i = 0,...,N − 1.

• In the case of noise free data, δi = 0, the stop criteria in (6) may never be

reached, i.e. kδ

∗ = ∞ for δi = 0 in (1).

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

• In the case of noisy data, we also propose a loping version of ITK, namely,

the L-ITK iteration. — Loping strategy: In the L-ITK iteration we omit an update of the ITK (within

• ne cycle) if the corresponding i-th residual is below some threshold.

— Stop criteria: the L-ITK method is not stopped until all residuals (within a cicle) are below the specified threshold.

• We provide convergence analysis for both ITK and L-ITK iterations.
• In particular we prove that L-ITK is a convergent regularization method in

the sense of [EngHanNeu96].

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

Outline

1

Introduction

2

Preliminary results

3

iTK Method: Exact data case

4

iTK Method: Noisy data case

5

L-iTK method

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

Assumptions

(A1) The operators Fi are weakly sequentially continuous and Fréchet diff. — The corresponding domains of definition Di are weakly closed. — There exists x0 ∈ X, M > 0, and ρ > 0 s.t.

F ′

i (x) ≤ M ,

x ∈ Bρ(x0) ⊂ N−1

i=0 Di .

(7) (A2) Local tangential cone condition [KalNeuSch08]

Fi(x)−Fi(¯

x)−F ′

i (¯

x)(x −¯ x)Y ≤ ηFi(x)−Fi(¯ x)Y , x,¯ x ∈ Bρ(x0) (8) holds for some η < 1. (A3) There exists an element x∗ ∈ Bρ/4(x0) such that F(x∗) = y, where y = (y0,...,yN−1) are exact data satisfying (1).

• Choice of the positive constants α and τ in (5), (6).

α > 16

3

δmax ρ 2 , τ > 1+η

1−η ≥ 1, (9) (δmax := maxj{δj}) (for linear problems: τ = 1) (for exact data: α > 0)

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

• Well-definiteness of the Tikhonov functionals

Jk(x) := F[k](x)− yδ

[k]2 +αx − xδ k 2.

(10) Lemma Let assumption (A1) be satisfied. Then each Tikhonov functional Jk in (10) attains a minimizer on X. Proof: [EngHanNeu96]. (notice that: xδ

k+1 ∈ argminJk(x))

• Estimate for the residual of the ITK iteration.

Lemma Let xδ

k and α be defined by (5) and (9) respectively. Then

F[k](xδ

k+1)− yδ [k]2 ≤ F[k](xδ k )− yδ [k]2 ,

k < kδ

∗ .

(11) Proof: Direct consequence of

F[k](xδ

k+1)− yδ [k]2 ≤ Jk(xδ k+1) ≤ Jk(xδ k ) ≤ F[k](xδ k )− yδ [k]2, k < kδ

∗.

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

• Auxiliary result (to prove monotony of the ITK iteration).

Lemma Let xδ

k and α be defined by (5) and (9) respectively. Moreover, assume that

(A1) - (A3) hold true. If xδ

k+1 ∈ Bρ(x0) for some k ∈ N, then

xδ

k+1 − x∗2 −xδ k − x∗2 ≤

≤ 2 αF[k](xδ

k+1)− yδ [k]

• (η− 1)F[k](xδ

k+1)− yδ [k]+(1+η)δ[k]

• .

(12)

• The proof of this lemma requires that xδ

k+1 ∈ Bρ(x0).

• In the next lemma we make sure that this assumption is satisfied.

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

Lemma Let xδ

k and α be defined by (5) and (9) respectively. Moreover, assume that

(A1), (A3) hold true. If xδ

k ∈ Bρ/4(x∗) for some k ∈ N, then xδ k+1 ∈ Bρ(x0).

• Monotony property:

Proposition Under the assumptions of Lemma 3, for all k < kδ

∗ the iterates xδ

k remain in

Bρ/4(x∗) ⊂ Bρ(x0) and satisfy (12). Moreover,

xδ

k+1 − x∗2 ≤ xδ k − x∗2 ,

k < kδ

∗ .

(13)

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

Outline

1

Introduction

2

Preliminary results

3

iTK Method: Exact data case

4

iTK Method: Noisy data case

5

L-iTK method

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

• Main goal:

To prove convergence of the ITK iteration for δi = 0, i = 0,...,N − 1.

• Exact data case: y = (y0,...,yN−1)

The iterates in (5) are denoted by xk. (to contrast with xδ

k : noisy data case)

• Minimal norm solution:

Lemma There exists an x0-minimal norm solution of (2) in Bρ/4(x0), i.e., a solution x†

• f (2) such that x† − x0 = inf{x − x0 : x ∈ Bρ/4(x0) and F(x) = y}.

Moreover, x† is the only solution of (2) in Bρ/4(x0)∩

• x0 + ker(F ′(x†))⊥

.

• x† denotes the x0-minimal norm solution of (2).

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

• Define ek := x† − xk.

Thus, ek is monotone non increasing. (follows from (13))

• From (12), it follows that

∞

∑

k=0

F[k](xk+1)− y[k]2 ≤ α

2(1−η)x0 − x†2 < ∞, (14)

• These two are the main ingredients to prove:

Theorem (Convergence for exact data) For exact data, the iteration (xk) converges to a solution of (2), as k → ∞. Moreover, if

N (F ′(x†)) ⊆ N (F (x))

for all x ∈ Bρ(x0), i = 0,...,N − 1, (15) then xk → x†.

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

Outline

1

Introduction

2

Preliminary results

3

iTK Method: Exact data case

4

iTK Method: Noisy data case

5

L-iTK method

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

• Main goal:

To prove that xδ

kδ

∗ converges to a solution of (2) as δ → 0.

(kδ

∗ is the stoping index in (6))

• Finiteness of the stopping index kδ

∗ .

Proposition Assume δmin := min{δ0,...δN−1} > 0. Then kδ

∗ defined in (6) is finite.

Proof: Contradiction argument based on (12) and on the choice of τ in (9).

• Additional assumption:

(A4) The operators Fi in (2) and it’s derivatives F ′

i are Lipschitz continuous

(i.e., Fi(x)− Fi(¯ x)+F ′

i (x)− F ′ i (¯

x) ≤ Lx − ¯ x, for all x,¯ x ∈ Bρ(x0)) — Moreover, α in (9) and M in (7) are such that (M + M)L < α. (M := M(ρ,x0,y,∆) := sup{Fi(x)− yδ

i : i = 0,...,N − 1, x ∈

Bρ(x0), yδ

i − yi ≤ δi , |δ| ≤ ∆})

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

• Continuity of xδ

k at δ = 0 for fixed k ∈ N.

Lemma Let δj = (δj,0,...,δj,N−1) ∈ (0,∞)N be given with limj→∞ δj = 0. Moreover, let yδj = (yδj

0 ,...,yδj N−1) ∈ Y N be a corresponding sequence of noisy data

satisfying

yδj

i − yi ≤ δj,i ,

i = 0,...,N − 1, j ∈ N. Then, for each fixed k ∈ N we have limj→∞ xδj

k+1 = xk+1.

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

• Convergence for noisy data:

Theorem Let δj = (δj,0,..., δj,N−1) be a given sequence in (0,∞)N with limj→∞ δj = 0, and let yδj = (yδj

0 ,...,yδj N−1) ∈ Y N be a corresponding sequence of noisy

data satisfying yδj

i − yi ≤ δj,i, i = 0,...,N − 1, j ∈ N. Denote by

kj

∗ := k∗(δj,yδj) the stopping index defined in (6) and assume that the

sequence {kj

∗}j∈N is unbounded. Then xδj

kj

∗ converges to a solution of (2), as

j → ∞. Moreover, if (15) holds, then xδj

kj

∗ → x†.

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

Outline

1

Introduction

2

Preliminary results

3

iTK Method: Exact data case

4

iTK Method: Noisy data case

5

L-iTK method

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

• Loping iterated Tikhonov-Kaczmarz method (L-ITK method) for solving (2):

xδ

k+1 = xδ k −α−1ωkF ′ [k](xδ k+1)∗(F[k](xδ k+1)− yδ [k]).

(16) where

ωk :=

• 1
• F[k](xδ

k+1)− yδ [k]

• ≥ τδ[k]
• therwise

.

(17) (The positive constants α and τ are defined as in (9).)

• Meaning of (16), (17):

— At each iterative step, compute xk+1/2 ∈ D[k] satisfying xk+1/2 = xδ

k −α−1F ′ [k](xk+1/2)∗(F[k](xk+1/2)− yδ [k])

— If F[k](xk+1/2)−yδ

[k] ≥ τδ[k], take xδ k+1 = xk+1/2, otherwise xδ k+1 = xδ k .

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

Remark

• Exact data (δ = 0): the L-ITK reduces to the ITK.
• Noisy data: the L-ITK method is fundamentally different from the ITK:

— The bang-bang relaxation parameter ωk effects that the iterates defined in (5) become stationary if all components of the residual vector

• Fi(xδ

k )− yδ i

• fall below a pre-specified threshold.

— This characteristic renders (5) a regularization method. Remark

• The iteration in (16), (17) corresponds to

xδ

k+1 ∈ argmin

• ωkF[k](x)− yδ

[k]2 +αx − xδ k

• and is not uniquely defined.
• For noisy data, a semi-convergence result is obtained under the smooth

assumption (A4) on the functionals Fi, which guarantees that the L-ITK iteration is uniquely defined.

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

• Stop criteria:

— The L-ITK iteration should be terminated when, for the first time, all xδ

k are

equal within a cycle. That is, we stop the iteration at kδ

∗ := min{lN ∈ N : xδ

lN = xδ lN+1 = ··· = xδ lN+N−1},

(18) — In other words: kδ

∗ is the smallest multiple of N such that

xδ

kδ

∗ = xδ

kδ

∗ +1 = ··· = xδ

kδ

∗ +N−1 .

(19)

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

• Convergence analysis:

— Monotony result follows analog to the ITK iteration.

• Before proving the main semiconvergence theorem we need two auxiliary

results: — the first result guarantees that, for noisy data, the stopping index kδ

∗ in (18)

is finite; — the second result is an stability result: for each k ∈ N fixed limj→∞ xδj

k+1 = xk+1.

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

Proposition Assume δmin := min{δ0,...δN−1} > 0. Then kδ

∗ in (18) is finite, and

Fi(xδ

kδ

∗ )− yδ

i < κτδi ,

i = 0,...,N − 1. (20) where κ := [(1+η)+ M2/α]/(1−η). Lemma Let δj = (δj,0,...,δj,N−1) ∈ (0,∞)N be given with limj→∞ δj = 0. Moreover, let yδj = (yδj

0 ,...,yδj N−1) ∈ Y N be a corresponding sequence of noisy data

satisfying

• yδj

i − yi

• ≤ δj,i ,

i = 0,...,N − 1, j ∈ N. Then, for each fixed k ∈ N we have limj→∞ xδj

k+1 = xk+1.

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

• Semiconvergence result for the L-ITK iteration.

Theorem Let δj = (δj,0,..., δj,N−1) be a given sequence in (0,∞)N with limj→∞ δj = 0, and let yδj = (yδj

0 ,...,yδj N−1) ∈ Y N be a corresponding sequence of noisy

data satisfying yδj

i − yi ≤ δj,i, i = 0,...,N − 1, j ∈ N.

Denote by kj

∗ := k∗(δj,yδj) the corresponding stopping index defined in (18).

Then xδj

kj

∗ converges to a solution x∗ of (2) as j → ∞.

Moreover, if (15) holds, then xδj

kj

∗ converges to x†.

Introduction Preliminary results iTK Method: Exact data case iTK Method: Noisy data case L-iTK method Bibliography

Bibliography

• A. DeCezaro, J.aumeister and A. Leitão, Modified iterated Tikhonov methods for solving

systems of nonlinear ill-posed equations, Inverse Problems and Imaging 5:2011, 1–17

• J. Baumeister, B. Kaltenbacher and A. Leitão, On Levenberg-Marquardt Kaczmarz

methods for regularizing systems of nonlinear ill-posed equations, Inverse Problems and Imaging, 4:2010, 335–350 H.W. Engl, M. Hanke and A. Neubauer, Regularization of Inverse Problems, Kluwer Academic Publishers Group, Dordrecht, 1996

• M. Hanke and C.W. Groetsch, Nonstationary iterated Tikhonov regularization, J. Optim.

Theory Appl.,98:1998, 37–53