Interplay between statistical and deterministic methods Mihaela - - PowerPoint PPT Presentation

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

Interplay between statistical and deterministic methods

Mihaela Pricop, Frank Bauer

Institute for Mathematical Stochastics University of Göttingen Department for Knowledge-Based Mathematical Systems, Johannes Kepler University Linz Vienna 2009

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

Inverse problem

Goal: To estimate a† ∈ X, not directly observable when u belonging to the separable Hilbert space Y is known and F : D(F) ⊂ X → Y F(a†) = u. F is a possibly nonlinear, injective operator. The problem is ill-posed in the sense that F −1 is not continuous.

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

Definitions

Definition

A continuous linear operator ǫ : Y → L2(Ω, K, P) is called a Hilbert-space process. The covariance covǫ : Y → Y of ǫ is the bounded operator defined by covǫφ1, φ2 = Cov(ǫ, φ1 , ǫ, φ2), φ1, φ2 ∈ Y.

Definition

ǫ is a white noise process if covǫ = I. Every Hilbert-space valued random variable Σ with finite second moment can be identified with a Hilbert-space process by the operator Φ → Φ, Σ. A Gaussian white noise process in an infinite-dimensional Hilbert space can not be identified with a Hilbert-space valued random variable with finite second moment.

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

Abstract noise model

Y = u + δξ + σǫ deterministic error ξ ∈ Y, ξY = 1 stochastic error ǫ is a Hilbert-space process with E ǫ, φY = 0, covǫ ≤ 1.

Remark

White noise models occur as limits of discrete noise models as the sample size n tends to infinity. We have σ ∼

1 √n.

Brown & Low, 1996 Mathé & Pereverzev (2003) Bissantz, Hohage, Munk & Ruymgaart (2007)

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

Aims in inverse problems

The discontinuous operator F −1 is approximated by a family of continuous operators {Rα : α > 0}. We also need a parameter choice rule α = α(Y, δ, σ) to

• btain an estimate

a = Rα(Y,δ,σ)(Y). Convergence for deterministic errors sup

ξ≤1

Rα(F(a†)+δξ,δ)(F(a†) + δξ) − a† δ→0 − → 0

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

Consistency for random noise P

• Rα(F(a†)+σǫ,σ)(F(a†) + σǫ) − a†2 > θ

σ→0 − → 0, ∀θ > 0 Quadratic risk for random noise ERα(F(a†)+σǫ,σ)(F(a†) + σǫ) − a†2 σ→0 − → 0 Probabilistic rates: ∀σ ∃t1(σ) σ→0 − → 0, t2(σ) σ→0 − → 0 such that P

• Rα(F(a†)+σǫ,σ)(F(a†) + σǫ) − a†2 ≥ t1(σ)
• ≤ t2(σ)

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

Rates of convergence

For ill-posed problems, rates of convergence can only be

• btained under further a-priori information on the solution,

e.g. that a† belongs to a smoothness class S. Deterministic optimal rate △(δ, S, F) = inf

R

sup

ξ≤1,a†∈S

Rα(F(a†)+δξ,δ)(F(a†) + δξ) − a† Statistical minimax risk △(σ, S, F) = inf

R sup a†∈S

• ERα(F(a†)+σǫ,σ)(F(a†) + σǫ) − a†2

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

Oracle inequalities (stochastic)

Oracle inequalities: ERα(F(a†)+σǫ,σ)(F(a†) + σǫ) − a†2 ≤ ρσ inf

α ERα(F(a†) + σǫ) − a†2 + O

• σ2

Donoho & Johnstone 1994, Cavalier &Golubev &Picard &Tsybakov 2002

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

Oracle inequalities (deterministic)

weak oracle inequality Rα(F(a†)+δξ,δ)(F(a†)+δξ)−a† ≤ C infα

• supξ≤1 Rα(F(a†)+

δξ) − Rα(F(a†)) + Rα(F(a†)) − a†

• + O (δ)

strong oracle inequality Rα(F(a†)+δξ,δ)(F(a†) + δξ) − a† ≤ c sup

ξ

inf

α Rα(F(a†) + δξ) − a† + O (δ)

(Raus & Hämarik, recent papers)

• racle inequality (Bauer, Kindermann 2008)

Rα(F(a†)+δξ,δ)(F(a†) + δξ) − a† ≤ C inf

α

• Rα(F(a†) + δξ) − Rα(F(a†)) + Rα(F(a†)) − a†

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

Oracle inequalities for linear inverse problems

Deterministic/Deterministic:

Knowing δ: deterministic oracle inequality (Lepski˘ ı balancing, Mathé & Pereverzev 2003, 2006) Restricted sets/saturation: oracle inequality (quasi-optimality, Bauer, Kindermann 2008)

Deterministic/Stochastic:

Knowing σ: oracle inequality loosing logarithmic factor (Lepski˘ ı balancing) Restricted sets/saturation: oracle inequality (quasi-optimality, Bauer, Kindermann 2008)

Bayesian/Stochastic:

Oracle inequality, even without knowing σ (quasi-optimality, Bauer, Reiß2008)

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

Nonlinear statistical inverse problems

O’Sullivan 1990: first convergence rate result (suboptimal rates with restrictive assumptions) Bissantz & Hohage & Munk 2004: consistency and optimal rates for one smoothness class for Tikhonov regularization Hohage & Pricop 2008: optimal rates in a range of smoothness classes for Tikhonov regularization Bauer & Hohage & Munk 2009: convergence rates for regularized Newton method and optimal rates

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

2-step method for nonlinear inverse problems

F : X → Y is a nonlinear, injective operator. An estimator u of u ∈ Y is chosen, Y a Hilbert space, such that

• E

u − u2

Y ≤ τ with known τ.

• a ∈ D(F) is an estimator of a†:
• a := argmina∈D(F){F(a) −

u2

Y + αa − a02 X }

Bissantz & Hohage & Munk 2004

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

Hilbert scales

Consider an operator L : D(L) → X unbounded, self-adjoint, strictly positive, D(L) ⊂ X dense. Xs := D(Ls), s ≥ 0, x, ys := Lsx, LsyX , x, y ∈ Xs For s < 0 we define Xs as completion of X under the norm xs :=< x, x >1/2

s

(Xs)s∈R is the Hilbert scale induced by L Natterer 1984: Rates of convergence for deterministic linear inverse problems

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

Tikhonov regularization in Hilbert scales

Nonlinear Inverse Problems

• a is the solution of

F(a) − u2

Y + αa − a02 s → min, a ∈ D(F) ∩ (a0 + Xs)

Assumptions

• 1. D(F) is convex, F is continuous, injective,

Fréchet-differentiable on X and weakly closed on Xs for some s ≥ 0.

• 2. F ′(a†)hY ∼ h−p, ∀h ∈ X, for some known p > 0.
• 3. There exists M > 0 such that a ∈ D(F) ∩ (a0 + Xs)

F ′(a†) − F ′(a)Y←X−p ≤ Ma† − a00 ≤ λ 2Λ.

Neubauer 1992: Deterministic convergence rate with · Y←X in Assumption 3.

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

A-priori choice of the regularization parameter

Theorem (Hohage, Pricop)

If the Assumptions 1 − 3 are fulfilled, a† − a0 ∈ Xq, q ∈ [s, p + 2s], s ≥ p and α ∼ τ

2(p+s) p+q

then E a − a†2

X = O

• τ

2q q+p

• .

For linear operators the upper bounds of the previous theorem agree with the minimax lower bound. For nonlinear operators lower bounds are not available and the upper bounds for the combined method are derived from case to case.

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

Convergence for Lepski˘ ı choice in deterministic setting

Lepski˘ ı 1990: adaptive choice of the regularization parameter for regression problems Mathé, Pereverzev 2003, 2006: the Lepski˘ ı principle for linear inverse problems We use the error splitting a† − a ≤ a† −aα+aα − a where aα := argmin a∈D(F)∩(a0+Xs)

• F(a) − F(a†)2

Y + αa − a02 s

• .

Theorem (Pricop)

Let Assumptions 1 − 3, a† − a0 ∈ Xq, q ∈ [s, p + 2s], s ≥ p and a deterministic noise model hold. Then it holds aα − a†X ≤ C1α

q 2(p+s)

• a − aαX ≤ c1(δα

−p 2(p+s) + α q 2(p+s) )

with the constants C1 and c1 depending on a†, p, q, s.

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

Balancing principle for deterministic nonlinear inverse problems

0.1 0.2 0.3 0.4 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 regularization parameter error approximation error data noise error total error

αt α*

0.1 0.2 0.3 0.4 0.2 0.4 0.6 0.8 1 regularization parameter error approximation error data noise error total error

αt α*

There exists an ’optimal’ stopping index k+ ∈ {0, 1, . . . , Kmax} and a known increasing function E : N → [0, ∞) such that

• ak − a† ≤ E(k)δ.

Stopping rule i+ = max

• i : ai − aj ≤ 4C∗δα

−p 2(s+p)

j

, j = 1, 2, . . . , i

• .

where αj = δ2(q2)j−1, q > 1, j = 1, . . . , Kmax, ai = aαi.

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

Balancing principle for deterministic nonlinear inverse problems

Theorem

Under the Assumptions 1 − 3, for deterministic noise model and for the choice of the regularization parameter α = α+, the

• rder-optimal error bound

a+ − a† ≤ 6C∗δ

q p+q

holds true, where a+ = aαi+. Shuai, Pereverzev, Ramlau 2007: the balancing principle for nonlinear inverse problems

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

Balancing principle for statistical nonlinear inverse problems

Let us assume a stochastic setting and choose i+ = max

• i : ai − ajX ≤ 4C∗τ ln 1

τ α

−p 2(s+p)

j

, j = 1, 2, . . . , i

• .

Theorem (Pricop)

If, besides the Assumptions 1 − 3 for stochastic setting, the probability distribution for the estimator u fulfills the exponential inequality P

• u − E

u2 ≥ (t − 1)E

• u − E

u2 ≤ C2 exp(−c2t) for any t > 1 and for C2, c2 > 0, then it holds E(a+ − a†2) ≤ 2qK

p+qτ

2q p+q ln 1

τ .

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

Iteratively regularized Gauss- Newton method

standard Newton method F ′( ak)( ak+1 − ak) = Y − F( ak) If F continuous and compact, this is an ill-posed problem. In general Y − F( ak) / ∈ R

• F ′(

ak)

• .

Tikhonov regularization in each Newton step

• ak+1 = argmin a∈X F ′(

ak)(a − a0) + F( ak) − Y2

Y + αka − a02 X

Bakushinskii 1992: local convergence results Tikhonov regularization has saturation 1 iterated Tikhonov, Landweber iteration

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

Generalized Gauss- Newton methods

We choose a regularization method with qualification µ0 ≥ 1

• ak+1 = gαk
• F ′(

ak)∗F ′( ak)

• F ′(

ak)∗ Y − F( ak) + F ′( ak)( ak − a0)

• +a0

with αk = α0qk, q ∈ (0, 1), α0 > 0. General source conditions a0 − a† = Γ

• F ′(a†)

∗F ′(a†)

• w, w ∈ X, w ≤ ρ

Γ : [0, F ′(a†)2] → R is a monotonic increasing function satisfying Γ(0) = 0. Γ(t) = tq or Γ(t) = (− ln(t))−q Engl & Hanke & Neubauer 1996, Hohage 1997

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

Balancing principle for deterministic setting k+ = min

• k ∈ {0, . . . , Kmax(δ)} :

ak − amX ≤ 2δ C3 αm

1 2

: ∀m = k + 1, . . . , Kmax(δ)}

Theorem

Let us assume σ = 0, the Hölder source conditions with

1 2 ≤ µ ≤ µ0, smallness assumptions for δ, L, ρ = w, 1 α0 and

F ′(a1) − F ′(a2)Y←X ≤ La1 − a2X . Then it holds

• ak+ − a†X ≤ Cρ

1 2q+1 δ 2q 2q+1 .

Bauer & Hohage 2005

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

Theorem

Let assume either the Hölder source conditions with 0 ≤ q ≤ 1

2

• r Γ(t) = (− ln(t))−q, q ∈ (0, ∞) and smallness assumptions for

δ, L, ρ = wX ,

1 α0 . For a stronger Lipschitz condition on F ′(a)

it holds

• ak+ − a†X ≤ Cρ

1 2q+1 δ 2q 2q+1 .

for Hölder source condition and

• ak+ − a†X ≤ Cρ
• − ln(δ

ρ) −q for logarithmic source conditions. Bauer & Hohage 2005

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

Assumptions on stochastic noise Let the variance term V(a, α) = gα (F ′(a)∗F ′(a)) F ′(a)∗ǫ2.

• N1. There exists a known function ϕnoi such that
• EV(a†, α)

1

2 ≤ ϕnoi, ∀α ∈ (0, α0], a† ∈ D(F)

• N2. There are constants 1 < γnoi < γnoi < ∞ such that

γnoi ≤ ϕnoi(αk+1) ϕnoi(αk) ≤ γnoi, k ∈ N

• N3. Exponential inequality

∃c3, C3 > 0 ∀a† ∈ D(F) ∀α ∈ (0, α0] ∀t ≥ 1 P

• V(a†, α) ≥ t2EV(a†, α)
• ≤ C3 exp
• −c3t2

Bissantz & Hohage & Munk & Ruymgaart 2007

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

Balancing principle in stochastic setting Let us consider the stopping rule k+ = min

• k :

ak − amX ≤ C3 ln σ−2σϕ(αm), m ∈ {k + 1, . . . , Kmax(σ)}

• .

Theorem

Let us assume the Hölder source conditions with 1

2 ≤ q, a

smallness assumptions for ak − a0 and F ′(a1) − F ′(a2)Y←X ≤ La1 − a2X . Then it holds (E( ak+ − a†2

X ))

1 2 ≤ K4 min

k∈N

• Errorapprox + (ln σ−1)σϕnoi(αk)
• .

For ϕnoi(α) = C5α−c, we have E( ak+ − a†2

X ) = O

• σ ln σ−1

2q 2q+c

• .

Bauer & Hohage & Munk 2009

Noise models for inverse problems Tikhonov regularization in Hilbert scales Regularized Newton methods

Thank you for your attention!

Acknowledgement to the SFB 755 project "Nanoscale Photonic Imaging" and the "Upper Austrian Technology and Research Promotion"