New Parameter Choice Rules for Regularization with Mixed Gaussian - - PowerPoint PPT Presentation

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary)

New Parameter Choice Rules for Regularization with Mixed Gaussian and Poissonian Noise

Elias Helou (joint work with ´ Alvaro De Pierro)

ICMC - USP elias@icmc.usp.br

1 de agosto de 2013

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary)

Sum´ ario

Regularization Conditioning and Regularization A Deterministic Approach A Frequentist Approach Frequentist Approaches to Parameter Selection Existing Methods The New Technique Numerical Results (preliminary) Tomography The (Still Preliminary) Results Concluding Remarks

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Ill-Conditioned Linear System

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Ill-Conditioned Linear System

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Ill-Conditioned Linear System

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Ill-Conditioned Linear System

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Ill-Conditioned Linear System

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Ill-Conditioned Linear System

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Ill-Conditioned Linear System

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Ill-Conditioned Linear System

x Ax = b

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Ill-Conditioned Linear System

x xǫ Axǫ = b + ǫ =: bǫ

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Ill-Conditioned Linear System

x xǫ xǫ − x ǫ = 25.00

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Ill-Conditioned Linear System

x xǫ κ(A) = 50.02

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Ill-Conditioned Linear System

◮ Ill-conditioned linear systems appear frequently in

applications;

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Ill-Conditioned Linear System

◮ Ill-conditioned linear systems appear frequently in

applications;

◮ Condition number will be much higher than example;

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Ill-Conditioned Linear System

◮ Ill-conditioned linear systems appear frequently in

applications;

◮ Condition number will be much higher than example; ◮ Under the presence of noise, severe loss of precision.

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Regularization

◮ Replace the original problem by a stable perturbed version

from a family {Pγ(bǫ)}γ∈Γ;

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Regularization

◮ Replace the original problem by a stable perturbed version

from a family {Pγ(bǫ)}γ∈Γ;

◮ Example: Tikhonov Regularization (Γ = R+)

xtik

γ (bǫ) := argmin x∈Rn

Ax − bǫ2

2 + γx2 2

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Regularization

◮ Replace the original problem by a stable perturbed version

from a family {Pγ(bǫ)}γ∈Γ;

◮ Example: Tikhonov Regularization (Γ = R+)

xtik

γ (bǫ) := argmin x∈Rn

Ax − bǫ2

2 + γx2 2

= (AT A + γI)−1AT bǫ;

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Regularization

◮ Replace the original problem by a stable perturbed version

from a family {Pγ(bǫ)}γ∈Γ;

◮ Example: Tikhonov Regularization (Γ = R+)

xtik

γ (bǫ) := argmin x∈Rn

Ax − bǫ2

2 + γx2 2

= (AT A + γI)−1AT bǫ;

◮ How to choose the regularization parameter γ?

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Deterministic Regularization Parameter Choice

◮ If ǫk → 0, then the parameter selection function

ℓ(bǫk, ǫk) must satisfy: xℓ(bǫk,ǫk)(bǫk) → x;

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Deterministic Regularization Parameter Choice

◮ If ǫk → 0, then the parameter selection function

ℓ(bǫk, ǫk) must satisfy: xℓ(bǫk,ǫk)(bǫk) → x;

◮ Drawback:

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Deterministic Regularization Parameter Choice

◮ If ǫk → 0, then the parameter selection function

ℓ(bǫk, ǫk) must satisfy: xℓ(bǫk,ǫk)(bǫk) → x;

◮ Drawback:

◮ Impossible if ǫ is not available (is it ever available?); Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Deterministic Regularization Parameter Choice

◮ If ǫk → 0, then the parameter selection function

ℓ(bǫk, ǫk) must satisfy: xℓ(bǫk,ǫk)(bǫk) → x;

◮ Drawback:

◮ Impossible if ǫ is not available (is it ever available?); ◮ Says nothing when ǫ is not small. Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Frequentist Regularization Parameter Choice

◮ A stochastic model for the problem may be available if it is

possible to estimate EǫǫT and we assume Eǫ = 0;

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Frequentist Regularization Parameter Choice

◮ A stochastic model for the problem may be available if it is

possible to estimate EǫǫT and we assume Eǫ = 0;

◮ For concreteness we use the model

EǫǫT = diag{Ax} + σ2I, for a known σ;

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Frequentist Regularization Parameter Choice

◮ A stochastic model for the problem may be available if it is

possible to estimate EǫǫT and we assume Eǫ = 0;

◮ For concreteness we use the model

EǫǫT = diag{Ax} + σ2I, for a known σ;

◮ That is, a mixed Gaussian-Poissonian model where

EǫǫT = E diag{bǫ} + σ2I.

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Noise in Imaging Technologies

◮ Imaging technologies often work by counting photon arrival;

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Noise in Imaging Technologies

◮ Imaging technologies often work by counting photon arrival; ◮ Examples: tomographic scanners, charge-coupled device

(ccd) sensors;

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Noise in Imaging Technologies

◮ Imaging technologies often work by counting photon arrival; ◮ Examples: tomographic scanners, charge-coupled device

(ccd) sensors;

◮ Photon emission is a Poisson process and the underlying

electronic circuitry adds Gaussian noise;

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Conditioning and Regularization A Deterministic Approach A Frequentist Approach

Noise in Imaging Technologies

◮ Imaging technologies often work by counting photon arrival; ◮ Examples: tomographic scanners, charge-coupled device

(ccd) sensors;

◮ Photon emission is a Poisson process and the underlying

electronic circuitry adds Gaussian noise;

◮ Tomographic image reconstruction and image deblurring are

ill-posed linear inverse problems.

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Existing Methods The New Technique

Regularization under Poissonian Noise

[Bardsley and Goldes(2009)] advocate to choose γ such that diag(Axγ)−1/2(Axγ − b)2

2 = m,

because E(Ax)i − bi2

2 = (Ax)i.

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Existing Methods The New Technique

Regularization under Poissonian Noise

[Bertero et al.(2010)] propose choosing γ such that KL(b, Axγ) = m 2 , because E

• KL(b, Ax)
• = m

2 +

m

• i=1

O

• (Ax)−1

i

• ,

where KL is the Kullback-Leibler entropic divergence: KL(x, y) =

n

• i=1

xi log xi yi − xi + yi.

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Existing Methods The New Technique

Regularization under Poissonian Noise

[Santos and De Pierro(2009)], instead, propose to minimize E (KL(Ax, Axγ)) ;

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Existing Methods The New Technique

Regularization under Poissonian Noise

[Santos and De Pierro(2009)], instead, propose to minimize E (KL(Ax, Axγ)) ; To this end, they provide an approximate estimator: E[KL(Ax, Axγ)] ≈ E m

• i=1

(Axγ)i

• − E
• bT log(Axγ)
• − E
• mωT diag(b){log(Ax+

γ ) − log(Ax− γ )}

δω2

2

• + K,

where ω ∼ N(0, I), δ > 0, and x±

γ is the solution obtained using

b ± δω as data.

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Existing Methods The New Technique

Regularization with Mixed Noise

Assume db (x, xγ) = Ub(x) + Vb(Ax)T Wb(xγ) + Xb(xγ) E

• db(Ax, Axγ)
• ≈ E
• Xb(xγ)
• +E
• Vb(b)T Wb(xγ)
• +K

− mE ωT diag(b + σ2I) [Fγ(b + δω) − Fγ(b − δω)] 2δω2

• ,

where Fγ(b) = V ′

b(b)T Wb(xγ), ω ∼ N(0, I) and K is a constant

which does not depend on γ.

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Existing Methods The New Technique

Special Case: Minimal Expected Squared Norm (mes)

If we use db(x, xγ) = Ax − Axγ in the above result, we obtain: E

• Ax − Axγ2

2

• ≈ K + E
• Axγ2

2

• − 2E[bT Axγ]

+ mE

• ωT diag(b + σ2I)
• Ax+

γ − Ax− γ

• δω2

2

• .

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Existing Methods The New Technique

Special Special Case: mes with Linear Regularization

If the regularization method is linear, the estimate is exact! E

• Ax − Axγ2

2

• = K + E
• Axγ2

2

• − 2E[bT Axγ]

+ mE

• ωT diag(b + σ2I)
• Ax+

γ − Ax− γ

• δω2

2

• .

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Tomography The (Still Preliminary) Results Concluding Remarks

Tomography

◮ Techniques for non-destructive visualization of

cross-sectional images;

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Tomography The (Still Preliminary) Results Concluding Remarks

Tomography

◮ Techniques for non-destructive visualization of

cross-sectional images;

◮ Applications in medicine; materials evaluation; botanics, etc;

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Tomography The (Still Preliminary) Results Concluding Remarks

Tomography

◮ Techniques for non-destructive visualization of

cross-sectional images;

◮ Applications in medicine; materials evaluation; botanics, etc; ◮ Measured data are approximate line integrals.

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Tomography The (Still Preliminary) Results Concluding Remarks

Tomography

t t R[f](θ, t)

1 2

θ 1

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Tomography The (Still Preliminary) Results Concluding Remarks

Applications in Tomography

lasq lakl

5.61 · 10−2 9.22 · 10−2 1.28 · 10−1 1.64 · 10−1 2.01 · 10−1 0.00 · 100 7.32 · 10−2 1.46 · 10−1

Ax − bǫ/Ax xγ − x xγ∗ − x − 1

Figura: Results for fbp regularization. Poissonian noise only (σ = 0).

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Tomography The (Still Preliminary) Results Concluding Remarks

Applications in Tomography

lasq lakl

5.88 · 10−2 1.20 · 10−1 1.82 · 10−1 2.43 · 10−1 3.05 · 10−1 0.00 · 100 4.26 · 10−2 8.51 · 10−2

Ax − bǫ/Ax xγ − x xγ∗ − x − 1

Figura: Results for fbp regularization. Mixed noise (σ = 5).

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Tomography The (Still Preliminary) Results Concluding Remarks

Conclusions and Future Research

◮ Very good methodology for parameter selection for linear

regularization;

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Tomography The (Still Preliminary) Results Concluding Remarks

Conclusions and Future Research

◮ Very good methodology for parameter selection for linear

regularization;

◮ Previous experiments indicates good performance for

nonlinear regularization methods as well;

Elias Helou Regularization Parameter Choice – 29◦ CBM

Regularization Frequentist Approaches to Parameter Selection Numerical Results (preliminary) Tomography The (Still Preliminary) Results Concluding Remarks

Conclusions and Future Research

◮ Very good methodology for parameter selection for linear

regularization;

◮ Previous experiments indicates good performance for

nonlinear regularization methods as well;

◮ Further experimentation required with other divergence

measures.

Elias Helou Regularization Parameter Choice – 29◦ CBM

References

Johnathan M. Bardsley and John Goldes. Regularization parameter selection methods for ill-posed Poisson maximum likelihood estimation. Inverse Problems, 25(9):095005, 2009. doi: 10.1088/0266-5611/25/9/095005. Mario Bertero, Patrizia Boccacci, Giorgio Talenti, Riccardo Zanella and Luca Zanni. A discrepancy principle for Poisson data. Inverse Problems, 26(10):105004–105023, 2010. doi: 10.1088/0266-5611/26/10/105004. Reginaldo J. Santos and ´ Alvaro Rodolfo De Pierro. A new parameters choice method for ill-posed problems with Poisson data and its application to emission tomographic imaging. International Journal of Tomography & Statistics, 11(s09):33–52, 2009. URL http://www.ceser.res.in/ceserp/index.php/ijts/issue/view/19.