The EM Algorithm for Positive Linear Inverse Problems Bernard A. Mair - - PowerPoint PPT Presentation

The EM Algorithm for Positive Linear Inverse Problems

Bernard A. Mair Department of Mathematics University of Florida

Outline

• Finite Dimensional Positive Linear Inverse

Problems

• Finite Dimensional EM Algorithm
• Positive Linear Inverse Problems
• The Infinite Dimensional EM Algorithm
• Previous Convergence Results
• New Developments

Finite Dimensional Poisson Linear Inverse Problems

, T 1 2 ,

, where ( ) , , Data: [ , ,..., ] , where ~ Poiss[ ] Normalization: 1, for 1,2, ,

i j I J I i i i j i

p d d d d y p j J

×

= = ≥ = = =

∑

y P x P x y d …

Application: Emission Tomography

Maximum Likelihood Estimation

Gaussian Errors ⇒ Least Squares Minimization Poisson Errors ⇒ Kullback-Leibler Minimization

PET Physics

detector tube detector ring

Each annihilation results in 2 photons traveling in (nearly) opposite directions along a uniformly random line Emission-detection model

Maximum Likelihood Estimation

log ( | ) ( [ ] log [ ] !) ( log( [ ] ) [ ] ) ( | ) ( )

i i i i i i i i i i i

Prob d d d d d c KL c L c = = − + − = − − + + = − + +

∑ ∑

d y P x Px Px Px Px d P x x

• ˆ

MLE: arg max ( ) arg min ( | ) L KL

≥ ≥

=

x 0 x 0

x x d P x

Finite Dimensional EM Algorithm

( 1) ( ) (0) ( )

ˆ ˆ ˆ KT optimality: ( ) ; ( ) EM Algorithm: , 0, flat [ ]

i ij n n j j n i i

L L d p x x

+

∇ = ∇ ≤ = >

∑

x x x x Px

• Analytic Derivation
• L. Shepp and Y

. Vardi (1982), IEEE Trans. Med. Imag..

( ) ( ) ( 1) ( ) ( )

For any data vector : (1) 0; ; ( ) ( ) (2) { } converges to a MLE (not unique)

n n n n j i j i n

x d L L

+

≥ > = ≥

∑ ∑

d x x x x

Image De‐blurring

Noisy data ⇒ EM converges to noisy image

⇒ Regularization needed

ˆ arg max ( ) ( )

R

L R β

≥

−

x 0

x x x

• Penalized MLE:

Automatic choice of β, R Fast optimization algorithms

Early Stopping of EM

Automatic choice of stopping iteration

• EM converges rapidly to exact image if data is exact

Positive Linear Inverse Problems

Vardi and Lee (1993), From image deblurring to optimal investments: maximum likelihood solutions for positive linear inverse problems, J. Royal Statist. Soc., B.

I nfinite Dimensional EM Algorithm

( ) ( , ) ( ) ( ), where , , 0; ( , ) 1, for all Given , estimate . g s p s x f x dx Pf s p f g p s x ds x g g f = ≥ = ≈

∫ ∫

• 1

( ) ( , ) ( ) ( ) ; 0, const. ( )

n n n

g s p s x x x ds P s λ λ λ λ

+

= >

∫

• Kondor (1983), Nuclear Instruments and Methods. (MCW for f ,g on [0,1])

ML Estimation

1

( ) ( , ) ( ) ( ), ,where 0, bdd., ( , ) 1, for all Given ( ), estimate . g s p s x f x dx Pf s s p p s x ds x g g L f

Ω Σ

= ∈Σ ≥ = ∈Ω ≈ ∈ Σ

∫ ∫

• ( )

( ) ( )

{ }

1 1 1

( ) ( | ) log( ( )) ( )

• Max. ( ) over M

( ) : 0, L f KL g Pf g s g s Pf s g s Pf s ds L f f L f f g

Σ

− = − − + ⎡ ⎤ ⎣ ⎦ ∈ Ω ≥ =

∫

ML Feasibility Set

{ }

( ) ( )

1 1 1 1 1 1

• Max. ( ) over M

( ) : 0, Reasons: EM iterates satisfy the constraint. M ( ) log( ( )) L f f L f f g Pf g Pf f g f L f g s g s Pf s ds

Σ

∈ Ω ≥ = = ⇒ = = ∈ ⇒ = −∫

• i

i

• Is this norm constraint necessary?
• Does not exist in finite dimensional case.
• Does max. without constraint satisfy the constraint?

EM Convergence

1

( ) ( , ) ( ) ( ) ; 0, ( ) ( )

n n n

g s p s x x x ds L P s λ λ λ λ λ

∞ + Ω

= > ∈ Ω

∫

• {

}

1 1 * 1 *

(1) 0, ( ), (2) ( ) is incr. and convergent. (3) If in , is a max. of over .

n n n n n

L g L L L M λ λ λ λ λ λ λ

∞

> ∈ Ω = →

[Multhei and Schorr (1987,89,93)]: Resmerita, Engl, Iusem (2007)

(4) If and has bdd. sol'n, then { } has a

• subseq. which conv. weakly in

, 1.

n p

g g Pf g L p λ = = ≥

Infinite Dimensional EM Algorithm: Issues

• Existence of maximizer of L.
• Definition of feasible set – different than

finite dimensional case.

• Convergence of EM algorithm for noisy data.

New Results

Existence of maximizer of L. Definition of feasible set – similar to finite dimensional case. Convergence of (subsequence of) EM iterates for noisy data.

All issues resolved (partially) in semi- infinite dimensional model

Semi‐infinite Model

1

( ) ( ) , 1,2, , ,where 0, cont., ( ) 1, for all Given , estimate .

i i i I i i i I

g p x f x dx Pf i I p p x x g g f

Ω = +

= = ≥ = ∈Ω ≈ ∈

∫ ∑

• …
• 1

Extend to the set of finite Borel measures on . ( ) ( | ) [ log( ) ] MLE: Maximize ( ) over .

I i i i i i i

P S L KL g P g g P g P L S λ λ λ λ λ

=

Ω − = − − +

∑

Existence of MLE

( )

(1) wkly-cpct ( ) , s.t.sup ( ) sup ( ) ˆ (2) arg max ( ) exists and satisfies ˆ ˆ ( ) , ( ) 1, for all ˆ (3) arg max ( ) where { : ( ) } ˆ (4) is MLE iff ( )

S S t S i i i i i i S i i

S t S L L L g g p x P x L S S g i g

λ λ λ λ

λ λ λ λ λ λ λ λ λ λ λ

∈ ∈ ∈ ∈

∃ ⊂ = = Ω = ≤ ∈Ω = = ∈ Ω =

∑ ∑ ∑

• ˆ

( ) 1, for all and ˆ ˆ ( ) ( ) 1, for a.e.

i i i i i i i i

p x P x ii g p x P x λ λ λ ≤ ∈Ω = − ∈Ω

∑ ∑

Mair et al (1996), Inverse Problems

Semi‐infinite EM Algorithm

1

( ) ( ) ( ) [ ] , 0, in

n n i i n i i

x x g p x P L λ λ λ λ

∞ +

= >

∑

Assumptions:

No is identically zero. If , then [ ] for some .

i n n i

p f Pf i

∞ → ∞

→ ∞ i i

Results:

1 1

{ } has a subseq. conv. weakly in to some . { ( )} incr. to ( ) { ( | )} is a decreasing sequence. If { } in , is an MLE (w.r.t. ).

n n n n

L L L L KL L S λ λ λ λ λ λ λ λ λ

∗ ∞ ∗ ∗ ∗ ∗

∈ → i i i i

END