The Mathematics of X-ray Tomography Tatiana A. Bubba Department of - - PowerPoint PPT Presentation

The Mathematics of X-ray Tomography Tatiana A. Bubba

Department of Mathematics and Statistics, University of Helsinki tatiana.bubba@helsinki.fi Summer School on Very Finnish Inverse Problems Helsinki, June 3-7, 2019

Finnish Centre of Excellence in Inverse Modelling and Imaging

2018-2025 2018-2025

Limited Data Tomography Problems

Despite ill-posedness, CT is well understood when comprehensive projection data are available: Analytical techniques: FBP, FDK Iterative techniques: ART-based methods, ML and LS approaches, MBIR

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Limited Data Tomography Problems

Despite ill-posedness, CT is well understood when comprehensive projection data are available: Analytical techniques: FBP, FDK Iterative techniques: ART-based methods, ML and LS approaches, MBIR However, concrete practical issues: lower the X-ray radiation dose shorten the scanning time take into account the non-stationarity

• f the target and the time-dependance
• f the measurements
• Limited Data tomography

Dynamic tomography These are severely ill-posed problems and state-of-the-art techniques from clas- sical CT perform poorly.

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Sparse Tomography

FBP with comprehensive data (1200 projections) FBP with sparse data (20 projections)

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Sparse Tomography

FBP with comprehensive data (1200 projections) Non-negative Tikhonov regularization (20 projections)

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Sparse Tomography

FBP with comprehensive data (1200 projections) Non-negative TV regularization (20 projections)

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Sparse Tomography

FBP with comprehensive data (1200 projections) Non-negative TGV regularization (20 projections)

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Sparse Tomography

FBP with comprehensive data (1200 projections) Non-negative Haar wavelets regularization (20 projections)

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Sparse Tomography

FBP with comprehensive data (1200 projections) Non-negative shearlets regularization (20 projections)

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Limited Data Tomography: Exterior CT

?

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Limited Data Tomography: Limited Angle CT

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Limited Data Tomography: Region-of-Interest CT

? ?

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Limited Data Tomography: Limited Angle ROI CT

? ?

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Limited Angle Tomography

Sample Rf(·, τ) on [−φ, φ] ⊂ [−π/2, π/2), denoted by Rφf = Rf|[−φ,φ]×R.

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Limited Angle Tomography

Sample Rf(·, τ) on [−φ, φ] ⊂ [−π/2, π/2), denoted by Rφf = Rf|[−φ,φ]×R.

φ = 90◦, filtered backprojection (FBP)

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Limited Angle Tomography

Sample Rf(·, τ) on [−φ, φ] ⊂ [−π/2, π/2), denoted by Rφf = Rf|[−φ,φ]×R.

φ = 75◦, filtered backprojection (FBP)

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Limited Angle Tomography

Sample Rf(·, τ) on [−φ, φ] ⊂ [−π/2, π/2), denoted by Rφf = Rf|[−φ,φ]×R.

φ = 60◦, filtered backprojection (FBP)

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Limited Angle Tomography

Sample Rf(·, τ) on [−φ, φ] ⊂ [−π/2, π/2), denoted by Rφf = Rf|[−φ,φ]×R.

φ = 45◦, filtered backprojection (FBP)

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Limited Angle Tomography

Sample Rf(·, τ) on [−φ, φ] ⊂ [−π/2, π/2), denoted by Rφf = Rf|[−φ,φ]×R.

φ = 30◦, filtered backprojection (FBP)

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Limited Angle Tomography

Sample Rf(·, τ) on [−φ, φ] ⊂ [−π/2, π/2), denoted by Rφf = Rf|[−φ,φ]×R.

φ = 15◦, filtered backprojection (FBP)

Observations:

• nly certain boundaries/features seem

to be “visible”, missing wedge creates artifacts, highly ill-posed inverse problem!

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Limited Angle CT in Dental Imaging

Image credits: Samuli Siltanen, VT device.

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Limited Angle CT in Breast Imaging

Image credits: Giotto Tomo.

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Limited Angle CT in Luggage Control

Image credits: Analogic COBRA Checkpoint CT.

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Microlocal Analysis & Wavefront Sets

Definition (Wavefront set, [H¨

• rmander, 1970])

Let f be a function. A tuple (x0, ξ0) ∈ R2×R2\

• is not in the wavefront set of f

iff ⊲ there exists a smooth cut-off function φ ∈ C∞

c (R2) with φ(x0) = 0

(localize in x0) ⊲ there is an open cone V containing ξ0 (microlocalize in ξ0) such that φ · f decays rapidly in V .

= ⇒ simultaneous description of location and direction of singularities of f!

E.T. Quinto, Singularities of the X-ray transform and limited data tomography in R2 and R3, SIAM J. on

• Math. Anal. 24 (5), 1215–1225.
• J. Frikel and E.T. Quinto, Characterization and reduction of artifacts in limited angle tomography, Inverse

Problems 29 (12), 125007.

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Microlocal Analysis & Wavefront Sets

Definition (Wavefront set, [H¨

• rmander, 1970])

Let f be a function. A tuple (x0, ξ0) ∈ R2×R2\

• is not in the wavefront set of f

iff ⊲ there exists a smooth cut-off function φ ∈ C∞

c (R2) with φ(x0) = 0

(localize in x0) ⊲ there is an open cone V containing ξ0 (microlocalize in ξ0) such that φ · f decays rapidly in V .

= ⇒ simultaneous description of location and direction of singularities of f! Main literature for limited data CT: characterization in sinogram: Quinto (1993) characterization in FBP, reduction of artifacts: Frikel & Quinto (2013)

E.T. Quinto, Singularities of the X-ray transform and limited data tomography in R2 and R3, SIAM J. on

• Math. Anal. 24 (5), 1215–1225.
• J. Frikel and E.T. Quinto, Characterization and reduction of artifacts in limited angle tomography, Inverse

Problems 29 (12), 125007.

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Visibility in CT

“visible”: singularities tangent “invisible”: singularities not tangent to sampled lines to sampled lines

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Visibility in CT

“visible”: singularities tangent “invisible”: singularities not tangent to sampled lines to sampled lines

Theorem ([Quinto, 1993])

Let L0 = L(θ0, s0) be a line in the plane. Let (x0, ξ0) ∈ WF(f) such that x0 ∈ L0 and ξ0 is a normal vector to L0. The singularity of f at (x0, ξ0) causes a unique singularity in WF(Rf) at (θ0, s0). Singularities of f not tangent to L(θ0, s0) do not cause singularities in Rf at (θ0, s0).

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Shearlets Comes into Play: What Are They?

Definition (Discrete shearlet system [Kutyniok & Labate, 2006])

Let ψ ∈ L2(R2).

• G. Kutyniok and D. Labate, Shearlets: Multiscale Analysis for Multivariate Data, 1st ed. New

York: Springer Verlag, 2012.

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Shearlets Comes into Play: What Are They?

Definition (Discrete shearlet system [Kutyniok & Labate, 2006])

Let ψ ∈ L2(R2). For j ∈ Z and k ∈ R, define the parabolic scaling matrix Aj and the shear matrix Sk by Aj := 22j 2j

• ,

(1) j = 0

• G. Kutyniok and D. Labate, Shearlets: Multiscale Analysis for Multivariate Data, 1st ed. New

York: Springer Verlag, 2012.

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Shearlets Comes into Play: What Are They?

Definition (Discrete shearlet system [Kutyniok & Labate, 2006])

Let ψ ∈ L2(R2). For j ∈ Z and k ∈ R, define the parabolic scaling matrix Aj and the shear matrix Sk by Aj := 22j 2j

• ,

(1) j = 1

• G. Kutyniok and D. Labate, Shearlets: Multiscale Analysis for Multivariate Data, 1st ed. New

York: Springer Verlag, 2012.

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Shearlets Comes into Play: What Are They?

Definition (Discrete shearlet system [Kutyniok & Labate, 2006])

Let ψ ∈ L2(R2). For j ∈ Z and k ∈ R, define the parabolic scaling matrix Aj and the shear matrix Sk by Aj := 22j 2j

• ,

(1) j = 2

• G. Kutyniok and D. Labate, Shearlets: Multiscale Analysis for Multivariate Data, 1st ed. New

York: Springer Verlag, 2012.

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Shearlets Comes into Play: What Are They?

Definition (Discrete shearlet system [Kutyniok & Labate, 2006])

Let ψ ∈ L2(R2). For j ∈ Z and k ∈ R, define the parabolic scaling matrix Aj and the shear matrix Sk by Aj := 22j 2j

• ,

Sk := 1 k 1

• .

(1) k = 0

• G. Kutyniok and D. Labate, Shearlets: Multiscale Analysis for Multivariate Data, 1st ed. New

York: Springer Verlag, 2012.

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Shearlets Comes into Play: What Are They?

Definition (Discrete shearlet system [Kutyniok & Labate, 2006])

Let ψ ∈ L2(R2). For j ∈ Z and k ∈ R, define the parabolic scaling matrix Aj and the shear matrix Sk by Aj := 22j 2j

• ,

Sk := 1 k 1

• .

(1) k = 1/4

• G. Kutyniok and D. Labate, Shearlets: Multiscale Analysis for Multivariate Data, 1st ed. New

York: Springer Verlag, 2012.

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Shearlets Comes into Play: What Are They?

Definition (Discrete shearlet system [Kutyniok & Labate, 2006])

Let ψ ∈ L2(R2). For j ∈ Z and k ∈ R, define the parabolic scaling matrix Aj and the shear matrix Sk by Aj := 22j 2j

• ,

Sk := 1 k 1

• .

(1) k = 1/2

• G. Kutyniok and D. Labate, Shearlets: Multiscale Analysis for Multivariate Data, 1st ed. New

York: Springer Verlag, 2012.

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Shearlets Comes into Play: What Are They?

Definition (Discrete shearlet system [Kutyniok & Labate, 2006])

Let ψ ∈ L2(R2). For j ∈ Z and k ∈ R, define the parabolic scaling matrix Aj and the shear matrix Sk by Aj := 22j 2j

• ,

Sk := 1 k 1

• .

(1) The system SH(ψ) := {ψj,k,m := 23j/2ψ(SkAj · −m) | j ∈ Z, k ∈ Z, m ∈ Z2} (2) is the discrete shearlet system defined by ψ.

• G. Kutyniok and D. Labate, Shearlets: Multiscale Analysis for Multivariate Data, 1st ed. New

York: Springer Verlag, 2012.

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Shearlets Comes into Play: What Are They?

Definition (Discrete shearlet system [Kutyniok & Labate, 2006])

Let ψ ∈ L2(R2). For j ∈ Z and k ∈ R, define the parabolic scaling matrix Aj and the shear matrix Sk by Aj := 22j 2j

• ,

Sk := 1 k 1

• .

(1) The system SH(ψ) := {ψj,k,m := 23j/2ψ(SkAj · −m) | j ∈ Z, k ∈ Z, m ∈ Z2} (2) is the discrete shearlet system defined by ψ. Analogous continuous setting with parameters a ∈ R+, s ∈ R, t ∈ R2.

• G. Kutyniok and D. Labate, Shearlets: Multiscale Analysis for Multivariate Data, 1st ed. New

York: Springer Verlag, 2012.

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Shearlets Comes into Play: What Are They?

Definition (Discrete shearlet system [Kutyniok & Labate, 2006])

Let ψ ∈ L2(R2). For j ∈ Z and k ∈ R, define the parabolic scaling matrix Aj and the shear matrix Sk by Aj := 22j 2j

• ,

Sk := 1 k 1

• .

(1) The system SH(ψ) := {ψj,k,m := 23j/2ψ(SkAj · −m) | j ∈ Z, k ∈ Z, m ∈ Z2} (2) is the discrete shearlet system defined by ψ. Analogous continuous setting with parameters a ∈ R+, s ∈ R, t ∈ R2. Many more evolved constructions (cone-adapted, bandlimited, compactly supported, . . .).

• G. Kutyniok and D. Labate, Shearlets: Multiscale Analysis for Multivariate Data, 1st ed. New

York: Springer Verlag, 2012.

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Shearlets and Wavefront Sets

Resolution of Wavefront Sets (simplified from [Kutyniok & Labate, 2006], [Grohs, 2011])

WF(f)c =

• (t0, s0) ∈ R2 × [−1, 1] : for (t, s) in neighborhood U of (t0, s0):

|SHψf(a, s, t)| = O(ak) as a − → 0, ∀k ∈ N, unif. over U

• where

L2(R2) ∋ f → SHψf(a, s, t) = f, ψa,s,t, (a, s, t) ∈ R+ × R × R2 denotes the continuous shearlet transform.

t0 x1 x2 s0

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Shearlets and Wavefront Sets

Resolution of Wavefront Sets (simplified from [Kutyniok & Labate, 2006], [Grohs, 2011])

WF(f)c =

• (t0, s0) ∈ R2 × [−1, 1] : for (t, s) in neighborhood U of (t0, s0):

|SHψf(a, s, t)| = O(ak) as a − → 0, ∀k ∈ N, unif. over U

• where

L2(R2) ∋ f → SHψf(a, s, t) = f, ψa,s,t, (a, s, t) ∈ R+ × R × R2 denotes the continuous shearlet transform. f is smooth in t0 and shearing direction s = ⇒ fast decay of shearlet coefficients = ⇒ sparsity! wavelets only characterize singular support

• f f (i.e., no directional information)

t0 x1 x2 s0

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Shearlets and Wavefront Sets

Resolution of Wavefront Sets (simplified from [Kutyniok & Labate, 2006], [Grohs, 2011])

WF(f)c =

• (t0, s0) ∈ R2 × [−1, 1] : for (t, s) in neighborhood U of (t0, s0):

|SHψf(a, s, t)| = O(ak) as a − → 0, ∀k ∈ N, unif. over U

• where

L2(R2) ∋ f → SHψf(a, s, t) = f, ψa,s,t, (a, s, t) ∈ R+ × R × R2 denotes the continuous shearlet transform. f is smooth in t0 and shearing direction s = ⇒ fast decay of shearlet coefficients = ⇒ sparsity! wavelets only characterize singular support

• f f (i.e., no directional information)

x1 x2 θ

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

∗-lets and Limited Angle Tomography I

Forward problem carefully analyzed by Frikel (2013) for curvelets The index set of curvelets can be split into a part that is “visible” under Rφ and a part that is “invisible”, i.e., Rφ ψj,k,l = 0 (via Fourier slice theorem).

• J. Frikel, Sparse regularization in limited angle tomography, Appl. Comp. Harm. Anal. 34 (1),

117–141, 2013.

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

∗-lets and Limited Angle Tomography I

Forward problem carefully analyzed by Frikel (2013) for curvelets The index set of curvelets can be split into a part that is “visible” under Rφ and a part that is “invisible”, i.e., Rφ ψj,k,l = 0 (via Fourier slice theorem). Holds for shearlets and other directional representation systems as well

Semi-visible ξ1 ξ2 ξ1 ξ2 Ch Ch Cv Cv R Semi-visible ξ1 ξ2 Wφ Invisible Semi-visible Visible Visible Wedge

• J. Frikel, Sparse regularization in limited angle tomography, Appl. Comp. Harm. Anal. 34 (1),

117–141, 2013.

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

∗-lets and Limited Angle Tomography II

Obtain f =

• (j,k,l)∈Ivis

f, ψj,k,l ψj,k,l +

• (j,k,l)∈Iinv

f, ψj,k,l ψj,k,l = fvis + finv. Use in: SHT

ψ

• argmin

z

z1,w + 1 2 Rφ SHT

ψ (z) − y2 2

• Tatiana Bubba

The Mathematics of X-ray Tomography Very Finnish IP2019

∗-lets and Limited Angle Tomography II

Obtain f =

• (j,k,l)∈Ivis

f, ψj,k,l ψj,k,l +

• (j,k,l)∈Iinv

f, ψj,k,l ψj,k,l = fvis + finv. Use in: SHT

ψ

• argmin

z∈Ivis

z1,w + 1 2 Rφ SHT

ψ (z) − y2 2

• dimension reduction

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

∗-lets and Limited Angle Tomography II

Obtain f =

• (j,k,l)∈Ivis

f, ψj,k,l ψj,k,l +

• (j,k,l)∈Iinv

f, ψj,k,l ψj,k,l = fvis + finv. Use in: SHT

ψ

• argmin

z∈Ivis

z1,w + 1 2 Rφ SHT

ψ (z) − y2 2

• dimension reduction

drawbacks: no positivity constraint, synthesis formulation

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

The Idea

Facts: parts of the WF are available

• nly “here and there”.

shearlets are proven to resolve the WF Idea: use shearlet coefficients to fill in the missing parts of the wavefront set.

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

The Idea

Facts: parts of the WF are available

• nly “here and there”.

shearlets are proven to resolve the WF Idea: use shearlet coefficients to fill in the missing parts of the wavefront set. ℓ1-minimization reconstruction “candy-wrap”

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

The Idea

Facts: parts of the WF are available

• nly “here and there”.

shearlets are proven to resolve the WF Idea: use shearlet coefficients to fill in the missing parts of the wavefront set. ℓ1-minimization reconstruction “candy-wrap” “candy-wrap” shearlet cube: visibile + invisible

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

The Idea

Facts: parts of the WF are available

• nly “here and there”.

shearlets are proven to resolve the WF Idea: use shearlet coefficients to fill in the missing parts of the wavefront set. ℓ1-minimization reconstruction “candy-wrap” “candy-wrap” shearlet cube: visibile + invisible fill in missing coeff. “inpainted” reconstruction

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

The Idea

Facts: parts of the WF are available

• nly “here and there”.

shearlets are proven to resolve the WF Idea: use shearlet coefficients to fill in the missing parts of the wavefront set. ℓ1-minimization reconstruction “candy-wrap” “candy-wrap” shearlet cube: visibile + invisible fill in missing coeff. “inpainted” reconstruction DL?

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Shearlet Cube

Numerically, the shearlet transform does the following: f ∈ Rn×n − → F ∈ Rn×n×L

f ∈ Rn×n

SHψ

F ∈ Rn×n×L

Each subband F (:, :, i) corresponds to the inner products f, ψj,k,·, F is referred to as the shearlet coefficients cube of f.

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

“Candy–wrap” Structure

Due to the sorting, shearlets coefficients follow a specific structure in each scale:

f ∈ Rn×n

SHψ

highest scale

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

“Candy–wrap” Structure

Due to the sorting, shearlets coefficients follow a specific structure in each scale:

f ∈ Rn×n

SHψ

highest scale

Invisibility of limited angle tomography “creates” holes in it:

(a) full-angle CT (b) limited angle CT

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

“Candy–wrap” Prior?

First Idea

Handcraft a prior that promotes this specific “candy–wrap” structure of the shearlet coefficients: enforce continuity of WF. Too complicated?

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

“Candy–wrap” Prior?

First Idea

Handcraft a prior that promotes this specific “candy–wrap” structure of the shearlet coefficients: enforce continuity of WF. Too complicated? “messy” shearlet coefficients easy rule for the human eye, hard to grasp mathematically

(a) shearlet cube for Shepp-Logan (b) “cleaned” shearlet cube

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

“Candy–wrap” Prior?

First Idea

Handcraft a prior that promotes this specific “candy–wrap” structure of the shearlet coefficients: enforce continuity of WF. Too complicated? “messy” shearlet coefficients easy rule for the human eye, hard to grasp mathematically

(a) shearlet cube for Shepp-Logan (b) “cleaned” shearlet cube

Idea

Train a deep neural network to fill in the gaps of the WF.

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Numerical Simulation: Verify the Concept of (In-)Visibility

f gt

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Numerical Simulation: Verify the Concept of (In-)Visibility

f gt FBP

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Numerical Simulation: Verify the Concept of (In-)Visibility

f gt ℓ1 shearlet solution f ∗

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Numerical Simulation: Verify the Concept of (In-)Visibility

f gt SHT

ψ

• SHψ(f FBP)Ivis + SHψ(f gt)Iinv
• Tatiana Bubba

The Mathematics of X-ray Tomography Very Finnish IP2019

Numerical Simulation: Verify the Concept of (In-)Visibility

f gt SHT

ψ

• SHψ(f ∗)Ivis + SHψ(f gt)Iinv
• Tatiana Bubba

The Mathematics of X-ray Tomography Very Finnish IP2019

Numerical Simulation: Verify the Concept of (In-)Visibility

f gt SHT

ψ

• SHψ(f ∗)Ivis + SHψ(f gt)Iinv
• SHψ(f ∗)Ivis close to SHψ(f gt)Ivis

⇒ learning the invisible coefficients should be sufficient

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Numerical Simulation: Verify the Concept of (In-)Visibility

f gt spoiler: SHT

ψ

• SHψ(f ∗)Ivis + F N N

Iinv

• SHψ(f ∗)Ivis close to SHψ(f gt)Ivis

⇒ learning the invisible coefficients should be sufficient

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Some Literature on DL for Inverse Problems

• n FBP:

Kang et al. (2017): contourlets of FBP + U-net, 2nd place Mayo low-dose challenge & many more works from this group! Zhang et al. (2016): 2-layer network on FBP Jin et al. (2017): U-Net on FBP

incorporating forward model via optimization scheme:

Hammernik et al. (2017): learning weights for FBP, then filtering Meinhardt et al. (2017): learning proximal operators Adler et al. (2017): learned primal dual

The Closest Method: Gu & Ye (2017)

“Based on the observation that the artifacts from limited angles have some directional property and are globally distributed, we propose a novel multi-scale wavelet domain residual learning architecture, which compensates for the artifacts.”

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Some Observations

Concerning “denoising” of the FBP (or its coefficients) with DL: missing theory, unclear what the NN really does:

entire image is processed which features are modified? lack of a clear interpretation (?)

NN needs to learn a lot of streaking artifacts (+ noise)

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Some Observations

Concerning “denoising” of the FBP (or its coefficients) with DL: missing theory, unclear what the NN really does:

entire image is processed which features are modified? lack of a clear interpretation (?)

NN needs to learn a lot of streaking artifacts (+ noise) We can do better (in limited angle CT)!

• nly the invisible boundaries need to be learned

shearlets help to access them the coefficients follow the “candy–wrap” structure

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Our Approach

Step 1, “recover the visible”: best available classical solution (little artifacts, denoised) f ∗ := argmin

f≥0

SHψ(f)1,w + 1 2 Rφ f − y2

2

Allows to access WF via sparsity prior on shearlets: for (j, k, l) ∈ Iinv: SHψ(f ∗)(j,k,l) ≈ 0 for (j, k, l) ∈ Ivis: SHψ(f ∗)(j,k,l) reliable and near perfect Step 2, “learn the invisible”: supervised learning of invisible coefficients NN θ : SHψ(f ∗)Iinv F

• !

≈ SHψ(f gt)Iinv

• Step 3, “combine”:

f LtI = SHT

ψ (SHψ(f ∗)Ivis + F )

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Our Approach

y fLtI SHT SH(f∗)Ivis + F Step 1

• nonlin. rec.
• f vis. coeff.

(ADMM) combine Step 3 both parts Σ SH(f∗) Step 2 PhantomNet NN θ learn the invisible F visible coeff. Ivis invisible coeff. Iinv learned coeff.

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Our Approach – Step 2: CNN PhantomNet

Convolutional Neural Network that minimizes over the empirical risk: min

θ

1 N

N

• j=1

NN θ(SH(f ∗

j)) − SH(f j)Iinv2 w,2. (512 × 512, 64) (256 × 256, 128) (128 × 128, 256) (64 × 64, 512) (512 × 512, 128) (256 × 256, 256) (128 × 128, 512) (64 × 64, 512) (128 × 128, 512) (256 × 256, 256) (512 × 512, 128) (512 × 512, 59) (128 × 128, 256) (256 × 256, 128) (512 × 512, 64) Convolution Trimmed-DenseBlock Transition Down Transition Up Copy and Concatenate

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Learning the Invisible

Model based & data driven: only learn what needs to be learned! Possible advantages: faithfulness by learning only what is not visible in the data better performance due to better input NN does not process entire image

less blurring by U-net fewer unwanted artifacts

better generalization

T.A. Bubba, G. Kutyniok, M. Lassas, M. M¨ arz, W. Samek, S. Siltanen and V. Srinivasan, Learning the Invisible: a hybrid deep learning-shearlets framework for limited angle computed tomography, Inverse Problems 2019 (in press).

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Learning the Invisible

Model based & data driven: only learn what needs to be learned! Possible advantages: faithfulness by learning only what is not visible in the data better performance due to better input NN does not process entire image

less blurring by U-net fewer unwanted artifacts

better generalization Disadvantage: speed: dominated by ℓ1-minimization

T.A. Bubba, G. Kutyniok, M. Lassas, M. M¨ arz, W. Samek, S. Siltanen and V. Srinivasan, Learning the Invisible: a hybrid deep learning-shearlets framework for limited angle computed tomography, Inverse Problems 2019 (in press).

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Setup

Experimental Scenarios:

Mayo Clinic5: human abdomen scans provided by the Mayo Clinic for the AAPM Low-Dose CT Grand Challenge.

10 patients (2378 slices of size 512 × 512 with thickness 3mm) 9 patients for training (2134 slices) and 1 patient for testing (244 slices) Mayo-60◦: missing wedge of 60◦ Mayo-75◦: missing wedge of 30◦

Lotus Root: real data measured with the µCT in Helsinki

to check generalization properties of our method (training is on Mayo-60◦) Lotus-60◦: missing wedge of 60◦ Lotus-75◦: missing wedge of 30◦

Operators:

Rφ: Astra toolbox (fanbeam geometry fits lotus root’s acquisition setup) SHψ: α-shearlet transform toolbox (bandlimited shearlets with 5 scales, i.e., 59 subbands of size 512 × 512)

5We would like to thank Dr. Cynthia McCollough, the Mayo Clinic, the American Association

• f Physicists in Medicine (AAPM), and grant EB01705 and EB01785 from the National Institute of

Biomedical Imaging and Bioengineering for providing the Low-Dose CT Grand Challenge data set.

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Mayo-60◦

f gt

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Mayo-60◦

f gt f FBP: RE = 0.50, HaarPSI=0.35

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Mayo-60◦

f gt f TV: RE = 0.21, HaarPSI=0.41

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Mayo-60◦

f gt f ∗: RE = 0.19, HaarPSI=0.43

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Mayo-60◦

f gt f [Gu & Ye, 2017]: RE = 0.22, HaarPSI=0.40

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Mayo-60◦

f gt NN θ(f FBP): RE = 0.16, HaarPSI=0.53

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Mayo-60◦

f gt NN θ(SH(f FBP)): RE = 0.16, HaarPSI=0.58

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Mayo-60◦

f gt f LtI: RE = 0.09, HaarPSI=0.76

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Mayo-75◦

f gt

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Mayo-75◦

f gt f FBP: RE = 0.30, HaarPSI=0.46

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Mayo-75◦

f gt f TV: RE = 0.10, HaarPSI=0.63

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Mayo-75◦

f gt f ∗: RE = 0.09, HaarPSI=0.64

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Mayo-75◦

f gt f [Gu & Ye, 2017]: RE = 0.21, HaarPSI=0.43

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Mayo-75◦

f gt NN θ(f FBP): RE = 0.09, HaarPSI=0.82

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Mayo-75◦

f gt NN θ(SH(f FBP)): RE = 0.06, HaarPSI=0.82

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Mayo-75◦

f gt f LtI: RE = 0.03, HaarPSI=0.92

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Lotus-60◦

f gt

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Lotus-60◦

f gt f FBP: RE = 0.49, HaarPSI=0.49

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Lotus-60◦

f gt f TV: RE = 0.21, HaarPSI=0.60

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Lotus-60◦

f gt f ∗: RE = 0.19, HaarPSI=0.61

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Lotus-60◦

f gt f [Gu & Ye, 2017]: RE = 0.42, HaarPSI=0.56

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Lotus-60◦

f gt NN θ(SH(f FBP)): RE = 0.27, HaarPSI=0.63

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Lotus-60◦

f gt f LtI: RE = 0.15, HaarPSI=0.74

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Lotus-75◦

f gt

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Lotus-75◦

f gt f FBP: RE = 0.30, HaarPSI=0.36

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Lotus-75◦

f gt f TV: RE = 0.12, HaarPSI=0.80

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Lotus-75◦

f gt f ∗: RE = 0.10, HaarPSI=0.79

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Lotus-75◦

f gt f [Gu & Ye, 2017]: RE = 0.25, HaarPSI=0.65

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Lotus-75◦

f gt NN θ(SH(f FBP)): RE = 0.19, HaarPSI=0.80

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Lotus-75◦

f gt f LtI: RE = 0.08, HaarPSI=0.88

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019

Take-Home Message

limited angle CT is a special inverse problem

visible and invisible features

ℓ1-minimization with shearlets

access visible part of WF negligible invisible part

learn the invisible parts with a deep NN

3D “inpainting” problem regularity assumptions on f

faithful approach: limit influence of DL

no explanation of DL but clearer concept what is happening!

Tatiana Bubba The Mathematics of X-ray Tomography Very Finnish IP2019