Compressive Imaging EE367/CS448I: Computational Imaging and Display - - PowerPoint PPT Presentation

Compressive Imaging

Gordon Wetzstein Stanford University EE367/CS448I: Computational Imaging and Display stanford.edu/class/ee367 Lecture 11

• whiteboard derivations of square matrix, over-determined, under-

determined

• least-norm solution

Motivation

Single Pixel Camera

Wakin et al. 2006

Single Pixel Camera

• riginal

10% 5% 2%

Wakin et al. 2006

Single Pixel Camera – Image Formation

… … , = = =

measurement matrix measurements

, ,

b

A

• Shannon: highest frequency in image with NxN pixels is N/2
• compressive sensing: only need M<N measurements!
• magic? à no, relies on sparsity of sampled image
• what’s sparsity? number of non-zero elements in image
• a signal is k-sparse if there are k non-zero elements

Beyond Shannon Nyquist

Natural Images are Sparse (in some transform domain)

wikipedia cnx.org

discrete cosine transform Haar wavelets

∇xx ∇yx x

Natural Images are Sparse (in some transform domain)

better: isotropic

∇xx

( )

2 + ∇yx

( )

2

x ∇xx

( )

2 +

∇yx

( )

2

easier: anisotropic

Natural Images are Sparse (in some transform domain)

• general idea:

Compressive Sensing – Synthesis Problem

b = Ax = AΨs

• ptics

computation

b

A Ψ s

Compressive Sensing – Analysis Problem

minimize

x

{ }

Kx 1 subject to b = Ax

we will use “K” as the matrix transforming the signal into some (sparse) domain

basis pursuit denoise

Compressive Sensing – Analysis Problem

minimize

x

{ }

Kx 1 subject to b = Ax b − Ax 2

2 ≤ ε

• r

basis pursuit

minimize

x

{ }

1 2 b − Ax 2

2 + λ Kx 1

Basis Pursuit Denoise / LASSO

minimize

x

{ }

Kx 1 subject to b − Ax 2

2 ≤ ε

• there is a that makes both problems equivalent, but we don’t

know what that is à manually tune parameters

λ

minimize

x

{ }

1 2 b − Ax 2

2 + λΓ x

( ) Regularized Image Reconstruction

some image prior, such as norm or others

ℓ1

data fidelity term

minimize

x

{ }

1 2 b − Ax 2

2 f (x)

! " # $ # + λΓ z

( )

g(z)

%

subject to Kx − z = 0

Regularized Image Reconstruction

• split into two parts à mathematically equivalent

Lρ x,z,y

( ) = f (x)+ g(z)+ yT Kx − z ( )+ ρ

2 Kx − z 2

2

Regularized Image Reconstruction

• Augmented Lagrangian

x ← prox ⋅ 2,ρ v

( ) = argmin

x

{ }

Lρ x,z,y

( ) = argmin

x

{ }

1 2 Ax − b 2

2 + ρ

2 Kx − v , v = z − u z ← proxΓ,ρ v

( ) = argmin

z

{ }

Lρ x,z,y

( ) = argmin

z

{ }

λΓ z

( )+ ρ

2 v − z , v = Kx + u u ← u + Kx − z

Regularized Image Reconstruction

• iterative updates - ADMM

repeat until converged

x ← prox ⋅ 2,ρ v

( ) = argmin

x

{ }

Lρ x,z,y

( ) = argmin

x

{ }

1 2 Ax − b 2

2 + ρ

2 Kx − v , v = z − u z ← proxΓ,ρ v

( ) = argmin

z

{ }

Lρ x,z,y

( ) = argmin

z

{ }

λΓ z

( )+ ρ

2 v − z , v = Kx + u u ← u + Kx − z

Regularized Image Reconstruction

• iterative updates - ADMM

repeat until converged

prox ⋅ 2,ρ v

( ) = argmin

x

{ }

1 2 Ax − b 2

2 + ρ

2 Kx − v

Regularized Image Reconstruction

… see whiteboard / notes ...

prox ⋅ 2,ρ v

( ) = argmin

x

{ }

1 2 Ax − b 2

2 + ρ

2 Kx − v

Regularized Image Reconstruction

• x-update: solve
• symmetric, positive definite matrix à conjugate gradient method
• … see whiteboard / notes ...

prox ⋅ 2,ρ v

( ) = AT A + ρK TK

A !

" # $ $ % $$ ⎛ ⎝ ⎜ ⎜ ⎞ ⎠ ⎟ ⎟

−1

ATb + ρK Tv

b &

" # $ % $ ⎛ ⎝ ⎜ ⎜ ⎞ ⎠ ⎟ ⎟ A !x = b

"

Regularized Image Reconstruction

… depends on prior ...

proxΓ,ρ v

( ) = argmin

z

{ }

λΓ z

( )+ ρ

2 v − z

Regularized Image Reconstruction

TV prior

proxΓ,ρ v

( ) = argmin

z

{ }

λΓ z

( )+ ρ

2 v − z prox ⋅ 1,ρ v

( ) = argmin

z

{ }

λ z 1 + ρ 2 v − z = Sλ/ρ v

( )

• also:
• K = D =

Dx Dy ⎡ ⎣ ⎢ ⎢ ⎤ ⎦ ⎥ ⎥

Regularized Image Reconstruction

NLM prior

proxΓ,ρ v

( ) = argmin

z

{ }

λΓ z

( )+ ρ

2 v − z proxNLM,ρ v

( ) = NLM v, λ

ρ ⎛ ⎝ ⎜ ⎞ ⎠ ⎟

• also:
• K = I

variance variance of denoising

Applied to Single Pixel Camera

Applied to Single Pixel Camera

Applied to Single Pixel Camera – TV Prior

1x 2x 4x 8x

Applied to Single Pixel Camera – NLM Prior

1x 2x 4x 8x

Applications of Compressive Imaging

• reduce acquisition time, radiation exposure, or allow for more

patients in same time, …

• examples: x-ray computed tomography and MRI

Compressive Medical Imaging

This slide has a 16:9 media window Computed Tomography (CT)

Image: Wikipedia

x-ray source x-ray sensor

3D Reconstruction Reconstructed 2D Slices

This slide has a 16:9 media window Computed Tomography – Fourier Slice Theorem

primal domain frequency domain

• measurements = Fourier slices
• compressive CT: e.g. fewer slices

Magnetic Resonance Imaging

frequency domain

• measurements = (random) Fourier coefficients
• compressive MRI: fewer Fourier coefficients

wikipedia

• people in bio-medical imaging often hesitant about priors:
• few guarantees for success
• if reconstruction breaks, not clear how exactly
• is that feature a reconstruction artifact or the thing I’m looking for?

Compressive Imaging: CT & MRI

Compressive Hyperspectral Imaging

• motivation:
• conventional: either scan over xy or over lambda!
• idea: capture hyperspectral datacube with a single, coded image

– use compressive sensing to reconstruct

• first approach: CASSI (coded aperture snapshot spectral imager),

Wagadarikar 2008

x y λ

Compressive Hyperspectral Imaging

Arce et al. 2014

Compressive Hyperspectral Imaging

Arce et al. 2014

Compressive Hyperspectral Imaging

Arce et al. 2014

• moderate quality for snapshot, but good quality for coded multi-shot
• applications: remote sensing, cultural heritage, …

Compressive Light Field Imaging

sensor lenslet

f

Fixed trade-off between spatial and angular resolution!

Integral Imaging – Lippmann 1908

Scene from Above

• Lenslet Array

[Lippman 1908], [Adelson and Wang 1992], [Ng et al. 2005] Integral Imaging

• # Sensor Pixels X
• # Sensor Pixels Y

Scene from Above

• Integral Imaging

Scene from Above

• # Sensor Pixels X / # Views X

# Sensor Pixels Y / # Views Y Integral Imaging: Spatio-Angular Resolution Tradeoff!

• Spatio-Angular Resolution Tradeoff!

Key Insight: Light Field is Redundant!

• Scene from

Above

Exploit Redundancy Computationally Sparse Reconstruction Overcomplete Dictionaries Light Field Atoms Sparse Representation Optimal Optical Setup Mask-based Light Field Coding

Compressive Light Field Photography

ACM SIGGRAPH 2013

Optical Preservation of Light Field

=

Coded projection

=

Overcomplete dictionary

Dictionary Coefficient vector Image Light field

Light field atoms

We need to be able to distinguish atoms from their projections

Scene from Above Previous Mask-Coded Light Field Projection

• Parallax Barriers

[Ives 1903] Sum of Sinusoids or MURA [Veeraraghavan 2007, Lanman 2008]

• Multiplexing + linear reconstruction
• Low resolution light fields similar to the lenslets design

“On Plenoptic Multiplexing and Reconstruction”, IJCV, Wetzstein et al. 2013

Scene from Above Proposed Technology: Optimized Mask w.r.t. the Dictionary

• =

Coded projection

=

Overcomplete dictionary

Dictionary Coefficients Image

Light field atoms

• Multiplexing + nonlinear reconstruction
• Higher spatial resolution

Imaging Lens Image sensor Polarizing Beamsplitter Virtual sensor LCoS Camera

Coded Attenuation Mask

Captured 2D Image 4D Reconstruction

=

Mask-Coded Projection 4D Light Field

Basis Pursuit Denoise:

Sparse Coefficients!

Compressive Light Field Photography

ACM SIGGRAPH 2013

b = Φx = ΦΨs

minimize

s

s 1 subject to b − ΦΨs 2

2 ≤ ε

Captured 2D Image

Compressive Light Field Photography

ACM SIGGRAPH 2013

• massively parallel à move to cloud?!
• better: replace with convolutional sparse coding!

• metamaterials
• THz imaging
• x-ray imaging
• thermal IR
• ultra-fast imaging
• not as much on compressive coherent imaging (could be interesting

for course projects: OCT, holography, …)

• …

Compressive Imaging Everywhere

Notes

• compressive imaging is an exploding area: check COSI, ICCP,

CVPR, ICCV conferences, other optics journals and conferences

• most variants of compressive imaging problems can be

implemented with ADMM (implemented in HW3)

• check lecture notes online to help with homework
• pen research questions: how to design good sensing matrices

given constraints of optics

References and Further Reading

• Michael Wakin, Jason Laska, Marco Duarte, Dror Baron, Shriram Sarvotham, Dharmpal Takhar, Kevin Kelly, and

Richard Baraniuk, An Architecture for Compressive Imaging (Proc. International Conference on Image Processing -- ICIP 2006, Atlanta, GA, Oct. 2006)

• E. J. Candes, J. K. Romberg, and T. Tao, "Stable Signal Recovery from Incomplete and Inaccurate Measurements,"

Communications on Pure and Applied Mathematics, Vol. LIX, 1207–1223 (2006)

• Candes, Wakin, “An Introduction to Compressive Sampling”, IEEE Signal Processing Magazine, 2008
• Aharon, Elad, Bruckstein, “K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse

Representation”, IEEE Trans. Im. Proc. 2006

• Arce, Brady, Carin, Arguello, Kittle “Compressive Coded Aperture Spectral Imaging: An Introduction”, IEEE Signal
• Proc. Magazine 2014
• Wagadarikar, R. John, R. Willett, and D. Brady, “Single disperser design for coded aperture snapshot spectral

imaging,” Appl. Opt., vol. 47, pp. B44–B51, Apr. 2008