Performance Bounds for Computational Imaging Oliver Cossairt - - PowerPoint PPT Presentation

Performance Bounds for Computational Imaging

Oliver Cossairt Assistant Professor Northwestern University

Collaborators: Mohit Gupta1, Changyin Zhou1, Daniel Miau1, Shree Nayar1, Kaushik Mitra2, Ashok Veeraraghavan2

1Columbia University 2Rice University

• Take multiple pictures and computationally combine

Computational Imaging: Increased Functionality

Others

• Multiview Stereo
• Depth from Focus/Defocus
• Tomography
• Structured Light
• Deconvolution microscopy
• etc.

HDR Imaging Panoramic Stitching Digital Holography

[Greenbum et al. ’12]

Image-Based Lighting

[Debevec et al. ’00]

Light Field Capture

[Wilburn et al. ’04]

Coded image capture for increased performance

Computational Imaging: Increased Performance

Reflectance

[Schechner ‘03] [Ratner ‘07] [Ratner ‘08]

Defocus Blur

[Hausler ’72] [Nagahara ’08] [Levin et al. ’07] [Zhou, Nayar ’08] [Dowski, Cathey ‘96]

Motion Blur

[Raskar ’06] [Levin ‘08] [Cho ’10]

Multi/Hyper-Spectral

[Sloane ’79] [Hanley ’99] [Baer ‘99] [Wetzstein et al., ’12]

Light Field Capture

[Lanman ’08] [Veeraraghavan ’07] [Liang ‘08]

Coded Aperture

[Mertz ’65] [Gottesman ’89]

Computational Imaging Performance

Time 50 millisec Short Exposure

Camera Exposure Camera Exposure Time 50 millisec

Long Exposure

Vs.

Computational Imaging Performance

Camera Exposure Time 50 millisec

Coded Exposure Time 50 millisec Short Exposure

Vs.

Deblurred Image

When does computational imaging improve performance?

Camera Exposure

[Raskar ’06]

Measuring Computational Imaging Performance

• No diffraction
• Fully determined

Image Formation Model

Assumption: A) Linear model of incoherent image formation Scene Coded Image Computational Camera Image Optical Coding Equation

Coded Image Coding Matrix Noise

• Signal dependent / independent noise
• Ignore Dark current, fixed pattern

Affine Noise Model

Assumption: B) Affine noise model (photon noise is Gaussian) Noise Variance at kth Pixel:

photon noise aperture, lighting, pixel size read noise electronics, ADC’s, quantization

• Photon noise modeled as Gaussian

(ok for more than 10 photons)

• Photon noise spatially averaged

Noise PDF:

Signal-level and photon noise depend on illumination

Lighting Conditions

Assumption: C) Naturally occurring light conditions for photography

Ex) q=.5, R = .5, F/8, t = 6ms, p=6um

Isrc (lux)

Quarter moon Full moon Twilight Indoor lighting Cloudy day Sunny day

10-2 1 10 102 103 104 8×10-3 0.8 8.1 81.4 814.3 8143

J (e-) Signal level Illumination

Illumination (lux) Aperture Exposure Time (s) Pixel Size (m) Quantum Efficiency Reflectivity Average Signal (e-)

[Cossairt et al. TIP ‘12]

For Gaussian noise, Mean-Squared-Error (MSE) can be computed analytically

Measuring Performance

Observation: 1) Multiplexing performance depends on coding matrix Ex) Coded Motion Deblurring

Long Exposure Coded Exposure

Impulse imaging (identity sampling)

Multiplexing vs. Impulse Imaging

Observation: 2) Multiplexing increases signal-dependent noise

Noise variance

Coded imaging (multiplexed sampling)

Noise variance Increased throughput

Observation: 3) Performance depends on multiplexing and signal prior

Multiplexing vs. Impulse Imaging

SNR Gain over impulse imaging:

Decreases with C Noise Dependent Increases with C Coding Dependent

Coded Aperture Astronomy

Decreasing contrast Increasing scene points

[Mertz ’65]

Fresnel zone plate

[Sloane ’79]

Hadamard Multiplexing:

No SNR gain for large signal

Assume we have a PDF for images, e.g.

Image Prior Models

Assumption: D) Signal prior models naturally occurring images MSE difficult to express analytically when Compute the Maximum A Posteriori (MAP) estimate

Data term Prior term

Other priors

• Total Variation (TV)
• Wavelet/sparsity prior
• Learned priors (K-SVD)

Power Spectra Prior

Image Priors and Noise

Observation: 4) Signal priors help more at low light levels

PSNR = 5.5 dB

Twilight (10 lux)

PSNR = 16.4 dB

Denoise

Daylight (10 lux)

5

PSNR = 35 dB PSNR = 35.9 dB

Denoise

When does computational imaging improve performance? Observations: 1) Multiplexing performance depends on coding matrix 2) Multiplexing helps most in low light 3) Performance depends on both multiplexing and signal prior 4) Signal priors help most in low light Assumptions A) Incoherent imaging B) Affine noise model C) Natural lighting conditions D) Natural image prior

Example: Motion Deblurring

Motion Deblurring vs. Impulse Imaging

Camera Exposure Time 50 millisec

Computational Imaging (Coded Exposure) Time 50 millisec Impulse Imaging (Short Exposure)

Vs.

What is the best possible coding performance we can get?

Optical efficiency (C) = total ‘on’ time

[Ratner ‘07]

When Does Motion Deblurring Improve Performance?

Motion Invariant Levin et al. Flutter Shutter Raskar et al.

Upper Bound on SNR Gain:

Performance depends only on lighting conditions!

[Cossairt et al. TIP ‘12]

Read Noise Average signal level

q=.5, R = .5, F/2.1, p = 1um,

Maximum object speed (pixels/sec)

Twilight (10 lux)

Flutter Shutter Simulation

Impulse (4ms) Flutter Shutter (180ms) Deblurred Cloudy Day (10 lux)

3

PSNR = 12.4 dB PSNR = 10.1 dB

q=.5, R = .5, F/2.1, pixel size = 1um, read noise

PSNR = -7.2 dB PSNR = -3.0 dB

Example: Extended DOF Imaging

F 1.4 F 2.8 F 5.6 F 8.0

Lens Image

Depth of Field and Noise

Small apertures have large depth of field and low SNR

Lens Sensor

Focal Sweep

400 600 900 1500 2000 1200 1700 (depth)

Point Spread Function (PSF)

[Hausler ‘72, Nagahara et al. ‘08]

Focal Sweep

400 600 900 1500 2000 (depth) 1200 1700

+ + + + + + =

t = 1 t = 2 t = 3 t = 4 t = 5 t = 6 t = 7

(400) (600) (900) (1500) (2000) (1200) (1700)

Integrated PSF

[Hausler ‘72, Nagahara et al. ‘08]

Lens Sensor

Quasi Depth Invariant PSF

Extended depth of field with a single deconvolution

1200mm 750mm 400mm 2000mm

• 25 0 25

0.016 0.012 0.008 0.004

1 60

1200mm

• 25 0 25

0.016 0.012 0.008 0.004

750mm 400mm 2000mm

×

Traditional Camera PSF Focal Sweep PSF

Focal Sweep: Captured

75 m 50 m

Extended Depth of Field Telescope

Meade LX200 8’’ Telescope

Traditional Image

75 m 50 m 75 m 50 m

Focal Sweep: Processed

2000mm FL

Focal Sweep (F/2.0) Traditional (F/2.0) Twilight (10 lux)

Pixel size = 5um Read noise

Focal Sweep Simulation

Daylight (10 lux)

5

PSNR = 35 db PSNR = 38.5 db

Traditional (F/20.0)

PSNR = 5.5 db PSNR = 18.5 db

Focal Sweep (F/2.0)

Focal Sweep Simulation (with Prior)

Daylight (10 lux)

5

PSNR = 35.9 dB / 35 dB PSNR = 39.6 dB / 38.5 dB

BM3D Algorithm: [Dabov et al. ‘06] Twilight (10 lux) Traditional (F/20.0)

PSNR = 16.4 dB / 5.5 dB PSNR = 22.8 dB / 18.5 dB

Traditional (F/2.0)

Pixel size = 5um Read noise

State-of-the-art priors are hard to analyze!

A Universal Image Prior

Gaussian Mixture Model (GMM) prior

Cluster weight Cluster mean Cluster covariance Number of clusters

• GMM parameters learned from database of 30 Million image patches
• Analytic expression for MSE depending only on

Coding matrix Noise variance GMM parameters

[Mitra et al., submitted to PAMI ‘13]

Focal Sweep Performance with GMM Prior

[Mitra et al., submitted to PAMI ‘13]

SNR gain (in dB) Illuminance (in lux)

Multiplexing gain with prior Multiplexing gain without prior

Full moon Twilight Indoor lighting Cloudy day Sunny day

Gain due to prior alone

q=.5, R = .5, t=6ms, p = 1um, Impulse camera: F/11, Focal Sweep: F/1

Defocus Deblurring Performance with GMM Prior

[Mitra et al., submitted to TIP ‘13]

Defocus Deblurring gain as high as 8dB for Cubic Phase Plate SNR gain (in dB)

Cubic Phase

Conclusions

• Results for Motion Deblurring, EDOF also applicable to many
• ther computational cameras
• Computational imaging performance should always be

measured relative to impulse imaging

• Computational imaging performance depends jointly on

multiplexing, noise, and signal priors

• Important question: “How much performance improvement

from multiplexing above and beyond use of signal priors?”

http://en.wikipedia.org/wiki/Focus_stacking

Depth of Field

Microscope Tachinid Fly Small DOF

http://en.wikipedia.org/wiki/Focus_stacking

Depth of Field

Microscope Tachinid Fly Large DOF

Telephoto Focal Sweep with Deformable Optics

Canon 800mm EFL Lens Deformable Lens Sensor

[Miau et al. ICCP ‘13]

Video Quality Comparison

Conventional EDOF (Deformable Lens)

Noise Variance: Impulse Camera

lens sensor A

Focal Sweep Performance

Mean-Squared Error: Noise Variance:

light increase

Focal Sweep

lens diffuser sensor C*A

Mean-Squared Error:

[Cossairt et al. TIP ‘12]

When Does Defocus Deblurring Improve Performance?

Performance depends only on lighting conditions!

[Cossairt et al. TIP ‘12]

Read Noise Average signal level

Focal sweep multiplexing gain can be expressed analytically

q=.5, R = .5, t = 20ms, p = 5um,

Lens Image

Focal Sweep Without Moving Parts

Focal Sweep

Lens Image

Diffusion Coding

Radial Diffuser

(No Moving Parts)

[Cossairt et al. Siggraph ‘10]

500 x 3 micron

What is the Optimal Aperture Size?

Too Noisy Impulse Camera F/16

1m 4m 2m depth

• f field

lens sensor

Pro: 64x Light (SNR ) Con: Deblur (SNR )

lens 2o diffuser sensor

Focal Sweep

1m 4m 2m depth

• f field

Performance is independent of aperture size

Diffusion Coding F/1.8 (Deblurred) Traditional F/1.8 Diffusion Coding F/1.8 (Captured)

Diffusion Coding vs. Traditional Camera

Traditional F/18 (Normalized)

Example: Deblurring vs. Denoising

Example: Computational Aberration Removal

Computational Gigapixel Camera

Ball Lens Lens Array Ball Lens Sensor Array

Computations

Gigapixel Image

Also See: MOSAIC Program, Duke, UCSD, Distant Focus

Ball Lens Sensor Pan/Tilt Motor

Computational Camera Design Prototype Camera

Resolution vs. Lens Scale

Pixels Point Spread Function

• 50

50

PSF size increases linearly

Resolution vs. Lens Scale

Pixels Point Spread Function

• 50

50

PSF size increases linearly

Scale (M) RMS Deblurring Error

1 12 25 37 50 62 75 87 100 0.02 0.04 0.06 0.08 0.10 0.12 0.14

Deblurring Error is sub-linear

[Cossairt et al. JOSA ‘11]