Burst Photography ! EE367/CS448I: Computational Imaging and Display ! stanford.edu/class/ee367 ! Lecture 7 ! Gordon Wetzstein ! Stanford University !
Motivation ! wikipedia ! exposure sequence ! -4 stops !
Motivation ! wikipedia ! exposure sequence ! -2 stops !
Motivation ! wikipedia ! exposure sequence ! 2 stops !
wikipedia ! 4 stops !
Motivation ! wikipedia ! HDR ! contrast reduction (scaling) !
wikipedia ! HDR ! local tone mapping !
Computational Photography - Overview � • high dynamic range � Debevec & Malik, 1997 � • super-resolution � • burst photography � • focal stack � • aperture stack � • confocal stereo � • blurry/noisy � • flash/no flash � • multi-flash �
High Dynamic Range Imaging ! • ! dynamic range: ratio between brightest and darkest value ! • ! quantization within that range is equally important ! ! from 8 bits (256 values) to 32 bits floating point ! Debevec & Malik, 1997 ! original photo ! simulation from LDR ! simulation from HDR ! motion blurred photo !
HDRI – Overview � • estimate camera response curve � • capture multiple low dynamic range (LDR) exposures � • fuse LDR images into 32 bit HDR image � • possibly convert to absolute radiance (global scaling) � • application specific use: � • image-based rendering lighting � • tone mapping � • … �
HDRI – Estimating the Response Curve � • not required when working with linear RAW images � • easiest option: use calibration chart �
HDRI – Estimating the Response Curve ! • ! not required when working with linear RAW images ! • ! easiest option: use calibration chart ! linear RAW ! 1 known reflectance ! 0 64 ! 128 ! 196 ! 255 ! pixel value !
HDRI – Estimating the Response Curve ! • ! not required when working with linear RAW images ! • ! easiest option: use calibration chart ! e.g. JPEG ! 1 known reflectance ! 0 64 ! 128 ! 196 ! 255 ! pixel value !
HDRI – Linearizing LDR Exposures ! • ! capture exposure, apply lookup table ! e.g. JPEG ! ( ) I lin = f ! 1 I 1 f ! 1 " ( ) relative radiance ! I 0 64 ! 128 ! 196 ! 255 ! pixel value !
� HDRI – Merging LDR Exposures � • start with LDR image sequence I i (only exposure time t i changes) � ( ) I i = f t i X • individual exposure is: , f is camera response function � Image from Debevec & Malik, 1997 �
� HDRI – Merging LDR Exposures � ( ) I lin i = f − 1 I i • undo the camera response: � ( ) = I 1/ γ ( ) = I γ f − 1 I → f I e.g. gamma function � Image from Debevec & Malik, 1997 �
! HDRI – Merging LDR Exposures ! • ! compute a weight (confidence) that a pixel is well-exposed ! ! (close to) saturated pixel = not confident, pixel in center of dynamic ( ) " % range = confident! ! 2 I lin ij ! 0.5 $ ' w ij = exp ! 4 or mean pixel value, ! $ ' 0.5 2 e.g. 127.5 if I in [0, 255] ! # &
HDRI – Merging LDR Exposures � ( ) ⎛ ⎞ • 2 compute per-color-channel-per-LDR-pixel weights � I lin ij − 0.5 ⎜ ⎟ w ij = exp − 4 ⎜ ⎟ 0.5 2 ⎝ ⎠
HDRI – Merging LDR Exposures � • define least-squares objective function in log-space à perceptually ( ) ( ) − log t i X ( ) ∑ 2 linear: � minimize O= w i log I lin i i X • equate gradient to zero: � ( ) ∂ O ( ) − log t i ∑ ( ) − log X ( ) ( ) = 2 = 0 w i log I lin i ∂ log X i ( ) ( ) − log t i ( ) ∑ ⎛ ⎞ w i log I lin i ! = exp ⎜ ⎟ i • X gives: � ∑ ⎜ ⎟ w i ⎝ ⎠ i
HDRI – Merging LDR Exposures ! • ! define least-squares objective function in log-space ! perceptually ( ) ( ) ! log t i X ( ) " 2 linear: ! minimize O= w i log I lin i i X • ! equate gradient to zero: ! ( ) ! O ( ) " log t i # ( ) " log X ( ) ( ) = 2 = 0 w i log I lin i ! log X i ( ) ( ) ! log t i ( ) " # & w i log I lin i ! = exp % ( i • ! X gives: ! " % ( w i $ ' i
! HDRI – Relative v Absolute Radiance ! • ! LDR to HDR only gives relative radiance (HW4!) ! • ! scale by reference radiance to get absolute! Image from Debevec & Malik, 1997 !
Image-based Lighting with Light Probes ! • ! text ! Paul Debevec !
Image-based Lighting with Light Probes ! • ! single light probe covers light incident from (almost) entire hemisphere! !
Paul Debevec, Renderign with Natural Light ! SIGGRAPH Electronic Theater 1998 ! Image Based Lighting !
HDRI – Tone Mapping � • how to display a high dynamic range image on an LDR display? � • tone mapping: fit into luminance range of display (or 0-255), while preserving image details � • HW4 �
HDRI – Tone Mapping � • sun overexposed � • foreground too dark � [Durand and Dorsey, 2002] �
HDRI – Global Tone Mapping � [Durand and Dorsey, 2002] � • gamma correction: � I = I γ • colors are washed out �
HDRI – Global Tone Mapping � [Durand and Dorsey, 2002] � • gamma in intensity only! � • intensity details lost �
HDRI – Gradient-domain Tone Mapping ! • ! compute gradients, scale them, integrate (Poisson eq.) ! [Fattal et al., 2002] ! HDR image (scaled) !
HDRI – Gradient-domain Tone Mapping ! • ! compute gradients, scale them, integrate (Poisson eq.) ! [Fattal et al., 2002] ! gradients ! HDR image (scaled) !
HDRI – Gradient-domain Tone Mapping ! [Fattal et al., 2002] ! gradient attenuation map ! tone mapped result !
HDRI – Tone Mapping with Bilateral Filter ! Input HDR image ! Output ! Large scale (base layer) ! Intensity ! Large scale ! [Durand and Dorsey, 2002] ! Reduce ! contrast ! Detail ! Fast ! Bilateral ! Preserve! ! Filter ! Color ! Detail ! Color !
HDRI – Tone Mapping with Bilateral Filter ! • ! difference is not too big ! [Durand and Dorsey, 2002] ! Gradient-space [Fattal et al.] ! Bilateral [Durand et al.] !
HDRI – Tone Mapping with Bilateral Filter ! • ! bilateral “looks” a bit better ! • ! no ground truth ! it’s up [Durand and Dorsey, 2002] ! to the user ! Gradient-space [Fattal et al.] ! Bilateral [Durand et al.] !
HW4, Q1 & Q2 � • Q1: HDR image fusion (from series of different LDR exposures) � • Q2: tone-map HDR image with � • global gamma correction on all color channels � • global gamma correction on intensity channel � • local tone mapping with bilateral filter �
Burst Photography - Overview ! • ! basic idea: capture and merge bursts of photos (2 or more): ! • ! • multiple exposures: HDR but also deblurring … multiple exposures: HDR but also deblurring … ! • ! multiple shifted low-res images: super-resolution ! • ! focal stack ! • ! aperture stack ! • ! noisy + blurry: denoising + deblurring ! • ! flash / no flash ! • ! multi-flash !
Pixel Super-Resolution � • increase “pixel count”, not related to diffraction limit � • idea: capture multiple low-res (LR) images and fuse them into a single super-resolved (SR) image � [Ben-Ezra et al., 2004] � Super-Resolution �
Pixel Super-Resolution ! light l16 !
light l16 !
Pixel Super-Resolution ! • ! LR must be sub-pixel shifted ! I 1 ! $ ! $ I 1 A 1 # & = # & I SR # & # & I 2 A 2 " % " % ! ! A b I SR stacked, measured ! downsampling & ! LR images ! phase shift ! I 2
Pixel Super-Resolution ! • ! example for 1D scanline ! I 1 ! = I SR I SR b A I 2
Pixel Super-Resolution � • in general: system is well-conditioned for non-integer pixel shifts and super-resolution factors of 2-3x (don’t necessarily need priors) � • HW 4, Q3: solve (large-scale) pixel super-resolution with gradient 1 descent to � 2 2 AI SR − b 2 minimize I SR
! HW4 – Q3 ! ( ) = x ! " A T r x = x ! " A T Ax ! b • ! gradient descent: ! • ! use matrix-free functions to implement matrix-vector multiplications! ! I SR I SR SR Ax() is already I 1 I 2 implemented, generate your I SR I SR own 4 low-res images, then SR SR implement Atx() and solve ! I 3 I 4
Overview of Other Techniques �
Focal Stack ! find highest gradient ! focal stack ! contributions ! all-in-focus image ! wikipedia ! • ! implemented in a range of products… !
Aperture Stack � • what changes? exposure and depth of field – extract HDR & depth! � refocus front � f/8 � [Hasinoff and Kutulakos 2007] � f/4 � refocus f/2 � rear � layered � depth map �
Confocal Stereo ! • ! idea: intensity of in-focus point remains constant for varying aperture ! [Hasinoff and Kutulakos, 2006] !
Confocal Stereo ! • ! aperture !" capture aperture and ( aperture ! i , focus f j ) ! focal stack ! [Hasinoff and Kutulakos, 2006] ! • ! for each pixel: find focus setting where aperture stack is most invariant ! focus f "
Confocal Stereo ! aperture !" [Hasinoff and Kutulakos, 2006] ! photograph ! estimated depth map ! focus f "
Recommend
More recommend
