C OMPUTATIONAL A SPECTS OF C OMPUTATIONAL D IGITAL P HOTOGRAPHY P - - PowerPoint PPT Presentation
CS 89.15/189.5, Fall 2015 C OMPUTATIONAL A SPECTS OF C OMPUTATIONAL D IGITAL P HOTOGRAPHY P HOTOGRAPHY Filtering & convolution Wojciech Jarosz wojciech.k.jarosz@dartmouth.edu Your Spanish castle illusions CS 89/189: Computational
Filtering & convolution
COMPUTATIONAL ASPECTS OF DIGITAL PHOTOGRAPHY
Wojciech Jarosz wojciech.k.jarosz@dartmouth.edu
COMPUTATIONAL PHOTOGRAPHY
CS 89.15/189.5, Fall 2015
Your Spanish castle illusions
CS 89/189: Computational Photography, Fall 2015 2
Timelapse photography in the news
CS 89/189: Computational Photography, Fall 2015 11
Today’s agenda
Linear filtering & convolution
Complexity analysis
Denoising from a single image
CS 89/189: Computational Photography, Fall 2015 13
Image processing motivation
Sharpen images Downsample images Fake depth of field Smooth out noise, skin blemishes ... We must understand convolution!
CS 89/189: Computational Photography, Fall 2015
15
After a slide by Frédo Durand
CS 89/189: Computational Photography, Fall 2015
Sharpening
16
After a slide by Frédo Durand
Yikes! Herringbone patterns
Downsampling
17
After a slide by Frédo Durand
Downsample by a scale of 0.2
We “randomly” pick a color in the high frequency pattern
Downsampling
Downsample by a scale of 0.2
18
After a slide by Frédo Durand
Solution: blur the pattern to get average color over new pixels
Downsampling
Blur Down sample
19
After a slide by Frédo Durand
Fake tilt shift
http://www.tiltshiftphotography.net/photoshop-tutorial.php
CS 89/189: Computational Photography, Fall 2015
20
After a slide by Frédo Durand
Blur in optics
Diffraction Lens aberrations Object movement Camera shake Can we remove blur computationally?
CS 89/189: Computational Photography, Fall 2015
21
After a slide by Frédo Durand
Lens diffraction
After a slide by Frédo Durand
http://luminous-landscape.com/ tutorials/understanding-series/u- diffraction.shtml
(heavily cropped) See also
http://www.cambridgeincolour.com/ tutorials/diffraction-photography.htm
Blur example: spherical aberration
Pixel value: weighted average of local color
CS 89/189: Computational Photography, Fall 2015
23 Sensor lens
color variation
After a slide by Frédo Durand
Remove optical artifacts
Calibrate lenses and remove blur e.g. DXO
CS 89/189: Computational Photography, Fall 2015
24
After a slide by Frédo Durand
CS 89/189: Computational Photography, Fall 2015
Removing camera shake
25
Original Naïve Sharpening
After a slide by Frédo Durand
Fergus et al’s algorithm
Blur as convolution
Replace each pixel by a linear combination of its neighbors.
The prescription for the linear combination is called the “convolution kernel”.
CS 89/189: Computational Photography, Fall 2015
27
local image data kernel modified image 10 5 3 4 5 1 1 1 7 0.5 1 0.5 7
After a slide by Frédo Durand
Linear shift-invariant filtering
Replace each pixel by a linear combination of its neighbors.
The prescription for the linear combination is called the “convolution kernel”.
CS 89/189: Computational Photography, Fall 2015
28
local image data kernel modified image 10 5 3 4 5 1 1 1 7 0.5 1 0.5 7
After a slide by Frédo Durand
Example of linear NON-shift invariant transformation?
e.g. neutral-density graduated filter (darken high y):
Formally, what does linear mean?
What does shift invariant mean?
CS 89/189: Computational Photography, Fall 2015
29
After a slide by Frédo Durand
CS 89/189: Computational Photography, Fall 2015
Questions?
30
Convolution algorithm
set output image to zero for all pixels (x,y) in output image for all (x’,y’) in kernel
(this assumes the kernel coordinates are centered)
CS 89/189: Computational Photography, Fall 2015
31
local image data kernel modified image 10 5 3 4 5 1 1 1 7 0.5 1 0.5 7
After a slide by Frédo Durand
Questions?
s for all pixels (x,y) in for all (x’,y’)
(this assumes the kernel coordinates are centered)
CS 89/189: Computational Photography, Fall 2015
32
local image data kernel modified image 10 5 3 4 5 1 1 1 7 0.5 1 0.5 7
After a slide by Frédo Durand
CS 89/189: Computational Photography, Fall 2015
Convolution (warm-up slide)
33
After a slide by Frédo Durand
1 pixel offset coefficient
⨂ =
CS 89/189: Computational Photography, Fall 2015
Convolution (warm-up slide)
34
After a slide by Frédo Durand
1 pixel offset coefficient filtered (no change)
= ⨂
CS 89/189: Computational Photography, Fall 2015
Convolution (warm-up slide)
35
After a slide by Frédo Durand
1 pixel offset coefficient filtered (no change)
= ⨂
δ f = f ⊗ δ f
CS 89/189: Computational Photography, Fall 2015
Convolution
36
After a slide by Frédo Durand
1 pixel offset coefficient
⨂ =
CS 89/189: Computational Photography, Fall 2015
Convolution
37
After a slide by Frédo Durand
1 pixel offset coefficient
⨂ =
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
CS 89/189: Computational Photography, Fall 2015
Convolution
38
After a slide by Frédo Durand
1/3 pixel offset coefficient
⨂ =
CS 89/189: Computational Photography, Fall 2015
Blurring
39
After a slide by Frédo Durand
1/3 pixel offset coefficient
⨂ =
blurred (applied in both dimensions)
CS 89/189: Computational Photography, Fall 2015
Blur examples
40
After a slide by Frédo Durand
1/3 pixel offset coefficient
⨂ =
8 8/3 filtered
impulse
CS 89/189: Computational Photography, Fall 2015
Blur examples
41
After a slide by Frédo Durand
1/3 pixel offset coefficient
⨂ =
8 8/3 filtered
impulse
1/3 pixel offset coefficient
⨂ =
8
edge
4 filtered 8 4
CS 89/189: Computational Photography, Fall 2015
Questions?
42
(I ⌦ g)(x) =
Z
x0 I(x0) g(x x0) dx0
More formally: Convolution
CS 89/189: Computational Photography, Fall 2015
44
g
After a slide by Frédo Durand
Questions?
CS 89/189: Computational Photography, Fall 2015
45
g
(I ⌦ g)(x) =
Z
x0 I(x0) g(x x0) dx0
After a slide by Frédo Durand
Convolution & probability
Convolution was first used by Laplace to study the probability of the sum of two random variables
CS 89/189: Computational Photography, Fall 2015
47
After a slide by Frédo Durand
Random variables
CS 89/189: Computational Photography, Fall 2015
48
X
30% 50% 20%
Y
1 50% 40% 10%
1
Probability?
How can X+Y=0?
After a slide by Frédo Durand
Sum of random variables
CS 89/189: Computational Photography, Fall 2015
49
X
30% 50% 20%
Y
1 50% 40% 10%
1
Probability?
How can X+Y=0?
P(X + Y = k) = ∑
k0
P(X = k0)P(Y = k k0)
After a slide by Frédo Durand
Questions?
CS 89/189: Computational Photography, Fall 2015
50
X
30% 50% 20%
Y
1 50% 40% 10%
1
Probability?
How can X+Y=0?
P(X + Y = k) = ∑
k0
P(X = k0)P(Y = k k0)
After a slide by Frédo Durand
Compare
CS 89/189: Computational Photography, Fall 2015
51
(I ⌦ g)(x) =
Z
x0 I(x0) g(x x0) dx0
Forward model: light goes from x to x+x’ Backward model: light at x comes from x-x’ P(X + Y = k) = ∑
k0
P(X = k0)P(Y = k k0)
After a slide by Frédo Durand
Image processing
I will often use the term “convolution” improperly and fail to flip the kernel
symmetric
CS 89/189: Computational Photography, Fall 2015
52
After a slide by Frédo Durand
CS 89/189: Computational Photography, Fall 2015
Questions?
53
http://graphics.stanford.edu/courses/cs178/applets/ convolution.html
CS 89/189: Computational Photography, Fall 2015 57 After a slide by Frédo Durand
Box filter
CS 89/189: Computational Photography, Fall 2015 58
⊗
After a slide by Frédo Durand
Nice and smooth: Gaussian
CS 89/189: Computational Photography, Fall 2015 59
⊗
After a slide by Frédo Durand
Gaussian formula
r is the distance to the center a is a normalization constant
after the fact
σ is the standard deviation and controls the width of the Gaussian
CS 89/189: Computational Photography, Fall 2015 60 http://en.wikipedia.org/wiki/Gaussian_function After a slide by Frédo Durand
ae− r2
2σ2
Gaussian formula
Gaussians have infinite support
but are often truncated
CS 89/189: Computational Photography, Fall 2015 61
ae− r2
2σ2
http://en.wikipedia.org/wiki/Gaussian_function After a slide by Frédo Durand
How can we sharpen?
Blurring was easy Sharpening is not as obvious
CS 89/189: Computational Photography, Fall 2015 63 After a slide by Frédo Durand
How can we sharpen?
Blurring was easy Sharpening is not as obvious Idea: amplify the stuff not in the blurry image
CS 89/189: Computational Photography, Fall 2015 64 After a slide by Frédo Durand
CS 89/189: Computational Photography, Fall 2015
Sharpening
65
= +k* high pass sharpened image input blurred high pass input
After a slide by Frédo Durand
Sharpening: kernel view
Recall f is the input f’ is a sharpened image g is a blurring kernel k is a scalar controlling the strength of sharpening
CS 89/189: Computational Photography, Fall 2015 66
f 0 = f + k ⇤ ( f f ⌦ g)
After a slide by Frédo Durand
Sharpening: kernel view
Recall Denote δ the Dirac kernel (pure impulse)
CS 89/189: Computational Photography, Fall 2015 67
f 0 = f + k ⇤ ( f f ⌦ g)
After a slide by Frédo Durand
f = f ⊗ δ
Sharpening: kernel view
Recall Sharpening is also a convolution
CS 89/189: Computational Photography, Fall 2015 68
f 0 = f + k ⇤ ( f f ⌦ g) f 0 = f ⌦ δ + k ⇤ ( f ⌦ δ f ⌦ g) f 0 = f ⌦ ((k + 1)δ g)
After a slide by Frédo Durand
Sharpening kernel
Note: many other sharpening kernels exist (just like we saw multiple blurring kernels) Amplify the difference between a pixel and its neighbors
CS 89/189: Computational Photography, Fall 2015 69
blue: positive red: negative
f 0 = f ⌦ ((k + 1)δ g)
After a slide by Frédo Durand
Alternate interpretation
CS 89/189: Computational Photography, Fall 2015 70
CS 89/189: Computational Photography, Fall 2015
Questions?
71
= +k*
Unsharp mask
Sharpening is often called “unsharp mask” because photographers used to sandwich a negative with a blurry positive film in order to sharpen
CS 89/189: Computational Photography, Fall 2015 73
http://www.tech-diy.com/UnsharpMasks.htm
After a slide by Frédo Durand
CS 89/189: Computational Photography, Fall 2015 74
http://www.tech-diy.com/images/unsharp2.jpg
After a slide by Frédo Durand
Unsharp mask
http://en.wikipedia.org/wiki/Unsharp_masking http://www.largeformatphotography.info/unsharp/ http://www.tech-diy.com/UnsharpMasks.htm http://www.cambridgeincolour.com/tutorials/unsharp-mask.htm
CS 89/189: Computational Photography, Fall 2015 75 After a slide by Frédo Durand
Problem with excess
Haloes around strong edges
CS 89/189: Computational Photography, Fall 2015 77 After a slide by Frédo Durand
CS 89/189: Computational Photography, Fall 2015
Oversharpening
78
coefficient
8 Sharpened (differences are accentuated; constant areas are left untouched). 11.2 1.7
8
After a slide by Frédo Durand
Bells and whistles
Apply mostly on luminance Old Clarity in Lightroom/Adobe Camera Raw
Only apply at edges
Sharpening chrominance as well
CS 89/189: Computational Photography, Fall 2015 79 After a slide by Frédo Durand
Lightroom demo
CS 89/189: Computational Photography, Fall 2015 80
Gradient: finite difference
horizontal gradient [[-1, 1]] vertical gradient: [[-1], [1]]
CS 89/189: Computational Photography, Fall 2015 82 After a slide by Frédo Durand
Gradient: finite difference
horizontal gradient [[-1, 1]] vertical gradient: [[-1], [1]]
CS 89/189: Computational Photography, Fall 2015 83
Gradient magnitude Vertical gradient (absolute value) Horizontal gradient (absolute value)
After a slide by Frédo Durand
e.g. Sobel [http://en.wikipedia.org/wiki/Sobel_operator]
Gradient
CS 89/189: Computational Photography, Fall 2015 84
Horizontal gradient Vertical gradient Magnitude
Gx =
−1 +1 −2 +2 −1 +1
⊗ A and Gy =
−1 −2 −1 +1 +2 +1
⊗ A
After a slide by Frédo Durand
Convolution cost?
set output image to zero for all pixels (x,y) in output image for all (x’,y’) in kernel
Cost?
CS 89/189: Computational Photography, Fall 2015
86
After a slide by Frédo Durand
Separability
Sometimes the 2D kernel can be decomposed into the convolution of a horizontal and a vertical filter. Example: box
same, but it’s the case here)
CS 89/189: Computational Photography, Fall 2015 88
=
⨂
After a slide by Frédo Durand
Separable box blur
CS 89/189: Computational Photography, Fall 2015
First blur horizontally using g(x) Then blur vertically using g(y)
89 After a slide by Frédo Durand
⊗ ⊗
Separable convolution cost?
for all pixels (x,y) in output image for all x’ in kernel
for all pixels (x,y) in output image for all y’ in kernel
Horizontal cost? Vertical cost? Total: Instead of:
CS 89/189: Computational Photography, Fall 2015 90
O(input.width * input.height * kernel.width) O(input.width * input.height * (kernel.height+kernel.width)) O(input.width * input.height * kernel.height) O(input.width * input.height * (kernel.height*kernel.width))
After a slide by Frédo Durand
Good news
Gaussians are separable too See Assignment 4!
CS 89/189: Computational Photography, Fall 2015 91
Box blur: Can we do even better?
Can we get even better asymptotic complexity? Very large kernel sizes?
CS 89/189: Computational Photography, Fall 2015 92
Box blur: Can we do even better?
Since 2D box is separable, let’s focus
The neighborhoods of pixel i and pixel i+1 are very similar In fact, they only differ by 2 pixels, so:
Asymptotically independent of kernel size, depends only on image size!
CS 89/189: Computational Photography, Fall 2015 93
pixel i neighborhood(i) neighborhood(i+1)
Box blur cost?
Naïve: Separable: Incremental:
CS 89/189: Computational Photography, Fall 2015 94
O(input.width * input.height * (kernel.height+kernel.width)) O(input.width * input.height * (kernel.height*kernel.width))
After a slide by Frédo Durand
O(input.width * input.height + (kernel.height+kernel.width)) O(input.width * input.height)
Repeated convolution
CS 89/189: Computational Photography, Fall 2015 95
Repeated convolution
CS 89/189: Computational Photography, Fall 2015 96
Repeated convolution
CS 89/189: Computational Photography, Fall 2015 97
Repeated convolution
CS 89/189: Computational Photography, Fall 2015 98
Repeated convolution
Convolution of two box kernels yields a tent kernel
CS 89/189: Computational Photography, Fall 2015 99
Repeated convolution
Yet another convolution with a box yields piecewise quadratic
CS 89/189: Computational Photography, Fall 2015 100
Repeated convolution
The pattern continues
cubic, and so on.
Each time we make the kernel smoother Taking this to the limit will yield a Gaussian
CS 89/189: Computational Photography, Fall 2015 101
delta 1D box box box ⨂ box box ⨂ box ⨂ box box ⨂ box ⨂ box ⨂ box box ⨂ box ⨂ box ⨂ box ⨂ box
Photoshops’ Gaussian not a true Gaussian
Gaussian blur as multi-box blur
Can approximate Gaussian blur with several box blurs Asymptotically independent of kernel size! Assignment 4 extra credit
CS 89/189: Computational Photography, Fall 2015 103
Best input to debug convolution
Impulse
CS 89/189: Computational Photography, Fall 2015 105 After a slide by Frédo Durand
Centering the kernel
Our images are defined with 0,0 in the upper left corner Kernels are usually assumed to have origin at the center
CS 89/189: Computational Photography, Fall 2015 106 After a slide by Frédo Durand
Normalization
As a rule of thumb, you want kernels to be normalized when you want the output to preserve the overall brightness of the image.
CS 89/189: Computational Photography, Fall 2015 107 After a slide by Frédo Durand
Denoising from 1 image
We can’t take average over multiple images
CS 89/189: Computational Photography, Fall 2015 109
Noisy input
After a slide by Frédo Durand
Denoising from 1 image
We can’t take average over multiple images Idea 1: take a spatial average
as their neighbor
pass
Here: Gaussian blur
CS 89/189: Computational Photography, Fall 2015 110
Noisy input
After a slide by Frédo Durand
Gaussian blur
CS 89/189: Computational Photography, Fall 2015 111
After Gaussian blur
After a slide by Frédo Durand
Gaussian blur
Noise is mostly gone But image is blurry
CS 89/189: Computational Photography, Fall 2015 112
After Gaussian blur
After a slide by Frédo Durand
Gaussian blur
Noise is mostly gone But image is blurry
Problem: not all neighbors have the same color Bilateral filter idea: only consider neighbors that have similar values
CS 89/189: Computational Photography, Fall 2015 114
After Gaussian blur
After a slide by Frédo Durand
Bilateral filter
Tomasi and Manduci 1998 http://www.cse.ucsc.edu/~manduchi/Papers/ ICCV98.pdf Developed for denoising Related to
chan01digital.html
Full survey: http://people.csail.mit.edu/sparis/publi/2009/fntcgv/ Paris_09_Bilateral_filtering.pdf
CS 89/189: Computational Photography, Fall 2015 115 After a slide by Frédo Durand
Bilateral filtering
Images are often piecewise constant with noise added
measurement of the same value
Simply blurring doesn’t work
We should blur only within each constant- colored region
CS 89/189: Computational Photography, Fall 2015 116 [Tomasi and Manduchi 1998]
⨂ =
After a slide by Marc Levoy
Bilateral filtering
If pixels are similar in intensity, they are probably from the same region of the scene Perform a “convolution” where the weight applied to nearby pixels falls off with:
blurred
being blurred
i.e. blur in domain and range dimensions!
CS 89/189: Computational Photography, Fall 2015 117 [Tomasi and Manduchi 1998]
⨂ =
After a slide by Marc Levoy
Here, input is a step function + noise
Start with Gaussian filtering
input
J
=
f
⊗
I
118 After a slide by Frédo Durand
Weight of ξ depends on distance to x
Gaussian filter as weighted average
input
ξ
∑
f (x,ξ)
I(ξ)
ξ x x ξ
J(x) =
119 After a slide by Frédo Durand
The problem of edges
input
x ξ Ι(ξ) I(x)
ξ
∑
f (x,ξ)
I(ξ)
J(x) =
Here, Ι(ξ) “pollutes” our estimate J(x) ¡ It is too different
120 After a slide by Frédo Durand
Penalty g on the intensity difference
Principle of Bilateral filtering
input
J(x) =
1 k(x)
ξ
∑ f (x,ξ)
g(I(ξ) − I(x)) I(ξ)
x Ι(ξ) I(x)
121 After a slide by Frédo Durand
Bilateral filtering
input
J(x) =
1 k(x)
ξ
∑ f (x,ξ)g(I(ξ) − I(x)) I(ξ)
x ξ x
Spatial Gaussian f
122
[Tomasi and Manduchi 1998]
After a slide by Frédo Durand
Spatial Gaussian f Gaussian g on the intensity difference
Bilateral filtering
input
J(x) =
1 k(x)
ξ
∑ f (x,ξ) g(I(ξ) − I(x))I(ξ)
x Ι(ξ) I(x)
123
[Tomasi and Manduchi 1998]
After a slide by Frédo Durand
k(x)=
Normalization factor
input
J(x) =
1 k(x)
ξ
∑ f (x,ξ)
g(I(ξ) − I(x)) I(ξ)
ξ
∑ f (x,ξ) g(I(ξ) − I(x))
124
[Tomasi and Manduchi 1998]
After a slide by Frédo Durand
Bilateral filtering is non-linear
input
The weights are different for each output pixel
125
[Tomasi and Manduchi 1998]
After a slide by Frédo Durand
Bilateral filter
Noisy input After bilateral filter
After a slide by Frédo Durand
Can we do better?
Noisy input After bilateral filter
chroma noise
After a slide by Frédo Durand
Chroma noise
Our visual system has different spatial frequency response to chrominance vs. luminance Perform Bilateral filtering in YUV Bigger spatial filter in U & V
CS 89/189: Computational Photography, Fall 2015 128 After a slide by Frédo Durand
Normal RGB Bilateral filter
Noisy input After bilateral filter
After a slide by Frédo Durand
YUV Bilateral filter
Noisy input After YUV bilateral filter
After a slide by Frédo Durand
Comparison
Noisy input Bilateral filter YUV bilateral filter
After a slide by Frédo Durand 131
Bilateral filtering
Also used to remove skin blemishes in portraits
(although box spatial kernel instead of Gaussian)
Useful for lots of other things
dynamic-range imaging
CS 89/189: Computational Photography, Fall 2015 132 After a slide by Frédo Durand
Photoshop surface blur
Note the radius and threshold controls
CS 89/189: Computational Photography, Fall 2015 133 After a slide by Frédo Durand
Assignment 4
Convolution Separable Unsharp mask Gradient Denoising YUV denoising
CS 89/189: Computational Photography, Fall 2015 135
Denoising
Bayesian coring in the wavelet domain
Big heuristics
NL means
Statistics of natural images
CS 89/189: Computational Photography, Fall 2015 137
References
http://www.cambridgeincolour.com/tutorials/image-noise.htm http://www.cambridgeincolour.com/tutorials/image-noise-2.htm http://www.clarkvision.com/imagedetail/does.pixel.size.matter2/ http://www.clarkvision.com/articles/digital.sensor.performance.summary/ http://books.google.com/books?id=OYFYt5C4N94C&pg=PA405&dq=binomial+film+grain+noise#v=onepage&q=binomial %20film%20grain%20noise&f=false http://en.wikipedia.org/wiki/Image_noise http://www.picturecode.com/noise.htm http://www.instructables.com/id/Avoiding-Camera-Noise-Signatures/ http://www.photoxels.com/tutorial_noise.html http://people.csail.mit.edu/hasinoff/hdrnoise/hasinoff-sensornoise-tutorial-iccp10.pptx http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/ http://www.imatest.com/docs/noise/ http://www.cambridgeincolour.com/tutorials/image-averaging-noise.htm http://www.petapixel.com/2012/02/21/a-simple-explanation-of-how-iso-works-in-digital-photography/
CS 89/189: Computational Photography, Fall 2015 138
Slide credits
Frédo Durand
CS 89/189: Computational Photography, Fall 2015 139