[PPT] - Computational Photography Si Lu Spring 2018 PowerPoint Presentation

SLIDE 1

Computational Photography

Si Lu

Spring 2018

http://web.cecs.pdx.edu/~lusi/CS510/CS510_Computati

nal_Photography.htm

04/10/2018

SLIDE 2

Last Time

Digital Camera

n History of Camera n Controlling Camera

Photography Concepts

SLIDE 3

Today

Filters and its applications

naïve denoising Gaussian blur better denoising edge-preserving filter noisy image

Slide credit: Sylvain Paris and Frédo Durand

SLIDE 4

The raster image (pixel matrix)

Slide credit: D. Hoiem

SLIDE 5

The raster image (pixel matrix)

0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99 0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91 0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92 0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95 0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85 0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33 0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74 0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93 0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99 0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97 0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93

Slide credit: D. Hoiem

SLIDE 6

Perception of Intensity

Slide credit: C. Dyer

SLIDE 7

Perception of Intensity

Slide credit: C. Dyer

SLIDE 8

Color Image

R G B

Slide credit: D. Hoiem

SLIDE 9

Image filtering: compute function of local

neighborhood at each pixel position

One type of “Local operator,” “Neighborhood operator,”

“Window operator”

Useful for:

n Enhancing images

Noise reduction, smooth, resize, increase contrast, etc.

n Extracting information from images

Texture, edges, distinctive points, etc.

n Detecting patterns

Template matching, e.g., eye template

Source: D. Hoiem

Image Filtering

Slide credit: C. Dyer

SLIDE 10

Source: http://lullaby.homepage.dk/diy-camera/bokeh.html

Bokeh: Blur in out-of-focus regions of image Camera shake

*

=

Source: Fergus, et al. “Removing Camera Shake from a Single Photograph”, SIGGRAPH 2006

Blurring in the Real World

Slide credit: C. Dyer

SLIDE 11

Image Correlation Filtering

Select a filter g

n g is called a filter, mask, kernel, or template

Center filter g at each pixel in image f
Multiply weights by corresponding pixels
Set resulting value in output image h
Linear filtering is sum of dot product at each

pixel position

Filtering operation called cross-correlation,

and denoted h = f  g

Slide credit: C. Dyer

SLIDE 12

1 1 1 1 1 1 1 1 1

] , [ g  

Example: Box Filter

Slide credit: David Lowe

SLIDE 13

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

Credit: S. Seitz

] , [ ] , [ ] , [

,

l n k m f l k g n m h

l k

   

[.,.] h [.,.] f

l n k m f l k g n m h

l k

   

Credit: S. Seitz

SLIDE 19

10 20 30 30 50 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

[.,.] h [.,.] f

Image Filtering

1 1 1 1 1 1 1 1 1

] , [ g  

?

] , [ ] , [ ] , [

,

l n k m f l k g n m h

l k

   

Credit: S. Seitz

SLIDE 20

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 10 20 30 30 30 20 10 20 40 60 60 60 40 20 30 60 90 90 90 60 30 30 50 80 80 90 60 30 30 50 80 80 90 60 30 20 30 50 50 60 40 20 10 20 30 30 30 30 20 10 10 10 10

[.,.] h [.,.] f

Image Filtering

1 1 1 1 1 1 1 1 1

] , [ g  

] , [ ] , [ ] , [

,

l n k m f l k g n m h

l k

   

Credit: S. Seitz

SLIDE 21

What does it do?

Replaces each pixel with

an average of its neighborhood

Achieves smoothing

effect (i.e., removes sharp features)

Weaknesses:
Blocky results
Axis-aligned streaks

1 1 1 1 1 1 1 1 1

Slide credit: David Lowe

] , [ g  

Box Filter

SLIDE 22

Smoothing with Box Filter

Slide credit: C. Dyer

SLIDE 23

Properties of Smoothing Filters

Smoothing

n Values all positive n Sum to 1  constant regions same as input n Amount of smoothing proportional to mask size n Removes “high-frequency” components n “low-pass” filter

Slide credit: C. Dyer

SLIDE 24

Weight contributions of neighboring pixels by nearness
Constant factor at front makes volume sum to 1
Convolve each row of image with 1D kernel to produce new image;

then convolve each column of new image with same 1D kernel to yield output image

0.003 0.013 0.022 0.013 0.003 0.013 0.059 0.097 0.059 0.013 0.022 0.097 0.159 0.097 0.022 0.013 0.059 0.097 0.059 0.013 0.003 0.013 0.022 0.013 0.003

5 x 5,  = 1

Slide credit: Christopher Rasmussen

Gaussian Filtering

SLIDE 25

Smoothing with a box

actually doesn’t compare at all well with a defocused lens

Most obvious difference is

that a single point of light viewed in a defocused lens looks like a fuzzy blob; but the averaging process would give a little square

Gaussian is isotropic (i.e.,

rotationally symmetric)

A Gaussian gives a good

model of a fuzzy blob

It closely models many

physical processes (the sum

f many small effects)

Slide by D.A. Forsyth

Smoothing with a Gaussian

SLIDE 26

What does Blurring take away?

riginal

Slide credit: C. Dyer

SLIDE 27

What does Blurring take away?

smoothed (5x5 Gaussian)

Slide credit: C. Dyer

SLIDE 28

Smoothing with Gaussian Filter

SLIDE 29

Smoothing with Box Filter

SLIDE 30

input

Slide by S. Paris

SLIDE 31

box average

Slide by S. Paris

SLIDE 32

Gaussian blur

Slide by S. Paris

SLIDE 33

What parameters matter here?
Standard deviation, , of Gaussian: determines extent
f smoothing

σ = 2 with 30 x 30 kernel σ = 5 with 30 x 30 kernel

Source: D. Hoiem

Gaussian Filters

SLIDE 34

Slide credit: C. Dyer

SLIDE 35

for sigma=1:3:10 h = fspecial('gaussian‘, hsize, sigma);

ut = imfilter(im, h);

imshow(out); pause; end

…

Parameter σ is the “scale” / “width” / “spread” of the Gaussian kernel, and controls the amount of smoothing

Smoothing with a Gaussian

Slide credit: C. Dyer

SLIDE 36

Gaussian filters

= 30 pixels = 1 pixel = 5 pixels = 10 pixels

Slide credit: C. Dyer

SLIDE 37

What parameters matter here?
Size of kernel or mask

σ = 5 with 10 x 10 kernel σ = 5 with 30 x 30 kernel

Gaussian Filters

Slide credit: C. Dyer

SLIDE 38

Gaussian function has infinite “support” but need a finite-size kernel
Values at edges should be near 0
98.8% of area under Gaussian in mask of size 5σ x 5σ
In practice, use mask of size 2k+1 x 2k+1 where k  3
Normalize output by dividing by sum of all weights

How big should the filter be?

Slide credit: C. Dyer

SLIDE 39

Original 1 1 1 1 1 1 1 1 1 2

?

(Note that filter sums to 1)

Source: D. Lowe

Sharpening Filters

SLIDE 40

Original 1 1 1 1 1 1 1 1 1 1

?

(Note that filter sums to 1)

Source: D. Lowe

Sharpening Filters

1

+

SLIDE 41

Original 1 1 1 1 1 1 1 1 1 2

Sharpening filter
Sharpen an out of focus image by

subtracting a multiple of a blurred version

Source: D. Lowe

Sharpening Filters

SLIDE 42

Source: D. Lowe

Sharpening

SLIDE 43

h = f + k(f * g) where k is a small positive constant

and g =

Called unsharp masking in photography

1 1

4

1 1 called a Laplacian mask

Sharpening by Unsharp Masking

SLIDE 44

Sharpening using Unsharp Mask Filter

Original Filtered result

Slide credit: C. Dyer

SLIDE 45

1

1

2

2

1

1 Vertical Edge (absolute value)

Sobel

Application: Edge Detection

Slide credit: C. Dyer

SLIDE 46

1
2
1

1 2 1 Horizontal Edge (absolute value)

Sobel

Application: Edge Detection

Slide credit: C. Dyer

SLIDE 47

Application: Hybrid Images

Gaussian Filter Laplacian Filter

A. Oliva, A. Torralba, P.G. Schyns, Hybrid Images, SIGGRAPH 2006

Gaussian unit impulse Laplacian of Gaussian I1 I2 G1 (1-G2) I1  G1

SLIDE 48

https://www.youtube.com/watch?v=gfvMU36fgKw

SLIDE 49

Application: XDoG Filters

Gaussian filtering results Winnemoller, H., XDoG: advanced image stylization with eXtended Difference-of-Gaussians NPAR 2011 Input XDoG Input XDoG

SLIDE 50

Application: Painterly Filters

Many methods have been

proposed to make a photo look like a painting

Today we look at one:

Painterly-Rendering with Brushes of Multiple Sizes

Basic ideas:

n Build painting one layer at a time, from biggest to smallest brushes n At each layer, add detail missing from previous layer

A. Hertzmann, Painterly rendering with curved brush strokes of multiple sizes, SIGGRAPH 1998.

Slide credit: S. Chenney

SLIDE 51

Algorithm 1

function paint(sourceImage,R1 ... Rn) // take source and several brush sizes { canvas := a new constant color image // paint the canvas with decreasing sized brushes for each brush radius Ri, from largest to smallest do { // Apply Gaussian smoothing with a filter of size const * radius // Brush is intended to catch features at this scale referenceImage = sourceImage * G(fs Ri) // Paint a layer paintLayer(canvas, referenceImage, Ri) } return canvas }

Slide credit: S. Chenney

SLIDE 52

Algorithm 2

procedure paintLayer(canvas,referenceImage, R) // Add a layer of strokes { S := a new set of strokes, initially empty D := difference(canvas,referenceImage) // euclidean distance at every pixel for x=0 to imageWidth stepsize grid do // step in size that depends on brush radius for y=0 to imageHeight stepsize grid do { // sum the error near (x,y) M := the region (x-grid/2..x+grid/2, y-grid/2..y+grid/2) areaError := sum(Di,j for i,j in M) / grid2 if (areaError > T) then { // find the largest error point (x1,y1) := max Di,j in M s :=makeStroke(R,x1,y1,referenceImage) add s to S } } paint all strokes in S on the canvas, in random order }

Slide credit: S. Chenney

SLIDE 53

Results

Original Biggest brush Medium brush added Finest brush added

Slide credit: S. Chenney

SLIDE 54

More filters (3 mins Break)

Slide credit: S. Chenney

SLIDE 55

Filter Re-cap

naïve denoising Gaussian blur better denoising edge-preserving filter noisy image

Slide credit: Sylvain Paris and Frédo Durand

SLIDE 56

Median Filter

Replace pixel by the

median value of its neighbors

No new pixel values

introduced

Removes spikes: good

for impulse, salt & pepper noise

Slide credit: C. Dyer

SLIDE 57

Salt and pepper noise Median filtered Plots of a row of the image Matlab: output im = medfilt2(im, [h w])

Median Filter

Slide credit: M. Hebert, C. Dyer

SLIDE 58

Median Filter

Median filter is edge preserving

Slide credit: C. Dyer

SLIDE 59

Slide credit: C. Dyer

SLIDE 60

19x19 median filter

input

utput

images by J. Plush Slide credit: C. Dyer

SLIDE 61

Bilateral filter

Tomasi and Manduci 1998

http://www.cse.ucsc.edu/~manduchi/Papers/I CCV98.pdf

Related to

n SUSAN filter [Smith and Brady 95] http://citeseer.ist.psu.edu/smith95susan.html n Digital-TV [Chan, Osher and Chen 2001] http://citeseer.ist.psu.edu/chan01digital.html n sigma filter http://www.geogr.ku.dk/CHIPS/Manual/f187.htm

Slide credit: F . Durand

SLIDE 62

Start with Gaussian filtering

Here, input is a step function + noise
utput

input

 J

 

 f

 

 I

Slide credit: F . Durand

SLIDE 63

Gaussian filter as weighted average

Weight of x depends on distance to x
utput

input

x x x x

Slide credit: F . Durand



x



 f (x,x)

 I(x)

 J(x)

 

SLIDE 64

The problem of edges

Here, “pollutes” our estimate J(x)
It is too different

x

Slide credit: F . Durand



x



 f (x,x)

 I(x)

 J(x)

 

utput

input

x x x x

SLIDE 65

Principle of Bilateral filtering

[Tomasi and Manduchi 1998]

Penalty g on the intensity difference
utput

input

 J(x)

 

 1 k(x)



x



 f (x,x)

 g(I(x)  I(x))

 I(x)

x x I(x)

Slide credit: F . Durand

SLIDE 66

Bilateral filtering

Spatial Gaussian f
utput

input

 J(x)

 

 1 k(x)



x



 f (x,x)

 g(I(x)  I(x))

 I(x)

x x x

Slide credit: F . Durand

[Tomasi and Manduchi 1998]

SLIDE 67

Bilateral filtering

Spatial Gaussian f
Gaussian g on the intensity difference
utput

input

 J(x)

 

 1 k(x)



x



 f (x,x)

 g(I(x)  I(x))  I(x)

x x I(x)

Slide credit: F . Durand

[Tomasi and Manduchi 1998]

SLIDE 68

Normalization factor

k(x)=
utput

input

 J(x)

 

 1 k(x)



x



 f (x,x)

 g(I(x)  I(x))

 I(x)



x



 f (x,x)

 g(I(x)  I(x))

[Tomasi and Manduchi 1998]

Slide credit: F . Durand

SLIDE 69

Blur from averaging across edges

* * *

input

utput

Same Gaussian kernel everywhere.

Slide credit: P . Sylvain

SLIDE 70

Bilateral filter: no averaging across edges

* * *

input

utput

The kernel shape depends on the image content.

Slide credit: P . Sylvain

SLIDE 71

s = 2 s = 6 s = 18 r = 0.1 r = 0.25 r = 

(Gaussian blur)

input

Slide credit: P . Sylvain

Parameter for spatial distance Gaussian f Parameter for intensity difference Gaussian g

SLIDE 72

s = 2 s = 6 s = 18 r = 0.1 r = 0.25 r = 

(Gaussian blur)

input

Slide credit: P . Sylvain

Parameter for spatial distance Gaussian f Parameter for intensity difference Gaussian g

SLIDE 73

Result

Input Output Tomasi and Manduchi 1998

SLIDE 74

Other view

The bilateral filter uses the 3D distance

Slide credit: F . Durand

SLIDE 75

Speed

Direct bilateral filtering is slow (minutes)
Accelerations exist:

n Subsampling in space & range

Durand & Dorsey 2002
Paris & Durand 2006

n Limit to box kernel & intelligent maintenance of histogram

Weiss 2006

Slide credit: F . Durand

SLIDE 76

Local filters

Compute a new value at each pixel using its

neighboring pixels

Box filter
Gaussian filter
Median filter
Bilateral filter

SLIDE 77

Non-local means filter

Compute a new value at each pixel from the

whole image

Buades, A., Coll, B., Morel, J.-M. A non-local algorithm for image denoising. CVPR 2005

weight of pixel j value at pixel j final value at pixel i

SLIDE 78

Weight

: patch centered at pixel i : patch centered at pixel j

Similar pixel neighborhoods give a large weight

Reprint from Buades et al. 2005

SLIDE 79

Input Gaussian Anisotropic Total variation Neighborhood NL-means

Reprint from Buades et al. 2005

SLIDE 80

Non-local means filter

High-quality
Slow

n Fast non-local means algorithms available

BM3D:

n http://www.cs.tut.fi/~foi/GCF-BM3D/

SLIDE 81

BM3D

Image denoising by sparse 3-D transform-

domain collaborative filtering (TIP 2007)

n http://www.cs.tut.fi/~foi/GCF-BM3D/

SLIDE 82

BM3D

Patch-based
Non-local method
Two-stage denoising
Collaborative denoising
Denoise in frequency domain

SLIDE 83

BM3D

SLIDE 84

BM3D

Results

An Analysis and Implementation of the BM3D Image Denoising Method

SLIDE 85

BM3D

An Analysis and Implementation of the BM3D Image Denoising Method

SLIDE 86

Video de-noise

We know how to de-noise an image
How about video?
E. P

. Bennett and L. McMillan. Video Enhancement using Per-pixel Virtual Exposures SIGGRAPH 2005

SLIDE 87

Gaussian filter in video cube

Blurring artifacts

n Not edge-preserving n Motion blur

SLIDE 88

Bilateral filter in video cube

Cannot remove shot noise

Reprint from [Bennett and McMillan 2005]

SLIDE 89

ASTA Filter [Bennett and McMillan ‘05]

Build upon bilateral filter
Find similar pixels in a video cube for filtering

n Patch-based similarity measurement

Adaptive Spatial-temporal Accumulation Filter

n Prefer temporal neighbors

SLIDE 90

Patch-based similarity measurement

frame pt frame st

SLIDE 91

Similarity measure

Reprint from [Bennett and McMillan 2005]

SLIDE 92

Adaptive Filtering

Reprint from [Bennett and McMillan 2005]

SLIDE 93

Results (filtering + tone mapping)

Input Naïve method ASTA

Reprint from [Bennett and McMillan 2005]

SLIDE 94

Next Time

Color
Lighting