Loop over all pixels in image pixel F[i,j] a b c 1/9 1/9 1/9 - - PDF document

Computational Photography

CS 4475/6475 Maria Hybinette

1 Maria Hybinette

Filtering: Smoothing

• Point-process and Neighboring Pixels

Computations on an Image for Image Smoothing

Maria Hybinette 2

Objectives

• Smooth an image while considering a

neighborhood n of pixels

– Average filtering (use the average of n) – Median filtering (use the median of n)

• special non-linear filtering and smoothing

approach

Maria Hybinette 3

From Pixel/Point Operations to Groups of Pixels

• Previously we used pixel to pixel arithmetic (e.g., add(), sub()))
• How to Smooth a Signal?
• Approach: Use a group of pixels : Neighboring Values of a

neighborhood.

1. Moving Average [1 1 1 1 1] X 1/5 2. Weighted Moving Average [1 4 6 4 1] X 1/16

Maria Hybinette 4

Example: 3x3 Kernel

Maria Hybinette 5

Example: 3x3 Kernel

• Smoothing Process over an Image using Averages
• 1*90/9=10, 2*90/9=20, ….

Maria Hybinette 6

Maria Hybinette 7

Results in Shades of Gray

• What about the edges?

Maria Hybinette 8

Edges

• Expand the image we are smoothing:

– Wrap around pixels. (3x3 filter needs one more pixel

• n each border, 5x5 needs 2)

– Copy edges

Maria Hybinette 9

Mathematical Notation, Observations, and Computing the Result Matrix

• A Small image ‘h’ is “rubbed” over

(larger) image ‘F’ that we are smoothing

• Kernel: h

– Example: 3x3 area around each

• riginal pixel used

– Neighborhood size – k=1, window size = 3. – k >1,

• Window size is 2k+1

– k=1, 2(1)+1 = 3 – Our kernel is 3x3 – Recall : A pixel is referenced by (i, j)

• Want Result Matrix ‘G’

– F à G using h. – G = h (operation) F

Maria Hybinette 10

Computing the Values of Result Matrix

• Let’s refer to

– F[i,j] as input and – G[i,j] as output – h[i,j] as the kernel

• Example: pixel (3,3) (expanded edges above)

– G[3, 3] = a*A + b*B + c*C + d*D +e*E + f*F + h*H + i*I – If all weights (a through i) are 1: – G[3, 3] = 1/9 (A + B + C + D + E + F + G + H + I )

Maria Hybinette 11 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9

Smoothing when a-i are all 1s.

• “Standardizes” the weight to

each pixel . (k=1, then 1/9)

• Loop over all pixels in

neighborhood around particular image pixel F[i,j]

– starting from upper right corner, -k, -k, if k=1, then 1 left, from center and up 1.

Maria Hybinette 12

G[3, 3] = 1/9 (A + B + C + D + E + F + G + H + I )

G[i, j]= 1 (2k +1)2 F[i+u, j + v]

v=−k k

∑

u=−k k

∑

20 20 10 20 10 20 10 10 13 30 30 20 90 90 90 90 90 20 20 90 90 90 90 20 10 90 90 90 90 90 10 10 90 90 90 90 90 10 10 90 90 90 90 90 10 20 20 20 20 10 20 10 20 10 10 13 20 20 10 20 10 20 10 10 13 30 30 20 A B C 90 90 20 20 D E F 90 90 20 10 G H I 90 90 10 10 90 90 90 90 90 10 10 90 90 90 90 90 10 20 20 20 20 10 20 10 20 10 10 13

Loop over all pixels in

a b c d e f g h i 1 (2k +1)2

F[i+u, j + v]

v=−k k

∑

u=−k k

∑

Generalize the uniform weight

• More Generally,
• Formula is Referred to Cross

Correlation more on this later.

Maria Hybinette 13

G[3, 3] = 1/9 (A + B + C + D + E + F + G + H + I )

G[i, j]= h[u,v]*F[i+u, j + v]

v=−k k

∑

u=−k k

∑

20 20 10 20 10 20 10 10 13 30 30 20 90 90 90 90 90 20 20 90 90 90 90 20 10 90 90 90 90 90 10 10 90 90 90 90 90 10 10 90 90 90 90 90 10 20 20 20 20 10 20 10 20 10 10 13 20 20 10 20 10 20 10 10 13 30 30 20 A B C 90 90 20 20 D E F 90 90 20 10 G H I 90 90 10 10 90 90 90 90 90 10 10 90 90 90 90 90 10 20 20 20 20 10 20 10 20 10 10 13

Loop over all pixels in

a b c d e f g h i G[i, j]= a* A+b*B+c*C +d *D+e*E + f *F +h*H +i*I

G[i, j]= 1 (2k +1)2 F[i+u, j + v]

v=−k k

∑

u=−k k

∑

Example: Box Filter for Smoothing

• Uniform Kernel (all 1s).

– Box Area above is 21x21 the larger the kernel the ‘smoother’ the result.

• The larger the box the smoother it gets.

Maria Hybinette 14

h7ps://en.wikipedia.org/wiki/Box_blur

Example: Median Filter for smoothing

• Median Filter: Nonlinear filter

– Reduces noise (the outliers are ‘basically’ noise and removed). – Preserves edges

Maria Hybinette 15

20 20 1 20 10 20 10 13 13 20 0 0 20 20 20 0 90 90 90 90 90 20 20 20 0 90 0 90 90 90 20 20 10 0 90 90 90 90 90 10 10 10 0 90 90 90 90 90 10 10 10 0 90 90 90 90 90 10 10 20 0 0 20 20 20 20 10 20 10 20 10 13 13 F 20 20 1 20 10 20 10 13 13 20 0 0 20 20 20 0 90 90 90 90 90 20 20 20 0 90 0 90 90 90 20 20 10 0 90 90 90 90 90 10 10 10 0 90 90 90 90 90 10 10 10 0 90 90 90 90 90 10 10 20 0 0 20 20 20 20 10 20 10 20 10 13 13

20 20 1 20 0 20 0 90

median( ) median( )

0 0 1 20 20 20 20 90

average( )

0 0 0 1 20 20 20 20 90

Median = 20 (removes outliers) Average=171/9 = 19 h7ps://en.wikipedia.org/wiki/Median_filter

Comparison

• Median Filter Preserves Edges (11x11 median filter)
• Sharper Edges

Maria Hybinette 16

Median Filter vs. Average Filter

• Median (11x11) on left, Average Filter on Right

Maria Hybinette 17

Example: Median Filter for Noise Removal

• Salt and Pepper Noise (3x3 median filter)

Maria Hybinette 18

Noise Removal Median vs. Average

• 3x3 Median Filter on left, 3x3 Average Filter on Right

Maria Hybinette 19

• Image Smoothing
• Applying a kernel to smooth an

image (cross correlation)

• Applying a function (median) to

smooth an image.

Maria Hybinette 20

Cross Correlation

• In signal processing, cross-correlation is a measure
• f similarity of two waveforms as a function of a

time-lag applied to one of them.

– Can find pattern in images with this approach, the kernel is the pattern we are looking for.

• Also known as a sliding dot product or sliding

inner-product.

Maria Hybinette 21

G[i, j]= h[u,v]*F[i+u, j + v]

v=−k k

∑

u=−k k

∑

Cross Correlation

• Filtering an image:

– Replace each pixel with a linear combination of its neighbors

• Filter “kernel” or “mask”

– h[u,v] is the prescription for weights in the linear combination

Maria Hybinette 22

G[i, j]= h[u,v]*F[i+u, j + v]

v=−k k

∑

u=−k k

∑

G = h ⊗ F

Revisit: Box Filter

• Box/Average Filter

– Size 21 x 21 – Values are uniform

Maria Hybinette 23

Filter Original Result

Example: Gaussian Filter

• Normal or Gaussian Distribution of kernel

– Give center values more importance.

Maria Hybinette 24

h7ps://en.wikipedia.org/wiki/Gaussian_blur

Average Filter vs. Gaussian Filter

• Average is more blurry
• Gaussian filter Gives more ‘detail’ to center pixel,

– Preserves edges more

Maria Hybinette 25

Varying the ‘Spread’ or Standard Deviation (σ)

• As sigma gets larger more blurry. Extends how far it

smoothens the images.

Maria Hybinette 26

Correlation to Convolution

• Example : Take Cross Correlation of Impulse

Kernel.

Maria Hybinette 27

Result of Impulse Function

• Notice : Reflected Result (switching rows and

column of the kernel into result image)

Maria Hybinette 28

Another way to look at result

• Reverse response

Maria Hybinette 29

Convolution Method

• Convolution is a mathematical operation on two

functions F and h,

• Produces a third function that is typically viewed as

a modified version

• Gives the area of overlap between the two

functions

• In a form of the amount that one of the original

functions is translated.

Maria Hybinette 30

Convolution Method (*)

• Flip filter in both dimensions:

– Bottom to top – Right to left

• Then apply cross-correlation ( (X) )
• Commutative : F * G = G * F
• Associative: (F*G) * H = F * (G*H)
• Identity (Unit Impulse E): F * E = F

Maria Hybinette 31

G[i, j]= h[u,v]*F[i −u, j − v]

v=−k k

∑

u=−k k

∑

a b c d e f g h i g h i d e f a b c i h g f e d c b a

Example: Linear Filter

• Nothing
• Shift by 1
• Average Box Blur

Maria Hybinette 32 1 1 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9

Example: Combing Linear Filters

• Magnify importance of middle pixel subtract a

smooth version of the image

– Sharper image.

Maria Hybinette 33

Linear Filters

• riginal,

64x64

• 0.11
• 0.11
• 0.11
• 0.11

1.9

• 0.11
• 0.11
• 0.11
• 0.11

1 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9

x 2 - =

Sources & Inspiration

Contributors of Course Material:

• Irfan Essa (main) & Frank Dellaert (Georgia

Tech) – Also early adopters

• Marc Levoy (Stanford)– taught computational

photography since 2002: – A leader in the field : Frankecamera

• Frédo Durand (MIT)
• Jack Tumblin (Northwestern)
• Wikipedia
• http://www.all-art.org/

history658_photography1.html

• “Photography”, London, Stone, Upton

Maria Hybinette 34