Loop over all pixels in image pixel F[i,j] a b c 1/9 1/9 1/9 - PDF document

Filtering: Smoothing • Point-process and Neighboring Pixels Computations on an Image for Image Smoothing Computational Photography CS 4475/6475 Maria Hybinette 1 Maria Hybinette 2 Maria Hybinette From Pixel/Point Operations to Objectives Groups of Pixels • Smooth an image while considering a neighborhood n of pixels – Average filtering (use the average of n) – Median filtering (use the median of n) • special non-linear filtering and smoothing • Previously we used pixel to pixel arithmetic (e.g., add(), sub())) approach • How to Smooth a Signal? • Approach : Use a group of pixels : Neighboring Values of a neighborhood. 1. Moving Average [1 1 1 1 1] X 1/5 2. Weighted Moving Average [1 4 6 4 1] X 1/16 3 Maria Hybinette 4 Maria Hybinette Example: 3x3 Kernel Example: 3x3 Kernel • Smoothing Process over an Image using Averages • 1*90/9=10, 2*90/9=20, …. 5 Maria Hybinette 6 Maria Hybinette

Results in Shades of Gray • What about the edges? 7 Maria Hybinette 8 Maria Hybinette Mathematical Notation, Observations, Edges and Computing the Result Matrix • A Small image ‘ h ’ is “rubbed” over (larger) image ‘ F ’ that we are smoothing • Kernel : h – Example: 3x3 area around each original pixel used – Neighborhood size – k=1, window size = 3. – k >1, • Window size is 2k+1 – k=1, 2(1)+1 = 3 – Our kernel is 3x3 – Recall : A pixel is referenced by (i, j) • Expand the image we are smoothing: • Want Result Matrix ‘ G ’ – Wrap around pixels. (3x3 filter needs one more pixel – F à G using h. on each border, 5x5 needs 2) – G = h (operation) F – Copy edges 9 Maria Hybinette 10 Maria Hybinette Smoothing when a-i are all 1s. Computing the Values of Result Matrix 20 20 10 20 10 20 10 10 13 20 20 10 20 10 20 10 10 13 G[3, 3] = 1/9 (A + B + C + D + E + F + G + H + I ) 30 30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 30 30 k k 1 ∑ ∑ 20 0 A B C 90 90 0 20 G [ i , j ] = F [ i + u , j + v ] 20 0 90 90 90 90 90 0 20 • Let’s refer to (2 k + 1) 2 u = − k v = − k 20 0 90 0 90 90 90 0 20 20 0 D E F 90 90 0 20 – F[i,j] as input and 1 10 0 90 90 90 90 90 0 10 • “Standardizes” the weight to 10 0 G H I 90 90 0 10 (2 k + 1) 2 10 0 90 90 90 90 90 0 10 – G[i,j] as output each pixel . (k=1, then 1/9) 10 0 90 90 90 90 90 0 10 10 0 90 90 90 90 90 0 10 – h[i,j] as the kernel 10 0 90 90 90 90 90 0 10 • Loop over all pixels in k k 20 0 0 0 0 0 0 0 20 ∑ ∑ F [ i + u , j + v ] • Example : pixel (3,3) (expanded edges above) 20 0 0 0 0 0 0 0 20 u = − k v = − k neighborhood around particular 20 20 10 20 10 20 10 10 13 20 20 10 20 10 20 10 10 13 – G[3, 3] = a*A + b*B + c*C + d*D +e*E + f*F + h*H + i*I Loop over all pixels in image pixel F[i,j] a b c 1/9 1/9 1/9 – starting from upper right corner, -k, -k, if 1/9 1/9 1/9 k=1, then 1 left, from center and up 1. d e f – If all weights (a through i) are 1: 1/9 1/9 1/9 g h i – G[3, 3] = 1/9 (A + B + C + D + E + F + G + H + I ) 11 Maria Hybinette 12 Maria Hybinette

Generalize the uniform weight Example: Box Filter for Smoothing 20 20 10 20 10 20 10 10 13 20 20 10 20 10 20 10 10 13 G[3, 3] = 1/9 (A + B + C + D + E + F + G + H + I ) 30 30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 30 30 k k 1 ∑ ∑ 20 0 A B C 90 90 0 20 G [ i , j ] = F [ i + u , j + v ] 20 0 90 90 90 90 90 0 20 (2 k + 1) 2 u = − k v = − k 20 0 90 0 90 90 90 0 20 20 0 D E F 90 90 0 20 • More Generally, 10 0 90 90 90 90 90 0 10 10 0 G H I 90 90 0 10 10 0 90 90 90 90 90 0 10 10 0 90 90 90 90 90 0 10 10 0 90 90 90 90 90 0 10 10 0 90 90 90 90 90 0 10 G [ i , j ] = a * A + b * B + c * C + d * D + e * E + f * F + h * H + i * I 20 0 0 0 0 0 0 0 20 20 0 0 0 0 0 0 0 20 k k 20 20 10 20 10 20 10 10 13 ∑ ∑ 20 20 10 20 10 20 10 10 13 G [ i , j ] = h [ u , v ]* F [ i + u , j + v ] • Uniform Kernel (all 1s). Loop over all pixels in u = − k v = − k a b c – Box Area above is 21x21 the larger the kernel the ‘smoother’ the • Formula is Referred to Cross result. • The larger the box the smoother it gets. d e f Correlation more on this later. g h i h7ps://en.wikipedia.org/wiki/Box_blur 13 Maria Hybinette 14 Maria Hybinette Comparison Example: Median Filter for smoothing median( ) 20 20 1 20 20 1 20 10 20 10 13 13 20 20 1 20 10 20 10 13 13 0 0 0 1 20 20 20 20 90 20 0 20 0 0 0 0 0 0 0 0 0 0 20 20 0 20 20 median( ) 20 0 0 20 0 90 90 90 90 90 20 20 20 0 90 90 90 90 90 20 20 Median = 20 (removes outliers) 20 0 90 0 90 90 90 20 20 20 0 90 0 90 90 90 20 20 20 0 90 10 0 90 90 90 90 90 10 10 10 0 90 90 90 90 90 10 10 10 0 90 90 90 90 90 10 10 10 0 90 90 90 90 90 10 10 average( ) 10 0 90 90 90 90 90 10 10 10 0 90 90 90 90 90 10 10 0 0 0 1 20 20 20 20 90 20 20 10 20 10 20 10 13 13 F 20 0 20 0 0 0 0 0 0 0 0 0 0 20 20 0 20 20 20 20 10 20 10 20 10 13 13 Average =171/9 = 19 • Median Filter : Nonlinear filter – Reduces noise (the outliers are ‘basically’ noise and removed). • Median Filter Preserves Edges (11x11 median filter) – Preserves edges • Sharper Edges h7ps://en.wikipedia.org/wiki/Median_filter 15 Maria Hybinette 16 Maria Hybinette Median Filter vs. Average Filter Example: Median Filter for Noise Removal • Salt and Pepper Noise (3x3 median filter) • Median (11x11) on left, Average Filter on Right 17 Maria Hybinette 18 Maria Hybinette

Noise Removal Median vs. Average • Image Smoothing • Applying a kernel to smooth an image (cross correlation) • Applying a function (median) to smooth an image. • 3x3 Median Filter on left, 3x3 Average Filter on Right 19 Maria Hybinette 20 Maria Hybinette Cross Correlation Cross Correlation k k k k ∑ ∑ ∑ ∑ G [ i , j ] = h [ u , v ]* F [ i + u , j + v ] G [ i , j ] = h [ u , v ]* F [ i + u , j + v ] u = − k v = − k u = − k v = − k G = h ⊗ F • In signal processing, cross-correlation is a measure • Filtering an image: of similarity of two waveforms as a function of a – Replace each pixel with a linear combination of its time-lag applied to one of them. neighbors – Can find pattern in images with this approach, the • Filter “ kernel ” or “ mask ” kernel is the pattern we are looking for. – h[u,v] is the prescription for weights in the linear • Also known as a sliding dot product or sliding combination inner-product. 21 Maria Hybinette 22 Maria Hybinette Revisit: Box Filter Example: Gaussian Filter • Normal or Gaussian Distribution of kernel • Box/Average Filter – Give center values more importance. – Size 21 x 21 – Values are uniform Filter h7ps://en.wikipedia.org/wiki/Gaussian_blur Original Result 23 Maria Hybinette 24 Maria Hybinette

Varying the ‘Spread’ or Standard Average Filter vs. Gaussian Filter Deviation ( σ ) • Average is more blurry • Gaussian filter Gives more ‘detail’ to center pixel, • As sigma gets larger more blurry. Extends how far it smoothens the images. – Preserves edges more 25 Maria Hybinette 26 Maria Hybinette Correlation to Convolution Result of Impulse Function • Example : Take Cross Correlation of Impulse Kernel. • Notice : Reflected Result (switching rows and column of the kernel into result image) 27 Maria Hybinette 28 Maria Hybinette Another way to look at result Convolution Method • Convolution is a mathematical operation on two functions F and h, • Produces a third function that is typically viewed as a modified version • Gives the area of overlap between the two functions • In a form of the amount that one of the original • Reverse response functions is translated. 29 Maria Hybinette 30 Maria Hybinette

Convolution Method (*) Example: Linear Filter k k ∑ ∑ G [ i , j ] = h [ u , v ]* F [ i − u , j − v ] u = − k v = − k • Flip filter in both dimensions: 0 0 0 0 0 0 1/9 1/9 1/9 a b c g h i i h g – Bottom to top 0 0 1 0 1 0 1/9 1/9 1/9 d e f d e f f e d 0 0 0 0 0 0 1/9 1/9 1/9 – Right to left g h i a b c c b a • Then apply cross-correlation ( (X) ) • Commutative : F * G = G * F • Associative: (F*G) * H = F * (G*H) • Nothing • Shift by 1 • Identity (Unit Impulse E): F * E = F • Average Box Blur 31 Maria Hybinette 32 Maria Hybinette Example: Combing Linear Filters Sources & Inspiration Linear Filters 0 0 0 1/9 1/9 1/9 Contributors of Course Material: x 2 - = 0 1 0 1/9 1/9 1/9 • Irfan Essa (main) & Frank Dellaert (Georgia 0 0 0 1/9 1/9 1/9 Tech) – Also early adopters original, -0.11 -0.11 -0.11 64x64 • Marc Levoy (Stanford)– taught computational -0.11 1.9 -0.11 photography since 2002: -0.11 -0.11 -0.11 – A leader in the field : Frankecamera • Frédo Durand (MIT) • Jack Tumblin (Northwestern) • Magnify importance of middle pixel subtract a • Wikipedia smooth version of the image • http://www.all-art.org/ – Sharper image. history658_photography1.html • “Photography”, London, Stone, Upton 33 Maria Hybinette 34 Maria Hybinette