Einfhrung in Visual Computing Unit 12: Local Operations http:// - - PowerPoint PPT Presentation

▶
einf hrung in visual computing
SMART_READER_LITE
LIVE PREVIEW

Einfhrung in Visual Computing Unit 12: Local Operations http:// - - PowerPoint PPT Presentation

Dec 07, 2022 46 likes •898 views

Einfhrung in Visual Computing Unit 12: Local Operations http:// www.caa.tuwien.ac.at/cvl/teaching/sommersemester/evc Content: Neighborhood Introduction to Local Operations Boundary Problem 2D Convolution Linear Image

slide-1
SLIDE 1

Einführung in Visual Computing

Unit 12: Local Operations

  • Content:
  • Neighborhood
  • Introduction to Local Operations
  • Boundary Problem
  • 2D Convolution
  • Linear Image Filters
  • Non-Linear Image Filters
  • Noise
  • Median Filter

http://www.caa.tuwien.ac.at/cvl/teaching/sommersemester/evc

1 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-2
SLIDE 2

Types of Operations

  • The operations that can be applied to digital images to transform

an input image a[m,n] into an output image b[m,n]

  • Point: the output value at a specific coordinate is dependent
  • nly on the input value at that same coordinate.
  • Local: the output value at a specific coordinate is dependent on

the input values in the neighborhood of that same coordinate

  • Global: the output value at a specific coordinate is dependent
  • n all the values in the input image

2 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-3
SLIDE 3

Types of Neighborhoods

  • Neighborhood operations play a key role in digital image

processing

  • Neighborhoods are basically of two types
  • 4-connected
  • 8-connected
  • Two types of sampling schemes are
  • Rectangular sampling - In most cases, images are sampled by

laying a rectangular grid over an image

  • Hexagonal sampling - An alternative sampling scheme

3 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-4
SLIDE 4

(a) (b) Types of neighborhood (a) 4 connected (b) 8 connected

Types of Neighborhoods

Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations 4

slide-5
SLIDE 5

Local Operations

5

slide-6
SLIDE 6

Local-Operations

  • Point Operators alter image pixel-wise I‘(u,v) = T(I(u,v)). A pixel in

the result image depends only on one single pixel in the original image.

  • Local Operators consider the neighborhood of the pixel too. The

locality N of a P(u,v) is given by the surrounding pixels of P(u,v)

  • The neighborhood is usually square-shaped
  • Local Operations are very important in image processing. (Usually

applied before image analyzing algorithms)

6 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-7
SLIDE 7

Local-Operators

The Gray level of the resulting pixel is dependent on several pixel in the

  • riginal image:
  • An operator window is placed around an actual pixel
  • Computation of the resulting pixel by combination of gray levels of

the actual pixel and its neighbors and saving it at the position of the actual pixel at its position in the result image

  • Computation of the complete image (by displacement of the operator

window by one pixel)

Operator window Operator window Actual pixel

7 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-8
SLIDE 8

Neighbourhood Operations

  • For each pixel in the origin image, the outcome is written on the

same location at the target image.

Origin x y Image f (x, y) (x, y) Neighbourhood Target Origin

8 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-9
SLIDE 9

Simple Neighbourhood Operations

  • Simple neighbourhood operations example:
  • Min: Set the pixel value to the minimum in the neighbourhood
  • Max: Set the pixel value to the maximum in the neighbourhood

Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations 9

slide-10
SLIDE 10

The Spatial Filtering Process

j k l m n

  • p

q r

Origin x y Image f (x, y)

eprocessed =

n*e + j*a + k*b + l*c + m*d + o*f + p*g + q*h + r*i

Filter (w)

Simple 3*3 Neighbourhood e 3*3 Filter

a b c d e f g h i

Original Image Pixels

*

The above is repeated for every pixel in the original image to generate the filtered image

10 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-11
SLIDE 11

Linear 2-D Filter

11 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-12
SLIDE 12

Spatial Filtering: Equation Form Filtering can be given in equation form as shown above Notations are based on the image shown to the left

∑∑

− = − =

+ + =

a a s b b t

t y s x f t s w y x g ) , ( ) , ( ) , (

12 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-13
SLIDE 13

Example 2D Filter

Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations 13

slide-14
SLIDE 14

Smoothing Spatial Filtering

One of the simplest spatial filtering operations we can perform is a smoothing operation

  • Simply average all of the pixels in a neighbourhood around a

central value

  • Especially useful

in removing noise from images

  • Also useful for

highlighting gross detail

1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9

Simple averaging filter

14 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-15
SLIDE 15

Smoothing Spatial Filtering

1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9

Origin x y Image f (x, y)

e =

1/9*106 + 1/9*104 + 1/9*100 + 1/9*108 + 1/9*99 + 1/9*98 + 1/9*95 + 1/9*90 + 1/9*85

= 98.3333

Filter

Simple 3*3 Neighbourhood

106

104 99 95 100 108 98 90 85

1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9

3*3 Smoothing Filter

104 100 108 99 106 98 95 90 85

Original Image Pixels

*

The above is repeated for every pixel in the original image to generate the smoothed image

15 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-16
SLIDE 16

Definition of a Local Operator

  • The convolution window is moved over the entire image.

Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations 16

slide-17
SLIDE 17

Image Smoothing Example

  • The image at the top left is an
  • riginal image of size 500*500

pixels

  • The subsequent images show the

image after filtering with an averaging filter of increasing sizes 3, 5, 9, 15 and 35

  • Notice how detail begins to

disappear

17 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-18
SLIDE 18

Image Smoothing Example

18 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-19
SLIDE 19

Image Smoothing Example

19 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-20
SLIDE 20

Image Smoothing Example

20 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-21
SLIDE 21

Image Smoothing Example

21 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-22
SLIDE 22

Image Smoothing Example

22 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-23
SLIDE 23

Image Smoothing Example

23 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-24
SLIDE 24

Weighted Smoothing Filters

  • More effective smoothing filters can be generated by allowing

different pixels in the neighbourhood different weights in the averaging function

  • Pixels closer to the

central pixel are more important

  • Often referred to as a

weighted averaging

1/16 2/16 1/16 2/16 4/16 2/16 1/16 2/16 1/16

Weighted averaging filter

24 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-25
SLIDE 25

Another Smoothing Example

By smoothing the original image we get rid of lots of the finer detail which leaves only the gross features for thresholding

Original Image Smoothed Image Thresholded Image

* Image taken from Hubble Space Telescope

25 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-26
SLIDE 26

Photoshop: Other Filters - Custom Filter

26 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-27
SLIDE 27

Boundary Problem

slide-28
SLIDE 28

Strange Things Happen At The Edges!

At the edges of an image we are missing pixels to form a neighbourhood

Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations 28

Origin x y Image f (x, y)

e e e e e e e

slide-29
SLIDE 29

Strange Things Happen At The Edges! (cont…)

  • There are a few approaches to dealing with missing edge pixels:
  • Omit missing pixels: Only works with some filters, Can add extra

code and slow down processing

  • Pad the image: Typically with either all white or all black pixels
  • Replicate border pixels
  • Truncate the image

Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations 29

slide-30
SLIDE 30

Boundary Problem

  • Problem: Border pixels cannot be

reached!

  • Possible solution:
  • Result image shrinks -2 Pixel in

width and height. However this solution is usually not feasible. The image size has to remain constant throughout the entire processing.

30 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-31
SLIDE 31

Boundary Problem

  • Another solution:
  • Border-Pixel are filled with an

intensity value.

  • Being completely ignorant we

could use black. In this case the image remains

  • constant. However a very ugly

black border remains.

31 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-32
SLIDE 32

Boundary Problem

  • A somewhat better solution:
  • Border-Pixel are not processed.

They are left untouched with their „old“intensity value. When large sized kernels are used boundary might get visible.

32 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-33
SLIDE 33

Convolution

slide-34
SLIDE 34

Linear Filter

  • Output signal at each point = weighted average of local region of

input signal

  • Depends on input signal, pattern of weights
  • “Filter” H(i,j) = function of weights for linear combination
  • Basic operation = move filter to some position u, add up i•j

times H

  • In general:

34 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

∑ ∑

− = − =

⋅ + + ←

1 1 1 1

) , ( ) , ( ) , ( '

i j

j i H j v i u I v u I

slide-35
SLIDE 35

Convolution

Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations 35

∫

∞ ∞ −

− = ⋅ dt t x g t f x g x f ) ( ) ( ) ( ) (

f(x) g(x)

slide-36
SLIDE 36

Convolution

  • f is called “signal” and g is “filter” or “kernel”, but the operation is

symmetric

  • Usually desirable to leave a constant signal

unchanged: choose g such that

1 ) ( =

∫

∞ ∞ −

dt t g

36 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-37
SLIDE 37

Two-dimensional Convolution

Function Function

Linerar Convolution

37 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-38
SLIDE 38

Difference: Linear Filter - Convolution

  • Correlation:
  • Convolution:

Or

  • Filterfunction is rotated by 180°

Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations 38

∑ ∑

− = − =

⋅ + + ←

1 1 1 1

) , ( ) , ( ) , ( '

i j

j i H j v i u I v u I

∑ ∑

∞ −∞ = ∞ −∞ =

⋅ − − =

i j

j i H j v i u I v u I ) , ( ) , ( ) , ( '

∑ ∑

∞ −∞ = ∞ −∞ =

− − ⋅ + + =

i j

j i H j v i u I v u I ) , ( ) , ( ) , ( '

slide-39
SLIDE 39

Properties of linear Filters

  • Commutative:
  • I ∗ H = H ∗ I
  • Linear:
  • (a · I) ∗ H = I ∗ (a · H) = a · (I ∗ H)
  • (I1 + I2) ∗ H = (I1 ∗ H) + (I2 ∗ H)
  • Associative:
  • A ∗ (B ∗ C) = (A ∗ B) ∗ C
  • Separabel: H = H1 ∗ H2 ∗ . . . ∗ Hn

Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations 39

slide-40
SLIDE 40

Separable Templates

  • Convolve with n x n template
  • n2 multiplications and additions
  • Convolve with two n x 1 templates
  • 2n multiplications and additions
  • Box Filter
  • Separated kernels

40 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

          = = 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 *

y x xy

H H H

[ ]

          = = 1 1 1 , 1 1 1 1 1

y x

H H

slide-41
SLIDE 41

Composite Filters

  • Convolution is distributive
  • Can create a composite filter and do a single convolution
  • Not convolve image with one filter and convolve result with

second.

  • Efficiency gain

( ) ( )

C B A C B A ⊗ ⊗ = ⊗ ⊗

41 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-42
SLIDE 42

Image Filters

  • Usually a uneven sized

kernel is chosen (e.g. 3x3, 5x5, ...) for symmetry reasons

42 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-43
SLIDE 43

Linear Image Filters

43

slide-44
SLIDE 44

Image Filters

  • Filter kernels are scaled in order to guarantee that the brightness

does not change generally. The scaling factor is computed by the sum of the coefficients of the kernel matrix.

  • Example:

 Intensity of a pixel is the sum of all of its 8

neighbors

 On a white image (all pixel = 1) new value 9!  Intensity of a pixel is the average of all of its 8

neighbors

 On a white image (all pixel = 1) new value 1!

44 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-45
SLIDE 45

Averaging Filter

  • Example 4x4 Averaging Filter on 64x64 Pixel Image:

45 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-46
SLIDE 46

Averaging Filter

  • Analytic function: statistical mean
  • Type: linear Operator
  • Goal: Image smoothing, Noise suppression

Original Mean

Mean Operator

46 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-47
SLIDE 47

Linear Filters: Examples

1 Original Identical image

Source: D. Lowe

*

=

47 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-48
SLIDE 48

Linear Filters: Examples

1 Original Shifted left By 1 pixel

Source: D. Lowe

*

=

48 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-49
SLIDE 49

Linear Filters: Examples

Original 1 1 1 1 1 1 1 1 1 Blur (with a mean filter)

Source: D. Lowe

*

=

49 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-50
SLIDE 50

Linear Filters: Examples

Original 1 1 1 1 1 1 1 1 1 2

  • Sharpening filter

(accentuates edges)

Source: D. Lowe

=

*

50 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-51
SLIDE 51

Photoshop: Other Filters - Custom Filter

51 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-52
SLIDE 52

Averaging Filter

  • Averaging Filter is a low pass filter.
  • Averaging Filter can smooth unstructured noise
  • Periodic noise cannot be suppressed properly (see frequency

response)

Frequency response of a 3x3 averaging filter Frequency response of a 5x5 averaging filter

52 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-53
SLIDE 53

2D Gaussian Filter

53 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-54
SLIDE 54

Gaussian Filter

  • Filter kernel can be approximated by convolution of two one-

dimensional binomial distributions.

Approximation to Gauss‘ bell-shape curve (2D) Example: 3x3 Gaussian Kernel

54 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-55
SLIDE 55

Gaussian Filter

55 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-56
SLIDE 56

Gaussian Filter

  • Gaussians are used because:
  • Smooth (infinitely differentiable)
  • Decay to zero rapidly
  • Simple analytic formula
  • Separable: multidimensional Gaussian = product of Gaussians in

each dimension

  • Convolution of 2 Gaussians = Gaussian
  • Limit of applying multiple filters is Gaussian (Central limit

theorem)

56 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-57
SLIDE 57

Sampled Signals

  • Can’t store continuous signal: instead store “samples”
  • Usually evenly sampled:

f0=f(x0), f1=f(x0+∆x), f2=f(x0+2∆x), f3=f(x0+3∆x), …

  • Instantaneous measurements of continuous signal
  • This can lead to problems

→

57 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-58
SLIDE 58

Aliasing

  • Reconstructed signal might be very different from original:

“aliasing”

  • Solution: smooth the signal before sampling

→ →

58 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-59
SLIDE 59

Non-Linear Filters

slide-60
SLIDE 60

Linear and Non-linear Filters

  • The image filters we looked at so far are all linear (given that

clipping does not occur).

  • Linearity of a Filter H means if applied to two images IA and IB

following equation is satisfied:

Linear filters (so far):

  • Low- and High-Pass filter
  • Gauss filter

Non-linear filter:

  • Min/Max filter
  • Median filter

60 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-61
SLIDE 61

Non-linear Filters

  • Linear filters may have disadvantages: Linear smoothing filter

suppresses noise but blurs the image at the same time.

  • Non-linear filters (rank value filters) do not have this
  • disadvantage. Therefore they are usually applied for noise

removal.

61 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-62
SLIDE 62

Minimum Filter

62 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-63
SLIDE 63

Minimum Filter

63 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-64
SLIDE 64

Maximum Filter

64 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-65
SLIDE 65

Noise

slide-66
SLIDE 66

Types of Noise

  • There are several characteristic sources of digital image noise:
  • Photon shot noise
  • Sensor read noise
  • Pattern noise
  • Thermal noise:
  • Pixel response non-uniformity (PRNU)
  • Quantization error

66 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-67
SLIDE 67

Photon Shot Noise

  • Light is discrete bundles of energy called photons -> more intense

light, higher number of photons per second.

  • Photons arrive discretely, independently and randomly described

by Poisson statistics.

Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations 67

The noise histogram is essentially a Gaussian distribution

slide-68
SLIDE 68

Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations 68

slide-69
SLIDE 69

Types of Noise

  • Random variable X Gaussian

distributed with mean µ and standard deviation σ.

  • In order to prevent intensities
  • utside of the interval [0, 1] in

the noisy image, a clipping function f ( ixy+X) is applied to the sum of the original image ixy and the random variable X.

69 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-70
SLIDE 70

Types of Noise

  • Random variable Y is uniformly

distributed within (0, 1).

  • Noise function n(h,y,d) is applied

to every pixel h of the input

  • image. Let y be a concrete
  • bservation of the random

variable Y and d the density parameter of the „salt and pepper noise“.

70 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-71
SLIDE 71

Image Noise

http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/index.html Original image Gaussian noise Salt and pepper noise

(each pixel has some chance of being switched to zero or one)

71 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-72
SLIDE 72

Gaussian Noise

= 1 pixel = 2 pixels = 5 pixels

Smoothing with larger standard deviations suppresses noise, but also blurs the image

72 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-73
SLIDE 73

Salt & Pepper Noise – Gaussian Blur

= 1 pixel = 2 pixels = 5 pixels

What’s wrong with the results?

p = 10%

73 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-74
SLIDE 74

Alternative Idea: Median Filtering

  • A median filter operates over a window by selecting the median

intensity in the window Is median filtering linear?

74 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-75
SLIDE 75

Median Filter

75

slide-76
SLIDE 76

Median Filter

The Median filter is a non-linear

  • filter. The kernel window selects a

pixel set S around the center coordinates (xc,yc) of the original

  • image. The new value C‘(xc,yc) is set

to the median value of the pixel set

  • S. That is C‘(xc,yc) = median( S ).

76 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-77
SLIDE 77

Median Filter: Outliers are eliminated

77 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-78
SLIDE 78

Example: 3 Median

  • Linear Ramp
  • Linear Ramp with

Spike

78 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-79
SLIDE 79

Treated like Outlier

Example: 3 Median

  • 1 Bit Pulse
  • 2 Bit Pulse

79 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-80
SLIDE 80

Salt & Pepper Noise – Median Filtering

= 1 pixel = 2 pixels = 5 pixels 3x3 window 5x5 window 7x7 window p = 10%

80 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-81
SLIDE 81

Example Lenna

81 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-82
SLIDE 82

Example Lenna

82 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-83
SLIDE 83

Lenna

  • The Lenna (or Lena) picture is one
  • f the most widely used standard

test images

  • ‘Lena' or ‘Lenna' is a digitized Playboy

centerfold, from November 1972: Lena Soderberg

  • Alexander Sawchuk was searching the lab for a good

image to scan for a colleague's conference paper. They wanted something glossy to ensure good output dynamic range, and they wanted a human face. Just then, somebody happened to walk in with a recent issue

  • f Playboy…….
  • More on: http://www.cs.cmu.edu/~chuck/lennapg/

Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations 83

slide-84
SLIDE 84

Median Filter Examples

  • Salt & Pepper

Noise:

  • Gaussian Noise:

3x3 Median 3x3 Median

84 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations

slide-85
SLIDE 85

Summary

  • Local Operations involve neighborhood of a pixel
  • Local Operations can be computed in spatial domain using

convolution

  • Filter can be used to improve image quality
  • Mean Filter
  • Gaussian Filter
  • Median Filter (non-linear)
  • Noise
  • Gaussian distributed noise
  • „Salt and Pepper“ noise

85 Robert Sablatnig, Computer Vision Lab, EVC-12: Local Operations