Lecture 2: Image filtering PS1 due next Tuesday Updated office - - PowerPoint PPT Presentation

▶
lecture 2 image filtering ps1 due next tuesday updated
SMART_READER_LITE
LIVE PREVIEW

Lecture 2: Image filtering PS1 due next Tuesday Updated office - - PowerPoint PPT Presentation

Dec 11, 2022 772 likes •1.54k views

Lecture 2: Image filtering PS1 due next Tuesday Updated office hours next week, due to holiday. New times will be on Piazza. Questions? Recall last week Extra edges Missing edges Input image Edges In this lecture What other

slide-1
SLIDE 1

Lecture 2: Image filtering

slide-2
SLIDE 2
  • PS1 due next Tuesday
  • Updated office hours next week, due to
  • holiday. New times will be on Piazza.
  • Questions?
slide-3
SLIDE 3

Recall last week…

Edges Input image

Missing edges Extra edges

slide-4
SLIDE 4

In this lecture

What other transformations can we do?

slide-5
SLIDE 5

Filtering

Source: Torralba, Freeman, Isola

Our goal: remove unwanted sources of variation, and keep the information relevant for whatever task we need to solve.

g[n, m] f[n, m]

slide-6
SLIDE 6

Linear filtering

Very general! For a filter, H, to be linear, it has to satisfy:

H(a[m, n] + b[m, n]) + H(a[m, n]) + H(b[m, n])

<latexit sha1_base64="bwjokjlr5CKTUP0e+sMmFK8tILY=">ACMXicbZDLSgMxFIYz9VbrbdSlm2ARWpQyo4JuhGI3XVawF+gMJZOmbWiSGZKMUIa+khvfRNx0oYhbX8K0nYpWDwS+8/85JOcPIkaVdpyJlVlZXVvfyG7mtrZ3dvfs/YOGCmOJSR2HLJStACnCqCB1TUjrUgSxANGmsGwMvWbD0QqGop7PYqIz1Ff0B7FSBupY1erBdTmZ8KHpzCYQdHQXISLJkgbz8tVCxX47d3AymK82LHzTsmZFfwLbgp5kFatYz973RDHnAiNGVKq7TqR9hMkNcWMjHNerEiE8BD1SdugQJwoP5ltPIYnRunCXijNERrO1J8TCeJKjXhgbnKkB2rZm4r/e1Y9679hIo1kTg+UO9mEdwml8sEslwZqNDCAsqfkrxAMkEdYm5JwJwV1e+S80zkvuRcm5u8yXb9M4suAIHIMCcMEVKIMqIE6wOARvIBX8GY9WRPr3fqYX81Y6cwh+FXW5xflFqLC</latexit>H(Ca[m, n]) = CH(a[m, n]) <latexit sha1_base64="bwjokjlr5CKTUP0e+sMmFK8tILY=">ACMXicbZDLSgMxFIYz9VbrbdSlm2ARWpQyo4JuhGI3XVawF+gMJZOmbWiSGZKMUIa+khvfRNx0oYhbX8K0nYpWDwS+8/85JOcPIkaVdpyJlVlZXVvfyG7mtrZ3dvfs/YOGCmOJSR2HLJStACnCqCB1TUjrUgSxANGmsGwMvWbD0QqGop7PYqIz1Ff0B7FSBupY1erBdTmZ8KHpzCYQdHQXISLJkgbz8tVCxX47d3AymK82LHzTsmZFfwLbgp5kFatYz973RDHnAiNGVKq7TqR9hMkNcWMjHNerEiE8BD1SdugQJwoP5ltPIYnRunCXijNERrO1J8TCeJKjXhgbnKkB2rZm4r/e1Y9679hIo1kTg+UO9mEdwml8sEslwZqNDCAsqfkrxAMkEdYm5JwJwV1e+S80zkvuRcm5u8yXb9M4suAIHIMCcMEVKIMqIE6wOARvIBX8GY9WRPr3fqYX81Y6cwh+FXW5xflFqLC</latexit>

Source: Torralba, Freeman, Isola

slide-7
SLIDE 7

A linear filter in its most general form can be written as (for a 1D signal of length N): In matrix form:

Linear filtering

Source: Torralba, Freeman, Isola

slide-8
SLIDE 8

Photo by Fredo Durand, slide by Torralba, Freeman, Isola

Why handle each spatial position differently? Want translation invariance!

slide-9
SLIDE 9

Photo by Fredo Durand

Image denoising

slide-10
SLIDE 10
  • Let’s replace each pixel with a weighted

average of its neighborhood

  • The weights are called the filter kernel
  • What are the weights for the average of a 


3x3 neighborhood?

1 1 1 1 1 1 1 1 1 “box filter”

Source: D. Lowe

Moving average

slide-11
SLIDE 11

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

Moving average

?

Filter kernel

Input Output

slide-12
SLIDE 12

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

Moving average

40

Filter kernel

Input Output

slide-13
SLIDE 13

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

Moving average

40 ?

Filter kernel

Input Output

slide-14
SLIDE 14

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

Moving average

40 60

Filter kernel

Input Output

slide-15
SLIDE 15

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

Moving average

40 60 ?

Filter kernel

Input Output

slide-16
SLIDE 16

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

Moving average

40 60 80

Filter kernel

Input Output

slide-17
SLIDE 17

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

Moving average

40 60 60 40 20 60 90 60 40 20 50 80 80 60 30 50 80 80 60 30 30 50 50 40 20

Filter kernel

Input Output

slide-18
SLIDE 18

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

Moving average

40 60 80

Filter kernel

Input Output

40 60 60 40 20 60 90 60 40 20 50 80 80 60 30 ? 50 80 80 60 30 30 50 50 40 20

slide-19
SLIDE 19

Handling boundaries

Source: Torralba, Freeman, Isola

slide-20
SLIDE 20

Handling boundaries

11x11 box = Zero padding

Source: Torralba, Freeman, Isola

slide-21
SLIDE 21

Handling boundaries

Input Output Error

Source: Torralba, Freeman, Isola

slide-22
SLIDE 22

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

Moving average

40 60 80

Filter kernel

Input Output

40 60 60 40 20 60 90 60 40 20 50 80 80 60 30 30 50 80 80 60 30 30 50 50 40 20

slide-23
SLIDE 23

Convolution

  • Let h be the image and g be the kernel. The output of

convolving h with g is:

h

Source: F. Durand

Convention: 
 kernel is “flipped”

slide-24
SLIDE 24

Properties of the convolution

Commutative Associative Distributive with respect to the sum

slide-25
SLIDE 25

Why flip the kernel?

Indexes go backward!

slide-26
SLIDE 26

Cross correlation

  • Sometimes called just correlation
  • Neither associative nor commutative
  • In the literature, people often just call both “convolution”
  • Filters often symmetric, so won’t matter

No flipping!

slide-27
SLIDE 27

Convolutional neural networks

  • Neural network with specialized connectivity structure
  • Mostly just convolutions!

(LeCun et al. 1989)

Source: Torralba, Freeman, Isola

slide-28
SLIDE 28

Filtering examples

slide-29
SLIDE 29

Practice with linear filters

1 Original

?

Source: D. Lowe

slide-30
SLIDE 30

Practice with linear filters

1 Original Filtered (no change)

Source: D. Lowe

“Impulse”

slide-31
SLIDE 31

Practice with linear filters

1 Original

?

Source: D. Lowe

“Translated Impulse”

slide-32
SLIDE 32

Practice with linear filters

1 Original Shifted left By 1 pixel

Source: D. Lowe

slide-33
SLIDE 33

Practice with linear filters

Original

?

1 1 1 1 1 1 1 1 1

Source: D. Lowe

slide-34
SLIDE 34

Practice with linear filters

Original 1 1 1 1 1 1 1 1 1 Blur (with a box filter)

Source: D. Lowe

slide-35
SLIDE 35

Practice with linear filters

Original 1 1 1 1 1 1 1 1 1 2

  • ?

(Note that filter sums to 1)

Source: D. Lowe

slide-36
SLIDE 36

Practice with linear filters

Original 1 1 1 1 1 1 1 1 1 2

  • Sharpening filter
  • Accentuates differences with

local average

Source: D. Lowe

slide-37
SLIDE 37

Sharpening

Source: D. Lowe

slide-38
SLIDE 38

Practice with linear filters

Original

?

Can you do this?

slide-39
SLIDE 39

Rectangular filter

⊗

g[m,n] h[m,n] = f[m,n]

Source: Torralba, Freeman, Isola

slide-40
SLIDE 40

Rectangular filter

⊗

g[m,n] h[m,n] = f[m,n]

Source: Torralba, Freeman, Isola

slide-41
SLIDE 41

Motion blur

“Naturally” occurring filters

Input image

Source: Torralba, Freeman, Isola

slide-42
SLIDE 42

“Naturally” occurring filters

Convolution output Convolution weights Input image

Source: Torralba, Freeman, Isola

slide-43
SLIDE 43

Camera shake

(from Fergus et al, 2007)

Source: Torralba, Freeman, Isola

slide-44
SLIDE 44

Blur occurs in many natural situations

Source: Torralba, Freeman, Isola

slide-45
SLIDE 45

Smoothing with box filter revisited

Source: D. Forsyth

  • What’s wrong with this picture?
  • What’s the solution?
slide-46
SLIDE 46

Smoothing with box filter revisited

  • What’s wrong with this picture?
  • What’s the solution?

“fuzzy blob”

Source: S. Lazebnik

  • To eliminate edge effects, weight contribution of

neighborhood pixels according to their closeness to the center

slide-47
SLIDE 47

Gaussian kernel

Source: K. Grauman

σ = 2 with 30 x 30 kernel σ = 5 with 30 x 30 kernel

  • Constant factor in front makes kernel sum to 1 (can

also omit it and just divide by sum of filter weights).

slide-48
SLIDE 48

Gaussian vs. box filtering

Source: S. Lazebnik

slide-49
SLIDE 49

Gaussian standard deviation

49

σ=2 σ=4 σ=8

Source: Torralba, Freeman, Isola

slide-50
SLIDE 50

Gaussian filters

  • Convolution with self is another Gaussian
  • Can smooth with small-σ kernel, repeat, get same result as

larger-σ

  • Convolving two times with Gaussian kernel with std. dev. σ 


is same as convolving once with kernel with std. dev.

Source: K. Grauman

2 σ

I blur(blur(I)) blur(blur(blur(I))) blur(blur(blur(blur(I))))

slide-51
SLIDE 51

Gaussian filters

  • It’s a separable kernel
  • Blur with 1D Gaussian in one direction, then the other.
  • Faster to compute. O(n) time for an n*n kernel instead of O(n2)
  • Learn more about this in Problem Set 1!

I blurx(I) blury(blurx(I))

slide-52
SLIDE 52

Edges: recall last lecture…

Image gradient: Approximation image derivative: Edge strength Edge orientation: Edge normal: I(x,y)

Slide credit: Antonio Torralba

slide-53
SLIDE 53

Discrete derivatives

Source: Torralba, Freeman, Isola

slide-54
SLIDE 54

[-1 1]

g[m,n] h[m,n] = f[m,n] [-1, 1]

Source: Torralba, Freeman, Isola

slide-55
SLIDE 55

[-1 1]T

g[m,n] h[m,n] = f[m,n] [-1, 1]T

Source: Torralba, Freeman, Isola

slide-56
SLIDE 56

Can we recover the image?

?

Source: Torralba, Freeman, Isola

slide-57
SLIDE 57

Reconstruction from 2D derivatives

[-1 1] [-1 1]T c c = c

In 2D, we have multiple derivatives (along n and m) and we compute the pseudo-inverse of the full matrix.

Source: Torralba, Freeman, Isola

slide-58
SLIDE 58

Reconstruction from 2D derivatives

[1 -1] [1 -1]T

Source: Torralba, Freeman, Isola

slide-59
SLIDE 59

Editing the edge image

[1 -1] [1 -1]T

Source: Torralba, Freeman, Isola

slide-60
SLIDE 60

Thresholding edges

Source: Torralba, Freeman, Isola

slide-61
SLIDE 61

Issues with derivative filters

[1 -1]

  • Sensitive to edges at small spatial scales
  • Also sensitive to noise
  • You'll see this in Problem Set 1
slide-62
SLIDE 62

Why is this happening?

Where is the edge?

Source: S. Seitz

Noisy input image

slide-63
SLIDE 63

Solution: smooth first

f h f * h

Source: S. Seitz

To find edges, look for peaks in

slide-64
SLIDE 64

Gaussian Derivative of Gaussian (x)

Source: N. Snavely

Derivative of Gaussian filter

slide-65
SLIDE 65

Derivative of Gaussian filter

x-direction y-direction

Source: N. Snavely

slide-66
SLIDE 66

Derivatives of Gaussians: Scale

σ=2 σ=4 σ=8

Source: Torralba, Freeman, Isola

slide-67
SLIDE 67

Picks up larger-scale edges

Source: Torralba, Freeman, Isola

slide-68
SLIDE 68

Computing a directional derivative

= ?

+ =

(From multivariable calculus) Directional derivative is a linear combination of partial derivatives

Source: N. Snavely

slide-69
SLIDE 69

Derivative of Gaussian filter

x-direction y-direction

+ =

Source: N. Snavely

slide-70
SLIDE 70

The Sobel operator

  • Common approximation to derivative of Gaussian
  • Where does this come from?

Source: N. Snavely

slide-71
SLIDE 71
  • Apply filter to itself repeatedly.
  • Converges to Gaussian, due to Central Limit Theorem

An approximation to the Gaussian

[1 1] [1 1] = [1 2 1] [1 1] [1 1] [1 1] = [1 3 3 1] b1 = [1 1] b2 = b3 =

Source: Torralba, Freeman, Isola

slide-72
SLIDE 72

Binomial filter

72

slide-73
SLIDE 73

Sobel operator: example

Source: N. Snavely / Wikipedia

slide-74
SLIDE 74

Next class: frequency