Image Processing II Computer Vision Fall 2018 Columbia University - - PowerPoint PPT Presentation

▶
image processing ii
SMART_READER_LITE
LIVE PREVIEW

Image Processing II Computer Vision Fall 2018 Columbia University - - PowerPoint PPT Presentation

Jan 14, 2023 121 likes •1.1k views

Image Processing II Computer Vision Fall 2018 Columbia University Convolution Review Cross Correlation 0 0 0 1 G [ x , y ] -1 0 1 F [ x , y ] 9 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90

slide-1
SLIDE 1

Image Processing II

Computer Vision Fall 2018 Columbia University

slide-2
SLIDE 2

Convolution Review

slide-3
SLIDE 3

Cross Correlation

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

F[x, y]

  • 1

1

1 9

G[x, y]

slide-4
SLIDE 4

Cross Correlation

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

F[x, y]

  • 1

1

1 9

G[x, y]

slide-5
SLIDE 5

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

F[x, y]

  • 1

1

1 9

Cross Correlation

G[x, y]

slide-6
SLIDE 6

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

F[x, y]

  • 1

1

1 9

Cross Correlation

G[x, y]

slide-7
SLIDE 7

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

F[x, y]

  • 1

1

1 9

Cross Correlation

G[x, y]

slide-8
SLIDE 8

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

F[x, y]

  • 1

1

1 9

Cross Correlation

G[x, y]

slide-9
SLIDE 9

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

F[x, y]

  • 1

1

1 9

Cross Correlation

G[x, y]

slide-10
SLIDE 10

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

F[x, y]

  • 1

1

1 9

Cross Correlation

G[x, y]

slide-11
SLIDE 11

90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90

F[x, y]

1

  • 1

1 9

Convolution

G[x, y]

slide-12
SLIDE 12

Convolution

(f * g)[x, y] = ∑

i,j

f[x − i, y − j]g[i, j]

f g

Flip LR, UD

slide-13
SLIDE 13

We flip for the nice properties

F ∗ H = H ∗ F (F ∗ H) ∗ G = F ∗ (H ∗ G) (F ∗ G) + (H ∗ G) = (F + H) ∗ G

Commutative: Associative: Distributive:

slide-14
SLIDE 14

Fourier Transforms

slide-15
SLIDE 15

Vision is Repetitive

slide-16
SLIDE 16

Joseph Fourier

Wikipedia

A bold idea (1807): Any univariate function can be rewritten as a weighted sum of sines and cosines of different frequencies. Don’t believe it? Neither did Lagrange, Laplace, Poisson and other bigwigs Not translated into English until 1878!

Source: James Hays

slide-17
SLIDE 17

How to build this 1D signal using sin waves?

slide-18
SLIDE 18

= + + + + +…

How to build this 1D signal using sin waves?

slide-19
SLIDE 19

=

slide-20
SLIDE 20

= +

slide-21
SLIDE 21

= + +

slide-22
SLIDE 22

= + + + + +…

slide-23
SLIDE 23

= + + + + +…

Where we are going…

slide-24
SLIDE 24

Background: Sinusoids

Source: Deva Ramanan

A = amplitude = phase f = frequency

ϕ

slide-25
SLIDE 25

Fourier Transform

Amplitude Phase Signal

slide-26
SLIDE 26

2D Fourier Transform

Amplitude Phase Signal

slide-27
SLIDE 27

Wikipedia – Unit Circle

sin t cos t

eift = cos ft + i sin ft

Sine/cosine and circle

Amplitude: Radius of circle Frequency: How fast you change t

slide-28
SLIDE 28

Square wave (approx.)

Mehmet E. Yavuz

slide-29
SLIDE 29

Sawtooth wave (approx.)

Mehmet E. Yavuz

slide-30
SLIDE 30

Wikipedia – Unit Circle

sin t cos t

eift = cos ft + i sin ft

Sine/cosine and circle

Amplitude: Radius of circle Frequency: How fast you change t

slide-31
SLIDE 31

Towards Fourier Transform

Wikipedia – Unit Circle

sin t cos t

Maps g(t) on to the unit circle with frequency f

g(t)

g(t)e−2πift

slide-32
SLIDE 32

The Fourier Transform

Wikipedia – Unit Circle

sin t cos t

G(f ) = ∫

∞ −∞

g(t)e−2πiftdt

How I think of it: You wrap g(t) around the circle with frequency f, then calculate average position of g(t)

slide-33
SLIDE 33

Signal that we want to compute FT on

g(t) = cos(t) + 1

slide-34
SLIDE 34

The Fourier Transform

g(t) = cos(t) + 1 g(t)e−2πift, t = 0…100π f = 0.001 G( f ) = ∫

∞ −∞

g(t)e−2πiftdt

slide-35
SLIDE 35

The Fourier Transform

G( f ) = ∫

∞ −∞

g(t)e−2πiftdt g(t) = cos(t) + 1 g(t)e−2πift, t = 0…100π f = 0.002

slide-36
SLIDE 36

The Fourier Transform

G( f ) = ∫

∞ −∞

g(t)e−2πiftdt g(t) = cos(t) + 1 g(t)e−2πift, t = 0…100π f = 0.003

slide-37
SLIDE 37

The Fourier Transform

G( f ) = ∫

∞ −∞

g(t)e−2πiftdt g(t) = cos(t) + 1 g(t)e−2πift, t = 0…100π f = 0.3

slide-38
SLIDE 38

The Fourier Transform

G( f ) = ∫

∞ −∞

g(t)e−2πiftdt g(t) = cos(t) + 1 g(t)e−2πift, t = 0…100π f = 0.4

slide-39
SLIDE 39

The Fourier Transform

G( f ) = ∫

∞ −∞

g(t)e−2πiftdt g(t) = cos(t) + 1 g(t)e−2πift, t = 0…100π f = 0.5

slide-40
SLIDE 40

The Fourier Transform

G( f ) = ∫

∞ −∞

g(t)e−2πiftdt g(t) = cos(t) + 1 g(t)e−2πift, t = 0…100π f = 0.317

slide-41
SLIDE 41

The Fourier Transform

G( f ) = ∫

∞ −∞

g(t)e−2πiftdt g(t) = cos(t) + 1 g(t)e−2πift, t = 0…100π f = 1 π

slide-42
SLIDE 42

The Fourier Transform

G( f ) = ∫

∞ −∞

g(t)e−2πiftdt g(t) = cos(t) + 1 g(t)e−2πift, t = 0…100π f = 1 2π

slide-43
SLIDE 43

The Fourier Transform

G( f ) = ∫

∞ −∞

g(t)e−2πiftdt g(t) = cos(t) + 1 g(t)e−2πift, t = 0…100π f = 1 2π G( f )

slide-44
SLIDE 44

The Fourier Transform

G( f ) = ∫

∞ −∞

g(t)e−2πiftdt g(t) = cos(t + 0.5) + 1 g(t)e−2πift, t = 0…100π f = 1 2π G( f )

slide-45
SLIDE 45

The Fourier Transform

G( f ) = ∫

∞ −∞

g(t)e−2πiftdt g(t) = 2 cos(t + 0.5) + 1 g(t)e−2πift, t = 0…100π f = 1 2π G( f )

slide-46
SLIDE 46

The Fourier Transform

Amplitude: Phase: ℜ[G(f )]2 + ℑ[G(f )]2 tan−1 ℑ[G(f )] ℜ[G(f )]

G(f ) = ∫

∞ −∞

g(t)e−2πiftdt

slide-47
SLIDE 47

Inverse Fourier Transform

Amplitude Phase Signal

slide-48
SLIDE 48

g(t) = ∫

∞ −∞

G(f )e2πiftdt

Inverse Fourier Transform:

G(f ) = ∫

∞ −∞

g(t)e−2πiftdt

Fourier Transform:

slide-49
SLIDE 49

Frequencies

Images are 64x64 pixels. The wave is a cosine (if phase is zero).

Source: Bill Freeman

slide-50
SLIDE 50

Image Amplitude

FT has peaks at spatial frequencies of repeated structure

Source: Deva Ramanan

slide-51
SLIDE 51

Source: Deva Ramanan

slide-52
SLIDE 52

Source: Deva Ramanan

slide-53
SLIDE 53

Source: Deva Ramanan

slide-54
SLIDE 54

Source: Deva Ramanan

Lunar Orbital Image (1966)

Amplitude

slide-55
SLIDE 55

Source: Deva Ramanan

Lunar Orbital Image (1966)

Amplitude Remove Peaks

slide-56
SLIDE 56

Let’s Practice

slide-57
SLIDE 57

Some important Fourier transforms

Image Magnitude DFT Phase DFT

Images are 64x64 pixels.

Source: Bill Freeman

slide-58
SLIDE 58

Some important Fourier transforms

Image Magnitude DFT Phase DFT

Source: Bill Freeman

slide-59
SLIDE 59

Some important Fourier transforms

Image Magnitude DFT Phase DFT

Source: Bill Freeman

slide-60
SLIDE 60

Some important Fourier transforms

Image Magnitude DFT Phase DFT

Source: Bill Freeman

slide-61
SLIDE 61

Image Magnitude DFT

Scale Small image
 details produce content in high spatial frequencies

Source: Bill Freeman

slide-62
SLIDE 62

Some important Fourier transforms

Image Magnitude DFT Phase DFT

Source: Bill Freeman

slide-63
SLIDE 63

Some important Fourier transforms

Image Magnitude DFT Phase DFT

Source: Bill Freeman

slide-64
SLIDE 64

Image Magnitude DFT

Orientation A line transforms to a line oriented perpendicularly to the first.

Source: Bill Freeman

slide-65
SLIDE 65

Game: find the right pairs

A B C 1 2 3

fx(cycles/image pixel size) fx(cycles/image pixel size) fx(cycles/image pixel size)

Images DFT
 magnitude

Source: Bill Freeman

slide-66
SLIDE 66

Magnitude Phase Reconstruction

slide-67
SLIDE 67

Magnitude Phase Reconstruction

Removing High Frequencies

slide-68
SLIDE 68

Magnitude Phase Reconstruction

Removing High Frequencies

slide-69
SLIDE 69

Magnitude Phase Reconstruction

slide-70
SLIDE 70

Magnitude Phase Reconstruction

What will happen?

?

slide-71
SLIDE 71

Magnitude Phase Reconstruction

Removing Low Frequencies

slide-72
SLIDE 72

Magnitude Phase Reconstruction

Removing Low Frequencies

slide-73
SLIDE 73

Compression

slide-74
SLIDE 74

Phase and Magnitude

Each color channel is processed in the same way.

Source: Bill Freeman

slide-75
SLIDE 75

Phase and magnitude

Using random
 amplitude does not
 look good.

Source: Bill Freeman

slide-76
SLIDE 76

Does phase always win?

76

slide-77
SLIDE 77

Convolution

  • Convolution in time space is multiplication in frequency

space:

  • Convolution in frequency space is multiplication in time

space:

g(x) * h(x) = ℱ−1 [ℱ[g(x)] ⋅ ℱ[h(x)]]

ℱ[g(x)] * ℱ[h(x)] = ℱ [g(x) ⋅ h(x)]

Which is more computationally efficient?

slide-78
SLIDE 78

FFT in Practice

Source: Deva Ramanan

slide-79
SLIDE 79

* =

ℱ−1

Image

ℱ

Amplitude

ℱ

Amplitude

=

⊙

Amplitude Gaussian Filter Filter Response

slide-80
SLIDE 80

FT of Gaussian is Gaussian

ℱ

Gaussian (sigma = 2) Gaussian

slide-81
SLIDE 81

Remember box filters?

slide-82
SLIDE 82

* =

ℱ−1

Image

ℱ

Amplitude

ℱ

Amplitude

=

⊙

Amplitude Box Filter Filter Response

slide-83
SLIDE 83

FT of Box Filters

ℱ ℱ

Amplitude Amplitude

slide-84
SLIDE 84

Remember Laplacian filters?

∂2f ∂x2 + ∂2f ∂y2

f

slide-85
SLIDE 85

FT of Laplacian Filters

ℱ

Amplitude Amplitude

ℱ

[−1,2,1]

slide-86
SLIDE 86

Creating an Illusion

slide-87
SLIDE 87

Some visual areas…

From M. Lewicky

Source: Aude Oliva

slide-88
SLIDE 88

Spatial Frequency Contrast

Source: Aude Oliva

slide-89
SLIDE 89

Source: Aude Oliva

slide-90
SLIDE 90

Spatial Frequency Contrast

Source: Aude Oliva

slide-91
SLIDE 91

Hybrid Images

Oliva & Schyns

slide-92
SLIDE 92

= +

slide-93
SLIDE 93

Hybrid Images

slide-94
SLIDE 94

Hybrid Images

slide-95
SLIDE 95

Source: Aude Oliva

slide-96
SLIDE 96

http://cvcl.mit.edu/ hybrid_gallery/gallery.html

Source: Aude Oliva

slide-97
SLIDE 97

Next Class: Image Formation