6.891 Computer Vision and Applications Prof. Trevor. Darrell - - PowerPoint PPT Presentation

1

6.891

Computer Vision and Applications

• Prof. Trevor. Darrell

Lecture 2:

– Linear Filters and Convolution (review) – Fourier Transform (review) – Sampling and Aliasing (review)

Readings: F&P Chapter 7.1-7.6

2

Recap: Cameras, lenses, and calibration

Last time:

• Camera models
• Projection equations
• Calibration methods

Images are projections of the 3-D world onto a 2-D plane…

3

Recap: pinhole/perspective

Pinole camera model - box with a small hole in it: Perspective projection:

Forsyth&Ponce

• z

y f y z x f x ' ' ' '

4

Recap: Intrinsic parameters

) sin( ) cot( v z y v u z y z x u Г

• Г
• 1

1 ) sin( ) cot( 1 1 z y x v u z v u

• Using homogenous coordinates,

we can write this as:

• r:
• P

K z p

• 1

5

Recap: Combining extrinsic and intrinsic calibration parameters

Forsyth&Ponce

W C W C W C

O P R P Г

• P

O R K z p

W C C W

• 1
• P

K z p

• 1
• P

Μ z p

• 1
• Intrinsic

Extrinsic

6

Other ways to write the same equation

• 1

. . . . . . . . . 1 1

3 2 1 z y x T T T

W W W m m m z v u

P M z p

• 1
• P

m P m v P m P m u

• 3

2 3 1

pixel coordinates world coordinates z is in the camera coordinate system, but we can solve for that, since , leading to:

z P m

• 3

1

7

Recap: Camera calibration

) ( ) (

3 2 3 1

• i

i i i

P m v m P m u m

• Stack all these measurements of i=1…n points

into a big matrix:

• 3

2 1 1 1 1 1 1 1

• m

m m P v P P u P P v P P u P

T n n T n T T n n T T n T T T T T T

P m P m v P m P m u

• 3

2 3 1

8

Today

Review of early visual processing

– Linear Filters and Convolution – Fourier Transform – Sampling and Aliasing

You should have been exposed to this material in previous courses; this lecture is just a (quick) review. Administrivia:

– sign-up sheet – introductions

9

What is image filtering?

• Modify the pixels in an image based on

some function of a local neighborhood of the pixels.

5 1 4 1 7 1 5 3 10 Local image data 7 Modified image data Some function

10

Linear functions

• Simplest: linear filtering.

– Replace each pixel by a linear combination of its neighbors.

• The prescription for the linear combination

is called the “convolution kernel”.

5 1 4 1 7 1 5 3 10 0.5 0.5 0 1 Local image data kernel 7 Modified image data

11

Convolution

• l

k

l k g l n k m I g I n m f

,

] , [ ] , [ ] , [

12

Linear filtering (warm-up slide)

• riginal

Pixel offset coefficient 1.0

?

13

Linear filtering (warm-up slide)

• riginal

Pixel offset coefficient 1.0 Filtered (no change)

14

Linear filtering

Pixel offset coefficient

• riginal

1.0

?

15

shift

Pixel offset coefficient

• riginal

1.0 shifted

16

Linear filtering

Pixel offset coefficient

• riginal

0.3

?

17

Blurring

Pixel offset coefficient

• riginal

0.3 Blurred (filter applied in both dimensions).

18

Blur examples

Pixel offset coefficient 0.3

• riginal

8 filtered 2.4 impulse

19

Blur examples

Pixel offset coefficient 0.3

• riginal

8 filtered 4 8 4 impulse edge Pixel offset coefficient 0.3

• riginal

8 filtered 2.4

20

Smoothing reduces noise

• Generally expect pixels to

“be like” their neighbours

– surfaces turn slowly – relatively few reflectance changes

• Generally expect noise

processes to be independent from pixel to pixel

• Implies that smoothing

suppresses noise, for appropriate noise models

• Scale

– the parameter in the symmetric Gaussian – as this parameter goes up, more pixels are involved in the average – and the image gets more blurred – and noise is more effectively suppressed

21

The effects of smoothing Each row shows smoothing with gaussians of different width; each column shows different realisations of an image of gaussian noise.

22

Linear filtering (warm-up slide)

• riginal

2.0

?

1.0

23

Linear filtering (no change)

• riginal

2.0 1.0 Filtered (no change)

24

Linear filtering

• riginal

2.0 0.33

?

25

(remember blurring)

Pixel offset coefficient

• riginal

0.3 Blurred (filter applied in both dimensions).

26

Linear filtering

• riginal

2.0 0.33

?

27

Sharpening

• riginal

2.0 0.33 Sharpened

• riginal

28

Sharpening example

coefficient

• 0.3
• riginal

8 Sharpened (differences are accentuated; constant areas are left untouched). 11.2 1.7

• 0.25

8

29

Sharpening

before after

30

Gradients and edges

• Points of sharp change

in an image are interesting:

– change in reflectance – change in object – change in illumination – noise

• Sometimes called

edge points

• General strategy

– linear filters to estimate image gradient – mark points where gradient magnitude is particularly large wrt neighbours (ideally, curves of such points).

31

Smoothing and Differentiation

• Issue: noise

– smooth before differentiation – two convolutions to smooth, then differentiate? – actually, no - we can use a derivative of Gaussian filter

• because differentiation is convolution, and convolution is

associative

32

The scale of the smoothing filter affects derivative estimates, and also the semantics of the edges recovered.

1 pixel 3 pixels 7 pixels

33

Oriented filters

Gabor filters (Gaussian modulated harmonics) at different scales and spatial frequencies Top row shows anti-symmetric (or odd) filters, bottom row the symmetric (or even) filters.

34

Linear image transformations

• In analyzing images, it’s often useful to

make a change of basis.

Fourier transform, or Wavelet transform, or Steerable pyramid transform

f U F

• Vectorized image

transformed image

35

Self-inverting transforms

F U F U f

• Г
• 1

Same basis functions are used for the inverse transform U transpose and complex conjugate

36

An example of such a transform: the Fourier transform

discrete domain

• Г

Г

• 1

1 ln

] , [ 1 ] , [

M k N l N M km i

e n m F MN l k f

• Inverse transform
• Г
• 1

1 ln

] , [ ] , [

M k N l N M km i

e l k f n m F

• Forward transform

37

To get some sense of what basis elements look like, we plot a basis element --- or rather, its real part --- as a function of x,y for some fixed u, v. We get a function that is constant when (ux+vy) is constant. The magnitude of the vector (u, v) gives a frequency, and its direction gives an orientation. The function is a sinusoid with this frequency along the direction, and constant perpendicular to the direction.

38

Here u and v are larger than in the previous slide.

39

And larger still...

40

Phase and Magnitude

• Fourier transform of a real

function is complex

– difficult to plot, visualize – instead, we can think of the phase and magnitude of the transform

• Phase is the phase of the

complex transform

• Magnitude is the

magnitude of the complex transform

• Curious fact

– all natural images have about the same magnitude transform – hence, phase seems to matter, but magnitude largely doesn’t

• Demonstration

– Take two pictures, swap the phase transforms, compute the inverse - what does the result look like?

41

42

This is the magnitude transform

• f the

cheetah pic

43

This is the phase transform

• f the

cheetah pic

44

45

This is the magnitude transform

• f the zebra

pic

46

This is the phase transform

• f the zebra

pic

47

Reconstruction with zebra phase, cheetah magnitude

48

Reconstruction with cheetah phase, zebra magnitude

49

Example image synthesis with fourier basis.

• 16 images

50

2

51

6

52

18

53

50

54

82

55

136

56

282

57

538

58

1088

59

2094

60

4052.

4052

61

8056.

62

15366

63

28743

64

49190.

65

65536.

66

Fourier transform magnitude

67

Masking out the fundamental and harmonics from periodic pillars

68

Name as many functions as you can that retain that same functional form in the transform domain

69

Forsyth&Ponce

70

Oppenheim, Schafer and Buck, Discrete-time signal processing, Prentice Hall, 1999

Discrete-time, continuous frequency Fourier transform

71

Discrete-time, continuous frequency Fourier transform pairs

Oppenheim, Schafer and Buck, Discrete-time signal processing, Prentice Hall, 1999

73

Bracewell, The Fourier Transform and its Applications, McGraw Hill 1978

Bracewell’s pictorial dictionary of Fourier transform pairs

74

Bracewell, The Fourier Transform and its Applications, McGraw Hill 1978

Bracewell’s pictorial dictionary of Fourier transform pairs

75

Bracewell, The Fourier Transform and its Applications, McGraw Hill 1978

Bracewell’s pictorial dictionary of Fourier transform pairs

76

Bracewell, The Fourier Transform and its Applications, McGraw Hill 1978

Bracewell’s pictorial dictionary of Fourier transform pairs

77

Why is the Fourier domain particularly useful?

• It tells us the effect of linear convolutions.

78

h g f

• Fourier transform of convolution

Consider a (circular) convolution of g and h

79

h g f

• h

g DFT n m F

• ]

, [

Fourier transform of convolution

Take DFT of both sides

80

h g f

• h

g DFT n m F

• ]

, [

Fourier transform of convolution

Write the DFT and convolution explicitly

• Г
• 1

1 ,

] , [ ] , [ ,

M u N v N vn M um i l k

e l k h l v k u g n m F

81

h g f

• h

g DFT n m F

• ]

, [

Fourier transform of convolution

Move the exponent in

• Г
• 1

1 ,

] , [ ] , [ ,

M u N v N vn M um i l k

e l k h l v k u g n m F

• Г
• 1

1 ,

] , [ ] , [

M u N v l k N vn M um i

l k h e l v k u g

82

h g f

• h

g DFT n m F

• ]

, [

Fourier transform of convolution

Change variables in the sum

• Г
• 1

1 ,

] , [ ] , [ ,

M u N v N vn M um i l k

e l k h l v k u g n m F

• Г
• 1

1 ,

] , [ ] , [

M u N v l k N vn M um i

l k h e l v k u g

• Г

Г Г

• 1

1 ,

] , [ ] , [

k M k l N l l k N n l M m k i

l k h e g

83

h g f

• h

g DFT n m F

• ]

, [

Fourier transform of convolution

Perform the DFT (circular boundary conditions)

• Г
• 1

1 ,

] , [ ] , [ ,

M u N v N vn M um i l k

e l k h l v k u g n m F

• Г
• 1

1 ,

] , [ ] , [

M u N v l k N vn M um i

l k h e l v k u g

• Г

Г Г

• 1

1 ,

] , [ ] , [

k M k l N l l k N n l M m k i

l k h e g

• Г
• l

k N M km i

l k h e n m G

, ln

] , [ ,

84

h g f

• h

g DFT n m F

• ]

, [

Fourier transform of convolution

Perform the other DFT (circular boundary conditions)

• Г
• 1

1 ,

] , [ ] , [ ,

M u N v N vn M um i l k

e l k h l v k u g n m F

• Г
• 1

1 ,

] , [ ] , [

M u N v l k N vn M um i

l k h e l v k u g

• Г

Г Г

• 1

1 ,

] , [ ] , [

k M k l N l l k N n l M m k i

l k h e g

• Г
• l

k N M km i

l k h e n m G

, ln

] , [ ,

• n

m H n m G , ,

85

Analysis of our simple filters

86

Analysis of our simple filters

• riginal

Pixel offset coefficient 1.0 Filtered (no change)

1 ] , [ ] , [

1 1 ln

• Г
• M

k N l N M km i

e l k f n m F

• 1.0 constant

87

Analysis of our simple filters

Pixel offset coefficient

• riginal

1.0 shifted

M m i M k N l N M km i

e e l k f n m F

• Г
• ]

, [ ] , [

1 1 ln

1.0 Constant magnitude, linearly shifted phase

88

Analysis of our simple filters

Pixel offset coefficient

• riginal

0.3 blurred

• Г
• Г
• M

m e l k f n m F

M k N l N M km i

• cos

2 1 3 1 ] , [ ] , [

1 1 ln

Low-pass filter 1.0

89

Analysis of our simple filters

• riginal

2.0 0.33 sharpened

• Г
• Г
• M

m e l k f n m F

M k N l N M km i

• cos

2 1 3 1 2 ] , [ ] , [

1 1 ln

high-pass filter 1.0 2.3

90

Sampling and aliasing

91

Sampling in 1D takes a continuous function and replaces it with a vector of values, consisting of the function’s values at a set of sample points. We’ll assume that these sample points are on a regular grid, and can place one at each integer for convenience.

92

Sampling in 2D does the same thing, only in 2D. We’ll assume that these sample points are on a regular grid, and can place one at each integer point for convenience.

93

A continuous model for a sampled function

• We want to be able to

approximate integrals sensibly

• Leads to

– the delta function – model on right Sample2D f (x,y)

• f (x, y)(x i, y j)

i

• i
• f (x,y)

(x i, y j)

i

• i

94

The Fourier transform of a sampled signal

F Sample2D f (x,y)

• F f (x, y)

(x i,y j)

i

• i
• F f (x,y)
• **F

(x i, y j)

i

• i
• F u i,v j
• j
• i

95

96

97

Aliasing

• Can’t shrink an image by taking every second

pixel

• If we do, characteristic errors appear

– In the next few slides – Typically, small phenomena look bigger; fast phenomena can look slower – Common phenomenon

• Wagon wheels rolling the wrong way in movies
• Checkerboards misrepresented in ray tracing
• Striped shirts look funny on colour television

98

Resample the checkerboard by taking

• ne sample at each circle.

In the case of the top left board, new representation is reasonable. Top right also yields a reasonable representation. Bottom left is all black (dubious) and bottom right has checks that are too big.

99

Constructing a pyramid by taking every second pixel leads to layers that badly misrepresent the top layer

100

Smoothing as low-pass filtering

• The message of the FT is

that high frequencies lead to trouble with sampling.

• Solution: suppress high

frequencies before sampling – multiply the FT of the signal with something that suppresses high frequencies

– or convolve with a low-pass filter

• A filter whose FT is a

box is bad, because the filter kernel has infinite support

• Common solution: use

a Gaussian

– multiplying FT by Gaussian is equivalent to convolving image with Gaussian.

101

Sampling without smoothing. Top row shows the images, sampled at every second pixel to get the next; bottom row shows the magnitude spectrum of these images.

102

Sampling with smoothing. Top row shows the images. We get the next image by smoothing the image with a Gaussian with sigma 1 pixel, then sampling at every second pixel to get the next; bottom row shows the magnitude spectrum of these images.

103

Sampling with smoothing. Top row shows the images. We get the next image by smoothing the image with a Gaussian with sigma 1.4 pixels, then sampling at every second pixel to get the next; bottom row shows the magnitude spectrum of these images.

104

Thought problem

Analyze crossed gratings…

105

Thought problem

Analyze crossed gratings…

106

Thought problem

Analyze crossed gratings…

107

Thought problem

Analyze crossed gratings… Where does perceived near horizontal grating come from?

108

What is a good representation for image analysis?

• Fourier transform domain tells you “what”

(textural properties), but not “where”.

• Pixel domain representation tells you

“where” (pixel location), but not “what”.

• Want an image representation that gives

you a local description of image events— what is happening where.