[PDF] - Recap: Cameras, lenses, and 6.891 calibration Computer Vision and PDF Document

SLIDE 1

1

6.891

Computer Vision and Applications

Prof. Trevor. Darrell

Lecture 2:

– Linear Filters and Convolution (review) – Fourier Transform (review) – Sampling and Aliasing (review)

Readings: F&P Chapter 7.1-7.6

2

Recap: Cameras, lenses, and calibration

Last time:

Camera models
Projection equations
Calibration methods

Images are projections of the 3-D world onto a 2-D plane…

3

Recap: pinhole/perspective

Pinole camera model - box with a small hole in it: Perspective projection:

Forsyth&Ponce

z

y f y z x f x ' ' ' '

4

Recap: Intrinsic parameters

) sin( ) cot( v z y v u z y z x u Г

Г
1

1 ) sin( ) cot( 1 1 z y x v u z v u

Using homogenous coordinates,

we can write this as:

r:
P

K z p

1
5

Recap: Combining extrinsic and intrinsic calibration parameters

Forsyth&Ponce

W C W C W C

O P R P Г

P

O R K z p

W C C W

1
P

K z p

1
P

Μ z p

1
Intrinsic

Extrinsic

6

Other ways to write the same equation

1

. . . . . . . . . 1 1

3 2 1 z y x T T T

W W W m m m z v u

P M z p

1
P

m P m v P m P m u

3

2 3 1

pixel coordinates world coordinates z is in the camera coordinate system, but we can solve for that, since , leading to: z P m

3

1

SLIDE 2

2

7

Recap: Camera calibration

) ( ) (

3 2 3 1

i

i i i

P m v m P m u m

Stack all these measurements of i=1…n points

into a big matrix:

3

2 1 1 1 1 1 1 1

m

m m P v P P u P P v P P u P

T n n T n T T n n T T n T T T T T T

P m P m v P m P m u

3

2 3 1 8

Today

Review of early visual processing

– Linear Filters and Convolution – Fourier Transform – Sampling and Aliasing

You should have been exposed to this material in previous courses; this lecture is just a (quick) review. Administrivia:

– sign-up sheet – introductions

9

What is image filtering?

Modify the pixels in an image based on

some function of a local neighborhood of the pixels.

5 1 4 1 7 1 5 3 10 Local image data 7 Modified image data Some function

10

Linear functions

Simplest: linear filtering.

– Replace each pixel by a linear combination of its neighbors.

The prescription for the linear combination

is called the “convolution kernel”.

5 1 4 1 7 1 5 3 10 0.5 0.5 0 1 0 0 Local image data kernel 7 Modified image data

11

Convolution

l

k

l k g l n k m I g I n m f

,

] , [ ] , [ ] , [

12

Linear filtering (warm-up slide)

riginal

Pixel offset coefficient 1.0

?

SLIDE 3

3

13

Linear filtering (warm-up slide)

riginal

Pixel offset coefficient 1.0 Filtered (no change)

14

Linear filtering

Pixel offset coefficient

riginal

1.0

?

15

shift

Pixel offset coefficient

riginal

1.0 shifted

16

Linear filtering

Pixel offset coefficient

riginal

0.3

?

17

Blurring

Pixel offset coefficient

riginal

0.3 Blurred (filter applied in both dimensions).

18

Blur examples

Pixel offset coefficient 0.3

riginal

8 filtered 2.4 impulse

SLIDE 4

4

19

Blur examples

Pixel offset coefficient 0.3

riginal

8 filtered 4 8 4 impulse edge Pixel offset coefficient 0.3

riginal

8 filtered 2.4

20

Smoothing reduces noise

Generally expect pixels to

“be like” their neighbours

– surfaces turn slowly – relatively few reflectance changes

Generally expect noise

processes to be independent from pixel to pixel

Implies that smoothing

suppresses noise, for appropriate noise models

Scale

– the parameter in the symmetric Gaussian – as this parameter goes up, more pixels are involved in the average – and the image gets more blurred – and noise is more effectively suppressed

21

The effects of smoothing Each row shows smoothing with gaussians of different width; each column shows different realisations of an image of gaussian noise.

22

Linear filtering (warm-up slide)

riginal

2.0

?

1.0

23

Linear filtering (no change)

riginal

2.0 1.0 Filtered (no change)

24

Linear filtering

riginal

2.0 0.33

?

SLIDE 5

5

25

(remember blurring)

Pixel offset coefficient

riginal

0.3 Blurred (filter applied in both dimensions).

26

Linear filtering

riginal

2.0 0.33

?

27

Sharpening

riginal

2.0 0.33 Sharpened

riginal

28

Sharpening example

coefficient

0.3
riginal

8 Sharpened (differences are accentuated; constant areas are left untouched). 11.2 1.7

0.25

8

29

Sharpening

before after

30

Gradients and edges

Points of sharp change

in an image are interesting:

– change in reflectance – change in object – change in illumination – noise

Sometimes called

edge points

General strategy

– linear filters to estimate image gradient – mark points where gradient magnitude is particularly large wrt neighbours (ideally, curves of such points).

SLIDE 6

6

31

Smoothing and Differentiation

Issue: noise

– smooth before differentiation – two convolutions to smooth, then differentiate? – actually, no - we can use a derivative of Gaussian filter

because differentiation is convolution, and convolution is

associative

32

The scale of the smoothing filter affects derivative estimates, and also the semantics of the edges recovered.

1 pixel 3 pixels 7 pixels

33

Oriented filters

Gabor filters (Gaussian modulated harmonics) at different scales and spatial frequencies Top row shows anti-symmetric (or odd) filters, bottom row the symmetric (or even) filters.

34

Linear image transformations

In analyzing images, it’s often useful to

make a change of basis.

Fourier transform, or Wavelet transform, or Steerable pyramid transform

f U F

Vectorized image

transformed image

35

Self-inverting transforms F U F U f

Г
1

Same basis functions are used for the inverse transform U transpose and complex conjugate

36

An example of such a transform: the Fourier transform

discrete domain

Г

Г

1

1 ln

] , [ 1 ] , [

M k N l N M km i

e n m F MN l k f

Inverse transform
Г
1

1 ln

] , [ ] , [

M k N l N M km i

e l k f n m F

Forward transform

SLIDE 7

7

37

To get some sense of what basis elements look like, we plot a basis element --- or rather, its real part --- as a function of x,y for some fixed u, v. We get a function that is constant when (ux+vy) is constant. The magnitude of the vector (u, v) gives a frequency, and its direction gives an orientation. The function is a sinusoid with this frequency along the direction, and constant perpendicular to the direction.

38

Here u and v are larger than in the previous slide.

39

And larger still...

40

Phase and Magnitude

Fourier transform of a real

function is complex

– difficult to plot, visualize – instead, we can think of the phase and magnitude of the transform

Phase is the phase of the

complex transform

Magnitude is the

magnitude of the complex transform

Curious fact

– all natural images have about the same magnitude transform – hence, phase seems to matter, but magnitude largely doesn’t

Demonstration

– Take two pictures, swap the phase transforms, compute the inverse - what does the result look like?

41 42

This is the magnitude transform

f the

cheetah pic

SLIDE 8

8

43

This is the phase transform

f the

cheetah pic

44 45

This is the magnitude transform

f the zebra

pic

46

This is the phase transform

f the zebra

pic

47

Reconstruction with zebra phase, cheetah magnitude

48

Reconstruction with cheetah phase, zebra magnitude

SLIDE 9

9

49

Example image synthesis with fourier basis.

16 images

50

2

51

6

52

18

53

50

54

82

SLIDE 10

10

55

136

56

282

57

538

58

1088

59

2094

60

4052.

4052

SLIDE 11

11

61

8056.

62

15366

63

28743

64

49190.

65

65536.

66

Fourier transform magnitude

SLIDE 12

12

67

Masking out the fundamental and harmonics from periodic pillars

68

Name as many functions as you can that retain that same functional form in the transform domain

69

Forsyth&Ponce

70

Oppenheim, Schafer and Buck, Discrete-time signal processing, Prentice Hall, 1999

Discrete-time, continuous frequency Fourier transform

71

Discrete-time, continuous frequency Fourier transform pairs

Oppenheim, Schafer and Buck, Discrete-time signal processing, Prentice Hall, 1999

73

Bracewell, The Fourier Transform and its Applications, McGraw Hill 1978

Bracewell’s pictorial dictionary of Fourier transform pairs

SLIDE 13

13

74

Bracewell, The Fourier Transform and its Applications, McGraw Hill 1978

Bracewell’s pictorial dictionary of Fourier transform pairs

75

Bracewell, The Fourier Transform and its Applications, McGraw Hill 1978

Bracewell’s pictorial dictionary of Fourier transform pairs

76

Bracewell, The Fourier Transform and its Applications, McGraw Hill 1978

Bracewell’s pictorial dictionary of Fourier transform pairs

77

Why is the Fourier domain particularly useful?

It tells us the effect of linear convolutions.

78

h g f

Fourier transform of convolution

Consider a (circular) convolution of g and h

79

h g f

h

g DFT n m F

]

, [

Fourier transform of convolution

Take DFT of both sides

SLIDE 14

14

80

h g f

h

g DFT n m F

]

, [

Fourier transform of convolution

Write the DFT and convolution explicitly

Г
1

1 ,

] , [ ] , [ ,

M u N v N vn M um i l k

e l k h l v k u g n m F

81

h g f

h

g DFT n m F

]

, [

Fourier transform of convolution

Move the exponent in

Г
1

1 ,

] , [ ] , [ ,

M u N v N vn M um i l k

e l k h l v k u g n m F

Г
1

1 ,

] , [ ] , [

M u N v l k N vn M um i

l k h e l v k u g

82

h g f

h

g DFT n m F

]

, [

Fourier transform of convolution

Change variables in the sum

Г
1

1 ,

] , [ ] , [ ,

M u N v N vn M um i l k

e l k h l v k u g n m F

Г
1

1 ,

] , [ ] , [

M u N v l k N vn M um i

l k h e l v k u g

Г

Г Г

1

1 ,

] , [ ] , [

k M k l N l l k N n l M m k i

l k h e g

83

h g f

h

g DFT n m F

]

, [

Fourier transform of convolution

Perform the DFT (circular boundary conditions)

Г
1

1 ,

] , [ ] , [ ,

M u N v N vn M um i l k

e l k h l v k u g n m F

Г
1

1 ,

] , [ ] , [

M u N v l k N vn M um i

l k h e l v k u g

Г

Г Г

1

1 ,

] , [ ] , [

k M k l N l l k N n l M m k i

l k h e g

Г
l

k N M km i

l k h e n m G

, ln

] , [ ,

84

h g f

h

g DFT n m F

]

, [

Fourier transform of convolution

Perform the other DFT (circular boundary conditions)

Г
1

1 ,

] , [ ] , [ ,

M u N v N vn M um i l k

e l k h l v k u g n m F

Г
1

1 ,

] , [ ] , [

M u N v l k N vn M um i

l k h e l v k u g

Г

Г Г

1

1 ,

] , [ ] , [

k M k l N l l k N n l M m k i

l k h e g

Г
l

k N M km i

l k h e n m G

, ln

] , [ ,

n

m H n m G , ,

85

Analysis of our simple filters

SLIDE 15

15

86

Analysis of our simple filters

riginal

Pixel offset coefficient 1.0 Filtered (no change)

1 ] , [ ] , [

1 1 ln

Г
M

k N l N M km i

e l k f n m F

1.0constant

87

Analysis of our simple filters

Pixel offset coefficient

riginal

1.0 shifted

M m i M k N l N M km i

e e l k f n m F

Г
]

, [ ] , [

1 1 ln

1.0 Constant magnitude, linearly shifted phase

88

Analysis of our simple filters

Pixel offset coefficient

riginal

0.3 blurred

Г
Г
M

m e l k f n m F

M k N l N M km i

cos

2 1 3 1 ] , [ ] , [

1 1 ln

Low-pass filter 1.0

89

Analysis of our simple filters

riginal

2.0 0.33 sharpened

Г
Г
M

m e l k f n m F

M k N l N M km i

cos

2 1 3 1 2 ] , [ ] , [

1 1 ln

high-pass filter 1.0 2.3

90

Sampling and aliasing

91

Sampling in 1D takes a continuous function and replaces it with a vector of values, consisting of the function’s values at a set of sample points. We’ll assume that these sample points are on a regular grid, and can place one at each integer for convenience.

SLIDE 16

16

92

Sampling in 2D does the same thing, only in 2D. We’ll assume that these sample points are on a regular grid, and can place one at each integer point for convenience.

93

A continuous model for a sampled function

We want to be able to

approximate integrals sensibly

Leads to

– the delta function – model on right Sample2D f (x,y)

f (x, y)(x i, y j)

i

i
f (x,y)

(x i, y j)

i

i
94

The Fourier transform of a sampled signal

F Sample2D f (x,y)

F f (x,y)

(x i,y j)

i

i
F f (x,y)
**F

(x i,y j)

i

i
F u i,v j
j
i
95

96 97

Aliasing

Can’t shrink an image by taking every second

pixel

If we do, characteristic errors appear

– In the next few slides – Typically, small phenomena look bigger; fast phenomena can look slower – Common phenomenon

Wagon wheels rolling the wrong way in movies
Checkerboards misrepresented in ray tracing
Striped shirts look funny on colour television

SLIDE 17

17

98

Resample the checkerboard by taking

ne sample at each circle.

In the case of the top left board, new representation is reasonable. Top right also yields a reasonable representation. Bottom left is all black (dubious) and bottom right has checks that are too big.

99

Constructing a pyramid by taking every second pixel leads to layers that badly misrepresent the top layer

100

Smoothing as low-pass filtering

The message of the FT is

that high frequencies lead to trouble with sampling.

Solution: suppress high

frequencies before sampling – multiply the FT of the signal with something that suppresses high frequencies

– or convolve with a low-pass filter

A filter whose FT is a

box is bad, because the filter kernel has infinite support

Common solution: use

a Gaussian

– multiplying FT by Gaussian is equivalent to convolving image with Gaussian.

101

Sampling without smoothing. Top row shows the images, sampled at every second pixel to get the next; bottom row shows the magnitude spectrum of these images.

102

Sampling with smoothing. Top row shows the images. We get the next image by smoothing the image with a Gaussian with sigma 1 pixel, then sampling at every second pixel to get the next; bottom row shows the magnitude spectrum of these images.

103

Sampling with smoothing. Top row shows the images. We get the next image by smoothing the image with a Gaussian with sigma 1.4 pixels, then sampling at every second pixel to get the next; bottom row shows the magnitude spectrum of these images.

SLIDE 18

18

104

Thought problem

Analyze crossed gratings…

105

Thought problem

Analyze crossed gratings…

106

Thought problem

Analyze crossed gratings…

107

Thought problem

Analyze crossed gratings… Where does perceived near horizontal grating come from?

108

What is a good representation for image analysis?

Fourier transform domain tells you “what”

(textural properties), but not “where”.

Pixel domain representation tells you

“where” (pixel location), but not “what”.

Want an image representation that gives