Image Features Sanja Fidler CSC420: Intro to Image Understanding 1 - - PowerPoint PPT Presentation

▶
image features
SMART_READER_LITE
LIVE PREVIEW

Image Features Sanja Fidler CSC420: Intro to Image Understanding 1 - - PowerPoint PPT Presentation

May 20, 2023 239 likes •944 views

Image Features Sanja Fidler CSC420: Intro to Image Understanding 1 / 1 Image Features Image features are useful descriptions of local or global image properties designed (or learned!) to accomplish a certain task You may want to choose di ff

slide-1
SLIDE 1

Image Features

Sanja Fidler CSC420: Intro to Image Understanding 1 / 1

slide-2
SLIDE 2

Image Features

Image features are useful descriptions of local or global image properties designed (or learned!) to accomplish a certain task You may want to choose different features for different tasks Depending on the problem we need to typically answer three questions:

Where to extract image features? What to extract (what’s the content of the feature)? How to use them for your task, e.g., how to match them?

Sanja Fidler CSC420: Intro to Image Understanding 2 / 1

slide-3
SLIDE 3

Image Features

Let’s watch a video clip

Sanja Fidler CSC420: Intro to Image Understanding 3 / 1

slide-4
SLIDE 4

Image Features

Where is the movie taking place?

Sanja Fidler CSC420: Intro to Image Understanding 4 / 1

slide-5
SLIDE 5

Image Features

Where is the movie taking place?

Sanja Fidler CSC420: Intro to Image Understanding 4 / 1

slide-6
SLIDE 6

Image Features

Where is the movie taking place?

Sanja Fidler CSC420: Intro to Image Understanding 4 / 1

slide-7
SLIDE 7

Image Features

Where is the movie taking place? We matched in: Distinctive locations: keypoints Distinctive features: descriptors

Sanja Fidler CSC420: Intro to Image Understanding 4 / 1

slide-8
SLIDE 8

Image Features

Tracking: Where to did the scene/actors move?

Sanja Fidler CSC420: Intro to Image Understanding 5 / 1

slide-9
SLIDE 9

Image Features

Tracking: Where to did the scene/actors move? We matched: Quite distinctive locations Quite distinctive features

Sanja Fidler CSC420: Intro to Image Understanding 5 / 1

slide-10
SLIDE 10

Image Features

A shot in a movie is a clip with a coherent camera (no sudden viewpoint changes)

Sanja Fidler CSC420: Intro to Image Understanding 6 / 1

slide-11
SLIDE 11

Image Features

A shot in a movie is a clip with a coherent camera (no sudden viewpoint changes) We matched: Globally – one descriptor for full image Descriptor can be simple, e.g. color

Sanja Fidler CSC420: Intro to Image Understanding 6 / 1

slide-12
SLIDE 12

Image Features

How could we tell which type of scene it is?

Sanja Fidler CSC420: Intro to Image Understanding 7 / 1

slide-13
SLIDE 13

Image Features

How could we tell which type of scene it is? We matched: Globally – one descriptor for full image (?) More complex descriptor: color, gradients, “deep” features (learned), etc

Sanja Fidler CSC420: Intro to Image Understanding 7 / 1

slide-14
SLIDE 14

Image Features

How would we solve this?

Sanja Fidler CSC420: Intro to Image Understanding 8 / 1

slide-15
SLIDE 15

Image Features

How would we solve this? We matched: One descriptor for full patch Descriptor can be simple, e.g. color

Sanja Fidler CSC420: Intro to Image Understanding 8 / 1

slide-16
SLIDE 16

Image Features

How would we solve this?

Sanja Fidler CSC420: Intro to Image Understanding 8 / 1

slide-17
SLIDE 17

Image Features

How would we solve this? We matched: At each location Compared pixel values

Sanja Fidler CSC420: Intro to Image Understanding 8 / 1

slide-18
SLIDE 18

Image Features

How would we solve this?

Sanja Fidler CSC420: Intro to Image Understanding 8 / 1

slide-19
SLIDE 19

Image Features

How would we solve this? We matched: Distinctive locations Distinctive features Affine invariant

Sanja Fidler CSC420: Intro to Image Understanding 8 / 1

slide-20
SLIDE 20

Image Features

How would we solve this?

Sanja Fidler CSC420: Intro to Image Understanding 8 / 1

slide-21
SLIDE 21

Image Features

Detection: Where to extract image features?

“Interesting” locations (keypoints, interesting regions) In each location (densely)

Description: What to extract?

What’s the spatial scope of the feature? What’s the content of the feature?

Matching: How to match them?

Sanja Fidler CSC420: Intro to Image Understanding 9 / 1

slide-22
SLIDE 22

Image Features

Detection: Where to extract image features?

“Interesting” locations (keypoints) TODAY In each location (densely)

Description: What to extract?

What’s the spatial scope of the feature? What’s the content of the feature?

Matching: How to match them?

Sanja Fidler CSC420: Intro to Image Understanding 9 / 1

slide-23
SLIDE 23

Image Features:

Interest Point (Keypoint) Detection

Sanja Fidler CSC420: Intro to Image Understanding 10 / 1

slide-24
SLIDE 24

Application Example: Image Stitching

[Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 11 / 1

slide-25
SLIDE 25

Local Features

Detection: Identify the interest points. Description: Extract feature vector descriptor around each interest point. Matching: Determine correspondence between descriptors in two views. [Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 12 / 1

slide-26
SLIDE 26

Goal: Repeatability of the Interest Point Operator

Our goal is to detect (at least some of) the same points in both images We have to be able to run the detection procedure independently per image We need to generate enough points to increase our chances of detecting matching points We shouldn’t generate too many or our matching algorithm will be too slow Figure: Too few keypoints ! little chance to find the true matches [Source: K. Grauman, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 13 / 1

slide-27
SLIDE 27

Goal: Distinctiveness of the Keypoints

We want to be able to reliably determine which point goes with which. [Source: K. Grauman, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 14 / 1

slide-28
SLIDE 28

What Points to Choose?

[Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 15 / 1

slide-29
SLIDE 29

What Points to Choose?

Textureless patches are nearly impossible to localize. [Adopted from: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 16 / 1

slide-30
SLIDE 30

What Points to Choose?

Textureless patches are nearly impossible to localize. Patches with large contrast changes (gradients) are easier to localize. [Adopted from: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 16 / 1

slide-31
SLIDE 31

What Points to Choose?

Textureless patches are nearly impossible to localize. Patches with large contrast changes (gradients) are easier to localize. But straight line segments cannot be localized on lines segments with the same orientation (aperture problem) [Adopted from: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 16 / 1

slide-32
SLIDE 32

What Points to Choose?

Textureless patches are nearly impossible to localize. Patches with large contrast changes (gradients) are easier to localize. But straight line segments cannot be localized on lines segments with the same orientation (aperture problem) Gradients in at least two different orientations are easiest, e.g., corners! [Adopted from: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 16 / 1

slide-33
SLIDE 33

What Points to Choose?

Textureless patches are nearly impossible to localize. Patches with large contrast changes (gradients) are easier to localize. But straight line segments cannot be localized on lines segments with the same orientation (aperture problem) Gradients in at least two different orientations are easiest, e.g., corners! [Adopted from: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 16 / 1

slide-34
SLIDE 34

Interest Points: Corners

How can we find corners in an image?

Sanja Fidler CSC420: Intro to Image Understanding 17 / 1

slide-35
SLIDE 35

Interest Points: Corners

We should easily recognize the point by looking through a small window. Shifting a window in any direction should give a large change in intensity. Figure: (left) flat region: no change in all directions, (center) edge: no change along the edge direction, (right) corner: significant change in all directions [Source: Alyosha Efros, Darya Frolova, Denis Simakov]

Sanja Fidler CSC420: Intro to Image Understanding 18 / 1

slide-36
SLIDE 36

Interest Points: Corners

Compare two image patches using (weighted) summed square difference Measures change in appearance of window w(x, y) for the shift [Source: J. Hays]

Sanja Fidler CSC420: Intro to Image Understanding 19 / 1

slide-37
SLIDE 37

Interest Points: Corners

Compare two image patches using (weighted) summed square difference Measures change in appearance of window w(x, y) for the shift [Source: J. Hays]

Sanja Fidler CSC420: Intro to Image Understanding 19 / 1

slide-38
SLIDE 38

Interest Points: Corners

Compare two image patches using (weighted) summed square difference Measures change in appearance of window w(x, y) for the shift [Source: J. Hays]

Sanja Fidler CSC420: Intro to Image Understanding 19 / 1

slide-39
SLIDE 39

Interest Points: Corners

Let’s look at EWSSD We want to find out how this function behaves for small shifts Remember our goal to detect corners:

Sanja Fidler CSC420: Intro to Image Understanding 20 / 1

slide-40
SLIDE 40

Interest Points: Corners

Using a simple first-order Taylor Series expansion: I(x + u, y + v) ⇡ I(x, y) + u · ∂I ∂x (x, y) + v · ∂I ∂y (x, y) And plugging it in our expression for EWSSD: EWSSD(u, v) = X

x

X

y

w(x, y) ⇣ I(x + u, y + v) I(x, y) ⌘2 ⇡ X

x

X

y

w(x, y) ⇣ I(x, y) + u · Ix + v · Iy I(x, y) ⌘2 = X

x

X

y

w(x, y) ⇣ u2I 2

x + 2u · v · Ix · Iy + v 2 I 2 y

⌘ = X

x

X

y

w(x, y) · ⇥u v⇤  I 2

x

Ix · Iy Ix · Iy I 2

y

u v

  • Sanja Fidler

CSC420: Intro to Image Understanding 21 / 1

slide-41
SLIDE 41

Interest Points: Corners

Since (u, v) doesn’t depend on (x, y) we can rewriting it slightly: EWSSD(u, v) = X

x

X

y

w(x, y) ⇥u v⇤  I 2

x

Ix · Iy Ix · Iy I 2

y

 u v

  • =

⇥ u v ⇤ ⇣ X

x

X

y

w(x, y)  I 2

x

Ix · Iy Ix · Iy I 2

y

⌘ | {z }

Let’s denotes this with M

u v

  • =

⇥u v⇤ M u v

  • M is a 2 ⇥ 2 second moment matrix computed from image gradients:

M = X

x

X

y

w(x, y)  I 2

x

Ix · Iy Ix · Iy I 2

y

  • Sanja Fidler

CSC420: Intro to Image Understanding 22 / 1

slide-42
SLIDE 42

How Do I Compute M?

Let’s say I have this image

Sanja Fidler CSC420: Intro to Image Understanding 23 / 1

slide-43
SLIDE 43

How Do I Compute M?

Let’s say I have this image I need to compute a 2 ⇥ 2 second moment matrix in each image location

Sanja Fidler CSC420: Intro to Image Understanding 23 / 1

slide-44
SLIDE 44

How Do I Compute M?

Let’s say I have this image I need to compute a 2 ⇥ 2 second moment matrix in each image location In a particular location I need to compute M as a weighted average of gradients in a window

Sanja Fidler CSC420: Intro to Image Understanding 23 / 1

slide-45
SLIDE 45

How Do I Compute M?

Let’s say I have this image I need to compute a 2 ⇥ 2 second moment matrix in each image location In a particular location I need to compute M as a weighted average of gradients in a window I can do this efficiently by computing three matrices, I 2

x , I 2 y and Ix · Iy, and

convolving each one with a filter, e.g. a box or Gaussian filter

Sanja Fidler CSC420: Intro to Image Understanding 23 / 1

slide-46
SLIDE 46

Interest Points: Corners

We now have M computed in each image location Our EWSSD is a quadratic function where M implies its shape EWSSD(u, v) = ⇥ u v ⇤ M u v

  • M =

X

x

X

y

w(x, y)  I 2

x

Ix · Iy Ix · Iy I 2

y

  • [Source: J. Hays]

Sanja Fidler CSC420: Intro to Image Understanding 24 / 1

slide-47
SLIDE 47

Interest Points: Corners

Let’s take a horizontal “slice” of EWSSD(u, v): ⇥ u v ⇤ M u v

  • = const

This is the equation of an ellipse

Figure: Different ellipses obtain by different horizontal “slices”

Sanja Fidler CSC420: Intro to Image Understanding 25 / 1

slide-48
SLIDE 48

Interest Points: Corners

Let’s take a horizontal “slice” of EWSSD(u, v): ⇥ u v ⇤ M u v

  • = const

This is the equation of an ellipse

Figure: Different ellipses obtain by different horizontal “slices”

Sanja Fidler CSC420: Intro to Image Understanding 25 / 1

slide-49
SLIDE 49

Interest Points: Corners

Our matrix M is symmetric: M = X

x

X

y

w(x, y)  I 2

x

Ix · Iy Ix · Iy I 2

y

  • And thus we can diagonalize it

(in Matlab: [V,D] = eig(M)): M = V λ1 λ2

  • V −1

Columns of V are major and minor axes of ellipse, λ−1/2 are radius [Source: J. Hays]

Sanja Fidler CSC420: Intro to Image Understanding 26 / 1

slide-50
SLIDE 50

Interest Points: Corners

Columns of V are principal directions λ1, λ2 are principal curvatures [Source: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 27 / 1

slide-51
SLIDE 51

Interest Points: Corners

The eigenvalues of M (λ1, λ2) reveal the amount of intensity change in the two principal orthogonal gradient directions in the window [Source: R. Szeliski, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 28 / 1

slide-52
SLIDE 52

Interest Points: Corners

How do the ellipses look like for this image? [Source: J. Hays]

Sanja Fidler CSC420: Intro to Image Understanding 29 / 1

slide-53
SLIDE 53

Interest Points: Corners

How do the ellipses look like for this image? [Source: J. Hays]

Sanja Fidler CSC420: Intro to Image Understanding 29 / 1

slide-54
SLIDE 54

Interest Points: Corners

[Source: K. Grauman, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 30 / 1

slide-55
SLIDE 55

Interest Points: Criteria to Find Corners

Harris and Stephens, ’88, is rotationally invariant and downweighs edge-like features where λ1 λ0 R = det(M) α · trace(M)2 = λ0λ1 α(λ0 + λ1)2 Why go via det and trace and not use a criteria with λ? α a constant (0.04 to 0.06) The corresponding detector is called Harris corner detector

Sanja Fidler CSC420: Intro to Image Understanding 31 / 1

slide-56
SLIDE 56

Interest Points: Criteria to Find Corners

Harris and Stephens, 88 is rotationally invariant and downweighs edge-like features where λ1 λ0 R = det(M) αtrace(M)2 = λ0λ1 α(λ0 + λ1)2 Shi and Tomasi, 94 proposed the smallest eigenvalue of A, i.e., λ−1/2 . Triggs, 04 suggested λ0 αλ1 also reduces the response at 1D edges, where aliasing errors sometimes inflate the smaller eigenvalue Brown et al, 05 use the harmonic mean det(A) trace(A) = λ0λ1 λ0 + λ1 [Source R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 32 / 1

slide-57
SLIDE 57

Harris Corner detector

1

Compute gradients Ix and Iy

2

Compute I 2

x , I 2 y , Ix · Iy

3

Average (Gaussian) ! gives M

4

Compute R = det(M) α trace(M)2 for each image window (cornerness score)

5

Find points with large R (R > threshold).

6

Take only points of local maxima, i.e., perform non-maximum suppression

Sanja Fidler CSC420: Intro to Image Understanding 33 / 1

slide-58
SLIDE 58

Example

[Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 34 / 1

slide-59
SLIDE 59

1) Compute Cornerness

[Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 35 / 1

slide-60
SLIDE 60

2) Find High Response

[Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 36 / 1

slide-61
SLIDE 61

3) Non-maxima Suppresion

[Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 37 / 1

slide-62
SLIDE 62

Results

[Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 38 / 1

slide-63
SLIDE 63

Another Example

[Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 39 / 1

slide-64
SLIDE 64

Cornerness

[Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 40 / 1

slide-65
SLIDE 65

Interest Points

[Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 41 / 1

slide-66
SLIDE 66

Interest Points – Ideal Properties?

We want corner locations to be invariant to photometric transformations and covariant to geometric transformations Invariance : Image is transformed and corner locations do not change Covariance : If we have two transformed versions of the same image, features should be detected in corresponding locations [Source: J. Hays]

Sanja Fidler CSC420: Intro to Image Understanding 42 / 1

slide-67
SLIDE 67

Properties of Harris Corner Detector

Shift? Harris corner detector is shift-covariant (our window functions shift) [Source: J. Hays]

Sanja Fidler CSC420: Intro to Image Understanding 43 / 1

slide-68
SLIDE 68

Properties of Harris Corner Detector

Rotation? Second moment ellipse rotates but its shape (i.e. eigenvalues) remains the same Harris corner detector is rotation-covariant [Source: J. Hays]

Sanja Fidler CSC420: Intro to Image Understanding 44 / 1

slide-69
SLIDE 69

Properties of Harris Corner Detector

Scale? Corner location is not scale invariant/covariant! [Source: J. Hays]

Sanja Fidler CSC420: Intro to Image Understanding 45 / 1

slide-70
SLIDE 70

Next Time

Can we also define keypoints that are shift, rotation and scale invariant/covariant? What should be our description around keypoint?

Sanja Fidler CSC420: Intro to Image Understanding 46 / 1