Scale Invariant Interest Point Detection Sanja Fidler CSC420: Intro - - PowerPoint PPT Presentation

▶
scale invariant interest point detection
SMART_READER_LITE
LIVE PREVIEW

Scale Invariant Interest Point Detection Sanja Fidler CSC420: Intro - - PowerPoint PPT Presentation

Dec 02, 2023 17 likes •900 views

Image Features: Scale Invariant Interest Point Detection Sanja Fidler CSC420: Intro to Image Understanding 1 / 30 Our Goal: Matching Objects / Images Our goal is to be able to match an object in different images where the object appears in

slide-1
SLIDE 1

Image Features:

Scale Invariant Interest Point Detection

Sanja Fidler CSC420: Intro to Image Understanding 1 / 30

slide-2
SLIDE 2

Our Goal: Matching Objects / Images

Our goal is to be able to match an object in different images where the

  • bject appears in different scale, rotation, viewpoints, etc. How?

Figure: We want to be able to match these two objects / images

Sanja Fidler CSC420: Intro to Image Understanding 2 / 30

slide-3
SLIDE 3

Our Goal: Matching Objects / Images

Our goal is to be able to match an object in different images where the

  • bject appears in different scale, rotation, viewpoints, etc. How?

Figure: But these shouldn’t be matched!

Sanja Fidler CSC420: Intro to Image Understanding 2 / 30

slide-4
SLIDE 4

Our Goal: Matching Objects / Images

Our goal is to be able to match an object in different images where the

  • bject appears in different scale, rotation, viewpoints, etc. How?

Find interest points on each image Figure: Find some interest points in an image

Sanja Fidler CSC420: Intro to Image Understanding 2 / 30

slide-5
SLIDE 5

Our Goal: Matching Objects / Images

Our goal is to be able to match an object in different images where the

  • bject appears in different scale, rotation, viewpoints, etc. How?

Find interest points on each image Figure: And independently in other images (independently: my algorithm only sees one image at a time – why is this a good idea?)

Sanja Fidler CSC420: Intro to Image Understanding 2 / 30

slide-6
SLIDE 6

Our Goal: Matching Objects / Images

Our goal is to be able to match an object in different images where the

  • bject appears in different scale, rotation, viewpoints, etc. How?

Find interest points on each image Figure: How can we match points??

Sanja Fidler CSC420: Intro to Image Understanding 2 / 30

slide-7
SLIDE 7

Our Goal: Matching Objects / Images

Our goal is to be able to match an object in different images where the

  • bject appears in different scale, rotation, viewpoints, etc. How?

Find interest points on each image Form a vector description of each point. How? Figure: We could match if we took a patch around each point, and describe it with a feature vector (we know how to compare vectors)

Sanja Fidler CSC420: Intro to Image Understanding 2 / 30

slide-8
SLIDE 8

Our Goal: Matching Objects / Images

Our goal is to be able to match an object in different images where the

  • bject appears in different scale, rotation, viewpoints, etc. How?

Find scale invariant interest points on each image Form a vector description of each point. How? Figure: What if my interest point detector tells me the size (scale) of the patch? We are hoping that this “canonical” size somehow reflects size of the object.

Sanja Fidler CSC420: Intro to Image Understanding 2 / 30

slide-9
SLIDE 9

Our Goal: Matching Objects / Images

Our goal is to be able to match an object in different images where the

  • bject appears in different scale, rotation, viewpoints, etc. How?

Find scale invariant interest points on each image Form a vector description of each point. How? Figure: And then we can form our feature vectors with respect to this size (how?)

Sanja Fidler CSC420: Intro to Image Understanding 2 / 30

slide-10
SLIDE 10

Our Goal: Matching Objects / Images

Our goal is to be able to match an object in different images where the

  • bject appears in different scale, rotation, viewpoints, etc. How?

Find scale invariant interest points on each image Form a vector description of each point. How? Matching Figure: Then life is easy: we find the best matches and compute a transformation (scale, rotation, etc) of the object – in the next lecture

Sanja Fidler CSC420: Intro to Image Understanding 2 / 30

slide-11
SLIDE 11

Our Goal: Matching Objects / Images

Our goal is to be able to match an object in different images where the

  • bject appears in different scale, rotation, viewpoints, etc. How?

Find scale invariant interest points on each image Form a vector description of each point. How? Matching Figure: And we are hoping that our feature vectors and our matching algorithm will be able to say that this image does not contain our object!

Sanja Fidler CSC420: Intro to Image Understanding 2 / 30

slide-12
SLIDE 12

Our Goal: Matching Objects / Images

Our goal is to be able to match an object in different images where the

  • bject appears in different scale, rotation, viewpoints, etc. How?

Find scale invariant interest points on each image Let’s do this first! Form a vector description of each point. How? Matching

Sanja Fidler CSC420: Intro to Image Understanding 2 / 30

slide-13
SLIDE 13

Scale Invariant Interest Points

How can we independently select interest points in each image, such that the detections are repeatable across different scales? [Source: K. Grauman, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 3 / 30

slide-14
SLIDE 14

Scale Invariant Interest Points

How can we independently select interest points in each image, such that the detections are repeatable across different scales? [Source: K. Grauman, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 3 / 30

slide-15
SLIDE 15

Scale Invariant Interest Points

How can we independently select interest points in each image, such that the detections are repeatable across different scales? Extract features at a variety of scales, e.g., by using multiple resolutions in a pyramid, and then matching features at the same level. When does this work?

Sanja Fidler CSC420: Intro to Image Understanding 3 / 30

slide-16
SLIDE 16

Scale Invariant Interest Points

How can we independently select interest points in each image, such that the detections are repeatable across different scales? More efficient to extract features that are stable in both location and scale. [Source: K. Grauman, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 3 / 30

slide-17
SLIDE 17

Scale Invariant Interest Points

How can we independently select interest points in each image, such that the detections are repeatable across different scales? Find scale that gives local maxima of a function f in both position and scale. [Source: K. Grauman, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 3 / 30

slide-18
SLIDE 18

Automatic Scale Selection

Function responses for increasing scale (scale signature).

Sanja Fidler CSC420: Intro to Image Understanding 4 / 30

slide-19
SLIDE 19

Automatic Scale Selection

Function responses for increasing scale (scale signature).

Sanja Fidler CSC420: Intro to Image Understanding 4 / 30

slide-20
SLIDE 20

Automatic Scale Selection

Function responses for increasing scale (scale signature).

Sanja Fidler CSC420: Intro to Image Understanding 4 / 30

slide-21
SLIDE 21

Automatic Scale Selection

Function responses for increasing scale (scale signature).

Sanja Fidler CSC420: Intro to Image Understanding 4 / 30

slide-22
SLIDE 22

Automatic Scale Selection

Function responses for increasing scale (scale signature).

Sanja Fidler CSC420: Intro to Image Understanding 4 / 30

slide-23
SLIDE 23

Automatic Scale Selection

Function responses for increasing scale (scale signature).

Sanja Fidler CSC420: Intro to Image Understanding 4 / 30

slide-24
SLIDE 24

Automatic Scale Selection

Function responses for increasing scale (scale signature).

Sanja Fidler CSC420: Intro to Image Understanding 4 / 30

slide-25
SLIDE 25

Automatic Scale Selection

Function responses for increasing scale (scale signature).

Sanja Fidler CSC420: Intro to Image Understanding 4 / 30

slide-26
SLIDE 26

Automatic Scale Selection

Function responses for increasing scale (scale signature).

Sanja Fidler CSC420: Intro to Image Understanding 4 / 30

slide-27
SLIDE 27

Automatic Scale Selection

Function responses for increasing scale (scale signature).

Sanja Fidler CSC420: Intro to Image Understanding 4 / 30

slide-28
SLIDE 28

Automatic Scale Selection

Function responses for increasing scale (scale signature).

Sanja Fidler CSC420: Intro to Image Understanding 4 / 30

slide-29
SLIDE 29

Automatic Scale Selection

Function responses for increasing scale (scale signature).

Sanja Fidler CSC420: Intro to Image Understanding 4 / 30

slide-30
SLIDE 30

What Can the Signature Function Be?

Lindeberg (1998): extrema in the Laplacian of Gaussian (LoG). Lowe (2004) proposed computing a set of sub-octave Difference of Gaussian filters looking for 3D (space+scale) maxima in the resulting structure. [Source: R. Szeliski, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 5 / 30

slide-31
SLIDE 31

What Can the Signature Function Be?

Lindeberg (1998): extrema in the Laplacian of Gaussian (LoG). Lowe (2004) proposed computing a set of sub-octave Difference of Gaussian filters looking for 3D (space+scale) maxima in the resulting structure. [Source: R. Szeliski, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 6 / 30

slide-32
SLIDE 32

Blob Detection – Laplacian of Gaussian

Laplacian of Gaussian: We mentioned it for edge detection ∇2g(x, y, σ) = ∂2g(x, y, σ) ∂x2 + ∂2g(x, y, σ) ∂y 2 , where g is a Gaussian It is a circularly symmetric operator (finds difference in all directions) It can be used for 2D blob detection! How? [Source: K. Grauman, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 7 / 30

slide-33
SLIDE 33

Blob Detection – Laplacian of Gaussian

Laplacian of Gaussian: We mentioned it for edge detection ∇2g(x, y, σ) = − 1 πσ4

  • 1 − x2 + y 2

2σ2

  • exp− x2+y2

2σ2

It is a circularly symmetric operator (finds difference in all directions) It can be used for 2D blob detection! How? [Source: K. Grauman, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 7 / 30

slide-34
SLIDE 34

Blob Detection – Laplacian of Gaussian

It can be used for 2D blob detection! How? [Source: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 8 / 30

slide-35
SLIDE 35

Blob Detection – Laplacian of Gaussian

It can be used for 2D blob detection! How? [Source: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 8 / 30

slide-36
SLIDE 36

Blob Detection – Laplacian of Gaussian

It can be used for 2D blob detection! How? [Source: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 8 / 30

slide-37
SLIDE 37

Blob Detection – Laplacian of Gaussian

It can be used for 2D blob detection! How? [Source: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 8 / 30

slide-38
SLIDE 38

Blob Detection – Laplacian of Gaussian

It can be used for 2D blob detection! How? [Source: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 8 / 30

slide-39
SLIDE 39

Blob Detection in 2D: Scale Selection

Laplacian of Gaussian = blob detector [Source: B. Leibe, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 9 / 30

slide-40
SLIDE 40

Characteristic Scale

We define the characteristic scale as the scale that produces peak (minimum or maximum) of the Laplacian response [Source: S. Lazebnik]

Sanja Fidler CSC420: Intro to Image Understanding 10 / 30

slide-41
SLIDE 41

Example

[Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 11 / 30

slide-42
SLIDE 42

Example

[Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 11 / 30

slide-43
SLIDE 43

Example

[Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 11 / 30

slide-44
SLIDE 44

Example

[Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 11 / 30

slide-45
SLIDE 45

Example

[Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 11 / 30

slide-46
SLIDE 46

Example

[Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 11 / 30

slide-47
SLIDE 47

Scale Invariant Interest Points

Sanja Fidler CSC420: Intro to Image Understanding 12 / 30

slide-48
SLIDE 48

Example

[Source: S. Lazebnik]

Sanja Fidler CSC420: Intro to Image Understanding 13 / 30

slide-49
SLIDE 49

Blob Detection – Laplacian of Gaussian

That’s nice. But can we do faster? Remember again the Laplacian of Gaussian: ∇2g(x, y, σ) = ∂2g(x, y, σ) ∂x2 +∂2g(x, y, σ) ∂y2 , where g is a Gaussian

Sanja Fidler CSC420: Intro to Image Understanding 14 / 30

slide-50
SLIDE 50

Blob Detection – Laplacian of Gaussian

That’s nice. But can we do faster? Remember again the Laplacian of Gaussian: ∇2g(x, y, σ) = ∂2g(x, y, σ) ∂x2 +∂2g(x, y, σ) ∂y2 , where g is a Gaussian So computing our interest points means two convolutions (one for each derivative) per scale Larger scale (σ), larger the filters (more work for convolution) Can we do it faster?

Sanja Fidler CSC420: Intro to Image Understanding 14 / 30

slide-51
SLIDE 51

Approximate the Laplacian of Gaussian

[Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 15 / 30

slide-52
SLIDE 52

Lowe’s DoG

Lowe (2004) proposed computing a set of sub-octave Difference of Gaussian filters looking for 3D (space+scale) maxima in the resulting structure [Source: R. Szeliski, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 16 / 30

slide-53
SLIDE 53

Lowe’s DoG

First compute a Gaussian image pyramid [Source: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 17 / 30

slide-54
SLIDE 54

Lowe’s DoG

First compute a Gaussian image pyramid Compute Difference of Gaussians [Source: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 17 / 30

slide-55
SLIDE 55

Lowe’s DoG

First compute a Gaussian image pyramid Compute Difference of Gaussians At every scale [Source: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 17 / 30

slide-56
SLIDE 56

Lowe’s DoG

First compute a Gaussian image pyramid Compute Difference of Gaussians At every scale Find local maxima in scale A bit of pruning of bad maxima and we’re done! [Source: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 17 / 30

slide-57
SLIDE 57

Lowe’s DoG

First compute a Gaussian image pyramid Compute Difference of Gaussians At every scale Find local maxima in scale A bit of pruning of bad maxima and we’re done! [Source: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 17 / 30

slide-58
SLIDE 58

Examples

Figure: Let’s first try out some synthetic images

Sanja Fidler CSC420: Intro to Image Understanding 18 / 30

slide-59
SLIDE 59

Examples

Figure: Detected interest points (kind of make sense)

Sanja Fidler CSC420: Intro to Image Understanding 18 / 30

slide-60
SLIDE 60

Examples

Figure: Other roundy objects

Sanja Fidler CSC420: Intro to Image Understanding 18 / 30

slide-61
SLIDE 61

Examples

Figure: Detected interest points

Sanja Fidler CSC420: Intro to Image Understanding 18 / 30

slide-62
SLIDE 62

Examples

Figure: Real images

Sanja Fidler CSC420: Intro to Image Understanding 18 / 30

slide-63
SLIDE 63

Examples

Figure: Detected interest points

Sanja Fidler CSC420: Intro to Image Understanding 18 / 30

slide-64
SLIDE 64

Examples

Figure: Detected interest points

Sanja Fidler CSC420: Intro to Image Understanding 18 / 30

slide-65
SLIDE 65

Other Interest Point Detectors (Many Good Options!)

Lindeberg: Laplacian of Gaussian Lowe: DoG (typically called the SIFT interest point detector) Mikolajczyk & Schmid: Hessian/Harris-Laplacian/Affine Tuyttelaars & Van Gool: EBR and IBR Matas: MSER Kadir & Brady: Salient Regions

Sanja Fidler CSC420: Intro to Image Understanding 19 / 30

slide-66
SLIDE 66

Summary – Stuff You Should Know

To match the same scene or object under different viewpoint, it’s useful to first detect interest points (keypoints) We looked at these interest point detectors: Harris corner detector: translation and rotation but not scale invariant Scale invariant interest points: Laplacian of Gaussians and Lowe’s DoG Harris’ approach computes I 2

x , I 2 y and IxIy, and blurs each one with a

  • Gaussian. Denote with: A = g ∗ I 2

x , B = g ∗ (IxIy) and C = g ∗ I 2 y . Then

Mxy = A(x, y) B(x, y) B(x, y) C(x, y)

  • characterizes the shape of EWSSD for a window

around (x, y). Compute “cornerness” score for each (x, y) as R(x, y) = det(Mxy) − α trace(Mxy)2. Find R(x, y) > threshold and do non-maxima suppression to find corners. Lowe’s approach creates a Gaussian pyramid with s blurring levels per

  • ctave, computes difference between consecutive levels, and finds local

extrema in space and scale

Sanja Fidler CSC420: Intro to Image Understanding 20 / 30

slide-67
SLIDE 67

Matching Results Based on SIFT Interest Points

Works pretty well in variety of settings Figure: Lowe’s interest point detector finds scale-invariant points that can be reliably matched across different images. (We will talk about how to do matching soon)

Sanja Fidler CSC420: Intro to Image Understanding 21 / 30

slide-68
SLIDE 68

Matching Results Based on SIFT Interest Points

Works pretty well in variety of settings Figure: Lowe’s interest point detector finds scale-invariant points that can be reliably matched across different images. (We will talk about how to do matching soon)

Sanja Fidler CSC420: Intro to Image Understanding 21 / 30

slide-69
SLIDE 69

Matching Results Based on SIFT Interest Points

Works pretty well in variety of settings Figure: Lowe’s interest point detector finds scale-invariant points that can be reliably matched across different images. (We will talk about how to do matching soon)

Sanja Fidler CSC420: Intro to Image Understanding 21 / 30

slide-70
SLIDE 70

Matching Results Based on SIFT Interest Points

What about in different lighting/weather conditions?

Sanja Fidler CSC420: Intro to Image Understanding 21 / 30

slide-71
SLIDE 71

Matching Results Based on SIFT Interest Points

[Pic from: Y. Verdie, K. M. Yi, P. Fua and V. Lepetit. TILDE: A Temporally Invariant Learned DEtector. CVPR’15]

Fails in very different lighting conditions Figure: Green point(s) are repeatable interest points, red are non-repeatable

Sanja Fidler CSC420: Intro to Image Understanding 21 / 30

slide-72
SLIDE 72

Machine Learning to Find Better Keypoints?

[Pic from: Y. Verdie, K. M. Yi, P. Fua and V. Lepetit. TILDE: A Temporally Invariant Learned DEtector. CVPR’15]

Can we use Machine Learning to detect interest points more reliably? SIFT Learned Interest Point Detector?

Sanja Fidler CSC420: Intro to Image Understanding 22 / 30

slide-73
SLIDE 73

Learning an Interest Point Detector:

  • Y. Verdie, K. M. Yi, P. Fua and V. Lepetit

Large TILDE: A Temporally Invariant Learned DEtector CVPR 2015

Paper: http://infoscience.epfl.ch/record/206786/files/top.pdf Project page & Code: http://cvlab.epfl.ch/research/tilde

Sanja Fidler CSC420: Intro to Image Understanding 23 / 30

slide-74
SLIDE 74

Training Data

What can we use?

Sanja Fidler CSC420: Intro to Image Understanding 24 / 30

slide-75
SLIDE 75

Training Data

What can we use? Data from webcams!

Sanja Fidler CSC420: Intro to Image Understanding 24 / 30

slide-76
SLIDE 76

Training Data

Now that we have training images, how shall we train the detector?

Sanja Fidler CSC420: Intro to Image Understanding 24 / 30

slide-77
SLIDE 77

Training the Detector

Detect e.g. SIFT Interest Points in images across time Keep only those that are repeatable across time. These are our (super reliable) positive training examples. What about negative examples?

Sanja Fidler CSC420: Intro to Image Understanding 25 / 30

slide-78
SLIDE 78

Training the Detector

Detect e.g. SIFT Interest Points in images across time Keep only those that are repeatable across time. These are our (super reliable) positive training examples. What about negative examples? All other points with some distance wrt positive points

Sanja Fidler CSC420: Intro to Image Understanding 25 / 30

slide-79
SLIDE 79

Training the Detector

Detect e.g. SIFT Interest Points in images across time Keep only those that are repeatable across time. These are our (super reliable) positive training examples. What about negative examples? All other points with some distance wrt positive points Take a patch around each point, extract some features on it. Train a classifier or a regressor

Sanja Fidler CSC420: Intro to Image Understanding 25 / 30

slide-80
SLIDE 80

Trained Filters

Remember from the lecture where we trained a classifier to detect edges: If we train a linear classifier on a patch, it can be seen as a filter

Sanja Fidler CSC420: Intro to Image Understanding 26 / 30

slide-81
SLIDE 81

Trained Filters

Remember from the lecture where we trained a classifier to detect edges: If we train a linear classifier on a patch, it can be seen as a filter Tiny lesson learned: Sometime our intermediate results (filters in this case) don’t look interpretable at all, but they still do the job

Sanja Fidler CSC420: Intro to Image Understanding 26 / 30

slide-82
SLIDE 82

Using the Learned Interest Point Detector

Now that we trained our detector, how can we use it on new images?

Sanja Fidler CSC420: Intro to Image Understanding 27 / 30

slide-83
SLIDE 83

Using the Learned Interest Point Detector

Apply our filter on each image patch (convolution, if it’s a linear classifier)

Sanja Fidler CSC420: Intro to Image Understanding 27 / 30

slide-84
SLIDE 84

Using the Learned Interest Point Detector

Apply our filter on each image patch (convolution, if it’s a linear classifier) This has response everywhere. How can we find the actual interest points?

Sanja Fidler CSC420: Intro to Image Understanding 27 / 30

slide-85
SLIDE 85

Using the Learned Interest Point Detector

Apply our filter on each image patch (convolution, if it’s a linear classifier) This has response everywhere. How can we find the actual interest points? Non-maxima suppression (keep only points that are local maxima)

Sanja Fidler CSC420: Intro to Image Understanding 27 / 30

slide-86
SLIDE 86

Qualitative Results (nice looking pictures)

Visually check how well we can now match with new interest points SIFT, SURF are hand-designed interest point detectors FAST is trained to detect corners fast: First employs a slow method to detect corners, then trains decision trees to detect them really fast

[E. Rosten and T. Drummond. Machine Learning for HighSpeed Corner Detection. ECCV 2006 Paper: http://www.edwardrosten.com/work/rosten_2006_machine.pdf] Sanja Fidler CSC420: Intro to Image Understanding 28 / 30

slide-87
SLIDE 87

Quantitative Results (performance numbers)

Every method is much more convincing if it shows quantitative performance. If there are baselines for the problem, rule is: the more baselines the better. The more datasets, the better

Sanja Fidler CSC420: Intro to Image Understanding 29 / 30

slide-88
SLIDE 88

Local Descriptors – Next Time

Detection: Identify the interest points. Description: Extract a feature descriptor around each interest point. Matching: Determine correspondence between descriptors in two views. [Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 30 / 30