Scale Invariant Interest Point Detection Sanja Fidler CSC420: Intro - - PowerPoint PPT Presentation

▶

scale invariant interest point detection

Scale Invariant Interest Point Detection Sanja Fidler CSC420: Intro - - PowerPoint PPT Presentation

Jan 27, 2023 123 likes •634 views

Image Features: Scale Invariant Interest Point Detection Sanja Fidler CSC420: Intro to Image Understanding 1 / 19 Scale Invariant Interest Points How can we independently select interest points in each image, such that the detections are

slide-1

SLIDE 1

Image Features:

Scale Invariant Interest Point Detection

Sanja Fidler CSC420: Intro to Image Understanding 1 / 19

slide-2

SLIDE 2

Scale Invariant Interest Points

How can we independently select interest points in each image, such that the detections are repeatable across different scales? [Source: K. Grauman, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 2 / 19

slide-3

SLIDE 3

Scale Invariant Interest Points

How can we independently select interest points in each image, such that the detections are repeatable across different scales? [Source: K. Grauman, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 2 / 19

slide-4

SLIDE 4

Scale Invariant Interest Points

How can we independently select interest points in each image, such that the detections are repeatable across different scales? Extract features at a variety of scales, e.g., by using multiple resolutions in a pyramid, and then matching features at the same level. When does this work?

Sanja Fidler CSC420: Intro to Image Understanding 2 / 19

slide-5

SLIDE 5

Scale Invariant Interest Points

How can we independently select interest points in each image, such that the detections are repeatable across different scales? More efficient to extract features that are stable in both location and scale. [Source: K. Grauman, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 2 / 19

slide-6

SLIDE 6

Scale Invariant Interest Points

How can we independently select interest points in each image, such that the detections are repeatable across different scales? Find scale that gives local maxima of a function f in both position and scale. [Source: K. Grauman, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 2 / 19

slide-7

SLIDE 7

Automatic Scale Selection

Function responses for increasing scale (scale signature).

Sanja Fidler CSC420: Intro to Image Understanding 3 / 19

slide-8

SLIDE 8

Automatic Scale Selection

Function responses for increasing scale (scale signature).

Sanja Fidler CSC420: Intro to Image Understanding 3 / 19

slide-9

SLIDE 9

Automatic Scale Selection

Function responses for increasing scale (scale signature).

Sanja Fidler CSC420: Intro to Image Understanding 3 / 19

slide-10

SLIDE 10

Automatic Scale Selection

Function responses for increasing scale (scale signature).

Sanja Fidler CSC420: Intro to Image Understanding 3 / 19

slide-11

SLIDE 11

Automatic Scale Selection

Function responses for increasing scale (scale signature).

Sanja Fidler CSC420: Intro to Image Understanding 3 / 19

slide-12

SLIDE 12

Automatic Scale Selection

Function responses for increasing scale (scale signature).

Sanja Fidler CSC420: Intro to Image Understanding 3 / 19

slide-13

SLIDE 13

Automatic Scale Selection

Function responses for increasing scale (scale signature).

Sanja Fidler CSC420: Intro to Image Understanding 3 / 19

slide-14

SLIDE 14

Automatic Scale Selection

Function responses for increasing scale (scale signature).

Sanja Fidler CSC420: Intro to Image Understanding 3 / 19

slide-15

SLIDE 15

Automatic Scale Selection

Function responses for increasing scale (scale signature).

Sanja Fidler CSC420: Intro to Image Understanding 3 / 19

slide-16

SLIDE 16

Automatic Scale Selection

Function responses for increasing scale (scale signature).

Sanja Fidler CSC420: Intro to Image Understanding 3 / 19

slide-17

SLIDE 17

Automatic Scale Selection

Function responses for increasing scale (scale signature).

Sanja Fidler CSC420: Intro to Image Understanding 3 / 19

slide-18

SLIDE 18

Automatic Scale Selection

Function responses for increasing scale (scale signature).

Sanja Fidler CSC420: Intro to Image Understanding 3 / 19

slide-19

SLIDE 19

What Can the Signature Function Be?

Lindeberg (1998): extrema in the Laplacian of Gaussian (LoG). Lowe (2004) proposed computing a set of sub-octave Difference of Gaussian filters looking for 3D (space+scale) maxima in the resulting structure. [Source: R. Szeliski, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 4 / 19

slide-20

SLIDE 20

What Can the Signature Function Be?

Lindeberg (1998): extrema in the Laplacian of Gaussian (LoG). Lowe (2004) proposed computing a set of sub-octave Difference of Gaussian filters looking for 3D (space+scale) maxima in the resulting structure. [Source: R. Szeliski, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 5 / 19

slide-21

SLIDE 21

Blob Detection – Laplacian of Gaussian

Laplacian of Gaussian: We mentioned it for edge detection ∇2g(x, y, σ) = ∂2g(x, y, σ) ∂x2 + ∂2g(x, y, σ) ∂y 2 , where g is a Gaussian It is a circularly symmetric operator (finds difference in all directions) It can be used for 2D blob detection! How? [Source: K. Grauman, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 6 / 19

slide-22

SLIDE 22

Blob Detection – Laplacian of Gaussian

Laplacian of Gaussian: We mentioned it for edge detection ∇2g(x, y, σ) = − 1 πσ4

1 − x2 + y 2

2σ2

exp− x2+y2

2σ2

It is a circularly symmetric operator (finds difference in all directions) It can be used for 2D blob detection! How? [Source: K. Grauman, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 6 / 19

slide-23

SLIDE 23

Blob Detection – Laplacian of Gaussian

It can be used for 2D blob detection! How? [Source: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 7 / 19

slide-24

SLIDE 24

Blob Detection – Laplacian of Gaussian

It can be used for 2D blob detection! How? [Source: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 7 / 19

slide-25

SLIDE 25

Blob Detection – Laplacian of Gaussian

It can be used for 2D blob detection! How? [Source: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 7 / 19

slide-26

SLIDE 26

Blob Detection – Laplacian of Gaussian

It can be used for 2D blob detection! How? [Source: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 7 / 19

slide-27

SLIDE 27

Blob Detection – Laplacian of Gaussian

It can be used for 2D blob detection! How? [Source: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 7 / 19

slide-28

SLIDE 28

Blob Detection in 2D: Scale Selection

Laplacian of Gaussian = blob detector [Source: B. Leibe, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 8 / 19

slide-29

SLIDE 29

Characteristic Scale

We define the characteristic scale as the scale that produces peak (minimum or maximum) of the Laplacian response [Source: S. Lazebnik]

Sanja Fidler CSC420: Intro to Image Understanding 9 / 19

slide-30

SLIDE 30

Example

[Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 10 / 19

slide-31

SLIDE 31

Example

[Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 10 / 19

slide-32

SLIDE 32

Example

[Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 10 / 19

slide-33

SLIDE 33

Example

[Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 10 / 19

slide-34

SLIDE 34

Example

[Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 10 / 19

slide-35

SLIDE 35

Example

[Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 10 / 19

slide-36

SLIDE 36

Scale Invariant Interest Points

Sanja Fidler CSC420: Intro to Image Understanding 11 / 19

slide-37

SLIDE 37

Example

[Source: S. Lazebnik]

Sanja Fidler CSC420: Intro to Image Understanding 12 / 19

slide-38

SLIDE 38

Blob Detection – Laplacian of Gaussian

That’s nice. But can we do faster? Remember again the Laplacian of Gaussian: ∇2g(x, y, σ) = ∂2g(x, y, σ) ∂x2 +∂2g(x, y, σ) ∂y2 , where g is a Gaussian

Sanja Fidler CSC420: Intro to Image Understanding 13 / 19

slide-39

SLIDE 39

Blob Detection – Laplacian of Gaussian

That’s nice. But can we do faster? Remember again the Laplacian of Gaussian: ∇2g(x, y, σ) = ∂2g(x, y, σ) ∂x2 +∂2g(x, y, σ) ∂y2 , where g is a Gaussian So computing our interest points means two convolutions (one for each derivative) per scale Larger scale (σ), larger the filters (more work for convolution) Can we do it faster?

Sanja Fidler CSC420: Intro to Image Understanding 13 / 19

slide-40

SLIDE 40

Approximate the Laplacian of Gaussian

[Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 14 / 19

slide-41

SLIDE 41

Lowe’s DoG

Lowe (2004) proposed computing a set of sub-octave Difference of Gaussian filters looking for 3D (space+scale) maxima in the resulting structure [Source: R. Szeliski, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 15 / 19

slide-42

SLIDE 42

Lowe’s DoG

First compute a Gaussian image pyramid [Source: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 16 / 19

slide-43

SLIDE 43

Lowe’s DoG

First compute a Gaussian image pyramid Compute Difference of Gaussians [Source: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 16 / 19

slide-44

SLIDE 44

Lowe’s DoG

First compute a Gaussian image pyramid Compute Difference of Gaussians At every scale [Source: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 16 / 19

slide-45

SLIDE 45

Lowe’s DoG

First compute a Gaussian image pyramid Compute Difference of Gaussians At every scale Find local maxima in scale A bit of pruning of bad maxima and we’re done! [Source: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 16 / 19

slide-46

SLIDE 46

Lowe’s DoG

First compute a Gaussian image pyramid Compute Difference of Gaussians At every scale Find local maxima in scale A bit of pruning of bad maxima and we’re done! [Source: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 16 / 19

slide-47

SLIDE 47

Other Interest Point Detectors (Many Good Options!)

Lindeberg: Laplacian of Gaussian Lowe: DoG (typically called the SIFT interest point detector) Mikolajczyk & Schmid: Hessian/Harris-Laplacian/Affine Tuyttelaars & Van Gool: EBR and IBR Matas: MSER Kadir & Brady: Salient Regions

Sanja Fidler CSC420: Intro to Image Understanding 17 / 19

slide-48

SLIDE 48

Summary – Stuff You Should Know

To match the same scene or object under different viewpoint, it’s useful to first detect interest points (keypoints) We looked at these interest point detectors: Harris corner detector: translation and rotation but not scale invariant Scale invariant interest points: Laplacian of Gaussians and Lowe’s DoG Harris’ approach computes I 2

x , I 2 y and IxIy, and blurs each one with a

Gaussian. Denote with: A = g ∗ I 2

x , B = g ∗ (IxIy) and C = g ∗ I 2 y . Then

Mxy = A(x, y) B(x, y) B(x, y) C(x, y)

characterizes the shape of EWSSD for a window

around (x, y). Compute “cornerness” score for each (x, y) as R(x, y) = det(Mxy) − α trace(Mxy)2. Find R(x, y) > threshold and do non-maxima suppression to find corners. Lowe’s approach creates a Gaussian pyramid with s blurring levels per

ctave, computes difference between consecutive levels, and finds local

extrema in space and scale

Sanja Fidler CSC420: Intro to Image Understanding 18 / 19

slide-49

SLIDE 49

Local Descriptors – Next Time

Detection: Identify the interest points. Description: Extract a feature descriptor around each interest point. Matching: Determine correspondence between descriptors in two views. [Source: K. Grauman]

Sanja Fidler CSC420: Intro to Image Understanding 19 / 19