Interest Point Detectors & RANSAC Instructor - Simon Lucey - - PowerPoint PPT Presentation

Interest Point Detectors & RANSAC

Instructor - Simon Lucey

16-423 - Designing Computer Vision Apps

Today

• Image Gradients.
• Interest Point Detectors
• Matching Regions.
• RANSAC.

I(x, y) I(x + 1, y + 1) I

Simoncelli & Olshausen 2001

I(x, y) I I(x + 8, y + 8)

Simoncelli & Olshausen 2001

I(x, y) I I(x + 16, y + 16)

Simoncelli & Olshausen 2001

I(x, y) I I(x + 50, y + 50)

Simoncelli & Olshausen 2001

I(x, y) I I(x + 50, y + 50)

Simoncelli & Olshausen 2001

Pixel Coherence

• The pixel coherence assumption demonstrates that pixels

within natural images are heavily correlated with one another within a local neighborhood (e.g., +/- 5 pixels).

• For example,

5

N

3 4 3 2 ? 5 5 4 2

3

Pixel Coherence

• The pixel coherence assumption demonstrates that pixels

within natural images are heavily correlated with one another within a local neighborhood (e.g., +/- 5 pixels).

• For example,

5

N

3 4 3 2 ? 5 5 4 2

3

“Typically assume that pixels within +/- 3, 5 or 7 pixels are highly correlated.”

IGxH x

Estimating Gradients

6

• Images are a discretely sampled representation of a

continuous signal,

(Black)

IGxH x

Estimating Gradients

7

• Images are a discretely sampled representation of a

continuous signal,

IGxH x

(Black)

• What if I want to know given that I have only the

appearance at and ?

I(x0 + ∆x)

Estimating Gradients

8

(Black)

∆x

IGxH x

IFxH x

x0

I(x0)

I(x0 + ∆x) I(x0)

• Again, simply take the Taylor series approximation?

Estimating Gradients

9

(Black)

IGxH x

IFxH x

x0

I(x0 + ∆x) I(x0)

I(x0 + ∆x) ≈ I(x0) + ∂I(x0) ∂x

T

∆x

• Again, simply take the Taylor series approximation?

Estimating Gradients

9

(Black)

IGxH x

IFxH x

x0

I(x0 + ∆x) I(x0)

I(x0 + ∆x) ≈ I(x0) + ∂I(x0) ∂x

T

∆x

∂I(x0) ∂x

• Traditional method for calculating gradients in vision is

through the use of edge filters. (e.g., Sobel, Prewitt).

• where,

∗ ∗

Gradients through Filters

10

“Horizontal” “Vertical”

∂I(x) ∂x = [Ix(x), Iy(x)]T

Iy Ix I

• Traditional method for calculating gradients in vision is

through the use of edge filters. (e.g., Sobel, Prewitt).

• where,

∗ ∗

Gradients through Filters

10

“Horizontal” “Vertical”

∂I(x) ∂x = [Ix(x), Iy(x)]T

Iy Ix I

“Often have to apply a smoothing filter as well.”

Today

• Natural Image Statistics.
• Interest Point Detectors.
• Matching Regions.
• RANSAC.

Interest Point Detectors

Interest Point Detectors

• Interest point detectors, in general, detect corners.
• A dense approach (using all pixels) will be far too slow.
• Matching lines suffers from the “aperture problem” (i.e. has

the line moved along itself?).

• Corners are:-
• relatively sparse
• well-localized
• good for matching
• reasonably cheap to compute
• appear quite robustly
• useful for geometry.

A(∆x, ∆y) = X

(xk,yk)∈N (x,y)

||I(xk, yk) − I(xk + ∆x, yk + ∆y)||2 N (∆x, ∆y)

Interest Point Detectors

• Approach is based on concept of auto-correlation,

14

N(x)

Interest Point Detectors

• Approach is based on concept of auto-correlation,

15

“in vector form” “local neighborhood”

A(∆x) = X

xk∈N (x)

||I(xk) − I(xk + ∆x)||2

A(∆x)

Interest Point Detectors

• Which patches match the auto-correlation responses?

16

A(∆x) = X

xk∈N (x)

||I(xk) − I(xk + ∆x)||2

A(∆x)

Interest Point Detectors

• Which patches match the auto-correlation responses?

16

A(∆x) = X

xk∈N (x)

||I(xk) − I(xk + ∆x)||2

Harris Detector

• Harris & Stephens (1988) proposed,

17

≈ X

i∈N

||I(xi) − I(xi) − ∂I(xi) ∂x

T

∆x||2 ≈ ∆xT H∆x H = X

i∈N

∂I(xi) ∂x ∂I(xi) ∂x

T

Harris Detector

• How can one characterize these three situations?

A(∆x) = X

xk∈N (x)

||I(xk) − I(xk + ∆x)||2

• We can simplify this through a linear approximation,
• where,

Harris Corner Detector

Make decision based on image structure tensor

H = X

i∈N

∂I(xi) ∂x ∂I(xi) ∂x

T

Harris Detector

20

Effects of Scale

• Consider a single row or column of the image

– Plotting intensity as a function of position gives a signal

Source: S. Seitz

Effects of Scale

• Consider a single row or column of the image

– Plotting intensity as a function of position gives a signal

• Where is the edge?

Source: S. Seitz

Search Multiple Scales

f

Source: S. Seitz

Search Multiple Scales

f g

Source: S. Seitz

Search Multiple Scales

f g f * g

Source: S. Seitz

Search Multiple Scales

f g f * g

Source: S. Seitz

Search Multiple Scales

• To find edges, look for peaks in

f g f * g

Source: S. Seitz

SIFT Detector

• Filter with difference of Gaussian filters at increasing scales
• Build image stack (scale space)
• Find extrema in this 3D volume

Coarse to Fine

24

SIFT Detector

Identified Corners Remove those

• n edges

Remove those where contrast is low

Today

• Natural Image Statistics.
• Interest Point Detectors.
• Matching Regions.
• RANSAC.

• Once we have detected local regions, we must then match

them.

Matching Regions

27

image 1 image 2

SSD : sum of square difference Small difference values similar patches

X

[∆x,∆y]T ∈N

||I1(x1 + ∆x, y1 + ∆y) − I2(x2 + ∆x, y2 + ∆y)||2

2

(x1, y1) (x2, y2)

Problems with SSD

• SSD measures are

sensitive to both,

• geometric and,
• photometric variation.
• Common practice to

use descriptors.

28

I1 I2

Planar Affine Patch Assumption

29

“View 1” “View 2”

• A number of technique have been proposed to detect affine

covariant regions (Schmid et al. 2004).

Rotation Invariance

• Estimation of the dominant orientation

– extract gradient orientation – histogram over gradient orientation – peak in this histogram

• Rotate patch in dominant direction

SIFT Descriptor

• 1. Compute image gradients
• 2. Pool into local histograms
• 3. Concatenate histograms
• 4. Normalize histograms

More on this in future lectures.

Why Pool?

Why Pool?

Why Pool?

“average”

Why Pool?

∗

“histogram” “pooling” “blurring”

Why Pool?

∗

“histogram” “pooling” “blurring”

MATLAB Example

• Example in MATLAB,

>> h = fspecial(‘gaussian’,[25,25],3); >> resp = imfilter(im, h);

∗

h resp

MATLAB Example

• Example in MATLAB,

>> h = fspecial(‘gaussian’,[25,25],3); >> resp = imfilter(im, h);

∗

h resp

Other Descriptors

• Since SIFT, a plethora of other descriptors have been

proposed.

• SIFT is sometimes problematic to use in practice as it is
• protected under existing patents.
• In OpenCV alone there are,
• SURF - (patent protected)
• BRIEF
• PHREAK
• See following link for tutorial in OpenCV.
• ORB
• BRISK
• FAST

Other Descriptors

• Since SIFT, a plethora of other descriptors have been

proposed.

• SIFT is sometimes problematic to use in practice as it is
• protected under existing patents.
• In OpenCV alone there are,
• SURF - (patent protected)
• BRIEF
• PHREAK
• See following link for tutorial in OpenCV.
• ORB
• BRISK
• FAST

Today

• Natural Image Statistics.
• Interest Point Detectors.
• Matching Regions.
• RANSAC.

Robust Estimation

• Least squares criterion is

not robust to outliers

• For example, the two
• utliers here cause the fitted

line to be quite wrong.

• One approach to fitting

under these circumstances is to use RANSAC – “Random sampling by consensus”

RANSAC

RANSAC

Fitting a homography with RANSAC

Original images Initial matches Inliers from RANSAC

Things to try in iOS - GPUImage

See - https://github.com/BradLarson/GPUImage for more details.

More to read…

• Prince et al.
• Chapter 13, Sections 2-3.
• Corke et al.
• Chapter 13, Section 3.
• E. Rublee et al. “ORB: an efficient alternative to SIFT or

SURF”, ICCV 2011.