Lecture 2 Local Interest Point Detectors Lin ZHANG, PhD School of - - PowerPoint PPT Presentation

Lin ZHANG, SSE, 2020

Lecture 2 Local Interest Point Detectors

Lin ZHANG, PhD School of Software Engineering Tongji University Spring 2020

Lin ZHANG, SSE, 2020

Content

• Local Invariant Features
• Motivation
• Requirements
• Invariance
• Harris Corner Detector
• Scale Invariant Point Detection
• Automatic scale selection
• Laplacian‐of‐Gaussian detector
• Difference‐of‐Gaussian detector

Lin ZHANG, SSE, 2020

Motivation

Lin ZHANG, SSE, 2020

Motivation

Application: Image Matching

Lin ZHANG, SSE, 2020

Motivation

Application: Image Matching

Lin ZHANG, SSE, 2020

Motivation

Application: Image Matching

NASA Mars Rover Images

Lin ZHANG, SSE, 2020

Motivation

Application: Image Matching

NASA Mars Rover images with SIFT matches (Look for tiny colored squares)

Lin ZHANG, SSE, 2020

Motivation

• Panorama stitching
• We have two images – how do we combine them?

Lin ZHANG, SSE, 2020

Motivation

• Panorama stitching
• We have two images – how do we combine them?

Lin ZHANG, SSE, 2020

Motivation

• Panorama stitching
• We have two images – how do we combine them?

Lin ZHANG, SSE, 2020

Motivation

• Panorama stitching
• We have two images – how do we combine them?

Lin ZHANG, SSE, 2020

General Approach for Image Matching

Source: B. Leibe

Lin ZHANG, SSE, 2020

Characteristics of Good Features

• Repeatability
• The same feature can be found in several images despite geometric and

photometric transformations

• Saliency
• Each feature has a distinctive description
• Compactness and efficiency
• Many fewer features than image pixels
• Locality
• A feature occupies a relatively small area of the image; robust to clutter and
• cclusion

Lin ZHANG, SSE, 2020

Invariance: Geometric Transformations

Lin ZHANG, SSE, 2020

Level of Geometric Invariance

Lin ZHANG, SSE, 2020

Invariance: Photometric Transformations

Lin ZHANG, SSE, 2020

Applications

Feature points are used for:

• Motion tracking
• Image alignment
• 3D reconstruction
• Object recognition
• Indexing and database retrieval
• Robot navigation

Lin ZHANG, SSE, 2020

Content

• Local Invariant Features
• Motivation
• Requirements
• Invariance
• Harris Corner Detector
• Scale Invariant Point Detection
• Automatic scale selection
• Laplacian‐of‐Gaussian detector
• Difference‐of‐Gaussian detector

Lin ZHANG, SSE, 2020

Finding Corners

My office, 5:30PM, Sep. 18, 2011

Lin ZHANG, SSE, 2020

Finding Corners

• Key property: in the region around a corner,

image gradient has two or more dominant directions

• Corners are repeatable and distinctive
• C. Harris and M. Stephens. “A Combined Corner and Edge Detector.“

Proceedings of the 4th Alvey Vision Conference: pages 147—151, 1988.

Lin ZHANG, SSE, 2020

Corner Detection: Basic Idea

• We should easily recognize the point by looking through

a small window

• Shifting a window in any direction should give a large

change in intensity

“edge”: no change along the edge direction “corner”: significant change in all directions “flat” region: no change in all directions

Lin ZHANG, SSE, 2020

‐ Demo of a point + with well distinguished neighborhood. Moving the window in any direction will result in a large intensity change.

+

> 0

Harris Detector: Basic Idea

Difference = 3

Lin ZHANG, SSE, 2020

• +

> 0

Harris Detector: Basic Idea

Demo of a point + with well distinguished neighborhood. Moving the window in any direction will result in a large intensity change. Difference = 2

Lin ZHANG, SSE, 2020

• +

> 0

Harris Detector: Basic Idea

Demo of a point + with well distinguished neighborhood. Moving the window in any direction will result in a large intensity change. Difference = 5

Lin ZHANG, SSE, 2020

• +

> 0

Harris Detector: Basic Idea

Demo of a point + with well distinguished neighborhood. Moving the window in any direction will result in a large intensity change. Difference = 2

Lin ZHANG, SSE, 2020

• +

> 0

Harris Detector: Basic Idea

Demo of a point + with well distinguished neighborhood. Moving the window in any direction will result in a large intensity change. Difference = 3

Lin ZHANG, SSE, 2020

• +

> 0

Harris Detector: Basic Idea

Demo of a point + with well distinguished neighborhood. Moving the window in any direction will result in a large intensity change. Difference = 2

Lin ZHANG, SSE, 2020

• +

> 0

Harris Detector: Basic Idea

Demo of a point + with well distinguished neighborhood. Moving the window in any direction will result in a large intensity change. Difference = 3

Lin ZHANG, SSE, 2020

• +

> 0

Harris Detector: Basic Idea

Demo of a point + with well distinguished neighborhood. Moving the window in any direction will result in a large intensity change. Difference = 2

Lin ZHANG, SSE, 2020

Harris Corner Detection: Mathematics

Change in appearance of a local patch (defined by a window w) centered at p for the shift :

Intensity Shifted intensity Window function

• r

Window function w= Gaussian 1 in window, 0 outside

2 ( , )

( , ) ( ( , ) ( , ))

i i

w i i i i x y w

S x y f x y f x x y y



     



 

( , ) x y  

Lin ZHANG, SSE, 2020

 

2 ( , ) 2 ( , ) 2 ( , )

( , ) ( ( , ) ( , )) ( , ) ( , ) ( , ) ( , ) , ( , ) ( , ) , ( , ) , (

i i i i i i

w i i i i x y w i i i i i i i i x y w i i i i x y w i i

S x y f x y f x x y y x f x y f x y f x y f x y y x y x f x y f x y y x y f x y x x y f x

  

                                                          

  

       

 

( , )

( , ) ( , ) , )

i i

i i i i i i x y w

x f x y f x y y y x y y x x y M y



                                                 



      (Due to )

2 T

 u u u

Harris Corner Detection: Mathematics

(1) (2)

Lin ZHANG, SSE, 2020

2 ( , ) ( , ) 2 ( , ) ( , )

( , ) ( , ) ( , ) ( , ) ( , ) ( , )

i i i i i i i i

i i i i i i x y w x y w i i i i i i x y w x y w

M f x y f x y f x y x x y f x y f x y f x y x y y

   

                                                      

   

• It is real symmetric

Harris Corner Detection

Lin ZHANG, SSE, 2020

actually is the ellipse equation.

 

( , ) , x S x y x y M y              ( , ) 1 S x y   

2 ( , ) ( , ) 2 ( , ) ( , )

( ) ( ) ( ) ( )

i i i i i i i i

x x y x y w x y w x y y x y w x y w

I I I M I I I

   

            

   

The shape of the ellipse is determined by M.

Harris Corner Detection

Lin ZHANG, SSE, 2020

The “cornerness” of the window w is reflected in M Suppose there are two local windows w1 and w2; consider the cases when the moving of the two windows leads to the intensity change equals to 1. The moving vector of each window satisfies the ellipse equation. Thus,

 

, x y  

Which window has higher cornerness?

Harris Corner Detection

 

2

, 1 x x y M y           

For w2,

x  y 

 

1

, 1 x x y M y           

For w1,

x  y 

Lin ZHANG, SSE, 2020

R R M       

 2 1 1

 

The axis lengths of the ellipse are determined by the eigenvalues and the orientation is determined by R

direction of the slowest change direction of the fastest change

(max)-1/2 (min)-1/2 Diagonalization of M: Why?

Harris Corner Detection

Lin ZHANG, SSE, 2020

Interpreting the eigenvalues

1 2 “Corner” 1 and 2 are large, 1 ~ 2; S increases in all

directions

1 and 2 are small; S is almost constant

in all directions

“Edge” 1 >> 2 “Edge” 2 >> 1 “Flat” region

Classification of image points using eigenvalues of M:

Lin ZHANG, SSE, 2020

Corner response function

Measure of corner response:

(k – empirical constant, k = 0.04‐0.06)

 

2

trace det M M k R  

2 1 2 1

trace det        M M

Lin ZHANG, SSE, 2020

Harris corner detector‐‐illustration

Ellipse with equation :



, 1 x x y M y           

Lin ZHANG, SSE, 2020

 

, 1 x x y M y           

Harris corner detector‐‐illustration

Ellipse with equation :

Lin ZHANG, SSE, 2020

Harris corner detector‐Algorithm

Lin ZHANG, SSE, 2020

Harris Detector: Steps

Lin ZHANG, SSE, 2020

Compute corner response R

Harris Detector: Steps

Lin ZHANG, SSE, 2020

Harris Detector: Steps

Find points with large corner response: R>threshold

Lin ZHANG, SSE, 2020

Take only the points of local maxima of R

Harris Detector: Steps

Lin ZHANG, SSE, 2020

Models of Image Change

Photometric

• Affine intensity change (I  a I + b)

Geometric

• Rotation
• Scale
• Affine

Lin ZHANG, SSE, 2020

Harris Detector: Some Properties

Rotation invariance

Ellipse rotates but its shape (i.e. eigenvalues) remains the same Corner response R is invariant to image rotation

Lin ZHANG, SSE, 2020

Harris Detector: Some Properties

Not invariant to image scale!

All points will be classified as edges

Corner ! The underlying reason is that Harris corner detection scheme does not provide an automatic and appropriate window size selection method!

Lin ZHANG, SSE, 2020

Local Descriptors for Harris Corners

p

vectorize the patch

1 2

n

v v v             

descriptor

• Descriptor for a Harris corner point
• Take a region with a fixed size around it
• Stack the region into a vector
• This vector serves as the descriptor
• When matching two descriptors in two different

images, usually the correlation coefficient is used

Lin ZHANG, SSE, 2020

Local Descriptors for Harris Corners

Correlation coefficient can be used to measure the similarity of two descriptors

1 1 2 2 1 2

[ ( )][ ( )] ( ) ( ) E E E D D     v v v v v v

• Descriptor for a Harris corner point
• Take a region with a fixed size around it
• Stack the region into a vector
• This vector serves as the descriptor
• When matching two descriptors in two different

images, usually the correlation coefficient is used

Lin ZHANG, SSE, 2020

Local Descriptors for Harris Corners

• Deficiencies of such simple descriptors
• Not rotation invariant
• Not scale invariant
• Descriptor for a Harris corner point
• Take a region with a fixed size around it
• Stack the region into a vector
• This vector serves as the descriptor
• When matching two descriptors in two different

images, usually the correlation coefficient is used

Lin ZHANG, SSE, 2020

Local Descriptors for Harris Corners

• We want:
• Rotation and scale invariant feature points
• Rotation and scale invariant feature descriptors
• Descriptor for a Harris corner point
• Take a region with a fixed size around it
• Stack the region into a vector
• This vector serves as the descriptor
• When matching two descriptors in two different

images, usually the correlation coefficient is used

Lin ZHANG, SSE, 2020

Content

• Local Invariant Features
• Motivation
• Requirements
• Invariance
• Harris Corner Detector
• Scale Invariant Point Detection
• Automatic scale selection
• Laplacian‐of‐Gaussian detector
• Difference‐of‐Gaussian detector

Lin ZHANG, SSE, 2020

From Points to Regions

• The Harris corner detector defines interest points
• Precise localization
• High repeatability
• In order to match those points, we need to compute a

descriptor over a region

• How can we define such a region in a scale invariant

manner?

• That is how can we detect sale invariant regions?

Lin ZHANG, SSE, 2020

Scale Invariant Region Selection

Lin ZHANG, SSE, 2020

Scale Invariant Region Selection

Lin ZHANG, SSE, 2020

Scale Invariant Region Selection

Lin ZHANG, SSE, 2020

Scale Invariant Region Selection

Lin ZHANG, SSE, 2020

What do we want to do next?

• Naïve approach for scale invariant local description is

not efficient (Detect Harris corners first, and then exhaustively searching for regions with appropriate sizes)

• Now we want to:
• Find scale invariant points in the image (location)
• At the same time, we want to know their characteristic

scales (used to determine the neighborhood for local description)

Lin ZHANG, SSE, 2020

Achieving scale covariance

• Goal: independently detect corresponding regions in

scaled versions of the same image

• Need scale selection mechanism for finding

characteristic region size that is covariant with the image transformation

Lin ZHANG, SSE, 2020

Automatic Scale Selection

Lin ZHANG, SSE, 2020

Automatic Scale Selection

• Common approach
• Take a local extremum of this function
• Observation: region size for which the extremum is achieved

should be covariant to image scale; this scale covariant region size is found in each image independently

Lin ZHANG, SSE, 2020

Automatic Scale Selection

Lin ZHANG, SSE, 2020

Automatic Scale Selection

Lin ZHANG, SSE, 2020

Automatic Scale Selection

Lin ZHANG, SSE, 2020

Automatic Scale Selection

Lin ZHANG, SSE, 2020

Automatic Scale Selection

Lin ZHANG, SSE, 2020

Automatic Scale Selection

Lin ZHANG, SSE, 2020

Automatic Scale Selection

Lin ZHANG, SSE, 2020

Automatic Scale Selection

• A good function for scale selection
• It should have one stable sharp peak response to

region size

Lin ZHANG, SSE, 2020

What is a useful signature function for scale?

Lin ZHANG, SSE, 2020

Characteristic Scale

Lin ZHANG, SSE, 2020

Another Fact

Spatial selection: the magnitude of the Laplacian response will achieve an extremum at the center of the blob, provided the scale of the Laplacian is “matched” to the scale

• f the blob

Lin ZHANG, SSE, 2020

Scale‐Invariant Point Detection

Local extremum in scale space

• f Laplacian of Gaussian

Lin ZHANG, SSE, 2020

Scale‐Invariant Point Detection

Local extremum in scale space

• f Laplacian of Gaussian

Lin ZHANG, SSE, 2020

Scale‐Invariant Point Detection

Local extremum in scale space

• f Laplacian of Gaussian

Lin ZHANG, SSE, 2020

Scale‐Invariant Point Detection

Local extremum in scale space

• f Laplacian of Gaussian

(Positions of extrema in the scale‐spatial space)

Lin ZHANG, SSE, 2020

We have got want we want!

Note: local extrema is obtained by comparing the examined location with all the other 26 points around it in the scale‐ space If the local extrema of LoG is achieved at p, two things of p can be determined: its spatial location and characteristic scale!

Lin ZHANG, SSE, 2020

Scale normalization

• The response of a derivative of Gaussian filter to a

perfect step edge decreases as σ increases

• To keep response the same (scale‐invariant),

must multiply Gaussian derivative by σ

• Laplacian is the second Gaussian derivative, so it

must be multiplied by σ2

Lin ZHANG, SSE, 2020

Blob detection in 2D

Laplacian of Gaussian: Circularly symmetric operator for blob detection in 2D

2 2 2 2 2

y g x g g       

, g is the Gaussian function

Lin ZHANG, SSE, 2020

Blob detection in 2D

Laplacian of Gaussian: Circularly symmetric operator for blob detection in 2D

              

2 2 2 2 2 2 norm

y g x g g 

Scale‐normalized:

Lin ZHANG, SSE, 2020

Scale‐Invariant Point Detection: Example

Lin ZHANG, SSE, 2020

Scale‐Invariant Point Detection: Example

Lin ZHANG, SSE, 2020

Scale‐Invariant Point Detection: Example

Lin ZHANG, SSE, 2020

Approximating the Laplacian with a difference of Gaussians:

 

2

( , , ) ( , , )

xx yy

L G x y G x y      ( , , ) ( , , ) DoG G x y k G x y    

(Laplacian) (Difference of Gaussians)

Efficient implementation

where Gaussian is

Assignment!

Lin ZHANG, SSE, 2020

DoG

Lin ZHANG, SSE, 2020

Scale‐Invariant Point Detection

Lin ZHANG, SSE, 2020

Examples

Lin ZHANG, SSE, 2020

Examples

Interest points found by DoG extrema What does the arrows mean? Next lecture!!

Lin ZHANG, SSE, 2020