CS4495/6495 Introduction to Computer Vision 4A-L2 Finding corners - - PowerPoint PPT Presentation

4A-L2 Finding corners

CS4495/6495 Introduction to Computer Vision

Feature points

Characteristics of good features

Repeatability/Precision

• The same feature can be found in several images

despite geometric and photometric transformations

Characteristics of good features

Saliency/Matchability

• Each feature has a distinctive description

Characteristics of good features

Compactness and efficiency

• Many fewer features than image pixels

Characteristics of good features

Locality

• A feature occupies a relatively small area of the

image; robust to clutter and occlusion

Corner Detection: Basic Idea

“edge”: no change along the edge direction “corner”: significant change in all directions with small shift “flat” region: no change in all directions

Source: A. Efros

Finding Corners

• Key property: in the region around a corner, image

gradient has two or more dominant directions

Finding Corners

• C. Harris and M. Stephens. "A Combined Corner and Edge

Detector,” Proceedings of the 4th Alvey Vision Conference: 1988

Corner Detection: Mathematics

 

2 ,

( , ) ( , ) ( , ) ( , )

x y

E u v w x y I x u y v I x y    



Intensity Shifted intensity Window function

• r

Window function w(x,y) = Gaussian 1 in window, 0 outside

Source: R. Szeliski

Change in appearance for the shift [u,v]:

Corner Detection: Mathematics

 

2 ,

( , ) ( , ) ( , ) ( , )

x y

E u v w x y I x u y v I x y    



I(x, y) E(u, v)

E(0,0) E(3,2)

Change in appearance for the shift [u,v]:

Corner Detection: Mathematics

 

2 ,

( , ) ( , ) ( , ) ( , )

x y

E u v w x y I x u y v I x y    



We want to find out how this function behaves for small shifts (u,v near 0,0)

Change in appearance for the shift [u,v]:

Corner Detection: Mathematics

Second-order Taylor expansion of E(u,v) about (0,0) (local quadratic approximation for small u,v):

 

2 ,

( , ) ( , ) ( , ) ( , )

x y

E u v w x y I x u y v I x y    



Change in appearance for the shift [u,v]:

Corner Detection: Mathematics

2 2 2

(0 ) 1 (0 ) ( ) (0 ) · · 2 d F d F F x F x x d x d x      

 

2 ,

( , ) ( , ) ( , ) ( , )

x y

E u v w x y I x u y v I x y    



Change in appearance for the shift [u,v]:

Corner Detection: Mathematics

(0, 0 ) (0, 0 ) (0, 0 ) 1 ( , ) (0, 0 ) [ ] [ ] (0, 0 ) (0, 0 ) (0, 0 ) 2

u u u u v v u v vv

E E E u E u v E u v u v E E E v                     

2 2 2

(0 ) 1 (0 ) ( ) (0 ) · · 2 d F d F F x F x x d x d x      

 

2 ,

( , ) ( , ) ( , ) ( , )

x y

E u v w x y I x u y v I x y    



Change in appearance for the shift [u,v]:

1 (0, 0 ) (0, 0 ) (0, 0 ) ( , ) (0, 0 ) [ ] [ ] (0, 0 ) (0, 0 ) (0, 0 ) 2

u u u u v v u v vv

E E E u E u v E u v u v E E E v                     

Second-order Taylor expansion of E(u,v) about (0,0):

 

2 ,

( , ) ( , ) ( , ) ( , )

x y

E u v w x y I x u y v I x y    



1 (0, 0 ) (0, 0 ) (0, 0 ) ( , ) (0, 0 ) [ ] [ ] (0, 0 ) (0, 0 ) (0, 0 ) 2

u u u u v v u v vv

E E E u E u v E u v u v E E E v                     

Second-order Taylor expansion of E(u,v) about (0,0):

 

2 ,

( , ) ( , ) ( , ) ( , )

x y

E u v w x y I x u y v I x y    



Need these derivatives…

 

,

( , ) 2 ( , ) ( , ) ( , ) ( , )

x y

u u v

w x y I x u y v E I x y x u y v

I

      

x

 

2 ,

( , ) ( , ) ( , ) ( , )

x y

E u v w x y I x u y v I x y    



Second-order Taylor expansion of E(u,v) about (0,0):

 

, ,

( , ) 2 ( , ) ( , ) ( , ) 2 ( , ) ( , ) ( , ) ( , )

x y x y

u u u v

w x y x u y v x u y v w x y I x u y v I x y x u v E y

I I I

            

x x xx

 

2 ,

( , ) ( , ) ( , ) ( , )

x y

E u v w x y I x u y v I x y    



Second-order Taylor expansion of E(u,v) about (0,0):

 

2 ,

( , ) ( , ) ( , ) ( , )

x y

E u v w x y I x u y v I x y    



Second-order Taylor expansion of E(u,v) about (0,0):

 

, ,

( , ) 2 ( , ) ( , ) ( , ) 2 ( , ) ( , ) ( , ) ( , )

x y x y

u v y x xy

E u v w x y x u y v x u y v w x y I x u y v I x y x u y v

I I I

            

Second-order Taylor expansion of E(u,v) about (0,0):

   

, , , , ,

( , ) 2 ( , ) ( , ) ( , ) ( , ) ( , ) 2 ( , ) ( , ) ( , ) 2 ( , ) ( , ) ( , ) ( , ) ( , ) 2 ( , ) ( , ) ( , ) 2 ( , ) ( , ) ( ,

u x x y u u x x x y xx x y u v y x x y x y

E u v w x y I x u y v I x y I x u y v E u v w x y I x u y v I x u y v w x y I x u y v I x y I x u y v E u v w x y I x u y v I x u y v w x y I x u y v I x y                               

 

) ( , )

xy

I x u y v   1 (0, 0 ) (0, 0 ) (0, 0 ) ( , ) (0, 0 ) [ ] [ ] (0, 0 ) (0, 0 ) (0, 0 ) 2

u u u u v v u v vv

E E E u E u v E u v u v E E E v                     

1 (0, 0 ) (0, 0 ) (0, 0 ) ( , ) (0, 0 ) [ ] [ ] (0, 0 ) (0, 0 ) (0, 0 ) 2

u u u u v v u v vv

E E E u E u v E u v u v E E E v                     

     

, , , , ,

(0, 0 ) 2 ( , ) ( , ) ( , ) ( , ) (0, 0 ) 2 ( , ) ( , ) ( , ) 2 ( , ) ( , ) ( , ) ( , ) (0, 0 ) 2 ( , ) ( , ) ( , ) 2 ( , ) ( , ) ( , ) ( , )

u x x y u u x x x y xx x y u v y x x y xy x y

E w x y I x y I x y I x y E w x y I x y I x y w x y I x y I x y I x y E w x y I x y I x y w x y I x y I x y I x y             

Evaluate E and its derivatives at (0,0):

= 0 = 0 = 0 = 0

Second-order Taylor expansion of E(u,v) about (0,0):

(0, 0 ) (0, 0 ) (0, 0 )

u v

E E E   

(0, 0 ) (0, 0 ) ( 1 (0, 0 ) (0, 0 ) ( , ) [ ] [ ] (0, 0 ) (0, 0 ) 2 0, )

u u u v u v vv u v

E E E E u E u v u v u v v E E E                     

, , ,

(0, 0 ) 2 ( , ) ( , ) ( , ) (0, 0 ) 2 ( , ) ( , ) ( , ) (0, 0 ) 2 ( , ) ( , ) ( , )

u u x x x y vv y y x y u v x y x y

E w x y I x y I x y E w x y I x y I x y E w x y I x y I x y      

2 , , 2 , ,

( , ) ( , ) ( , ) ( , ) ( , ) ( , ) [ ] ( , ) ( , ) ( , ) ( , ) ( , )

x x y x y x y x y y x y x y

w x y I x y w x y I x y I x y u E u v u v w x y I x y I x y w x y I x y v                   

(0, 0 ) (0, 0 ) (0, 0 )

u v

E E E   

, , ,

(0, 0 ) 2 ( , ) ( , ) ( , ) (0, 0 ) 2 ( , ) ( , ) ( , ) (0, 0 ) 2 ( , ) ( , ) ( , )

u u x x x y vv y y x y u v x y x y

E w x y I x y I x y E w x y I x y I x y E w x y I x y I x y      

Second-order Taylor expansion of E(u,v) about (0,0):

Corner Detection: Mathematics

The quadratic approximation simplifies to

2 2 ,

( , )

x x y x y x y y

I I I M w x y I I I         



where M is a second moment matrix computed from image derivatives:

( , ) [ ] u E u v u v M v       

The second moment matrix M:

2 2 ,

( , )

x x y x y x y y

I I I M w x y I I I         



Can be written (without the weight):

Each product is a rank 1 2x2

( )

x T x x x y x y y x y y y

I I I I I M I I I I I I I I I                               

     

The surface E(u,v) is locally approximated by a quadratic form.

Interpreting the second moment matrix

( , ) [ ] u E u v u v M v       

2 2 ,

( , )

x x y x y x y y

I I I M w x y I I I        

Consider a constant “slice” of E(u, v):

Interpreting the second moment matrix

[ ] co n st u u v M v       

This is the equation of an ellipse.

2 2 2 2

2

x x y y

I u I I u v I v k      

2 1 2 , 2

( , )

x x y x y x y y

I I I M w x y I I I                 

First, consider the axis-aligned case where gradients are either horizontal or vertical If either λ is close to 0, then this is not a corner, so look for locations where both are large.

Interpreting the second moment matrix

Interpreting the second moment matrix

1 1 2

M R R  

 

     

The axis lengths of the ellipse are determined by the eigenvalues and the orientation is determined by R Diagonalization of M:

Interpreting the second moment matrix

direction of the slowest change direction of the fastest change (max)-1/2 (min)-1/2

Interpreting the eigenvalues

1 2

“Corner” 1 and 2 are large, 1 ~ 2; E increases in all directions 1 and 2 are small; E is almost constant in all directions “Edge” 1 >> 2 “Edge” 2 >> 1

“Flat” region

Classification of image points using eigenvalues of M:

Harris corner response function

“Corner” R > 0 “Edge” R < 0 “Edge” R < 0 “Flat” region |R| small

α: constant (0.04 to 0.06)

2 2 1 2 1 2

d et( ) trace( ) ( ) R M M           

Harris corner response function

“Corner” R > 0 “Edge” R < 0 “Edge” R < 0 “Flat” region |R| small

R depends only on eigenvalues of M, but don’t compute them (no sqrt, so really fast even in the ’80s).

2 2 1 2 1 2

d et( ) trace( ) ( ) R M M           

Harris corner response function

“Corner” R > 0 “Edge” R < 0 “Edge” R < 0 “Flat” region |R| small

2 2 1 2 1 2

d et( ) trace( ) ( ) R M M           

R is large for a corner R is negative with large magnitude for an edge |R| is small for a flat region

Low texture region

Gradients have small magnitude => small 1, small 2

( )

T

M I I    

Edge

Large gradients, all the same => large 1, small 2

( )

T

M I I    

High textured region

Gradients different, large magnitudes => large 1, large 2

( )

T

M I I    

Harris Detector: Algorithm

1.

Compute Gaussian derivatives at each pixel

2.

Compute second moment matrix M in a Gaussian window around each pixel

3.

Compute corner response function R

4.

Threshold R

5.

Find local maxima of response function (nonmaximum suppression)

• C. Harris and M. Stephens. "A Combined Corner and Edge Detector.“

Proceedings of the 4th Alvey Vision Conference: pages 147—151, 1988.

Harris Detector: Workflow

Harris Detector: Workflow

Compute corner response R

Harris Detector: Workflow

Find points with large corner response: R>threshold

Harris Detector: Workflow

Take only the points of local maxima of R

Harris Detector: Workflow

Other corners:

Shi-Tomasi ’94:

“Cornerness” = min (λ1, λ2) Find local maximums cvGoodFeaturesToTrack(...) Reportedly better for region undergoing affine deformations

Other corners:

• Brown, M., Szeliski, R., and Winder, S. (2005):
• There are others…

1 1

d et tr M M      