Overview Overview Introduction to local features Introduction to - - PowerPoint PPT Presentation

Overview Overview

Introduction to local features

• Introduction to local features

H i i t t i t + SSD ZNCC SIFT

• Harris interest points + SSD, ZNCC, SIFT

S l & ffi i i t i t t i t d t t

• Scale & affine invariant interest point detectors

E l i d i f diff d

• Evaluation and comparison of different detectors
• Region descriptors and their performance

Scale invariance - motivation Scale invariance motivation

• Description regions have to be adapted to scale changes

I t t i t h t b t bl f l h

• Interest points have to be repeatable for scale changes

Harris detector + scale changes Harris detector + scale changes

| } ) ) ( ( | ) {( | H dist b a b a  

Repeatability rate

|) | |, max(| | } ) ), ( ( | ) , {( | ) (

i i i i i i

H dist R b a b a b a    

Scale adaptation Scale adaptation

Scale change bet een t o images       Scale change between two images                          

1 1 2 2 2 2 1 1 1

sy sx I y x I y x I Scale adapted derivative calculation

Scale adaptation Scale adaptation

Scale change bet een t o images       Scale change between two images                          

1 1 2 2 2 2 1 1 1

sy sx I y x I y x I

   

Scale adapted derivative calculation

) ( ) (

1 1

2 2 2 1 1 1

  s G y x I s G y x I

n n

i i n i i  

                  

 s

n

s

   

Scale adaptation Scale adaptation

  ) ( ) (

2

  L L L          ) ( ) ( ) ( ) ( ) ~ (

2 

   

y y x y x x

L L L L L L G

) (

i

L where are the derivatives with Gaussian convolution

Scale adaptation Scale adaptation

  ) ( ) (

2

  L L L          ) ( ) ( ) ( ) ( ) ~ (

2 

   

y y x y x x

L L L L L L G

) (

i

L where are the derivatives with Gaussian convolution

Scale adapted auto-correlation matrix

     ) ( ) ( ) ~ (

2 2

  s L L s L G

y x x

Scale adapted auto correlation matrix

     ) ( ) ( ) (

2 2

   s L s L L s G s

y y x y x x

Harris detector – adaptation to scale Harris detector adaptation to scale

} ) ), ( ( | ) , {( ) (    

i i i i

H dist R b a b a

Multi-scale matching algorithm Multi scale matching algorithm

1  s 3  s 5  s

Multi-scale matching algorithm Multi scale matching algorithm

1  s

8 matches 8 matches

Multi-scale matching algorithm Multi scale matching algorithm

1  s

3 matches

Robust estimation of a global affine transformation

3 matches

Multi-scale matching algorithm Multi scale matching algorithm

1  s

3 matches 3 matches

3  s

4 t h 4 matches

Multi-scale matching algorithm Multi scale matching algorithm

1  s

3 matches 3 matches

3  s

4 t h 4 matches

5  s

correct scale

highest number of matches

16 matches

Matching results Matching results

Scale change of 5 7 Scale change of 5.7

Matching results Matching results

100% t t h (13 t h ) 100% correct matches (13 matches)

Scale selection Scale selection

• We want to find the characteristic scale of the blob by

convolving it with Laplacians at several scales and looking for the maximum response H L l i d l

• However, Laplacian response decays as scale

increases:

increasing σ

• riginal signal

(radius=8)

Why does this happen?

Scale normalization Scale normalization

The response of a derivative of Gaussian filter to a perfect

• The response of a derivative of Gaussian filter to a perfect

step edge decreases as σ increases

1   2 1

Scale normalization Scale normalization

The response of a derivative of Gaussian filter to a perfect

• The response of a derivative of Gaussian filter to a perfect

step edge decreases as σ increases

• To keep response the same (scale invariant) must
• To keep response the same (scale-invariant), must

multiply Gaussian derivative by σ

• Laplacian is the second Gaussian derivative so it must be

Laplacian is the second Gaussian derivative, so it must be multiplied by σ2

Effect of scale normalization Effect of scale normalization

Unnormalized Laplacian response Original signal Scale-normalized Laplacian response maximum

Blob detection in 2D Blob detection in 2D

Laplacian of Gaussian: Circularly symmetric operator for

• Laplacian of Gaussian: Circularly symmetric operator for

blob detection in 2D

2 2 2 2 2

g g g     

2 2

y x g  

Blob detection in 2D Blob detection in 2D

Laplacian of Gaussian: Circularly symmetric operator for

• Laplacian of Gaussian: Circularly symmetric operator for

blob detection in 2D

                

2 2 2 2 2 2 norm

g g g 

Scale-normalized:

     

2 2 norm

y x g

Scale selection Scale selection

The 2D Laplacian is given by

• The 2D Laplacian is given by

2 2 2

2 / ) ( 2 2 2

) 2 (





y x

e y x

 

 

(up to scale)

• For a binary circle of radius r, the Laplacian achieves a

y , p maximum at

2 / r  

e n respons

r

Laplacian 2 / r image scale (σ)

Characteristic scale Characteristic scale

We define the characteristic scale as the scale that

• We define the characteristic scale as the scale that

produces peak of Laplacian response

characteristic scale

• T. Lindeberg (1998). Feature detection with automatic scale selection.

International Journal of Computer Vision 30 (2): pp 77--116.

Scale selection Scale selection

For a point compute a value (gradient Laplacian etc ) at

• For a point compute a value (gradient, Laplacian etc.) at

several scales Normali ation of the al es ith the scale factor

• Normalization of the values with the scale factor

e.g. Laplacian

| ) ( |

2 yy xx

L L s 

• Select scale at the maximum → characteristic scale



s

| ) ( |

2 yy xx

L L s 

E lt h th t th L l i i b t lt

scale

• Exp. results show that the Laplacian gives best results

Scale selection Scale selection

Scale invariance of the characteristic scale

• Scale invariance of the characteristic scale

s

p.

• norm. La

scale

Scale selection Scale selection

Scale invariance of the characteristic scale

• Scale invariance of the characteristic scale

s

p. p.

• norm. La
• norm. La

 

scale scale

  



2 1

s s s

• Relation between characteristic scales

Scale-invariant detectors Scale invariant detectors

Harris Laplace (Mikolajczyk & Schmid’01)

• Harris-Laplace (Mikolajczyk & Schmid’01)
• Laplacian detector (Lindeberg’98)
• Difference of Gaussian (Lowe’99)

Harris-Laplace Laplacian

Harris-Laplace Harris Laplace

multi-scale Harris points selection of points at maximum of Laplacian invariant points + associated regions [Mikolajczyk & Schmid’01]

Matching results Matching results

213 / 190 detected interest points 213 / 190 detected interest points

Matching results Matching results

58 points are initially matched 58 points are initially matched

Matching results Matching results

32 points are matched after verification – all correct 32 points are matched after verification all correct

LOG detector LOG detector

Convolve image with scale Convolve image with scale- normalized Laplacian at several scales several scales

)) ( ) ( (

2

 

yy xx

G G s LOG  

Detection of maxima and minima

• f Laplacian in scale space

Hessian detector Hessian detector

      

xy xx

L L L L x H ) (

Hessian matrix

 

yy xy

L L

2 xy yy xx

L L L DET  

Determinant of Hessian matrix Penalizes/eliminates long structures g

• with small derivative in a single direction

Efficient implementation Efficient implementation

• Difference of Gaussian (DOG) approximates the

Difference of Gaussian (DOG) approximates the Laplacian

) ( ) (   G k G DOG  

• Error due to the approximation

DOG detector DOG detector

• Fast computation scale space processed one octave at a
• Fast computation, scale space processed one octave at a

time

David G. Lowe. "Distinctive image features from scale-invariant keypoints.”IJCV 60 (2).

Local features - overview Local features overview

• Scale invariant interest points
• Affine invariant interest points
• Evaluation of interest points
• Descriptors and their evaluation

Affine invariant regions - Motivation Affine invariant regions Motivation

Scale invariance is not sufficient for large baseline changes

• Scale invariance is not sufficient for large baseline changes

detected scale invariant region g

A

j t d i i i t h l ll projected regions, viewpoint changes can locally be approximated by an affine transformation A

Affine invariant regions - Motivation Affine invariant regions Motivation

Affine invariant regions - Example Affine invariant regions Example

Harris/Hessian/Laplacian-Affine Harris/Hessian/Laplacian Affine

• Initialize with scale invariant Harris/Hessian/Laplacian
• Initialize with scale-invariant Harris/Hessian/Laplacian

points

• Estimation of the affine neighbourhood with the second

moment matrix [Lindeberg’94]

• Apply affine neighbourhood estimation to the scale-

invariant interest points [Mikolajczyk & Schmid’02 invariant interest points [Mikolajczyk & Schmid 02, Schaffalitzky & Zisserman’02]

• Excellent results in a comparison [Mikolajczyk et al.’05]

Affine invariant regions Affine invariant regions

Based on the second moment matrix (Lindeberg’94)

• Based on the second moment matrix (Lindeberg’94)

     ) , ( ) , ( ) ( ) (

2 2 D y x D x

L L L G M   x x          ) , ( ) , ( ) , ( ) , ( ) ( ) , , (

2 2 D y D y x D y x D x I D D I

L L L G M        x x x

• Normalization with eigenvalues/eigenvectors

1

x x

2

M  

Affine invariant regions Affine invariant regions

x x A 

L R

x x A 

L 2 1 L

x x

L

M  

R 2 1 R

x x

R

M  

  

L R

Rx x

Isotropic neighborhoods related by image rotation

Affine invariant regions - Estimation

• Iterative estimation

initial points

Affine invariant regions Estimation

• Iterative estimation – initial points

Affine invariant regions - Estimation

• Iterative estimation

iteration #1

Affine invariant regions Estimation

• Iterative estimation – iteration #1

Affine invariant regions - Estimation

• Iterative estimation

iteration #2

Affine invariant regions Estimation

• Iterative estimation – iteration #2

Affine invariant regions - Estimation

• Iterative estimation

iteration #3 #4

Affine invariant regions Estimation

• Iterative estimation – iteration #3, #4

Harris-Affine versus Harris-Laplace Harris Affine versus Harris Laplace

H i L l Harris-Laplace Harris-Affine

Harris/Hessian-Affine Harris/Hessian Affine

H i Affi Harris-Affine Hessian-Affine

Harris-Affine Harris Affine

Hessian-Affine Hessian Affine

Matches Matches

22 correct matches

Matches Matches

33 correct matches

Maximally stable extremal regions (MSER) [Matas’02] Maximally stable extremal regions (MSER) [Matas 02]

Extremal regions: connected components in a thresholded

• Extremal regions: connected components in a thresholded

image (all pixels above/below a threshold)

• Maximally stable: minimal change of the component

(area) for a change of the threshold i e region remains (area) for a change of the threshold, i.e. region remains stable for a change of threshold

• Excellent results in a comparison [Mikolajczyk et al.’05]

Maximally stable extremal regions (MSER) Maximally stable extremal regions (MSER) E l f th h ld d i Examples of thresholded images

high threshold low threshold

MSER MSER

Overview Overview

Introduction to local features

• Introduction to local features

H i i t t i t + SSD ZNCC SIFT

• Harris interest points + SSD, ZNCC, SIFT

S l & ffi i i t i t t i t d t t

• Scale & affine invariant interest point detectors

E l ti d i f diff t d t t

• Evaluation and comparison of different detectors
• Region descriptors and their performance

Evaluation of interest points Evaluation of interest points

Quantitative evaluation of interest point/region detectors

• Quantitative evaluation of interest point/region detectors

– points / regions at the same relative location and area

• Repeatability rate : percentage of corresponding points
• Two points/regions are corresponding if

– location error small location error small – area intersection large

• [K. Mikolajczyk, T. Tuytelaars, C. Schmid, A. Zisserman, J. Matas,
• F. Schaffalitzky, T. Kadir & L. Van Gool ’05]

Evaluation criterion Evaluation criterion

H

% 100 # #   regions detected regions ing correspond ity repeatabil

Evaluation criterion Evaluation criterion

H

% 100 # #   regions detected regions ing correspond ity repeatabil

% 100 ) 1 (    union

• n

intersecti error

• verlap

2% 10% 20% 30% 40% 50% 60%

Dataset Dataset

• Different types of transformation
• Different types of transformation

– Viewpoint change – Scale change – Image blur – JPEG compression Light change – Light change

• Two scene types

yp

– Structured – Textured

• Transformations within the sequence (homographies)

– Independent estimation – Independent estimation

Viewpoint change (0-60 degrees ) Viewpoint change (0-60 degrees )

structured scene textured scene

Zoom + rotation (zoom of 1-4) Zoom + rotation (zoom of 1-4)

structured scene textured scene

Blur compression illumination Blur, compression, illumination

blur - structured scene blur - textured scene light change - structured scene jpeg compression - structured scene

Comparison of affine invariant detectors Comparison of affine invariant detectors

Viewpoint change - structured scene Viewpoint change structured scene

90 100 Harris−Affine Hessian−Affine

repeatability %

60 70 80

ility %

Hessian−Affine MSER IBR EBR Salient 30 40 50

repeatabilit

15 20 25 30 35 40 45 50 55 60 65 10 20

viewpoint angle viewpoint angle

reference image 20 60 40

Comparison of affine invariant detectors

Scale change

Comparison of affine invariant detectors

Scale change

repeatability % repeatability %

reference image 4 reference image 2.8

Conclusion - detectors

• Good performance for large viewpoint and scale changes

Conclusion detectors

• Good performance for large viewpoint and scale changes
• Results depend on transformation and scene type, no one best

Results depend on transformation and scene type, no one best detector

• Detectors are complementary

– MSER adapted to structured scenes H i d H i d t d t t t d – Harris and Hessian adapted to textured scenes

• Performance of the different scale invariant detectors is very similar
• Performance of the different scale invariant detectors is very similar

(Harris-Laplace, Hessian, LoG and DOG)

• Scale-invariant detector sufficient up to 40 degrees of viewpoint

change

Overview Overview

Introduction to local features

• Introduction to local features

H i i t t i t + SSD ZNCC SIFT

• Harris interest points + SSD, ZNCC, SIFT

S l & ffi i i t i t t i t d t t

• Scale & affine invariant interest point detectors

E l i d i f diff d

• Evaluation and comparison of different detectors
• Region descriptors and their performance

Region descriptors Region descriptors

• Normalized regions are

– invariant to geometric transformations except rotation – not invariant to photometric transformations

Descriptors Descriptors

• Regions invariant to geometric transformations except

rotation

– rotation invariant descriptors – normalization with dominant gradient direction – normalization with dominant gradient direction

• Regions not invariant to photometric transformations

– invariance to affine photometric transformations p – normalization with mean and standard deviation of the image patch

Descriptors Descriptors

Extract affine regions Normalize regions Eliminate rotational + illumination Compute appearance descriptors SIFT (Lowe ’04)

Descriptors Descriptors

Gaussian derivative based descriptors

• Gaussian derivative-based descriptors

– Differential invariants (Koenderink and van Doorn’87) – Steerable filters (Freeman and Adelson’91) Steerable filters (Freeman and Adelson 91)

• SIFT (Lowe’99)

( )

• Moment invariants [Van Gool et al.’96]
• Shape context [Belongie et al.’02]

p

[ g ]

• SIFT with PCA dimensionality reduction
• SURF descriptor [Bay et al.’08]

SURF descriptor [Bay et al. 08]

• DAISY descriptor [Tola et al.’08, Windler et al’09]

Comparison criterion Comparison criterion

• Descriptors should be

p

– Distinctive – Robust to changes on viewing conditions as well as to errors of th d t t the detector

• Detection rate (recall)
• Detection rate (recall)

– #correct matches / #correspondences

• False positive rate

1

False positive rate

– #false matches / #all matches

• Variation of the distance threshold

– distance (d1, d2) < threshold

1

[K Mikolajczyk & C Schmid PAMI’05] [K. Mikolajczyk & C. Schmid, PAMI 05]

Viewpoint change (60 degrees) Viewpoint change (60 degrees)

esift

* *

shape context gradient pca cross correlation steerable filters gradient moments sift

0.9 1

gradient pca complex filters har−aff esift

0.6 0.7 0.8 0.9

101

0.3 0.4 0.5 0.6

#correct / 2101

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0.2 0.3 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

1−precision

Scale change (factor 2 8)

esift

* *

Scale change (factor 2.8)

shape context gradient pca cross correlation steerable filters gradient moments sift

0.9 1

gradient pca complex filters har−aff esift

0.6 0.7 0.8 0.9

086

0.3 0.4 0.5 0.6

#correct / 208

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0.2 0.3 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

1−precision

Conclusion - descriptors Conclusion descriptors

SIFT based descriptors perform best

• SIFT based descriptors perform best

Si ifi t diff b t SIFT d l di i

• Significant difference between SIFT and low dimension

descriptors as well as cross-correlation

• Robust region descriptors better than point-wise

descriptors descriptors

• Performance of the descriptor is relatively independent of
• Performance of the descriptor is relatively independent of

the detector

Available on the internet Available on the internet

h //l i i l f / f http://lear.inrialpes.fr/software

• Binaries for detectors and descriptors

– Building blocks for recognition systems

• Carefully designed test setup

– Dataset with transformations – Evaluation code in matlab B h k f d t t d d i t – Benchmark for new detectors and descriptors