Learning Covariant Feature Detectors Karel Lenc and Andrea Vedaldi - - PowerPoint PPT Presentation

Learning Covariant Feature Detectors

Karel Lenc and Andrea Vedaldi University of Oxford GMDL Workshop, ECCV 2016

Local Feature Detection for Image Matching

2

(x)

...

x (x’)

...

x’

The challenges of learning feature detectors

Detector goals select a small number of features → learn what to detect consistently with a viewpoint change → learn covariance

3

selection covariance

[A. Zisserman]

Prior work

Learning detectors

Anchor Points defined a-priory:

• J. Sochman and J. Matas: Learning fast emulators of binary decision processes.

IJCV 2009. (Haar Cascade, Hessian Detector).

• E. Rosten, R. Porter, and T. Drummond: Faster and better: a machine learning

approach to corner detection. TPAMI 2010 (Decision tree, Harris Detector). Detect points stable over time:

• Y. Verdie et al: TILDE: A Temporally Invariant Learned DEtector, CVPR 2015.

4

Extensive prior work

Learning descriptors

• S. Winder et al: Picking the best DAISY. CVPR 2009
• K. Simonyan et al: Learning local feature descriptors using convex optimization.

TPAMI 2014

• S. Zagoruyko et al: Learning to Compare Image Patches via Convolutional Neural
• Networks. CVPR 2015
• M. Paulin et al: Local Convolutional Features with Unsupervised Training for Image
• Retrieval. ICCV 2015
• E. Simo-Serra et al: Discriminative learning of deep convolutional feature point
• descriptors. ICCV 2015
• V. Balntas et al: Learning local feature descriptors with triplets and shallow

convolutional neural networks. BMVC 2016

• F. Radenovic et al. CNN Image Retrieval Learns from BoW: Unsupervised Fine-

Tuning with Hard Examples. ECCV 2016 See Also the Local Features: State of the art, open problems and performance evaluation Workshop

5

Detector ≠ descriptor

Bypassing the feature selection problem

Detection by regression

Advantages: feature selection is performed implicitly by the function Ψ the function Ψ can be implemented by a CNN this simplifies learning formulation

6

CNN Ψ T x T-1x Detection by regression: Design a function Ψ that translates the centre of an image window x on top of the nearest feature.

A convolutional approach

From regression to detection

Apply the function Ψ at all locations, convolutionally Accumulate votes for the feature locations Perform non-maxima suppression in the vote map

7

input image estimated translations voted features

The CNN normalizes the patch by putting the feature in the center

Covariance constraint

8

CNN Ψ T x T-1x

The CNN normalises the patch by putting the feature in the centre

Covariance constraint

9

CNN Ψ T x T-1x

Two translated patches become the same after normalization

Covariance constraint

10

CNN Ψ T x T-1x CNN Ψ Tʹ xʹ Tʹ-1x identity map T-1 = -T I Tʹ T’ ○ I ○ -T

Two translated patches become the same after normalization

Covariance constraint

11

CNN Ψ T x T-1x CNN Ψ Tʹ xʹ Tʹ-1x identity map Tʹ － T

We do not know a-priori what the normalized patch looks like

? ?

Covariance constraint

12

CNN Ψ T x CNN Ψ Tʹ xʹ identity map Tʹ － T

But we may know the transformation Tgt between the patches

Covariance constraint

13

Tgt ＝ Tʹ － T ＝ Ψ(xʹ) － Ψ(x)

? ?

T Tʹ identity map Tgt x xʹ

Satisfied if the normalised patches are identical

Covariance constraint

14

Tgt ＝ Tʹ － T ＝ Ψ(xʹ) － Ψ(x) T Tʹ Tgt x xʹ Different

⨉

Satisfied if the normalised patches are identical

Covariance constraint

15

Tgt ＝ Tʹ － T ＝ Ψ(xʹ) － Ψ(x) T Tʹ Tgt x xʹ Same

✔

Using synthetic translations of real patches

Learning formulation

20

Tgt ＝ Ψ(xʹ) － Ψ(x) ‖Tgt － Ψ(xʹ) ＋ Ψ(x)‖2 ≅ 0 x1 Tgt,1 x1 ＝ Tgt,1x1 ‖Tgt,1 － Ψ(Tgt,1x1) ＋ Ψ(x1)‖2 Loss terms x2 Tgt,2 x1 ＝ Tgt,1x1 ‖Tgt,2 － Ψ(Tgt,2x2) ＋ Ψ(x2)‖2 x3 Tgt,3 x3 ＝ Tgt,3x3 ‖Tgt,3 － Ψ(Tgt,3x3) ＋ Ψ(x3)‖2 … … …

Generalisation: affine covariant detectors

Features are oriented ellipses and transformations are affinities.

21

x Ggt ＝ Ψ(xʹ) ○ Ψ(x)-1 identity map Ψ(xʹ) ○ Ψ(x) -1 1 Ggt Ψ(x) ＝ (A,T) Ψ(xʹ) ＝ (Aʹ,Tʹ)

1 1

A very general framework

By choosing different transformation groups, we obtain different detector types:

22

Covariance group Residual group Feature Example Translation － Point FAST Similarity － Pointed disk DoG + Orientation detection Affine － Pointed ellipse Harris Affine + Orientation detection Rotation － Line bundle Orientation detection Euclidean Rotation Point Harris Translation 1D translation Line Edge detector Simlarity Rotation Disk SIFT Affine Rotation Ellipse Harris Affine

Residual transformations

See the paper for a full theoretical characterisation.

23

Ψ(xʹ) ＝ (sʹ,Tʹ) residual Ψ(xʹ) ○ Q ○ Ψ(x)-1 Q Tgt ＝ (sgt, Rgt, Tgt) Ψ(x) ＝ (s,T) DoG is covariant with scale, rotation, translation, but fixes only scale and translation. Solution: replace identity with a residual transformation Q supplanting the missing information.

By choosing different transformation groups, we obtain different detector types:

Covariance group Residual group Feature Example Translation － Point FAST Similarity － Pointed disk DoG + Orientation detection Affine － Pointed ellipse Harris Affine + Orientation detection Rotation － Line bundle Orientation detection Euclidean Rotation Point Harris Translation 1D translation Line Edge detector Simlarity Rotation Disk DoG Affine Rotation Ellipse Harris Affine

A very general framework

24

An example CNN architecture

DeteNet: A simple Siamese CNN architecture with six layers to estimate T Training uses synthetic patch pairs generated from ILSVRC12 train Uses Laplace operator to sample salient image patches of size 28 x 28

25

Conv 5 ⨉ 5 ⨉ 40 Pool ↓2 Conv 5 ⨉ 5 ⨉ 100 Pool ↓2 Conv 4 ⨉ 4 ⨉ 300 Conv 1 ⨉ 1 ⨉ 500 Conv 1 ⨉ 1 ⨉ 500 Conv 1 ⨉ 1 ⨉ 2

Ψ(x) x

Conv 5 ⨉ 5 ⨉ 40 Pool ↓2 Conv 5 ⨉ 5 ⨉ 100 Pool ↓2 Conv 4 ⨉ 4 ⨉ 300 Conv 1 ⨉ 1 ⨉ 500 Conv 1 ⨉ 1 ⨉ 500 Conv 1 ⨉ 1 ⨉ 2

x’ Ψ(x')

‖Tgt － Ψ(xʹ) ＋ Ψ(x)‖2 Tgt

Loss Shared Weights

DetNet Easy Test

26

DetNet Hard Test

27

Detection

Detection

Detection

Detection

DoG

Harris

Detection

Repeatability on DTU

DetNet results

37

Repeatability on DTU

DetNet results

38

Repeatability on DTU

DetNet results

39

RotNet Architecture

L2 Norm places the regressed location on an unit circle

40

Conv 5 ⨉ 5 ⨉ 40 Pool ↓2 Conv 5 ⨉ 5 ⨉ 100 Pool ↓2 Conv 4 ⨉ 4 ⨉ 300 Conv 1 ⨉ 1 ⨉ 500 Conv 1 ⨉ 1 ⨉ 500 Conv 1 ⨉ 1 ⨉ 2

Ψ(x) x

Conv 5 ⨉ 5 ⨉ 40 Pool ↓2 Conv 5 ⨉ 5 ⨉ 100 Pool ↓2 Conv 4 ⨉ 4 ⨉ 300 Conv 1 ⨉ 1 ⨉ 500 Conv 1 ⨉ 1 ⨉ 500 Conv 1 ⨉ 1 ⨉ 2

x’ Ψ(x')

‖Rgt － Ψ(xʹ) ＋ Ψ(x)‖2 (cos(𝜄gt), sin(𝜄gt))

Loss Shared Weights

L2 Norm L2 Norm

RotNet Easy Test

41

RotNet Hard Test

42

Summary

Detection by regression bypasses the feature selection problem formulated as a learnable convolutional regressor Covariance constraint covariance is all you need to learn features naturally integrates with detection by regression a very general formulation encompassing all detector types bonus: a nice theory of features Results very good detectors can be learned however, not yet dramatically better than existing ones This is just the beginning a whole new approach many details still need ironing out DepDet Models available at: https://github.com/lenck/ddet

44

Repeatability on VGG-Affine

DetNet results

45

Angle error and matching score VGG-Affine

RotNet example patch pairs

46