Place recognition with instance search from hand-crafted to - - PowerPoint PPT Presentation

Place recognition with instance search

from hand-crafted to learning-based methods

Giorgos Tolias

Tutorial on “Large-Scale Visual Place Recognition and Image-Based Localization” Tolias Sattler Brachmann ICCV 2019, Seoul

Outline

• Place recognition with instance search
• Benchmarks
• Hand-crafted approaches
• Learning-based approaches
• Descriptor whitening
• Privacy preserving search

Place recognition

Place recognition

Place recognition

Place recognition via instance search

• Huge number of classes
• each “named” location
• each landmark
• GPS coordinates
• Low intra-class variability
• Large-scale instance search
• nearest neighbor classifiers
• kernel density estimation

representation of database images

Place recognition via instance search

query image query representation nearest neighbor search

Benchmarks

Geo-localization

• Tokyo 24/7 [Torii et al., CVPR‘15]
• Pittsburg dataset [Torii et al., CVPR‘13]
• San Francisco dataset [Chen et al., CVPR’11]

GPS-based ground truth

image from Tokyo 24 / 7

Geo-localization

GPS-based ground truth IM2GPS [Hays & Efros, CVPR’08]

images from [Hays & Efros]

Instance retrieval (buildings, landmarks)

• Oxford buildings [Philbin et al., CVPR’07]
• Paris [Philbin et al., CVPR’08]
• Oxford/Paris revisited + 1M distractors

[Radenovic et al., CVPR’18] Manually constructed ground truth

http://cmp.felk.cvut.cz/revisitop/

Landmark recognition and retrieval

Google Landmarks Dataset Crowd-sourced ground truth

https://github.com/cvdfoundation/google-landmark

• Recognition training set

4.1m images 200k landmarks

• Retrieval index set

762k images (1⁄3 decrease) 101k landmarks

• Test set

118k images about 1% depicts landmarks

Global descriptors & voting approaches

Global descriptor

• Instance search reduces to similarity search in d-dimensional space
• Compatible with efficient nearest neighbor techniques

Local descriptors

• Pairwise similarity
• Index local descriptors separately

Match kernel – voting approach

Embedding function: Image similarity: Local similarity function:

Voting approach & global descriptor

Linear matching of local descriptors

Voting approach & global descriptor

Linear matching of local descriptors voting approach

Voting approach & global descriptor

Linear matching of local descriptors voting approach

Voting approach & global descriptor

Linear matching of local descriptors global descriptor voting approach

From local to global

process & aggregate post-processing (whitening) local descriptor set global descriptor global descriptor

Hand-crafted methods

Locally invariant features

• Local feature detectors

SIFT [ICCV’99] SURF [ECCV’06] MSER [BMVC’03] Hessian-Affine [IJCV’04], …

• Local descriptors

SIFT [ICCV’99] SURF [ECCV’06], …

Bag-of-Words (BoW)

• Descriptor quantization
• Clustering  visual codebook
• Cluster  visual word
• Matching only within cluster

[Sivic, ICCV’03] [Csurka, ECCVW’04]

Bag-of-Words (BoW)

• Descriptor quantization
• Clustering  visual codebook
• Cluster  visual word
• Matching only within cluster

[Sivic, ICCV’03] [Csurka, ECCVW’04]

Bag-of-Words (BoW)

not all pairs contribute

• Descriptor quantization
• Clustering  visual codebook
• Cluster  visual word
• Matching only within cluster

[Sivic, ICCV’03] [Csurka, ECCVW’04]

Bag-of-Words (BoW)

global descriptor voting approach i-th element

Vector of locally aggregated descriptors VLAD

Aggregate residual vectors, instead of counts

[Jegou et al., CVPR’10]

Dense features

[Zhao et al., BMVC’13]

… … … … … …

Aggregating dense features  fixed dimensionality of global descriptor DenseVLAD [Torii et al, CVPR’15] [Tolias et al., ECCV’14]

Global descriptors

global descriptor voting approach

• Global descriptor preferred for small codebooks (8-512 centroids)

Large codebooks and inverted files

1: 2: … i: ... K: Codebook size in the order of 106 [Philbin et al., CVPR’07]

Large codebooks and inverted files

1: 2: … i: ... K: Codebook size in the order of 106 [Philbin et al., CVPR’07]

Large codebooks and inverted files

1: 2: … i: ... K: Codebook size in the order of 106 [Philbin et al., CVPR’07]

Large codebooks and inverted files

1: 2: … i: ... K: Codebook size in the order of 106 [Philbin et al., CVPR’07]

Large codebooks and inverted files

1: 2: … i: ... K: … … … … Codebook size in the order of 106 [Philbin et al., CVPR’07]

Large codebooks and inverted files

1: 2: … i: ... K: … … … … Codebook size in the order of 106 [Philbin et al., CVPR’07]

Large codebooks and inverted files

1: 2: … i: ... K: … … … … Codebook size in the order of 106 [Philbin et al., CVPR’07]

Selective Match Kernel

Non-linear local similarity Normalized residuals

[Tolias et al. IJCV’16]

Aggregated Selective Match Kernel (ASMK)

Normalized aggregated residuals

[Tolias et al. IJCV’16]

(or binarized aggregated residuals for efficiency)

Aggregated Selective Match Kernel (ASMK)

[Tolias et al. IJCV’16] same color = same visual word typical codebook size: 65k

Spatial verification

tentative correspondences inliers with fast spatial verification [Philbin et al., CVPR’07] [Philbin et al., CVPR’07]

Learning-based methods

Global descriptors with CNNs

• Descriptor / network architecture
• Loss function
• Supervision / training data

Global descriptors with CNNs

embedding & aggregation

fully convolutional network

descriptor:

BoW with CNN features

[Mohedano et al. ICMR’16]

• Used with pre-trained features and hard assignment
• Soft assignment needed for training

NetVLAD

soft-assignment

Codebook centroids are trainable parameters

[Arandjelovic et al., CVPR’16]

• Descriptor
• Pair-wise similarity
• Simple but works

 discriminative power of CNN activations

Sum pooling – SPoC descriptor

[Babenko & Lempitsky, ICCV’15]

Weighted sum pooling – CroW descriptor

α: weight based on L2 norm of local descriptors β: inverted-document-frequency weight example of α [Kalantidis et al., ECCV’16]

Max pooling – MAC descriptor

Input image conv5 filter 1 conv5 filter 2 …. conv5 filter i …. conv5 filter K

[Razavian et al., MTA’16] [Tolias et al., ICLR’16]

Max pooling – MAC descriptor

Input image conv5 filter 1 conv5 filter 2 …. conv5 filter i …. conv5 filter K

maximum activation

[Razavian et al., MTA’16] [Tolias et al., ICLR’16]

Max pooling – MAC descriptor

[Razavian et al., MTA’16] [Tolias et al., ICLR’16] regions for top matching components different color per component pair 1 pair 3 pair 2

Generalized mean pooling – GeM descriptor

𝑞 → ∞ max pool (MAC) 𝑞 = 1 avg pool (SPoC) [Radenovic et al., PAMI’19]

Hybrid – R-MAC descriptor

MAC descriptor

[Tolias et al., ICLR’16]

Hybrid – R-MAC descriptor

MAC descriptor

[Tolias et al., ICLR’16]

Hybrid – R-MAC descriptor

MAC descriptor

[Tolias et al., ICLR’16]

Hybrid – R-MAC descriptor

MAC descriptor

[Tolias et al., ICLR’16]

Hybrid – R-MAC descriptor

MAC descriptor

• Normalize descriptors
• Sum aggregate

[Tolias et al., ICLR’16]

Performance comparison

10 20 30 40 50 60 70 80 ResNet101 pre-trained ResNet101 fine-tuned SPoC CroW MAC GeM R-MAC

Precision@10 on R-Oxford+1M distractors

Performance comparison

10 20 30 40 50 60 70 80 ResNet101 pre-trained ResNet101 fine-tuned SPoC CroW MAC GeM R-MAC

Precision@10 on R-Oxford+1M distractors Fine-tuning improvement for GeM: +26.6%

Attention maps

Descriptor Image similarity [Kim et al., CVPR’17]

image from Kim et al.

learned attention: 83.2 vs CroW attention : 80.1 NetVLAD top-1 accuracy, San Francisco learned attention : 75.2 vs no attention : 71.8 NetVLAD top-1 accuracy, Tokyo 24/7

DELF

Descriptor Image similarity [Noh et al., ICCV’17]

image from Noh et al.

DELF

[Noh et al., ICCV’17]

Training time:

• Aggregate local activations into global descriptor
• Global descriptor in the loss

Test time:

• No aggregation into global descriptor
• Index strongest local descriptors separately

image from Noh et al.

mAP on Oxford5k Global descriptor 77.9 Voting + SP 83.8

(not in the paper, thanks to authors for sharing)

Training loss

Loss function for N-way classification

cross entropy loss Discrete labels  difficult to obtain at instance-level FC layer soft-max global descriptor

Loss functions for metric learning

anchor negative positive

Contrastive loss Triplet loss

far enough as close as possible large enough

• Sampling from discrete class labels
• problem: large intra-class variability
• Need automatic ways for pair-wise labels

Average precision loss

The larger the batch the better  no need to sample [Revaud et al., ICCV’19]

Average Precision Loss

[Revaud et al., ICCV’19]

Training data

Training data from GPS: negatives

anchor candidate negatives

Training data from GPS: negatives

anchor candidate positives camera orientation (unkown)

Training data from GPS: negatives

anchor candidate positives camera orientation (unkown)

Descriptor distance to resolve:  pick the closest [Arandjelovic et al., CVPR’16]

Training data from SfM

7.4M images  713 training 3D models [Schonberger et al. CVPR’15] [Radenovic et al. CVPR’16]

Training data from SfM

camera orientation known number of inliers known

7.4M images  713 training 3D models [Schonberger et al. CVPR’15] [Radenovic et al. CVPR’16]

Training data from SfM: hard negatives

anchor

Negative examples: images from different 3D models than the query Hard negatives: closest negative examples to the query [Radenovic et al. PAMI’19]

Training data from SfM: hard negatives

anchor the most similar CNN descriptor

Negative examples: images from different 3D models than the query Hard negatives: closest negative examples to the query [Radenovic et al. PAMI’19]

Training data from SfM: hard negatives

anchor the most similar CNN descriptor naive hard negatives top k by CNN

Negative examples: images from different 3D models than the query Hard negatives: closest negative examples to the query

increasing CNN descriptor distance to the query

[Radenovic et al. PAMI’19]

Training data from SfM: hard negatives

anchor the most similar CNN descriptor naive hard negatives top k by CNN

Negative examples: images from different 3D models than the query Hard negatives: closest negative examples to the query

increasing CNN descriptor distance to the query

[Radenovic et al. PAMI’19]

Training data from SfM: hard negatives

anchor the most similar CNN descriptor naive hard negatives top k by CNN diverse hard negatives top k: one per 3D model

Negative examples: images from different 3D models than the query Hard negatives: closest negative examples to the query

increasing CNN descriptor distance to the query

[Radenovic et al. PAMI’19]

Training data from SfM: hard positives

anchor

Positive examples: images that share 3D points with the query Hard positives: positive examples not close enough to the query

[Radenovic et al. PAMI’19]

Training data from SfM: hard positives

anchor top 1 by CNN

Positive examples: images that share 3D points with the query Hard positives: positive examples not close enough to the query

[Radenovic et al. PAMI’19]

Training data from SfM: hard positives

anchor top 1 by CNN

Positive examples: images that share 3D points with the query Hard positives: positive examples not close enough to the query

[Radenovic et al. PAMI’19]

Training data from SfM: hard positives

anchor top 1 by CNN top 1 by inliers

harder positives

Positive examples: images that share 3D points with the query Hard positives: positive examples not close enough to the query

[Radenovic et al. PAMI’19]

Training data from SfM: hard positives

anchor top 1 by CNN top 1 by inliers random from top k by inliers

harder positives

Positive examples: images that share 3D points with the query Hard positives: positive examples not close enough to the query

[Radenovic et al. PAMI’19]

Positive and negative training images

[Radenovic et al. PAMI’19]

Positive and negative training images

Oxford 5k Paris 6k

Off-the-shelf

44.2 51.6

[Radenovic et al. PAMI’19]

Positive and negative training images

Oxford 5k Paris 6k

Off-the-shelf top 1 CNN + top k CNN

44.2 51.6 56.2 63.1

[Radenovic et al. PAMI’19]

Positive and negative training images

Oxford 5k Paris 6k

Off-the-shelf top 1 CNN + top k CNN

44.2 51.6 56.2 63.1

[Radenovic et al. PAMI’19]

Positive and negative training images

Oxford 5k Paris 6k

Off-the-shelf top 1 CNN + top k CNN top 1 CNN + top 1 / model CNN

44.2 51.6 56.2 63.1 56.7 63.9

[Radenovic et al. PAMI’19]

Positive and negative training images

Oxford 5k Paris 6k

Off-the-shelf top 1 CNN + top k CNN top 1 CNN + top 1 / model CNN top 1 inliers + top 1 / model CNN

44.2 51.6 56.2 63.1 56.7 63.9 59.7 67.1

[Radenovic et al. PAMI’19]

Positive and negative training images

Oxford 5k Paris 6k

Off-the-shelf top 1 CNN + top k CNN top 1 CNN + top 1 / model CNN top 1 inliers + top 1 / model CNN random(top k inliers) + top 1 / model CNN

44.2 51.6 56.2 63.1 56.7 63.9 59.7 67.1 60.2 67.5

[Radenovic et al. PAMI’19]

Class labels + cleaning

[Gordo et al. IJCV’18]

Use classical computer vision to collect training data:  Bag-of-Words and spatial verification

Class labels + cleaning

[Gordo et al. IJCV’18]

Class labels + cleaning

[Gordo et al. IJCV’18]

classification loss vs ranking loss

PlaNet

adaptive partitioning into k=26,263 cells N-way classification training [Weyand et al., ICCV’17] Very compact model (377 MB)! But is it better than instance search?

Revisiting IM2GPS

[Vo et al., CVPR’17]

• A. Classification with globe partitioning
• best at coarse level, bad at fine level
• very compact model

Evaluation at different scales IM2GPS dataset Fine street (1km) city (25km) Coarse scale region (250km) country (750km) continent (7500km)

Revisiting IM2GPS

[Vo et al., CVPR’17]

• A. Classification with globe partitioning
• best at coarse level, bad at fine level
• very compact model
• B. Descriptors from A used for instance search
• improves for fine level
• all descriptors in memory

Evaluation at different scales IM2GPS dataset Fine street (1km) city (25km) Coarse scale region (250km) country (750km) continent (7500km)

Revisiting IM2GPS

[Vo et al., CVPR’17]

• A. Classification with globe partitioning
• best at coarse level, bad at fine level
• very compact model
• B. Descriptors from A used for instance search
• improves for fine level
• all descriptors in memory
• C. Fine-tuning A with ranking loss, use for instance search
• no improvements
• high intra class variability / not challenging pairs

Evaluation at different scales IM2GPS dataset Fine street (1km) city (25km) Coarse scale region (250km) country (750km) continent (7500km)

Revisiting IM2GPS

[Vo et al., CVPR’17]

• A. Classification with globe partitioning
• best at coarse level, bad at fine level
• very compact model
• B. Descriptors from A used for instance search
• improves for fine level
• all descriptors in memory
• C. Fine-tuning A with ranking loss, use for instance search
• no improvements
• high intra class variability / not challenging pairs
• D. Global descriptor (MAC) trained with SfM data [Radenovic et al.]
• the best for fine level
• all descriptors in memory

Evaluation at different scales IM2GPS dataset Fine street (1km) city (25km) Coarse scale region (250km) country (750km) continent (7500km)

Google landmark recognition challenge

Combining global GeM with local DELF-ASMK

GeM-based recognition

GeM common mistakes

GeM common mistakes

Recognition with ASMK-DELF + spatial verification

Spatial verification – common mistakes

Spatial verification – common mistakes

Combined classifier

23.3 21.1 29.3 5 10 15 20 25 30 35 GeM ASMK-DELF+SP Combined

Global descriptor vs voting approach

DELF with ASMK*+SP

• Prec@10: 81.1

GeM fine-tuned with SfM, D = 512

• Prec@10: 68.6

R-Oxford+1M distractors [Radenovic et al., CVPR’18]

Global descriptor vs voting approach

DELF with ASMK*+SP

• Prec@10: 81.1
• DB mem. (GB): 10.3

GeM fine-tuned with SfM, D = 512

• Prec@10: 68.6
• DB mem. (GB): 1.92

R-Oxford+1M distractors [Radenovic et al., CVPR’18]

Global descriptor vs voting approach

DELF with ASMK*+SP

• Prec@10: 81.1
• DB mem. (GB): 10.3

GeM fine-tuned with SfM, D = 512

• Prec@10: 68.6
• DB mem. (GB): 1.92

R-Oxford+1M distractors [Radenovic et al., CVPR’18]

Global descriptor vs voting approach

DELF with ASMK*+SP

• Prec@10: 81.1
• DB mem. (GB): 10.3
• Query time, extraction + search (sec): 0.42 (GPU) + 0.98 (CPU)

GeM fine-tuned with SfM, D = 512

• Prec@10: 68.6
• DB mem. (GB): 1.92
• Query time, extraction + search (sec): 0.23 (GPU) + 0.56 (CPU)

R-Oxford+1M distractors [Radenovic et al., CVPR’18]

Global descriptors & quantization

fine-tuned R-MAC ResNet101 Uncompressed (8192 bytes): 82.8 [Gordo et al. IJCV’18] Product Quantization (PQ) [Jegou et al., PAMI’10]

Descriptor whitening

Post-processing with whitening

process & aggregate post-processing (whitening) local descriptor set global descriptor global descriptor

Post-processing with whitening

process & aggregate post-processing (whitening) local descriptor set global descriptor global descriptor pre-trained

Post-processing with whitening

process & aggregate post-processing (whitening) local descriptor set global descriptor global descriptor learned end-to-end

Descriptor processing with PCA

eigen-vectors as columns global descriptor mean vector

PCA and power-law normalization

jointly down-weigh co-occurring features

[Perronnin et al., CVPR’10]

PCA whitening

[Jegou & Chum, ECCV’12]

jointly down-weigh co-occurring features

PCA whitening with shrinkage

jointly down-weigh co-occurring features

[Mukundan et al., IJCV’19]

Supervised whitening

Covariance from matching and non-matching pairs Obtain the whitening from covariance of matching

• whitening matrix:

Obtain the rotation by PCA on whitened covariance of non-matching: [Mikolajczyk & Matas, ICCV’07]

Results – descriptor post-processing

MAC 31.6 mAP on R-Oxford pre-trained ResNet101

Results – descriptor post-processing

MAC 31.6 PCA+power-law 40.1 mAP on R-Oxford pre-trained ResNet101

Results – descriptor post-processing

MAC 31.6 PCA+power-law 40.1 mAP on R-Oxford pre-trained ResNet101

Results – descriptor post-processing

MAC 31.6 PCA+power-law 40.1 PCA whitening 41.7 mAP on R-Oxford pre-trained ResNet101

Results – descriptor post-processing

MAC 31.6 PCA+power-law 40.1 PCA whitening 41.7 mAP on R-Oxford pre-trained ResNet101

Results – descriptor post-processing

MAC 31.6 PCA+power-law 40.1 PCA whitening 41.7 PCA wh.+shrink 43.5 mAP on R-Oxford pre-trained ResNet101

Results – descriptor post-processing

MAC 31.6 PCA+power-law 40.1 PCA whitening 41.7 PCA wh.+shrink 43.5 mAP on R-Oxford pre-trained ResNet101

Results – descriptor post-processing

MAC 31.6 PCA+power-law 40.1 PCA whitening 41.7 PCA wh.+shrink 43.5 supervised whitening 46.9 mAP on R-Oxford pre-trained ResNet101

Results – descriptor post-processing

MAC 31.6 PCA+power-law 40.1 PCA whitening 41.7 PCA wh.+shrink 43.5 supervised whitening 46.9 mAP on R-Oxford pre-trained ResNet101

Results – descriptor post-processing

MAC 31.6 PCA+power-law 40.1 PCA whitening 41.7 PCA wh.+shrink 43.5 supervised whitening 46.9 mAP on R-Oxford pre-trained ResNet101 SPoC 24.5 42.7

Results – descriptor post-processing

MAC 31.6 PCA+power-law 40.1 PCA whitening 41.7 PCA wh.+shrink 43.5 supervised whitening 46.9 mAP on R-Oxford pre-trained ResNet101 SPoC 24.5 42.7 GeM 31.6 50.1

Results – descriptor post-processing

MAC 31.6 PCA+power-law 40.1 PCA whitening 41.7 PCA wh.+shrink 43.5 supervised whitening 46.9 mAP on R-Oxford pre-trained ResNet101 SPoC 24.5 42.7 GeM 31.6 50.1

Results – descriptor post-processing

MAC 31.6 PCA+power-law 40.1 PCA whitening 41.7 PCA wh.+shrink 43.5 supervised whitening 46.9 mAP on R-Oxford pre-trained ResNet101 SPoC 24.5 42.7 GeM 31.6 50.1 fine-tuned ResNet101 GeM 52.9 64.1

Results – descriptor post-processing

MAC 31.6 PCA+power-law 40.1 PCA whitening 41.7 PCA wh.+shrink 43.5 supervised whitening 46.9 mAP on R-Oxford pre-trained ResNet101 SPoC 24.5 42.7 GeM 31.6 50.1 fine-tuned ResNet101 GeM 52.9 64.1 some pooling operations benefit more than others additional insight about whitening in [Mukundan et al. IJCV’19]

Post-processing with whitening

process & aggregate post-processing (whitening) local descriptor set global descriptor global descriptor learned end-to-end

Post-processing with whitening

process & aggregate FC (whitening) local descriptor set global descriptor global descriptor learned end-to-end

https://github.com/filipradenovic/cnnimageretrieval-pytorch

Summary

Instance search for place recognition Better than classification-based approaches Classification loss can handle large intra-class variability ranking loss needs selection of clean & challenging pairs Global descriptors vs voting approaches speed/compactness vs better accuracy Surprising effectiveness of descriptor whitening

Privacy preserving search

Tolias, Radenovic, Chum: Targeted mismatch adversarial attacks. ICCV 2019

Private content – personal data

Access to place recognition / retrieval systems  sharing of private content

Concealed queries

Original queries

Concealed queries

Original queries Concealed queries

Adversarial attacks

+ = plane plane Non-targeted misclassification

Adversarial attacks

+ = plane kangaroo Targeted misclassification

Adversarial attacks

+ = dissimilar descriptors Non-targeted mismatch

Adversarial attacks

+ = similar descriptors Targeted mismatch carrier attack target perturbation concealed query initial query

Typical descriptor for retrieval

Targeted mismatch attacks

Known: FCN, Pooling & Normalization (PN), Re-sampling Unknown: Post-processing

Targeted mismatch attacks

Known: FCN, Re-sampling Unknown: Pooling & Normalization (PN), Post-processing

Targeted mismatch attacks

Known: FCN, Re-sampling Unknown: Pooling & Normalization (PN), Post-processing

Targeted mismatch attacks

Known: FCN Unknown: Re-sampling, Pooling & Normalization (PN), Post-processing multi-scale attack Key ingredient: blurring before re-sampling

target carrier 0.782

target carrier 0.782 1.000 GeM, λ=0

target carrier 0.782 1.000 GeM, λ=0 tens, λ=0 0.999

target carrier 0.782 1.000 GeM, λ=0 tens, λ=0 0.999

target carrier 0.782 1.000 GeM, λ=0 tens, λ=0 0.999 tens, λ=1 0.997

target carrier 0.782 1.000 GeM, λ=0 tens, λ=0 0.999 tens, λ=1 0.997

target carrier 0.782 1.000 GeM, λ=0 tens, λ=0 0.999 tens, λ=1 0.997

target carrier 0.782 1.000 GeM, λ=0 tens, λ=0 0.999 tens, λ=1 0.997

target carrier 0.782 1.000 GeM, λ=0 tens, λ=0 0.999 tens, λ=1 0.997 1.000 hist, λ=0

target carrier 0.782 1.000 GeM, λ=0 tens, λ=0 0.999 tens, λ=1 0.997 1.000 hist, λ=0

Unknown resolution and multi-scale attack

performance similarity to target

Unknown resolution and multi-scale attack

multi-scale attack with dense resolution sampling performance similarity to target

Unseen network

Targeted mismatch adversarial attacks

https://github.com/gtolias/tma Online code