Hardness-Aware Deep Metric Learning Wenzhao Zheng , Zhaodong Chen, - - PowerPoint PPT Presentation

Hardness-Aware Deep Metric Learning

Department of Automation, Tsinghua University, China http://ivg.au.tsinghua.edu.cn/

Wenzhao Zheng, Zhaodong Chen, Jiwen Lu and Jie Zhou IEEE CVPR 2019

Deep Metric Learning

2

• Group similar images together

Deep Metric Learning

3

• Contrastive Paradigm
• Triplet Paradigm

J(T(x, x+, x−)) = [D(x, x+) − D(x, x−) + m]+

J(T(xi, xj)) = 1{yi = yj}D2(xi, xj) + 1{yi 6= yj}[α D(xi, xj)]2

+

Deep Metric Learning

4

• Fewer and fewer hard negatives as training goes!
• Sampling is a vital component in DML
• Hard negative mining is widely applied
• The need for informative samples

Motivation

5

• Resources are going to be exhausted?
• Turn easy negatives to hard negatives
• Alchemy!

Intuition

6

• Hardness-Aware Augmentation
• Hardness-and-Label-Preserving Synthesis

feature embedding

e y−

y−

y+

y

ˆ z−

ˆ y−

e z−

z−

z+

z

What?

e y−

y−

y+

y

ˆ y−

y−

y+

y

ˆ y−

?

Proposed Approach

7

• Hardness-Aware Augmentation

Metric

A P N e N

N

• Metric
• A

P N e N

N

Proposed Approach

8

• Hardness-Aware Deep Metric Learning

CNN

Reconstruct

• Same

Category

• y

z

ˆ z

y0 y0 e y e y

• Label-preserving

Triplet loss, N-pair loss, … Hardness-preserving Employ the hardness-aware synthetics

• Generator objective:

Formulation

9

Jgen = Jrecon + λJsoft = c(Y, Y0) + λJsoft( e Y, L) = X

y2Y y02Y0

||y − y0||2 + λ X

e y2 e Y l2L

jsoft(e y, l)

Jmetric = e−

β Jgen Jm + (1 − e− β Jgen )Jsyn

= e−

β Jgen J(T) + (1 − e− β Jgen )J(e

T)

• Metric objective:

• CUB-200-2011: 11,788

images of 200 bird species

• Cars196: 16,185 images of

196 cars models

• Stanford Online Products:

120,053 images of 22,634 products from eBay.com

Experiments

10

Ablation Study

11

• Comparisons of different settings

Ablation Study

12

• Comparisons of using different pulling factors

Quantitative Results

13

• CUB-200-2011 Dataset

Quantitative Results

14

• Cars196 Dataset

Quantitative Results

15

• Stanford Online Products Dataset

Qualitative Results

16

• CUB-200-2011 Dataset

• The number of informative samples decreases gradually as

training proceeds

• The proposed HDML generates hardness-aware synthetics

as complements to the training data

• Apply
• ur

framework to the more general data augmentation problem

Summary and Future Work

17