SLIDE 1
Deep Semantic Ranking Based Hashing for Multi-Label Image Retrieval
Presenter: MinKu Kang
1
Fang Zhao, et al. CVPR 2015
2
Hashing for Multi-Label Image Retrieval Fang Zhao, et al. CVPR - - PowerPoint PPT Presentation
Hashing for Multi-Label Image Retrieval Fang Zhao, et al. CVPR - - PowerPoint PPT Presentation
Deep Semantic Ranking Based Hashing for Multi-Label Image Retrieval Fang Zhao, et al. CVPR 2015 Presenter: MinKu Kang 1 Previous Presentation Presented by Youngki Kwon 2 Introduction Ranking Based Image Retrieval Similarity based on
SLIDE 2
SLIDE 3
3
Introduction – Ranking Based Image Retrieval
Similarity based on # common labels
SLIDE 4
Previous Work - Metric Learning
4
Pairwise Similarity Similar Dissimilar
Assumed each image contains a single representative label
SLIDE 5
5
Previous Work - Metric Learning
Assumed each image contains a single representative label
SLIDE 6
6
Previous Work – Triplet Network
CNN
. . .
CNN
. . .
CNN
. . .
𝐽 𝐽+ 𝐽− 𝐺(𝐽) 𝐺(𝐽+) 𝐺(𝐽−)
Weights are shared. Weights are shared. Triplet Ranking Loss
Assumed each image contains a single representative label
SLIDE 7
Multi-label based Ranking
7
Count the number of common labels
more-similar less-similar
SLIDE 8
8
𝑠
1 = 3
𝑠
2 = 2
𝐵𝑜𝑑ℎ𝑝𝑠
𝑢𝑠𝑓𝑓, 𝑡𝑣𝑜, 𝑡𝑙𝑧 𝑢𝑠𝑓𝑓, 𝑡𝑣𝑜, 𝑡𝑙𝑧 𝑢𝑠𝑓𝑓, 𝑡𝑙𝑧
Ranking Score
Similar dissimilar
SLIDE 9
9
𝑠
1 = 3
𝑠
2 = 2
𝐵𝑜𝑑ℎ𝑝𝑠
𝑢𝑠𝑓𝑓, 𝑡𝑣𝑜, 𝑡𝑙𝑧 𝑢𝑠𝑓𝑓, 𝑡𝑣𝑜, 𝑡𝑙𝑧 𝑢𝑠𝑓𝑓, 𝑡𝑙𝑧
Triplet Loss Function
Similar dissimilar
SLIDE 10
10
𝑠
1 = 3
𝑠
2 = 2
𝑠
3 = 0 𝑢𝑠𝑓𝑓, 𝑡𝑣𝑜, 𝑡𝑙𝑧 𝑢𝑠𝑓𝑓, 𝑡𝑣𝑜, 𝑡𝑙𝑧 𝑢𝑠𝑓𝑓, 𝑡𝑙𝑧 𝑐𝑣𝑗𝑚𝑒𝑗𝑜, 𝑑𝑏𝑠 𝑠
1 = 3
𝒙𝒊𝒇𝒐 𝒋 = 𝟐 𝑠
2 = 2
𝑠
3 = 0 𝑢𝑠𝑓𝑓, 𝑡𝑣𝑜, 𝑡𝑙𝑧 𝑢𝑠𝑓𝑓, 𝑡𝑙𝑧
𝑐𝑣𝑗𝑚𝑒𝑗𝑜, 𝑑𝑏𝑠 𝑠
1 = 3
𝒙𝒊𝒇𝒐 𝒋 = 𝟑
Possible Triplets
Constructing Triplets
𝒓 𝒚𝒋 𝒚𝒌 𝒚𝒌 Loss for an anchor =
SLIDE 11
Final Loss Function
11
Loss for an anchor Loss for all anchors
SLIDE 12
12
Final Loss Function – Regularizers
: Encourages each bit averaged over the training data to be mean-zero : Penalized large weights
…
SLIDE 13
Skipping Layer
13
Bypassing Connection Skipping Layer
Utilize diverse feature information biased toward visual appearance
SLIDE 14
14
Additional Relaxations
discontinuous smooth, differentiable Relaxation
SLIDE 15
15
Additional Relaxations
discontinuous smooth, differentiable Relaxation
But, the sigmoid function had a bad influence on the convergence of the network Many Deep Learning Libraries support automatic, symbolic differentiations
SLIDE 16
16
Additional Modification on Loss Function
𝑠
1 = 3
𝑠2 = 2 𝑠
1 = 3
𝑠3 = 1 Higher weight Ranking Discrepancy
SLIDE 17
17
Experiments – Multi-Labeled Dataset
MIRFLICKR-25K : Multi-label(24) images from social photography website NUS-WIDE: Multi-label(81) images
SLIDE 18
18
Experiments - Measure
For top-p retrieved images
Average Ranking
SLIDE 19
19
MIRFLICKR-25K NUS-WIDE
Experimental Results
SLIDE 20
20
Experimental Results – Effect of Skipping Layer / Weighting Scheme
MIRFLICKR-25K NUS-WIDE
SLIDE 21
Summary
21
SLIDE 22
1. What is the most appropriate role of the bypassing connection? a) to increase the capacity of the network b) to utilize the diverse feature information c) to make the training procedure efficient. d) to prevent the overfitting training dataset 2. Choose the term which is not included in the final loss function. a) mean-zero relaxation b) sign function => sigmoid function term c) large-weight penalizing term d) L1 regularization term for the hash codes
22
Quiz
SLIDE 23
- Deep Semantic Ranking Based Hashing for Multi-Label Image
Retrieval, Fang Zhao, et al., CVPR 2015
23