and Hash Coding with Deep Neural Networks Presenter: MinKu Kang 1 - - PowerPoint PPT Presentation

Simultaneous Feature Learning and Hash Coding with Deep Neural Networks

Presenter: MinKu Kang

1

Why did I choose this paper?

• Efficient image retrieval via binary encoding of images
• efficient bitwise operations.
• space-efficient storage.
• There are many useful techniques which can also readily be used in
• ther research fields.
• An advanced Neural Net (shared network) structure is used from

which I can learn a lot.

2

Background - Similarity-Preserving Hashing

3

Related Work

Two-Stage Framework

Learn binary hash codes Learn Binary hashing functions

4

Decomposing Similarity Matrix

5

Learning Hash Functions

Used as ground truth

6

Two Stage Framework

Analytical

7

Optimization

Cost Function

8

Augmenting Output Layer with Binary Class Labels

9

Dataset for Experiments

10

Results on CIFAR-10

For 48 bits hash codes

11

For 48 bits hash codes

Results on NUS-WIDE

12

as much as two orders of magnitude

13

Related Work – Metric Learning Based Hashing

Haomiao et al., Deep Supervised Hashing for Fast Image Retrieval, CVPR 2016

14

Similarity-Preserving Loss Function

Loss for similar pairs Loss for dissimilar pairs

…

15

Relaxation of the Loss Function

But they didn’t use sigmoid layer because it slows down the convergence Typically, this layer is replaced by a sigmoid activation layer for binary-like outputs A regularizer encouraging the output values in the vicinity of range (-1 ~ 1) Weaker constraints compared to [0, 1]-sigmoid layer, but shows better performance.

16

More peaked, More binary-like # values in the Output layer Output value

Effect of Regularizer

Results with sigmoid output layer

17

Effect of Regularizer

Retrieval performance (mAP) of models under different settings of parameters

Results on CIFAR-10

18

19

Background - Metric Learning

Siamese Network Triplet Network x is more similar to x+ than to x- (high-order relationship) Similarity of x1 and x2 (pairwise)

Triplet Loss based Network

less-similar

20

Hanjiang et al., Simultaneous Feature Learning and Hash Coding with Deep Neural Networks, CVPR 2015

Pairwise versus Triplet Ranking

21

Pairwise Similarity Triplet Ranking Similar Dissimilar More-similar Less-similar query

𝐽 𝐽+ 𝐽−

𝑗𝑛𝑏𝑕𝑓 𝐽 𝑗𝑡 𝒏𝒑𝒔𝒇 𝒕𝒋𝒏𝒋𝒎𝒃𝒔 𝑢𝑝 𝑗𝑛𝑏𝑕𝑓 𝐽+ 𝑢ℎ𝑏𝑜 𝑢𝑝 𝑗𝑛𝑏𝑕𝑓𝐽−

22

Training Architecture

CNN

. . .

CNN

. . .

CNN

. . .

𝐽 𝐽+ 𝐽− 𝐺(𝐽) 𝐺(𝐽+) 𝐺(𝐽−)

Weights are shared. Weights are shared. Triplet Ranking Loss Sigmoid activation layer restricts the output values in the range [0, 1].

Triplet Ranking Loss

23

• 1

loss The term

24

CNN

. . .

CNN

. . .

CNN

. . .

𝐽 𝐽+ 𝐽− 𝐺(𝐽) 𝐺(𝐽+) 𝐺(𝐽−)

Weights are shared. Weights are shared. Triplet Ranking Loss

Training Architecture

𝑋

𝑗𝑘

𝑋

𝑗𝑘

𝑋

𝑗𝑘

𝜖𝑋

𝑗𝑘

𝜖 𝑋

𝑗𝑘 = 𝑋 𝑗𝑘 + 𝛽

Weight Update

25

𝜖𝑚 𝐺 𝐽 , 𝐺 𝐽+ , 𝐺 𝐽− 𝜖𝑋

𝑗𝑘

= 𝜖𝑚 𝐺 𝐽 , 𝐺 𝐽+ , 𝐺 𝐽− 𝜖𝐺 𝐽 ∗ 𝜖𝐺 𝐽 𝜖 … ∗ 𝜖 … 𝜖𝑋

𝑗𝑘

+ 𝜖𝑚 𝐺 𝐽 , 𝐺 𝐽+ , 𝐺 𝐽− 𝜖𝐺 𝐽+ ∗ 𝜖𝐺 𝐽+ 𝜖 … ∗ 𝜖 … 𝜖𝑋

𝑗𝑘

+ 𝜖𝑚 𝐺 𝐽 , 𝐺 𝐽+ , 𝐺 𝐽− 𝜖𝐺 𝐽− ∗ 𝜖𝐺 𝐽− 𝜖 … ∗ 𝜖 … 𝜖𝑋

𝑗𝑘

Updating 𝑋

𝑗𝑘 requires

values from the three networks !

𝜖𝑚 𝐺 𝐽 , 𝐺 𝐽+ , 𝐺 𝐽− 𝜖𝑋

𝑗𝑘

= 𝒈(𝑶𝑶, 𝑶𝑶+ , 𝑶𝑶−) Analytically differentiated.

26

Weight Update

Updating 𝑋

𝑗𝑘 requires

values from the three networks !

𝜖𝑚 𝐺 𝐽 , 𝐺 𝐽+ , 𝐺 𝐽− 𝜖𝑋

𝑗𝑘

= 𝒈(𝑶𝑶, 𝑶𝑶+ , 𝑶𝑶−)

• We need three forward-propagations for each training triplet.
• We need to maintain the three weight-shared copies of the network in the memory
• Weight update is computationally expensive compared to a typical single network structure.
• Possible combinations of triplets given a training set are many.
• There is a dedicated paper (*) – training time improvement about two-orders of magnitude.

* Bohan et al, Fast Training of Triplet-based Deep Binary Embedding Networks, CVPR 2016 .

Divide-and-Encode Module

27

Enforces Independency property

• Each hash bit is generated from a separated slice of features
• the output hash codes may be less redundant to each other.
• No Mathematical Proof.

hash codes hash codes

28

Overall Structure

Quantization Divide-and-Encode CNN Input Image

In test time, a trained single network is used

29

Results on SVNH

Results on CIFAR10

30

31

Results on NUS-WIDE

Divide-and-Encode versus Fully-Connected-Encode

32

DSH(pairwise) verse DNNH(triplet)

33

Triplet Pairwise In DSH paper, they said they implemented DNNH themselves. In DSH paper, they said divide and encode structure largely degraded the retrieval mAP on CIFAR-10 Training inefficiencies in Training Triplet Network may have resulted in inferior performance

Conclusion

• While Triplet Network can learn higher-order relationship between

training samples, there are training inefficiencies

• Practically, pairwise metric learning based method shows better

performance

• Efficient sampling strategies for triplets are needed.
• Solving training inefficiencies in Triplet Network could be a key for

better results

• End-to-End architecture is preferred.

34

References & Acknowledgement

• RongkaiXiae et al., Supervised Hashing for Image Retrieval via Image

Representation Learning

• Haomiao et al., Deep Supervised Hashing for Fast Image Retrieval
• Hanjiang et al., Simultaneous Feature Learning and Hash Coding with

Deep Neural Networks

35

Quiz

• 1. What is the advantage of Triplet Network over Pairwise

Network? a) fast training speed b) low complexity of the architecture c) capturing high-order relationships between training samples

• 2. Why did the authors design the Divide and Encode Module?

a) to enhance the training speed b) to enforce the independency property between hash functions c) to lower down the complexity of the problem

36