HARDNET: CONVOLUTIONAL NETWORK FOR LOCAL IMAGE DESCRIPTION A n a - - PowerPoint PPT Presentation
HARDNET: CONVOLUTIONAL NETWORK FOR LOCAL IMAGE DESCRIPTION A n a s t a s i i a Mi s h c h u k , D m y t r o Mi s h k i n , F i l i p R a d e n o v i c J i r i Ma t a s OUTLINE Short review
A n a s t a s i i a Mi s h c h u k , D m y t r
s h k i n , F i l i p R a d e n
i c J i r i Ma t a s
Short review of methods for learning
The L2-Net HardNet loss and architecture Benchmarks
2
3
D i s c r i mi n a n t L e a r n i n g
L
a l I ma g e D e s c r i p t
s B r
n e t a l , P A MI 2 1
3 s e t s , 4 k p a t c h e s e a c h :
i b e r t y ( s h
n )
r e d a me
e mi t e S i z e : 6 4 x 6 4 , g r a y s c a l e . O b t a i n e d f r
S f M mo d e l , 3 D p
n t → D
k e y p
n t s U s e d i n a l l l e a r n e d d e s c r i p t
s me a n t i
e d i n t h i s p r e s e n t a t i
4
CONVEXOPT (SIMONYAN ET AL, 2012)
Global margin loss
S i mo n y a n e t a l , E C C V 2 1 2
C
v e x
t i mi z a t i
p r
l e m
5
MATCHNET
H a n e t a l , C V P R 2 1 5 . Wo r k s w e l l , b u t r e l y
me t r i c n e t w
k . A p p r
i ma t e k N N me t h
s , e . g . F L A N N c a n n
b e a p p l i e d d i r e c t l y
7x7 Conv pad 1
64 24
ReLU
1 24
5x5 Conv pad 2
64
ReLU
64
3x3 Conv pad 1 ReLU
64
2x2 MP/2
32 32
2x2 MP/2
16 96 16
3x3 Conv pad 1 ReLU
96 16
3x3 Conv pad 1 ReLU
64 16
3x3 MP/2
64 8
8x8 Conv ReLU
1 128
1x1 Conv ReLU
1 256
1x1 Conv ReLU
1 256
1x1 Conv
Sofumax
1 2
6
DEEPCOMPARE
Z a g
u y k
n d K
d a k i s , C V P R 2 1 5 Wo r k s w e l l , b u t r e l y
me t r i c n e t w
k . A p p r
i ma t e k N N me t h
s , e . g . F L A N N c a n n
b e a p p l i e d d i r e c t l y
7x7 Conv pad 3
64 96
ReLU
1 96
5x5 Conv pad 2
192
ReLU
192
3x3 Conv pad 1 ReLU
64
2x2 MP/2
32 32
2x2 MP/2
16 256 16 8
8x8 Conv ReLU
1 1 256
1x1 Conv ReLU 256 1x1 Conv
Sigmoid 1
2x2 MP/2
256
9 S i mo
e r r a e t a l , I C C V 2 1 5 . B a l n t a s e t a l , B MV C 2 1 6 32 32
7x7 Conv
26
TanH
1
2x2 MP/2
13
6x6 Conv TanH
8 64
8x8 Conv TanH
1 128 32
( B a l n t a s e t a l , 2 1 6 )
v e n s h a l l
e r a n d f a s t e r C N N ,
a r d
e g a t i v e mi n i n g : b y a n c h
s w a p i n t r i p l e t .
r i p l e t ma r g i n l
s
L 2 d i s t a n c e 1 64
7x7 Conv
58 32
TanH 2x2 L2pool/2
29
6x6 Conv TanH
23 64
5x5 Conv TanH
4 128 32
3x3 L2Pool/3 8
64
4x4 L2Pool/4 1
128
( S i mo
e r r a e t a l , 2 1 5 ) R e l a t i v e l y s h a l l
a n d f a s t C N N . H a r d n e g a t i v e mi n i n g : C
t r a s t i v e l
s
L 2 d i s t a n c e
1
D e s c r . # l a y e r s w/ p a r a ms L
s Ha r d mi n i n g K d
r e e r e a d y C
v e x O p t 1 G l
a l ma r g i n
D e e p D e s c 3 C
t r a s t i v e + + T F e a t 3 T r i p l e t ma r g i n + /
Ma t c h N e t 8 C r
s e n t r
y
e e p C
5 H i n g e
a l n t a s e t a l , B MV C 2 1 6
1 1
32 32 16 16
3x3 Conv pad 1
32 32
BN + ReLU
1
3x3 Conv pad 1
32
BN + ReLU 3x3 Conv pad 1 /2
64
BN + ReLU 3x3 Conv pad 1
64
BN + ReLU 3x3 Conv pad 1 /2 BN + ReLU
8 128
3x3 Conv pad 1 BN + ReLU
8 128
8x8 Conv BN+ L2Norm
1 128
1 3
S
t ma x
e r r
/ c
u mn
d i s t a n c e ma t r i x
1 4
S
t ma x
e r r
/ c
u mn
d i s t a n c e ma t r i x P e n a l t y
d e s c r i p t
c
e n t s c
r e l a t i
1 5
S
t ma x
e r r
/ c
u mn
d i s t a n c e ma t r i x S
t ma x
e r r
/ c
u mn
d i s t a n c e ma t r i x
i n t e r me d i a t e f e a t u r e s P e n a l t y
d e s c r i p t
c
e n t s c
r e l a t i
1 6
Triplet margin loss for hard negative P e n a l t y
d e s c r i p t
c h a n n e l s c
r e l a t i
S
t ma x
e r r
/ c
u mn
d i s t a n c e ma t r i x
i n t e r me d i a t e f e a t u r e s
1 7
3x3 Conv pad 1
32 32
BN + ReLU
1
3x3 Conv pad 1
32
BN + ReLU 3x3 Conv pad 1 /2
64
BN + ReLU 3x3 Conv pad 1
64
BN + ReLU 3x3 Conv pad 1 /2 BN + ReLU
8 128
3x3 Conv pad 1 BN + ReLU
8 128
8x8 Conv BN+ L2Norm
1 128
1 8
1 9
D e s c r . # l a y e r s w/ p a r a ms L
s Ha r d mi n i n g K d
r e e r e a d y C
v e x O p t 1 G l
a l ma r g i n
D e e p D e s c 3 C
t r a s t i v e + + T F e a t 3 T r i p l e t ma r g i n + /
Ma t c h N e t 8 C r
s e n t r
y
e e p C
5 H i n g e
2 N e t 7 S
t Ma x + + Ha r d N e t 7 T r i p l e t ma r g i n + +
2
2 1
2 2
N
r a d i e n t f r
n e g a t i v e e x a mp l e S ma l l g r a d i e n t s
2 3
Contrastive Softmax (L2Net) Triplet margin FPR, Brown Yos 0.009 0.009 0.006 mAUC, W1BS 0.072 0.083 0.083 mAUC, HP-T 0.153 0.157 0.164
2 4
2 5
2 6
Mi s h k i n e t a l , B MV C 2 1 5 N u i s a n c e f a c t
: A p p e a r a n c e G e
t r y L i g h t i n g S e n s
2 7
D
, H e s s i a n , H a r r i s – i n r e f . i ma g e ~ 1 3 p a t c h e s p e r i ma g e k e p t . R e p r
e c t e d t
h e r i ma g e s w i t h 3 l e v e l s
“ a ffjn e f r a me n
s e ” a d d e d V : 5 7 i ma g e s i x p l e t s – p h
t r i c c h a n g e s I : 5 9 i ma g e s i x p l e t s – g e
t r i c c h a n g e s B a l n t a s e t a l , C V P R 2 1 7
2 8
2 9
D a t a s e t s a r e a l r e a d y s a t u r a t e d
O n p a r wi t h R
S I F T S t i l l c h a l l e n g i n g d u e t
l t i p l e n u i s a n c e f a c t
s
Z i t n i c k a n d R a mn a t h , 2 1 1 , Mi s h k i n e t a l 2 1 5 , Mi k
a j c z y k e t a l . 2 1 3 , H a u a g g e a n d S n a v e l y , 2 1 2 , K e l ma n e t a l , 2 7 , F e r n a n d
t a l . 2 1 4
3
P h i l b i n e t a l 2 7 , P h i l b i n e t a l 2 8
3 1
PDF: https://arxiv.org/abs/1705.10872 Source and models: https://github.com/DagnyT/hardnet 3 2