Recurrent Transformer Networks for Semantic Correspondence - - PowerPoint PPT Presentation

Neural Information Processing Systems (NeurIPS) 2018

Recurrent Transformer Networks for Semantic Correspondence

Seungryong Kim1, Stepthen Lin2, Sangryul Jeon1, Dongbo Min3, Kwanghoon Sohn1

• Dec. 05, 2018

1) 2) 3)

Semantic Correspondence

• Establishing dense correspondences between semantically similar images,

i.e., different instances within the same object or scene categories

• For example, the wheels of two different cars, the body of people or animals

2

Seungryong Kim et al., Recurrent Transformer Networks for Semantic Correspondence, NeurIPS, 2018

In Introduction

Challenges in Semantic Correspondence

3

Seungryong Kim et al., Recurrent Transformer Networks for Semantic Correspondence, NeurIPS, 2018

In Introduction

Pho hotometric De Deformations

• Intra-class appearance and

attribute variations

• Etc.

Ge Geometric De Deformations

• Different viewpoint or baseline
• Non-rigid shape deformations
• Etc.

Lac Lack of

• f Sup

Supervis ision

• Labor-intensive of annotation
• Degraded by subjectivity
• Etc.

?

Objective How to estimate locally-varying affine transformation fields without ground-truth supervision?

4

Seungryong Kim et al., Recurrent Transformer Networks for Semantic Correspondence, NeurIPS, 2018

Problem Formulation

𝑗 𝐔𝑗 = 𝐁𝑗, 𝐠𝑗 𝑗′ = T𝑗𝑗

Methods for Geometric Invariance in Feature Extraction Step

• UCN [Choy et al., NeurIPS’16]
• CAT-FCSS [Kim et al., TPAMI’18]
• Etc.

5

Seungryong Kim et al., Recurrent Transformer Networks for Semantic Correspondence, NeurIPS, 2018

Background

Spatial Transformer Networks (STNs)-based methods [Jaderberg et al., NeurIPS’15] 𝐁𝑗 is learned wo/𝐁𝑗

∗

But, 𝐠𝑗 is learned w/𝐠𝑗

∗

Geometric inference based on

• nly source or target image

Methods for Geometric Invariance in Regularization Step

• GMat. [Rocco et al., CVPR’17]
• GMat. w/Inl. [Rocco et al., CVPR’18]
• Etc.

6

Seungryong Kim et al., Recurrent Transformer Networks for Semantic Correspondence, NeurIPS, 2018

Background

𝐔𝑗 is learned wo/𝐔𝑗

∗

using self- or meta-supervision Geometric Inference using source/target images Globally-varying geometric Inference only Only fixed, untransformed versions of the features

Networks Configuration

• To weaves the advantages of STN-based methods and geometric

matching methods by recursively estimating geometric transformation residuals using geometry-aligned feature activations

7

Seungryong Kim et al., Recurrent Transformer Networks for Semantic Correspondence, NeurIPS, 2018

Recurrent Transformer Networks (RTNs)

Feature Extraction Networks

• Input images 𝐽𝑡 and 𝐽𝑢 are passed through Siamese convolution networks with

parameters 𝐗𝐺 such that 𝐸𝑗 = 𝐺 𝐽 𝐗𝐺

• Using CAT-FCSS, VGGNet (conv4-4), ResNet (conv4-23)

8

Seungryong Kim et al., Recurrent Transformer Networks for Semantic Correspondence, NeurIPS, 2018

Recurrent Transformer Networks (RTNs)

Recurrent Geometric Matching Networks

• Constraint correlation volume construction

𝐷(𝐸𝑗

𝑡, 𝐸𝑢(𝐔𝑘)) =< 𝐸𝑗 𝑡, 𝐸𝑢(𝐔𝑘) >/ < 𝐸𝑗 𝑡, 𝐸𝑢(𝐔𝑘) >2

9

Seungryong Kim et al., Recurrent Transformer Networks for Semantic Correspondence, NeurIPS, 2018

Recurrent Transformer Networks (RTNs)

Source Target

Recurrent Geometric Matching Networks

• Recurrent geometric inference

𝐔𝑗

𝑙 − 𝐔𝑗 𝑙−1 = 𝐺(𝐷(𝐸𝑗 𝑡, 𝐸𝑢(𝐔𝑗 𝑙−1))|𝐗𝐻)

10

Seungryong Kim et al., Recurrent Transformer Networks for Semantic Correspondence, NeurIPS, 2018

Recurrent Transformer Networks (RTNs)

Source Target

• Iter. 1
• Iter. 2
• Iter. 3
• Iter. 4

Weakly-supervised Learning

• Intuition: matching score between the source 𝐸𝑡 at each pixel 𝑗 and the target

𝐸𝑢(𝐔𝑗) should be maximized while keeping the scores of other candidates low

• Loss Function:

𝑀 𝐸𝑗

𝑡, 𝐸𝑢 𝐔

= − ෍

𝑘∈𝑁𝑗

𝑞𝑘

∗log(𝑞(𝐸𝑗 𝑡, 𝐸𝑢(𝐔𝑘)))

where the function 𝑞(𝐸𝑗

𝑡, 𝐸𝑢(𝐔𝑘)) is a Softmax probability

𝑞(𝐸𝑗

𝑡, 𝐸𝑢(𝐔𝑘)) =

exp(𝐷(𝐸𝑗

𝑡, 𝐸𝑢(𝐔𝑘)))

σ𝑚∈𝑁𝑗 exp(𝐷(𝐸𝑗

𝑡, 𝐸𝑢(𝐔𝑚)))

where 𝑞𝑘

∗ denotes a class label defined as 1 if 𝑘 = 𝑗, 0 otherwise

11

Seungryong Kim et al., Recurrent Transformer Networks for Semantic Correspondence, NeurIPS, 2018

Recurrent Transformer Networks (RTNs)

Results on the TSS Benchmark

12

Seungryong Kim et al., Recurrent Transformer Networks for Semantic Correspondence, NeurIPS, 2018

Experimental Results

Source images Target images SCNet

[Han et al., ICCV’17]

• GMat. w/Inl.

[Rocco et al., CVPR’18]

RTNs

Results on the PF-PASCAL Benchmark

13

Seungryong Kim et al., Recurrent Transformer Networks for Semantic Correspondence, NeurIPS, 2018

Experimental Results

Source images Target images SCNet

[Han et al., ICCV’17]

• GMat. w/Inl.

[Rocco et al., CVPR’18]

RTNs

Results on the PF-PASCAL Benchmark

14

Seungryong Kim et al., Recurrent Transformer Networks for Semantic Correspondence, NeurIPS, 2018

Experimental Results

Source images Target images SCNet

[Han et al., ICCV’17]

• GMat. w/Inl.

[Rocco et al., CVPR’18]

RTNs

15

Seungryong Kim et al., Recurrent Transformer Networks for Semantic Correspondence, NeurIPS, 2018

Concluding Remarks

• RTNs learn to infer locally-varying geometric fields for semantic

correspondence in an end-to-end and weakly-supervised fashion

• The key idea is to utilize and iteratively refine the transformations and

convolutional activations through matching between the image pair

• A technique is presented for weakly-supervised training of RTNs

Seungryong Kim, Ph.D.

Digital Image Media Lab. Yonsei University, Seoul, Korea

Tel: +82-2-2123-2879 E-mail: srkim89@yonsei.ac.kr Homepage: http://diml.yonsei.ac.kr/~srkim/

Thank you!

See you at 210 & 230 AB #119