Supervision-by-Registration: An Unsupervised Approach to Improve the - - PowerPoint PPT Presentation

Supervision-by-Registration: An Unsupervised Approach to Improve the Precision of Facial Landmark Detectors

Xuanyi Dong1, Shoou-I Yu2, Xinshuo Weng2, Shih-En Wei2, Yi Yang1, Yaser Sheikh2

1Cai University of Technology Sydney, 2Oculus Research, Facebook

CVPR 2018, Salt Lake City

Facial Landmark Detection

A Challenging Problem

Poses

(expressions/viewpoints)

Sagonas et al. 300 Faces in-the-Wild Challenge: The first facial landmark localization Challenge. ICCV, 2013.

Identity Temporal Consistency

Landmark Detection Methods

Image-based Detection

• DeepReg [Shi et al, NNLS’ 14]
• Convolutional Pose Machine [Wei et al,

CVPR’ 16]

• Hourglass Network [Newell et al, ECCV’

16]

• ….
• Pros

○ Accurate across poses/identity

• Cons

○ Lack of temporal consistency (jittering)

Video-based Detection

Landmark Detection Methods

Video-based Detection

• Recurrent Encoder-Decoder Network

[Peng et al, ECCV’ 16]

• Two-Streams Transformer [Liu et al,

TPAMI’ 17]

• Supervision-by-Registration [Ours]
• ….
• Pros

○ Temporal-consistent

• Cons

○ Require per-frame annotations, difficult to scale up

Image-based Detection

• DeepReg [Shi et al, NNLS’ 14]
• Convolutional Pose Machine [Wei et al,

CVPR’ 16]

• Hourglass Network [Newell et al, ECCV’

16]

• ….
• Pros

○ Accurate across poses/identity

• Cons

○ Lack of temporal consistency (jittering)

What is Supervision-by-Registration?

Lucas-Kanade Tracking Operation: Differentiable

Registration Loss: Forward-Backward Scheme

Noh et al. Learning Deconvolution Network for Semantic Segmentation? ICCV, 2015.

Soft-Argmax Differentiable Operation

Sample Heatmap Output

Implementation

• Used VGG16 as the backbone architecture
• Used CPM as the base facial landmark detector (can be replaced by others.

E.g., stacked hourglass network)

• Operate LK tracking on images/conv1 features

Results: on Image Datasets

Results: on Video Datasets

• AUC@0.08 error for each individual video of 300-VW category C. The

numbers are percentages.

Demo

Take Home Messages

• Registration can be a free supervision signal to enforce temporal consistency
• More generally, self-supervision is powerful!