Facial Landmark Tracking for Mobile Instructor - Simon Lucey - - PowerPoint PPT Presentation

Facial Landmark Tracking for Mobile

Instructor - Simon Lucey

16-623 - Designing Computer Vision Apps

Thatcher Effect

Thompson, P. (1980). "Margaret Thatcher: a new illusion". Perception. 9 (4)

Thatcher Effect

Thompson, P. (1980). "Margaret Thatcher: a new illusion". Perception. 9 (4)

Evil Twin App

Snapchat’s Filters

Taken from http://petapixel.com/2016/06/30/snapchats-powerful-facial-recognition-technology-works/

Snapchat’s Filters

Taken from http://petapixel.com/2016/06/30/snapchats-powerful-facial-recognition-technology-works/

Facial Landmark Alignment History

• Active Appearance Models (AAMs) - LK inspired

Cootes, Edwards and Taylor, 1998 (Active Appearance Models) Matthews and Baker, 2004 (Active Appearance Models Revisited) Cootes and Taylor, 1992 (Active Shape Models) Cristinacce and Cootes, 2004. (Constrained Local Models) Zhou, Gu, and Zhang, 2003 (Bayesian Tangent Shape Models)

• Active Shape Models (ASMs)

How many points?

(a) MultiPIE/IBUG (b) XM2VTS (c) FRGC-V2 (d) AR (e) LFPW (f) HELEN (g) AFW (h) AFLW

Sagonas, Christos, et al. "300 faces in-the-wild challenge: Database and results." Image and Vision Computing 47 (2016): 3-18.

Human Annotator Error

Sagonas, Christos, et al. "300 faces in-the-wild challenge: Database and results." Image and Vision Computing 47 (2016): 3-18.

300 Faces in the Wild DB

• Contains 300 images from indoor and outdoor conditions.

Sagonas, Christos, et al. "300 faces in-the-wild challenge: Database and results." Image and Vision Computing 47 (2016): 3-18.

“Indoor” “Outdoor”

Today

• Constrained Local Models
• Supervised Descent Models
• Efficient Sparse SDMs
• Efficient Dense SDMs

Constrained Local Models

10

• Start with an initial

Constrained Local Models

10

• Start with an initial

Constrained Local Models

10

• Start with an initial
• Warp to a fixed size template (e.g., 110x110 pixels).

CLM - Extracting Patch Responses

11

(110x110 pixels)

• Try to fit a single point.

CLM - Extracting Patch Responses

11

(110x110 pixels)

• Try to fit a single point.

CLM - Extracting Patch Responses

11

(110x110 pixels) “Current Estimate for point n”

• Try to fit a single point.

CLM - Extracting Patch Responses

11

(110x110 pixels) “Current Estimate for point n”

• Try to fit a single point.

“Groundtruth for point n”

Constrained Local Models

12

(110x110 pixels) “nth Constrained Local Search Area”

ith Patch Expert

ith Constrained Local Search Area

Constrained Local Models

13

(e.g., 30x30 pixels) “nth Patch Response”

“Uses Convolution“

∗

CLM - Extracting Patch Responses

(110x110 pixels)

• 68 points in total.

14

CLM - Extracting Patch Responses

(110x110 pixels)

• 68 points in total.

14

D1(p1) = DN(pN) =

CLM - Extracting Patch Responses

(110x110 pixels)

• Get responses for all N patch experts.

.............................

15

Step 1. Pre-compute N local responses.

Dt

i(pi)

∆p∗ = arg min

∆p N

X

i=1

Dt

i(pi + ∆pi) + λ R(p + ∆p)

pt+1 pt ∆p∗

CLM Algorithm

• Unlike Graph Fitting, CLM algorithm is iterative:-

Step 2. Step 3.

“Update the patch center positions.”

Step 4. repeat steps 1 - 3.

“Iterate until convergence.”

(a)

231

................. “Generate responses.” “Optimization step.”

16

Step 1. Pre-compute N local responses.

Dt

i(pi)

∆p∗ = arg min

∆p N

X

i=1

Dt

i(pi + ∆pi) + λ R(p + ∆p)

pt+1 pt ∆p∗

CLM Algorithm

• Unlike Graph Fitting, CLM algorithm is iterative:-

Step 2. Step 3.

“Update the patch center positions.”

Step 4. repeat steps 1 - 3.

“Iterate until convergence.”

(a)

231

................. “Generate responses.” “Optimization step.”

Solving Step 2 is crucial, and most challenging component!!!

16

Point Distribution Model

W(x; p)

R(p + ∆p)

Point Distribution Model

W(x; p)

R(p + ∆p)

Point Distribution Model

W(x; p)

∆pT L∆p

Mode Seeking

• One can interpret the problem as seeking the “mode” across

all responses that is consistent with the global geometry.

18

Mode Seeking

• One can interpret the problem as seeking the “mode” across

all responses that is consistent with the global geometry.

18

• Gaussian assumptions simplify the problem but are often too

strong.

Mode Seeking

• One can interpret the problem as seeking the “mode” across

all responses that is consistent with the global geometry.

18

• Gaussian assumptions simplify the problem but are often too

strong.

• If non-Gaussian, however, mode(s) MUST be found

iteratively.

• Uses variations of the EM-Algorithm,
• Can be slow (iterative).
• Guaranteed of only local minima.

Constrained Mean Shift

19

Constrained Mean Shift

19

• Saragih et al. proposed constrained mean shift.
• Can be applied extremely efficiently using LUT.
• Vary kernel density estimate (KDE) to perform coarse to fine

fitting.

“Constrained Mean Shift”

Saragih, Lucey and Cohn, ICCV 2009. (Constrained Mean Shifts)

Constrained Mean Shift

20

Constrained Mean Shift

20

Saragih, Lucey and Cohn, ICCV 2009. (Constrained Mean Shifts)

Results

21

ASM CQF BTSM CMS ASM CQF

Results

21

ASM CQF BTSM CMS ASM CQF

Results

22

0.2 0.4 0.6 0.8 1 2 4 6 8 10 ProportionofImages ShapeRMSError ASM(88ms) CQF(98ms) GMM(2410ms) KDE(121ms)

ASM (88ms) CQF (98ms) BTSM (2410ms) CMS (121ms) Shape RMS Error Proportion of Images

Saragih, Lucey, Cohn, ICCV 2009. Saragih, Lucey, Cohn, IJCV 2011.

23

CMS Results

Saragih, Lucey, Cohn, AFGR 2011.

Saragih, Lucey, Cohn, AFGR 2011.

CLM Extensions

• Numerous extensions to CLMs

have been proposed over the years.

• A variant - which is referred to

as - Conditional Local Neural Fields (CLNF) obtains almost state of the art performance for facial landmark tracking.

• A package called Open-Face

provides an open source implementation.

http://www.cl.cam.ac.uk/research/rainbow/projects/openface/

• T. Baltrusaitis, L.-P. Morency, and P. Robinson. Constrained local neural fields for robust facial landmark detection in the wild. In ICCVW, 2013.

CLM Drawbacks

• Although exhibiting state of the art performance excellent

performance CLM’s real-time performance is heavily linked to a large number of convolution operations.

• T. Baltrusaitis, L.-P. Morency, and P. Robinson. Constrained local neural fields for robust facial landmark detection in the wild. In ICCVW, 2013.
• This makes the approach ill-suited for mobile device

application due to its computational cost.

Problem

27

⊗

CPU GPU NCC

Solution

28

CPU GPU Multiplication x R =

Solution

28

CPU GPU Multiplication x R =

Today

• Constrained Local Models
• Supervised Descent Models
• Efficient Sparse SDMs
• Efficient Dense SDMs

• Xiong & De La Torre proposed the Supervised Descent Method

(SDM) to explicitly learn .

Reminder: SDMs

R

X, Xuehan, and F. De la Torre. "Supervised descent method and its applications to face alignment." CVPR 2013.

• Xiong & De La Torre proposed the Supervised Descent Method

(SDM) to explicitly learn .

• can be learned from data

Reminder: SDMs

R

X, Xuehan, and F. De la Torre. "Supervised descent method and its applications to face alignment." CVPR 2013.

• Xiong & De La Torre proposed the Supervised Descent Method

(SDM) to explicitly learn .

• can be learned from data
• One can generate a synthetic training set

from given data by sampling


Reminder: SDMs

R

X, Xuehan, and F. De la Torre. "Supervised descent method and its applications to face alignment." CVPR 2013.

• Xiong & De La Torre proposed the Supervised Descent Method

(SDM) to explicitly learn .

• can be learned from data
• One can generate a synthetic training set

from given data by sampling


Reminder: SDMs

R

X, Xuehan, and F. De la Torre. "Supervised descent method and its applications to face alignment." CVPR 2013.

• Xiong & De La Torre proposed the Supervised Descent Method

(SDM) to explicitly learn .

• can be learned from data
• One can generate a synthetic training set

from given data by sampling


• Training objective:
• Learns to directly predict the geometric displacement conditioned on appearance.

Reminder: SDMs

R

X, Xuehan, and F. De la Torre. "Supervised descent method and its applications to face alignment." CVPR 2013.

31

Ground-Truth Artificial Noise

31

Ground-Truth Artificial Noise

Reminder: SDMs

• Like IC-LK, SDM assumes a linear relationship between appearance

and geometry:

∆p = R[I(p) − T (0)]

X, Xuehan, and F. De la Torre. "Supervised descent method and its applications to face alignment." CVPR 2013.

Reminder: SDMs

• Like IC-LK, SDM assumes a linear relationship between appearance

and geometry:

∆p = R[I(p) − T (0)]

• Iteratively updates until convergence

X, Xuehan, and F. De la Torre. "Supervised descent method and its applications to face alignment." CVPR 2013.

Reminder: SDMs

• Like IC-LK, SDM assumes a linear relationship between appearance

and geometry:

∆p = R[I(p) − T (0)]

• Iteratively updates until convergence

X, Xuehan, and F. De la Torre. "Supervised descent method and its applications to face alignment." CVPR 2013.

Reminder: SDMs

• Like IC-LK, SDM assumes a linear relationship between appearance

and geometry:

∆p = R[I(p) − T (0)]

• Iteratively updates until convergence

X, Xuehan, and F. De la Torre. "Supervised descent method and its applications to face alignment." CVPR 2013.

Reminder: SDMs

• Like IC-LK, SDM assumes a linear relationship between appearance

and geometry:

∆p = R[I(p) − T (0)]

• Iteratively updates until convergence

X, Xuehan, and F. De la Torre. "Supervised descent method and its applications to face alignment." CVPR 2013.

Reminder: SDMs

• Like IC-LK, SDM assumes a linear relationship between appearance

and geometry:

∆p = R[I(p) − T (0)]

• Iteratively updates until convergence

X, Xuehan, and F. De la Torre. "Supervised descent method and its applications to face alignment." CVPR 2013.

Reminder: SDMs

• Like IC-LK, SDM assumes a linear relationship between appearance

and geometry:

∆p = R[I(p) − T (0)]

• Iteratively updates until convergence

X, Xuehan, and F. De la Torre. "Supervised descent method and its applications to face alignment." CVPR 2013.

Reminder: SDMs

• Like IC-LK, SDM assumes a linear relationship between appearance

and geometry:

…

∆p = R[I(p) − T (0)]

• Iteratively updates until convergence

X, Xuehan, and F. De la Torre. "Supervised descent method and its applications to face alignment." CVPR 2013.

Reminder: SDMs

• Like IC-LK, SDM assumes a linear relationship between appearance

and geometry:

… …

∆p = R[I(p) − T (0)]

• Iteratively updates until convergence
• However, is iteration specific.

R(k)

X, Xuehan, and F. De la Torre. "Supervised descent method and its applications to face alignment." CVPR 2013.

SDMs vs. CLMs

• SDMs are inherently more computationally efficient than CLMs.
• Computational cost is associated only with image warping, feature

extraction, and matrix multiplication (no convolution).

IntraFace

http://www.humansensing.cs.cmu.edu/intraface

IntraFace

http://www.humansensing.cs.cmu.edu/intraface

• SIFT Feature - SDMs

• SIFT Feature - SDMs
• 1. Compute image gradients
• 2. Pool into local histograms
• 3. Concatenate histograms
• 4. Normalize histograms

• SIFT Feature - SDMs
• 1. Compute image gradients
• 2. Pool into local histograms
• 3. Concatenate histograms
• 4. Normalize histograms

Calculating SIFT for each landmark

• n a mobile device is in general too

costly for real-time performance.

SIFT Speed - Mac vs. iPhone 7

• A. Fagg, S. Sridharan, and S. Lucey, "Fast, Dense Feature SDM on an iPhone" Arix 2016.

Reminder: Binary Features

• Proposed by Calonder et al. ECCV 2010 (BRIEF).
• Borrows idea that binary comparison is very fast on modern

chipset architectures,

1 : I(x + ∆1) > I(x + ∆2) 0 : otherwise

{

ψI(x, ∆1, ∆2) =

• Combine features together compactly,

ψI(x) = X

i

2i−1ψI(x, ∆(i)

1 , ∆(i) 2 )

Why do Binary Features Work?

• Success of binary features says something about

perception itself.

• Absolute values of pixels do not matter.
• Makes sense as variable lighting source will effect the

gain and bias of pixels, but the local ordering should remain relatively constant.

Reminder: BRIEF Descriptor

• Do not need to “touch” all pixels, can choose pairs

randomly and sparsely,

{∆1, ∆2}

Hamming distance in least squares

1 0 1 1 0 0 1 0 0 0 1 1 0 0 1 0 1 0 0 0 0 0 0 0 178 50 128 Hamming distance Squared distance

1

vs.

16384

• H. Alismail, B. Browning, S. Lucey. "Enhancing Direct Camera Tracking with Feature Descriptors" ACCV 2016.

• Local Binary Feature - SDMs
• S. Ren, et al. "Face alignment at 3000 fps via regressing local binary features." CVPR 2014.

• Local Binary Feature - SDMs
• S. Ren, et al. "Face alignment at 3000 fps via regressing local binary features." CVPR 2014.

• Local Binary Feature - SDMs

φ1 =

> ∈ [0, 1]

• S. Ren, et al. "Face alignment at 3000 fps via regressing local binary features." CVPR 2014.

• Local Binary Feature - SDMs

φ1 =

> ∈ [0, 1]

. . .

> ∈ [0, 1]

φF =

• S. Ren, et al. "Face alignment at 3000 fps via regressing local binary features." CVPR 2014.

Local Binary Feature - SDMs

Local Binary Feature - SDMs

• φ1

1

Local Binary Feature - SDMs

• φ1

φ2

1 1

Local Binary Feature - SDMs

• φ1

φ2

φ3

1 1 1

Local Binary Feature - SDMs

• φ1

φ2

φ3

1 1 1

[0 1 0]

Local Binary Feature - SDMs

• …

concatenating

• (a)

(b)

…

• S. Ren, et al. "Face alignment at 3000 fps via regressing local binary features." CVPR 2014.

Local Binary Feature - SDMs

• Learning Linear

Projection Estimated Shape

• Train

Test

• Learning Feature

Mapping Local Binary Features Feature Mapping Linear Projection Estimated Shape

• S. Ren, et al. "Face alignment at 3000 fps via regressing local binary features." CVPR 2014.

Local Binary Feature - SDMs

70

• 21

300-W (68 landmarks)2 Method Fullset Common Subset Challenging Subset FPS ESR[5] 7.58 5.28 17.00 120 SDM[32] 7.52 5.60 15.40 70 LBF 6.32 4.95 11.98 320 LBF fast 7.37 5.38 15.50 3100

atasets, respectively. The errors of ESR and SDM are from our

• S. Ren, et al. "Face alignment at 3000 fps via regressing local binary features." CVPR 2014.

LBF-SDMs - Drawbacks

• Offer unprecedented speed, making its application ideal for

mobile face tracking.

• In reality, however, performance is often noisy/jittery as the

approach is only touching a sparse set of pixels.

Binary Approximated SIFT

• Recently, Fagg et al. proposed Binary Approximated SIFT

for SDMs.

• A. Fagg, S. Sridharan, and S. Lucey, "Fast, Dense Feature SDM on an iPhone" Arxiv 2016.

Binary Approximated SIFT

• Recently, Fagg et al. proposed Binary Approximated SIFT

for SDMs.

θ = atan2(ryI(x), rxI(x))

• A. Fagg, S. Sridharan, and S. Lucey, "Fast, Dense Feature SDM on an iPhone" Arxiv 2016.

Binary Approximated SIFT

• Recently, Fagg et al. proposed Binary Approximated SIFT

for SDMs.

θ = atan2(ryI(x), rxI(x))

ryI(x)

rxI(x)

• A. Fagg, S. Sridharan, and S. Lucey, "Fast, Dense Feature SDM on an iPhone" Arxiv 2016.

Binary Approximated SIFT

• Recently, Fagg et al. proposed Binary Approximated SIFT

for SDMs.

ryI(x)

rxI(x) ryI(x) > 0 rxI(x) > 0 |ryI(x)| > |rxI(x)| 23 = 8 orientation bins

• A. Fagg, S. Sridharan, and S. Lucey, "Fast, Dense Feature SDM on an iPhone" Arxiv 2016.

Binary Approximated SIFT

• Recently, Fagg et al. proposed Binary Approximated SIFT

for SDMs.

• The approach approximates binning through binary

comparison.

8 orientations

1 1 1 1

• A. Fagg, S. Sridharan, and S. Lucey, "Fast, Dense Feature SDM on an iPhone" Arxiv 2016.

• BA-SIFT Feature - SDMs

1 1 1 1

• BA-SIFT Feature - SDMs

Calculating BASIFT extremely efficient.

1 1 1 1

Inspiration from the FFT

• A. Fagg, S. Sridharan, and S. Lucey, "Fast, Dense Feature SDM on an iPhone" Arxiv 2016.

Sparse Compositional SDM

• A. Fagg, S. Sridharan, and S. Lucey, "Fast, Dense Feature SDM on an iPhone" Arxiv 2016.

Speed Comparison

• A. Fagg, S. Sridharan, and S. Lucey, "Fast, Dense Feature SDM on an iPhone" Arxiv 2016.

90 FPS - iPhone 7

• A. Fagg, S. Sridharan, and S. Lucey, "Fast, Dense Feature SDM on an iPhone" Arxiv 2016.

L L L

90 FPS - iPhone 7

• A. Fagg, S. Sridharan, and S. Lucey, "Fast, Dense Feature SDM on an iPhone" Arix 2016.

More Examples.....

More Examples.....

More Examples.....

More Examples.....

More Examples.....