2D Face Image Analysis Probabilistic Morphable Model Fitting - - PDF document

06.06.2019 1

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

2D Face Image Analysis

Probabilistic Morphable Model Fitting Basel2019 University of Basel

1

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Modeling of 2D Images

06.06.2019 2

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Morphable Models for Image Registration

Output

R = Rendering Function ρ = Parameters for Pose, Illumination, ...

Optimization Problem: Find optimal α, β, ρ !

R               

β1 + β2 + β3 + ⋯ α1 + α2 + α3 + ⋯

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Contents

Landmarks Fitting Image Fitting Observed Landmarks in 2D Observed Image

06.06.2019 3

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

2D Face Image Analysis

𝑄 𝜄 𝐽 ∝ ℓ 𝜄; 𝐽 𝑄(𝜄)

Morphable Model adaptation to explain image

Bayesian Inference Setup

Face & Feature point detection

Integration of fast bottom-up methods 𝐺

Image Likelihood

Image as observation

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Computer Graphics: Rendering Faces

2D Image 2D Face Examples 3D Face Scans 2D Images

w1 * + w2 * + w3 * + w4 * +. . . R =

Faces: GP models for shape & color: 𝑡𝛽 = 𝜈 + 𝑉𝐸𝛽 𝛽~ 𝑂 0, 𝐽𝑒 𝑑β = 𝜈 + 𝑉𝐸β β~ 𝑂 0, 𝐽𝑒

06.06.2019 4

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Computer Graphics Overview

• Geometry

ry (result of shape modelling)

• Camera & Proje
• jecti

tion

Transformations in space and projection Maps 3D space and 2D image plane

• Ra

Rasterizati tion

Correspondence: image pixels ↔ surface Z-Buffer: Hidden surface removal

• Shad

hading

Illumination simulation models

• Illum

umination

• n

Phong: Ambient, diffuse & specular Global Illumination

7

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Face-to-Image Transformations

• Mod
• del-View

𝑈

𝑁𝑊 𝑦 = 𝑆𝜒,𝜔,𝜘 𝒚 + 𝒖

• Proj
• jectio

tion

𝒬 𝑦 = 𝑔 𝑨 𝑦 𝑧

• Vie

iewport

𝑈

𝑊𝑄(𝑦) =

𝑥 2 (𝑦 + 1) ℎ 2 (1 − 𝑧) + 𝒖𝑞𝑞

• 9 Parameters:
• (3) Translation 𝒖
• (3) Rotation 𝜒, 𝜔, 𝜘
• (1) Focal length 𝑔
• (2) Image Offset 𝒖𝑞𝑞
• 2 Constants:
• (2) Image size / sampling

8

06.06.2019 5

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Perspective Effect

• Perspective division distorts image non-linearly
• Effect depends on relation of object depth and camera distance

9

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Rasterization

• Camera: 3D → 2D

transformation for points

• Raster Image in image plane
• Establishes correspondence

to 3D surface for each pixel

• Basis: geometric primitives

10

𝑥 ℎ (0,0)

(4,2)

Pixel grid, cell-centered

06.06.2019 6

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Phong Illumination Model

• Combination of three illumination

contributions:

• Lambert (diffuse)
• Specular
• Ambient (global)
• Ambient is a scene average light intensity 𝐽𝐵
• Lambert and specular part for each light source

11

𝐽′ = 𝑙amb ∗ 𝐽

𝐵 + 𝑙diff ∗ 𝐽𝑀 ∗ cos 𝑀, 𝑂 + 𝑙spec ∗ 𝐽𝑀 ∗ cos R, V 𝑜

𝑙diff ∗ 𝐽𝑀 ∗ cos 𝑀, 𝑂 𝑙spec ∗ 𝐽𝑀 ∗ cos R, V 𝑜 𝑙amb ∗ 𝐽

𝐵

usually colored

N   L V



R

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Phong Illumination Model

• Combination of three

illumination contributions:

• Lambert (diffuse)
• Specular
• Ambient (global)
• Ambient is a scene average

light intensity 𝐽𝐵

• Lambert and specular part

for each light source

12

𝐽′ = 𝑙amb ∗ 𝐽

𝐵 + 𝑙diff ∗ 𝐽𝑀 ∗ cos 𝑀, 𝑂 + 𝑙spec ∗ 𝐽𝑀 ∗ cos R, V 𝑜

𝑙diff ∗ 𝐽𝑀 ∗ cos 𝑀, 𝑂 𝑙spec ∗ 𝐽𝑀 ∗ cos R, V 𝑜 𝑙amb ∗ 𝐽

𝐵

usually colored

06.06.2019 7

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Environment Maps

• Mapping of incoming light

intensity from every direction 𝐽𝑀

RGB 𝜄, 𝜒

• Modeled at infinity
• Typically empirically captured
• Shading with environment

maps requires integration

• ver all incoming directions

13

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Environment Maps

14

Grace Cathedral (San Francisco)

• P. Debevec

White surface in Grace Cathedral

06.06.2019 8

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Spherical Harmonics Illumination

• Expand map 𝐽𝑀

RGB 𝜄, 𝜒 with

basis functions

• Choose Spherical Harmonics:

Eigenfunctions of Laplace

• perator on sphere surface

𝑍

𝑚𝑛(𝜄, 𝜒)

• Corresponds to Fourier transform
• Integration becomes

multiplication of coefficients (→ fast convolution)

• Low frequency part is sufficient

for Lambertian reflectance

15 Inigo.quilez Ramamoorthi, Ravi, and Pat Hanrahan. "An efficient representation for irradiance environment maps." Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, 2001.

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Environment Map Illumination

16

06.06.2019 9

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

i bi

model(

, ) , ,

x y r g b

p p I I I  I

, , r g b

Illumination Model Color Transformation

, ,

r g b

I I I , , x y z ,

x y

p p

Perspective Projection Rigid Transformation Normals

Image Formation: at each Vertex k

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

3D Face Reconstruction

18

06.06.2019 10

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Probabilistic Inference for Image Registration

• Generative image explanation: How to find 𝜄 explaining I ?

𝑞 𝜄 𝐽 = ℓ(𝜄; 𝐽) 𝑞(𝜄) 𝑂(𝐽) 𝑂 𝐽 = න ℓ(𝜄; 𝐽)𝑞(𝜄)d𝜄

• ----> Normalization intractable in our setting
• What can be done:

1. Accept MAP as the only option 2. Approximate posterior distribution (e.g. use sampling methods)

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

MH Inference of the 3DMM

• Parameters
• Shape:

50 – 200, low-rank parameterized GP shape model

• Color:

50 – 200, low-rank parameterized GP color model

• Pose/Camera:

9 parameters, pin-hole camera model

• Illumination:

9*3 Spherical Harmonics illumination/reflectance ≈ 300 dimensions (!!)

21

• Target distribution is our “po

posterio ior”: 𝑄: ෨ 𝑄 𝜄 𝐽 = ℓ 𝜄; 𝐽 𝑄 𝜄

• Unnormalized
• Point-wise evaluation only

06.06.2019 11

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Metropolis Algorithm

23

𝑅(𝜄′|𝜄) 𝑄(𝜄′|𝐽)

𝜄′

Proposal

Accept with probability

reject

draw proposal 𝜄′

𝜄

Update 𝜄 ← 𝜄′ 𝛽 = min 𝑄(𝜄′|𝐽) 𝑄(𝜄|𝐽) , 1 1 − 𝛽

• Asymptotically generates samples 𝜄𝑗 ∼ 𝑄(𝜄|𝐽): 𝜄1, 𝜄2, 𝜄3, …
• Markov chain Monte Carlo (MCMC) Method
• Works with unnormalized, point-wise posterior

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Proposals

• Choose simple Gaussian random walk proposals (Metropolis)

"𝑅 𝜄′|𝜄 = 𝑂(𝜄′|𝜄, Σ𝜄)"

• Normal perturbations of current state
• Block-wise to account for different parameter types
• Shape

𝑂(𝜷′|𝜷, 𝜏𝑇

2𝐹𝑡)

• Color

𝑂(𝜸′|𝜸, 𝜏𝐷

2𝐹𝐷)

• Camera

σ𝑑 𝑂(𝜄𝑑

′|𝜄𝑑, 𝜏𝑑 2)

• Illumination

σ𝑗 𝑂(𝜄𝑀

′|𝜄𝑀, 𝜏𝑀,𝑗 2 𝐹𝑀)

• Large mixture distributions, e.g.

In practice, we often add more complicated proposals, e.g. shape scaling, a direct illumination estimation and decorrelation

24

2 3 𝑅𝑄 𝜄′ 𝜄 + 1 3 ෍

𝑗

𝜇𝑗𝑅𝑗

𝑀(𝜄′|𝜄)

06.06.2019 12

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Landmarks Fitting

Projection Variable Parameters

• Pose
• Shape

Likelihood ℓ 𝜄; ෥ 𝒚 ∝ 𝑄 ෥ 𝒚 𝒚 𝜄 Target Landmarks Rendered Landmarks Face Model Prior 𝑄 𝜄

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

3DMM Landmarks Likelihood

Simple models: Ind ndependent t Gaussia ians

• Observation of landmark locations in image
• Single landmark position model:

𝒚𝑗

2D 𝜄

= TVP ∘ Pr ∘ TMV 𝒚𝑗

3D

ℓ𝑗 𝜄; ෥ 𝒚𝑗

2D = 𝑂 ෥

𝒚𝑗

2D|𝒚𝑗 2D 𝜄 , 𝜏LM 2 Independence and Gaussian are just simple models (questionable)

27

TMV 𝒚 = 𝑆𝜒,𝜔,𝜘 𝒚 + 𝒖 (TVP ∘ Pr)(𝒚) = 𝑥 2 ∗ 𝑦 𝑨 − ℎ 2 ∗ 𝑧 𝑨 + 𝒖𝑞𝑞

• Independent model

ℓ 𝜄; {෥ 𝒚𝑗

2D}𝑗

= ෑ

𝑗

ℓ 𝜄; ෥ 𝒚𝑗

2D

06.06.2019 13

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Landmarks: Samples

28

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Results: 2D Landmarks

• Landmarks posterior:

Manual labelling: 𝜏LM = 4pix Image: 512x512

• Certainty of pose fit?
• Influence of ear points?
• Frontal better than side-view?

Yaw, σ𝐌𝐍 = 4pix wi with th ears w/o w/o ears Frontal 1.4∘ ± 𝟏. 𝟘∘ −0.8∘ ± 𝟑. 𝟖∘ Side view 24.8∘ ± 𝟑. 𝟔∘ 25.2∘ ± 𝟓. 𝟏∘

06.06.2019 14

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Face Model Fitting

31

Parametric face model Target Image 𝐽 Rendered Image 𝐽 𝜄 Likelihood ℓ 𝜄; 𝐽 ∝ 𝑄 𝐽 𝐽 𝜄 Face Model Reconstruction: Analysis-by-Synthesis 𝜄 = 𝜘, 𝛽, 𝛾 : 𝜘 Scene Parameters, 𝛽 Face shape, 𝛾 Face color

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Independent Pixels Likelihood

∗ ⋯ 𝒪( | , 𝜏2𝐽3) 𝒪( | , 𝜏2𝐽3) ∗

ℓ 𝜄; ሚ 𝐽 =

ℓ 𝜄; ሚ 𝐽 = ෑ

𝑗∈𝐺

𝒪 ෩ 𝐽𝑗 | 𝐽𝑗 𝜄 , 𝜏2𝐽3

𝐺

Standard choice Corresponds to least squares fitting

06.06.2019 15

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Image Likelihood

Backgr grou

• und

d mode del is requ quired ed

The face model does not cover the complete target image and shows self-occlusion.

Collec ective e likel eliho hood

• d model

del

Pixels are not independent. We can also model the empirical distribution of image distance ℎ 𝑒 𝑒 = ‖ − ‖ ℎ(𝑒)

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Background Model

• Variable alignment of model with the image
• Projected size and self-occlusion
• Shrinking or misalignment
• Model background pixels explicitly

34

Face covers only parts of the image – background must not be ignored

ℓ 𝜄; ሚ 𝐽 = ෑ

𝑗∈𝐺

ℓF 𝜄; ෩ 𝐽𝑗 ෑ

𝑘∈𝐶

𝑐BG ෩ 𝐽𝑗

Arbitrary background: The explicit background model needs to be based on generic and simple assumptions: Constant Histogram Schönborn et al. 2015 «Background modeling for generative image models», Computer Vision and Image Understanding, Volume 136

06.06.2019 16

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Posterior Samples: Fitting Result

• Model instances with comparable reconstruction quality
• Remaining uncertainty of model representation
• Integration of uncertain detection directly into model adaptation

35

Posterior using collective likelihood

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Results: Image

Yaw angle: 1.9∘ ± 0.2∘

36

06.06.2019 17

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Image: Samples

37

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Posterior Shape Variation

Landmarks posterior, sd[mm] Image posterior, sd[mm]

38

06.06.2019 18

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Fitting Results

39

Images from: Huang, Gary B., et al. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Vol. 1. No. 2. Technical Report 07-49, University of Massachusetts, Amherst, 2007. Images from: Köstinger, Martin, et al. "Annotated facial landmarks in the wild: A large-scale, real-world database for facial landmark localization." Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on. IEEE, 2011.

LFW AFLW

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Automatic Fitting

• Detection of face and feature points
• Scanning window & classifier
• Uncertain results
• Feed-forward: early hard decisions
• Integration concept
• Bayesian integration

→ Filter ering ng

• Metropolis sampling

→ Pr Propo

• pose

e & ve verify fy

40

Which box contains the face?

Schönborn, Sandro, et al. "Markov Chain Monte Carlo for Automated Face Image Analysis." International Journal of Computer Vision (2016): 1-24.

06.06.2019 19

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Random Forest Detection

• Scanning Window
• Random Forest Classifier

41

𝑔

2 𝐽𝑞

𝑔

1 𝐽𝑞

𝑔

3 𝐽𝑞

   

• Haar Features
• Information gain splitting
• Bagging many trees, depth ~16
• ~200k training patches (AFLW)

> 𝜄 ≤ 𝜄

• Classify each patch: face or not
• Search over image
• Search over scales
• Histogram equalization

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Bayesian Integration

• Different modality
• Box 𝐺: position & size
• Landmarks 𝐸: certainty
• Detection is uncertain
• Likelihood models
• Detection is observation
• Different observation models
• Conceptual uncertainty

42

Observation likelihood 𝑄 𝜄 𝐺, 𝐸 = ℓ 𝜄; 𝐺, 𝐸 𝑄 𝜄 𝑂(𝐺, 𝐸) ℓ 𝜄; 𝐺, 𝐸 = 𝑄 𝐺|𝜄 𝑄 𝐸|𝜄 Bayesian inference

Detection data Bayesian integration

06.06.2019 20

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Integration by Filtering

• Step-by-step Bayesian inference
• Condition on observations one after the other
• Posterior of first observation becomes prior for next step
• Each step adds an observation through conditioning with its likelihood
• Equivalent to single-step Bayesian inference

44

𝑄 𝜄 𝑄(𝜄| ) 𝑄(𝜄| )

ℓ 𝜄; 𝐺, 𝐸 ℓ 𝜄; 𝐽

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Filtering: Multiple Metropolis Decisions

θ′

MH-Filter: Pr Prio ior

Q θ′|θ

𝑞𝑏𝑑𝑑𝑓𝑞𝑢

reject θ𝑝𝑚𝑒 → θ′

update θ′ → θ MH-Filter: Face

e Box

𝑞𝑏𝑑𝑑𝑓𝑞𝑢

reject θ𝑝𝑚𝑒 → θ′

MH-Filter: Ima mage

𝑞𝑏𝑑𝑑𝑓𝑞𝑢

reject θ𝑝𝑚𝑒 → θ′ θ′ 𝑄0 𝜄

𝑚 𝜄,𝐺𝐶

𝑄 𝜄|𝐺𝐶

𝑚 𝜄,𝐽

𝑄 𝜄|𝐺𝐶, 𝐽

• Step-wise Bayesian inference: Needs ℓ 𝜄 for each step
• Saves computation time if properly ordered

06.06.2019 21

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

49

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Summary

• Fitting as probabilistic inference
• Probabilistic inference is often intractable
• Sampling methods approximate by simulation
• MCMC methods provide a powerful sampling framework
• Markov Chain with target distribution as equilibrium distribution
• General algorithms, e.g. Metropolis-Hastings
• Fitting of the 3DMM as a real inference problem
• MH algorithm to integrate information: Framework
• Filtering: Uncertain information as observation, step-by-step
• Propose-and-verify: Alternatives, multiple hypotheses, heuristics

50

06.06.2019 22

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

51

Occlu lusio ion-aware 3D 3D Mor

• rphable

le Face Mod

• dels

ls

Bernhard Egger, Sandro Schönborn, Andreas Schneider, Adam Kortylewski, Andreas Morel-Forster, Clemens Blumer and Thomas Vetter Intern ernation

• nal Journ

rnal of Computer r Vision

• n, 2018

018

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Face Image Analysis under Occlusion

52 Source: AFLW Database Source: AR Face Database

06.06.2019 23

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

There is nothing like: no background model

ℓ 𝜄; 𝐽 = ෑ

𝑦 ∈ 𝐽

ℓ 𝜄; 𝐽 𝑦 = ෑ

𝑗∈𝐺

𝑚𝑔𝑏𝑑𝑓(𝜄; ෩ 𝐽𝑗) ෑ

𝑗`∈𝐶

𝑐(෩ 𝐽𝑗`)

“Background Modeling for Generative Image Models” Sandro Schönborn, Bernhard Egger, Andreas Forster, and Thomas Vetter Computer Vision and Image Understanding, Vol 113, 2015. > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Occlusion-aware Model

𝑚 𝜄; ሚ 𝐽, 𝑨 = ෑ

𝑗

𝑚𝑔𝑏𝑑𝑓 𝜄; ෩ 𝐽𝑗

𝑨 ∙ 𝑚𝑜𝑝𝑜−𝑔𝑏𝑑𝑓 𝜄; ෩

𝐽𝑗

1−𝑨

06.06.2019 24

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Inference

56

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Initialisation: Robust Illumination Estimation

57

Init 𝜄𝑚𝑗𝑕ℎ𝑢 Init 𝑨 Init 𝜄𝑑𝑏𝑛𝑓𝑠𝑏

06.06.2019 25

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Results: Qualitative

Source: AR Face Database

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Results: Qualitative

59 Source: AFLW Database

06.06.2019 26

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS2019 ¦ BASEL

Results: Applications

60 Source: LFW Database