[PDF] - Probabilistic Morphable Models Thomas Vetter > DEPARTMENT OF PDF Document

SLIDE 1

1

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Probabilistic Morphable Models

Thomas Vetter

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Photo by Pete Souza/The White House via Getty Images.

Male 60-70 Blue eyes Wide nose Mouth closed …… “Robert Gates”

SLIDE 2

2

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Analysis by Synthesis

3D World Image Analysis Synthesis Image Model Image Description

model parameter

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Example based image modeling of faces

2D Image 2D Face Examples 2D Image 3D Face Scans

= w1 * + w2 * + w3 * + w4 * +. . .

SLIDE 3

3

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Morphable Models for Image Registration

Output

R = Rendering Function ρ = Parameters for Pose, Illumination, ...

Optimization Problem: Find optimal α, β, ρ !

R               

β1 + β2 + β3 + ⋯ α1 + α2 + α3 + ⋯

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Probabilistic Morphable Models

1. Model-based image registration using Gaussian Processes for

shape deformations

2. “Probabilistic registration”: Find the distribution of possible

transformations h(𝜄) that transforms 𝐽𝑆 to 𝐽𝑈 .

?

𝑄(𝜄|𝐽𝑈, 𝐽𝑆)

SLIDE 4

4

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Gaussian Process Morphable Models:

A Gaussian process ℎ ~ 𝐻𝑄 𝜈, 𝑙

n 𝑌 is completely defined by

its mean function 𝜈 ∶ 𝑌 → ℝ3 and covariance function 𝑙 ∶ 𝑌 × 𝑌 → ℝ3×3 A low rank approximation can by computed using the Nyström approximation. ℎ 𝜄 ≈ 𝜈 + σ𝑗

𝑒 𝜄𝑗

𝜇𝑗Φ𝑗 with 𝜄 ~ 𝑂(0, 𝐽𝑒)

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Advantage of Gaussian Process Morphable Models

Probabilistic formalism ! Extremely flexible concept. By varying the covariance function k a variety of ‘different’ algorithms of deformation modelling are included.  Thin Plate Splines  Free Form deformations  …  Standard PCA-Model

“Scalismo” an open source library by Marcel Lüthi see also our MOOC on FutureLearn “Statistical Shape Modlling”

SLIDE 5

5

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Surface Data Prediction

as Gaussian Process Regression 3D Surface Data Base Analysis 3D Input Statistical Prediction Original

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Surface Data Prediction

as Gaussian Process Regression

SLIDE 6

6

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Application

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Disclocation of the patella

SLIDE 7

7

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Femur Patella MRI-Slice

Example use-case: Trochlea dysplasia

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Trochlea-Dysplasia

SLIDE 8

8

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Surgical intervention: Increase goove

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Surgical intervention: Augment bony structure

SLIDE 9

9

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Posterior Shape Models

T. Albrecht, M. Lüthi, T. Gerig, T. Vetter,

Medical Image Analysis, 2013

Automatic inference of pathology

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Probabilistic Inference for Image Registration

Generative image explanation: How to find 𝜄 explaining I ? 𝑞 𝜄 𝐽 = ℓ(𝜄; 𝐽) 𝑞(𝜄) 𝑂(𝐽) 𝑂 𝐽 = න ℓ(𝜄; 𝐽)𝑞(𝜄)d𝜄

----> Normalization intractable in our setting

What can be done:

1. Accept MAP as the only option 2. Approximate posterior distribution (e.g. use sampling methods)

SLIDE 10

10

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

The Metropolis-Hastings Algorithm

Need a distribution which can generate samples: 𝑅 𝜄′ 𝜄) Algorithm transforms samples from 𝑅 into samples from 𝑄:

1. Draw a sample 𝜄′ from 𝑅 𝜄′ 𝜄) 2. Accept 𝜄′ as new state 𝜄 with probability 𝑞𝑏𝑑𝑑𝑓𝑞𝑢 = min

𝑄 𝜄′ 𝑄 𝜄 𝑅 𝜄|𝜄′ 𝑅 𝜄′|𝜄 , 1

3. State 𝜄 is current sample, repeat for next sample

--> Generates unbiased but correlated samples from 𝑄

Markov Chain Monte Carlo Sampling: Result: 𝜄1, 𝜄2, 𝜄3, … …

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

MH Inference of the 3DMM

 Target distribution is our “posterior”:  𝑄: ෨ 𝑄 𝜄 𝐽𝑈 = ℓ 𝜄|𝐽𝑈, 𝐽𝑆 𝑞 𝜄  Unnormalized  Point-wise evaluation only  Parameters  Shape: 50 – 200, low-rank parameterized GP shape model  Color: 50 – 200, low-rank parameterized GP color model  Pose/Camera: 9 parameters, pin-hole camera model  Illumination: 9*3 Spherical Harmonics for illumination/reflectance  ≈ 300 dimensions (!!)

SLIDE 11

11

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Metropolis Filtering

θ′

MH-Filter:

Q θ′|θ

𝑞𝑏𝑑𝑑𝑓𝑞𝑢

reject θ𝑝𝑚𝑒 → θ′

θ′ θ′

update θ′ → θ

Markov Chain Monte Carlo Sampling: Result: 𝜄1, 𝜄2, 𝜄3, … …

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Results: 2D Landmarks

Landmarks posterior:

Manual labelling: 𝜏LM = 4pix Image: 512x512

Certainty of pose fit?

Influence of ear points? Frontal better than side-view?

Yaw, σ𝐌𝐍 = 4pix with ears w/o ears Frontal 1.4∘ ± 𝟏. 𝟘∘ −0.8∘ ± 𝟑. 𝟖∘ Side view 24.8∘ ± 𝟑. 𝟔∘ 25.2∘ ± 𝟓. 𝟏∘

SLIDE 12

12

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Integration of Bottom-Up

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Metropolis Filtering

θ′

MH-Filter: Prior

Q θ′|θ

𝑞𝑏𝑑𝑑𝑓𝑞𝑢

reject θ𝑝𝑚𝑒 → θ′

update θ′ → θ

MH-Filter: Face Box

𝑞𝑏𝑑𝑑𝑓𝑞𝑢

reject θ𝑝𝑚𝑒 → θ′ MH-Filter: Image

𝑞𝑏𝑑𝑑𝑓𝑞𝑢

reject θ𝑝𝑚𝑒 → θ′

θ′ 𝑄0 𝜄

𝑚 𝜄,𝐺𝐶

𝑄 𝜄|𝐺𝐶

𝑚 𝜄,𝐽

𝑄 𝜄|𝐺𝐶, 𝐽

SLIDE 13

13

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

35

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Face analysis

Roger F. asian caucasian blue eyes brown eyes wide nose male mustache gaze Hor yaw pitch roll 0.34 0.52 0.19 0.69 0.70 0.52 0.13 20° 34°

8°

4°

SLIDE 14

14

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

40

Occlusion-aware 3D Morphable Face Models

Bernhard Egger, Sandro Schönborn, Andreas Schneider, Adam Kortylewski, Andreas Morel-Forster, Clemens Blumer and Thomas Vetter International Journal of Computer Vision, 2018

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Face Image Analysis under Occlusion

41

Source: AFLW Database Source: AR Face Database

SLIDE 15

15

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

ℓ 𝜄; 𝐽 = ෑ

𝑞𝑗𝑦𝑓𝑚

ℓ 𝜄; 𝐽 𝑦

There is nothing like: no background model

“Background Modeling for Generative Image Models” Sandro Schönborn, Bernhard Egger, Andreas Forster, and Thomas Vetter Computer Vision and Image Understanding, Vol 113, 2015.

= ෑ

𝑦∈𝐺𝑕

ℓ 𝜄; 𝐽 𝑦 × ෑ

𝑦∈𝐶𝑕

ℓ 𝜄; 𝐽 𝑦 ℓ 𝜄; 𝐽 = ෑ

𝑦 ∈ 𝐽

ℓ 𝜄; 𝐽 𝑦

Maximum Likelihood Formulation:

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Occlusion-aware Model

𝑚 𝜄; ሚ 𝐽, 𝑨 = ෑ

𝑗

𝑚𝑔𝑏𝑑𝑓 𝜄; ෩ 𝐽𝑗

𝑨 ∙ 𝑚𝑜𝑝𝑜−𝑔𝑏𝑑𝑓 𝜄; ෩

𝐽𝑗

1−𝑨

SLIDE 16

16

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Inference

46

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Initialisation: Robust Illumination Estimation

47 Init 𝜄𝑚𝑗𝑕ℎ𝑢 Init 𝑨 Init 𝜄𝑑𝑏𝑛𝑓𝑠𝑏

SLIDE 17

17

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Results: Qualitative

Source: AR Face Database > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Results: Qualitative

49

Source: AFLW Database

SLIDE 18

18

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Results: Applications

50

Source: LFW Database > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Modeling of 2D Images

SLIDE 19

19

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Computer can learn to model faces according to „human“ categories.

Aggressive Trustworthy

Portraits made to Measure

SLIDE 20

20

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Modeling the Appearance of Faces

Which directions code for specific attributes ?

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Learning from Examples

SLIDE 21

21

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Portraits made to Measure

Trustworthiness Social Skills Risk Seeking Likeability Extroversion Aggressiveness

% Correct ratings

100 90 80 70 60 50 40 30 20 10

Personality traits

Portraits made to measure: Mirella Walker and Thomas Vetter Journal of Vision, 9(11):12, 1-13, 2009

.

Aggressiveness Extroversion Likeability Risk Seeking Social Skills Trustworthiness Original Face Aggressiveness Aggressiveness Extroversion Extroversion Likeability Likeability Risk Seeking Risk Seeking Social Skills Social Skills Trustworthiness Trustworthiness Original Face Original Face

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Simulation of Aging of Human Faces in Images

SLIDE 22

22

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Aging model:

model predicts perceived age

Labeled / True age

20 years 70 years

Predicted age

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Ageing: linear shape model only

SLIDE 23

23

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Example-based: The Problem

+ 5 years + 5 years

Target Image AGE: 40 Shape and Skin of donor AGE: 45 Shape and Skin of donor AGE: 50

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Parametric Pigmentation Model

SLIDE 24

24

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018

Aging Model

Shape: continuous Pigmentation: stochastic Wrinkles: example based

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018