Probabilistic Morphable Models Thomas Vetter > DEPARTMENT OF - PDF document

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Probabilistic Morphable Models Thomas Vetter > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Male 60-70 Blue eyes Wide nose Mouth closed …… “Robert Gates” Photo by Pete Souza/The White House via Getty Images. 1

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Analysis by Synthesis 3D Image Image Description World model parameter Analysis Image Model Synthesis > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Example based image modeling of faces 2D Image 3D Face Scans 2D Image 2D Face Examples = w 1 * + w 2 * + w 3 * + w 4 * +. . . 2

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Morphable Models for Image Registration   α 1 + α 2 + α 3 + ⋯      R     β 1 + β 2 + β 3 + ⋯      R = Rendering Function ρ = Parameters for Pose, Illumination, ... Optimization Problem: Find optimal α , β , ρ ! Output > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Probabilistic Morphable Models 1. Model-based image registration using Gaussian Processes for shape deformations ? 2. “ Probabilistic registration”: Find the distribution of possible transformations h( 𝜄 ) that transforms 𝐽 𝑆 to 𝐽 𝑈 . 𝑄( 𝜄 |𝐽 𝑈 , 𝐽 𝑆 ) 3

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Gaussian Process Morphable Models:  A Gaussian process ℎ ~ 𝐻𝑄 𝜈, 𝑙 on 𝑌 is completely defined by its mean function 𝜈 ∶ 𝑌 → ℝ 3 and covariance function 𝑙 ∶ 𝑌 × 𝑌 → ℝ 3×3  A low rank approximation can by computed using the Nyström approximation. 𝑒 𝜄 𝑗 ℎ 𝜄 ≈ 𝜈 + σ 𝑗 𝜇 𝑗 Φ 𝑗 with 𝜄 ~ 𝑂(0, 𝐽 𝑒 ) > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Advantage of Gaussian Process Morphable Models  Probabilistic formalism !  Extremely flexible concept. By varying the covariance function k a variety of ‘different’ algorithms of deformation modelling are included.  Thin Plate Splines  Free Form deformations  …  Standard PCA-Model “ Scalismo ” an open source library by Marcel Lüthi see also our MOOC on FutureLearn “Statistical Shape Modlling ” 4

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Surface Data Prediction as Gaussian Process Regression 3D Surface Data Base Analysis Statistical Original 3D Input Prediction > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Surface Data Prediction as Gaussian Process Regression 5

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Application > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Disclocation of the patella 6

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Example use-case: Trochlea dysplasia MRI-Slice Patella Femur > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Trochlea-Dysplasia 7

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Surgical intervention: Increase goove > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Surgical intervention: Augment bony structure 8

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Automatic inference of pathology Posterior Shape Models T. Albrecht, M. Lüthi, T. Gerig, T. Vetter, Medical Image Analysis, 2013 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Probabilistic Inference for Image Registration  Generative image explanation: How to find 𝜄 explaining I ? 𝑞 𝜄 𝐽 = ℓ(𝜄; 𝐽) 𝑞(𝜄) 𝑂 𝐽 = න ℓ(𝜄; 𝐽)𝑞(𝜄)d𝜄 𝑂(𝐽) -----> Normalization intractable in our setting  What can be done: 1. Accept MAP as the only option 2. Approximate posterior distribution (e.g. use sampling methods) 9

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 The Metropolis-Hastings Algorithm  Need a distribution which can generate samples: 𝑅 𝜄 ′ 𝜄)  Algorithm transforms samples from 𝑅 into samples from 𝑄 : Draw a sample 𝜄 ′ from 𝑅 𝜄 ′ 𝜄) 1. 𝑄 𝜄 ′ 𝑅 𝜄|𝜄 ′ Accept 𝜄 ′ as new state 𝜄 with probability 𝑞 𝑏𝑑𝑑𝑓𝑞𝑢 = min 2. 𝑅 𝜄 ′ |𝜄 , 1 𝑄 𝜄 State 𝜄 is current sample, repeat for next sample 3. ---> Generates unbiased but correlated samples from 𝑄  Markov Chain Monte Carlo Sampling: Result: 𝜄 1 , 𝜄 2 , 𝜄 3 , … … > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 MH Inference of the 3DMM  Target distribution is our “posterior”: ෨  𝑄: 𝑄 𝜄 𝐽 𝑈 = ℓ 𝜄|𝐽 𝑈 , 𝐽 𝑆 𝑞 𝜄  Unnormalized  Point-wise evaluation only  Parameters  Shape: 50 – 200, low-rank parameterized GP shape model  Color: 50 – 200, low-rank parameterized GP color model  Pose/Camera: 9 parameters, pin-hole camera model  Illumination: 9*3 Spherical Harmonics for illumination/reflectance  ≈ 300 dimensions (!!) 10

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Metropolis Filtering MH-Filter: θ ′ θ ′ Q θ ′ |θ 𝑞 𝑏𝑑𝑑𝑓𝑞𝑢 reject θ 𝑝𝑚𝑒 → θ ′ θ ′ update θ ′ → θ  Markov Chain Monte Carlo Sampling: Result: 𝜄 1 , 𝜄 2 , 𝜄 3 , … … > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Results: 2D Landmarks  Landmarks posterior: Manual labelling: 𝜏 LM = 4 pix Image: 512x512  Certainty of pose fit?  Influence of ear points?  Frontal better than side-view? Yaw, σ 𝐌𝐍 = 4pix with ears w/o ears 1.4 ∘ ± 𝟏. 𝟘 ∘ −0.8 ∘ ± 𝟑. 𝟖 ∘ Frontal 24.8 ∘ ± 𝟑. 𝟔 ∘ 25.2 ∘ ± 𝟓. 𝟏 ∘ Side view 11

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Integration of Bottom-Up > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Metropolis Filtering MH-Filter: Prior MH-Filter: Face Box MH-Filter: Image θ ′ θ ′ Q θ ′ |θ 𝑞 𝑏𝑑𝑑𝑓𝑞𝑢 𝑞 𝑏𝑑𝑑𝑓𝑞𝑢 𝑞 𝑏𝑑𝑑𝑓𝑞𝑢 reject reject reject θ 𝑝𝑚𝑒 → θ ′ θ 𝑝𝑚𝑒 → θ ′ θ 𝑝𝑚𝑒 → θ ′ update θ ′ → θ 𝑚 𝜄,𝐺𝐶 𝑚 𝜄,𝐽 𝑄 0 𝜄 𝑄 𝜄|𝐺𝐶 𝑄 𝜄|𝐺𝐶, 𝐽 12

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 35 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Face analysis Roger F. asian 0.34 caucasian 0.52 blue eyes 0.19 brown eyes 0.69 wide nose 0.70 male 0.52 mustache 0.13 gaze Hor 20° yaw 34° pitch -8° roll 4° 13

40 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Occlusion-aware 3D Morphable Face Models Bernhard Egger, Sandro Schönborn, Andreas Schneider, Adam Kortylewski, Andreas Morel-Forster, Clemens Blumer and Thomas Vetter International Journal of Computer Vision, 2018 41 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Face Image Analysis under Occlusion Source: AFLW Database Source: AR Face Database 14

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 There is nothing like: no background model Maximum Likelihood Formulation: ℓ 𝜄; 𝐽 = ℓ 𝜄; 𝐽 = ෑ ෑ ℓ 𝜄; 𝐽 𝑦 ℓ 𝜄; 𝐽 𝑦 = ෑ ℓ 𝜄; 𝐽 𝑦 × ෑ ℓ 𝜄; 𝐽 𝑦 𝑞𝑗𝑦𝑓𝑚 𝑦 ∈ 𝐽 𝑦∈𝐺𝑕 𝑦∈𝐶𝑕 “ Background Modeling for Generative Image Models ” Sandro Schönborn, Bernhard Egger, Andreas Forster, and Thomas Vetter Computer Vision and Image Understanding, Vol 113, 2015. > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Occlusion-aware Model 𝑨 ∙ 𝑚 𝑜𝑝𝑜−𝑔𝑏𝑑𝑓 𝜄; ෩ 1−𝑨 𝑚 𝜄; ሚ 𝑚 𝑔𝑏𝑑𝑓 𝜄; ෩ 𝐽, 𝑨 = ෑ 𝐽 𝑗 𝐽 𝑗 𝑗 15

46 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Inference > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Initialisation: Robust Illumination Estimation Init 𝑨 Init 𝜄 𝑑𝑏𝑛𝑓𝑠𝑏 Init 𝜄 𝑚𝑗𝑕ℎ𝑢 47 16

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Results: Qualitative Source: AR Face Database 49 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Results: Qualitative Source: AFLW Database 17

50 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Results: Applications Source: LFW Database > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Modeling of 2D Images 18

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Portraits made to Measure  Computer can learn to model faces according to „human“ categories. Aggressive Trustworthy 19

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Modeling the Appearance of Faces  Which directions code for specific attributes ? > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Cape Town | 2018 Learning from Examples 20