Face Identification by Image Comparison done by pixel analysis ? - - PDF document

1

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Face Image Analysis Applications

Probabilistic Morphable Model Fitting Basel2018 Thomas Vetter University of Basel

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Face Identification by Image Comparison

? But which pixel to compare with which ? … done by pixel analysis Shape information tells us which pixel to compare

2

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Analysis by Synthesis

3D World Image Analysis Synthesis Image Model Image Description

model parameter

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Change Your Image ...

3

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Analysis by Synthesis

3D World Image Analysis Synthesis Image Model Image Description

model parameter

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

THE BIG QUESTION: How is this Image Model structured?

Possibly, there is no final answer! Is it: 2D, an image based rendering model?

• r

3D, a full 3D computer graphics model?

• r ….

4

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Linear Object Class Idea

Linear Object Classes and Image Synthesis from a Single Example Image. Thomas Vetter and Tomaso Poggio IEEE P AMI 1997, 19(7), 733-742.

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Separating shape and texture in 2D images

5

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

2D Morphable Face Image Model

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Linear Object Class Idea

6

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Image based rendering

7

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Synthesis of novel views from a single face image. Thomas Vetter, IJCV 1998, 28(2), 103-116. > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Morphable 2D Face Model

=

α1𝑆 + α2𝑆 + α3𝑆 + α4𝑆 + … β1𝑆 + β2𝑆 + β3𝑆 + β4𝑆 + …

8

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Morphable 3D Face Model

R               

α1 + α2 + α3 + α4 + ⋯ β1 + β2 + β3 + β4 + ⋯

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Morphable Models for Image Registration

Output

R = Rendering Function ρ = Parameters for Pose, Illumination, ...

Optimization Problem: Find optimal α, β, ρ !

R               

β1 + β2 + β3 + ⋯ α1 + α2 + α3 + ⋯

9

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Face Recognition

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

18

Normalizing for pose, illumination and …

?

Shape recovery Illumination inversion Shape recovery Illumination inversion

10

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Face recognition

Images: CMU-PIE database. (2002)

Complex Changes in Appearance

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

3D Morphable Model

11

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Identification by shape and texture coefficients only

Gallery … ,

i i

 

Model- Fitting

,

i i

 

Model- Fitting

,

i i

 

Model- Fitting

Test ,

i i

 

Model- Fitting compare Identity

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Face analysis

Roger F. asian caucasian blue eyes brown eyes wide nose male mustache gaze Hor yaw pitch roll 0.34 0.52 0.19 0.69 0.70 0.52 0.13 20° 34°

• 8°

4°

12

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Multi-PIE: Face recognition

60 70 80 90 100 15° 30° 45°

3DGEM [16] 3DMM [17] 3DMM ours [18]

[16] Prabhu et al., “Unconstrained Pose-Invariant Face Recognition using 3D Generic Elastic Models”, PAMI 2011 [17] Schönborn et al., “A Monte Carlo Strategy to Integrate Detection and Model-Based Face Analysis”, GCPR 2013 [18] Egger et al., “Pose Normalization for Eye Gaze Estimation and Facial Attribute Description ”, GCPR 2014

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Try a new hairstyle!

3D Geomety and Texture 3D Pose, Position Illumination, Foreground, Background

13

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Try a new hairstyle!

3D Geomety and Texture 3D Pose, Position Illumination, Foreground, Background

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Image Preprocessing for FRVT 2002

14

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Image Preprocessing for FRVT 2002

15

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Skin Detail Analysis for Face Recognition Jean Sebastian Pierrard , Thomas Vetter CVPR 2007

Skin Detail Analysis for Face Recognition

16

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Overview

Characterizing moles  Appearance Blob detection  Location Skin segmentation  Importance Saliency measure  Reference Systsem Recognition Morphable Model

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Data used

 Results based on subset of FERET-data base Gray scale Medium resolution (10-20k pixels face area) Mole sizes: 2-20 pixels

17

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Morphable Model for Correspondence

Fitting Fitting

Correspondence

3D reconst.

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

3DMM maps visible region on a common reference

Fitting Fitting

Correspondence

3D reconst.

18

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Morphable Model for Correspondence II

Fitting Fitting 3D reconst. Rendering

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Mole Detection: Shading Problem

 Template matching is sensitive to intensity gradients !

19

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Illumination Compensation

( )

ic

E x



( ), ( ) I x z x

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Mole Detection: Shading Problem

0.59cc 0.56cc 0.82cc 0.75cc

Local fitting

20

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

False Positives

 Templates also match common facial features  Sporadic hits due to hairstyle, beard, …  We need to mask out non-skin regions / outliers  3DMM is not sufficient

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Selection by Saliency

21

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Recognition

 Find matching pairs of moles in reference frame  Identification score: weighted sum of saliencies from matched points

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Face Recognition

• Based only on mole locations and saliency.

22

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Manipulation of Faces

Modeler

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Modeling of 2D Images

23

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Face Exchange Tasks

Source Target

Blend artificial edges

(3DMM)

Overlay target

• cclusions

Remove outliers from source texture Color balance & Illumination

(3DMM)

Scale & Orientation

(3DMM)

Difficult problem, even for humans. Has never be automated !

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

24

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Change Your Image ...

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Continuous Modeling in Face Space

Caricature Anti Face Average

Original

25

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Modeling the Appearance of Faces

Which directions code for specific attributes ?

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Learning from Examples

26

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Attributes of Faces

Gender Weight Original

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Computer can learn to model faces according to „human“ categories.

Aggressive Trustworthy

Portraits made to Measure

27

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Portraits made to Measure

Trustworthiness Social Skills Risk Seeking Likeability Extroversion Aggressiveness

% Correct ratings

100 90 80 70 60 50 40 30 20 10

Personality traits

Portraits made to measure: Mirella Walker and Thomas Vetter Journal of Vision, 9(11):12, 1-13, 2009

.

Aggressiveness Extroversion Likeability Risk Seeking Social Skills Trustworthiness Original Face Aggressiveness Aggressiveness Extroversion Extroversion Likeability Likeability Risk Seeking Risk Seeking Social Skills Social Skills Trustworthiness Trustworthiness Original Face Original Face

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Expressions

Original

28

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Simulation of Aging of Human Faces in Images

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Aging model:

model predicts perceived age

Labeled / True age

20 years 70 years

Predicted age

29

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Ageing: linear shape model only

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Example-based aging

Target Image Shape and Skin of donor transferred to target Donor Image

30

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Example-based Texture: The Problem

+ 5 years + 5 years

Target Image AGE: 40 Shape and Skin of donor AGE: 45 Shape and Skin of donor AGE: 50

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Parametric Pigmentation Model

𝜏 regulates the spread 𝑣, 𝑤 learned freakle position from example data 𝛻 The parameters 𝜏, 𝑣, 𝑤 and different freckle shapes are learned by detecting freckles in given faces

𝜏, 𝑣, 𝑤

Facial texture source detected freckles learned distribution parameters

𝜍(𝑦, 𝑧, σ) = ෍

𝑣,𝑤∈𝛻

𝒪 ((x−u,y−v)𝑈, σ)

31

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Parametric Pigmentation Model

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Aging Model

Shape: continuous Pigmentation: stochastic Wrinkles: example based

32

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

= smile

• Original:

+ smile = Novel Face:

Transfer of Facial Expressions

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Expression Invariant 3D Face Recognition with a Morphable Model

Brian Amberg, Reinhard Knothe and Thomas Vetter IN: IEEE Proceedings FG2008: 8th International Conference Automatic Face and Gesture Recognition, Amsterdam, The Netherlands, 2008.

33

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Input 3D Scans

Expression Invarant 3D Face Recognition

Approximation of Input with Morphable Expression Model Expression Normalized

Expression Invariant 3D Face Recognition with a Morphable Model Brian Amberg, Reinhard Knothe and Thomas Vetter, IEEE FG2008

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Linear Expression Model

Modeling facial expressions in a separate subspace

e e e

( , )

n n n

F M M         ( ) F M      ( ) ( ( , ))

n e

Data R F T     

Face Scans differ in Orientation and Translation

34

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Expression Transfer

Id           Xp                                

Fitting Fitting

1 1 1

, ,

ID XP

  

2 2 2

, ,

ID XP

  

1 2 1

, ,

ID XP

  

Rendering

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

3D Scans of Visemes

aao r th ea @@ ch fv ii kgnl

• -ou

pbm uh uu w w-au tdsz Reference

35

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Mouth Mesh

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Principal Components

Mouth Modeler

36

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Principal Components

Mouth Modeler

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Mouth Modeler

Principal Components

37

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Speech Animation

Text Audio Phonemes Visemes Morph Targets Video

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Retargeting Face Motions

38

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE

Animation of Images