Lecture: Face Recognition Juan Carlos Niebles and Ranjay Krishna - - PowerPoint PPT Presentation

Face identification

St Stanfor

• rd University

05-Nov-2019 1

Lecture: Face Recognition

Juan Carlos Niebles and Ranjay Krishna Stanford Vision and Learning Lab

Face identification

St Stanfor

• rd University

05-Nov-2019 2

CS 131 Roadmap

Pixels Images

Convolutions Edges Descriptors

Segments

Resizing Segmentation Clustering Recognition Detection Machine learning

Videos

Motion Tracking

Web

Neural networks Convolutional neural networks

Face identification

St Stanfor

• rd University

05-Nov-2019 3

Let’s recap

• A simple object recognition pipeline with kNN
• PCA

Face identification

St Stanfor

• rd University

05-Nov-2019 4

Object recognition: a classification framework

• Apply a prediction function to a feature representation of

the image to get the desired output:

f( ) = “apple” f( ) = “tomato” f( ) = “cow”

Slide credit: L. Lazebnik Dataset: ETH-80, by B. Leibe

Face identification

St Stanfor

• rd University

05-Nov-2019 5

A simple pipeline - Training

Training Images Image Features

Face identification

St Stanfor

• rd University

05-Nov-2019 6

A simple pipeline - Training

Training Labels Training Images Training Image Features

Face identification

St Stanfor

• rd University

05-Nov-2019 7

A simple pipeline - Training

Training Labels Training Images Training Image Features Learned Classifier

Face identification

St Stanfor

• rd University

05-Nov-2019 8

A simple pipeline - Training

Training Labels Training Images Training Image Features Image Features

Test Image

Learned Classifier

Face identification

St Stanfor

• rd University

05-Nov-2019 9

Prediction

A simple pipeline - Training

Training Labels Training Images Training Image Features Image Features

Test Image

Learned Classifier Learned Classifier

Face identification

St Stanfor

• rd University

05-Nov-2019 10

Prediction

A simple pipeline - Training

Training Labels Training Images Training Image Features Image Features

Test Image

Learned Classifier Learned Classifier

Face identification

St Stanfor

• rd University

05-Nov-2019 11

Image features

Input image

Color: Quantize RGB values

Invariance? ? Translation ? Scale ? Rotation ? Occlusion

Face identification

St Stanfor

• rd University

05-Nov-2019 12

Image features

Input image

Color: Quantize RGB values

Invariance? ? Translation ? Scale ? Rotation (in-planar) ? Occlusion

Face identification

St Stanfor

• rd University

05-Nov-2019 13

Image features

Input image

Color: Quantize RGB values

Invariance? ? Translation ? Scale ? Rotation (in-planar) ? Occlusion

Global shape: PCA space

Invariance? ? Translation ? Scale ? Rotation (in-planar) ? Occlusion

Face identification

St Stanfor

• rd University

05-Nov-2019 14

Image features

Input image

Color: Quantize RGB values

Invariance? ? Translation ? Scale ? Rotation (in-planar) ? Occlusion

Global shape: PCA space

Invariance? ? Translation ? Scale ? Rotation (in-planar) ? Occlusion

Face identification

St Stanfor

• rd University

05-Nov-2019 15

Image features

Input image

Color: Quantize RGB values

Invariance? ? Translation ? Scale ? Rotation ? Occlusion

Global shape: PCA space

Invariance? ? Translation ? Scale ? Rotation ? Occlusion

Local shape: shape context

Invariance? ? Translation ? Scale ? Rotation (in-planar) ? Occlusion

Face identification

St Stanfor

• rd University

05-Nov-2019 16

Image features

Input image

Color: Quantize RGB values

Invariance? ? Translation ? Scale ? Rotation ? Occlusion

Global shape: PCA space

Invariance? ? Translation ? Scale ? Rotation ? Occlusion

Local shape: shape context

Invariance? ? Translation ? Scale ? Rotation (in-planar) ? Occlusion

Face identification

St Stanfor

• rd University

05-Nov-2019 17

Image features

Input image

Color: Quantize RGB values

Invariance? ? Translation ? Scale ? Rotation ? Occlusion

Global shape: PCA space

Invariance? ? Translation ? Scale ? Rotation ? Occlusion

Local shape: shape context

Invariance? ? Translation ? Scale ? Rotation (in-planar) ? Occlusion

Texture: Filter banks

Invariance? ? Translation ? Scale ? Rotation (in-planar) ? Occlusion

Face identification

St Stanfor

• rd University

05-Nov-2019 18

Image features

Input image

Color: Quantize RGB values

Invariance? ? Translation ? Scale ? Rotation ? Occlusion

Global shape: PCA space

Invariance? ? Translation ? Scale ? Rotation ? Occlusion

Local shape: shape context

Invariance? ? Translation ? Scale ? Rotation (in-planar) ? Occlusion

Texture: Filter banks

Invariance? ? Translation ? Scale ? Rotation (in-planar) ? Occlusion

Face identification

St Stanfor

• rd University

05-Nov-2019 19

Prediction

A simple pipeline - Training

Training Labels Training Images Training Image Features Image Features

Test Image

Learned Classifier Learned Classifier

Face identification

St Stanfor

• rd University

05-Nov-2019 20

Classifiers: Nearest neighbor

Training examples from class 1 Training examples from class 2

Slide credit: L. Lazebnik

Face identification

St Stanfor

• rd University

05-Nov-2019 21

Prediction

A simple pipeline - Training

Training Labels Training Images Training Image Features Image Features

Test Image

Learned Classifier Learned Classifier

Face identification

St Stanfor

• rd University

05-Nov-2019 22

Classifiers: Nearest neighbor

Test example Training examples from class 1 Training examples from class 2

Slide credit: L. Lazebnik

Face identification

St Stanfor

• rd University

05-Nov-2019 23

Let’s recap

• A simple object recognition pipeline with kNN
• PCA

Face identification

St Stanfor

• rd University

05-Nov-2019 24

PCA compression: 144D -> 6D

Face identification

St Stanfor

• rd University

05-Nov-2019 25

2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12

6 most important eigenvectors

Face identification

St Stanfor

• rd University

05-Nov-2019 26

PCA compression: 144D ) 3D

Face identification

St Stanfor

• rd University

05-Nov-2019 27

2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12

3 most important eigenvectors

Face identification

St Stanfor

• rd University

05-Nov-2019 28

What we will learn today

• Introduction to face recognition
• The Eigenfaces Algorithm
• Linear Discriminant Analysis (LDA)

Turk and Pentland, Eigenfaces for Recognition, Journal of Cognitive Neuroscience 3 (1): 71–86.

• P. Belhumeur, J. Hespanha, and D. Kriegman. "Eigenfaces vs. Fisherfaces: Recognition

Using Class Specific Linear Projection". IEEE Transactions on pattern analysis and machine intelligence 19 (7): 711. 1997.

Face identification

St Stanfor

• rd University

05-Nov-2019 29

Courtesy of Johannes M. Zanker

“Faces” in the brain

Face identification

St Stanfor

• rd University

05-Nov-2019 30

“Faces” in the brain fusiform face area

Kanwisher, et al. 1997

Face identification

St Stanfor

• rd University

05-Nov-2019 31

Detection versus Recognition

Detection finds the faces in images Recognition recognizes WHO the person is

Face identification

St Stanfor

• rd University

05-Nov-2019 32

Face Recognition

• Digital photography

Face identification

St Stanfor

• rd University

05-Nov-2019 33

Face Recognition

• Digital photography
• Surveillance

Face identification

St Stanfor

• rd University

05-Nov-2019 34

Face Recognition

• Digital photography
• Surveillance
• Album organization

Face identification

St Stanfor

• rd University

05-Nov-2019 35

Face Recognition

• Digital photography
• Surveillance
• Album organization
• Person tracking/id.

Face identification

St Stanfor

• rd University

05-Nov-2019 36

Face Recognition

• Digital photography
• Surveillance
• Album organization
• Person tracking/id.
• Emotions and expressions

Face identification

St Stanfor

• rd University

05-Nov-2019 37

Face Recognition

• Digital photography
• Surveillance
• Album organization
• Person tracking/id.
• Emotions and expressions
• Security/warfare
• Tele-conferencing
• Etc.

Face identification

St Stanfor

• rd University

05-Nov-2019 38

The Space of Faces

• An image is a point in a high

dimensional space

– If represented in grayscale intensity, an N x M image is a point in RNM – E.g. 100x100 image = 10,000 dim

Slide credit: Chuck Dyer, Steve Seitz, Nishino

Face identification

St Stanfor

• rd University

05-Nov-2019 39

100x100 images can contain many things other than faces!

Face identification

St Stanfor

• rd University

05-Nov-2019 40

The Space of Faces

• An image is a point in a high

dimensional space

– If represented in grayscale intensity, an N x M image is a point in RNM – E.g. 100x100 image = 10,000 dim

• However, relatively few high

dimensional vectors correspond to valid face images

• We want to effectively model the

subspace of face images

Slide credit: Chuck Dyer, Steve Seitz, Nishino

ɸ1

Face identification

St Stanfor

• rd University

05-Nov-2019 41

Where have we seen something like this before?

ɸ1

Face identification

St Stanfor

• rd University

05-Nov-2019 42

Image space Face space

• Maximize the scatter of the training images in face space
• Compute n-dim subspace such that the projection of the data points onto the

subspace has the largest variance among all n-dim subspaces.

Face identification

St Stanfor

• rd University

05-Nov-2019 43

• So, compress them to a low-dimensional subspace that

captures key appearance characteristics of the visual DOFs.

Key Idea

• USE PCA for estimating the sub-space

(dimensionality reduction)

• Compare two faces by projecting the images into the subspace

and measuring the EUCLIDEAN distance between them.

Face identification

St Stanfor

• rd University

05-Nov-2019 44

What we will learn today

• Introduction to face recognition
• The Eigenfaces Algorithm
• Linear Discriminant Analysis (LDA)

Turk and Pentland, Eigenfaces for Recognition, Journal of Cognitive Neuroscience 3 (1): 71–86.

• P. Belhumeur, J. Hespanha, and D. Kriegman. "Eigenfaces vs. Fisherfaces: Recognition

Using Class Specific Linear Projection". IEEE Transactions on pattern analysis and machine intelligence 19 (7): 711. 1997.

Face identification

St Stanfor

• rd University

05-Nov-2019 45

Eigenfaces: key idea

• Assume that most face images lie on a low-dimensional subspace determined by

the first k (k<<d) directions of maximum variance

• Use PCA to determine the vectors or “eigenfaces” that span that subspace
• Represent all face images in the dataset as linear combinations of eigenfaces
• M. Turk and A. Pentland, Face Recognition using Eigenfaces, CVPR 1991

Face identification

St Stanfor

• rd University

05-Nov-2019 46

Training images: x1,…,xN

Face identification

St Stanfor

• rd University

05-Nov-2019 47

Top eigenvectors: ɸ1,…,ɸk

Mean: μ

Face identification

St Stanfor

• rd University

05-Nov-2019 48

Visualization of eigenfaces

Principal component (eigenvector) ɸk μ + 3σkɸk μ – 3σkɸk

Face identification

St Stanfor

• rd University

05-Nov-2019 49

Eigenface algorithm

• Training
• 1. Align training images x1, x2, …, xN
• 2. Compute average face
• 3. Compute the difference image (the centered data matrix)

Note that each image is formulated into a long vector!

µ = 1 N xi

∑

Face identification

St Stanfor

• rd University

05-Nov-2019 50

Eigenface algorithm

4. Compute the covariance matrix 5. Compute the eigenvectors of the covariance matrix Σ 6. Compute each training image xi ‘s projections as 7. Visualize the estimated training face xi

xi → xi

c ⋅φ1, xi c ⋅φ2,..., xi c ⋅φK

( ) ≡ a1,a2, ...,aK

( )

xi ≈ µ + a1φ1 + a2φ2 +...+ aKφK

Face identification

St Stanfor

• rd University

05-Nov-2019 51

Eigenface algorithm

6. Compute each training image xi ‘s projections as 7. Visualize the reconstructed training face xi

xi → xi

c ⋅φ1, xi c ⋅φ2,..., xi c ⋅φK

( ) ≡ a1,a2, ...,aK

( )

xi ≈ µ + a1φ1 + a2φ2 +...+ aKφK

a1φ1 a2φ2 aKφK ...

Reconstructed training face

𝑦"

Face identification

St Stanfor

• rd University

05-Nov-2019 52

Eigenvalues (variance along eigenvectors)

Face identification

St Stanfor

• rd University

05-Nov-2019 53

• Only selecting the top K eigenfaces à reduces the dimensionality.
• Fewer eigenfaces result in more information loss, and hence less

discrimination between faces.

Reconstruction and Errors

K = 4 K = 200 K = 400

Face identification

St Stanfor

• rd University

05-Nov-2019 54

Eigenface algorithm

• Testing

1. Take query image t 2. Project into eigenface space and compute projection 3. Compare projection w with all N training projections

• Simple comparison metric: Euclidean
• Simple decision: K-Nearest Neighbor

(note: this “K” refers to the k-NN algorithm, is different from the previous K’s referring to the # of principal components)

t → (t −µ)⋅φ1, (t −µ)⋅φ2,..., (t −µ)⋅φK

( ) ≡ w1,w2,...,wK ( )

Face identification

St Stanfor

• rd University

05-Nov-2019 56

Shortcomings

• Requires carefully controlled data:

–All faces centered in frame –Same size –Some sensitivity to angle

• Alternative:

–“Learn” one set of PCA vectors for each angle –Use the one with lowest error

• Method is completely knowledge free

–(sometimes this is good!) –Doesn’t know that faces are wrapped around 3D objects (heads) –Makes no effort to preserve class distinctions

Face identification

St Stanfor

• rd University

05-Nov-2019 57

Summary for Eigenface

Pros

• Non-iterative, globally optimal solution

Limitations

• PCA projection is optimal for reconstruction from a low

dimensional basis, but may NOT be optimal for discrimination… Is there a better dimensionality reduction?

Face identification

St Stanfor

• rd University

05-Nov-2019 58

Besides face recognitions, we can also do Facial expression recognition

Face identification

St Stanfor

• rd University

05-Nov-2019 59

Happiness subspace (method A)

Face identification

St Stanfor

• rd University

05-Nov-2019 60

Disgust subspace (method A)

Face identification

St Stanfor

• rd University

05-Nov-2019 61

Facial Expression Recognition Movies (method A)

Face identification

St Stanfor

• rd University

05-Nov-2019 62

What we will learn today

• Introduction to face recognition
• The Eigenfaces Algorithm
• Linear Discriminant Analysis (LDA)

Turk and Pentland, Eigenfaces for Recognition, Journal of Cognitive Neuroscience 3 (1): 71–86.

• P. Belhumeur, J. Hespanha, and D. Kriegman. "Eigenfaces vs. Fisherfaces: Recognition

Using Class Specific Linear Projection". IEEE Transactions on pattern analysis and machine intelligence 19 (7): 711. 1997.

Face identification

St Stanfor

• rd University

05-Nov-2019 63

Which direction will is the first principle component?

Face identification

St Stanfor

• rd University

05-Nov-2019 64

Fischer’s Linear Discriminant Analysis

• Goal: find the best separation between two classes

Slide inspired by N. Vasconcelos

Face identification

St Stanfor

• rd University

05-Nov-2019 65

Difference between PCA and LDA

• PCA preserves maximum variance
• LDA preserves discrimination

– Find projection that maximizes scatter between classes and minimizes scatter within classes

Face identification

St Stanfor

• rd University

05-Nov-2019 66

Illustration of the Projection

Poor Projection

x1 x2 x1 x2

• Using two classes as example:

Good

Face identification

St Stanfor

• rd University

05-Nov-2019 67

Basic intuition: PCA vs. LDA

Face identification

St Stanfor

• rd University

05-Nov-2019 68

LDA with 2 variables

• We want to learn a projection W such that the projection converts all the

points from x to a new space (For this example, assume m == 1):

• Let the per class means be:
• And the per class covariance matrices be:
• We want a projection that maximizes:

𝑨 = 𝑥&𝑦

Face identification

St Stanfor

• rd University

05-Nov-2019 69

Fischer’s Linear Discriminant Analysis

Slide inspired by N. Vasconcelos

Between class scatter Within class scatter

Face identification

St Stanfor

• rd University

05-Nov-2019 70

LDA with 2 variables

The following objective function: Can be written as

Face identification

St Stanfor

• rd University

05-Nov-2019 71

LDA with 2 variables

• We can write the between class scatter as:
• Also, the within class scatter becomes:

Slide inspired by N. Vasconcelos

Face identification

St Stanfor

• rd University

05-Nov-2019 72

LDA with 2 variables

• We can plug in these scatter values to our objective function:
• And our objective becomes:

Slide inspired by N. Vasconcelos

Face identification

St Stanfor

• rd University

05-Nov-2019 73

LDA with 2 variables

• The scatter variables

Slide inspired by N. Vasconcelos

Face identification

St Stanfor

• rd University

05-Nov-2019 74

2

S

1

S

B

S

2 1

S S SW + =

x1 x2

Within class scatter Between class scatter

Visualization

Face identification

St Stanfor

• rd University

05-Nov-2019 75

Linear Discriminant Analysis (LDA)

• Maximizing the ratio
• Is equivalent to maximizing the numerator while keeping the

denominator constant, i.e.

• And can be accomplished using Lagrange multipliers, where

we define the Lagrangian as

• And maximize with respect to both w and λ

Slide inspired by N. Vasconcelos

Face identification

St Stanfor

• rd University

05-Nov-2019 76

Linear Discriminant Analysis (LDA)

• Setting the gradient of

With respect to w to zeros we get

• r
• This is a generalized eigenvalue problem
• The solution is easy when

exists

Slide inspired by N. Vasconcelos

Face identification

St Stanfor

• rd University

05-Nov-2019 77

Linear Discriminant Analysis (LDA)

• In this case
• And using the definition of SB
• Assuming that (μ1-μ0)Tw=α is a scalar, this can be written as
• and since we don’t care about the magnitude of w

Slide inspired by N. Vasconcelos

Face identification

St Stanfor

• rd University

05-Nov-2019 78

LDA with N variables and C classes

Face identification

St Stanfor

• rd University

05-Nov-2019 79

Variables

• N Sample images:
• C classes:
• Average of each class:
• Average of all data:

{ }

N

x x , ,

1 !

å =

= N k k

x N

1

1 µ

å =

Î

i k

x k i i

x N

c

µ 1

Face identification

St Stanfor

• rd University

05-Nov-2019 80

Scatter Matrices

• Scatter of class i:

å

=

=

c i i W

S S

1

• Within class scatter:
• Between class scatter:

( )( )

T i k x i k i

x x S

i k

µ µ

c

• = å

Î

𝑇( = )

"*+ ,

)

• ."

(𝜈" − 𝜈-)(𝜈" − 𝜈-)&

Face identification

St Stanfor

• rd University

05-Nov-2019 81

Mathematical Formulation

• Recall that we want to learn a projection W

such that the projection converts all the points from x to a new space z:

• After projection:

– Between class scatter – Within class scatter

• So, the objective becomes:

W S W S

B T B =

~ W S W S

W T W =

~

W S W W S W S S W

W T B T W B

• pt

W W

max arg ~ ~ max arg = =

𝑨 = 𝑥&𝑦

Face identification

St Stanfor

• rd University

05-Nov-2019 82

Mathematical Formulation

• Solve generalized eigenvector problem:

W S W W S W W

W T B T

• pt

W

max arg =

m i w S w S

i W i i B

, , 1 ! = = l

Face identification

St Stanfor

• rd University

05-Nov-2019 83

Mathematical Formulation

• Solution: Generalized Eigenvectors
• Rank of Wopt is limited

– Rank(SB) <= |C|-1 – Rank(SW) <= N-C

m i w S w S

i W i i B

, , 1 ! = = l

Face identification

St Stanfor

• rd University

05-Nov-2019 84

PCA vs. LDA

• Eigenfaces exploit the max scatter of the training images in

face space

• Fisherfaces attempt to maximise the between class scatter,

while minimising the within class scatter.

Face identification

St Stanfor

• rd University

05-Nov-2019 85

Basic intuition: PCA vs. LDA

Face identification

St Stanfor

• rd University

05-Nov-2019 86

Results: Eigenface vs. Fisherface

• Variation in Facial Expression, Eyewear, and Lighting
• Input: 160 images of 16 people
• Train: 159 images
• Test:

1 image

With glasses Without glasses 3 Lighting conditions 5 expressions

Face identification

St Stanfor

• rd University

05-Nov-2019 87

Eigenface vs. Fisherface

Face identification

St Stanfor

• rd University

05-Nov-2019 88

What we have learned today

• Introduction to face recognition
• The Eigenfaces Algorithm
• Linear Discriminant Analysis (LDA)

Turk and Pentland, Eigenfaces for Recognition, Journal of Cognitive Neuroscience 3 (1): 71–86.

• P. Belhumeur, J. Hespanha, and D. Kriegman. "Eigenfaces vs. Fisherfaces: Recognition

Using Class Specific Linear Projection". IEEE Transactions on pattern analysis and machine intelligence 19 (7): 711. 1997.