Automatic Face Recognition in Weakly Constrained Environments - - PowerPoint PPT Presentation

Automatic Face Recognition in Weakly Constrained Environments

Fabien Cardinaux

cardinau@idiap.ch

Automatic Face Recognition in Weakly Constrained Environments – p.1/33

Outline

• Problem of Face Recognition in Weakly

Constrained Environments

• Traditional approaches for Face Recognition
• Databases and Evaluation
• Face Verification System based on Generative Models

(GMM)

• Future Plan

Automatic Face Recognition in Weakly Constrained Environments – p.2/33

Face Verification vs Identification

• Face Identification (FI): Find the identity of a given person out of a

pool of people

• Face Verification (FV): Authenticate the claimed identity based on

the face image

Recognition (Who is he?) Verification (Is he Mr X?)

Mr X Yes/No

Automatic Face Recognition in Weakly Constrained Environments – p.3/33

Weakly Constrained Environment

• Applications:
• Access control
• Banking transaction authentication
• Advanced video surveillance
• Weakly Constrained Environment:
• Unrestricted head pose
• Various in lighting conditions
• Background change
• Problems

Automatic Face Recognition in Weakly Constrained Environments – p.4/33

Challenge and Problems

• Wide variability between face images of the same

identity (due to the expression, lighting and head position changes)

• Limited number of reference images available by

identity => can not cover all possible variabilities

Automatic Face Recognition in Weakly Constrained Environments – p.5/33

Outline

• Problem of Face Recognition in Weakly Constrained

Environments

• Traditional approaches for Face Recognition
• Databases and Evaluation
• Face Verification System based on Generative Models

(GMM)

• Future Plan

Automatic Face Recognition in Weakly Constrained Environments – p.6/33

Face Recognition system

Generally, a full face recognition system can be decomposed into four stages:

• Face detection: Find the position of the face in the image
• Normalization: Reduce variabilities in the face images. Illumination

normalization and geometric normalization

• Feature extraction: Extract relevant information in the images
• Classification: Differs according to the specific task (identification
• r verification)

Automatic Face Recognition in Weakly Constrained Environments – p.7/33

Feature extraction (1/2)

Holistic Representation

• Principal Component Analysis (PCA): Projection is based on the

directions of largest variance of the face population.

• Linear Discriminant Analysis (LDA): More discriminant features

than the PCA subspace.

Automatic Face Recognition in Weakly Constrained Environments – p.8/33

Feature extraction (2/2)

Local representation

• Local PCA: Compute principal component of a set of sub-windows
• f images. The first p principal components were then used to filter

the full images

• 2D Gabor Filters: Face is represented by outputs of Gabor filters at

multiple scales, orientations, and spatial locations

• 2D DCT: Images analysed on a block by block basis. Each block is

decomposed in terms of 2D Discrete Cosine Transform (DCT) basis functions

Automatic Face Recognition in Weakly Constrained Environments – p.9/33

Classification

Computation of a score

• ✁

corresponding to an opinion on the probe

to be the identity

.

• Face identification:

• ✟
• ✁

(1)

• Face verification: Given a threshold

, the claim is accepted when

• ✁

and rejected when

• ✁

.

Automatic Face Recognition in Weakly Constrained Environments – p.10/33

Classification approaches

• Classification based on distance measures; e.g.: euclidean

distance between the feature vector of the probe image and the feature vector of a reference image

• Elastic Graph Matching
• Discriminant classifier: MLP or SVM
• Probabilistic/Generative approach: GMM or HMM

Automatic Face Recognition in Weakly Constrained Environments – p.11/33

Outline

• Problem of Face Recognition in Weakly Constrained

Environments

• Traditional approaches for Face Recognition
• Databases and Evaluation
• Face Verification System based on Generative Models

(GMM)

• Future Plan

Automatic Face Recognition in Weakly Constrained Environments – p.12/33

Databases

• XM2VTS (295 subjects):
• Multimodal: Face and speech
• Protocol for face verification: Lausanne Protocol
• FERET (1199 subjects):
• Designed for face recognition
• different camera, lighting, head pose
• BANCA (52 subjects) :
• Multimodal database: Face and Speech
• Protocol for face verification
• Three different scenarios (controlled, degraded and adverse)

Automatic Face Recognition in Weakly Constrained Environments – p.13/33

Evaluation of face verification systems

• Two types of errors:
• false acceptances (FA): The system accepts an impostor
• false rejections (FR): The system rejects a client
• To measure the performance of the system:
• ✁

• ✁☛✡

• ✂

• ✂

• Half Total Error Rate (HTER):

• ✁

• ✂

Automatic Face Recognition in Weakly Constrained Environments – p.14/33

Outline

• Problem of Face Recognition in Weakly Constrained

Environments

• Traditional approaches for Face Recognition
• Databases and Evaluation
• Face Verification System based on Generative

Models (GMM)

• Future Plan

Automatic Face Recognition in Weakly Constrained Environments – p.15/33

FV based on Generative Models

• 1. Face image decomposed into a sequence of overlaping windows
• 2. To reduce the dimensions of the observation vectors and to reduce

noise, we perform a feature extraction (such as 2D-DCT)

• 3. We train a GMM using a large set of face images from different

identities

• Universal Background Model (UBM). Trained by EM

algorithm

• 4. Client models are trained by adapting the UBM using MAP

adaptation

• 5. Opinion on the claim:
• ✁

Automatic Face Recognition in Weakly Constrained Environments – p.16/33

Feature extraction

Face extraction

Analyze on block by block Feature Vectors

DCT-mod2 Components

• Face extracted from image using location of the eyes (manually or

automatically located)

• Face image analysed on a block by block basis. Each block is

• (

) and overlaps neighbouring blocks by 50%

• Each block is decomposed in terms of DCT-mod2 (extension of 2D

DCT)

• ne face is represented by a set of feature vectors

Automatic Face Recognition in Weakly Constrained Environments – p.17/33

GMM training

• The UBM is trained by Expectation Maximization (EM) algorithm

using training data from all identities

• GMM parameters for each client model are found by adapting the

UBM using a maximum a posteriori (MAP) adaptation (In practice we adapt only the means) :

This approach deals with the problem of lack of training data for each identity.

Automatic Face Recognition in Weakly Constrained Environments – p.18/33

Verification decision

An opinion on the claim is found using:

• ✁

(2)

Since the UBM is a good representation of many clients, it is also used to find the likelihood of the claimant being an impostor, i.e.:

UBM

(3)

The verification decision is reached as follows: given a threshold

, the claim is accepted when

• ✁

and rejected when

• ✁

.

Automatic Face Recognition in Weakly Constrained Environments – p.19/33

Results on the XM2VTS db (LP1)

Manually located faces:

Approach FAR FRR HTER

GMM (IDIAP)

• 1.69

2.25 1.97 MLP (IDIAP)

• 3.22

3.50 3.36 NC (U. Surrey)

• 3.5

2.8 3.15

Automatically located faces:

Approach FAR FRR HTER

GMM (IDIAP)

• 2.15

2.75 2.45 MLP (IDIAP)

• 7.98

9.75 8.86 NC (U. Surrey)

• 7.6

6.8 7.2 EGM (U. Thessaloniki)

• 8.2

6.0 7.1

• Results from Face Recognition Contest ICPR2000
• Accepted for publication in AVPBA03 conference

Automatic Face Recognition in Weakly Constrained Environments – p.20/33

Outline

• Problem of Face Recognition in Weakly Constrained

Environments

• Traditional approaches for Face Recognition
• Databases and Evaluation
• Face Verification System based on Generative Models

(GMM)

• Future Plan

Automatic Face Recognition in Weakly Constrained Environments – p.21/33

Future Plan

• Face Recognition using HMM classifiers
• Application in a Weakly Controlled

Environment (Meeting Room)

• Face Recognition with unrestricted Head

Pose

Automatic Face Recognition in Weakly Constrained Environments – p.22/33

HMM approach for Face Recognition

Motivations:

• Use spatial information: This may increase

discrimination, as the spatial information is currently not used in the GMM approach

• Perform an alignement
• Good results in [Samaria94], [Nefian98] and

[Eickeler99]

• All results on ORL database

(non-challenging database)

• No adaptation approaches

Automatic Face Recognition in Weakly Constrained Environments – p.23/33

WCE - Meeting Room application

• Meeting recorded at IDIAP (Smart Meeting Room project)
• Evaluation of traditional approaches in a weakly controlled

environment (WCE)

• Developement of new approaches to deal with the problems posed

by WCE (such as head pose change)

• Integration of a face recognition system in a meeting annotation

application (Who was present at a particular meeting?)

Automatic Face Recognition in Weakly Constrained Environments – p.24/33

FR with unrestricted Head Pose

• Train several models corresponding to different head pose (out of

plane rotation)

• Weight the result of each expert using the likelihood on the head

pose

Automatic Face Recognition in Weakly Constrained Environments – p.25/33

Publications so far

[1] Face Verification Using MLP and SVM, Fabien Cardinaux and Sebastien Marcel, in "XI Journees NeuroSciences et sciences pour l’Ingenieur (NSI 2002)", 2002. [2] Comparison of MLP and GMM Classifiers for Face Verification

• n XM2VTS, Fabien Cardinaux, Conrad Sanderson, and Sébastien

Marcel, 4th International Conference on AUDIO- and VIDEO-BASED BIOMETRIC PERSON AUTHENTICATION, 2003. [3] Speech & Face Based Biometric Authentication at IDIAP , C. Sanderson, S.Bengio, H. Bourlard, J. Mariethoz, R. Collobert, M. F . BenZeghiba, F . Cardinaux, and S. Marcel, In International Conference

• n Multimedia and Expo, ICME, 2003.

Automatic Face Recognition in Weakly Constrained Environments – p.26/33

Thank You !

Automatic Face Recognition in Weakly Constrained Environments – p.27/33

Additional slides

Automatic Face Recognition in Weakly Constrained Environments – p.28/33

2D Discrete Cosine Transform (1)

• Face image is analyzed on a block by block basis
• Each block is 8
• 8
• 50% overlap
• Blocks decomposed in terms of 2D DCT basis functions

(64)

• Coefficients are ordered according to a zig-zag pattern,

reflecting the amount of information stored

1 u 2 v 1 2 3 3 Automatic Face Recognition in Weakly Constrained Environments – p.29/33

2D Discrete Cosine Transform (2)

• For block located at
• ✁✄✂

, the DCT feature vector is composed of:

• Only need to retain 15 coefficients (24%)
• Face is described by a set of vectors
• 56
• 64 (rows
• columns)

195 vectors

• Works well in non-challenging conditions
• Problem: illumination changes (intensity, direction)

Automatic Face Recognition in Weakly Constrained Environments – p.30/33

DCT-mod2 (1)

• Most affected coefficients:

• Throw them out?
• reduces performance
• ✁

• contain discriminative information
• Replace

• with their deltas
• In simplest form, deltas are differences between coefficients from neighbouring

blocks

Automatic Face Recognition in Weakly Constrained Environments – p.31/33

DCT-mod2 (2)

• Modify 2D DCT feature extraction by replacing the first 3 coefficients with their

horizontal and vertical deltas:

• ✁

• ✂

• ✁

• ✂

• ✁

• ✂

• ✏

• Refer to this approach as DCT-mod2
• Compare DCT-mod2 with DCT, PCA, PCA with histogram equalization and

2D Gabor wavelet based features

• Use an artificial illumination direction change:
• left:

(no change); middle:

; right:

Automatic Face Recognition in Weakly Constrained Environments – p.32/33

DCT-mod2 (3)

• Results obtained using a GMM based classifier in a verification scenario:

10 20 30 40 50 60 70 80 5 10 15 20 25 30 35 40 45 δ EER (%) PCA PCA + hist. equ. DCT GABOR DCT−MOD2 Automatic Face Recognition in Weakly Constrained Environments – p.33/33