Facial Expression Recognition Ian Endres April 10, 2009 Expression - - PowerPoint PPT Presentation

Facial Expression Recognition

Ian Endres April 10, 2009

Expression Overview

◮ Given image or video ◮ Identify 6 principal emotions: happiness, sadness, surprise,

fear, anger, disgust

Applications

◮ Robotics ◮ User/operator feedback

Method

◮ 1996 - Hand initialization, parametric flow, hand selected

predicates, simple hand written rules

◮ 2008 - Single image (Actor Dependent)

Dimensionality reduction (using DLLE), SVM classification

◮ 2008 - Motion (Actor Independent)

Automatic initialization, keypoint tracking, keypoint flow and displacement, nearest neighbor classification

Initialization

◮ Initialize initial frame, automatically track throughout ◮ Black - Mark box around features, head ◮ Yang

◮ Face detection - model skin with Gaussian color model ◮ Feature detection - Fit model to eye/mouth contours, iris edges

Motion Model (Black) Affine Model

u(x, y) = ao + a1x + a2y v(x, y) = a3 + a4x + a5y divergence = a1 + a5 curl = −a2 + a4 deformation = a1 − a5

Motion Model (Black) Planar, Curvature Models

Additional planar terms u : pox2 + p1xy v : poxy + p1y2 Additional curvature term: v : cx2

Model Fitting

Write model as (Affine, Planar, Curvature):

◮ Color constancy constraint: I(x, t) = I(x − X(x)P, t + 1) ◮ Thus minimize over P: x∈f ρ(∇I · (X(x)P) + It, σ) ◮ Error norm ρ(x, σ) is used to ignore outliers

Motion Model (Yang)

◮ Energy Model

◮ Distance from neutral : potential energy ◮ Velocity from optical flow : kinetic energy

Features

◮ Black - Predicates based on flow model parameters

Figure: Predicates for motion of 1) Mouth, eyebrows, eyes 2) Head

◮ Yang (Static) - Image embedding in R2, each person separate ◮ Yang (Motion) - Energy values for each image point

Learning/Inference

◮ Black - Detect expression change via conjunctions of

predicates

◮ Yang (Static)

◮ Train 1v1 polynomial kernel SVM ◮ 20 training examples per expression per person ◮ Inference using these classifiers

◮ Yang (Motion)

◮ “Apex” of expression as region of constant kinetic energy ◮ Label using potential energy states, using exemplar/nearest

neighbor method

Learning/Inference (Cont)

Figure: 1) Predicate Rules for Black 2) Temporal behavior of expressions

Results (Black)

Results (Yang, Static)

Drawbacks/Limitations

◮ Artifical Datasets ◮ Explicit initialization ◮ Black

◮ No machine learning

◮ Yang

◮ Requires head motion and fast face motions to be avoided ◮ No quantitative results for motion!