Sparse Coding Trees with Application to Emotion Classification - - PowerPoint PPT Presentation

Sparse Coding Trees with Application to Emotion Classification

Kevin H.C. Chen Marcus Z. Comiter

• H. T. Kung

Bradley McDanel AMFG 2015

Harvard University

Business Applications

User feedback systems, advertising, security systems

Politics

Automatic derivation

• f voter preferences,

focus group testing

Medicine

Additional metrics for patient care, helping children with autism

Application Motivation

Emotion Classification for IoT and Beyond

Methodology Motivation

Machine Learning and Unsupervised Feature Extraction

Feature 1 Feature 2

• Sparse coding makes data

more linearly separable

• Labels are not required

Sparse Coding Pipeline for Classification

Sparse Coding Classifier (e.g., SVM)

Transform data into feature representation Prediction with simple classifiers such as SVM

x z

label

Unsupervised Supervised

Representation by Sparse Coding

argminz | | x - Dz| |

2 + λ|

| z| |

2 Express the input signal (x) as the weighted (z) sum of a few features (D)

Note: we can also penalize L1 norm instead of L0 norm

Dictionary Learning

argminD,Z | | X - DZ| |

2 + λ|

| Z | |

Sparse Coefficients

• Finds common patterns in training data
• Solved by alternating updates of D and Z

2

Our Enhancement to SC

Sparse Coding tree (SC-tree)

to learn features with hierarchy

Non-negative constraints

to mitigate over-fitting in SC

Mirroring

to increase variation tolerance

Sparse Coding Tree

Learning Features for Hard Cases

• Some discriminating features can be subtle
• Finding clusters within clusters, similar to how

hierarchical k-means works

Fear can be confused with happiness because they both display teeth

Constructing Sparse Coding Tree

Input Sparse Coding Classifier (e.g., SVM) Group/Label Assignment label label label

If certain classes get confused consistently, put them through another layer of feature extraction

anger contempt sadness disgust fear happiness surprise

Branching in Sparse Coding Tree

Based on the confusion matrix from the coarse predictor

Features Learned in SC-tree

Features learned in the root node Features learned in a happy v.s. fear node

happy Could be happy or fear fear

Input label label label

max pooling split Sparse coding (LASSO/OMP) flip Sparse coding (LASSO/OMP)

Mirroring for Reflection Invariance

Using max pooling to capture the horizontal symmetry inherent in emotion classification

A reflected image would get the exact same representation

Improved Robustness with Mirroring

With max pooling, we always pick up response from the side of face with stronger features

Nonnegative Sparse Coding

argminz | | x - Dz| |

2 + λ|

| z| |

2

s.t. D ≥ 0, z ≥ 0

D with NN-constraint D without NN-constraint

Tends to learn regional components

Nonnegativity prevents cancelation of components, and therefore mitigates over-fitting

Datasets

Cohn-Kanade Extended Dataset (CK+) Emotions in the Wild Dataset (EitW) GENKI-4K Dataset AM-FED Dataset

CK+ GENKI AM-FED

Multi-class Multi-class Binary Binary

after pre-processing

• riginal data

Performance on Emotion Classification

Results reported in average recall

The sparse coding tree improves the performance of

• ur pipeline consistently.

79.9

SC NNSC MNNSC

75.1 70.1 73.6 71.5 76.8

w/SC Tree w/o SC Tree CK+ dataset

Performance on Emotion Classification

Results reported in average recall

The sparse coding tree improves the performance of

• ur pipeline consistently.

33.0

SC NNSC MNNSC

29.4 26.5 28.6 28.1 29.7

w/SC Tree w/o SC Tree EitW dataset

MNNSC Performance

Results reported in area under curve

with Mirroring and the non-negativity constraint, even greedy methods like OMP (L0) can be competitive

90.0

L0-min L1-min

best reported 96.1

L0-min L1-min

best reported 95.1 96.7 92.3 96.2 95.7 93.1 91.2 89.7 92.1 88.8 86.0 97.0

sparse coding Non-negativity Mirroring

GENKI-4K AM-FED

Applying Sparse Coding Tree to Action Recognition

Tested on KTH dataset

with SC Tree 92.13 % without:

86.57 %

Conclusion

Sparse coding, as an effective feature extraction method, can be enhanced by these techniques:

Sparse Coding tree (SC-tree)

to learn features with hierarchy

Non-negative constraints

to mitigate over-fitting in SC

Mirroring

to increase variation tolerance