PhD Dissertation Presentation Accurate and Robust Methodology for - PowerPoint PPT Presentation

PhD Dissertation Presentation Accurate and Robust Methodology for Pose and Spontaneous based Facial Expression Recognition Mr. Muhammad Hameed Siddiqi Department of Computer Engineering Kyung Hee University, South Korea Email: siddiqi@oslab.khu.ac.kr Advisor: Prof. Sungyoung Lee, PhD

Contents Facial Expression Recognition Background Introduction & Motivation  Problem Statement, Goals, and Challenges  Existing Works and their Limitations Proposed Methodology Face Detection and Extraction  Feature Extraction  Recognition Model  Experimental Setups & Results Uniqueness and Contributions Conclusion and Future Research 2/58

Introduction Facial Expression Recognition (FER) In our daily conversation, FER has 55% contribution [1]. Spontaneous- Pose-based FER based FER [5,6] [2,3] Artificially made by the subjects, i.e., observed on a day-to-day basis, e.g., during they are forced to perform [4] conversations or while watching TV [4] 3/58

Motivation The features mentioned in the figure are Therefore, a system is required highly merged due to similarity among the to capable of handling posed and spontaneous  FER issues expressions that results in high within-class to incorporate more intelligent capabilities to variance and low between-class variance  transform experimental prototypes into actual usable applications. to identify and characterize some of the most  relevant limitations in the FER Spontaneous Pose based FER System based 3D-feature plot for six different types of facial expressions. 4/58

Problem Statement How do we accurately and robustly recognize facial expressions in complex environments? How do we extract and select the best features from the contributing parts of the face?  To what extend is the FER methodology able to predict the label of the facial expressions in dynamic  environments? Goals To design, implement, and evaluate an accurate and robust methodology for pose and spontaneous based FER, which has the following objectives reduce classes similarities, accurate face detection, best features extraction and selection, complex distribution-based recognition in  naturalistic environment Challenges To minimize and maximize the variances within and between the classes To maintain higher accuracy in pose and spontaneous FER domains Robustness of the system against different domains 5/58

Research Taxonomy Human Emotion Recognition Physiological Audio Based Video Based Based Face Feature Feature Classification Detection Extraction Selection Stepwise Linear Discriminant Analysis Face Detection Geometric Appearance Muscle Motion Sequence Head Pose Frame Based & Extraction Based Based Based Based Hidden Conditional Active Contour Level- Wavelet Transform Random Fields set based Model Accurate Expressions Chan-Vese Energy Function Optical Flow Classification Bhattacharyya Distance Function 6/58

Related Work and their Limitations FER System Modules Methods Datasets Accuracy Limitations The performance of appearance-based methods degrades with the environmental change [27]. A prior knowledge is required for these methods, i.e., at the time of implementation for these techniques, it is compulsory to decide randomly which intensity information will be Appearance-based [7-10] important [28]. Yale B Feature-based methods [11-15], The performance of geometric-based approaches degrades with the variation in lighting FEI Geometric-based methods [16-18], conditions and viewpoint [29]. Face Detection CK 80-91% Knowledge-based methods [19-21], For knowledge-based approaches it is very hard for these approaches to build an CK+ Skin tone-based methods [22-25], and appropriate set of rules. If the rules are too general then there could be several false AT&T Template-based methods [26]. positives, or there could be false negatives if the rules are in too detail [30]. Skin tine-based methods are very sensitive to illumination like under varying lighting conditions [31]. Template-based methods are very sensitive to pixel misalignment in sub-image areas and depends on facial component detection [32]. Some holistic methods such as Nearest Features Line-based Subspace Analysis [33], Eigenfaces and Eigenvector [34], [35] and [36], Fisherfaces [37], global CK 77-85% features [38], Independent Component Analysis (ICA) [39, 40], Principal JAFFE Component Analysis (PCA) [41-43], frequency-based methods [44], Gabor MMI Holistic methods are very sensitive to variations in pose, illumination, occlusion, aging, and wavelet [45]. CMU-PIE rotation changes of the face [46], [47]. Feature Extraction Some local feature-based methods such as Local Feature Analysis (LFA) [48], Yale B+ The performance of local feature-based methods degrade in non-monotonic illumination Gabor features [49], Non-negative Matrix Factorization (NMF) and Local non- USTC-NVIE change, noise variation, change in pose, and expression conditions [28]. negative Matrix Factorization (LNMF) [50], and Local Binary Pattern (LBP) [51, FEI 72-92% 52], CK+ Local Transitional Pattern (LTP) [53], Local Directional Pattern (LDP) [54]. PCA has poor discriminating power [57]. LDA-based methods suffer from the limitations that their optimality criteria are not directly Principal Component Analysis (PCA) [56], CK associated to the classification capability of the achieved feature representation [59]. Linear Discriminant Analysis (LDA) [58], JAFFE Feature Selection 83-87% KDA does not have the capability to provide better performance in the case if the face Kernel Discriminant Analysis (KDA) [60], CK images of the same subjects are scattered rather than dispersed as clusters [61]. Generalized Discriminant Analysis (GDA) [62] CK GDA might not be stable and perhaps is not optimal in terms of the discriminant ability if there is small sample size data in the training data [63].

Related Work and their Limitations FER System Methods Accuracy FER System Methods Accuracy Module Module Gabor wavelet [16], Local Binary Appearance-based [7-10], Skin tone- Pattern [17,18], Local Transitional Feature Extraction 72-92% Face Detection based methods [11-14], and 80-91% Pattern [19], Local Directional Template-based methods [15] Pattern [20] Limitations Limitations The performance of these methods degrade during illumination change, • The performance of appearance-based methods degrades with the • noise variation, change in pose, occlusion, and expression conditions environmental change [21] [21] Skin tine and template based methods are very sensitive to illumination • [22] and to pixel misalignment in sub-image areas [23] respectively FER System Methods Accuracy FER System Methods Accuracy Module Module Linear Discriminant Analysis [24], Support vector machine [26], hidden Kernel Discriminant Analysis, 84-87% Classification 85-90% Feature Selection Markov model [27] Generalized Discriminant Analysis [25] Limitations Limitations SVM has no direct estimation of the probability [30] • The optimality criteria of LDA is not directly associated to the • HMMs presume that the current state depends only on the previous • classification capability of the achieved feature representation [28] state. Because of this assumption, labels of two contiguous states must If there is small sample size data in the training data, GDA is not optimal • hypothetically occur consecutively in the observed sequence. However, in terms of the discriminant [29] this presumption is not always true in reality 8/58

Related Work and their Limitations       exp , , f Y S X     | ; S p Y X    , z X            Pr   log , ,      u y Y Pr , , , , f Y S X y y y Y   y y  y 1        Tr log A { ss } S ,     T      ss ss          Tr , , , { } , f Y S X s s s s ss S      2 1   ss t t 1 log 2       Occ 2  s  , 1 t  s s 2 2     T    s      Occ , , , , f Y S X s s s S  s t   M s ,  1 1 t  s 2   T s         M , , , , f Y S X s s x s S 1 1 , s t t    M 2  t 1  s 2 2 s   T             M 2 T , , , ,     f Y S X s s x s S 2      2 exp , , ( ) , ( , , ) f Y S X u s A s s N x s t t  1 t 1 t t s s   t 1 t t t 1 S S     M   exp , , , f Y S m X      1 | ; S m p Y X    , z X These equations imply that with a particular set of values, the observation density at each state will converge to Gaussian form. Unfortunately, there is an algorithm that could guarantee this conversion. Therefore, these assumptions may result in a decrease of accuracy 9/58