Towards Automated Recognition of Human Emotions Using EEG Haiyan Xu - PowerPoint PPT Presentation

Towards Automated Recognition of Human Emotions Using EEG Haiyan Xu The Edward S. Rogers Sr. Department of Electrical and Computer Engineering, University of Toronto July 12, 2012

Research Objective To investigate whether EEG signal is feasible for affect detection analysis, especially using consumer grade EEG devices. 1/23

Outline 1 Overview of Affect Classification 2 Affect Detection System Overview Preprocessing: Filtering Based on Instantaneuous Frequency Feature Analysis Channel Selection: Genetic Algorithm (GA) 3 Experimental Setup 4 Simulation Results 5 Summary of Research Contributions 6 Future Works 2/23

Overview of Affect Classification Overview of Affect Classification Affective Computing Emotion is an unique, personal expression that differs under social context, culture background and personal experience Affective is the raw neurophysiological expression of emotion Affect Sensitive Applications Human Machine Interface (HMI) Health and rehabilitation applications Multimedia content indexing and retrieving 3/23

Overview of Affect Classification Research Motivations Prior Assessment Methods Facial expression, Voice, Physiological signals: heart rate, GSR, .. Disadvantages Not available due to inability to express emotions (autism disorders) Not reliable due to noisy source (crowded place) Interference due to non-emotional factors Affect Computing and EEG EEG signal originates from the Central Nervous system Brain networks in the limbic system are associated with affect expression Works with inaccessible and non-coorperative cases (autism disorders) Less influenced by non-emotional factors 4/23

Overview of Affect Classification Technical Challenges of Affect Detection Analysis Using EEG General Challenges EEG signals are originated from a non-linear, non-stationary processes Most signal processing systems are linear, use predefined bases, and assume stationary signals Lack of understanding on the dynamics of brain networks and correlation with emotional states Challenges Faced by Using Consumer-grade EEG Devices Much fewer number of electrodes available and conventional spatial analysis is not applicable Optimal sensor configuration for such applicaiton is not studied Constraints on computational complexity 5/23

Overview of Affect Classification Research Contributions Main Contributions Novel signal representation for EEG based on Instantaneuous Frequency Automatic determination of optimal sensor configuration for affect application that can be generalized easily to other applications Additional Contributions First person to design and implement a compelete EEG signal processing system for affect detection at our university Investigated the implications on how key parameters. (e.g., sampling rate) affects system performance 6/23

Affect Detection System Overview System Overview Test EEG Signals 7/23

Affect Detection System Overview Preprocessing: Filtering Based on Instantaneuous Frequency Preprocessing: Novel Filtering Based on Instantaneuous Frequency Output Class labels Feature Input signals Preprocessing Classifiers Analysis Purposes: to reduce noises, artefact and other external interferences Fourier and Wavelet Based Methods use a priori basis Multivariate Empirical Mode Decomposition (MEMD) is Instantaneuous Frequency based To obtain meaningful IF, Hibert Transform requires signal to be monocomponent, zero-mean locally [1] Data-driven, suitable for non-stationary signals from non-linear processes Decompose original signal into multiple time-varying frequency content, Intrinsic Mode Functions (IMFs) [1] L. Cohen. Time-Frequency Analysis, Prentice Hall, Englewood Cliffs (1995). 8/23

Affect Detection System Overview Preprocessing: Filtering Based on Instantaneuous Frequency Preprocessing Cont.: MEMD as Filter bank MEMD is useful as time-varying filtering technique (preprocessing), particularly for multicomponent signals. Frequencies of interest Alpha wave 8 − 12 Hz Beta wave 13 − 30 Hz Reconstruct EEG signal using sum of IMFS 0.6 IMFs obtained using Multivariate EMD 0.4 166.15 0.2 −2 10 5 10 15 20 0.6 0.4 107.17 0.2 5 10 15 20 0.6 0.4 64.12 0.2 −4 10 5 10 15 20 2 PSD in Log scale 25.05 1 0.4 5 10 15 20 14.93 0.2 −6 10 5 10 15 20 0.6 0.4 8.67 0.2 5 10 15 20 0.4 6.95 0.2 −8 5 10 15 20 10 0.6 0.4 5.51 0.2 0 5 10 15 20 0.6 4.81 0.4 0.2 0 1 2 5 10 15 20 0.3 4.52 0.2 0.1 5 10 15 20 0.4 8.09 0.2 5 10 15 20 Time 9/23

Affect Detection System Overview Feature Analysis Feature Analysis Output Class labels Feature Input signals Preprocessing Classifiers Analysis Purpose Dimension and computational complexity reduction Increase class separability Feature Analysis Algorithms Used Oscillation pattern variation in the time domain Six statistical features: e.g., Mean, Std, Skewness, Kurtosis.. Higher Order Crossings (HOC): zero crossing counts with iterative filtering process Event-related energy variation Spectral domain: narrow band energy (1Hz resolution, 8 − 30 Hz range) Time-spectral: wavelet-based energy and entropy analysis 10/23

Affect Detection System Overview Channel Selection: Genetic Algorithm (GA) Channel Selection: Genetic Algorithm (GA) Application of GA For 54 channels, there are 2 54 combinations, impossible for exhaustive EEG Features search Genetic Algoritm is a non-ranking Initial global optimization method Population Binary string representation for Fitness Calculation channel information Correct classification rate was used as Continue fitness function No Evolution? Channels were selectd on results from Yes 10 runs of GA Crossover Optimal Mutation Features Updated Population 11/23

Affect Detection System Overview Channel Selection: Genetic Algorithm (GA) Classifiers Output Class labels Feature Input signals Preprocessing Classifiers Analysis Purpose: Determine optimal decision boundaries between classes Classifiers Two simple classifiers were used to reduce computational complexity Linear Discriminate Analysis (LDA): optimal linear boundary between classes k Nearest Neightbors (kNN): majority voting, Euclidean distance was used as the distance metric 10 fold cross validation process was used to obtain the averaged simulation results shown in later sections A portion of the training samples are used for training, and the remaining, unseen samples used for testing purposes 12/23

Experimental Setup Experimental Setup: Database Description Arousal Emobraine-eNTERFACE06 Positively Negatively Excited Excited Validated public dataset, ideal for Valence results comparison Calm 5 subjects, aged 22-38 , 3 classes of affects, stimuli: IAPS images Biosemi Active II: 64 channels EEG headset 3 sessions and 30 trials each session 13/23

Experimental Setup Experimental Setup: Experiment Description Raw EEG Signal setup Multivariate EMD EEG Reconstruction Three experiments were conducted: with and without channel reduction Two testing cases: Genetic Referenced Algorithm Emotive Suject Specific: Training and testing samples are from the same Feature Analysis subject Cross Subject:Training and testing Classification: kNN, LDA samples are from all subjects Emotions 14/23

Experimental Setup Experimental Setup: Emotive Epoch v.s Biosemi Active II Biosemi Active II is a medical grade EEG headset Emotive Epoch is a popular consumer grade EEG headset Device Biosemi Active 2 Emotive EPOC SDK Data Format EDF MAT Resolution 24 bits ADC 16 bits (14 bits effective) Sampling Rate 1024Hz 128 SPS (2018 Hz internal) Channels 64 14 Channels in common AF 3, F 7, F 3, FC 5, FC 6, F 4, F 8, AF 4 15/23

Simulation Results Simulation Results Using all Channels Cross-Subject Emotion Recognition Classifier Statistical Narrow-bands Power HOC Wavelet 5NN 81.39 82.62 90.77 77.44 LDA 59.18 63.49 79.64 55.90 This results show that EEG signal is feasible for affect analysis even with simple classfiers kNN classifier provides better results which implies the features are not linearly seperable HOC is most representative for affect analysis 16/23

Simulation Results Simulation Results with Channel Reduction Referecend to Emotive Table: Cross-subject Recognition rate using only 8 electrodes Classifier Statistical Narrow-bands Power HOC Wavelet 5NN 68.15 78.15 89.64 58.87 LDA 38.77 39.90 43.13 37.74 Currently in market consumer grade EEG headset is feasible for affect detection applications HOC and kNN combination provides the best performance 17/23

Simulation Results Channels selected using GA Given the cost and usability, consumer grade EEG headsets typically have less than 14 channels. Selected 8 channels Selected 14 channels FPz FPz FP1 FP2 FP1 FP2 AF7 AF8 AF7 AF8 AF3 AF4 AF3 AF4 AFz AFz F7 F8 F7 F8 F5 F6 F5 F6 F3 F4 F3 F4 F1 F2 F1 F2 Fz Fz FC7 FC7 FC8 FC8 FC5 FC6 FC5 FC6 FC3 FC1 FCz FC2 FC4 FC3 FC1 FCz FC2 FC4 C5 C5 T7 C3 C1 Cz C2 C4 C6 T8 T7 C3 C1 Cz C2 C4 C6 T8 CP1 CPz CP2 CP1 CPz CP2 CP3 CP4 CP3 CP4 CP5 CP6 CP5 CP6 TP7 TP8 TP7 TP8 P1 Pz P2 P1 Pz P2 P3 P3 P4 P4 P5 P6 P5 P6 P7 P8 P7 P8 P9 P10 P9 P10 POz POz PO3 CMS DRL PO4 PO3 CMS DRL PO4 PO7 PO8 PO7 PO8 O1 O2 O1 O2 Oz Oz Iz Iz 18/23