Towards Automated Recognition of Human Emotions Using EEG Haiyan Xu - - PowerPoint PPT Presentation

Towards Automated Recognition of Human Emotions Using EEG

Haiyan Xu

The Edward S. Rogers Sr. Department of Electrical and Computer Engineering, University

• f Toronto

July 12, 2012

Research Objective

To investigate whether EEG signal is feasible for affect detection analysis, especially using consumer grade EEG devices.

1/23

Outline

1 Overview of Affect Classification 2 Affect Detection System Overview

Preprocessing: Filtering Based on Instantaneuous Frequency Feature Analysis Channel Selection: Genetic Algorithm (GA)

3 Experimental Setup 4 Simulation Results 5 Summary of Research Contributions 6 Future Works

2/23

Overview of Affect Classification

Overview of Affect Classification

Affective Computing Emotion is an unique, personal expression that differs under social context, culture background and personal experience Affective is the raw neurophysiological expression of emotion Affect Sensitive Applications Human Machine Interface (HMI) Health and rehabilitation applications Multimedia content indexing and retrieving

3/23

Overview of Affect Classification

Research Motivations

Prior Assessment Methods Facial expression, Voice, Physiological signals: heart rate, GSR, .. Disadvantages Not available due to inability to express emotions (autism disorders) Not reliable due to noisy source (crowded place) Interference due to non-emotional factors Affect Computing and EEG EEG signal originates from the Central Nervous system Brain networks in the limbic system are associated with affect expression Works with inaccessible and non-coorperative cases (autism disorders) Less influenced by non-emotional factors

4/23

Overview of Affect Classification

Technical Challenges of Affect Detection Analysis Using EEG

General Challenges EEG signals are originated from a non-linear, non-stationary processes Most signal processing systems are linear, use predefined bases, and assume stationary signals Lack of understanding on the dynamics of brain networks and correlation with emotional states Challenges Faced by Using Consumer-grade EEG Devices Much fewer number of electrodes available and conventional spatial analysis is not applicable Optimal sensor configuration for such applicaiton is not studied Constraints on computational complexity

5/23

Overview of Affect Classification

Research Contributions

Main Contributions

Novel signal representation for EEG based on Instantaneuous Frequency Automatic determination of optimal sensor configuration for affect application that can be generalized easily to other applications

Additional Contributions

First person to design and implement a compelete EEG signal processing system for affect detection at our university Investigated the implications on how key parameters. (e.g., sampling rate) affects system performance

6/23

Affect Detection System Overview

System Overview

Test EEG Signals

7/23

Affect Detection System Overview Preprocessing: Filtering Based on Instantaneuous Frequency

Preprocessing: Novel Filtering Based on Instantaneuous Frequency

Preprocessing Input signals Feature Analysis Classifiers Output Class labels

Purposes: to reduce noises, artefact and other external interferences Fourier and Wavelet Based Methods use a priori basis Multivariate Empirical Mode Decomposition (MEMD) is Instantaneuous Frequency based

To obtain meaningful IF, Hibert Transform requires signal to be monocomponent, zero-mean locally [1] Data-driven, suitable for non-stationary signals from non-linear processes Decompose original signal into multiple time-varying frequency content, Intrinsic Mode Functions (IMFs)

[1] L. Cohen. Time-Frequency Analysis, Prentice Hall, Englewood Cliffs (1995).

8/23

Affect Detection System Overview Preprocessing: Filtering Based on Instantaneuous Frequency

Preprocessing Cont.: MEMD as Filter bank

MEMD is useful as time-varying filtering technique (preprocessing), particularly for multicomponent signals. Frequencies of interest

Alpha wave 8 − 12Hz Beta wave 13 − 30Hz

Reconstruct EEG signal using sum of IMFS

1 2

10

−8

10

−6

10

−4

10

−2

PSD in Log scale IMFs obtained using Multivariate EMD

5 10 15 20 0.2 0.4 0.6 5 10 15 20 0.2 0.4 0.6 5 10 15 20 0.2 0.4 0.6 5 10 15 20 1 2 5 10 15 20 0.2 0.4 5 10 15 20 0.2 0.4 0.6 5 10 15 20 0.2 0.4 5 10 15 20 0.2 0.4 0.6 5 10 15 20 0.2 0.4 0.6 5 10 15 20 0.1 0.2 0.3 5 10 15 20 0.2 0.4 Time

166.15 107.17 64.12 25.05 14.93 8.67 6.95 5.51 4.81 4.52 8.09

9/23

Affect Detection System Overview Feature Analysis

Feature Analysis

Preprocessing Input signals Feature Analysis Classifiers Output Class labels

Purpose Dimension and computational complexity reduction Increase class separability Feature Analysis Algorithms Used Oscillation pattern variation in the time domain

Six statistical features: e.g., Mean, Std, Skewness, Kurtosis.. Higher Order Crossings (HOC): zero crossing counts with iterative filtering process

Event-related energy variation

Spectral domain: narrow band energy (1Hz resolution, 8 − 30Hz range) Time-spectral: wavelet-based energy and entropy analysis

10/23

Affect Detection System Overview Channel Selection: Genetic Algorithm (GA)

Channel Selection: Genetic Algorithm (GA)

For 54 channels, there are 254 combinations, impossible for exhaustive search Genetic Algoritm is a non-ranking global optimization method

Binary string representation for channel information Correct classification rate was used as fitness function Channels were selectd on results from 10 runs of GA

Application of GA

Initial Population EEG Features Continue Evolution? Optimal Features Fitness Calculation Crossover Mutation Updated Population

No Yes

11/23

Affect Detection System Overview Channel Selection: Genetic Algorithm (GA)

Classifiers

Preprocessing Input signals Feature Analysis Classifiers Output Class labels

Purpose: Determine optimal decision boundaries between classes Classifiers Two simple classifiers were used to reduce computational complexity

Linear Discriminate Analysis (LDA): optimal linear boundary between classes k Nearest Neightbors (kNN): majority voting, Euclidean distance was used as the distance metric

10 fold cross validation process was used to obtain the averaged simulation results shown in later sections

A portion of the training samples are used for training, and the remaining, unseen samples used for testing purposes

12/23

Experimental Setup

Experimental Setup: Database Description

Emobraine-eNTERFACE06 Validated public dataset, ideal for results comparison 5 subjects, aged 22-38 , 3 classes of affects, stimuli: IAPS images Biosemi Active II: 64 channels EEG headset 3 sessions and 30 trials each session

Valence

Negatively Excited Positively Excited Calm

Arousal

13/23

Experimental Setup

Experimental Setup: Experiment Description

setup Three experiments were conducted: with and without channel reduction Two testing cases:

Suject Specific: Training and testing samples are from the same subject Cross Subject:Training and testing samples are from all subjects

Raw EEG Signal Multivariate EMD Genetic Algorithm Classification: kNN, LDA Feature Analysis Emotions Referenced Emotive EEG Reconstruction

14/23

Experimental Setup

Experimental Setup: Emotive Epoch v.s Biosemi Active II

Biosemi Active II is a medical grade EEG headset Emotive Epoch is a popular consumer grade EEG headset Device Biosemi Active 2 Emotive EPOC SDK Data Format EDF MAT Resolution 24 bits ADC 16 bits (14 bits effective) Sampling Rate 1024Hz 128 SPS (2018 Hz internal) Channels 64 14 Channels in common AF3, F7, F3, FC5, FC6, F4, F8, AF4

15/23

Simulation Results

Simulation Results Using all Channels

Cross-Subject Emotion Recognition Classifier Statistical Narrow-bands Power HOC Wavelet 5NN 81.39 82.62 90.77 77.44 LDA 59.18 63.49 79.64 55.90 This results show that EEG signal is feasible for affect analysis even with simple classfiers kNN classifier provides better results which implies the features are not linearly seperable HOC is most representative for affect analysis

16/23

Simulation Results

Simulation Results with Channel Reduction Referecend to Emotive

Table: Cross-subject Recognition rate using only 8 electrodes

Classifier Statistical Narrow-bands Power HOC Wavelet 5NN 68.15 78.15 89.64 58.87 LDA 38.77 39.90 43.13 37.74 Currently in market consumer grade EEG headset is feasible for affect detection applications HOC and kNN combination provides the best performance

17/23

Simulation Results

Channels selected using GA

Given the cost and usability, consumer grade EEG headsets typically have less than 14 channels. Selected 8 channels

FPz FP2 AF4 AF8 AFz AF3 FP1 AF7 Fz F2 F4 F8 F6 F1 F3 F5 F7 FCz FC2 FC4 FC6 FC8 FC1 FC3 FC5 FC7 Cz C2 C4 C6 T8 T7 CPz CP2 CP4 CP6 TP8 TP7 C5 C3 C1 CP5 CP3 CP1 Pz P2 P4 P6 P8 P10 P1 P3 P5 P7 P9 POz PO3 PO7 PO4 PO8 Oz O2 O1 CMS DRL Iz

Selected 14 channels

FPz FP2 AF4 AF8 AFz AF3 FP1 AF7 Fz F2 F4 F8 F6 F1 F3 F5 F7 FCz FC2 FC4 FC6 FC8 FC1 FC3 FC5 FC7 Cz C2 C4 C6 T8 T7 CPz CP2 CP4 CP6 TP8 TP7 C5 C3 C1 CP5 CP3 CP1 Pz P2 P4 P6 P8 P10 P1 P3 P5 P7 P9 POz PO3 PO7 PO4 PO8 Oz O2 O1 CMS DRL Iz

18/23

Simulation Results

Simulation Results Summary

Channels 5NN LDA All 54 channels 90.77 79.64 8 - Referenced to commercial device 89.64 43.13 Channel selected using GA 18 89.23 62.05 10 88.77 56.31 6 89.79 53.21 The experimental results indicate the following: Recordings from higher number of channels give more classification accuracy Headsets with much smaller number of electrodes are feasible for affect detection analysis, however the choice of classifier is very important Linear classifier provides better classification accuracy on EEG sensor configuration selected through GA

19/23

Simulation Results

Recognition Rates Compared to the State of The Art Results

Studies Database Preprocessing Method Features Maximum Correct Classificaiton Rate Khalili et al. [1] eNTERFACE06, IASP, 3 emotions 4−45Hz bandpass filter Statistical 40%( LDA), 51% (kNN)

Comparing results from Khalili study, our approach is able to increase the performance from 51% to 81.39% through the use of an more effective preprocessing algorithm, and a maximum recognition rate of 90.77% Limitations: lack of international standard database for comparing results

[1] Z. Khalili and M. Moradi, Emotion detection using brain and peripheral signals,” in Biomedical Engineering Conference, 2008. CIBEC 2008. Cairo International, pp. 1-4,

• dec. 2008.

20/23

Simulation Results

Effects of Sampling Rate on Correct Recognition Rate

Sampling rate vs. correct recognition rate using all electrodes and LDA, kNN classiffier for all four feature types

256 512 1024 10 20 30 40 50 60 70 80 90 100 Sampling Frequency Correct Recognition Rate Sampling Frequency vs. Correct Recognition Rate (all electrodes, LDA) statistical narrow−band HOC wavelet−based 256 512 1024 10 20 30 40 50 60 70 80 90 100 Correct Recognition Rate vs. Sampling Rate (all electrodes, kNN) Sampling Rate Correct Recognition Rate statistical narrow−band hoc wavelet−based

Down sampling degrades the performance of linear classifier more than kNN Performance using HOC features degrades significantly when down sampled from 512Hz to 256 Hz

21/23

Summary of Research Contributions

Summary of Research Contributions

Applied a novel signal processing algorithm, Multivariate Empirical Mode Decomposition (MEMD) as a time-varying filtering technique Defined emotion specific channels using Genetic Algorithm (GA), which will be useful for future headset design Implemented a framework for Affect detection using EEG signals, a EEG signal processing system was designed and 4 feature extraction algorithms along with 2 classifiers were implemented for affect detection Proposed solutions for various practical issues, such as emotion elicitation, sampling rate implications, artefact reduction

22/23

Future Works

Future Works

To generalize my research findings

Proposed system should be tested with a larger dataset under natural conditions Optimal fusion techniques at the feature and decision level should be included for better performance

To broaden the application scenarios

Multimodality analysis should be studied, e.g., using face images for data labeling real-time testing Real time processing concerns should be addressed

23/23

Future Works

Publications

1 Haiyan Xu, Konstantinos N. Plataniotis, Affect Recognition Using

EEG Signal, 2012 IEEE International Workshop on Multimedia Signal Processing, Banff (Canada), 2012 (accepted).

2 Haiyan Xu, Mohammad Shahin Mahanta, Chris Aimone,

Konstantinos N. Plataniotis, ”A Real Time Portable System for Classification of Meditation States Using EEG Signals” 2012 IEEE International Workshop on Multimedia Signal Processing, Banff (Canada), 2012 (under review).

3 Haiyan Xu, Gaurav Jain, Konstantinos N. Plataniotis, ”Automated

Affect Detection using Facial Images and EEG Signals”, The IEEE Transactions on Affective Computing (TAC).(To be submitted)

24/23

Genetic Algorithm

Crossover Point Parent #1 1 0 0 1 0 1 0 0 1 1 Parent #2 0 1 0 1 1 1 1 0 1 1 Child #1 0 1 0 1 0 1 0 0 1 1 Child #2 0 1 0 1 1 1 1 0 1 1 Parent #1 1 0 0 1 0 1 0 0 1 1 Child #1 1 0 0 1 1 1 0 0 1 1 Mutation Point Mutation Process Crossover Process Initial Population [1001010101] ……… [010100111]

24/23

Experimental Setup Flowchart

Initial Signal MEMD-IMFs Extrac tion IMFs Selection Signa l Reconstruct ion Statistical Na rrow - ba nd w avelet HOC MEMD Filtering Classification Affe ct Re cognition Rate Feature Analysis

1 1 1

LDA, kNN Fe ature Ve ctor

• Pos. exited, Neg. ex cited, Neutra l

24/23

Multivariate EMD Algorithm

24/23

Multivariate EMD Algorithm

Algorithm 2. Multivariate extension of EMD Algorithm (MEMD)

• 1. Choose a suitable pointset for sampling on an (n − 1) sphere.
• 2. Calculate a projection, denoted by Pθk(t)T

t=1, of the input signal {v(t)}T t=1 along the direction

vector xθk, for all k (the whole set of direction vectors), giving pθk(t)K

k=1 as the set of projections.

• 3. Find the time instants
• tθk

i

• corresponding to the maxima of the set of projected signals

pθk(t)K

k=1.

• 4. Interpolate
• tθk

i , v(tθk i )

• to obtain multivariate envelope curves eθk(t)K

k=1.

• 5. For a set of K direction vectors, the mean m(t) of the envelope curves is calculated as

m(t) = 1 K

K

• k=1

eθk(t) (1)

• 6. Extract the ’detail’ d(t) using d(t) = x(t) − m(t). If the ’detail’ d(t) fulfills the stoppage

criterion for a multivariate IMF, apply the above procedure to x(t) − d(t), otherwise apply it to d(t).

23/23

Feature Anlalysis Comparison

Strength of each feature analysis algorithm statistical: HOC:intuitive, analyzing the up and down movement of the EEG signal.

Process is iterative, the higher the order, the more computational expensive Filter the time domain signal (backward difference operator). count the number of zero crossings A measure of EEG oscillation; the more pronounced the oscillation, the higher the expected number of zero-crossings is and vice versa. Zerocrossings and spectrum: number of zero crossings indicates the dominate frequency

Wavelet-based Entropy analysis: the entropy feature is basically a non-linear in nature and captures the nonlinearity of the EEG signals

• ver different emotions than other statistical features.

23/23

Statistical Features

The statistical features used to form the proposed FVs are defined as (Xi, i = 1 · · · N is the raw N-sample EEG signal) given in the following.

1 The mean of the raw signal

µx = 1 T

T

• t=1

X(t) = X(t) (2)

2 The standard deviation of the raw signal

σx =

• 1

T

T

• t=1

(X(t) − µx)2 (3)

3 The mean of the absolute values of the first differences of the raw signal

δx = 1 T − 1

T−1

• t=1

|X(t + 1) − X(t)| (4)

4 The mean of the absolute values of the first differences of the standardized signal

δx = 1 T − 1

T−1

• t=1
• X(t + 1) − X(t)
• = δx

σx (5)

5 The mean of the absolute values of the second differences of the raw signal

γx = 1 T − 2

T−2

• t=1

|X(t + 2) − X(t)| (6)

6 The mean of the absolute values of the second differences of the standardized signal

γx = 1 T − 2

T−2

• t=1
• X(t + 2) − X(t)
• = γx

δx (7) 23/23

Higher Order Crossings

Higher Order Crossing Features Let X1, X2, ..., XN be a zero-mean stationary time series, the zero-crossing count in discrete time is defined as the number of symbol changes in the corresponding clipped binary time series Zt = 1, ifXt ≥ 0 0, ifXt < 0 (8) The number of zero-crossings, denoted by D, is defined in terms of Zt D =

N

• t=2

[Zt − Zt−1]2, 0 ≤ D ≤ N − 1 (9)

23/23

Wavelet-based Features

Daubechies fourth-order orthonormal bases (db4) was employed to calculate the wavelet coefficients at the lth scale, CX(l, n), that correspond to the alpha band (812Hz)and Beta band (13 − 30Hz) were used to estimate the wavelet energy and wavelet entropy, given by ENGl =

2S−l−1

• n=1

|CX(l, n)|2 , N = 2S, 1 < l < S. (10) ENTl = −

2S−l−1

• n=1

|CX(l, n)|2 log(|CX(l, n)|2), N = 2S, 1 < l < S. (11) The parameters of 10 and 11 were used as a feature vector fw (i.e., f j

w,i = [ENGl, ENTl]).

23/23