Support Vector Machine Classification and Psychophysiological - - PowerPoint PPT Presentation

Support Vector Machine Classification and Psychophysiological Evaluation of Mental Workload and Engagement of Intuition- and Analysis-Inducing Tasks

Presenter: Joseph Nuamah

Department of Industrial and Systems Engineering Advisor: Younho Seong

March 3, 2017

Presenter: Joseph Nuamah March 3, 2017 1 / 34

1

Outline

2

Introduction

3

EEG

4

Aim

5

Hypotheses

6

Methodology

7

Results

8

Discussion and Conclusion

Presenter: Joseph Nuamah March 3, 2017 2 / 34

Introduction: Human Factors(HF) Issues in Autonomous Vehicles(AV)

Appropriate Levels of Automation Human operator in-the-loop for failure mode operations Fail-safe mechanisms into AV Usefulness of UV interfaces

Presenter: Joseph Nuamah March 3, 2017 3 / 34

Introduction: Problems with Improper Design of Automation

Increased monitoring demands Cognitive overload Mis-calibration of trust in automation Inability to resume manual control Loss of situation awareness Degraded manual skills due to lack of practice

Presenter: Joseph Nuamah March 3, 2017 4 / 34

Introduction: System Monitoring

Increased UV operator’s role of monitoring and supervising automation Increased monitoring requirements add to cognitive load Vigilance - Failure detection worse under passive monitoring than under active control (Wickens & Kessel, 1980) UVs likely to contain more displays and instruments:

Camera-fed screens Screens following up flight plan Instruments indicating intact communication requirements between Air Traffic Control operator and UV operator

Presenter: Joseph Nuamah March 3, 2017 5 / 34

Introduction: Human Supervisory Control

Human interactions with environment mediated by technological interfaces Supervisory control - operators oversee automated process, and continuously determine basis need to re-enter control loop Ongoing assessment based on comparison of actual and intended system performance UV operator in supervisory role

requires information about target parameters decides how automation should proceed to achieve targets communicate appropriate instructions monitor process to ensure commands are understood and executed

Components of Human Supervisory Control System:

Human operator Interface Automation

Presenter: Joseph Nuamah March 3, 2017 6 / 34

Introduction: Decision Making

Design of interface requires understanding of

judgment characteristics that they are to support effect of design on operator judgment

Task characteristics play important role in determining cognitive mode likely to be used Higher correspondence between task characteristics and cognitive characteristics correlate with operator’s judgment accuracy Dual process theories of intuition and analysis used to explicate human cognitive system

Presenter: Joseph Nuamah March 3, 2017 7 / 34

Introduction: Decision Making (cont’d)

Table: Attributes of Intuition and Analysis Cognition (adapted from Evans & Stanovich, 2013)

Intuition Analysis Does not require working memory Requires working memory Fast Slow High capacity Capacity limited Parallel Serial Nonconscious Conscious Automatic Controlled Holistic Analytic Relatively undemanding of cognitive capacity Demanding of cognitive capacity Experience-based decision making Consequential decision making

Table: Inducement of Intuition and Analysis by Task Conditions (adapted from Hammond et al., 1987)

Task Characteristic Intuition-Inducing State of Task Characteristic Analysis-Inducing State of Task Characteristic Number of cues Large (>5) Small Measurement of cues Perceptual measurement Objective, reliable measurement Distribution of cue values Continuous highly variable distribution Unknown distribution; cues are dichotomous; values are discrete Redundancy among cues High redundancy Low redundancy Decomposition of task Low High Availability of organizing principle Unavailable Available Degree of certainty in task Low certainty High certainty Time period Brief Long

Presenter: Joseph Nuamah March 3, 2017 8 / 34

Introduction: Decision Making (cont’d)

Behavioral and subjective traditionally measures used to measure judgment and decision making performance

May not produce much information on the operator’s state

Physiological measurements may be used

Continuously available and collection does not interfere with operator’s task performance Measures range from blood flow or neural activity in brain to heart rate variability and eye movements Methods include:

Electroencephalography (EEG), Functional Near-Infrared Spectroscopy (FNIRs), etc Skin conductance, cardiovascular responses, muscle activity, pupil diameter, eye blinks,eye movements, etc

Presenter: Joseph Nuamah March 3, 2017 9 / 34

EEG

EEG signals represent summed postsynaptic potentials of neurons firing a rate of milliseconds Graph of time varying voltage difference between active electrode attached to scalp and reference electrode

Table: Lobes and corresponding electrode label

Lobe Electrode Frontal F Temporal T Central C Parietal P Occipital O

Presenter: Joseph Nuamah March 3, 2017 10 / 34

EEG (cont’d)

Figure: Time domain EEG signal Figure: Main EEG waves Figure: Waveform showing several ERP components Table: Lobes and corresponding electrode label

Frequency Band (Hz) Associated Tasks & Behaviors Delta (0.1-3) Lethargic, not moving, not attentive Theta (4-8) Creative, intuitive, distracted, unfocused Alpha (8-12) Meditation, no action Beta (12-30) Mental activity Gamma (>30) High-level information

Presenter: Joseph Nuamah March 3, 2017 11 / 34

EEG Indexes

Spectral composition of EEG changes in response to changes in task difficulty or level of alertness Alpha, theta, beta all related to task engagement Task Engagement Index (TEI) is given by beta power alpha power + theta power Task Load Index (TLI) is given by frontal midline theta parietal alpha

Presenter: Joseph Nuamah March 3, 2017 12 / 34

Aim

Employ TLI to provide insight into cognitive load Employ TEI to provide insight into engagement Employ SVM to discriminate EEG signals recorded during execution

• f intuition-inducing and analysis-inducing tasks

Employ objective measures (reaction time and percent correct), and subjective measure (NASA-Task Load Index) to validate objective EEG measures (TLI and TEI)

Presenter: Joseph Nuamah March 3, 2017 13 / 34

Hypotheses

Engagement required for analysis-inducing tasks would be different from that required for intuition-inducing tasks Mental effort required for analysis-inducing tasks would be different from that required for intuition-inducing tasks

Presenter: Joseph Nuamah March 3, 2017 14 / 34

Methodology: Materials and Method

Participants:

Six participants (1 female, 5 males) Ages between 18 and 35 years All right-handed Normal vision No history of neuropsychiatric disorders

Equipment:

g.HIamp-256 channel biosignal amplifier g.GAMMAcap Electrode Type: AgCl Active electrode connector box comes with 64 channels g.Recorder used to record the EEG signals Presentation Software for stimuli delivery

Presenter: Joseph Nuamah March 3, 2017 15 / 34

Methodology: Stimuli

Baseline:

Participants were instructed to relax and fixate on a blank screen for 60 s

Intuition-inducing task:

For each stimulus, two objects presented: fixation on left, and flashing face on right Participants were instructed to press LEFT mouse button if they thought face on RIGHT was a happy face, or press RIGHT mouse button if they thought face on RIGHT was face of someone who was afraid Stimuli taken from FACE database established by Ebner et al. (2010) Stimulus duration was approx. 6 s Inter Stimulus Time (ISI) was approx. 2 s Two blocks, each containing 30 trials

Presenter: Joseph Nuamah March 3, 2017 16 / 34

Methodology: Stimuli (cont’d)

Analysis-inducing task:

For each stimulus, two multiplications were presented Participants instructed to determine which

• f two

multiplications was larger Participants instructed to press LEFT mouse button if they thought multiplication on LEFT was larger

• r press RIGHT

mouse button if they thought

Presenter: Joseph Nuamah March 3, 2017 17 / 34

Methodology: NASA-TLX

Subjective workload assessment tool Overall workload score based on weighted average of ratings on six subscales:

Mental Demand Physical Demand Temporal Demand Performance Effort Frustration

Presenter: Joseph Nuamah March 3, 2017 18 / 34

Methodology: Procedure

Sign informed consent and complete demographic questionnaire Fit g.GAMMAcap on scalp: 20 electrodes used, ear lobe as reference Calibrate electrode impedance Present experimental conditions Record Response time (RT) Complete NASA-TLX questionnaire

Presenter: Joseph Nuamah March 3, 2017 19 / 34

Methodology: Signal Preprocesssing

Raw EEG signals recorded at sampling rate of 256 Hz with Butterworth filter (0.01Hz high pass - 100Hz low pass) Notch filter with 60 Hz cutoff frequency to remove line noise Data re-referenced to average EEG epochs time-locked to stimulus presentation Data with amplitudes outside of range of

• 50 µV to +50 µV rejected

Independent component analysis (ICA) correct EEG data contaminated by signals of non-neural origin

SASICA software used to reject artifact independent components before EEG data analysis

Presenter: Joseph Nuamah March 3, 2017 20 / 34

Methodology: Signal Preprocesssing

Presenter: Joseph Nuamah March 3, 2017 21 / 34

Methodology: Signal Preprocesssing

Data submitted to ICA are EEG channel recordings arranged in a matrix of n channels (rows) by t time points (columns) data values ICA performs blind source separation of X with assumption that source time courses (U) are independent ICA finds component umixing matrix (W) that when multiplied by original data (X), yields matrix (U) of IC time courses U = WX X = W −1U W −1 is n by n component mixing matrix whose columns contain relative weights with which component projects to each scalp channel Portion of original data X forms the ith IC(Xi) is (outer) product of two vectors: ith column of W and ith row of U

Presenter: Joseph Nuamah March 3, 2017 22 / 34

Methodology: Signal Processing

Parametric approach - assume a model for EEG signals and estimate parameters of model

AutoRegressive-Moving-Average (ARMA)

Nonparametric approach - do not require a model of signal

Fast Fourier Transform (FFT)

Assumes signals are stationary

Short-Time Fourier Transform (STFT)

Allows for depiction of nonstationary signals as stationary ones by use

• f window function

Limitation - window too narrow implies poor frequency resolution, window too wide implies imprecise time localization

Wavelet Transform (WT)

Presenter: Joseph Nuamah March 3, 2017 23 / 34

Methodology: Signal Processing

Wavelet - waveform of effectively limited duration, has average value of zero Wavelet analysis-breaking up of a signal into shifted and scaled versions of

• riginal (or mother) wavelet

Two types: Continuous Wavelet Transforms (CWT) and Discrete Wavelet Transform (DWT) CWT defined as sum over all time of a signal x(t) multiplied by scaled, shifted versions of the wavelet function ψ C(scale, location) =

∞

−∞

x(t)ψ(scale, location)dt Two approaches differ in how they discretize scale parameter DWT choose scales and positions based on powers of 2, CWT uses exponential scales with a base smaller than 2

Presenter: Joseph Nuamah March 3, 2017 24 / 34

Methodology: Signal Processing

CWT expressed as C(a, b) = 1 √a

∞

−∞

x(t)ψ∗

t − b

a

• dt

where a, b ∈ R, a = 0 a is the scale parameter, b is the location parameter, ψ(t) is the mother wavelet

1 √a used to normalize energy such that it stays as same level for different

values of a and b Choice of mother wavelet depends on kind of features to be extracted CWT scales and frequency are related by Fa = Fc a.∆ where a is a scale ∆ is the sampling period (1/256) Fa is the pseudo-frequency corresponding to the scale a, in Hz center frequency of a Morlet wavelet is 0.8125

Presenter: Joseph Nuamah March 3, 2017 25 / 34

Methodology: Signal Processing

Coefficients of CWT for scale of 1.5 to 80 with a scale step of 0.1 Computed geometric mean energy of wavelet coefficients of each scale using Ej = 1 n

n

• i=1

|xi|2, j = 1, ..., 786 where xi’s are computed coefficients of signal at each scale n is window size 786 is total number of scales

Presenter: Joseph Nuamah March 3, 2017 26 / 34

Methodology: Signal Processing

Table: CWT SCALE RANGE AND CORRESPONDING PSEUDO-FREQUENCY AND EEG BAND

Scale Range PseudoFrequency (Hz) EEG Band 1.5–3.5 30–60 Gamma 3.5–8.5 13–30 Beta 8.5–13.0 8–13 Alpha 13.0–26.0 4–8 Theta 26.0–80.0 1–4 Delta Corresponding scale ranges for delta, theta, alpha, beta, and gamma bands used in present study are shown in Table Mean value of geometric means at each scale gives their corresponding absolute energy, Eband For an EEG signal, total energy Et across all five bands is given by Et = Edelta + Etheta + Ealpha + Ebeta + Egamma Relative wavelet energy is computed as ρband = Eband Et Computed five relative wavelet energy features -resulted in 100 features (5 features x 20 channels) per trial per task

Presenter: Joseph Nuamah March 3, 2017 27 / 34

Methodology: SVM Classification

Performed all classifications offline Designed six separate SVMs to classify two cognitive tasks for each participant SVM was tested with Radial Bias Function (RBF) kernels 10-fold cross-validation to find the best C and γ

Presenter: Joseph Nuamah March 3, 2017 28 / 34

Methodology: Block Diagram

Presenter: Joseph Nuamah March 3, 2017 29 / 34

Results

Multivariate Analysis of Variance (MANOVA) technique used Dependent variables:

mean reaction time (RT), mean NASA-TLX, mean percent correct, mean TLI across all channels for all participants, mean TEI across all channels for all participant

Results there was a significant overall treatment effect on NASA-TLX, Percent Correct and RT when analyzed simultaneously

Wilk’s Λ = 0.02269714, F(3,8) = 114.82, p- value < 0.0001

Indicates at least one effect of three task types on NASA TLX, Percent Correct and RT is different from others

Univariate ANOVA to study effects of the two task levels on each dependent variable

NASA TLX-enough evidence to conclude that the two different treatment levels did not have the same treatment effect on TLI, F(1,10)= 388.31, p- value < 0.0001 Percent Correct-enough evidence to conclude that the two different treatment levels did not have same treatment effect on TLI, F (1,10) = 27.86, p-value < 0.0004 RT-enough evidence to conclude that the two different treatment levels did not have same treatment effect on TLI, F(1,10) = 28.32, p-value < 0.0003

Presenter: Joseph Nuamah March 3, 2017 30 / 34

Results (cont’d)

Table: DESCRIPTIVE STATISTICS FOR EFFECT OF TASK TYPE ON NASA TLX, PERCENT CORRECT, AND RT

Task Type Dependent Variable Mean NASA-TLX 60.42 ± 4.16 Analysis-Inducing Percent Correct 91.61 ± 3.07 RT 22681.67 ± 7594.11 NASA TLX 21.39 ± 2.4914 Intuition-Inducing Percent Correct 98.60 ± 1.04 RT 6155.17 ± 438.09

Presenter: Joseph Nuamah March 3, 2017 31 / 34

Results (cont’d)

Table: DESCRIPTIVE STATISTICS FOR EFFECT OF TASK TYPE ON TLI AND TEI

Task Type Dependent Variable Mean Analysis-Inducing TLI 5.23 ± 1.73 TEI 0.47 ± 0.58 Intuition-Inducing TLI 2.42 ± 0.38 TEI 0.58 ± 0.75 Baseline TLI 3.80 ± 1.61 TEI 0.49 ± 0.58

Presenter: Joseph Nuamah March 3, 2017 32 / 34

Results (cont’d)

Presenter: Joseph Nuamah March 3, 2017 33 / 34

Discussion and Conclusion

Results from statistical analysis were consistent with hypothesis TEI for each participant across all 20 EEG channels revealed that TEI for baseline was generally higher than intuition-inducing and analysis-inducing tasks for all participants Higher value of TEI generated by intuition-inducing tasks in part as result of flashing nature of stimuli presented Negative correlation found between TLI and TEI - TLI measures mental workload, while TEI measures alertness and engagement Average classification accuracy of 86.36 % Analysis-inducing tasks appear to impose higher cognitive loads than intuition-inducing tasks SVM may be employed to classify EEG signals recorded during execution of intuition- and analysis-inducing tasks, and by extension high and low cognitive loads

Presenter: Joseph Nuamah March 3, 2017 34 / 34