Development & evaluation of human-centered technology basic - - PDF document

1/13/17 1

Brain-Computer Interfaces (BCIs):

Research in Communication, Control and Human Cognition

January 13, 2017

Chang S. Nam, Ph.D., CHFP

Associate Professor, Director Brain-Computer Interface (BCI) Lab Edward P. Fitts Department of Industrial & Systems Engineering North Carolina State university

Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary

Agenda

1 3 2 4

Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU

Human-centered computing research @ NCSU Motivations for interfacing human brain to computer

• Unlocking the locked-in
• Decoding people’s thoughts
• Factoring in human factors

BCI research @ NCSU

• P300-based BCIs for communication
• Control by BCIs

ü SSVEP-based BCI to support collaborative work ü SMR BCI-controlled FES for hand-wrist motor function ü SSSEP-based hybrid BCI for behaviorally non-responsive patients

• Neural correlates of human cognition

Summary

• Require active involvement of HF/E

Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary

1/13/17 2

Development & evaluation of human-centered technology

…basic & applied research on human factors/ergonomics (HF/E) issues associated with design, development, and evaluation of human-centered technology

Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary

Human-Automation Interaction Human-Centered Computing Model-based Remote Healthcare Intelligent Adaptive User Interface Neuroergonomics Brain-Computer Interface & Affective Computing

Suffering from severe motor disabilities…

q People who are totally paralyzed, or “locked-in”

Ø Nearly 2 million Amyotrophic lateral sclerosis (ALS), or Lou Gehrig's disease in USA, affecting 5 out of every 100K people worldwide

ü 1 in 50 people living with paralysis – approximately 6 million people

Ø Approx. 800K/yr people suffering from a new or recurrent stroke in USA, and 33 million worldwide

(Mozaffarian et al., 2015)

Ø Socio-economic impacts

ü Impossible to do routine tasks such as going up steps, getting out of a chair, or swallowing ü Reduced independence & quality of life

Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU Unlocking the locked-in Decoding people’s thoughts Factoring in human factors Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary

1/13/17 3

How it feels like to be paralyzed?

Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU

Try it! Loss of voluntary muscle control, but Cognitively intact !!!

Unlocking the locked-in Decoding people’s thoughts Factoring in human factors Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary

Brain-computer interface (BCI): General architecture

A non-muscular communication and control system that does not depend on the brain’s normal output pathways of peripheral nerves and muscles

Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU

Feedback Signal Acquisition Classification Signal Processing Brain Signal Extracting Features Spatial Filtering Temporal Filtering Classifier Building Classifier Features Feedback Training Task Online Task Offline Analysis

Unlocking the locked-in Decoding people’s thoughts Factoring in human factors Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary

1/13/17 4

Neural mechanisms of human cognition

Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU Unlocking the locked-in Decoding people’s thoughts Factoring in human factors Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary 8

Still long way to go: HF/E matters

Unlocking the locked-in Decoding people’s thoughts Factoring in human factors Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU

Lack of understanding of, or inattention to user’s interaction with BCI systems:

Effects of task, environmental & individual differences

1/13/17 5

[1] How does a P300-based BCI work?

q P300 ERP: Physical & Behavioral Properties

Ø A positive peak in voltage of an event-related potential (ERP) at a maximum of roughly 300 ms at Pz > Cz > Fz ü Oddball paradigm, where two stimuli are presented with different probabilities: frequent standard stimuli and infrequent task stimuli ü Related to the engagement of attention and the processing of novelty ü Fairly stable in locked in patients; No initial training

(Serby et al., 2005) Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU P300-based BCIs for communication Control by BCIs Neural correlates of human cognition Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary

Effects of task, environmental & individual differences

Ø To assess how background noise and screen size affect task performance, neural activity and cortical integration of users with and without severe motor disabilities Can they use BCIs in a mall food court, city park, or street café? Research Goal

Source: Li et al., (2011); Li & Nam (2011); Nam et al. (2009; 2010; 2012)

P300-based BCIs for communication Control by BCIs Neural correlates of human cognition Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU

q Participants

Ø 10 healthy subjects: 8M & 2F (M = 27.9 / 3.6 yrs) Ø 7 CP and 3 ALS patients with speech difficulties (M = 35.8 / 13.3 yrs)

q Data acquisition and preprocessing

Ø To spell 6, 10-character phrases, using 16 Chs. Ø Referenced/grounded to right/left mastoid with amplifier Ø Sampling rate of 256Hz; band-pass filtered 0.5 - 30Hz

1/13/17 6

q EEG Coherence

Ø Synchronization of oscillatory cortical activity between brain regions within a certain frequency band Ø An increase in coherence is thought to reflect reduced cortical differentiation and specialization, when two coherences are compared

q Cxy(ω): coherence value between signals x and y

§ Φxy(ω): value of cross-correlation power spectrum of signals x, y § Φxx(ω): value of auto-correlation power spectrum of signal x § Φyy(ω): value of auto-correlation power spectrum of signal y

Effects of task, environmental & individual differences

P300-based BCIs for communication Control by BCIs Neural correlates of human cognition Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU

Effects of intra-individual differences on neural activity

q Major findings

Ø Task and environmental factor effects on real-world applicability Ø Differences between people with (~75%) and without (~95%) severe motor disabilities Ø Variations among people with motor disabilities Performance ALS (Good) CP (Moderate) CP (Bad) r2 Topography P300 Pattern

Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU P300-based BCIs for communication Control by BCIs Neural correlates of human cognition Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary

1/13/17 7

Effects of inter-individual differences on coherence

Results Implications

q Severe motor disability affected functional cortical integration

Ø Significant difference in coherence between the two groups. Ø Participants with severe neuromuscular impairments, as compared with the able-bodied group, were obliged to recruit more cortical regions, reflecting a less efficient operating strategy for the task.

P300-based BCIs for communication Control by BCIs Neural correlates of human cognition Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU

q Sensory motor rhythm (SMR) at C3, Cz, & C4: An indicator of cortical activation/deactivation

Ø Reduction/Enhancement in 8-14Hz alpha band and 15-25Hz beta band

Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU

[2] How does a motor imagery-based BCI work?

P300-based BCIs for communication Control by BCIs Neural correlates of human cognition Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary Source: Jeon et al (2011); Nam et al (2011)

Spatial Dynamics Temporal Dynamics

1/13/17 8

SMR BCI-controlled FES for hand-wrist motor function

Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU P300-based BCIs for communication Control by BCIs Neural correlates of human cognition Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary

v Functional electrical stimulation (FES)

Ø A technique that causes a muscle to contract through the use of low level electrical currents to restore or improve its function Ø Common physical therapy in stroke rehabilitation (Miller et al., 2010)

Source: Choi et al (2016)

v Why interfacing FES to BCI?

Ø OT with FES showing better rehabilitation outcomes than OT alone, such as reduced spasticity, improved muscle strength, increased range of motion, etc. (Sabut et al., 2011; Jackson et al., 2006) Ø However, controlled by physical therapists would lack the patient’s intention and efforts. ü Neuroplasticity of the patients may not be promoted well, which is important to benefit rehabilitation.

Ø (Individualized) BCI-FES can better promote neuroplasticity by the user’s intention and effort Grasping vs. Opening

Signal acquisition & pre-processing

P300-based BCIs for communication Control by BCIs Neural correlates of human cognition Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU

v Recorded with 11 electrodes at 256 Hz sampling rate

Ø For MI analysis, 4 electrodes placed anterior and posterior to C3 and C4 Ø For SSVEP analysis, O1 and O2

v Signal preprocessing

Ø Band-pass filtered between 5 - 35 Hz Ø Notch-filtered at 60Hz Ø ICA for noise filtering Ø Surface Laplacian method around C3 and C4

v Participants

Ø 4 health subjects (3M; 1F; Avg. age 30.5±3.9) Ø Currently testing with 24 stroke patients (12 M; 12F) from local rehabilitation centers and NC Stroke Association

1/13/17 9

Feature extraction & classification

P300-based BCIs for communication Control by BCIs Neural correlates of human cognition Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU

v ERD/ERS (over C3 & C4) at α and β bands (Pfurtscheller & Neuper, 2006)

Ø 8 time epochs (0 to 4 seconds with 1, 2, and 4-second intervals ) Ø 9 frequency bands (5, 8, and 11 Hz, with ending frequency of either 20, 25, or 30 Hz) è 72 combinations relative to 2s reference

v Fisher’s Linear Discriminant Analysis (fLDA) used to build a classifier that activates the FES

Ø 10-fold cross-validation to compute the classification accuracy

Results: Classification accuracy of grasping vs. opening

P300-based BCIs for communication Control by BCIs Neural correlates of human cognition Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU

v High accuracy of Ref vs. MI (89.48%, 88.16%, 90.37%, and 93.93%) to detect the presence of SMR existence v All subjects showed higher classification results for SG vs. FO than FG vs. SO, except for subject 3. è The proposed hybrid BCI-FES system successfully control hand-wrist function in real-time.

è Confirmed use of user-specific features (frequency bands & time epochs) due to individual differences

S1 S4

1/13/17 10

[3] SSVEP BCI-driven prosthetic for rehabilitation

Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU P300-based BCIs for communication Control by BCIs Neural correlates of human cognition Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary

v What is steady-state visually evoked potential (SSVEP)?

Ø Steady-state visual evoked potentials evoked by a visual stimulus at certain frequency

(Niedermeyer & da Silva, 2005)

[3] SSVEP BCI-driven prosthetic for rehabilitation

Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU P300-based BCIs for communication Control by BCIs Neural correlates of human cognition Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary

1/13/17 11

Can BCIs support collaborative work?

v Research goals (Source: Li & Nam, 2016)

Ø Can people with severe motor disability perform a task jointly with other people (with or without severe motor disability) only through means of their brain activity?

ü How do these issues vary across users working under differing task conditions? ü What design considerations should we take into account for collaborative BCIs?

P300-based BCIs for communication Control by BCIs Neural correlates of human cognition Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU

v Accuracy: Overall 76%

Ø Significantly higher accuracy in Simultaneous Mode than Individual Mode & Sequential Mode (F2,24 = 36.31, p < 0.0001). ü Simultaneous (M = 0.868) > Individual (M = 0.684): F1,12 = 66.63, p < 0.0001 ü Simultaneous (M = 0.868) > Sequential (M = 0.728): F1,12 = 38.43, p < 0.0001

22

Results and discussion

v Spectral Power

Ø Significantly bigger spectral power in Simultaneous Mode and Sequential Mode than Individual Mode (F2,24 = 8.70, p = 0.0014). ü Simultaneous (M = 0.972) > Individual (M = 0.787): F1,12 = 8.15, p = 0.0087 ü Sequential (M = 1.049) > Individual (M = 0.787): F1,12 = 16.48, p = 0.0005

P300-based BCIs for communication Control by BCIs Neural correlates of human cognition Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU

1/13/17 12

v Effects of Collaboration Mode

Ø Significant effects on task performance, consistent with previous research (Wang &

Jung, 2011; Eckstein et al., 2012)

ü Simultaneous mode leads to significantly less completion time than individual mode. ü Significantly bigger power in Simultaneous and Sequential Mode than Individual Mode (STFT plot).

Ø The presence of teammate elevated individual performance (Aiello & Douthitt, 2001). Ø Simultaneous mode is more efficient because of the error cancellation property

• f team work (Baron et al., 2012; Brown, 2000).

Ø ALS patients could use a C-BCI as efficient as able-bodied people

Results and discussion

P300-based BCIs for communication Control by BCIs Neural correlates of human cognition Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU

Behavioral and neural correlations of two executive working memory functions between inhibition and updating

P300-based BCIs for communication Control by BCIs Neural correlates of human cognition Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU

Ø To investigate the hypothesis that the level of memory load could determine the impact of task interference ü To analyze information flow between anatomically-localized sources of brain activity during correct responses, and in particular error commission Research Goal

ACC

IC8: posterior rostral cingulate zone/caudal cingulate zone IC11: anterior rostral ACC IC38: SMA IC13: posterior parietal cortex

1/13/17 13

Correct Response Wrong Response

Transient theta (θ) information flow during error commission: temporal & spatial dynamics of brain activity

P300-based BCIs for communication Control by BCIs Neural correlates of human cognition Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU

Start (-525 ms) Middle (49 ms) End (623 ms) Correct Response Wrong Response

Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU

1/13/17 14

v Some patients recovered even many years after their trauma!!!

Could we establish communication?

Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU http://www.cbsnews.com/news/brain-heal-thyself/

The American Terry Wallis, who suffered a car accident in 1984, recovered from the minimally conscious state in 2003, 19 years later, he started talking.

https://www.youtube.com/watch?v=lQtmaBzFlh0

Man wakes up after 12 years in ‘vegetative state’: reveals, ‘I was aware of everything’

Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary Hybrid BCI for behaviorally non-responsive patients A model-based remote healthcare system

Funding Source: NSF BRAIN Initiative (IIS-1421948)

v Some of patients misdiagnosed (20-40%) as behaviorally non-responsive (e.g., VS, MCS) are conscious

Ø A patient apparently VS was able to modulate the BOLD response using MI (Owen et al., 2006) Ø 3 of 16 followed command and performed the MI task (Cruse et al., 2011) Ø 4 out of 24 patients (17%) in VS were consciously aware; answer yes and no questions (Monti et al., 2012)

Motivation 1: Misdiagnosis in disorders of consciousness

Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU Hybrid BCI for behaviorally non-responsive patients A model-based remote healthcare system Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary

1/13/17 15

q 20-30% of BCI users do not reach the level needed for control - 70% binary classification (Kübler et al., 2004; Vidaurre et al., 2011). q “Hybrid” BCIs can improve communication by utilizing advantages of each conventional BCI system and overcoming its disadvantages

Motivation 2: Visual-oriented BCIs and BCI illiteracy

Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU Hybrid BCI for behaviorally non-responsive patients A model-based remote healthcare system Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary

v Steady-State Somatosentory Evoked Potentials (SSSEPs)

Ø Sinusoidal electrophysiological brain response elicited from mechanical vibrotactile stimulation, modulated by selective spatial attention

ü Help those who have visual impairments and are unable to use visual BCIs ü Task options can be easily expanded spatially by stimulating body parts.

Development: Hybrid BCIs based on SSSEPs and P300s

Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU Hybrid BCI for behaviorally non-responsive patients A model-based remote healthcare system Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary

1/13/17 16

Selective spatial attention to vibrotactile stimuli

Prelim results: Feasibility of tactile-based hybrid BCI

Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU Source: Choi et al., 2015 Hybrid BCI for behaviorally non-responsive patients A model-based remote healthcare system Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary

Wearable sensor-based remote monitoring system:

Lack of (1) brain bio-metrics and (2) clinically relevant information

Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU Hybrid BCI for behaviorally non-responsive patients A model-based remote healthcare system

Funding Source: Rehabilitation Engineering Center; NIOSH

Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary

1/13/17 17

S L T

A model-based remote healthcare system

Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU

Biomechanics Models temporal and spatial dynamic models

A wireless sensor-based system detecting and transmitting neural and motion data from human brain and body, respectively, and extracting clinically relevant information based on models

Hybrid BCI for behaviorally non-responsive patients A model-based remote healthcare system Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary

1/13/17 18

Summary: Requiring active involvement of HF/E

q Brain-computer interfaces for communication vs. control and medical vs. non- medical applications: P300/SMR/SSVEP/SSSEP/hybrid BCIs

Ø Unlocking the locked-in Ø Decoding people’s thoughts

q Lack of understanding of, or inattention to HFE issues in terms of effects of task, environmental & individual differences q However, still require active involvement of HF/E

Ø “… Interest and work in BCI research by somebody with a HF background is still quite rare….. the field is now beginning to apply BCI technology to real-world problems, so that work by HF researchers is becoming increasingly important.” (from Dr. Schalk, Wadsworth Center, NY State Dept of Health)

q Require a model-based approach to wireless health monitoring systems; extracting “information” from “data”

Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU Human-Centered Computing Research @NCSU Motivations for Interfacing Brain to Computer BCI Research @NCSU Summary

§ BCI Lab at NCSU

Ø Ellen Wittenberg (Ph.D. student) Ø CK Choi (Ph.D. student) Ø Na Young Kim (Ph.D. student) Ø Farrokh Mohammadzadeh (Ph.D. student) Ø HeeSun Choi (Ph.D., PSY) Ø Kyle Bond (MS student) Ø Douglas Bryant (UG, PSY)

§ Support

Ø NSF BRAIN Initiative (IIS-1421948) Ø NSF BCS-1551688 Ø NAS/LAS (2014) Ø REC (2015; 2016)

§ Collaborators

Ø Carmen Graffagnino, M.D. (Duke) Ø Kristine Lindquist, Ph.D. (UNC-CH) Ø Yueqing Li, Ph.D. (Lamar Univ.) Ø Dean Krusienski, Ph.D. (ODU) Ø Brendan Allison, Ph.D. (USC) Ø Jing Feng, Ph.D. (NCSU) Ø Myoung-don, Oh, M.D. (Korea) Ø Junichi Ushiba, Ph.D. (Japan) Ø Takufumi Yanagisawa, M.D., Ph.D. (Japan) Ø Anton Nijholt, Ph.D. (Netherlands) Ø Fabien Lotte, Ph.D. (France) Ø Christopher Guger (Austria)

Acknowledgments

Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU

1/13/17 19 How would you feel if you can turn off the annoying alarm by just thinking to stop it?

Thank You!!!

Edward P. Fitts Dept of Industrial & Systems Engineering Brain-Computer Interface (BCI) @ NCSU