User-Centred BCI I for Mechatronic Actuation by Spatio-Temporal - PowerPoint PPT Presentation

IWES 2018 Siena, September 13-14, 2018 User-Centred BCI I for Mechatronic Actuation by Spatio-Temporal P300 Monitoring Daniela De Venuto* 1 , Giovanni Mezzina 1 , Valerio F. Annese 2 1 Dept. of Electrical and Information Engineering, Politecnico di Bari, Bari, Italy 2 School of Engineering, University of Glasgow, Glasgow, United Kingdom *(daniela.devenuto@poliba.it)

Outline ❑ Introduction: the “Brain Computer Interface” ❑ Methods: the Overall Architecture and Algorithm ▪ Machine Learning ▪ Features Management ▪ Classification ❑ Experimental Results ❑ Conclusions Siena, Italy September 13-14, 2018 IWES 2018

❑ Introduction The “Brain Computer Interface” ❑ Methods ❑ Results ❑ Conclusions General BCI Control System A "Brain-Computer Interface" (BCI) Signal Feature Intentions is the control loop platform Acquisition Extraction Recognition between the human brain and Commands mechanical devices . BCI Application Feedback Goal: To create enabling technology, even for disabled people, controlling devices by their mind Siena, Italy September 13-14, 2018 IWES 2018

❑ Introduction The “Brain Computer Interface” ❑ Methods ❑ Results ❑ Conclusions The BCI is based on the recognition of a particular Brain Activity Pattern (BAP) , that is excited during a particular mental task. Some of the most used (state of the art): Cursors and Speller ❑ Event related potentials (ERP) ❑ Slow cortical potentials (SCP) ❑ Event-related synchronization potentials (ERD/ERS) ❑ Steady state visual potentials Hochberg et al.(2006) (SSVP) Car Driving ❑ Sensorimotor rhythms (SMR) Duan Feng et al. (2015) Siena, Italy September 13-14, 2018 IWES 2018

❑ Introduction The “Brain Computer Interface” ❑ Methods ❑ Results ❑ Conclusions The BCI is based on the recognition of a particular Brain Activity Pattern (BAP) , that is excited during a particular mental task. Some of the most used (state of the art): Cursors and Speller Wheelchairs ❑ Event related potentials (ERP) ❑ Slow cortical potentials (SCP) ❑ Event-related synchronization potentials (ERD/ERS) ❑ Steady state visual potentials Hochberg et al.(2006) Tanaka et al. (2015) (SSVP) Car Driving ❑ Sensorimotor rhythms (SMR) Duan Feng et al. (2015) Siena, Italy September 13-14, 2018 IWES 2018

❑ Introduction The “Brain Computer Interface” ❑ Methods ❑ Results ❑ Conclusions The BCI is based on the recognition of a particular Brain Activity Pattern (BAP) , that is excited during a particular mental task. Some of the most used (state of the art): Prothesis Cursors and Speller Wheelchairs ❑ Event related potentials (ERP) ❑ Slow cortical potentials (SCP) ❑ Event-related synchronization Ortner et al. (2011) potentials (ERD/ERS) ❑ Steady state visual potentials Hochberg et al.(2006) Tanaka et al. (2015) (SSVP) Car Driving ❑ Sensorimotor rhythms (SMR) Duan Feng et al. (2015) Siena, Italy September 13-14, 2018 IWES 2018

❑ Introduction The “Brain Computer Interface” ❑ Methods ❑ Results ❑ Conclusions The BCI is based on the recognition of a particular Brain Activity Pattern (BAP) , that is excited during a particular mental task. Some of the most used (state of the art): Prothesis Cursors and Speller Wheelchairs ❑ Event related potentials (ERP) ❑ Slow cortical potentials (SCP) ❑ Event-related synchronization Ortner et al. (2011) potentials (ERD/ERS) ❑ Steady state visual potentials Hochberg et al.(2006) Tanaka et al. (2015) (SSVP) Neuro-games Car Driving ❑ Sensorimotor rhythms (SMR) Duan Feng et al. (2015) « Neuro-Pong» (2010) Siena, Italy September 13-14, 2018 IWES 2018

❑ Introduction The “Brain Computer Interface” ❑ Methods ❑ Results ❑ Conclusions The BCI is based on the recognition of a particular Brain Activity Pattern (BAP) , that is excited during a particular mental task. Some of the most used (state of the art): Prothesis Cursors and Speller Wheelchairs ❑ Event related potentials (ERP) ❑ Slow cortical potentials (SCP) ❑ Event-related synchronization Ortner et al. (2011) potentials (ERD/ERS) ❑ Steady state visual potentials Hochberg et al.(2006) Robotics Control Tanaka et al. (2015) (SSVP) Neuro-games Car Driving ❑ Sensorimotor rhythms (SMR) Bogue et al. (2014) Duan Feng et al. (2015) « Neuro-Pong» (2010) Siena, Italy September 13-14, 2018 IWES 2018

❑ Introduction ❑ Methods State of the Art ❑ Results ❑ Conclusions Mean 1 Physiological Phenomena Number of Training Signal Transfer rate (Occurrence Time) choices Time Accuracy (Opt: ≥4) (Opt: ≤1h) (Opt: ≥30 bits/ min) (Opt: >80%) Neural activity elicited by a visual 60-100 SSVP (or VEP) <12 Hours 80% stimulus (~10-70ms - AS ) bits/min Slow Cortical Potentials are shifts 5-12 SCP in the cortical electrical activity 2 -4 Weeks 86% bits/min (200ms BS to 300 ms AS ) Positive peaks due to the 20-25 P300 occurrence of single or rare <9 Hours 84% bits/min stimulus (~150-450ms AS) Modulations in sensorimotor 3-20 SMR 2-5 Weeks 85% rhythms (up to 8s AS) bits/min 1 Mean accuracy evaluated on work that operates on single trial classification; AS: after stimulus; BS: before stimulus Not in line with the BCI needs Could be improved In line with the BCI needs Siena, Italy September 13-14, 2018 IWES 2018

❑ Introduction Our Aim is … ❑ Methods ❑ Results ❑ Conclusions Number of Training Transfer Mean Signal Physiological Phenomena choices Time rate Accuracy Positive peaks due to single 20-25 P300 <9 Hours 84% or rare stimulus bits/min Create a P300-based BCI system for the remote control of mechatronic device, which ensures: ❑ High accuracy in detection ❑ Fast User-Centered Machine Learning Stage ❑ Computationally easy algorithms for portable hardware (Raspberry Pi, Microcontrollers, FPGAs, etc.) ❑ No brain signals modulation request ❑ Quick and accurate intention recognition Siena, Italy September 13-14, 2018 IWES 2018

❑ Introduction ❑ Methods The architecture ❑ Results ❑ Conclusions O FF -L INE M ACHINE L EARNING 1. Symmetry @ Ch i 2. Convexity@ Ch i Dimensionality SVM-based Decision 3. Triangle Area@ Ch i Reduction Boundaries Extraction 4. Peak to Max @ Ch i 5. Num changes @ Ch i S 1 6. Cumsum @ Ch i S j 6x6fts Features Extraction 6xEEG (n_ch*6 fts) t-RIDE Nfts nec NCA Features Algorithm 1. n=6 Upper Ampl. Extraction of highly Selection for Ti 2. n=6 Lower Ampl. characterizing area 3. n=6 Latency for Ti SVM Boundaries Functional based Classification Features Extraction Nfts nec (Nfts nec ) O N -L INE C LASSIFICATION Siena, Italy September 13-14, 2018 IWES 2018

❑ Introduction ❑ Methods The Hardware & Environment ❑ Results ❑ Conclusions The adopted stimulation protocol is a custom visual oddball paradigm : ❑ visual stimulation . ❑ random flash on a display ( 25% occurrence). ❑ inter-stimuli ( ISI ) time 500ms . Siena, Italy September 13-14, 2018 IWES 2018

❑ Introduction ❑ Methods The Hardware & Environment ❑ Results ❑ Conclusions Stimulation Terminal BCI Simulink Control System EEG Headset EEG Prototype Car Base Station System (PCS) ATMega328 P-PU Ultrasonic Sensors PCS Core Bluetooth Interface Siena, Italy September 13-14, 2018 IWES 2018

❑ Introduction ❑ Methods The Machine Learning Stage ❑ Results ❑ Conclusions T HE P RE -P ROCESSING Filtering: ❑ Bandpass Filtering (8th order Butterworth Filter: 0.5 – 30 Hz) ❑ 4th order Notch Butterworth : 48 – 52 Hz ❑ 4th order Low Pass Butterworth : 13 Hz Data slicing: The EEG data are decomposed in data blocks (observation) of 600ms. Siena, Italy September 13-14, 2018 IWES 2018

❑ Introduction ❑ Methods The Machine Learning Stage ❑ Results ❑ Conclusions T -RIDE: P300 CHARACTERIZATION The ML stage is entrusted to the tuned - Residue Iterative Decomposition (t-RIDE) approach [1]. It is based on the hypotesis of well-structured brain response. t-RIDE divides the signal into two (or three) components: ❑ Stimulus recognition ❑ Stimulus Classification: P300 ❑ (Optional) Active Response [1] D. De Venuto, V. F. Annese and G. Mezzina, "Remote Neuro-Cognitive Impairment Sensing Based on P300 Spatio-Temporal Monitoring," in IEEE Sensors Journal , vol. 16, no. 23, pp. 8348- 8356, Dec.1, 2016.doi: 10.1109/JSEN.2016.2606553 Siena, Italy September 13-14, 2018 IWES 2018

❑ Introduction ❑ Methods The Machine Learning Stage ❑ Results ❑ Conclusions T HE P300 F EATURE E XTRACTION Fea eature #1 #1: Sym ymmetry ry Fea eature #3 #3: IT ITA Fea eature #2 #2: Con Convexity Fea eature #4 #4: PPD PPD Fea eature #5 #5: NSC SC Fea eature #6 #6: Cu Cumulative Su Sum Siena, Italy September 13-14, 2018 IWES 2018