Adaptive Collaborative Jamison Heard Vanderbilt University PhD - PowerPoint PPT Presentation

Adaptive Collaborative Jamison Heard Vanderbilt University PhD Candidate Robotics March 7 th 2019

2 Health Care Human-Robot Interaction Social Robotics Military Manufacturing Space

3 Develop new methodologies that facilitate in intelligent and effective robot collaborations wit ith a human.

Dynamic Environments 4 Supervisory Peer

Workload and Performance 5 Good Performance Acceptable Level Poor Underload Normal Load Overload Overall Workload

Workload Components 6 Overall Workload Cognitive Physical Auditory Visual Speech

Envisioned System 7 Adaptive Human-Robot Teaming System Architecture Future Tasks Activity Recognition Task Current Tasks (Re-)Allocations i-CiFHaR Workload Models Workload Component Workload Interaction Estimates Metrics Decision Workload Assessment Framework Algorithm Predicted Interaction Performance Changes Communication Performance Modality Prediction Assess and Predict Determine Adaptations Apply Adaptations

Envisioned System 8 Adaptive Human-Robot Teaming System Architecture Future Tasks Activity Recognition Task Current Tasks (Re-)Allocations i-CiFHaR Workload Models Workload Component Workload Interaction Estimates Metrics Decision Workload Assessment Framework Algorithm Predicted Interaction Performance Changes Communication Performance Modality Prediction Assess and Predict Determine Adaptations Apply Adaptations

Outline 9 • Prior Work • Workload Assessment • Activity Recognition • Future Work • Adaptive Interactions • Research at RIT • Potential Funding Sources • Teaching Experience

Envisioned System 10 Adaptive Human-Robot Teaming System Architecture Future Tasks Activity Recognition Task Current Tasks (Re-)Allocations i-CiFHaR Workload Models Workload Component Workload Interaction Estimates Metrics Decision Workload Assessment Framework Algorithm Predicted Interaction Performance Changes Communication Performance Modality Prediction Assess and Predict Determine Adaptations Apply Adaptations

Objective Workload Metrics 11 Cognitive Physical Postural Measures Brain-Activity Measures Heart-Rate Cardiovascular Measures Skin Temperature Visual Auditory Speech Noise Level Eye-Tracking Measures Response Time Speech-Based Measures Respiration-Rate Noise Level

Challenges with Physiological Signals 12 Age Individual Differences Physical Fitness Training Circadian Rhythms Day-to-Day Variability Stressful Events

Related Work 13 Workload Assessment Algorithm Structure Physiological Machine Workload Signals Learning Classification J. Heard, C. E. Harriott, and J. A. Adams. A Survey of Workload Assessment Algorithms. IEEE Transactions on Human-Machine Systems. 2018.

Algorithmic Limitations 14 • Typically only assess Cognitive Workload • Do not detect the Underload State • Do not account for Individual Differences • Only analyzed in a Single Task Domain J. Heard, C. E. Harriott, and J. A. Adams. A Survey of Workload Assessment Algorithms. IEEE Transactions on Human-Machine Systems. 2018.

Objective Workload Metrics 15 Cognitive Physical Postural Measures Brain-Activity Measures Heart-Rate Cardiovascular Measures Skin Temperature Visual Auditory Speech Noise Level Eye-Tracking Measures Response Time Speech-Based Measures Respiration-Rate Noise Level

Workload Assessment Algorithm 16 Objective Workload Metrics Heart-Rate Heart-Rate Variability Respiration Rate Posture Magnitude Signal Processing and Neural Networks Feature Extraction Noise Level Speech-Rate Pitch Speech Intensity Workload Estimates J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.

Signal Processing 17 30 Second Adaptive Window Exponential Filtering J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.

Feature Extraction 19 Features Mean StDev. 30 Second Adaptive Window Exponential Filtering Slope Gradient J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.

Neural Network Architecture 20 Input Layer Hidden Layers Output Layer 4 x N + 2 128 128 128 128 1 Neurons Neurons Neurons Neurons Neurons Neuron N = number of objective Fully connected layers Workload Component workload metrics TANH activation functions Estimate J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.

Workload Estimates 21 Cognitive Physical Neural Networks Overall ∑ Auditory Workload Speech Workload Models Visual J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.

Workload State Classification 22 Underload Normal Load Overall Workload Threshold Overload J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.

Experimental Design 23 Supervisory-Based Evaluation Peer-Based Evaluation Search and Rescue Remotely Piloted Aircraft J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.

Algorithmic Analysis 24 • Three Trained Algorithms: PEER, SUP, and BOTH • Four Analysis: • Population Generalizability • Train on 70% of the participants, test on the other 30% • Cross-Teaming Generalizability • Train on peer evaluation Data, Test on supervisory • Vice-Versa • Emulated Real-World Conditions • Train on supervisory data, Test on second day supervisory data • Real-Time • Train on second day supervisory data, test in real-time J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.

Peer-Based Evaluation 25 96 96 95 95 100 90 90 88 Classification Accuracy 81 80 63 60 40 20 0 Cognitive Workload Physical Workload Overall Workload PEER SUP BOTH J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.

Supervisory-Based Evaluation 26 100 100 99 98 98 98 100 Classification Accuracy 80 67 66 63 60 40 20 0 Cognitive Workload Physical Workload Overall Workload SUP PEER BOTH J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.

Workload Transitions 27 Underload Normal Load Overload Underload Overload Normal Load Underload J. Heard and J. A. Adams. Multi-Dimensional Human Workload Assessment for Supervisory Human-Machine Teams. Journal of Cognitive Engineering and Decision Making. 2019.

Real-Time Evaluation 28 Communications Tracking Resource System Management Monitoring

Real-Time Results 29

Real-Time Results 30

Summary ry 31 • Assess Overall Workload and Contributing Components • Assess workload in Peer and Supervisory Human Robot Teams • Assess Workload Transitions • Real-Time Workload Assessments

Outline 32 • Prior Work • Workload Assessment • Activity Recognition • Future Work • Teaching Experience

Envisioned System 33 Adaptive Human-Robot Teaming System Architecture Future Tasks Activity Recognition Task Current Tasks (Re-)Allocations i-CiFHaR Workload Models Workload Component Workload Interaction Estimates Metrics Decision Workload Assessment Framework Algorithm Predicted Interaction Performance Changes Communication Performance Modality Prediction Assess and Predict Determine Adaptations Apply Adaptations

EMS Procedures 34 Intravenous Airway Therapy Management • Nasal Airway • Placing an IV • Oral Airway • Administer IV Medication • Intubation • IO Line Wound and Vital Fractures Checking High Trauma • Tourniquet • Stethoscope • Combat Gauze • Placing Monitoring Equipment • Chest Decompression • Splinting • Cardiopulmonary Resuscitation (CPR)

Hierarchical Task Analysis 35 Check for Breathing Preparation Lift Patient’s Chin Use Bag-Valve Mask CPR Give Breaths Use Mouth Compressions 30 Chest Compressions

Wearable Sensors 36 Apple Watch MYO • Accelerometer • Accelerometer • Gyroscope • Gyroscope • EMG

Accelerometer Data Example 37 CPR Compressions Breath

Video: OpenPose 38

OpenPose Challenges 39