Adaptive Collaborative Jamison Heard Vanderbilt University PhD - - PowerPoint PPT Presentation

▶
adaptive collaborative
SMART_READER_LITE
LIVE PREVIEW

Adaptive Collaborative Jamison Heard Vanderbilt University PhD - - PowerPoint PPT Presentation

Nov 14, 2023 275 likes •842 views

Adaptive Collaborative Jamison Heard Vanderbilt University PhD Candidate Robotics March 7 th 2019 2 Health Care Human-Robot Interaction Social Robotics Military Manufacturing Space 3 Develop new methodologies that facilitate in

slide-1
SLIDE 1

Adaptive Collaborative Robotics

Jamison Heard Vanderbilt University PhD Candidate March 7th 2019

slide-2
SLIDE 2

2

Social Robotics Manufacturing Space Military Health Care

Human-Robot Interaction

slide-3
SLIDE 3

3

Develop new methodologies that facilitate in intelligent and effective robot collaborations wit ith a human.

slide-4
SLIDE 4

Peer Supervisory

4

Dynamic Environments

slide-5
SLIDE 5

Workload and Performance

Overall Workload Performance

Poor Good

Underload Overload Normal Load Acceptable Level

5

slide-6
SLIDE 6

Workload Components

Overall Workload Cognitive Physical Auditory Visual Speech

6

slide-7
SLIDE 7

Workload Assessment Algorithm

Determine Adaptations Assess and Predict

Performance Prediction

i-CiFHaR

Interaction Decision Framework Workload Component Estimates Predicted Performance Task (Re-)Allocations Interaction Changes Workload Metrics Activity Recognition Workload Models

Adaptive Human-Robot Teaming System Architecture

Current Tasks Future Tasks Communication Modality

Apply Adaptations

Envisioned System

7

slide-8
SLIDE 8

Workload Assessment Algorithm

Determine Adaptations Assess and Predict

Performance Prediction

i-CiFHaR

Interaction Decision Framework Workload Component Estimates Predicted Performance Task (Re-)Allocations Interaction Changes Workload Metrics Activity Recognition Workload Models

Adaptive Human-Robot Teaming System Architecture

Current Tasks Future Tasks Communication Modality

Apply Adaptations

Envisioned System

8

slide-9
SLIDE 9

Outline

  • Prior Work
  • Workload Assessment
  • Activity Recognition
  • Future Work
  • Adaptive Interactions
  • Research at RIT
  • Potential Funding Sources
  • Teaching Experience

9

slide-10
SLIDE 10

Workload Assessment Algorithm

Determine Adaptations Assess and Predict

Performance Prediction

i-CiFHaR

Interaction Decision Framework Workload Component Estimates Predicted Performance Task (Re-)Allocations Interaction Changes Workload Metrics Activity Recognition Workload Models

Adaptive Human-Robot Teaming System Architecture

Current Tasks Future Tasks Communication Modality

Apply Adaptations

Envisioned System

10

slide-11
SLIDE 11

Objective Workload Metrics

Cognitive Physical Auditory Visual Speech

Brain-Activity Measures Cardiovascular Measures Eye-Tracking Measures Speech-Based Measures Respiration-Rate Noise Level Noise Level Response Time Postural Measures Heart-Rate Skin Temperature

11

slide-12
SLIDE 12

Challenges with Physiological Signals

Individual Differences Day-to-Day Variability Age Physical Fitness Training Circadian Rhythms Stressful Events

12

slide-13
SLIDE 13

Related Work

13

Physiological Signals Machine Learning Workload Classification

  • J. Heard, C. E. Harriott, and J. A. Adams. A Survey of Workload Assessment Algorithms. IEEE Transactions on Human-Machine Systems. 2018.

Workload Assessment Algorithm Structure

slide-14
SLIDE 14

Algorithmic Limitations

  • Typically only assess Cognitive Workload
  • Do not detect the Underload State
  • Do not account for Individual Differences
  • Only analyzed in a Single Task Domain

14

  • J. Heard, C. E. Harriott, and J. A. Adams. A Survey of Workload Assessment Algorithms. IEEE Transactions on Human-Machine Systems. 2018.
slide-15
SLIDE 15

Objective Workload Metrics

Cognitive Physical Auditory Visual Speech

Brain-Activity Measures Cardiovascular Measures Eye-Tracking Measures Speech-Based Measures Respiration-Rate Noise Level Noise Level Response Time Postural Measures Heart-Rate Skin Temperature

15

slide-16
SLIDE 16

Workload Assessment Algorithm

16

  • J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and

Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.

Heart-Rate Heart-Rate Variability Respiration Rate Posture Magnitude Noise Level Speech-Rate Pitch Speech Intensity Signal Processing and Feature Extraction Neural Networks Workload Estimates

Objective Workload Metrics

slide-17
SLIDE 17

Signal Processing

17

30 Second Window Adaptive Exponential Filtering

  • J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and

Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.

slide-18
SLIDE 18

Feature Extraction

19

30 Second Window Adaptive Exponential Filtering Mean StDev. Slope Gradient

  • J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and

Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.

Features

slide-19
SLIDE 19

Neural Network Architecture

20

4 x N + 2 Neurons

Input Layer

N = number of objective workload metrics 128 Neurons 128 Neurons 128 Neurons 128 Neurons 1 Neuron

Hidden Layers Output Layer

Fully connected layers TANH activation functions Workload Component Estimate

  • J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and

Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.

slide-20
SLIDE 20

Workload Estimates

21

Neural Networks Cognitive Physical Auditory Speech

  • J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and

Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.

Visual Workload Models Overall Workload ∑

slide-21
SLIDE 21

Workload State Classification

22

Overall Workload

  • J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and

Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.

Threshold

Underload Normal Load Overload

slide-22
SLIDE 22

Experimental Design

Peer-Based Evaluation Supervisory-Based Evaluation

23

  • J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and

Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.

Search and Rescue Remotely Piloted Aircraft

slide-23
SLIDE 23

Algorithmic Analysis

  • Three Trained Algorithms: PEER, SUP, and BOTH
  • Four Analysis:
  • Population Generalizability
  • Train on 70% of the participants, test on the other 30%
  • Cross-Teaming Generalizability
  • Train on peer evaluation Data, Test on supervisory
  • Vice-Versa
  • Emulated Real-World Conditions
  • Train on supervisory data, Test on second day supervisory data
  • Real-Time
  • Train on second day supervisory data, test in real-time

24

  • J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and

Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.

slide-24
SLIDE 24

25

Peer-Based Evaluation

  • J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and

Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.

96 90 96 88 63 90 95 81 95

20 40 60 80 100

Cognitive Workload Physical Workload Overall Workload Classification Accuracy PEER SUP BOTH

slide-25
SLIDE 25

26

Supervisory-Based Evaluation

  • J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and

Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.

98 99 100 63 66 67 98 98 100

20 40 60 80 100

Cognitive Workload Physical Workload Overall Workload Classification Accuracy SUP PEER BOTH

slide-26
SLIDE 26

Workload Transitions

27

  • J. Heard and J. A. Adams. Multi-Dimensional Human Workload Assessment for Supervisory Human-Machine Teams.

Journal of Cognitive Engineering and Decision Making. 2019.

Underload Normal Load Overload Underload Overload Normal Load Underload

slide-27
SLIDE 27

Real-Time Evaluation

28

Tracking Communications Resource Management System Monitoring

slide-28
SLIDE 28

Real-Time Results

29

slide-29
SLIDE 29

Real-Time Results

30

slide-30
SLIDE 30

Summary ry

  • Assess Overall Workload and Contributing Components
  • Assess workload in Peer and Supervisory Human Robot Teams
  • Assess Workload Transitions
  • Real-Time Workload Assessments

31

slide-31
SLIDE 31

Outline

  • Prior Work
  • Workload Assessment
  • Activity Recognition
  • Future Work
  • Teaching Experience

32

slide-32
SLIDE 32

Workload Assessment Algorithm

Determine Adaptations Assess and Predict

Performance Prediction

i-CiFHaR

Interaction Decision Framework Workload Component Estimates Predicted Performance Task (Re-)Allocations Interaction Changes Workload Metrics Activity Recognition Workload Models

Adaptive Human-Robot Teaming System Architecture

Current Tasks Future Tasks Communication Modality

Apply Adaptations

Envisioned System

33

slide-33
SLIDE 33

EMS Procedures

34

Airway Management

  • Placing an IV
  • Administer IV Medication
  • IO Line
  • Nasal Airway
  • Oral Airway
  • Intubation
  • Chest Decompression
  • Cardiopulmonary Resuscitation (CPR)
  • Tourniquet
  • Combat Gauze
  • Splinting
  • Stethoscope
  • Placing Monitoring Equipment

Intravenous Therapy Vital Checking Wound and Fractures High Trauma

slide-34
SLIDE 34

Hierarchical Task Analysis

35

CPR Preparation Give Breaths Compressions Check for Breathing Lift Patient’s Chin Use Bag-Valve Mask Use Mouth 30 Chest Compressions

slide-35
SLIDE 35

Wearable Sensors

Apple Watch

  • Accelerometer
  • Gyroscope

MYO

  • Accelerometer
  • Gyroscope
  • EMG

36

slide-36
SLIDE 36

Accelerometer Data Example

Compressions Breath CPR

37

slide-37
SLIDE 37

Video: OpenPose

38

slide-38
SLIDE 38

OpenPose Challenges

39

slide-39
SLIDE 39

Automatic Procedure Detection

40

Wearable Sensors Video Feature Extraction OpenPose Active Body Region Detection Random Forest Clinical Procedure

slide-40
SLIDE 40

Preliminary Results

41

0.2 0.4 0.6 0.8 1 1.2

Classification Accuracy by Procedure and Known Body Region Condition

Unknown Body Region Known Body Region

  • J. Heard, R. Paris, D. Scully, C. McNaughton, J. Ehrenfeld, J. Coco, D. Fabbri, B. Bodenheimer, and J. A. Adams. Automatic Clinical

Procedure Detection for Emergency Services. IEEE Conference on Engineering in Medicine and Biology. (In Review)

slide-41
SLIDE 41

Summary

  • Hierarchal Task Analysis can decompose activities into more

informative representations for machine-learning

  • Combining wearable sensors with video data can increase

activity recognition performance

42

slide-42
SLIDE 42

Outline

  • Prior Work
  • Future Work
  • Adaptive Interactions

43

slide-43
SLIDE 43

Workload Assessment Algorithm

Determine Adaptations Assess and Predict

Performance Prediction

i-CiFHaR

Interaction Decision Framework Workload Component Estimates Predicted Performance Task (Re-)Allocations Interaction Changes Workload Metrics Activity Recognition Workload Models

Adaptive Human-Robot Teaming System Architecture

Current Tasks Future Tasks Communication Modality

Apply Adaptations

Envisioned System

44

slide-44
SLIDE 44

MATB: Alarms and Communications

45

System Monitoring Communications

  • Visual
  • Auditory
  • Speech

Improve Reaction Time and Accuracy in the Underload and Overload states

slide-45
SLIDE 45

System Monitoring Performance

46

  • Auditory
  • Visual

UI Improvement:

slide-46
SLIDE 46

System Monitoring Performance

47

  • Auditory

Underload Overload

  • Visual

Adaption: Overall Workload State

  • Auditory or Visual?

Normal Load

slide-47
SLIDE 47

System Monitoring Performance

48

Adaption: Workload Component States

Is the human’s Speech or Auditory Channel Loaded?

  • Auditory

No

Wait x seconds

  • Visual

Yes

slide-48
SLIDE 48

Summary

  • UI or system improvement can be ineffective
  • Using the overall workload state to adapt interactions is

possible, but sub-optimal

  • Using workload component estimates to adapt allows for more

intelligent and effective interactions

49

slide-49
SLIDE 49

Outline

  • Prior Work
  • Future Work
  • Adaptive Interactions
  • Next Five Years

50

slide-50
SLIDE 50

Workload Assessment Algorithm

Determine Adaptations Assess and Predict

Performance Prediction

i-CiFHaR

Interaction Decision Framework Workload Component Estimates Predicted Performance Task (Re-)Allocations Interaction Changes Workload Metrics Activity Recognition Workload Models

Adaptive Human-Robot Teaming System Architecture

Current Tasks Future Tasks Communication Modality

Apply Adaptations

Envisioned System

51

slide-51
SLIDE 51

Research Problems

  • Improving the Workload Assessment Algorithm
  • Allocating Tasks to Robots and Humans
  • Extending the Adaptive Human-Robot Teaming Architecture

52

slide-52
SLIDE 52

Research Problem #1

53

Vision-Based Physiological Measurements Kalman Filter for Varying Workload Estimate Sampling Rates Deep Learning Estimating Visual Workload

slide-53
SLIDE 53

Research Problem #2

54

Adaptive Automation Multiple UAVs

slide-54
SLIDE 54

Research Problem #3

55

Adaptive Interactions in Collaborative Roles

slide-55
SLIDE 55

Collaborators

56 Funding Sources NASA Cooperative Agreement No. NNX16AB24A Department of Defense Contract Number W81XWH-17-C- 0252 from the CDMRP Defense Medical Research and Development Program

  • Dr. Julie A. Adams

David Greiner (MS) Julian Fortune Rachel Heald Anjali Vashisht

  • Dr. Bobby Bodenheimer
  • Dr. Daniel Fabbri

Richard Paris (Not Shown)

slide-56
SLIDE 56

Questions?

57