Skeletal Posture Estimation Shane Transue, Phuc Nguyen, Tam Vu, and - - PowerPoint PPT Presentation

Thermal-Depth Fusion for Occluded Body Skeletal Posture Estimation

Shane Transue, Phuc Nguyen, Tam Vu, and Min-Hyung Choi University of Colorado

IEEE Conference on Connected Health: Applications, Systems and Engineering Technologies

NON-CONTACT RESPIRATORY MONITORING

Sleep-based Supervised Respiration  Rate, tidal volume, apnea, COPD  Automated Radar Solutions

 Orthogonal radar monitoring  Region-based chest movements

 Automated Camera Solutions

 Monitors chest surface deformations  Computes changes in volume/behavior

Automated Solutions Require Posture

 Patient chest orientation  Occlusion detection

[P. Nguyen et al, IEEE INFOCOM’16] [S. Transue et al, IEEE/ACM CHASE’16]

Camera-based Tidal-volume Estimation Radar-based Tidal-volume Estimation

2/20

DEPTH AND THERMAL POSTURE ESTIMATION

Occluded Skeletal Tracking  Problem: Identifying occluded joints

 Blankets, clothing, hide joint positions  Depth-image is ambiguous  No ground-truth training/labeling/scoring

 Prior: Depth-based Skeletal Tracking  Prior: Thermal Posture Imaging (low)

3/20

Blanket induced occluded body (example) [F. Achilles et al., MICCAI’16]

THERMAL DEPTH FUSION IMAGING

Core Idea: Fuse Depth + Thermal

• Depth for 3D surface reconstruction
• Thermal for patient heat tracking

Monitoring Devices:  Microsoft Kinect2 (512x424@30[fps])  FLIR C2 (80x60@15[fps])  Alignment Bracket

Prototype Device and Experimental Design

4/20

MODELING THERMAL VOLUME POSTURE

Occlusion Implications

 Ambiguous Depth + Thermal data  Partial skeletal data (c)  Disconnected skeletal components (c)

Model Assumptions

 Predefined Skeletal Structure (b)  Enclosed volume (patient on surface)  Thermal volume reconstruction (a)

5/20 IEEE/ACM Chase 2017

THERMAL POSTURE GROUND-TRUTH

Occluded Skeletal Tracking

• Visual markers are occluded
• Skeletal structure may be incomplete
• Require a method for capturing thermal markers

Solution: Thermal Motion Tracking

• Borrows from traditional motion-capture
• Markers are defined by thermal spheres
• Interchangeable Thermal Suit

Thermal Posture Tracking (training only)

6/20

• Heated markers
• Markers are detachable
• Flexible Design
• Fixed joint count

DEPTH + THERMAL POSTURE MODELING (1)

7/20

Depth + Thermal Fusion

Proposed thermal posture estimation: Thermal + Depth to CNN Classification

DEPTH + THERMAL POSTURE MODELING (2)

8/20

Depth + Thermal Fusion TEGI Heat Propagation

Proposed thermal posture estimation: Thermal + Depth to CNN Classification

DEPTH + THERMAL POSTURE MODELING (3)

9/20

Depth + Thermal Fusion Thermal Volume Reconstruction TEGI Heat Propagation

Proposed thermal posture estimation: Thermal + Depth to CNN Classification

DEPTH + THERMAL POSTURE MODELING (4)

10/20

Depth + Thermal Fusion Thermal Volume Reconstruction Occluded Estimate Posture TEGI Heat Propagation

Proposed thermal posture estimation: Thermal + Depth to CNN Classification

THERMAL VOLUME RECONSTRUCTION

Posture Volume Assumptions

 Posture is enclosed  Human Body is a Connected-component  Propagate from known location (head)  Generate internal structure (enclosed volume)

Solution: Thermal Sphere Hierarchy

• Sphere-packing
• Boundary Conditions

(1) Surface Boundary (2) Thermal threshold

11/20

Infrared Image of Posture (enclosed volume under surface) 2D Thermal Sphere Packing

2D-3D INVERSE THERMAL PROPAGATION

How can we map 2D surface thermal information to 3D voxels? Solution: Thermal Extended Gaussian Images

• Maps 2D thermal data to 3D volumes
• Parametrized by distance, heat, etc.
• Computed by spherical projection

12/20

TEGI Point-to-volume Mapping

3D THERMAL VOLUME RESULT

13/20

Volume Reconstruction Pipeline

(1) Surface thermal-cloud (2) Volume enclosure (3) Sphere-packing and heat propagation (4) Voxel-grid representation

Depth + Thermal Enclosed Volume Heat Propagation Thermal Volume

3D THERMAL MONITORING (VIDEO)

14/20

POSTURE MONITORING APPLICATION

15/20

LABELING, TRAINING, AND CLASSIFICATION

Training: Correlate skeletal posture to volumetric thermal data

• Volumetric data provided as 3D image to CNN
• Classification based on 3D distribution
• Coarse-grain posture from classifications

16/20

Training Components (training-data + labeling) Runtime data

POSTURE CLASSIFICATION RESULTS

17/20

Note: Classification labels correspond to confusion matrix results

CLASSIFICATION RESULTS

Standard Posture Confusion Matrices

18/20

Patient-specific training/classification Cross-patient training/classification

CONCLUSION AND FUTURE WORK

Conclusion

• Fuse Depth + Fusion for occluded patient tracking
• 3D Patient heat distribution
• Occluded skeletal estimation
• Sleep-study patient tracking/posture
• Automated respiratory monitoring
• Long-term studies

Future Work

• Feature-based Training (RDF)
• Improve image resolution
• Address challenges/ambiguity
• Other Depth + Thermal applications

19/20

THANK YOU