Inference Tolerant to Missing Wearable Sensors Yang Liu 1 , Zhenjiang - PowerPoint PPT Presentation

Real-time Arm Skeleton Tracking and Gesture Inference Tolerant to Missing Wearable Sensors Yang Liu 1 , Zhenjiang Li 1 , Zhidan Liu 2 , Kaishun Wu 2 City University of Hong Kong 1 , Shenzhen University 2

Understanding Human Arm Motions • How is the arm moving? 4 1 7 2 • • What is the meaning of What is the meaning 6 5 this arm motion? this arm motion? 3 8 Running unnin

Elderly Care Last treatment Elderly diseases • Parkinson • Alzheimer Several weeks Problems with arm • Slow motion • Repeated motion • Instability • … Next treatment

Other Applications Template α Compare User’s skeleton α’ 80USD/hour

Other Applications Template Template α α Compare Compare User’s skeleton User’s skeleton α’ α’ 80USD/hour 80USD/hour 80USD/hou HCI Smart home Smart car Gaming

Existing Solutions • Service coverage • System cost • Privacy • Convenience • User-friendly

Existing Solutions • Service coverage Few • System cost • Privacy • Convenience • User-friendly

Existing Solutions • Service coverage Few • System cost • Privacy • Convenience • User-friendly

Key Problem On body Elbow (Fixed offset) Wrist

Key Problem How? Elbow

Tracking Principle Z Y X Y X Z X Y Z For a given wrist orientation, possible elbow locations are within a limited range [1]. [1] “I am a smartwatch and I can track my user’s arm”, in Proc. of ACM MobiSys, 2016.

Tracking Principle t-1 t t+1 … … Location: … … Velocity: … … Acceleration: Measured Derived ___ (t) acc elb (t) acc elb Ranges across time [1] [1] “I am a smartwatch and I can track my user’s arm”, in Proc. of ACM MobiSys, 2016.

Latency Activity duration 10x on desktop 98.2s 10s Recovery delay 30s 289.3s 289.3s 1min 9.1min Time delay of existing work [1] Time [1] “I am a smartwatch and I can track my user’s arm”, in Proc. of ACM MobiSys, 2016.

Latency Activity duration Activity duration 10x on desktop 98.2s 98.2s 10s 10s Recovery delay Recovery delay • Real-time 30s 30s 289.3s 289.3s 289.3s • Without impairing accuracy 1min 1min 9.1min 9.1min Time delay of existing work [1] Time delay of existing work [1] Time Time • Our solution [ArmTroi]: • HMM state reconstruction One search • Hierarchical search space [1] “I am a smartwatch and I can track my user’s arm”, in Proc. of ACM MobiSys, 2016.

?

? Our idea: exclude the unlikely locations using as little effort as possible

Focus on more likely candidates

Hierarchical Search Tracked Elbow Location Second-layer search First-layer search T Tracked Tracked T T Tracke Tracke Tracke Tracked k e e Center Point Center P Center Point Center P Center P Center P Center Point P P t k-1 t k Original size • Real-time � � • Time complexity: � � � � � � � • Without impairing accuracy � � Size of region

Understanding Human Arm Motions • How is the arm moving? • • What is the meaning of What is the meaning this arm motion? this arm motion? Running unnin

Motion Inference Spatial Spatial LSTM LSTM LSTM LSTM Prob. Temp. Temp. LSTM LSTM Left Left A arm arm 1 Softmax layer A 2 Running LSTM LSTM �� Incline LSTM �� A n Torso LSTM Spatial LSTM LSTM Temp. Right arm Non-scalable 6 combinations of missing inputs Cost-inefficient

Motion Inference Spatial Spatial LSTM LSTM LSTM LSTM Prob. Temp. Temp. LSTM LSTM Left Left A arm arm 1 Softmax layer A 2 Running LSTM LSTM �� Incline LSTM �� A n Torso LSTM Spatial LSTM LSTM Temp. Right arm No Non-scalable on n- scala e scala b b le le e 6 combinations of 6 combinations of ombinations of ombinations of ombinations of ombinations of Handle all combinations using one network? missing inputs missing inputs missing inputs missing inputs missing inputs missing inputs issing inputs Cost-inefficient Co Cos os st st t-i inefficie inefficien inefficient t

Our idea Padding • Adaptive design … LSTM f 1 f 1 f c Fixed w weight a 11 a 12 a 13 Weight … c f 2 LSTM LSTM 0.05 0.6 0.35 + + Input f 1 f f 1 f 2 f 3 f 3 … LSTM + + = 1 a 11 a 12 a 13 Features

Attention -based network adaption y t Attention 1 LSTM Features: � � � � � � � � � � � � � � RNN (rl 3 ) y t Input: � � z t y t x t 2 LSTM LSTM � � � � � : Weighted fusion t α y t � � � � � � � � h t-1 � � � � � � ����� 3 f LSTM ��� RNN Updated weights (fl 2 ) (rl 2 ) t- 1 x • Weight update t- 1 α • aligning with the activity f t- 1 y h t- 1 z t- 1 ��� descriptor � ��� � � � � ��� � � � � ��� x t t α ��� � � ���� � � � �� ��� � � y t f h t z t � � � � �������� ����� � � ���

ArmTroi Implementation Skeleton Tracking Applications Applications Skeleton Recover Kinetic Model Acceleration Elderly Elderly Point Clouds Care Care Arm Torso Raw Data Data E-Health Skeletons Label Gesture Inference HCI . . Network . . . . Structure Design Behavior DNN Attention-based Analysis Adaptation

Experiment setup • Participants: 7 volunteers • Dataset: Daily activities • Training: Intel i7-6700 CPU and Nvidia GTX 1080Ti GPU • Running: SAMSUNG Galaxy S7

Evaluation • Skeleton tracking • ArmTrak [1] • Elbow: 12.94cm • Wrist: 14.91cm • ArmTroi • Elbow: 10.53cm • Wrist: 12.94cm [1] “I am a smartwatch and I can track my user’s arm”, in Proc. of ACM MobiSys, 2016.

Evaluation • Skeleton tracking • • ArmTrak [1] ArmTrak [1] • • Elbow: 12.94cm Elbow: 12.94cm • • Wrist: 14.91cm Wrist: 14.91cm • • ArmTroi ArmTroi • • Elbow: 10.53cm Elbow: 10.53cm • • Wrist: 12.94cm Wrist: 12.94cm • Our latency • Desktop: 0.15x • Phone: 0.47x [1] “I am a smartwatch and I can track my user’s arm”, in Proc. of ACM MobiSys, 2016.

Evaluation • Motion inference • Baseline: MULT • Each combination of missing input • Accuracy with full set • FW: 92.7% vs 92.3% • DA: 91.4% vs 91.8%

Evaluation • Motion inference • • Baseline: MULT • Each combination of f missing input • • Accuracy with full set • FW: 92.7% vs 92.3% • DA: 91.4% vs 91.8% • Weight updating • Available input: Left Arm • LA’s weight increases

Conclusion 1, 2, 3 1. One goal: • Understanding human arm motions 2. Two aspects: • Real-time tracking • Motion inference tolerant to missing inputs 3. Three techniques: • HMM state reorganization • Hierarchical search • Attention-based network adaption