Inference Tolerant to Missing Wearable Sensors Yang Liu 1 , Zhenjiang - - PowerPoint PPT Presentation

Real-time Arm Skeleton Tracking and Gesture Inference Tolerant to Missing Wearable Sensors

Yang Liu1, Zhenjiang Li1, Zhidan Liu2, Kaishun Wu2 City University of Hong Kong1, Shenzhen University2

Understanding Human Arm Motions

• What is the meaning of

this arm motion?

• How is the arm moving?
• What is the meaning

this arm motion?

Running unnin

3 2 1 4 5 6 7 8

Elderly Care

Elderly diseases

• Parkinson
• Alzheimer

Several weeks Last treatment Next treatment

Problems with arm

• Slow motion
• Repeated motion
• Instability
• …

Other Applications

Template User’s skeleton Compare α α’ 80USD/hour

Other Applications

Smart home Smart car HCI Gaming Template User’s skeleton Compare α α’ 80USD/hour Template User’s skeleton Compare α α’ 80USD/hour 80USD/hou

Existing Solutions

• Service coverage
• System cost
• Privacy
• Convenience
• User-friendly

Existing Solutions

• Service coverage
• System cost
• Privacy
• Convenience
• User-friendly

Few

Existing Solutions

• Service coverage
• System cost
• Privacy
• Convenience
• User-friendly

Few

Key Problem

Elbow

On body Wrist (Fixed offset)

Key Problem

Elbow

How?

Tracking Principle

[1] “I am a smartwatch and I can track my user’s arm”, in Proc. of ACM MobiSys, 2016.

For a given wrist orientation, possible elbow locations are within a limited range [1].

Y X Z Y X Z X Y Z

Tracking Principle

Acceleration:

… …

Velocity:

… …

Derived Measured

… …

t t-1 t+1

Location:

accelb

___ (t)

accelb(t)

[1] “I am a smartwatch and I can track my user’s arm”, in Proc. of ACM MobiSys, 2016.

Ranges across time [1]

Latency

30s 98.2s 1min 289.3s 10s 9.1min

Time Time delay of existing work [1]

289.3s

10x on desktop

Activity duration Recovery delay

[1] “I am a smartwatch and I can track my user’s arm”, in Proc. of ACM MobiSys, 2016.

Latency

• Our solution [ArmTroi]:
• HMM state reconstruction
• Hierarchical search

30s 98.2s 1min 289.3s 10s 9.1min

Time Time delay of existing work [1]

10x on desktop

Activity duration Recovery delay

[1] “I am a smartwatch and I can track my user’s arm”, in Proc. of ACM MobiSys, 2016.

One search space 30s 98.2s 1min 10s 9.1min

Time Time delay of existing work [1]

Activity duration 289.3s 289.3s Recovery delay

• Real-time
• Without impairing accuracy

?

?

Our idea: exclude the unlikely locations using as little effort as possible

Focus on more likely candidates

Hierarchical Search

tk-1 tk

Tracked Center Point e P e P Tracked Center Point Tracke Center P T k T Tracke Tracke T Center P Center P Center P Tracked Center Point Tracked Elbow Location

• Time complexity:
• First-layer

search Second-layer search

Original size Size of region

• Without impairing accuracy
• Real-time

Understanding Human Arm Motions

• What is the meaning of

this arm motion?

• How is the arm moving?
• What is the meaning

this arm motion?

Running unnin

LSTM

Softmax layer

Spatial

• Prob.
• LSTM

LSTM LSTM LSTM LSTM LSTM

1

A

2

A

n

A

Temp. Spatial Temp.

Left arm

Incline

LSTM

Torso Right arm

Motion Inference

6 combinations of missing inputs Non-scalable Cost-inefficient

Running

LSTM

Spatial

LSTM LSTM

Temp.

Left arm

LSTM

LSTM

Softmax layer

Spatial

• Prob.
• LSTM

LSTM LSTM LSTM LSTM LSTM

1

A

2

A

n

A

Temp. Spatial Temp.

Left arm

Incline

LSTM

Torso Right arm

Motion Inference

6 combinations of missing inputs Non-scalable Cost-inefficient

Running

LSTM

Spatial

LSTM LSTM

Temp.

Left arm

LSTM

No

• n

n- e le b scala e le b scala Co Cos

• s

st st t-i t inefficient inefficien inefficie

• mbinations of

issing inputs

• mbinations of

missing inputs 6 combinations of missing inputs missing inputs

• mbinations of

missing inputs Handle all combinations using one network?

• mbinations of

missing inputs

Our idea

Features Input

Fixed weight

… … …

w

LSTM LSTM LSTM LSTM

f1 f2 f3

c

f1 f

a11 a12 a13

+ + = 1

• Adaptive design

c f1 f2 f3 a11 a12 a13

Weight 0.05 0.6 0.35

+ +

f1 f

Padding

Attention-based network adaption

LSTM LSTM LSTM LSTM

zt xt y t

1

y t

2

y t

3

yt

RNN (rl2) (fl2) RNN (rl3)

Attention α

t

ht-1 f

• Features:
• : Weighted fusion
• Weight update
• aligning with the activity

descriptor

• Updated weights
• Input:

z t-1 z t ht-1 ht

α

t

xt α

t-1

x

t-1

y

t-1

yt

f

• f

ArmTroi Implementation

Skeleton Tracking

Raw Data Kinetic Model Point Clouds

Skeleton Recover

Arm Acceleration Torso Skeletons

Gesture Inference

DNN

Network Structure Design Attention-based Adaptation

.

Elderly Care

Applications

. . .

Elderly Care

Applications

. .

E-Health HCI Behavior Analysis

Label Data

Experiment setup

• Participants: 7 volunteers
• Dataset:
• Training: Intel i7-6700 CPU and Nvidia GTX 1080Ti GPU
• Running: SAMSUNG Galaxy S7

Daily activities

Evaluation

• Skeleton tracking

[1] “I am a smartwatch and I can track my user’s arm”, in Proc. of ACM MobiSys, 2016.

• ArmTrak [1]
• Elbow: 12.94cm
• Wrist: 14.91cm
• ArmTroi
• Elbow: 10.53cm
• Wrist: 12.94cm

Evaluation

• Skeleton tracking

[1] “I am a smartwatch and I can track my user’s arm”, in Proc. of ACM MobiSys, 2016.

• ArmTrak [1]
• Elbow: 12.94cm
• Wrist: 14.91cm
• ArmTroi
• Elbow: 10.53cm
• Wrist: 12.94cm
• Our latency
• Desktop: 0.15x
• Phone: 0.47x
• ArmTrak [1]
• Elbow: 12.94cm
• Wrist: 14.91cm
• ArmTroi
• Elbow: 10.53cm
• Wrist: 12.94cm

Evaluation

• Motion inference
• Baseline: MULT
• Each combination of

missing input

• Accuracy with full set
• FW: 92.7% vs 92.3%
• DA: 91.4% vs 91.8%

Evaluation

• Weight updating
• Available input: Left Arm
• LA’s weight increases
• Motion inference
• Baseline: MULT
• Each combination of

missing input

• Accuracy with full set
• FW: 92.7% vs 92.3%
• DA: 91.4% vs 91.8%
• f

Conclusion 1, 2, 3

• 1. One goal:
• Understanding human arm motions
• 2. Two aspects:
• Real-time tracking
• Motion inference tolerant to missing inputs
• 3. Three techniques:
• HMM state reorganization
• Hierarchical search
• Attention-based network adaption