Human Gesture Recognition for Drone Control Drones are cool - - - PowerPoint PPT Presentation

SLIDE 1

Human Gesture Recognition for Drone Control

SLIDE 2

Drones are cool - Flying is hard

2

SLIDE 3

Drone Controllers

3

SLIDE 4

Proposed Solution: Gesture Control

• Drones already have cameras
• No additional HW required
• Use human body as the controller
• Visible from long distances
• Requires little or no training
• Control commands that are universal and standardized

4

SLIDE 5

Our Solution

5

SLIDE 6

Our Solution (Skeleton Extraction)

6

SLIDE 7

Samplei

Joint Based Feature Extraction - Raw Features

7

...

Framet

Frame1 (F1) F2 F3

x1 y1 x2 y2 x3 y3 ... ... ... ... ... ... ... ... ... ... xn-

1

yn-

1

xn yn

Keypoints (Upper body 8x2 + left palm 21x2 + right palm 21x2 = 100)

SLIDE 8

C19 C2

Clustering

8

C1

9

C2 C2 C1

9

F1 F2 F3 F4 ... C2 C1

9

F5 F6 ...

Sample17 Sample18

SLIDE 9

Gesture Graphs

9

G1 C1 C1 C2 C2 C7 C3 C1

9

F1 F2 F3 F4 F5 F6 F7 ... ...

Sample1

G7 C1

9

C2 C2 C1

9

F1 F2 F3 F4 ... ...

Sample2

SLIDE 10

Maximum Entropy Markov Model

Discriminative model

10

SLIDE 11

Training: Compute Probabilities

11

G7 C1

9

C2 C2 C1

7

F1 F2 F3 F4 ... ...

Sample2 P(Ct

2 | Ct-1 19 , G7)

P(Ct

19 | Ct-1 2 , G7)

P(Ct

2 | Ct-1 19 , G7)

P(Ct

19 | F1)

P(Ct

2 | F2)

P(Ct

19 | F3)

P(Ct

2 | F4)

SLIDE 12

Trained Probabilities

Inference

12 P(Ct

2 | Ct-1 17 , Gn)

P(Ct

19 | Ct-1 2 , Gn)

P(Ct

2 | Ct-1 19 , Gn)

P(Ct

17 | F1)

P(Ct

2 | F2)

P(Ct

19 | F3)

P(Ct

2 | F4)

Gn C1

7

C2 C2 C1

9

F1 F2 F3 F4 ... ...

Sampletest P(Ct

2 | Ct-1 19 , G7)

P(Ct

19 | Ct-1 2 , G7)

P(Ct

2 | Ct-1 19 , G7)

P(Ct

19 | F1)

P(Ct

2 | F2)

P(Ct

19 | F3)

P(Ct

2 | F4)

P(Ct

2 | Ct-1 19 , G7)

P(Ct

19 | Ct-1 2 , G7)

P(Ct

2 | Ct-1 19 , G7)

P(Ct

2 | Ct-1 19 , G7)

P(Ct

19 | Ct-1 2 , G7)

P(Ct

2 | Ct-1 19 , G7)

P(Ct

19 | F1)

P(Ct

2 | F2)

P(Ct

19 | F3)

P(Ct

2 | F4)

P(Ct

19 | F1)

P(Ct

2 | F2)

P(Ct

19 | F3)

P(Ct

2 | F4)

SLIDE 13

Baseline and Comparison

13 votes

i=1 14

SLIDE 14

Dataset

• Source: Isolated Gesture Recognition (ICPR '16)
• RGB-D gesture videos = 47,933 (1 video = 1 gesture)
• Gestures labels 249
• Different individuals 21
• Contains 9 Air Marshalling gestures (along with others)
• Data Samples Split: Train 1399, Valid 200, Test 300
• Used OpenPose to extract 2D skeleton data from RGB videos

14

Move backward

SLIDE 15

Results

15 Model Precision Recall Accuracy MEMM 0.80 0.80 0.80 Multilayer Perceptron 0.77 0.77 0.74 Eigen Joints 0.80 0.79 0.76 HMM Tuned 0.67 0.71 0.66 HMM Baseline 0.32 0.37 0.38

SLIDE 16

Challenges and Error Analysis

16 MEMM HMM

SLIDE 17

Conclusion

• We proposed a real-time gesture recognition for drone control using

structured prediction.

• Our proposed model achieved an improvement in accuracy of ~15% over the

baseline (tuned).

• Future Work
• Combining multiple graphical probabilistic models.
• Adding the hand joints for the HMM baseline.

17

SLIDE 18

Thanks :)

18

SLIDE 19

Appendix

19

SLIDE 20

Challenges and Our approach

• Detecting Human Body Pose
• Maintaining Visibility
• Gestures Selection

20

• OpenPose to extract skeleton

data from RGB camera

• Drone will rotate (yaw) to always

face the commander

• Use Aircraft Marshalling

gestures

SLIDE 21

Drone Dynamics

21

SLIDE 22

For Paper

22

zg=1 yk=

1

yk=

1

yk=

2

yk=

2

yk=

7

yk=

3

yk=19 xt=1 xt=2 xt=3 xt=4 xt=5 xt=6 xt=7 ... ...

si=12