Gesture Recognition: Hand Pose Estimation Adrian Spurr Ubiquitous Computing Seminar FS2014 27.05.2014 1
What is hand pose estimation? Input Computer-usable form 2
Augmented Reality Gaming PC Control Robot Control 3 3
Data glove • Utilizes optical flex sensors to measure finger bending. • Advantage: High accuracy, can provide haptic feedback. • Disadvantages: invasive, long calibration time, unnatural feeling, heavily instrumented. 4 4
Thanks to cheap depth cameras... Depth Camera RGB Camera 5 5
...and increase in GPU Power 6
Problems occuring • • Segmentation Noisy data 7
Problems occuring • Self-occlusion and viewpoint change: 8
Problems occuring • 27 Degrees of freedom per hand -> 280 trillion hand poses: 9
Problems occuring • Performance: For practical use, must be real time. 10
Principle of operation Algorithm 11
Existing schools of thought • • Model-based: Discriminative: Keeps internally track of Maps directly from current pose. observation to pose. Updates pose according “Learn” from training data to current pose and and apply knowledge to observation. unseen data. Processing 12
Short intro to Random Forests Ensemble learning A decision tree: Classification and Regression Consists of decision trees 13
Short intro to Random Forests Data in feature space Features = «Properties» of data 14
Short intro to Random Forests Data in feature space Features = «Properties» of data 15
Short intro to Random Forests Data in feature space Features = «Properties» of data 16
Short intro to Random Forests Data in feature space Features = «Properties» of data 17
Short intro to Random Forests Data in feature space Features = «Properties» of data 18
Building a classification tree 19
Building a classification tree 20
Building a classification tree 21
Random feature sampling Choose 𝑈 𝑘 which splits the data with maximum information gain. 22
Bagging 23
Prediction 24
RF for pose estimation Why Random Forests? How should we use them? • Robust • Must choose what to split on. • Fast • What should the labels be? • Thorougly studied 25
Advanced body pose recognition [Shotton2011] 26
Advanced body pose recognition Discriminative approach. Used in the Kinect. First paper to use synthetic training data. Basis for many future papers. [Shotton2011] 27
Creating synthetic data [Shotton2011] 28
Split funtion : Depth at position x 29 [Shotton2011]
Joint prediction [Shotton2011] 30
Per-class accuracy vs. tree depth • Accuracy increases as depth of tree increases. • Overfitting occurs for 15k training images. • More training images leads to higher accuracy and less overfitting. [Shotton2011] 31 31
Negative Results • Failure due to self-occlusion: • Failure due to unseen pose: [Shotton2011] 32
Unresolved issues • To capture all possible poses, need to generate huge amount of training data. • Training RF on big training set means more trees and deeper trees. • Big amount of memory needed. 33
Unresolved issues • To capture all possible poses, need to generate huge amount of training data. • Training RF on big training set means more trees and deeper trees. • Big amount of memory needed. • Solution: Divide training data into sub-sets and solve classification for each set separately. 34
Multi-layered Random Forest Cluster training data based on similarity. Train RF on and for each cluster. First layer assigns input to proper cluster. Second layer gives the final hand part label distribution. [Keskin2012] 35
Clustering training data Cluster based on weighted differences. Penalize differences of viewpoint, finger positions. Label each cluster, labels refer to hand shape. Train Random Forest on clusters. 36
Experts Use hand part labels. Train for each cluster a separate Random Forest. Each forest is called Expert. 37
Two prediction methods Global Expert Network: Feed input to first layer of Random Forest, average input, get hand shape label. Feed input to corresponding expert, get hand part distribution. 38
Two prediction methods Local Expert Network Feed input to first layer of Random Forest, get hand shape label for each pixel. Feed each pixel to its corresponding expert, get hand part distribution. 39
Parts distribution to pose • RDF returns the hand part distribution. • Get centre of each distribution by utilizing mean shift. 40
American Sign Language 41
First layer accuracy on ASL • 2-fold cross-validation: 97.8% • Confusion occurs for (m,n), (m,t) and (n,t) 42 42
Confusions • Confusion occurs for (m,n), (m,t) and (n,t) 43
Second layer accuracy Q = Number of clusters 44
Problems Not feasible to capture all possible variations of hand with synthetic data. Methods using only synthetic data suffer from synthetic- realistic discrepancies. But: Using realistic training data expensive, due to manually labelling them. 45 Synthetic Real
Problems Not feasible to capture all possible variations of hand with synthetic data. Methods using only synthetic data suffer from synthetic- realistic discrepancies. But: Using realistic training data expensive, due to manually labelling them. Solution: Transductive Learning. 46
Transductive Random Forest Transductive learning: learn from labelled data, apply knowledge transform to related unlabelled data Estimate pose based on knowledge gained from both labelled and unlabelled data. 47
Overview 48
Training data a = «Front» Training data consists of p = «Thumb» labelled real data and v = (3x16) synthetic data, and coordinates unlabelled real data Labelled elements are image patches, not pixels Label consists of tuple (a,p,v): a = Viewpoint p = Label of the closest joint v = Vector containing all positions of joint 49
Quality Function • Randomly choose between the two: Transductive Term Classification-Regression Term 50
Quality Function • 𝑅 𝑏 : Measures quality of split with respect to viewpoint a • 𝑅 𝑞 : Measures quality of split with respect to joint label p • 𝑅 𝑤 : Measures compactness of vote vector v 51
Quality Function Parameter Measures the “purity” of the node with respect to either the viewpoint a, or the joint label p 52
Quality Function • 𝑅 𝑢 : Measures image similarity between real data patches • 𝑅 𝑣 : Measures purity based on the association between the labelled and unlabelled data 53
Kinematic Refinement • Hands are biomechanically constrained on the poses it can do. • Use this for our advantage. • Utilize kinematic refinement to enforce these constraints. 54
Some results 55
Joint prediction accuracy 56
Estimating pose of two hands? Just apply single hand pose estimator twice? What if both hands are strongly interacting? Additional occlusion must be accounted for. 57
Dual hand pose estimation Model-based approach. Set up parameter space representing all degrees of freedom for both hands. Employ PSO to find best parameters suiting observation and current configuration with respect to a cost function. 58
Sample parameter space z - Yaw y - Pitch x - Roll 59
Cost function over param. space 60
Initialization Random sample of n particles with random velocities. 61
Iterating over parameter space Update particle velocities Update particle position with regards to: according to velocity Current velocity Local best position Global best position 62
Tracking Use RGB image to create skin map. Segment depth image according to skin map. 63
Tracking Cost function to optimize: P(h): Penalizes invalid finger positions. D(O,h,C): Penalizes discrepancies between hypothesis h and observation O. 64
Applying PSO Change particle velocity according to: = Best known position of particle i in generation k. = Best known position of all particles in generation k. Apply PSO for each observation O. Exploit temporal information by sampling particles around previous hypothesis. 65
Some results 66
Accuracy 67 67
Future of Hand Pose estimation • Academically solved • Further research in areas of recovering more than pose, such as hand model or 3D skin models. Including RGB image for prediction increases accuracy. Use of real data reduces synthetic-realistic discrepancies. 68
Thank you for your attention! 69 69
Recommend
More recommend
