Kinematic Model of the Hand using Computer Vision Edgar Sim o - - PowerPoint PPT Presentation

Kinematic Model of the Hand using Computer Vision

Edgar Sim´

• Serra

Barcelona, Spain Barcelona - April, 2011

Summary

Obtain hand movements through computer vision Reconstruct hand kinematics from data

Index

1 Overview 2 Kinematic Hand Model 3 Hand Detection 4 Kinematic Synthesis 5 Non-linear solver 6 Results

Overview

Motivation

EMG Signals ⇐ ⇒ Hand Movement Mapping of Electromyographic (EMG) signals Understanding hand movements Prosthesis design Exoskeleton design

Objectives

1 Design of a theoretical kinematic model of the

hand

2 Detection of real hand characteristics with

computer vision

3 Solver to adjust the theoretical model to the

experimental data

Kinematic Hand Model

State of the Art

Robotic Grippers Prosthetic Hands Match function not form Simplified kinematic models

State of the Art

Robotic Grippers Prosthetic Hands Match function not form Simplified kinematic models

State of the Art

Robotic Grippers Prosthetic Hands Match function not form Simplified kinematic models

Hand Anatomy

5 Fingers 14 Joints 27 Bones Complex articulated system

Hand Topology

3 common joints (wrist) Two fingers with 4 joints (index, middle) Three fingers with 5 joints (third, fourth, thumb) Thumb has different joint structure

Hand Topology

(a) Full hand. (b) Index finger. (c) Middle finger. (d) Third finger. (e) Fourth finger. (f) Thumb.

Hand Model

26 degrees of freedom All revolute joints Strong similarity with bone structure

Detection of Hand Characteristics

State of the Art

Contact approach

Low portability High cost Need for calibration Low resolution

State of the Art

Contact approach

Low portability High cost Need for calibration Low resolution

Non-contact Approach

Focus on realtime Low precision Applications in augmented reality

Approach

Tracking of nails

Non-deformable Fixed to hand structure Easily identifiable Not easy to identify rotation

Markers for robust design

Allows obtaining of pose (position and rotation)

Markers

Simple for robustness Convey pose information

Markers

Simple for robustness Convey pose information

Detecting Markers

1 Detect candidate points with Corner and edge

detector (Harris 1998)

2 Match candidate points with heuristics 3 Use DAISY descriptors to do tracking (Tola

2010)

Detecting Markers

1 Detect candidate points with Corner and edge

detector (Harris 1998)

2 Match candidate points with heuristics 3 Use DAISY descriptors to do tracking (Tola

2010)

PnP: Pose Estimation Problem

u = [A][R|t]p Very important problem in computer vision Identify camera position given n 2D-3D correspondences u ↔ p

Solving the Pose Estimation

Specific case 4 coplanar points: P4P Single unique solution (Yuan 1989) Usage of combination of algorithms

Orthogonal Iteration (Hager et al. 2000) Robust Pose Estimation (Schweighofer et al. 2006)

Error Analysis

50 100 150 200 50 100

Pose (0 degrees)

50 100 150 50 100

Pose (10 degrees)

50 100 150 50 100

Pose (20 degrees)

50 100 150 50 100

Pose (30 degrees)

−50 50 100 150 50 100

Pose (40 degrees)

−50 50 100 150 50 100

Pose (50 degrees)

−50 50 100 150 50 100

Pose (60 degrees)

−50 50 100 150 50 100

Pose (70 degrees)

−50 50 100 150 50 100

Pose (80 degrees)

Error Analysis

5 10 15 50 100 Noise Standard Deviation (px) Error (percent)

• T. Error (0 degrees)

Interval (99%) Mean 5 10 15 50 100 Noise Standard Deviation (px) Error (percent)

• T. Error (10 degrees)

5 10 15 50 100 Noise Standard Deviation (px) Error (percent)

• T. Error (20 degrees)

5 10 15 50 100 Noise Standard Deviation (px) Error (percent)

• T. Error (30 degrees)

5 10 15 50 100 Noise Standard Deviation (px) Error (percent)

• T. Error (40 degrees)

5 10 15 50 100 Noise Standard Deviation (px) Error (percent)

• T. Error (50 degrees)

5 10 15 50 100 Noise Standard Deviation (px) Error (percent)

• T. Error (60 degrees)

5 10 15 50 100 Noise Standard Deviation (px) Error (percent)

• T. Error (70 degrees)

5 10 15 50 100 Noise Standard Deviation (px) Error (percent)

• T. Error (80 degrees)

Error Analysis

5 10 15 20 40 60 Noise Standard Deviation (px) Rotation Error (percent)

• R. Error (0 degrees)

Interval (99%) Mean 5 10 15 20 40 60 Noise Standard Deviation (px) Rotation Error (percent)

• R. Error (10 degrees)

5 10 15 20 40 60 Noise Standard Deviation (px) Rotation Error (percent)

• R. Error (20 degrees)

5 10 15 20 40 60 Noise Standard Deviation (px) Rotation Error (percent)

• R. Error (30 degrees)

5 10 15 20 40 60 Noise Standard Deviation (px) Rotation Error (percent)

• R. Error (40 degrees)

5 10 15 20 40 60 Noise Standard Deviation (px) Rotation Error (percent)

• R. Error (50 degrees)

5 10 15 20 40 60 Noise Standard Deviation (px) Rotation Error (percent)

• R. Error (60 degrees)

5 10 15 20 40 60 Noise Standard Deviation (px) Rotation Error (percent)

• R. Error (70 degrees)

5 10 15 20 40 60 Noise Standard Deviation (px) Rotation Error (percent)

• R. Error (80 degrees)

Detection of Hand

Reliable and robust Low error in pose estimation with little noise Experimental dataset for adjusting theoretical hand model

Kinematic Synthesis

Kinematic Synthesis

Generalized inverse kinematics problem Changing structural parameters to solve problem Motion-to-form problem Very complex and generally unsolved problem

State of the Art

Simplifying the problem

Gro¨ ebner bases Elimination theory Linear product decomposition

State of the Art

Simplifying the problem

Gro¨ ebner bases Elimination theory Linear product decomposition

Revolute joint structures solved

2R: 2 solutions (A. Perez and J.M. McCarthy 2003)

State of the Art

Simplifying the problem

Gro¨ ebner bases Elimination theory Linear product decomposition

Revolute joint structures solved

2R: 2 solutions (A. Perez and J.M. McCarthy 2003) 3R: 13 solutions in an interval (Lee et al. 2004)

State of the Art

Simplifying the problem

Gro¨ ebner bases Elimination theory Linear product decomposition

Revolute joint structures solved

2R: 2 solutions (A. Perez and J.M. McCarthy 2003) 3R: 13 solutions in an interval (Lee et al. 2004) 4R: Unsolved

State of the Art

Simplifying the problem

Gro¨ ebner bases Elimination theory Linear product decomposition

Revolute joint structures solved

2R: 2 solutions (A. Perez and J.M. McCarthy 2003) 3R: 13 solutions in an interval (Lee et al. 2004) 4R: Unsolved ... 26R: Unsolved (our problem)

Kinematic Chain

F [G] S1 S2 S3 a12 d12 a23 [H] Θ12 α12 α23 M

Articulated system formed by joints Different topologies

Serial Parallel Loop Tree (hand model)

F [G] Common Joints Branch 2 Branch 1 S1 S2 S3 S4 S4

Pl¨ ucker Coordinates

Line representation: S = s

• rientation

+ǫ s0

• moment

Only 2 representations per line (s and −s)

Dual Quaternions

Homogeneous Matrix [H]

• 16 components

− → Dual Quaternion ˆ Q

• 8 components

Element of Clifford algebra Unit dual quaternions

Represents 3D displacements, points and lines Two constraints: ˆ qˆ q∗ = 1 and ˆ q · ˆ q0 = 0

Exponential Map

Based on Lie Algebra Allows joints to be written as 1-parameter screws Joint parameters: ˆ θ = θ + ǫd

Prismatic: θ = 0 Revolute: d = 0

d Θ S

ˆ S(ˆ θ) = e

ˆ θ 2S = cos

ˆ θ 2 + sin ˆ θ 2S

Forward Kinematics

Determines end effector position based on joints Direct calculation for serial kinematic topologies ˆ T(θ) = ˆ S1(ˆ θ1)ˆ S2(ˆ θ2) · · · ˆ Sn(ˆ θn)ˆ G = e

ˆ θ1 2 S1e ˆ θ2 2 S2 · · · e ˆ θn 2 Sn ˆ

G = (cos ˆ θ1 2 + sin ˆ θ1 2 S1) · · · (cos ˆ θn 2 + sin ˆ θn 2 Sn)ˆ G

Forward Kinematics

∆ˆ θ = θ − θref + ǫ(d − dref ) ˆ Q(∆ˆ θ) = ˆ S1(∆ˆ θ1)ˆ S2(∆ˆ θ2) · · · ˆ Sn(∆ˆ θn) = e

∆ˆ θ1 2 S1e ∆ˆ θ2 2 S2 · · · e ∆ˆ θn 2 Sn

= (cos ∆ˆ θ1 2 + sin ∆ˆ θ1 2 S1) · · · (cos ∆ˆ θn 2 + sin ∆ˆ θn 2 Sn)

Design Equations

Input: task defined by m − 1 relative transformations ˆ P1j = ˆ Tj ˆ T −1

1 , j = 2, . . . , m

Variables: joint axes and joint angles Equations: ˆ S1j(∆ˆ θ1j)ˆ S2j(∆ˆ θ2j)ˆ S3j(∆ˆ θ3j) ˆ Sk

4j(∆ˆ

θk

4j) · · · ˆ

Sk

7j(∆ˆ

θk

7j) − ˆ

Pk

1j = 0,

k ∈ 1, 2, ˆ S1j(∆ˆ θ1j)ˆ S2j(∆ˆ θ2j)ˆ S3j(∆ˆ θ3j) ˆ Sk

4j(∆ˆ

θk

4j) · · · ˆ

Sk

8j(∆ˆ

θk

8j) − ˆ

Pk

1j = 0,

k ∈ 3, 4, 5, j = 2, ..., m

Non-linear Solver

System Dimension

Relations between variables and equations decrease dimension Equations use 3 algebraic sets: Set Comp Ind Notes θ 1 1 Periodic with period 2π S = s + ǫs0 6 4 s · s0 = 0 , s = 1 ˆ Q = ˆ q + ǫˆ q0 8 6 ˆ q · ˆ q0 = 0 , ˆ qˆ q∗ = 1

System Dimension

System unknowns: n0

x = r(

4

• Structural

+ (m − 1)

• Joint

) nx = r(6 + (m − 1)) System functions: n0

f = b6(m − 1)

nf = b8(m − 1)

System Dimension

System must be solvable: n0

f ≥ n0 x

m = 4r 6b − r + 1 > 0 6b − r > 0 Multiple solution possibilities

System Reduction

System can be reduced to subsystems that can be solved Allows reducing the dimension of the system b r n0

x = n0 f

nx nf m Notes 5 26 780 832 1092 27 Full hand model 2 11 528 550 726 45 (4R, 4R) fingers 1 5 120 130 170 21 5R, no common

Solution Bound

High dimensionality of the system Important non-linearity of the equations Bezout’s theorem: D = d1d2 · · · dn

Serial chain: d = (3r)6 (A. Perez 2003)

di = 216(m−1) , i ∈ 1, 2, di = 246(m−1) , i ∈ 3, 4, 5, D =

5

• i=1

di ≈ 101058

Numerical Solver

Global optimizer Allow overdetermined systems

Numerical Solver

Global optimizer Allow overdetermined systems Levenberg-Marquadt Algorithm

Local optimizer Least squares Very local Very difficult to parallelize

Numerical Solver

Global optimizer Allow overdetermined systems Levenberg-Marquadt Algorithm

Local optimizer Least squares Very local Very difficult to parallelize

Genetic Algorithm

“Global” optimizer (meta-heuristics) Easy to implement Very hard to tune Extremely parallel

Implementation

Hybrid solver

Reduces search space by minimizing with Levenberg-Marquadt algorithm Global optimizer by using genetic algorithm

Implementation

Hybrid solver

Reduces search space by minimizing with Levenberg-Marquadt algorithm Global optimizer by using genetic algorithm

Implementation

Hybrid solver

Reduces search space by minimizing with Levenberg-Marquadt algorithm Global optimizer by using genetic algorithm

Results

Solver

Parallel algorithm Bottleneck in QR decomposition from Levenberg-Marquadt algorithm Average run: Generations Runtime (hours) 38 60

Solver

Mutation Solution

Solutions

Non-anthropomorphic structure Moves finger positions exactly the same as a hand Needs support for constraints to distinguish

More complicated solver

Solutions

Non-anthropomorphic structure Moves finger positions exactly the same as a hand Needs support for constraints to distinguish

More complicated solver

Solutions

(a) Full hand. (b) Index finger. (c) Middle finger. (d) Third finger. (e) Fourth finger. (f) Thumb.

Results

Software

libdq: Library for manipulating dual quaternions Kinematic hand model solver Software to track fingernail markers

Publication

• E. Simo-Serra, F. Moreno-Noguer and A.

Perez-Gracia, “Design of non-anthropomorphic robotic hands for anthropomorphic tasks”, ASME-IDETC, accepted.

Conclusions

Major problems in engineering

PnP or pose estimation problem Motion-to-form or dimensional kinematic synthesis problem

Results reflect maturity in both problems

Conclusions

Major problems in engineering

PnP or pose estimation problem Motion-to-form or dimensional kinematic synthesis problem

Results reflect maturity in both problems Hand detection

Low computation cost Reliable with slow movements

Conclusions

Major problems in engineering

PnP or pose estimation problem Motion-to-form or dimensional kinematic synthesis problem

Results reflect maturity in both problems Hand detection

Low computation cost Reliable with slow movements

Kinematic synthesis

Very high complexity Large number of solutions Non-anthropomorphic solutions found

Conclusions

Objectives

1 Defined theoretical hand model 2 Detected real hand characteristics with computer

vision

3 Adjusted theoretical model to experimental data

Conclusions

Objectives

1 Defined theoretical hand model 2 Detected real hand characteristics with computer

vision

3 Adjusted theoretical model to experimental data

Anthropomorphic solution not found

First project to attempt dimensional kinematic synthesis of tree-like topologies Kinematic synthesis is generally unsolved (3R is not algebraically solved) Upper bound of 101058 solutions

Future Work

Flexible generic tree-like topology dimensional kinematic synthesis solver

Exchangeable backends Scripting support Support for constraints

Future Work

Flexible generic tree-like topology dimensional kinematic synthesis solver

Exchangeable backends Scripting support Support for constraints

Study of design equations

How many solutions are there? What causes loss of dimensionality? How are solutions (and local minima) distributed? Can the a reduced model of the search space be used for initial estimation? Are matrix representation better at convergence?

Thanks

(a) Full hand. (b) Index finger. (c) Middle finger.

Questions?

(d) Third finger. (e) Fourth finger. (f) Thumb.