Robust Articulated-ICP for Real-Time Hand Tracking Andrea - - PowerPoint PPT Presentation

▶
robust articulated icp for real time hand tracking
SMART_READER_LITE
LIVE PREVIEW

Robust Articulated-ICP for Real-Time Hand Tracking Andrea - - PowerPoint PPT Presentation

Aug 17, 2023 154 likes •567 views

Robust Articulated-ICP for Real-Time Hand Tracking Andrea Tagliasacchi* Matthias Schrder* Anastasia Tkach Sofien Bouaziz Mario Botsch Mark Pauly * equal contribution Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea

slide-1
SLIDE 1

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

/36 1

Robust Articulated-ICP for Real-Time Hand Tracking

* equal contribution

Andrea Tagliasacchi* Sofien Bouaziz Matthias Schröder* Mario Botsch Anastasia Tkach Mark Pauly

slide-2
SLIDE 2

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Real-Time Tracking Setup

2

Intel RealSense PrimeSense (Carmine) SoftKinetic completely discards small portions of geometry low SNR along in depth (z-axis)

Data from (single) RGBD Sensors

slide-3
SLIDE 3

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Motivation? Augmented Reality

3

Microsoft HoloLens - PR Video (hololens.com) Intel Perceptual SDK Oculus Research (VR) MagicLeap (AR) HoloLens (AR)

slide-4
SLIDE 4

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Previous Work

4

Appearance-based

(guess solely based on current frame)

Model-based

(registers model of previous frame)

slide-5
SLIDE 5

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Model “geometry” Appearance “vision”

Previous Work

5

[Keskin ICCV’12] [Qian CVPR’14] [Oikono. CVPR’14] [Melax I3D’13] [Tang CVPR’14] [Schroder ICRA’14] [Sridhar 3DV’14] [Tompson SIG’14]

slide-6
SLIDE 6

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Contributions

  • combined 2D/3D registration (within ICP)
  • occlusion-aware correspondences (ICP)
  • regularization with statistical pose-space prior
  • extensible and unified real-time solver (>60fps)
  • revamping ICP for articulated tracking

6

slide-7
SLIDE 7

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi update correspondences

ICP: Iterative Closest Point

7

for more details please refer to Sparse-ICP [Bouaziz, Tagliasacchi, Pauly SGP’13] update correspondences update transformation update transformation

  • Step 1: optimizing correspondences
  • Step 2: optimizing transformations
slide-8
SLIDE 8

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Robust Articulated ICP

8

[Qian et al. CVPR’14] It uses alternate and gradient based optimization, converges fast, and is suitable for realtime applications. However, it can be easily trapped in poor local optima and cannot handle non-rigid

  • bjects well. Yet, it is still insufficient for

high-dimensional articulated hands, especially under free viewpoints. [Wei et al. SIGA’12] our tracking process successfully tracks the entire motion sequence while ICP fails to track most of

  • frames. This is because ICP is often

sensitive to initial poses and prone to local minimum, particularly involving tracking high-dimensional human body poses from noisy depth data. [Zhang et al. SIGA’14] The accompanying video clearly shows that our tracking process is much more robust than the ICP algorithm […] our tracking process successfully tracks the entire motion sequence while ICP fails to track most of frames.

slide-9
SLIDE 9

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

System Overview

9

linear solve

data fitting converged? 3D correspondences 2D correspondences

yes no

bounds pose space temporal collision silhouette posed model distance trans. point cloud lost tracking?

yes no

color image depth image reinitialize

min

θ

Epoints + Esilh.+Ewrist | {z }

Fitting terms

+ Epose + Ekin. + Etemporal | {z }

Prior terms

slide-10
SLIDE 10

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Preprocessing

10

  • color to identify the Region-of-Interest
  • demo: assumption on picking “+y” for PCA

Ss - sensor silhouette

  • … but all this could be learned!!

[Tompson et al. TOG’14]

slide-11
SLIDE 11

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

min

θ

Epoints + Esilh.+Ewrist | {z }

Fitting terms

+ Epose + Ekin. + Etemporal | {z }

Prior terms

linear solve

data fitting converged? 3D correspondences 2D correspondences

yes no

bounds pose space temporal collision silhouette posed model distance trans. point cloud lost tracking?

yes no

color image depth image reinitialize

Data Fitting Energies

11

slide-12
SLIDE 12

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

3D Registration (w/ occlusions)

12

M - cylinder hand model

Xs - sensor point cloud

3D Registration

= ΠM( x

Epoints = ω1 X

x∈Xs

kx ΠM(x, θ)k1

2

  • ur method (case #2)

ICP (case #2) ICP (case #1)

slide-13
SLIDE 13

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

13

Lesson #1: insufficient to render

M - cylinder hand model

Xs - sensor point cloud

3D Registration

c1 c2

Ground Truth Motion (finger 2 comes out of occlusion)

c1 c2

Correspondences of [Wei et al. SIGA’12] (renders the hand model into a point cloud)

c2 c1

Correspondences of [Our Method] (computes correspondences in close form)

slide-14
SLIDE 14

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

2D/3D Registration

14

3D Registration

M - cylinder hand model

Xs - sensor point cloud

slide-15
SLIDE 15

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

2D/3D Registration

15

3D Registration

M - cylinder hand model

Xs - sensor point cloud

2D Registration

Ss - sensor silhouette Sr - rendered silhouette

slide-16
SLIDE 16

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

16

Ss - sensor silhouette Sr - rendered silhouette

2D Registration

Esilhouette = ω2 X

p∈Sr

kp ΠSs(p, θ)k2

2

x

ΠSs

2D Registration

slide-17
SLIDE 17

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Model Prior Energies

17

linear solve

data fitting converged? 3D correspondences 2D correspondences

yes no

bounds pose space temporal collision silhouette posed model distance trans. point cloud lost tracking?

yes no

color image depth image reinitialize

min

θ

Epoints + Esilh.+Ewrist | {z }

Fitting terms

+ Epose + Ekin. + Etemporal | {z }

Prior terms

slide-18
SLIDE 18

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Joint Bounds Energy

18

slide-19
SLIDE 19

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Collision Energy

19

slide-20
SLIDE 20

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Temporal Coherence Energy

20

slide-21
SLIDE 21

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Statistical Pose Prior

21

encodes correlation across joints

slide-22
SLIDE 22

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Statistical Pose Prior

22

Recorded by VICON tracking system [Schroder ICRA’14] (they are accurate… for the person that have been recorded for)

slide-23
SLIDE 23

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

µ θ ˜ θ

Statistical Pose Prior (Soft)

23

Epose = ω4kθ (µ + ΠP ˜ θ)k2

2

+ ω5kΣ˜ θk2

2

we optimize the current pose So that it is similar to a reconstructed pose from the low dimensional subspace but when DOF are unconstrained we would like to restore the neutral (i.e. mean) pose.

slide-24
SLIDE 24

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Statistical Pose Prior (Hard)

24

Epose = ω4kθ (µ + ΠP ˜ θ)k2

2

+ ω5kΣ˜ θk2

2

… even with a high number of bases the animation remains stiff!!!

  • nly optimize in the subspace 


(i.e. variable replacement)

ω4 = ∞

slide-25
SLIDE 25

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

CPU/GPU Optimization

25

min

δθ

X

i

kfi + Jiδθk2

2

δθ = ⇣ X

i

JT

i Ji

⌘−1⇣ X

i

−JT

i fi

⌘

¯

  • Esilh. = ω2

X

p∈Sr

(nT (Jpersp(x)Jskel(x)δθ + (p − ΠSs(p, θ)))2

|Sr| ≈ 20k!!!!

|Jsilh| ≈ 20k × 26 |JT

silhJsilh| = 26 × 26

slide-26
SLIDE 26

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Results and Limitations

26

slide-27
SLIDE 27

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Motion Transfer to Rig

27

slide-28
SLIDE 28

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Tracking with Fast Motion

28

slide-29
SLIDE 29

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Tracking of Interacting Hands

29

slide-30
SLIDE 30

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Limitation: Calibration

30

slide-31
SLIDE 31

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Limitations: Fist Rotation

31

slide-32
SLIDE 32

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

State of the Art - Evaluations

32

[Sridhar ICCV’13] EM Tracker [Tompson TOG’14] ConvNets [Melax’14] Intel Perceptual SDK [Shroder ICRA’14] Subspace ICP [Tang CVPR’14] Forest Classifiers [Qian CVPR’14] ICP/PSO Hybrid

Qualitative Quantitative

adbadd flexex1 pinch count tigergrasp wave random [Tang et al. 2014] [Sridhar et al. 2013] [Sridhar et al. 2014] [ours] [ours] + re-init.

10 20 mm (re-initialization helps because Dexter-1 is a low frame-rate dataset… only 30Hz)

Dexter-1 Dataset (MPI)

slide-33
SLIDE 33

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Qualitative Comparisons

33

Convex Dynamics (Intel SDK) Subspace ICP

slide-34
SLIDE 34

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Conclusions

  • combined 2D/3D registration (within ICP)
  • occlusion-aware correspondences (ICP)
  • regularization with statistical pose-space prior
  • extensible and unified real-time solver (>60fps)
  • fully open source!

34

https://github.com/OpenGP/htrack

slide-35
SLIDE 35

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Shameless Advertisement

35

Where?

https://www.csc.uvic.ca/Program_Information/Graduate_Studies/msc_program.htm

Who pays? Who? Prof. Andrea Tagliasacchi and Brian Wyvill What? MSc (PhD) Language? English

slide-36
SLIDE 36

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Thank you!!!

36

Live demo at SGP’15!!! Don’t be shy!!

Mark Anastasia Sofien Matthias https://github.com/OpenGP/htrack

slide-37
SLIDE 37

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Extra Slides

37

slide-38
SLIDE 38

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Statistical Prior: Side Effects

38

Epose = ω4kθ (µ + ΠP ˜ θ)k2

2

+ ω5kΣ˜ θk2

2

as this prior correlates joint angles the convergence speed increases by about 3x

3 6 10 1 2 3

slide-39
SLIDE 39

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Reinitialization from Failure

39

xy-fingers z-fingers Exploration

  • f corresp. energy

Detection by ~[Qian et al. CVPR’14]

✏1 = X

x∈Xs

kx ΠM(x, θ)k2,

✏2 = Sr ∩ Ss Sr ∪ Ss

Determine tracking failure [Melax’13]

2 ≥ 1 + e−↵T [✏1,✏2] → tracking ok!

Logistic Regressor (optimize alpha s.t.):

slide-40
SLIDE 40

Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi

Particle Swarm

40