Capturing full body motion Antoine Kaufmann - - PowerPoint PPT Presentation

Capturing full body motion

Antoine Kaufmann antoinek@student.ethz.ch

April 9, 2013 Distributed Systems Seminar 1

What is Motion Capture?

April 9, 2013 Distributed Systems Seminar 2

What is Motion Capture?

Motion capture is the process of recording the movement of

• bjects or people... In filmmaking and video game

development, it refers to recording actions of human actors, and using that information to animate digital character models in 2D or 3D computer animation. [Wikipedia: Motion Capture]

Sources: http://lukemccann.wordpress.com/motion-capture/

April 9, 2013 Distributed Systems Seminar 3

What is it used for?

April 9, 2013 Distributed Systems Seminar 4

Applications: Filmmaking

Sources: http://www.fxguide.com/featured/the-hobbit-weta/

http://www.ugo.com/therush/avatar-moments-that-give-us-pause-6-gallery http://www.animationmagazine.net/events/ted-ruffles-feathers-at-the-oscars/

April 9, 2013 Distributed Systems Seminar 5

Applications: Game development

Sources: http://www.shipwrckd.com/2012/09/capturedinto-ubisofts-motion-capture-studio-launch-

party/ http://gamerant.com/bioshock-infinite-elizabeth-trailer/

April 9, 2013 Distributed Systems Seminar 6

Applications: Gaming

Sources: http://123kinect.com/kinect-sports-season-achivements/25170/

http://pikigeek.com/2012/03/06/peter-molyneux-says-we-need-more-kinect-games-or-we-will-die/

April 9, 2013 Distributed Systems Seminar 7

Applications: Virtual Reality

Sources: http://www.wired.com/dangerroom/2012/01/army-virtual-reality/

http://www.doolwind.com/blog/where-is-virtual-reality/

April 9, 2013 Distributed Systems Seminar 8

Applications: Biomechanics

Sources: http://orthopedics.childrenscolorado.org/our-programs/center-for-gait-and-movement-

analysis http://www.motionanalysis.com/html/movement/movement.html

April 9, 2013 Distributed Systems Seminar 9

How can it be done?

April 9, 2013 Distributed Systems Seminar 10

System Types: Mechanical

• Potentiometers and exoskeleton
• Accurate
• Post-processing straight forward
• No global root-motion
• Restricts range of motion

Sources: http://www.metamotion.com/gypsy/gypsy-motion-capture-system-workflow.htm

April 9, 2013 Distributed Systems Seminar 11

System Types: Inertial

• Multiple inertial measurement units (IMUs)

to track motion

• Gyroscopes or accelerometers
• Only relative position
• Problems with drift

Sources: Source:http://www.xsens.com/en/general/mvn

April 9, 2013 Distributed Systems Seminar 12

Problem: Drift

• Occurs if only relative measurements are available
• Due to inaccuracies of sensors and calculations

Sources: Practical Motion Capture in Everyday Surroundings, Vlasic et al. 2007

April 9, 2013 Distributed Systems Seminar 13

System Types: Optical

• Usually cameras and markers
• Markers are
• passive (reflective) or
• active (controlled LEDs)
• Can achieve high level of detail

Sources: http://beforevfx.tumblr.com/image/44047276135

April 9, 2013 Distributed Systems Seminar 14

System Types: Optical

Sources: http://www.cgadvertising.com/pages/posts/vicon-technologies-give-usc-students-hands-

• n-motion-capture-experience128.php
• Problems:
• Occlusion
• Marker-swapping
• Requires good contrast and lighting

April 9, 2013 Distributed Systems Seminar 15

System Types

• Mechanical
• Inertial
• Optical
• Image based
• Magnetic
• Acoustic
• Radio / Electromagnetic

Motion Tracking: No SIlver Bullet but a Respectable Arsenal by Greg Welch and Eric Foxlin

April 9, 2013 Distributed Systems Seminar 16

What’s wrong with existing systems?

April 9, 2013 Distributed Systems Seminar 17

Problems with existing systems

• Heavy instrumentation of user and/or environment
• Require line of sight
• Limited range
• Inaccurate
• High latency
• Expensive

April 9, 2013 Distributed Systems Seminar 18

Prakash

Prakash: Lignting Aware Motion Capture using Photosensing Markers and Multiplexed Illuminators Ramesh Raskar (Mitsubishi Electric Research Labs) et al.

April 9, 2013 Distributed Systems Seminar 19

Prakash

Address the following problems:

• Expensive high-speed cameras required
• Limited number of markers
• Marker swapping
• Special clothing and lighting required

Sources: http://www.vicon.com/products/cameras.html

http://parasite.usc.edu/?p=403

April 9, 2013 Distributed Systems Seminar 20

Prakash Main Idea: Use most basic optical devices

• LEDs as transmitters that are fixed in the scene
• Photosensors as tags to be tracked
• Cheap components

Sources: http://de.wikipedia.org/wiki/Datei:Uvled_highres_macro.jpg

https://de.wikipedia.org/wiki/Datei:Photodiode-closeup.jpg

April 9, 2013 Distributed Systems Seminar 21

Prakash: Receiver Tags

• Photosensors, a micro controller and a transmitter
• Multiple photosensors used for different measurements

Sources: Prakash paper

April 9, 2013 Distributed Systems Seminar 22

Prakash: Projectors

• Projectors built from multiple LEDs
• Labelling space with binary mask

Sources: Prakash paper

April 9, 2013 Distributed Systems Seminar 23

Prakash: Location

Sources: Prakash paper and presentation

• Basically binary search
• Accuracy doubled by every LED
• 3 projectors needed for 3D location

April 9, 2013 Distributed Systems Seminar 24

Prakash: Orientation

• Multiple fixed beacons
• Analog photosensor without lens
• Cosine fall-off for estimation of angles

Sources: Prakash paper

April 9, 2013 Distributed Systems Seminar 25

Prakash: Illumination

• Measure RGB illumination
• Use one photosensor per color

Sources: Prakash presentation

April 9, 2013 Distributed Systems Seminar 26

Prakash: Advantages

• IDs for markers: no swapping
• Not sensitive to lighting conditions
• Imperceptible tags in regular clothing
• Orientation and illumination information
• Faster than regular cameras

April 9, 2013 Distributed Systems Seminar 27

Prakash: Drawbacks

• No solution for occlusion
• Wires on tags
• Not suitable if motion is too fast
• Simultaneity assumption: Tags don’t move while lit by projector

April 9, 2013 Distributed Systems Seminar 28

Prakash: My thoughts

• Elegant idea using simple means
• Nice fit for filmmaking
• Basically a distributed system
• How does it scale in practice?

April 9, 2013 Distributed Systems Seminar 29

Body-Mounted Cameras

Motion Capture from Body-Mounted Cameras Takaaki Shiratori (Disney Research) et al.

April 9, 2013 Distributed Systems Seminar 30

Body-Mounted Cameras

Address the following problems:

• Heavy instrumentation of environment
• System confined to studios

Sources: http://www.creativeplanetnetwork.com/the_wire/2008/07/29/vicon-house-of-moves-builds-

new-motion-capture-sound-stage-expands-staff-with-full-service-animation-team/

April 9, 2013 Distributed Systems Seminar 31

Body-Mounted Cameras

• Attach multiple outward looking cameras to the subject
• No instrumentation of the environment
• Use structure-from-motion to recover movements from camera

footage

Sources: Body-Mounted Camera Paper

April 9, 2013 Distributed Systems Seminar 32

Body-Mounted Cameras: Approach

Sources: Body-Mounted Camera Paper

April 9, 2013 Distributed Systems Seminar 33

Body-Mounted Cameras: Approach

Sources: Body-Mounted Camera Paper

April 9, 2013 Distributed Systems Seminar 34

Body-Mounted Cameras: Approach

Sources: Body-Mounted Camera Paper

April 9, 2013 Distributed Systems Seminar 35

Body-Mounted Cameras: Global Optimization

• Why global optimization?
• Keep motion smooth and minimize reprojection error

Sources: Body-Mounted Camera Paper

April 9, 2013 Distributed Systems Seminar 36

Body-Mounted Cameras: Advantages

• No instrumentation of the environment
• Works outside: no limited range
• Motion of skeleton and global root motion
• 3D structure of scene as byproduct

April 9, 2013 Distributed Systems Seminar 37

Body-Mounted Cameras: Drawbacks

• Heavy instrumentation of user
• Very long processing time
• Problems with motion-blur
• Motion in the scene problematic

April 9, 2013 Distributed Systems Seminar 38

Body-Mounted Cameras: My thoughts

• Useful as soon as cameras are significantly smaller
• Main application: biomechanics research and sports

April 9, 2013 Distributed Systems Seminar 39

Humantenna

Humantenna: Using the Body as an Antenna for Real-Time Whole-Body Interaction Gabe Cohn (Microsoft Research) et al.

April 9, 2013 Distributed Systems Seminar 40

Humantenna

Address the following problems:

• Heavy instrumentation of environment and/or user
• High latency
• Portability

Sources: Body-Mounted Camera Paper

April 9, 2013 Distributed Systems Seminar 41

Humantenna

• Body as an antenna for receiving EM noise
• No instrumentation of the environment
• Minimal instrumentation of the user
• Goals:
• Gesture recognition
• Location classification

Sources: http://en.wikipedia.org/wiki/File:Antistatic_wrist_strap.jpg

http://mizzoumagarchives.missouri.edu/2011-Summer/features/the-struggle-for-signal/index.php

April 9, 2013 Distributed Systems Seminar 42

Humantenna

Sources of electromagnetic noise:

Sources: Humantenna Paper

April 9, 2013 Distributed Systems Seminar 43

Humantenna: Gesture Recognition

• Predefined whole body

gestures

• Offline approach (initially)
• Manual hints for start/end of

gesture

• Machine learning for

classifying gestures

Sources: Humantenna Paper

April 9, 2013 Distributed Systems Seminar 44

Humantenna: Gesture Recognition

Sources: Humantenna Paper

• Three steps for recognizing a gesture:

1. Segmentation 2. Feature extraction 3. Classification

April 9, 2013 Distributed Systems Seminar 45

Humantenna: Gesture Recognition

Sources: Humantenna Paper

Step 1: Segmentation 1. Down-sample (to 244S/s) and low-pass filter → DC waveform 2. Divide waveform into ≈ 100ms windows 3. Check every window if it is active 4. Everything between first and last active window is the gesture

April 9, 2013 Distributed Systems Seminar 46

Humantenna: Gesture Recognition

Sources: Humantenna Paper

Step 2: Feature extraction 1. Divide gesture into 5+2 windows

April 9, 2013 Distributed Systems Seminar 47

Humantenna: Gesture Recognition

Step 2: Feature extraction 2. Compute features for each window

• Mean of the DC waveform
• Apply high-pass and compute root-mean-square
• Compute FFT, frequencies 0 - 500Hz.

Sources: Humantenna Paper

April 9, 2013 Distributed Systems Seminar 48

Humantenna: Gesture Recognition

Step 3: Classification 1. Use support vector machine to classify gesture

Sources: http://docs.opencv.org/doc/tutorials/ml/introduction_to_svm/introduction_to_svm.html

April 9, 2013 Distributed Systems Seminar 49

Humantenna: Gesture Recognition

Experimental results

• Performed experiments in multiple homes and rooms
• Different participant used for every home
• Trained classifier on 36 examples, tested on 4

µ σ Min Max 92% 3% 86% 98%

Table: Accuracy across homes and participants

• A training set of only 4 gestures still results in 84% accuracy

April 9, 2013 Distributed Systems Seminar 50

Humantenna: Gesture Recognition

Sources: Humantenna Paper

April 9, 2013 Distributed Systems Seminar 51

Humantenna: Location Classification

• Just use the first 0.5 seconds instead of segmentation
• Use same DC and RMS features as for gesture recognition
• High frequency peaks provide a lot of information about the

location

• Use frequencies up to 125kHz

Sources: http://activerain.com/blogsview/1022299/viewpoint-midtown-buy-one-get-one-free-what-

a-deal-

April 9, 2013 Distributed Systems Seminar 52

Humantenna: Location Classification

Experimental results

• Three participants and two homes, two participants per home
• 8 locations per home, each participant performed gestures in 5
• System achieves an accuracy of 99.6% (σ = 0.4%)
• Classifier works across users with an accuracy of 96%

April 9, 2013 Distributed Systems Seminar 53

Humantenna: Advantages

• No instrumentation of environment
• Minimal instrumentation of the user
• Gesture recognition and location classification in real time

April 9, 2013 Distributed Systems Seminar 54

Humantenna: Drawbacks

• By design coarse-grained
• Training is necessary
• Only works inside the home

April 9, 2013 Distributed Systems Seminar 55

Humantenna: My thoughts

• Again a simple and elegant solution
• Weak evaluation in the paper
• Might be very interesting in connection with a smart phone

April 9, 2013 Distributed Systems Seminar 56

Conclusions: Basics

Prakash

• Optical, inside-looking-out
• Instrumentation of user and environment

Body-Mounted Cameras

• Optical, inside-looking-out
• Instrumentation of user

Humantenna

• Electromagnetic, inside-looking-out
• Minimal instrumentation of user

April 9, 2013 Distributed Systems Seminar 57

Conclusions: Accuracy

Prakash

• As accurate as desired
• Fast movements possible

Body-Mounted Cameras

• Limited by camera performance
• Difficulty with fast movements

Humantenna

• Coarse-grained: only classification of predefined gestures

April 9, 2013 Distributed Systems Seminar 58

Conclusions: Latency and Cost

Latency Costs Prakash Very low: 1ms Cheap Body-Mounted Cameras Very high: days Expensive Humantenna Low: 0.5s Cheap

April 9, 2013 Distributed Systems Seminar 59

Conclusions Motion Tracking: No Silver Bullet, but a Respectable Arsenal

April 9, 2013 Distributed Systems Seminar 60

Questions?

April 9, 2013 Distributed Systems Seminar 61