Motion Capture and Animation (01) RNDr. Martin Madaras, PhD. - - PowerPoint PPT Presentation

Virtual and Augmented Reality

Motion Capture and Animation (01)

• RNDr. Martin Madaras, PhD.

madaras@skeletex.xyz

Outline

2

 Principles of animation  Keyframe animation  Articulated figures  Skeletal animation  Motion capture  Projects…

3

• Ask questions, please!!!
• Be communicative
• www.slido.com #VAR01
• More active you are, the better for you!

How the lectures should look like #1

Manual animation

4

 Stop-motion animation  e.g. Coraline, Wallace & Gromit, etc.

Computer Animation

5

 What is animation?

 Make objects change over time according to scripted actions

 What is simulation?

 Predict how object change over time according to physical laws

Computer Animation

6

 Animation pipeline

 3D Modeling  Articulation  Motion specification  Motion simulation  Shading  Lighting  Rendering  Postprocessing

Keyframe Animation

7

 Define character poses at specific time steps called

“keyframes”

Inbetweening (“tweening”)

8

 Computing missing values based on existing surrounding

values

45

Inbetweening (“tweening”)

9

 Linear (constant)  Ease-in, ease-out

Keyframe Animation

10

 Inbetweening:

 Linear interpolation – usually not enough continuity

Spline Continuity

11

 How to ensure curves are “smooth”  Generally we have three levels of continuity  C0 - The curves meet  C0 & C1 - The tangents are shared  C0 & C1 & C2 - The “speed” is the same

C0 Continuity

12

 Zero order parametric continuity

C0 & C1 Continuity

13

 First order parametric continuity

C0 & C1 & C2 Continuity

14

 Second order parametric continuity

Implications for animation

15

 Linear interpolation is only C0

 Movement changes instantly at keyframes  Very unnatural looking

 We need at least C0 & C1 continuity

 Hermite interpolation  Spline interpolation  “Smoothstep” function

Smoothstep

16

Smoothstep

17

float smootherstep(float edge0, float edge1, float x) { // Scale, and clamp x to 0..1 range x = clamp((x - edge0)/(edge1 - edge0), 0.0, 1.0); // Evaluate polynomial return x*x*x*(x*(x*6 - 15) + 10); }

Keyframe Animation

18

 Inbetweening:

 Spline interpolation – may be visually good enough  May not follow physical laws

Keyframe Animation

19

 Inbetweening:

 Inverse kinematics or dynamics

Articulated Figures

20

 Character poses described by set of rigid bodies

connected by “joints”

Articulated Figures

21

 Well suited for humanoid characters

Keyframe Animation

22

 Inbetweening:

 Compute angles between keyframes

Example: Walk Cycle

23

 Inbetweening:

 Compute angles between keyframes

Example: Walk Cycle

24

 Hip joint orientation

Example: Walk Cycle

25

 Knee joint orientation

Example: Walk Cycle

26

 Ankle joint orientation

Animation Hierarchies

27

 Animate objects in relation to their parent

 Sun matrix is Ms  Earth matrix is MsMe  Moon matrix is MsMeMm

Kinematics and Dynamics

28

 Kinematics

 Considers only motion  Determined by positions, velocities, accelerations

 Dynamics

 Considers underlaying forces  Capture motion from initial positions and physics

Example: 2-Link Structure

29

 Two links connected by rotational joints

Forward Kinematics

30

 Animators specifies angles Θ1 and Θ2  Computer finds position of end effector: X

Forward Kinematics

31

 Joint motion can be specified by spline curves

Joint motion can be specified by spline curves

Forward Kinematics

32

 Joint motions can be specified by initial conditions

Joint motion can be specified by spline curves

Example: 2-Link Structure

33

 What if animator knows position of “end effector”

Inverse Kinematics

34

 Animator specifies end effector positions: X  Computer finds joint angles Θ1 and Θ2

Inverse Kinematics

35

 End-effector positions can be specified by splines

Inverse Kinematics

36

 Problem with more complex structures

 System with equation is usually under-defined  Multiple solutions

Inverse Kinematics

37

 Solution for more complex structures

 Find best solution (eg. minimize energy in motion)  Non-linear optimization

Inverse Kinematics

38

 Forward Kinematics

 Specify conditions (joint angles)  Compute conditions of end effectors

 Inverse Kinematics

 “Goal-directed” motion  Specify goal positions of end effectors  Compute conditions required to achieve goal

39

Character Animation (Linear Blend Skinning) (Skeletal Animation)

Skeletal Animation

40

 Hierarchical graph structure called Skeleton

 Nodes and edges (bones)

Skeletal Animation

41

 Graph structure can be disconnected in space

Real-time skeletal skinning

42

https://www.youtube.com/watch?v=DfIfcQiC2oA

Facial animation

43

 Facial expressions  Lips to speech synchronization  Controllers  Skinning  Morphing

http://www.anzovin.com/products/tfm1maya.html

Reusable animation

44

 One skeleton – different models

http://www.studiopendulum.com/alterego/

Motion capture

45

 Markers on actor’s body  Optical / magnetic sensors  3D reconstruction of markers’ position  Motion mapping

to virtual character

Outline

46

 Principles of animation  Keyframe animation  Articulated figures  Skeletal animation  Motion capture  Projects…

47

• Ask questions, please!!!
• Be communicative
• www.slido.com #VAR01
• More active you are, the better for you!

How the lectures should look like #2

Motion Capture

48

 Capturing of actor’s movement  Natural animation of CG characters

Optical Mocap

49

 Estimation of marker positions

Optical Mocap Problems

50

 Complicated setup  Problems with occlusions  Limited space

Inertial Mocap

51

 Inertial Measurement Unit  Bone rotations from IMU sensors

Mocap Suit Problems

52

 Custom implementation of walking algorithm  Inaccurate for sliding, complex jumping etc.  Recalibration needed over time

Mocap Walking Algorithm

53

Walking Algorithm Fail-Case

54

Optinertial

55

Optical-Inertial Synchronization of MoCap Suit with Single Camera Setup for Reliable Position Tracking VISIGRAPP 2018 (Adam Riečický, Martin Madaras, Michal Piovarči)

Motivation

56

 Combine optical and inertial data  Use camera image to find the actor in the scene  Use data streamed from the mocap suit

Main Idea

57

 Synchronize inertial mocap suit and camera data

 Suit joint rotations  Camera-based position estimation

 Overcome disadvantage of each method  Get reliable position tracking and mocap rotations  How to merge and synchronize the data ?

58

Related work

RGB Camera-based Mocap / Tracking

59

 Multi-view RGB – sum of spatial Gaussians

 Cheung et al. 2003  Stoll et al. 2011

 RGB camera probabilistic methods

 Wojek et al. 2009

 Optical flow + Chamfer matching

 Dimitrijevic et al. 2006  Katz and Aghajan 2008

Related Hybrid Methods

60

 Laser scanner + robot + IMUs

 Ziegler et al. 2011

 Template-based + IMUs + RGB cameras

 Pons-Moll et al. 2017  von Marcard et al. 2016

 Depth camera +IMUs + template

 Helten et al. 2013

Related Work Problems

61

 Mesh templates  Depth cameras  Multiple RGB cameras  Complicated setup  Position estimation only  Our Optinertial system needs only RGB or

monochromatic camera and the suit data

62

Optical-Inertial Synchronization

Our Idea

63

Optinertial Pipeline

64

 Chamfer matching

Actor Image

65

 Distance transform image  Background subtraction / contrast suit color

Optinertial Pipeline

66

 Template image rendering

Mocap Suits

67

 High number of IMUs vs. affordable price

Virtual Scene

68

 Real-time update of the virtual scene  Frame rendering  Camera calibration / pose estimation

Camera Calibration

69

 OpenCV ChArUco board  Estimation of origin/camera position in world space

Base-mesh Rendering

70

 SQM algorithm (skeleton, radii)

 Baerentzen et al. 2012

Base-mesh Rendering

71

 Actor image database  Skinning of base-mesh

Optinertial Pipeline

72

 Minimizing DT for the best match

Actor Movement Detection

73

 Find the best match for actor shift vector  Distance transform minimization

Image-space Optimization

74

 Silhouette + kernel integration

Optimization Implementation

75

 Parallel matrix multiplication on GPU  The best match gives the actor motion vector

Optinertial Pipeline

76

 Skeleton root translation

Results - Video

77

78

Evaluation

Evaluation Scene

79

 HTC Vive ground truth tracking

Evaluation Paths

80

Data Smoothing

81

 Post-processing  Gaussian filtering

Results - Error

82

 Suit walking algorithm vs. our approach

Results – Visualization

83

 Sliding sequence

Conclusion

84

 Optical-inertial tracking and motion capture  Monochromatic camera only  Solid position estimation during jumping and sliding  Our method may produce noisy results, however it can

be improved using Gaussian filtering

Limitations

85

 Camera occlusions are still problem  Fails if subject is too close to camera  Cannot run real-time for higher database sampling

Inertial + Optical Mocap

86

Capturing of Movement During Music Performance Bachelor thesis 2018

• Bc. Dana Škorvánková

Goals

• Implement a framework for

inertial motion capture – forearms, hands, fingers captured while playing the guitar

• Apply the captured motion on

a 3D humanoid character

• Synchronization with optical

mocap – 3D camera

87

Technologies

88

 HTC Vive tracker  3D camera by Photoneo Motion capture glove by

Synertial

Process

 Kinexact

Hand (calibration tool)

 Optical + inertial data synchronization  Datasets recording  Merging BVH files Updating structures Inertial data streamer Attaching hands to full- body animation

89

 Calibration of the

performer‘s hand skeleton

 C++  Update –

compatibility with new structures (representing skeleton – new organization of the skeletal nodes)

Kinexact Hand – calibration tool

root node in-hand nodes

90

Recording datasets

Glove setup

91

Glove + Vive tracker

data visualization

92

Optical + inertial data

Optical data

PhoXi 3D camera by Photoneo

Inertial data

Motion capture glove by Synertial

PhoXi Streamer app

C++

added glove data streaming

Pointcloud + skeleton + BVH

In each frame

skeleton format ?

93

Optical + inertial data

In each frame generating

• model pointcloud
• skeleton pointcloud

+ BVH file (the entire dataset)

94

Optical + inertial data

Reality

MoCap glove

60 fps

PhoXi 3D camera

• ca. 10 fps

(relative to PC performance)

Synchronization not stable

=> Potential future work

95

Captured by PhoXi 3D camera by PhotoNeo

Pointcloud generation

96

Before & after

attaching hands to body skeleton

97

98

Acknowledgements

 Thanks to all the people, whose work is shown here and whose

slides were used as a material for creation of these slides:

Matej Novotný, GSVM lectures at FMFI UK Peter Drahoš, PPGSO lectures at FIIT STU Output of all the publications and great team work Very best data from 3D cameras

99

www.skeletex.xyz madaras@skeletex.xyz

Questions ?!