Learning Character-Agnostic Motion for Motion Retargeting in 2D - PowerPoint PPT Presentation

Learning Character-Agnostic Motion for Motion Retargeting in 2D Kfir Aberman, Rundi Wu, Dani Lischinski, Baoquan Chen, Daniel Cohen-Or

Outline - Motivation - Approach - Results - Application

Outline - Motivation - Approach - Results - Application

Motion Retargeting in 3D

Related Work [Gleicher et. al., 1998] [Aristidou et.al., 2018] [Villegas et.al., 2018]

Motivation

Motion Retargeting in 2D View- Angle Character Agnostic Motion Skeleton

Outline - Motivation - Approach - Results - Application

Approach Motion Source 3D motion 2D 3D Video 3D 2D Retargeting Skeleton Output Estimated Video Target Camera Video Parameters Character Agnostic Motion Source Video Output Static Video Parameters Target Video

Architecture ( p i,j ) ˆ p i,j Dynamic latent space p i,j ˆ E M p i,j ∝ T Concat D T T Static Tile latent space 2 J E S 6/ T k D ( E M ( p i,j ) , E S ( p i,j )) � p i,j k 2 ⇤ ⇥ L rec = E p i,j ∼ P .

Architecture p i,j T T T T 8 8 2 4 T E M E M T Global Pooling 1 T 1 T T 8 4 2 2 J 2 J E s E S

Decompose and Re-compose k D ( E M ( p i,j ) , E S ( p k,l )) � p i,l k 2 ⇤ ⇥ L cross = E p i,j , p k,l ∼ P × P k D ( E M ( p k,l ) , E S ( p i,j )) � p k,j k 2 ⇤ ⇥ + E p i,j , p k,l ∼ P × P

Synthetic Data

Synthetic Data

Learning Clusters Implicitly L rec + λ L cross Skeleton View-Angle Latent Space Latent Space 90 � 60 � 30 � 0 � − 30 � − 60 � − 90 �

Implicitl Clusters Learning Motion Motion Latent Space - Latent Space View Angle labels 90 � 60 � 30 � 0 � − 30 � − 60 � − 90 �

Triplet Loss Motion Motion Latent Space Latent Space With Triplet loss Without Triplet loss

Foot Velocity Loss Root Global centered Velocity positions p i,j T 2 J 2 J

Supporting Videos in the wild Augmentation (Temporal trimming, flips, rotation, scale) Adding noise to the training data Reconstruct real videos using (only) the reconstruction loss.

Outline - Motivation - Approach - Results - Application

Results-skeleton

Results - view

Interpolation

Comparison

Outline - Motivation - Approach - Results - Application

Applications-performance cloning

Applications-performance cloning

Applications - Motion Retrieval

Applications - Motion Retrieval

Failure cases

Failure cases

Conclusions Take home message: Deep networks can constitute a better solution for specific sub-tasks, which do not strictly require a full 3D reconstruction. Synthetic data can really help with deep neural network training.

Questions?