Learning Character-Agnostic Motion for Motion Retargeting in 2D - - PowerPoint PPT Presentation

▶
learning character agnostic motion for motion retargeting
SMART_READER_LITE
LIVE PREVIEW

Learning Character-Agnostic Motion for Motion Retargeting in 2D - - PowerPoint PPT Presentation

Dec 25, 2022 119 likes •468 views

Learning Character-Agnostic Motion for Motion Retargeting in 2D Kfir Aberman, Rundi Wu, Dani Lischinski, Baoquan Chen, Daniel Cohen-Or Outline - Motivation - Approach - Results - Application Outline - Motivation - Approach - Results -

slide-1
SLIDE 1

Learning Character-Agnostic Motion for Motion Retargeting in 2D

Kfir Aberman, Rundi Wu, Dani Lischinski, Baoquan Chen, Daniel Cohen-Or

slide-2
SLIDE 2

Outline

  • Motivation
  • Approach
  • Results
  • Application
slide-3
SLIDE 3

Outline

  • Motivation
  • Approach
  • Results
  • Application
slide-4
SLIDE 4

Motion Retargeting in 3D

slide-5
SLIDE 5

Related Work

[Villegas et.al., 2018] [Aristidou et.al., 2018] [Gleicher et. al., 1998]

slide-6
SLIDE 6

Motivation

slide-7
SLIDE 7

Motion Retargeting in 2D

Character Agnostic Motion Skeleton View- Angle

slide-8
SLIDE 8

Outline

  • Motivation
  • Approach
  • Results
  • Application
slide-9
SLIDE 9

Approach

Estimated Camera Parameters

3D motion Retargeting

Motion Skeleton

2D 3D 3D 2D

Output Video Source Video Target Video Source Video Target Video Output Video

Character Agnostic Motion Static Parameters

slide-10
SLIDE 10

Architecture

T

Dynamic latent space Static latent space Concat

Tile

ˆ pi,j

pi,j

T

∝ T

6/ T

EM

ES

D

2J

Lrec = Epi,j∼P ⇥ kD(EM(pi,j), ES(pi,j)) pi,jk2⇤ . (pi,j) ˆ pi,j

slide-11
SLIDE 11

Architecture

T 8

T

2J

pi,j

1

EM Es

Global Pooling

T 2 T 2

T 8

T 8 T 4 T 4

1

T 2J

EM

ES

slide-12
SLIDE 12

Decompose and Re-compose

Epi,j,pk,l∼P×P ⇥ kD(EM(pi,j), ES(pk,l)) pi,lk2⇤ Epi,j,pk,l∼P×P ⇥ kD(EM(pk,l), ES(pi,j)) pk,jk2⇤ Lcross = +

slide-13
SLIDE 13

Synthetic Data

slide-14
SLIDE 14

Synthetic Data

slide-15
SLIDE 15

Learning Clusters Implicitly

−90 −60 −30 30 60 90

Skeleton Latent Space View-Angle Latent Space

Lrec + λLcross

slide-16
SLIDE 16

Implicitl Clusters Learning

−90 −60 −30 30 60 90

Motion Latent Space Motion Latent Space - View Angle labels

slide-17
SLIDE 17

Triplet Loss

Motion Latent Space Without Triplet loss Motion Latent Space With Triplet loss

slide-18
SLIDE 18

Foot Velocity Loss

pi,j

2J

T 2J

Global Velocity Root centered positions

slide-19
SLIDE 19

Supporting Videos in the wild

Augmentation (Temporal trimming, flips, rotation, scale) Adding noise to the training data Reconstruct real videos using (only) the reconstruction loss.

slide-20
SLIDE 20

Outline

  • Motivation
  • Approach
  • Results
  • Application
slide-21
SLIDE 21

Results-skeleton

slide-22
SLIDE 22

Results - view

slide-23
SLIDE 23

Interpolation

slide-24
SLIDE 24

Comparison

slide-25
SLIDE 25

Outline

  • Motivation
  • Approach
  • Results
  • Application
slide-26
SLIDE 26

Applications-performance cloning

slide-27
SLIDE 27

Applications-performance cloning

slide-28
SLIDE 28

Applications - Motion Retrieval

slide-29
SLIDE 29

Applications - Motion Retrieval

slide-30
SLIDE 30

Failure cases

slide-31
SLIDE 31

Failure cases

slide-32
SLIDE 32

Conclusions

Take home message:

Deep networks can constitute a better solution for specific sub-tasks, which do not strictly require a full 3D reconstruction. Synthetic data can really help with deep neural network training.

slide-33
SLIDE 33

Questions?