Towards a Virtual Stuntman Xue Bin (Jason) Peng UC Berkeley - - PowerPoint PPT Presentation

▶
towards a virtual stuntman
SMART_READER_LITE
LIVE PREVIEW

Towards a Virtual Stuntman Xue Bin (Jason) Peng UC Berkeley - - PowerPoint PPT Presentation

Oct 10, 2023 663 likes •1.82k views

Towards a Virtual Stuntman Xue Bin (Jason) Peng UC Berkeley Animation Animation Computer Animation [Geijtenbeek et al. 2013] [Brown et al. 2013] [Ju et al. 2013] [Tan et al. 2014] [Kwon and Hodgins 2017] [Peng et al. 2018] Physics-Based

slide-1
SLIDE 1

Towards a Virtual Stuntman

Xue Bin (Jason) Peng UC Berkeley

slide-2
SLIDE 2

Animation

slide-3
SLIDE 3

Animation

slide-4
SLIDE 4

Computer Animation

[Ju et al. 2013] [Brown et al. 2013] [Tan et al. 2014] [Geijtenbeek et al. 2013] [Kwon and Hodgins 2017] [Peng et al. 2018]

slide-5
SLIDE 5

Physics-Based Animation

[Coros et al. 2011]

slide-6
SLIDE 6

Deep RL

[Schulman et al. 2016] [Mnih et al. 2015] [Chebotar et al. 2017]

slide-7
SLIDE 7

Motion Quality

[Heess et al. 2017] [Schulman et al. 2016]

slide-8
SLIDE 8

Motivation

slide-9
SLIDE 9

Which is Mocap?

slide-10
SLIDE 10

Simulation Mocap

Which is Mocap?

slide-11
SLIDE 11

Overview

+ +

Character Reference Motion Task: Hit Target

slide-12
SLIDE 12

Overview

slide-13
SLIDE 13

Overview

slide-14
SLIDE 14

Overview

slide-15
SLIDE 15

Reference Motion

slide-16
SLIDE 16

Reference Motion

slide-17
SLIDE 17

Reference Motion

slide-18
SLIDE 18

Reference Motion

slide-19
SLIDE 19

Reference Motion

slide-20
SLIDE 20

Reference Motion

slide-21
SLIDE 21

Reference Motion

𝑏0 𝑏1 𝑏2 𝑏3 𝑏4

slide-22
SLIDE 22

State + Action

State:

  • link positions
  • link velocities
slide-23
SLIDE 23

State + Action

State:

  • link positions
  • link velocities
slide-24
SLIDE 24

State + Action

State:

  • link positions
  • link velocities

Action:

  • PD targets
slide-25
SLIDE 25

State + Action

State:

  • link positions
  • link velocities

Action:

  • PD targets
slide-26
SLIDE 26

Reward

slide-27
SLIDE 27

Reward

Imitation Objective

slide-28
SLIDE 28

Reward

Imitation Objective

slide-29
SLIDE 29

Reward

Imitation Objective

slide-30
SLIDE 30

Reward

Imitation Objective

slide-31
SLIDE 31

Reward

Imitation Objective Task Objective

slide-32
SLIDE 32

Proximal Policy Optimization (PPO)

[Schulman et al. 2017]

slide-33
SLIDE 33

Proximal Policy Optimization (PPO)

[Schulman et al. 2017]

slide-34
SLIDE 34

Proximal Policy Optimization (PPO)

[Schulman et al. 2017]

slide-35
SLIDE 35

Proximal Policy Optimization (PPO)

[Schulman et al. 2017]

slide-36
SLIDE 36

Proximal Policy Optimization (PPO)

[Schulman et al. 2017]

slide-37
SLIDE 37

Humanoid: Walk

slide-38
SLIDE 38

Humanoid: Run

slide-39
SLIDE 39

Comparison

[Merel et al. 2017] Ours

slide-40
SLIDE 40

No Reference Motion

slide-41
SLIDE 41

Locomotion

slide-42
SLIDE 42

Humanoid: Cartwheel

slide-43
SLIDE 43

Humanoid: Backflip

slide-44
SLIDE 44

Humanoid: Frontflip

slide-45
SLIDE 45

Humanoid: Roll

slide-46
SLIDE 46

Humanoid: Crawl

slide-47
SLIDE 47

Humanoid: Dance A

slide-48
SLIDE 48

Humanoid: Kip-Up

slide-49
SLIDE 49

Humanoid: Headspin

slide-50
SLIDE 50

Humanoid: Vault 1-Handed

slide-51
SLIDE 51

Humanoid: Flare

slide-52
SLIDE 52

20+ Skills

slide-53
SLIDE 53

Keyframe Animation

slide-54
SLIDE 54

T-Rex: Walk

Simulated Character

slide-55
SLIDE 55

Dragon: Walk

Simulated Character

slide-56
SLIDE 56

Lion: Run

Simulated Character

slide-57
SLIDE 57

Lion++

slide-58
SLIDE 58

Tasks

slide-59
SLIDE 59

Tasks

Reference Motion

slide-60
SLIDE 60

Tasks

Reference Motion

+

Task

slide-61
SLIDE 61

Humanoid: Spinkick - Strike

slide-62
SLIDE 62

Humanoid: Baseball Pitch - Throw

slide-63
SLIDE 63

No Reference Motion

slide-64
SLIDE 64

Humanoid: Balance Beam

slide-65
SLIDE 65

Humanoid: Run – Dense Gaps

slide-66
SLIDE 66

Retargeting

slide-67
SLIDE 67

Character Retargeting

+

Reference Motion Atlas

slide-68
SLIDE 68

Atlas: Spinkick

slide-69
SLIDE 69

Atlas: Run

slide-70
SLIDE 70

Atlas: Getup-Facedown

slide-71
SLIDE 71

Atlas: Backflip

slide-72
SLIDE 72

Multi-Clip Integration

slide-73
SLIDE 73

Multi-Clip Integration

slide-74
SLIDE 74

Multi-Clip Integration

slide-75
SLIDE 75

Learning from Mocap

slide-76
SLIDE 76

Mocap is a Hassle

[Holden 2018]

slide-77
SLIDE 77

Skills From Videos

slide-78
SLIDE 78

Simulation

Learning from Videos

Video Simulation

slide-79
SLIDE 79

Overview

slide-80
SLIDE 80

Overview

slide-81
SLIDE 81

Overview

slide-82
SLIDE 82

Overview

slide-83
SLIDE 83

Overview

slide-84
SLIDE 84

Pose Estimation

slide-85
SLIDE 85

Pose Estimation

Pose Prediction Video: Handspring A

slide-86
SLIDE 86

Human Mesh Recovery (HMR)

[Kanazawa et al., 2018]

slide-87
SLIDE 87

Pose Estimation

Pose Prediction Video: Backflip A

slide-88
SLIDE 88

Motion Imitation

slide-89
SLIDE 89

Motion Imitation

slide-90
SLIDE 90

Motion Imitation

slide-91
SLIDE 91

Motion Imitation

slide-92
SLIDE 92

Humanoid: Cartwheel B

Video: Cartwheel B Reference Motion Policy

slide-93
SLIDE 93

Humanoid: Jumping Jack

Video: Jumping Jack Policy

slide-94
SLIDE 94

Humanoid: Backflip B

Video: Backflip B Policy

slide-95
SLIDE 95

Humanoid: Frontflip

Video: Frontflip Policy

slide-96
SLIDE 96

Humanoid: Roll

Video: Roll Policy

slide-97
SLIDE 97

Humanoid: Spin

Video: Spin Policy

slide-98
SLIDE 98

Humanoid: Kip-Up

Video: Kip-Up Policy

slide-99
SLIDE 99

Humanoid: Vault

Video: Vault Policy

slide-100
SLIDE 100

Atlas: Handspring A

Video: Handspring A Policy

slide-101
SLIDE 101

Atlas: Jump

Video: Jump Policy

slide-102
SLIDE 102

Atlas: Vault

Video: Vault Policy

slide-103
SLIDE 103

Atlas: Dance

Video: Dance Policy

slide-104
SLIDE 104

Environment Retargeting

slide-105
SLIDE 105

Environment Retargeting

slide-106
SLIDE 106

Failure Cases

slide-107
SLIDE 107
slide-108
SLIDE 108

Skills from Videos

Policy Video: Backflip A

slide-109
SLIDE 109

Concluding Remarks

  • Simple method can learn a large repertoire of skills
  • Minimizing tracking error works (surprisingly) well
slide-110
SLIDE 110

Concluding Remarks

  • Simple method can learn a large repertoire of skills
  • Minimizing tracking error works (surprisingly) well
  • A lot of room for improvement for video imitation

– More end-to-end approach – Outdoor sports – Multiple actors

slide-111
SLIDE 111

Concluding Remarks

  • Simple method can learn a large repertoire of skills
  • Minimizing tracking error works (surprisingly) well
  • A lot of room for improvement for video imitation

– More end-to-end approach – Outdoor sports – Multiple actors

  • Code: https://github.com/xbpeng/DeepMimic
slide-112
SLIDE 112

Collaborators

Pieter Abbeel Sergey Levine Michiel van de Panne Angjoo Kanazawa Jitendra Malik

slide-113
SLIDE 113

Questions?