Simulation & Animation Iasonas Niloy Mitra Paul Guerrero - - PowerPoint PPT Presentation

▶
simulation animation
SMART_READER_LITE
LIVE PREVIEW

Simulation & Animation Iasonas Niloy Mitra Paul Guerrero - - PowerPoint PPT Presentation

Mar 19, 2023 240 likes •770 views

Simulation & Animation Iasonas Niloy Mitra Paul Guerrero Vladimir Kim Nils Thuerey Leonidas Guibas Kokkinos UCL/ UCL/ UCL/ Stanford/ Adobe TUM Adobe Ariel AI Adobe FAIR Timetable Niloy Iasonas Paul Nils Leonidas

slide-1
SLIDE 1

Niloy Mitra Iasonas Kokkinos Paul Guerrero Vladimir Kim Nils Thuerey Leonidas Guibas UCL/ Adobe UCL/ 
 Ariel AI UCL/
 Adobe Adobe TUM Stanford/
 FAIR

Simulation & Animation

slide-2
SLIDE 2

CreativeAI: Deep Learning for Graphics

Timetable

2

Niloy Iasonas Paul Nils Leonidas Introduction 9:00 X Neural Network Basics ~9:15 X Supervised Learning in CG ~9:50 X Unsupervised Learning in CG ~10:20 X Learning on Unstructured Data ~10:55 X Learning for Animation ~11:35 X Discussion 12:05 X X X X X

slide-3
SLIDE 3

CreativeAI: Deep Learning for Graphics

Computer Animation

3

  • Feature detection (image features, point features)

  • Denoising, Smoothing, etc. 

  • Embedding, Distance computation

  • Rendering

  • Animation

  • Physical simulation

  • Motion over time
  • Lots of data - expensive…
  • Relationships between spatial

and temporal changes

slide-4
SLIDE 4

CreativeAI: Deep Learning for Graphics

  • Target character rigs
  • Natural reactions and transitions
  • Reinforcement Learning

Character Animation

4

slide-5
SLIDE 5

CreativeAI: Deep Learning for Graphics

Physics-Based Animation

  • Leverage physical models
  • Examples:
  • Rigid bodies
  • Cloth
  • Deformable objects
  • Fluids

5

slide-6
SLIDE 6

CreativeAI: Deep Learning for Graphics

Character Animation

6

slide-7
SLIDE 7

CreativeAI: Deep Learning for Graphics

Existing Approaches

  • Motion Representations
  • Controllers

7

slide-8
SLIDE 8

CreativeAI: Deep Learning for Graphics 8

Learned Motion Manifolds

[Phase-functioned neural networks for character control, SIGGRAPH 2017] [Learning Motion Manifolds with Convolutional Autoencoders, SGA 2015 Tech. Briefs]

slide-9
SLIDE 9

Learned Motion Manifolds

9

[Phase-functioned neural networks for character control, SIGGRAPH 2017]

slide-10
SLIDE 10

CreativeAI: Deep Learning for Graphics 10

Reinforcement Learning

  • Goal: maximize reward by performing actions in an environment

Observations Reward Action

slide-11
SLIDE 11
  • Learn Controllers that steer character rigs
  • Smooth and natural transitions
  • Reactions to changes in the environment

RL for Animation

[DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills, SIGGRAPH 2018]

11

[Terrain-Adaptive Locomotion Skills Using Deep Reinforcement Learning, SIGGRAPH 2016]

slide-12
SLIDE 12

Reinforcement Learning

12

[DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills, SIGGRAPH 2018]

slide-13
SLIDE 13

CreativeAI: Deep Learning for Graphics

Physics-Based Animation

13

slide-14
SLIDE 14

CreativeAI: Deep Learning for Graphics

Physics-Based Animation

14

Experiment Theory Computation

Skip Theory with Deep Learning?

[No! More on that later…]

Observations / data Model equations Discrete representation

slide-15
SLIDE 15

CreativeAI: Deep Learning for Graphics

Physics-Based Animation

  • Better goal: support solving suitable physical models
  • Nature = Partial Differential Equations (PDEs)
  • Hence can aim for solving PDEs with deep learning (DL)
  • Requirement: “regularity” of the targeted function

15

“Bypass the solving of evolution equations when these equations conceptually exist but are not available or known in closed form.” [Kevrekidis et al.]

slide-16
SLIDE 16

CreativeAI: Deep Learning for Graphics

Physics-Based Deep Learning

16

Machine Learning Physics Numerical Methods

Combined Solvers Physics-based Optimization Physics-based Design Physics-based Constraints

Physics Applications based on DL

slide-17
SLIDE 17

CreativeAI: Deep Learning for Graphics

Partial Differential Equations

  • Typical problem formulation: unknown

function

  • PDE of the general form:
  • Solve in domain , with boundary

conditions on boundary

  • Traditionally: discretize & solve numerically.

Here: also discretize, but solve with DL…

17

f ⇣ x1, ..., xn; ∂u ∂x1 , ..., ∂u ∂xn ; ∂2u ∂2x1 , ∂2u ∂x1∂x2 , ... ⌘ = 0 u(x1, ..., xn)

Ω

Ω Γ

Γ

slide-18
SLIDE 18

CreativeAI: Deep Learning for Graphics

Methodology 1

  • Viewpoints: holistic or partial

[partial also meaning “coarse graining” or “sub-grid / up-res”]

  • Influences complexity and non-linearity of solution space
  • Trade off computation vs accuracy:
  • Target most costly parts of solving
  • Often at the expense of accuracy

18

slide-19
SLIDE 19

CreativeAI: Deep Learning for Graphics

Methodology 2

19

  • Consider dimensionality & structure of discretization
  • Small & unstructured
  • Fully connected NNs only choice
  • Only if necessary…
  • Large & structured
  • Employ convolutional NNs
  • Usually well suited
slide-20
SLIDE 20

CreativeAI: Deep Learning for Graphics

Solving PDEs with DL

  • Practical example: airfoil flow
  • Given boundary conditions solve stationary flow problem on grid
  • Fully replace traditional solver
  • 2D data, no time dimension
  • I.e., holistic approach with structured data

20

slide-21
SLIDE 21

CreativeAI: Deep Learning for Graphics

Solving PDEs with DL

  • Data generation
  • Large number of pairs: input (BCs) - targets (solutions)

21 Airfoil profile Generated mesh Full simulation domain

Inference region

Different free stream Velocities

slide-22
SLIDE 22

CreativeAI: Deep Learning for Graphics

Solving PDEs with DL

  • Data generation
  • Example pair
  • Note - boundary conditions (i.e.

input fields) are typically constant

  • Rasterized airfoil shape present

in all three input fields

22

Target

Pressure Velocity X Velocity Y

128 x 128 x 1 128 x 128 x 1 128 x 128 x 1

Freestream X

Boundary Conditions

Freestream Y Mask

128 x 128 x 1 128 x 128 x 1 128 x 128 x 1

slide-23
SLIDE 23

CreativeAI: Deep Learning for Graphics

Solving PDEs with DL

23

Reduce spatial dimensions Skip connections Increase spatial dimensions

  • U-net NN architecture
slide-24
SLIDE 24

CreativeAI: Deep Learning for Graphics

Solving PDEs with DL

  • Unet structure highly suitable for PDE solving
  • Makes boundary condition information available throughout
  • Crucial for inference of solution

24

  • U-net NN architecture
slide-25
SLIDE 25

CreativeAI: Deep Learning for Graphics

Solving PDEs with DL

  • Training: 80.000 iterations with ADAM optimizer
  • Convolutions with enough data - no dropout necessary
  • Learning rate decay stabilizes models

25

slide-26
SLIDE 26

CreativeAI: Deep Learning for Graphics

Results

26

Target (A) Regular data (B) Dimension less Pressure Velocity X Velocity Y

  • Use knowledge about

physics to simplify space of solutions: make quantities dimension- less

  • Significant gains in

inference accuracy

slide-27
SLIDE 27

CreativeAI: Deep Learning for Graphics

Solving PDEs with DL

  • Validation and test accuracy for different model sizes

27

Saturated, little gain from weights and data

slide-28
SLIDE 28

Code example

Solving PDEs with DL

28

slide-29
SLIDE 29

CreativeAI: Deep Learning for Graphics

Existing Approaches

  • Elasticity
  • Cloth
  • Fluids

29

slide-30
SLIDE 30
  • Learn correction of regular FEM simulation for complex materials
  • Numerical simulation with flexible NN for material behavior

Neural Material - Elasticity

30

[Neural Material: Learning Elastic Constitutive Material and Damping Models from Sparse Data, arXiv 2018] [NNWarp: Neural Network-based Nonlinear Deformation, TVCG 2018]

slide-31
SLIDE 31

CreativeAI: Deep Learning for Graphics

Neural Material - Elasticity

  • Learn correction of regular FEM simulation for complex materials

31

[Neural Material: Learning Elastic Constitutive Material and Damping Models from Sparse Data, arXiv 2018]

slide-32
SLIDE 32
  • Learn flexible reduced representation for physics problems

Latent Spaces

32

[Latent-space Physics: Towards Learning the Temporal Evolution of Fluid Flow, EG 2019] [Deep Fluids: A Generative Network for Parameterized Fluid Simulations, EG 2019]

slide-33
SLIDE 33
  • Learn flexible reduced representation for physics problems
  • Employ Encoder part (E) of Autoencoder network to reduce dimensions
  • Predict future state in latent space with FC network
  • Use Decoder (D) of Autoencoder to retrieve volume data

Latent Spaces

33

E

t t-1 t-2 . . .

E E FC

t+1

D

[Latent-space Physics: Towards Learning the Temporal Evolution of Fluid Flow, EG 2019] [Deep Fluids: A Generative Network for Parameterized Fluid Simulations, EG 2019]

slide-34
SLIDE 34

CreativeAI: Deep Learning for Graphics

Latent Spaces

34

  • Learn flexible reduced representation for physics problems

[Latent-space Physics: Towards Learning the Temporal Evolution of Fluid Flow, EG 2019] [Deep Fluids: A Generative Network for Parameterized Fluid Simulations, EG 2019]

slide-35
SLIDE 35

CreativeAI: Deep Learning for Graphics

Latent Spaces

35

  • In combination with Reinforcement Learning

[Fluid Directed Rigid Body Control using Deep Reinforcement Learning, SIGGRAPH 2018]

slide-36
SLIDE 36

CreativeAI: Deep Learning for Graphics 36

[Latent-space Dynamics for Reduced Deformable Simulation, EG 2019]

Latent Spaces

  • For elasticity problems
slide-37
SLIDE 37

CreativeAI: Deep Learning for Graphics

Latent Spaces

37

  • For elasticity problems

[Latent-space Dynamics for Reduced Deformable Simulation, EG 2019]

slide-38
SLIDE 38

CreativeAI: Deep Learning for Graphics 38

[Learning-Based Animation of Clothing for Virtual Try-On, EG 2019]

Latent Spaces

  • For cloth (adaptation to different body shapes)
slide-39
SLIDE 39
  • Generative model for 3D plus time
  • Input domain: low resolution 3D volumes
  • Output: high-resolution 3D volumes
  • Auxiliary goal: match temporal evolution of target domain (high-res. data)

Temporal Data

39

[tempoGAN: A Temporally Coherent, Volumetric GAN for Super-resolution Fluid Flow , SIGGRAPH 2018]

slide-40
SLIDE 40

Temporal Data

40

[tempoGAN: A Temporally Coherent, Volumetric GAN for Super-resolution Fluid Flow , SIGGRAPH 2018]

G

xa

slide-41
SLIDE 41

Temporal Data

41

[tempoGAN: A Temporally Coherent, Volumetric GAN for Super-resolution Fluid Flow , SIGGRAPH 2018]

G D_s

ya

G(xa) G(x ) G(x ) G(x )

xa

slide-42
SLIDE 42

42

[tempoGAN: A Temporally Coherent, Volumetric GAN for Super-resolution Fluid Flow , SIGGRAPH 2018]

G D_s D_t

xa xt-1 xt xt+1 ya yt-1 yt yt+1

G(xa) G(xt-1) G(xt) G(xt+1)

Temporal supervision

Temporal Data

slide-43
SLIDE 43

43

[tempoGAN: A Temporally Coherent, Volumetric GAN for Super-resolution Fluid Flow , SIGGRAPH 2018]

G D_s D_t

xa xt-1 xt xt+1 ya yt-1 yt yt+1

G(xa) G(xt-1) G(xt) G(xt+1)

Temporal Data

Discretized advection operator included in loss for G

slide-44
SLIDE 44

44

[tempoGAN: A Temporally Coherent, Volumetric GAN for Super-resolution Fluid Flow , SIGGRAPH 2018]

G D_s D_t

xa xt-1 xt xt+1 ya yt-1 yt yt+1

G(xa) G(xt-1) G(xt) G(xt+1)

Temporal Data

slide-45
SLIDE 45

Temporal Data

45

[tempoGAN: A Temporally Coherent, Volumetric GAN for Super-resolution Fluid Flow , SIGGRAPH 2018]

Low-res Input Result

slide-46
SLIDE 46

Temporal Data

46

[tempoGAN: A Temporally Coherent, Volumetric GAN for Super-resolution Fluid Flow , SIGGRAPH 2018]

Replay

slide-47
SLIDE 47

SIGGRAPH Asia Course CreativeAI: Deep Learning for Graphics

  • Checklist for solving PDEs with DL:

✓ Model? (Typically given) ✓ Data? Can enough training data be generated? ✓ Which NN Architecture? ✓ Fine tuning: learning rate, number of layers & features? ✓ Hyper-parameters, activation functions etc.?

Summary

47

slide-48
SLIDE 48

SIGGRAPH Asia Course CreativeAI: Deep Learning for Graphics

  • Approach PDE solving with DL like solving with traditional numerical

methods:

  • Find closest example in literature
  • Reproduce & test
  • Then vary, adjust, refine …
  • Main change: Data pipeline

Summary

48

slide-49
SLIDE 49

SIGGRAPH Asia Course CreativeAI: Deep Learning for Graphics 49

slide-50
SLIDE 50

SIGGRAPH Asia Course CreativeAI: Deep Learning for Graphics

  • DL provides a powerful computational tool
  • Open challenges:
  • Theoretical guarantees
  • Ethical questions
  • “Next level” of representation learning

50

Deep Learning - Outlook

slide-51
SLIDE 51

CreativeAI: Deep Learning for Graphics 51

Course Information (slides/code/comments)

http://geometry.cs.ucl.ac.uk/creativeai/

The End - Thank you!