Creative AI Combining Knowledge, Learning and Control for - - PowerPoint PPT Presentation

Creative AI

Combining Knowledge, Learning and Control for Expressive Modeling & Animation

Marie-Paule Cani

Ecole Polytechnique Paris, France

Visual representations

Mandatory to understand and create!

“We should think about graphic designs as cognitive tools, enhancing and extending our brains.” Colin Ware, Visual Thinking for Design, 2008 Leonardo da Vinci @ Renaud Chabrier

3D contents creation: Computer Graphics Interactive modeling… A failure?

3D modeling software

Editing DOFs of complex models Only usable by trained artists Refrains direct design !

Example: use for other sciences

• Vision from a scientist
• Explained to an artist…
• Multiple trials and errors!

Pre-created contents. The scientist cannot interact with them !

In this talk : Creative AI More expressive ways to model & animate?

A revolution of digital content creation

• 1. Gesture-based creation in 3D
• 2. Interactive models embedding knowledge & learning
• 3. Extension to animated virtual worlds

From mental visions to 3D, for general users?

Creative AI

Principles : gestural control + knowledge & learning

• Example: 3D shapes from a sketch

1. Expressive modeling principles 2. Extension to Virtual worlds 3. Animation

Without knowledge Interpret 2D shapes to 3D With knowledge Can we create a tree in a few gestures?

[Bernhardt 2008]

Creative AI Example: desiging a tree

Principle : Combining gestural control, knowledge & learning Inspiration → build on multi-resolution sketches

• Add knowledge & learning : perception, biology, statistics

1. Expressive modeling principles 2. Extension to Virtual worlds 3. Animation

Sketched Sketched Non Sketched [Prusinkiewicz et al., 01]

Solution

• Structure from silhouette!
• Use rules from biology, perception, statistics to :

– Infer plausible sub-structures – Duplicate them – Extend branches to 3D

Expressive modeling

1. Expressive modeling principles 2. Extension to Virtual worlds 3. Animation

Expressive modeling

Results

Eucalyptus

1. Expressive modeling principles 2. Extension to Virtual worlds 3. Animation

[Wither et al, EG 2009]

Expressive modeling

Gestures + knowledge

• Sculpt a castle as if it was clay?

[Milliez 2013]

Sculpting gestures

• Modeling virtual clay

[Kry 2008] 1. Expressive modeling principles 2. Extension to Virtual worlds 3. Animation

Embedding knowledge

Sculpting Structured Shapes

Man-made shapes

• Detected self similarities
• Local symmetries

a

b

c Replace a / d

d

Puzzle shape grammar

1. Expressive modeling principles 2. Extension to Virtual worlds 3. Animation

Embedding knowledge

Sculpting Structured Shapes

Solution Mutable elastic models

• Energy minimization
• Rules applied on the fly

1. Expressive modeling principles 2. Extension to Virtual worlds 3. Animation

[Milliez et al, EG 2013]

Expressive modeling

Extension to full virtual worlds?

Lots of elements + rules to be maintained

 Shapes: laws from biology, geology, statics  Motion: dynamic laws, mass preservation

Three challenges

• Matching rules with providing control
• Creating distributions of elements
• Expressive design of animated contents

1. Expressive modeling principles 2. Extension to Virtual worlds 3. Animation

Gestures + Control principle

Sculpting mountains

Could we sculpt mountains as if they were clay? – Constant volume – Folds, various wavelengths – Erosion & growth Volumetric earth-crust model

• A layered model coupling these phenomena

(uplift + erosion)

1. Expressive modeling principles 2. Extension to Virtual worlds 3. Animation

Sculpting Mountains

Sculpting interface Visible soil layers on eroded cliffs

[Cordonnier, IEEE TVGC 2018]

1. Expressive modeling principles 2. Extension to Virtual worlds 3. Animation

Designing networks of rivers & falls

Challenge

• Water flow uniquely depends from the terrain

Can we combine consistency & control ?

Sketching mountains… too indirect to control waterfalls !

1. Expressive modeling principles 2. Extension to Virtual worlds 3. Animation

Insight

Leave waterfalls sculpt the terrain!

Principle 2: Interleave user control & rule-based generation

• 1. The user sketches a network
• 2. Consistent flows are computed
• 3. The user selects a refinement type
• 4. The terrain deforms & details are added

1. Expressive modeling principles 2. Extension to Virtual worlds 3. Animation

Designing waterfall scenes

Validation Iron hole falls La réunion

[Emilien Poulin Cani, CGF 2015]

1. Expressive modeling principles 2. Extension to Virtual worlds 3. Animation

Populating virtual worlds ?

Distributions of vegetation & rocks

Principle 3 : Learning from user-specified examples Color = {Statistics on distributions of objects} (trees, stones …)  Learnt from a user-defined exemplar  Correlated with slope  Stored in a « palette » A variety of tools Pipette, brush, deform, gradient….

Exemplar r r

1. Expressive modeling principles 2. Extension to Virtual worlds 3. Animation

Learning and painting distributions

[Emilien et al. SIGGRAPH 2015]

Challenges

• Small

examples

• Interpolating

distributions

1. Smart models for shapes 2. Extension to Virtual worlds 3. Animation

Realistic Ecosystems? Learning from Simulation results

Idea: Combine simulation with world-brush – Multi-dimensional terrain clustering – Sand-box simulations for each cluster – Learn statistics – Synthesis : Semantic brushes: age, density…

[Gain et al. Eurographics 2017] 1. Smart models for shapes 2. Extension to Virtual worlds 3. Animation

Challenge Learning disc distributions!

Challenges

• Position and canopy size are correlated
• Overlapping behaviors are to be learnt!

Our solution: A new, normalized metric for disks

• Distinguishing disjoint, tangent, overlapping, nested disks

100x100m per cluster

Pair correlation functions 1. Smart models for shapes 2. Extension to Virtual worlds 3. Animation

Challenge Learning disc distributions!

Pair correlation functions

[Ecormier-Nocca, Eurographics 2019]

1. Smart models for shapes 2. Extension to Virtual worlds 3. Animation

Input Output

Animated virtual worlds

Expressive design of animations ?

Waterfalls : stationary motion only!

1. Smart models for shapes 2. Extension to Virtual worlds 3. Animation

Individual motion design?

Motion design only available to trained artists (@ E. Charleroy)

1. Smart models for shapes 2. Virtual worlds 3. Extension to animation

Expressive methods to pose characters

Line of action  Expressive C or S shapes  Aligned in position and/or orientation Posing a character in a single gesture

• LOA interpreted as a projective constraint

@The Estate of Preston Blair 1. Smart models for shapes 2. Extension to Virtual worlds 3. Animation

[Guay et al, SA 2013]

Could we “Sculpt” Motion?

Fast creation + progressive refinement

Keyframing is user intensive!

• Many key-frames needed
• Not easy to tune timing

Dynamic lines of action

• Defined in a single gesture?
• Enabling to control motion rhythm as well ?

Inspiration

1. Smart models for shapes 2. Extension to Virtual worlds 3. Animation

Space-time sketching of character motion

[Guay et al, SIGGRAPH 2015]

Group motion? Standard methods do not ease authoring!

• Pre-computed clips for individual motion
• “Steering behaviors”: Particles obeying interaction rules

→ No direct control of group shape, distribution & motion

Trading port Malaysia, 1800, British empire.

V

B

V

A1

V

A2

[Lim et al, Digital Heritage 2013]

1. Smart models for shapes 2. Extension to Virtual worlds 3. Animation

Populating virtual worlds

New multi-scale approach

Idea : Example-based design of herd animation

• Key-frame herd motion from photographs
• Learn herd distribution, density map, orientation field

1. Smart models for shapes 2. Extension to Virtual worlds 3. Animation

• Input photos
• Used with any
• nb. of animals !

Populating virtual worlds

New multi-scale approach

[Ecormier-Nocca et al. CASA 2019]

1. Smart models for shapes 2. Extension to Virtual worlds 3. Animation

Conclusion Creative AI vs. an AI that creates

AI systems are able to learn & combine preexisting contents

• Is this what we want?

Creative AI: Build on AI to make humans more creative

• Control to the user & Smart models to help

– Interpreting gestures – Duplicating details – Maintaining constraints

• Knowledge & light examples (added on the fly !)

[Liu & al 2015]

Deep learning for Creative AI?

Learning how to interact better with human users!

First example : Group Editing [Lun et al, Siggraph Asia 2017]

Element shape Structure of layout