Creative AI Combining Knowledge, Learning and Control for - PowerPoint PPT Presentation

Creative AI Combining Knowledge, Learning and Control for Expressive Modeling & Animation Marie-Paule Cani Ecole Polytechnique Paris, France

Visual representations Mandatory to understand and create! @ Renaud Chabrier Leonardo da Vinci “ We should think about graphic designs as cognitive tools, enhancing and extending our brains .” Colin Ware, Visual Thinking for Design, 2008

3D contents creation: Computer Graphics Interactive modeling… A failure? 3D modeling software Editing DOFs of complex models Only usable by trained artists Refrains direct design ! Example: use for other sciences • Vision from a scientist • Explained to an artist… • Multiple trials and errors! Pre-created contents. The scientist cannot interact with them !

In this talk : Creative AI More expressive ways to model & animate? A revolution of digital content creation 1. Gesture-based creation in 3D 2. Interactive models embedding knowledge & learning 3. Extension to animated virtual worlds From mental visions to 3D, for general users?

1. Expressive modeling principles 2. Extension to Virtual worlds 3. Animation Creative AI Principles : gestural control + knowledge & learning • Example: 3D shapes from a sketch [Bernhardt 2008] With knowledge Without knowledge Can we create a tree in a few gestures? Interpret 2D shapes to 3D

1. Expressive modeling principles Creative AI 2. Extension to Virtual worlds 3. Animation Example: desiging a tree Principle : Combining gestural control, knowledge & learning Inspiration → build on multi -resolution sketches • Add knowledge & learning : perception, biology, statistics Non Sketched Sketched Sketched [Prusinkiewicz et al., 01]

1. Expressive modeling principles 2. Extension to Virtual worlds 3. Animation Expressive modeling Solution • Structure from silhouette! • Use rules from biology, perception, statistics to : – Infer plausible sub-structures – Duplicate them – Extend branches to 3D

1. Expressive modeling principles 2. Extension to Virtual worlds 3. Animation Expressive modeling Results Eucalyptus [Wither et al, EG 2009]

1. Expressive modeling principles Expressive modeling 2. Extension to Virtual worlds 3. Animation Gestures + knowledge Sculpting gestures • Modeling virtual clay • Sculpt a castle as if it was clay? [Kry 2008] [Milliez 2013]

1. Expressive modeling principles 2. Extension to Virtual worlds Embedding knowledge 3. Animation Sculpting Structured Shapes Man-made shapes • Detected self similarities • Local symmetries Puzzle shape b c a d grammar Replace a / d

1. Expressive modeling principles Embedding knowledge 2. Extension to Virtual worlds 3. Animation Sculpting Structured Shapes Solution Mutable elastic models • Energy minimization • Rules applied on the fly [Milliez et al, EG 2013]

1. Expressive modeling principles 2. Extension to Virtual worlds Expressive modeling 3. Animation Extension to full virtual worlds? Lots of elements + rules to be maintained  Shapes: laws from biology, geology, statics  Motion: dynamic laws, mass preservation Three challenges • Matching rules with providing control • Creating distributions of elements • Expressive design of animated contents

1. Expressive modeling principles Gestures + Control principle 2. Extension to Virtual worlds 3. Animation Sculpting mountains Could we sculpt mountains as if they were clay? – Constant volume – Folds, various wavelengths – Erosion & growth Volumetric earth-crust model • A layered model coupling these phenomena (uplift + erosion)

1. Expressive modeling principles 2. Extension to Virtual worlds Sculpting Mountains 3. Animation Sculpting interface Visible soil layers on eroded cliffs [Cordonnier, IEEE TVGC 2018]

1. Expressive modeling principles 2. Extension to Virtual worlds 3. Animation Designing networks of rivers & falls Challenge • Water flow uniquely depends from the terrain Can we combine consistency & control ? Sketching mountains… too indirect to control waterfalls !

1. Expressive modeling principles 2. Extension to Virtual worlds Insight 3. Animation Leave waterfalls sculpt the terrain! Principle 2: Interleave user control & rule-based generation 1. The user sketches a network 2. Consistent flows are computed 3. The user selects a refinement type 4. The terrain deforms & details are added

1. Expressive modeling principles Designing waterfall scenes 2. Extension to Virtual worlds 3. Animation Validation Iron hole falls La réunion [Emilien Poulin Cani, CGF 2015]

1. Expressive modeling principles Populating virtual worlds ? 2. Extension to Virtual worlds 3. Animation Distributions of vegetation & rocks Principle 3 : Learning from user-specified examples Color = {Statistics on distributions of objects } (trees, stones …)  Learnt from a user-defined exemplar Exemplar  Correlated with slope  Stored in a « palette » r r A variety of tools Pipette, brush, deform, gradient….

1. Smart models for shapes 2. Extension to Virtual worlds Learning and painting distributions 3. Animation Challenges • Small examples • Interpolating distributions [Emilien et al. SIGGRAPH 2015]

1. Smart models for shapes 2. Extension to Virtual worlds Realistic Ecosystems? 3. Animation Learning from Simulation results Idea: Combine simulation with world-brush – Multi-dimensional terrain clustering – Sand-box simulations for each cluster – Learn statistics – Synthesis : Semantic brushes: age, density… [Gain et al. Eurographics 2017]

1. Smart models for shapes Challenge 2. Extension to Virtual worlds 3. Animation Learning disc distributions! 100x100m per cluster Challenges • Position and canopy size are correlated • Overlapping behaviors are to be learnt! Our solution: A new, normalized metric for disks • Distinguishing disjoint, tangent, overlapping, nested disks Pair correlation functions

1. Smart models for shapes Challenge 2. Extension to Virtual worlds 3. Animation Learning disc distributions! Output Input [Ecormier-Nocca, Eurographics 2019] Pair correlation functions

1. Smart models for shapes 2. Extension to Virtual worlds Animated virtual worlds 3. Animation Expressive design of animations ? Waterfalls : stationary motion only!

1. Smart models for shapes 2. Virtual worlds Individual motion design? 3. Extension to animation Motion design only available to trained artists ( @ E. Charleroy)

1. Smart models for shapes 2. Extension to Virtual worlds 3. Animation Expressive methods to pose characters Line of action  Expressive C or S shapes  Aligned in position and/or orientation @The Estate of Preston Blair Posing a character in a single gesture • LOA interpreted as a projective constraint [Guay et al, SA 2013]

1. Smart models for shapes 2. Extension to Virtual worlds Could we “Sculpt” Motion? 3. Animation Fast creation + progressive refinement Keyframing is user intensive! • Many key-frames needed • Not easy to tune timing Dynamic lines of action • Defined in a single gesture? • Enabling to control motion rhythm as well ? Inspiration

Space-time sketching of character motion [Guay et al, SIGGRAPH 2015]

1. Smart models for shapes Group motion? 2. Extension to Virtual worlds 3. Animation Standard methods do not ease authoring! Trading port Malaysia, 1800, British empire. V B V V A1 A2 [Lim et al, Digital Heritage 2013] • Pre-computed clips for individual motion • “Steering behaviors”: Particles obeying interaction rules → No direct control of group shape, distribution & motion

1. Smart models for shapes 2. Extension to Virtual worlds Populating virtual worlds 3. Animation New multi-scale approach Idea : Example-based design of herd animation • Key-frame herd motion from photographs • Learn herd distribution, density map, orientation field • Input photos • Used with any nb. of animals !

1. Smart models for shapes 2. Extension to Virtual worlds Populating virtual worlds 3. Animation New multi-scale approach [Ecormier-Nocca et al. CASA 2019]

Conclusion Creative AI vs. an AI that creates AI systems are able to learn & combine preexisting contents • Is this what we want? [Liu & al 2015] Creative AI: Build on AI to make humans more creative • Control to the user & Smart models to help – Interpreting gestures – Duplicating details – Maintaining constraints • Knowledge & light examples ( added on the fly !)

Deep learning for Creative AI? Learning how to interact better with human users! First example : Group Editing Element shape Structure of layout [Lun et al, Siggraph Asia 2017]