Learning to Model the Functionality and Physics of 3D Shapes - PowerPoint PPT Presentation

Learning to Model the Functionality and Physics of 3D Shapes Presenter: Shilin Zhu CSE Department, UCSD

Logic Flow • How to obtain the interaction context? • How to apply the forces? • How to infer physical properties?

How to obtain the interaction context? [SIGGRAPH 2018] Predictive and Generative Neural Networks for Object Functionality Ruizhen Hu, Zihao Yan, Jingwen Zhang, Oliver Van Kaick, Ariel Shamir, Hao Zhang, Hui Huang Shenzhen University, Carleton University, Simon Fraser University

Key Observations • Human can predict functionality of objects without surroundings • Experience allows to hallucinate interactions • One provides functionality and another one consumes it

What is an interaction context? • Central objects interact with surroundings that reflects functionalities • Goal: prediction and synthesis

Complete Framework • Goal: prediction and synthesis • fSIM-Net: predict functional similarity • iGEN-Net: generate interaction context • iSEG-Net: obtain individual objects and interaction types

fSIM-Net • fSIM-Net: predict functional similarity (triplet network) • Object’s latent space: GMM • Principle • (X, Y+): Push close • (X, Y-): Pull apart

iGEN-Net • iGEN-Net: generate interaction context • Two branches • Interaction context generation • Object placement

iSEG-Net • iSEG-Net: obtain individual objects and interaction types • Two branches • Segmentation on the scene context • Object placement

Results: Retrieval

Results: Multi-Functionality Synthesis

Results: Segmented Interaction Contexts

How to apply the forces? [ArXiV 2019] Learning Generalizable Physical Dynamics of 3D Rigid Objects Davis Rempe, Srinath Sridhar, He Wang, Leonida J. Guibas Stanford University

Key Observations • Human has intuitive knowledge of dynamics • The knowledge allows predicting the e ff ect of physical interactions • The dynamics should obey physics law that can generalize

One Simple Case • Predicting the rest pose after an impulse force applied on the 3D point cloud • Physics says:

One Simple Case • Input: 3D shape represented by point cloud

Model Architecture • Predicting the rest pose after an impulse force applied on the 3D point cloud

Results: Resting Pose

How to infer physical properties? [ECCV 2018] Physical Primitive Decomposition Zhijian Liu, William T. Freeman, Joshua B. Tenenbaum, Jiajun Wu MIT, Google Research

Key Observations • Human can decompose object into physical primitives • We need a physical shape representation modeling geometry and physics • The model should infer both geometrical primitive and physical parameter

What is Physical Primitive Decomposition? • It is more than geometry

What is Physical Primitive Decomposition? • It is challenging: appearance and physical motion both contribute to physical parameters (i.e., density)

Model Architecture • The model jointly combines shape, texture and motion into consideration • Classification and sampling on physical parameters leads to better results

Results: Density Estimation

Results: Analysis Confusion Matrix

Recall: Logic Flow • How to obtain the interaction context? • How to apply the forces? • How to infer physical properties?

Summary • Making machines understand interaction and physics is non-trivial • Deep neural network behaves far from human perception on dynamics • Complexity of real dynamics and functionality definition makes the problem hard to model