SLIDE 1
Learning to Model the Functionality and Physics of 3D Shapes
Presenter: Shilin Zhu CSE Department, UCSD
Logic Flow
- How to obtain the interaction context?
- How to apply the forces?
- How to infer physical properties?
Learning to Model the Functionality and Physics of 3D Shapes - - PowerPoint PPT Presentation
Learning to Model the Functionality and Physics of 3D Shapes - - PowerPoint PPT Presentation
Learning to Model the Functionality and Physics of 3D Shapes Presenter: Shilin Zhu CSE Department, UCSD Logic Flow How to obtain the interaction context? How to apply the forces? How to infer physical properties? How to obtain the
SLIDE 2
SLIDE 3
How to obtain the interaction context?
[SIGGRAPH 2018] Predictive and Generative Neural Networks for Object Functionality
Ruizhen Hu, Zihao Yan, Jingwen Zhang, Oliver Van Kaick, Ariel Shamir, Hao Zhang, Hui Huang Shenzhen University, Carleton University, Simon Fraser University
SLIDE 4
Key Observations
- Human can predict functionality of objects without surroundings
- Experience allows to hallucinate interactions
- One provides functionality and another one consumes it
SLIDE 5
What is an interaction context?
- Central objects interact with surroundings that reflects functionalities
- Goal: prediction and synthesis
SLIDE 6
Complete Framework
- Goal: prediction and synthesis
- fSIM-Net: predict functional similarity
- iGEN-Net: generate interaction context
- iSEG-Net: obtain individual objects and interaction types
SLIDE 7
fSIM-Net
- fSIM-Net: predict functional similarity (triplet network)
- Object’s latent space: GMM
- Principle
- (X, Y+): Push close
- (X, Y-): Pull apart
SLIDE 8
iGEN-Net
- iGEN-Net: generate interaction context
- Two branches
- Interaction context generation
- Object placement
SLIDE 9
iSEG-Net
- iSEG-Net: obtain individual objects and interaction types
- Two branches
- Segmentation on the scene context
- Object placement
SLIDE 10
Results: Retrieval
SLIDE 11
Results: Multi-Functionality Synthesis
SLIDE 12
Results: Segmented Interaction Contexts
SLIDE 13
How to apply the forces?
[ArXiV 2019] Learning Generalizable Physical Dynamics of 3D Rigid Objects
Davis Rempe, Srinath Sridhar, He Wang, Leonida J. Guibas Stanford University
SLIDE 14
Key Observations
- Human has intuitive knowledge of dynamics
- The knowledge allows predicting the effect of physical interactions
- The dynamics should obey physics law that can generalize
SLIDE 15
One Simple Case
- Predicting the rest pose after an impulse force applied on the 3D point cloud
- Physics says:
SLIDE 16
One Simple Case
- Input: 3D shape represented by point cloud
SLIDE 17
Model Architecture
- Predicting the rest pose after an impulse force applied on the 3D point cloud
SLIDE 18
Results: Resting Pose
SLIDE 19
How to infer physical properties?
[ECCV 2018] Physical Primitive Decomposition
Zhijian Liu, William T. Freeman, Joshua B. Tenenbaum, Jiajun Wu MIT, Google Research
SLIDE 20
Key Observations
- Human can decompose object into physical primitives
- We need a physical shape representation modeling geometry and physics
- The model should infer both geometrical primitive and physical parameter
SLIDE 21
What is Physical Primitive Decomposition?
- It is more than geometry
SLIDE 22
What is Physical Primitive Decomposition?
- It is challenging: appearance and physical motion both contribute to
physical parameters (i.e., density)
SLIDE 23
Model Architecture
- The model jointly combines shape, texture and motion into consideration
- Classification and sampling on physical parameters leads to better results
SLIDE 24
Results: Density Estimation
SLIDE 25
Results: Analysis
Confusion Matrix
SLIDE 26
Recall: Logic Flow
- How to obtain the interaction context?
- How to apply the forces?
- How to infer physical properties?
SLIDE 27
Summary
- Making machines understand interaction and
physics is non-trivial
- Deep neural network behaves far from human
perception on dynamics
- Complexity of real dynamics and functionality
definition makes the problem hard to model