SLIDE 1 Visual Grounding of Learned Physical Models
http://visual-physics-grounding.csail.mit.edu/ (* indicates equal contribution)
Yunzhu Li Toru Lin* Kexin Yi* Daniel M. Bear Daniel L.K. Yamins Jiajun Wu Joshua B. Tenenbaum Antonio Torralba
ICML 2020
SLIDE 2 Intuitive Physics
(1) Distinguish between different instances (2) Recognize objects’ physical properties (3) Predict future movements
(Wu et al., Learning to See Physics via Visual De-animation)
SLIDE 3 Intuitive Physics
(1) Distinguish between different instances (2) Recognize objects’ physical properties (3) Predict future movements
(Wu et al., Learning to See Physics via Visual De-animation)
SLIDE 4 For example
Different physical parameters lead to different motions. Estimating physical parameter by comparing mental simulation with observation
Larger stiffness Smaller stiffness Larger gravity Smaller gravity
SLIDE 5
Physical reasoning of deformable objects is challenging.
SLIDE 6
Physical reasoning of deformable objects is challenging. Particle-based Representation General & Flexible
SLIDE 7
Physical reasoning of deformable objects is challenging. Particle-based Representation General & Flexible We propose a model that jointly (1) Estimates the physical properties (2) Refines the particle locations using (1) a learned visual prior (2) a learned dynamics prior
SLIDE 8
Visually Grounded Physics Learner (VGPL)
SLIDE 9
Visually Grounded Physics Learner (VGPL)
SLIDE 10
Visually Grounded Physics Learner (VGPL)
Visual Grounding
SLIDE 11
Visually Grounded Physics Learner (VGPL)
Visual Grounding
SLIDE 12
We evaluate our model in environments involving interactions between rigid objects, elastic materials, and fluids.
SLIDE 13
We evaluate our model in environments involving interactions between rigid objects, elastic materials, and fluids. Within a few observation steps, our model is able to (1) refine the state estimation and reason about the physical properties (2) make predictions into the future.
SLIDE 14 Related Work
Li, Wu, Tedrake, Tenenbaum, Torralba. ICLR’19 Mrowca, Zhuang, Wang, Haber, Fei-Fei, Tenenbaum, Yamins. NeurIPS’18
Learning-based particle dynamics
Sanchez-Gonzalez, Godwin, Pfaff, Ying, Leskovec,
Ummenhofer, Prantl, Thuerey, Koltun. ICLR’20 Battaglia, Pascanu, Lai, Rezende,
SLIDE 15 Related Work
Li, Wu, Tedrake, Tenenbaum, Torralba. ICLR’19 Mrowca, Zhuang, Wang, Haber, Fei-Fei, Tenenbaum, Yamins. NeurIPS’18
Learning-based particle dynamics
Sanchez-Gonzalez, Godwin, Pfaff, Ying, Leskovec,
Ummenhofer, Prantl, Thuerey, Koltun. ICLR’20 Battaglia, Pascanu, Lai, Rezende,
Questions remains: (1) How well they handle visual inputs? (2) How to adapt to scenarios of unknown physical parameters?
SLIDE 16 Related Work
Hu, Liu, Spielberg, Tenenbaum, Freeman, Wu, Rus, Matusik. ICRA’19
Differentiating through physics-based simulators
Schenck, Fox. CoRL’18 Degrave, Hermans, Dambre, Wyffels. Frontiers in Neurorobotics 2019 Belbute-Peres, Smith, Allen, Tenenbaum, Kolter. NeurIPS’18 Liang, Lin, Koltun. NeurIPS’19
SLIDE 17 Related Work
Hu, Liu, Spielberg, Tenenbaum, Freeman, Wu, Rus, Matusik. ICRA’19
Differentiating through physics-based simulators
Schenck, Fox. CoRL’18 Degrave, Hermans, Dambre, Wyffels. Frontiers in Neurorobotics 2019 Belbute-Peres, Smith, Allen, Tenenbaum, Kolter. NeurIPS’18 Liang, Lin, Koltun. NeurIPS’19
Questions remains: (1) Make strong assumptions on the structure of the system (2) Usually time-consuming (2) Prone to local optimum (3) Lacking ways to handle visual inputs
SLIDE 18
Our Work
We proposed Visually Grounded Physics Learner (VGPL) to (1) bridge the perception gap, (2) enable physical reasoning from visual perception, and (3) perform dynamics-guided inference to directly predict the optimization results, which allows quick adaptation to environments with unknown physical properties.
SLIDE 19
Problem Formulation
Consider a system that contains objects and particles.
SLIDE 20
Problem Formulation
Consider a system that contains objects and particles. : Visual observ.
SLIDE 21
Problem Formulation
Consider a system that contains objects and particles. Visual prior : Particle position : Instance grouping : Visual observ.
SLIDE 22
Problem Formulation
Consider a system that contains objects and particles. Visual prior Dynamics prior : Particle position : Instance grouping : Visual observ.
SLIDE 23
Problem Formulation
Consider a system that contains objects and particles. Visual prior Dynamics prior : Particle position : Instance grouping : Visual observ. : Rigidness of each instance
SLIDE 24
Problem Formulation
Consider a system that contains objects and particles. Visual prior Dynamics prior : Particle position : Instance grouping : Visual observ. : Rigidness of each instance : Physical parameters
SLIDE 25
Problem Formulation
Consider a system that contains objects and particles. Visual prior Dynamics prior Inference module : Particle position : Instance grouping : Rigidness of each instance : Physical parameters : Visual observ.
SLIDE 26
Problem Formulation
Consider a system that contains objects and particles. Visual prior Dynamics prior Inference module : Particle position : Instance grouping : Rigidness of each instance : Physical parameters : Position refinement : Visual observ.
SLIDE 27
Problem Formulation
Consider a system that contains objects and particles. Visual prior Dynamics prior Inference module Objective function : Particle position : Instance grouping : Rigidness of each instance : Physical parameters : Position refinement : Visual observ.
SLIDE 28
Visual Prior
Visual observations :
SLIDE 29
Visual Prior
Particle locations : Instance grouping : Visual observations :
SLIDE 30
Visual Prior
Particle locations : Instance grouping : Objective function Visual observations :
SLIDE 31
Results of the Visual Prior
Visual Inputs Visual Inputs Prediction Prediction
SLIDE 32
Dynamics Prior
: Particle position : Instance grouping
SLIDE 33
Dynamics Prior
: Particle position : Instance grouping : Rigidness of each instance : Physical parameters
SLIDE 34 Dynamics Prior
: Particle position : Instance grouping : Rigidness of each instance : Physical parameters
Li, Wu, Tedrake, Tenenbaum, Torralba, “Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids,” ICLR’19
SLIDE 35
Results of the Dynamics Prior
SLIDE 36
Dynamics-Guided Inference
SLIDE 37
Dynamics-Guided Inference
: Rigidness of each instance : Physical parameters
SLIDE 38
Dynamics-Guided Inference
: Particle position : Instance grouping : Rigidness of each instance : Physical parameters
SLIDE 39
Dynamics-Guided Inference
: Particle position : Instance grouping : Rigidness of each instance : Physical parameters : Position refinement
SLIDE 40
Results
We will mainly investigate how accurate the following estimations are and whether they help with future prediction: (1) : Rigidness estimation (2) : Parameter estimation (3) : : Position refinement
SLIDE 41
Qualitative results on Rigidness Estimation
SLIDE 42 Quantitative results on Rigidness Estimation
Mean accuracy Mean accuracy
SLIDE 43
Qualitative results on Parameter Estimation
SLIDE 44
Quantitative results on Parameter Estimation
SLIDE 45
Qualitative results on Position Refinement
SLIDE 46
Quantitative results on Position Refinement
SLIDE 47
Quantitative results on Future Prediction
SLIDE 48
In summary
We proposed Visually Grounded Physics Learner (VGPL) to (1) simultaneously reason about physics and make future predictions based on visual and dynamics priors.
SLIDE 49
In summary
We proposed Visually Grounded Physics Learner (VGPL) to (1) simultaneously reason about physics and make future predictions based on visual and dynamics priors. (2) We employ a particle-based representation to handle rigid bodies, deformable objects, and fluids.
SLIDE 50
In summary
We proposed Visually Grounded Physics Learner (VGPL) to (1) simultaneously reason about physics and make future predictions based on visual and dynamics priors. (2) We employ a particle-based representation to handle rigid bodies, deformable objects, and fluids. (3) Experiments show that our model can infer the physical properties within a few observations, which allows the model to quickly adapt to unseen scenarios and make accurate predictions into the future.
SLIDE 51
Thank you for watching!
