SLIDE 1
Planning with Learned Models
Agrawal et al., 2016; Finn & Levine, 2016; Ebert et al., 2018 Watter et al., 2015, Banijamali et al. 2017, Zhang et al. 2017
Learning Latent Dynamics for Planning from Pixels Danijar Hafner, - - PowerPoint PPT Presentation
Learning Latent Dynamics for Planning from Pixels Danijar Hafner, - - PowerPoint PPT Presentation
Learning Latent Dynamics for Planning from Pixels Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, James Davidson @danijar h danijar.com/planet Planning with Learned Models Watter et al., 2015, Banijamali
SLIDE 2
SLIDE 3
Visual Control Tasks
Some model-free methods can solve these tasks but need up to 100,000 episodes partially
- bservable
contacts many joints sparse reward balance
SLIDE 4
Visual Control Tasks
Some model-free methods can solve these tasks but need up to 100,000 episodes partially
- bservable
contacts many joints sparse reward balance
SLIDE 5
Recipe for scalable model-based reinforcement learning Efficient planning in latent space with large batch size Reaches top performance using 200X fewer episodes
We introduce PlaNet
1 2 3
SLIDE 6
Latent Dynamics Model
encode images
SLIDE 7
Latent Dynamics Model
encode images predict states
SLIDE 8
Latent Dynamics Model
encode images predict states decode images
SLIDE 9
Latent Dynamics Model
encode images predict states decode images decode rewards
SLIDE 10
Recurrent State Space Model
s1 z1 s2 z2 s3 z3
stochastic deterministic
h1 h2 h3 z1 z3 z2 h1 h2 h3
Recurrent Neural Network State Space Model Recurrent State Space Model
SLIDE 11
Unguided Video Predictions by Single Agent
5 frames context and 45 frames predicted
SLIDE 12
Unguided Video Predictions by Single Agent
5 frames context and 45 frames predicted
SLIDE 13
Planning in Latent Space
SLIDE 14
Planning in Latent Space
SLIDE 15
Planning in Latent Space
SLIDE 16
Planning in Latent Space
SLIDE 17
Planning in Latent Space
SLIDE 18
Planning in Latent Space
SLIDE 19
Comparison to Model-Free Agents
Training time 1 day on a single GPU
SLIDE 20
Enabling More Model-Based RL Research
Explore dynamics without supervision Distill the planner to save computation Value function to extend planning horizon
SLIDE 21
Learning Latent Dynamics for Planning from Pixels
Website with code, videos, blog post, animated paper: danijar.com/planet