Maximum Entropy Framework: Inverse RL, Soft Optimality, and More - - PowerPoint PPT Presentation

Maximum Entropy Framework: Inverse RL, Soft Optimality, and More

Chelsea Finn and Sergey Levine UC Berkeley

5/20/2017

Introductions

Chelsea Finn PhD student

Sergey Levine assistant professor

• 1. A World without Rewards
• 2. A Probabilistic Model of Behavior
• 3. Application: Inverse RL
• 4. GANs and Energy-Based Models
• 5. Application: Soft-Q Learning

Outline

• 1. A World without Rewards
• 2. A Probabilistic Model of Behavior
• 3. Application: Inverse RL
• 4. GANs and Energy-Based Models
• 5. Application: Soft-Q Learning

Outline

Mnih et al. ’15 video from Montessori New Zealand

what is the reward?

reinforcement learning agent

reward

In the real world, humans don’t get a score.

reward function is essential for RL real-world domains: reward/cost often difficult to specify

• robotic manipulation
• autonomous driving
• dialog systems
• virtual assistants
• and more…

Kohl & Stone, ’04 Mnih et al. ’15 Silver et al. ‘16 Tesauro ’95

One approach: Mimic actions of human expert

• but no reasoning about outcomes or dynamics
• the expert might have different degrees of freedom
• the expert might not be always optimal

Can we reason about human decision-making? behavioral cloning + simple, sometimes works well

• 1. A World without Rewards
• 2. A Probabilistic Model of Behavior
• 3. Application: Inverse RL
• 4. GANs and Energy-Based Models
• 5. Application: Soft-Q Learning

Outline

Op&mal Control as a Model of Human Behavior

Mombaur et al. ‘09 Muybridge (c. 1870) Ziebart ‘08 Li & Todorov ‘06

• pQmize this to explain the data

What if the data is no not op&mal?

some mistakes maTer more than others! behavior is stochas'c but good behavior is sQll the most likely

A probabilis&c graphical model of decision making

no assumpQon of opQmal behavior!

Inference = planning

how to do inference?

A closer look at the backward pass

“opQmisQc” transiQon (not a good idea!) Ziebart et al. ‘10 “Modeling InteracQon via the Principle of Maximum Causal Entropy”

Stochas&c op&mal control (MaxCausalEnt) summary

variants: summary:

• 1. ProbabilisQc graphical

model for opQmal control

• 2. Control = inference

(similar to HMM, EKF, etc.)

• 3. Very similar to dynamic

programming, value iteraQon,

• etc. (but “soc”)

• 1. A World without Rewards
• 2. A Probabilistic Model of Behavior
• 3. Application: Inverse RL
• 4. GANs and Energy-Based Models
• 5. Application: Soft-Q Learning

Outline

Under reward, we can model how human can sub-optimally maximize reward. How can this help us with learning?

Inverse Optimal Control / Inverse Reinforcement Learning: infer cost/reward function from demonstrations

Challenges underdefjned problem difficult to evaluate a learned reward demonstrations may not be precisely optimal

given:

• state & action space
• roll-outs from π*
• dynamics model [sometimes]

goal:

• recover reward function
• then use reward to get policy

(Kalman ’64, Ng & Russell ’00)

(IOC/IRL)

Early IRL Approaches

• deterministic MDP
• alternative between solving MDP & updating reward
• heuristics for handling sub-optimality

Ng & Russell ‘00: expert actions should have higher value than

• ther actions, larger gap is better

Abbeel & Ng ’04: expert policy w.r.t. cost should match feature counts of expert trajectories Ratliff et al. ’06: max margin formulation between value of expert actions and other actions How to handle ambiguity and suboptimality?

Whiteboard

Maximum Entropy Inverse RL

(Ziebart et al. ’08)

Notation:

: reward with parameters [linear case ] : dataset of demonstrations

handle ambiguity using probabilistic model of behavior

Maximum Entropy Inverse RL

(Ziebart et al. ’08)

Whiteboard

What about unknown dynamics?

Goals:

• remove need to solve MDP in the inner loop
• be able to handle unknown dynamics
• handle continuous state & actions spaces

Case Study: Guided Cost Learning

ICML 2016

Update reward using samples & demos generate policy samples from π update π w.r.t. reward

x x x

1 2 n

h h h

1 2 k

h h h

1 2 p

(1) ( 1 ) (1) ( 2 ) ( 2 ) (2)

h h h

1 2 m

( 3 ) ( 3 ) ( 3 )

c (x)

θ

policy π reward r guided cost learning algorithm

policy π0

Update reward using samples & demos generate policy samples from π update π w.r.t. reward (partially optimize) generator

x x x

1 2 n

h h h

1 2 k

h h h

1 2 p

(1) ( 1 ) (1) ( 2 ) ( 2 ) (2)

h h h

1 2 m

( 3 ) ( 3 ) ( 3 )

c (x)

θ

policy π reward r discriminator guided cost learning algorithm update reward in inner loop of policy optimization

policy π0

Update reward using samples & demos generate policy samples from π generator Ho et al., ICML ’16, NIPS ‘16

x x x

1 2 n

h h h

1 2 k

h h h

1 2 p

(1) ( 1 ) (1) ( 2 ) ( 2 ) (2)

h h h

1 2 m

( 3 ) ( 3 ) ( 3 )

c (x)

θ

policy π reward r discriminator guided cost learning algorithm update π w.r.t. reward (partially optimize)

policy π0

GCL Experiments

dish placement pouring almonds Real-world Tasks

state includes goal plate pose state includes unsupervised visual features [Finn et al. ’16] action: joint torques

Update reward using samples & demos generate policy samples from q

x x x

1 2 n

h h h

1 2 k

h h h

1 2 p

(1) ( 1 ) (1) ( 2 ) ( 2 ) (2)

h h h

1 2 m

( 3 ) ( 3 ) ( 3 )

c (x)

θ

reward r Comparisons Relative Entropy IRL (Boularias et al. ‘11) Path Integral IRL (Kalakrishnan et al. ‘13)

Dish placement, demos

Dish placement, standard cost

Dish placement, RelEnt IRL

• video of dish baseline method

Dish placement, GCL policy

• video of dish our method - samples & reoptimizing

Pouring, demos

• video of pouring demos

Pouring, RelEnt IRL

• video of pouring baseline method

Pouring, GCL policy

• video of pouring our method - samples

Conclusion: We can recover successful policies for new positions. Is the reward function also useful for new scenarios?

Dish placement - GCL reopt.

• video of dish our method - samples & reoptimizing

Pouring - GCL reopt.

• video of pouring our method - reoptimization

Note: normally the GAN discriminator is discarded

Guided Cost Learning & Generative Adversarial Imitation Learning

Strengths

• can handle unknown dynamics
• scales to neural net rewards
• efficient enough for real robots

Limitations

• adversarial optimization is hard
• can’t scale to raw pixel observations of demos
• demonstrations typically collected with kinesthetic

teaching or teleoperation (fjrst person)

• 1. A World without Rewards
• 2. A Probabilistic Model of Behavior
• 3. Application: Inverse RL
• 4. GANs and Energy-Based Models
• 5. Application: Soft-Q Learning

Outline

Generative Adversarial Networks

Similarly, GANs learn an objective for generative modeling. real generated

(Goodfellow et al. ’14)

Zhu et al. ‘17 Isola et al. ‘17 Arjovsky et al. ‘17

noise

G D

Connection between Inverse RL and GANs

policy π~q(τ) generator G reward r discriminator D Finn*, Christiano*, Abbeel, Levine, arXiv ‘16 trajectory τ sample x discriminator only needs to learn data distribution, θ independent of generator density discriminator

Connection between Inverse RL and GANs

policy π~q(τ) generator G cost c discriminator D Finn*, Christiano*, Abbeel, Levine, arXiv ‘16 trajectory τ sample x generator objective is entropy-regularized RL generator

GANs for training EBMs

Finn*, Christiano*, Abbeel, Levine, arXiv ‘16 energy E discriminator D sampler q(x) generator G MaxEnt IRL is an energy-based model Use the generator’s density q(x) to form a consistent estimator of the energy function

Dθ(x) =

1 Z exp(−Eθ(x)) 1 Z exp(−Eθ(x)) + q(x)

Dai et al., ICLR submission ‘17

Kim & Bengio ICLR Workshop ’16; Zhao et al. arXiv ’16; Zhai et al. ICLR sub ‘17

• 1. A World without Rewards
• 2. A Probabilistic Model of Behavior
• 3. Application: Inverse RL
• 4. GANs and Energy-Based Models
• 5. Application: Soft-Q Learning

Outline

Stochas&c models for learning control

• How can we track both

hypotheses?

StochasQc energy-based policies

Tuomas Haarnoja Haoran Tang

SoK Q-learning

Tractable amor&zed inference for con&nuous ac&ons

Wang & Liu, ‘17

StochasQc energy-based policies aid exploraQon

StochasQc energy-based policies provide pretraining

Stochas&c Op&mal Control & MaxEnt in RL

Sallans & Hinton. Using Free Energies to Represent Q-values in a MulQagent Reinforcement Learning Task. 2000. Nachum et al. Bridging the Gap Between Value and Policy Based Reinforcement Learning. 2017. Peters et al. RelaQve Entropy Policy Search. 2010. O’Donoghue et al. Combining Policy Gradient and Q-Learning. 2017

Applica&ons of inverse reinforcement learning

Kitani et al. ‘14: Model human pedestrian interacQons Ziebart et al. ‘08: Predict taxi driver route preferences Dragan et al. ‘13: GeneraQng human-legible moQon (beyond roboQc manipulaQon and control) Li et al. ‘17: Learn objecQve for dialog generaQon

Concluding remarks