Overview of Robot Decision Making Prof. Yuke Zhu Fall 2020 CS391R: - - PowerPoint PPT Presentation

CS391R: Robot Learning (Fall 2020)

Overview of Robot Decision Making

1

• Prof. Yuke Zhu

Fall 2020

CS391R: Robot Learning (Fall 2020) 3

Today’s Agenda

• What is Robot Decision Making?
• Mathematical Framework of Sequential Decision Making
• Learning for Decision Making

○ reinforcement learning (model-free vs. model-based) ○ imitation learning (behavior cloning, DAgger, IRL, and adversarial learning)

• Research Frontiers

○ compositionality, learning to learn, …

CS391R: Robot Learning (Fall 2020) 4

[Levine et al. JMLR 2016] [Bohg et al. ICRA 2018] [Sa et al. IROS 2014] Perceive Act Perceive Act Act Perceive

Ro Robot Learning is to close the perception-action loop.

CS391R: Robot Learning (Fall 2020) 5

What is Robot Decision Making?

Choosing the action a robot should perform in the physical world…

Assistive Robots (Companions) Outer Space (Explorers) Autonomous Driving (Transporters)

CS391R: Robot Learning (Fall 2020) 6

What is Robot Decision Making?

Choosing the action a robot should perform in the physical world…

• Behaviors can’t be easily programmed
• Safety and robustness under uncertainty
• Imperfect sensing and actuation

[Source: Boston Dynamics]

CS391R: Robot Learning (Fall 2020) 7

Robot Decision Making vs. Playing Games

Robot decision making is embodied, active, and environmentally situated.

[Source: Boston Dynamics] [Source: DeepMind’s AlphaGo]

CS391R: Robot Learning (Fall 2020) 8

Before We Dive In…

• This lecture is intended to provide a high-level, bird-eye

view on (robot) decision making.

• The goal is not to go through all technical details:

○

We will re-visit them through paper reading in the following weeks.

○

Study the parts that you are less familiar with from online resources.

• Take related courses and read textbooks to learn this

subject in depth (see the last slide).

CS391R: Robot Learning (Fall 2020) 9

Mathematical Framework: Marko kov Decisi sion Processe sses

A Markov Decision Process is defined by a tuple M = hS, A, P, R, γi

S

: state space A

: action space P

: transition probability R

: reward function γ

: a discount factor γ ∈ [0, 1]

(st ∈ S)

(at ∈ A)

Pa

ss0 = Pr[st+1 | st, at]

r(s, a) = E[rt+1|s = st, a = at]

CS391R: Robot Learning (Fall 2020) 10

Mathematical Framework: Marko kov Decisi sion Processe sses

A Markov Decision Process is defined by a tuple M = hS, A, P, R, γi

A policy maps states to actions

Goal of (robot) decision making

Choose policy that maximizes cumulative reward

π : S → A

π∗ = arg max

π

E[ X

t≥0

γtr(st, π(st))]

CS391R: Robot Learning (Fall 2020) 11

Mathematical Framework: Marko kov Decisi sion Processe sses

We define two functions given a policy

Value function: the expected cumulative

discounted reward when acting according to the policy from a given state

π

V π(s) = E ⇥ X

t=0

γtr(st, π(st)) | s0 = s ⇤

Q function: the expected cumulative

discounted reward when acting according to the policy from a given state and taki king a give ven action Qπ(s, a) = r(s, a) + γ X

s02S

P(s0|s, a)V π(s0)

Pi*(a|s) = arg max_a Q*(s, a) V*(s) = max_a Q*(s, a) à Pi*

CS391R: Robot Learning (Fall 2020) 12

Solving MDPs with Known Models

When we know the model of the MDP M = hS, A, P, R, γi

Value Iteration Policy Iteration

• 1. Estimate optimal value function
• 1. Start with random policy
• 2. Compute optimal policy from optimal value function
• 2. Iteratively improve it until convergence to optimal policy

Use ideas from Dynamic Programming using model using model

CS391R: Robot Learning (Fall 2020) 13

Solving MDPs with Known Models

When we know the model of the MDP M = hS, A, P, R, γi

Optimal Control (LQR)

A special case: exact solution is easily to solve Assume linear transi sitions s and qu quadr drati tic r reward fu d functi tions

Sampling-based Planning

Evaluate outcomes of sampled actions with models Choose the action that leads to the best (predicted) outcome π∗

Linear transi sition st+1 = Atst + Btat

Qu Quadrati tic r reward

always negative

Ext xtensi sions: s: LQG (Gaussian noise), iLQR (non-linear transition)

Monte-Carlo Tree Search (MCTS) for Tic-Tac-Toe

r(st, at) = −s>

t Utst − a> t Wtat

CS391R: Robot Learning (Fall 2020) 14

Solving MDPs with Learned Models

Model is known in restricted domains: ga game mes, si simulated robots, si simple mechanics When model is not known, we can learn the model from data.

A key role of learning in model- based approaches τ = {(si, ai, ri) | i = 0, . . . , H}

agent’s experience learned model mo model-base sed RL RL

ˆ P

Can be represented by Gaussi ssian Processe sses, Ne Neural l Networks ks, GM GMMs, etc.

Use planning and

• ptimization methods

for known models (previous two slides)

CS391R: Robot Learning (Fall 2020) 15

Solving MDPs with Learned Models

Model structure is known (e.g., simulator). We tune some model parameters (e.g., mass and friction).

System Identification

[Ramos et al. RSS’19]

Week 12 Tue

Sensor-Space Model Latent-Space Model

Week 8 Tue Week 8 Tue

[Hafner et al. ICLR’20] [Finn et al. ICRA’17]

Predicting future raw sensory data Predicting future latent state

f(st+1 | st, at)

ht = g(st) f(ht+1 | ht, at)

Pr[µ | D]

CS391R: Robot Learning (Fall 2020) 16

Examples of Mo Model-Base sed Reinforcement Learning

“Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation.” Fang, Zhu, Garg, Savarese, Fei-Fei, CoRL 2019

CS391R: Robot Learning (Fall 2020) 17

Solving MDPs without Models

τ = {(si, ai, ri) | i = 0, . . . , H}

agent’s experience

• ptimal policy

mo model-fre free RL RL

π∗ = arg max

π

E[ X

t≥0

γtr(st, π(st))]

τ

When model is unknown and hard to estimate, we can learn policy directly from the agent’s trajectories from interacting with an MDP.

CS391R: Robot Learning (Fall 2020) 18

Solving MDPs without Models

Model-free Value-based RL

Optimality condition (Bellman equation) Q⇤(s, a) = r(s, a) + γ Es0|s,a[max

a0 Q⇤(s0, a0)]

Q-learning rule (temporal different learning)

Week k 7 Thu

π⇤(a|s) = arg max

a0 Q⇤(s, a0)

De Deep Q-Network k (DQN): Represent Q with neural networks

CS391R: Robot Learning (Fall 2020) 19

Solving MDPs without Models

Objective Function

J(θ) = Eτ∼p(τ|θ)[r(τ)]

θ

policy parameterized by

θ

trajectories under policy

πθ

Total reward of trajectory

policy gradient theorem

Different ways of computing the Q values lead to different PG variants: Monte-Carlo estimates (REINFORCE), learning value functions (Actor-Critic)

rθJ(θ) = Eπ[Qπ(s, a)rθ log πθ(a|s)]

Model-free Policy-Gradient RL Week k 7 Thu

CS391R: Robot Learning (Fall 2020) 20

Solving MDPs without Models

Model-free Policy-Gradient RL Week k 7 Thu

CS391R: Robot Learning (Fall 2020) 21

Examples of Mo Model-Fr Free ee Rei einf nfor

• rcem

cement ent Lear Learni ning ng

“QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation.” Kalashnikov et al. CoRL 2018

Week 13 Thu, Nov 19

CS391R: Robot Learning (Fall 2020) 22

Solving MDPs without Models

Model-free Value-based RL Model-free Policy-Gradient RL

Learns policy directly – often more stable Works for continuous action spaces Needs data from current policy to compute policy gradient (“on-policy” algorithm) – data inefficient Gradient estimates can be very noisy Can learn Q function from any interaction data, not just trajectories gathered using the current policy (“off-policy” algorithm) Relatively data-efficient (can reuse old interaction data) Need to optimize over actions: hard to apply to continuous action spaces Optimal Q function can be complicated, hard to learn

CS391R: Robot Learning (Fall 2020) 23

Mathematical Framework: Marko kov Decisi sion Processe sses

Reinforcement learning optimizes policy by trial and error in an MDP. Goal: To maximize the long-term rewards

S

: state space A

: action space P

: transition probability R

: reward function γ

: a discount factor γ ∈ [0, 1]

Fundamental assumption of RL: reward function (st ∈ S)

(at ∈ A)

Pa

ss0 = Pr[st+1 | st, at]

r(s, a) = E[rt+1|s = st, a = at]

CS391R: Robot Learning (Fall 2020) 24

Mathematical Framework: Marko kov Decisi sion Processe sses

Imitation learning optimizes policy by imitating the expert in an MDP. Goal: To match the behavioral distributions

S

: state space A

: action space P

: transition probability R

: reward function γ

: a discount factor γ ∈ [0, 1]

: set of demonstrations drawn from the expert policy πE

D

(st ∈ S)

(at ∈ A)

Pa

ss0 = Pr[st+1 | st, at]

r(s, a) = E[rt+1|s = st, a = at]

CS391R: Robot Learning (Fall 2020) 25

Mathematical Framework: Marko kov Decisi sion Processe sses

Imitation learning optimizes policy by imitating the expert in an MDP. Goal: To match the behavioral distributions Two basic ideas

• Direct estimation of the expert policy from expert data

(behavioral cloning)

• Reconstruct a reward function (inverse RL) and then learn

a policy from the reward (RL)

CS391R: Robot Learning (Fall 2020) 26

Imitation as Supervised Learning

Idea 1: Direct estimation of the expert policy from expert data

π∗ = arg min

π

X

st∈D

L ✓ π(st), πE(st) ◆

This can be cast as a supervised learning problem, called behavioral cloning

action from expert policy Distance metric that measures the discrepancy between the expert action and the policy action (e.g., KL-divergence)

Week k 8 Thu

CS391R: Robot Learning (Fall 2020) 27

Imitation as Supervised Learning

Idea 1: Direct estimation of the expert policy from expert data This can be cast as a supervised learning problem, called behavioral cloning

Week k 8 Thu

What can go wrong? compounding errors

How to fix: Asking expert for more data (DAgger)

CS391R: Robot Learning (Fall 2020) 28

Examples of Supervi vise sed Imitation Learning

"Agile Autonomous Driving using End-to-End Deep Imitation Learning" Pan, Cheng, Saigol, Lee, Yan, Theodorou, Boots. RSS 2018

CS391R: Robot Learning (Fall 2020) 29

Inverse Reinforcement Learning

Idea 2: Reconstruct a reward function and then learn a policy from the reward

Week k 9 Tue

Solvi ving fu full-fl fledged RL RL in in th the in inner lo loop To solve efficiently, IRL methods often assume:

v Known dynamics (for comparing and efficiently) v Linear reward function

π

π∗

r(s, a) = w>φ(s)

Problem: IRL is generally ill-posed – many reward functions under which the expert policy is optimal. How can we address it?

CS391R: Robot Learning (Fall 2020) 30

Examples of Inve verse se Reinforcement Learning

CS391R: Robot Learning (Fall 2020) 31

Adversarial Imitation Learning

Week k 9 Thu

IL reward rIL

Dψ(

score on policy

1 [Goodfellow et al. 2014; Ho & Ermon, 2016]

discriminator policy generated trajectories environment demonstration trajectories IL reward:

rIL(st, at) = − log(1 − Dψ(st))

strong discriminator bad policy weak discriminator good policy predicts 0 if policy and 1 if demo

− Dψ(st))

disc scriminator objective ve

− Dψ

CS391R: Robot Learning (Fall 2020) 32

Adversarial Imitation Learning

Week k 9 Thu

[Goodfellow et al. 2014; Ho & Ermon, 2016]

discriminator policy generated trajectories environment demonstration trajectories IL reward:

rIL(st, at) = − log(1 − Dψ(st))

predicts 0 if policy and 1 if demo

− Dψ(st))

disc scriminator objective ve

− Dψ

v Represent complex x reward function by neural networks v More iterative ve approaches to update reward and policy (no need to run full RL before updating the reward function) v We don’t know the dynamics but have access to a si simulator to compare with and .

π

π∗

CS391R: Robot Learning (Fall 2020) 33

Examples of Adve versa sarial Imitation Learning

• 2x

4x 2x

• “Reinforcement and Imitation Learning for Diverse Visuomotor Skills.” Zhu et al. RSS 2018

CS391R: Robot Learning (Fall 2020) 34

Robotics and Decision Making: Landsc scape

le learnin ing so source re reward rd fu functi tion exp xpert demonst stration have ve mo model? no no ye yes

• p
• ptimal

mal cont control rol & & planning

le learn mo model?

mod model el-base sed rei reinf nforcement

• rcement learni

earning ng

ye yes

mod model el-fr free rei reinf nforcement

• rcement learni

earning ng Week 7 Thu Week 8 Tue

no no est stimate re reward rd? no no ye yes

im imit itatio ion as su supervi vise sed learning Week 8 Thu

kn known dyn ynamics? s? no no ye yes

inve verse se reinforcement learning Week 9 Tue adve versa sarial imitation learning Week 9 Thu

CS391R: Robot Learning (Fall 2020) 35

Robotics and Decision Making: Fr Front

• ntier

ers

Learning from rich data sources

Language, preferences, instruction

• videos. Suboptimal demonstrations.

Object variations and long-horizon tasks.

Efficient learning of new tasks

Fast learning from limited experience. Representing and transferring past knowledge.

Safety and robustness

Probabilistic and formal guarantees

• f the robot behaviors during learning

and inference

Week 10 (Tue, Thu): Learning to Learn Week 11 (Tue, Thu): Compositionality

CS391R: Robot Learning (Fall 2020) 36

Resources

Related courses at UTCS

• CS342: Neural Networks
• CS394R: Reinforcement Learning: Theory and Practice

Other Course Materials and Textbooks

• UCL Course on RL by David Silver
• Berkeley CS 294: Deep Reinforcement Learning
• Reinforcement Learning: An Introduction, Sutton and Barto
• Reinforcement Learning and Optimal Control, Bertsekas