Lecture 25: Embodied vision 1 Today Formalisms for intelligent - - PowerPoint PPT Presentation

Lecture 25: Embodied vision

1

Today

• Formalisms for intelligent agents (environment, state, action, policy)
• Imitation learning
• Reinforcement learning
• Markov Decision Processes
• Policy gradient algorithm
• Just a high-level overview. See Sutton & Barto book [http://

incompleteideas.net/book/RLbook2018.pdf] for much more complete treatment

2

Source: Isola, Torralba, Freeman

Announcements

• Sign up for final presentation timeslot by tonight!
• Email us ASAP if no time works for your group

3

[Silver et al., 2016] [Jaderberg et al. 2018]

4

Source: Isola, Torralba, Freeman

The whole purpose of visual perception, in humans, is to make good motor decisions. We are sensorimotor systems.

5

“We move in order to see and we see in order to move” — J. J. Gibson

Source: Isola, Torralba, Freeman

Agent Environment

Observations Actions

Intelligent agents

6

Source: Isola, Torralba, Freeman

Environment

State Actions

Intelligent agents

Policy

f : st, at → st+1

π : st → at

st

at

7

Source: Isola, Torralba, Freeman

state: pixels!

How to represent a state? How to represent policy?

1

π

policy: action classifier

8

Source: Isola, Torralba, Freeman

→

Learner

→

Training data

Learning from examples

{x1, y1} {x2, y2} {x3, y3}

· · ·

f : X → Y

(aka supervised learning)

f ∗ = arg min

f∈F N

X

i=1

L(f(xi), yi)

9

Source: Isola, Torralba, Freeman

Imitation learning

(still just supervised learning, applied to learn policies)

→

Learner

Training data

· · ·

→

{s1, a1}

{s2, a2}

{s3, a3}

π∗ = arg min

π∈Π N

X

i=1

L(π(si), ai)

π : s → a

10

Source: Isola, Torralba, Freeman

n

• ,

n

• ,

n

• ,

…

d3 ♝b4+ ♘f3 ♞c6 0-0 ♝xc3

→

Learner

Hypothesis space

Convolutional neural net

Optimizer

Stochastic gradient descent

→

π

Imitation learning

s

a

Objective

L(a, π(s)) = H(a, π(s))

π(s) = softmax(gθ(s))

11

Source: Isola, Torralba, Freeman

Imitation learning

12

Imitation learning

13

From images to actions

Auxiliary measurements, e.g. speed. Goal capturing expert’s intentions, e.g. “Turn right at the next intersection.”

[See Codevilla et al., “End-to-end driving via conditional imitation learning”, 2018]

14

From images to actions

[See Zhou et al., “Does computer vision matter for action?”, 2019]

• Can use mid-level representations like depth, motion.
• Or do transfer learning from pretained net

15

Reinforcement learning

What’s a good policy? (what’s the learning objective?)

→

Learner

→

Policy · · ·

Data

{τ1, R1}

{τ2, R2}

{τ3, R3}

Behavior (“Trajectory”) Score (“Return”)

π : s → a

τ = {s1, a1, s2, a2, . . .}

16

Source: Isola, Torralba, Freeman

Policy Environment

Observations Actions

Reinforcement learning

Rewards

Learn a policy that takes actions that maximize reward

17

Source: Isola, Torralba, Freeman

Imitation learning Reinforcement learning

Hand-curated training data + Instructive examples + Follows a curriculum

• Expensive
• Limited to teacher’s knowledge

No training data, have to play around and collect the data yourself + No need for labeled data + Can learn things no human knows how to do

• Less instructive
• No curriculum
• Have to explore

18

Source: Isola, Torralba, Freeman

Environment

State, Reward Actions

Policy

f : st, at → st+1

π : st → at

at

st+1, rt

Reinforcement learning

19

Source: Isola, Torralba, Freeman

Environment

State, Reward Actions

Policy

f : st, at → st+1

π : st → at

at

st+1, rt

Reinforcement learning

τ = (s0, a0, r0, s1, a1, r1, . . .)

A sample from the MPD is called a Trajectory

(

Markov decision process (MDP)

Learned

r

Action State

time

a

s

Reward

20

Source: Isola, Torralba, Freeman

Reinforcement learning

τ = (s0, a0, r0, s1, a1, r1, . . .)

Trajectory

R(τ) =

∞

X

t=0

γtrt, γ ∈ (0, 1)

Discounted Returns

Environment

State, Reward Actions

Policy

f : st, at → st+1

π : st → at

at

st+1, rt

Learn a policy that takes actions that maximize expected reward

π∗ = arg max

π

Eτ∼π[R(τ)]

21

Source: Isola, Torralba, Freeman

Reinforcement learning →

Learner

Optimizer

Objective

Hypothesis space

R(τ)

These days: deep net ?

π

Data

Environment

→

τ

Can’t, in general, backprop through env!

22

Source: Isola, Torralba, Freeman

Environment is not differentiable! — How to optimize?

Policy gradients: Run a policy for a while. See what actions led to high rewards. Increase their probability.

[Adapted from Andrej Karpathy: http://karpathy.github.io/2016/05/31/rl/] 1

π

Pong

23

Source: Isola, Torralba, Freeman

Policy gradients: Run a policy for a while. See what actions led to high rewards. Increase their probability.

[Adapted from Andrej Karpathy: http://karpathy.github.io/2016/05/31/rl/]

24

Source: Isola, Torralba, Freeman

Prediction

dolphin cat grizzly bear angel fish chameleon iguana elephant clown fish

ˆ y

1

Ground truth label y

dolphin cat grizzly bear angel fish chameleon iguana elephant clown fish

…

x

1

…

f

fθ : X → RK

Loss

• <latexit sha1_base64="efCHYzZUrW5FJbsCkY/M8lzlGQ=">ACP3icfZDPSgMxEMaz/rf+rR69LFZBRMquCHos6sGLqGCt0C0ym05rMJsyaxYlj6DV30cH8Mn8CZevZnWClrFgcCPb74kM1+cSmEpCJ69kdGx8YnJqenCzOzc/MJicenC6sxwrHItbmMwaIUCqskSOJlahCSWGItvjno9Wu3aKzQ6pw6KTYSaCvREhzISdVINzVdLZaCctAv/zeEAyixQZ1eFb21qKl5lqAiLsHaehik1MjBkOASu4Uos5gCv4E21h0qSNA28v60X/dKU2/pY07ivy+v1GDom1nSR2zgTo2g73euJfvXpGrb1GLlSaESr+VErkz5pv7e63xQGOcmOA+BGuFl9fg0GOLmACoXoEN0yBo/dwycpGiBtNvMITDuBu65brh1t9eg/o1BfRkcu13A4xd9wsV0Og3J4tlOq7A8SnmIrbJVtsJDtsgo7YqesyjgT7J49sEfvyXvxXr23T+uIN7izH6U9/4BJ7WvgQ=</latexit><latexit sha1_base64="efCHYzZUrW5FJbsCkY/M8lzlGQ=">ACP3icfZDPSgMxEMaz/rf+rR69LFZBRMquCHos6sGLqGCt0C0ym05rMJsyaxYlj6DV30cH8Mn8CZevZnWClrFgcCPb74kM1+cSmEpCJ69kdGx8YnJqenCzOzc/MJicenC6sxwrHItbmMwaIUCqskSOJlahCSWGItvjno9Wu3aKzQ6pw6KTYSaCvREhzISdVINzVdLZaCctAv/zeEAyixQZ1eFb21qKl5lqAiLsHaehik1MjBkOASu4Uos5gCv4E21h0qSNA28v60X/dKU2/pY07ivy+v1GDom1nSR2zgTo2g73euJfvXpGrb1GLlSaESr+VErkz5pv7e63xQGOcmOA+BGuFl9fg0GOLmACoXoEN0yBo/dwycpGiBtNvMITDuBu65brh1t9eg/o1BfRkcu13A4xd9wsV0Og3J4tlOq7A8SnmIrbJVtsJDtsgo7YqesyjgT7J49sEfvyXvxXr23T+uIN7izH6U9/4BJ7WvgQ=</latexit><latexit sha1_base64="efCHYzZUrW5FJbsCkY/M8lzlGQ=">ACP3icfZDPSgMxEMaz/rf+rR69LFZBRMquCHos6sGLqGCt0C0ym05rMJsyaxYlj6DV30cH8Mn8CZevZnWClrFgcCPb74kM1+cSmEpCJ69kdGx8YnJqenCzOzc/MJicenC6sxwrHItbmMwaIUCqskSOJlahCSWGItvjno9Wu3aKzQ6pw6KTYSaCvREhzISdVINzVdLZaCctAv/zeEAyixQZ1eFb21qKl5lqAiLsHaehik1MjBkOASu4Uos5gCv4E21h0qSNA28v60X/dKU2/pY07ivy+v1GDom1nSR2zgTo2g73euJfvXpGrb1GLlSaESr+VErkz5pv7e63xQGOcmOA+BGuFl9fg0GOLmACoXoEN0yBo/dwycpGiBtNvMITDuBu65brh1t9eg/o1BfRkcu13A4xd9wsV0Og3J4tlOq7A8SnmIrbJVtsJDtsgo7YqesyjgT7J49sEfvyXvxXr23T+uIN7izH6U9/4BJ7WvgQ=</latexit><latexit sha1_base64="efCHYzZUrW5FJbsCkY/M8lzlGQ=">ACP3icfZDPSgMxEMaz/rf+rR69LFZBRMquCHos6sGLqGCt0C0ym05rMJsyaxYlj6DV30cH8Mn8CZevZnWClrFgcCPb74kM1+cSmEpCJ69kdGx8YnJqenCzOzc/MJicenC6sxwrHItbmMwaIUCqskSOJlahCSWGItvjno9Wu3aKzQ6pw6KTYSaCvREhzISdVINzVdLZaCctAv/zeEAyixQZ1eFb21qKl5lqAiLsHaehik1MjBkOASu4Uos5gCv4E21h0qSNA28v60X/dKU2/pY07ivy+v1GDom1nSR2zgTo2g73euJfvXpGrb1GLlSaESr+VErkz5pv7e63xQGOcmOA+BGuFl9fg0GOLmACoXoEN0yBo/dwycpGiBtNvMITDuBu65brh1t9eg/o1BfRkcu13A4xd9wsV0Og3J4tlOq7A8SnmIrbJVtsJDtsgo7YqesyjgT7J49sEfvyXvxXr23T+uIN7izH6U9/4BJ7WvgQ=</latexit>

H(y, ˆ y) = −

K

X

k=1

yk log ˆ yk

25

Source: Isola, Torralba, Freeman

Policy output Eventual return

Up Down

π

1

Action

Up +10 points 0 points +10 points +100 points +10 points

…

+10 points

π(a|s)

= probability of choosing action a given state s

s

26

Source: Isola, Torralba, Freeman

Policy output Eventual return

Up Down

π

1

Action

Down 0 points 0 points +10 points 0 points 0 points

…

+10 points

s

π(a|s)

= probability of choosing action a given state s

27

Source: Isola, Torralba, Freeman

Policy gradient

∂ ∂θEτ∼πθ[R(τ)] = ∂ ∂θ Z

τ

p(τ|θ)R(τ)dτ

= Z

τ

p(τ|θ)  ∂ ∂θ log(p(τ|θ))

• R(τ)dτ

= Eτ∼πθ  ∂ ∂θ log(p(τ|θ))R(τ)

• <latexit sha1_base64="YiCLZgx465nwtT5blhotIWcaRk=">ACXHicbVFda9swFJW9bkvTdc02MteLguD5CXY7WB7GZSNwR670bSFyARZuU5EZVtI14Pg+U/urS/7K62cmNGvC4Jz7kf0lFqtHIURVdB+GTn6bPnvd3+3ov9lweDV6/PXFlZiVNZ6tJepMKhVgVOSZHGC2NR5KnG8/TyW6uf/0brVFmc0tpgkotloTIlBXlqPnDwBXguaJWm9fdmXnMSFXCncuBGtekKSTQN15jRDHhmhay5EZaU0M1/BF0dANflcmRG7Zg/W3I8hl+bfAzcquWKkvlgGE2iTcBDEHdgyLo4mQ/+8kUpqxwLklo4N4sjQ0nd7pYamz6vHBohL8USZx4WIkeX1BtzGvjgmQVkpfWnINiwtztqkTu3zlNf2frg7mst+Zg2qyj7nNSqMBVhIbeLskoDldA6DQtlUZJeyCkVf6uIFfCG0j+P/rehPj+kx+Cs8NJfDSJfn4cHn/t7Oixd+w9G7GYfWLH7Ac7YVMm2RW7DnrBbvAv3An3wv1taRh0PW/YnQjf3gAzd7YH</latexit>
• Do actions with high rewards more often, and low rewards less often
• This is called the REINFORCE algorithm.
• Converges very slowly, in comparison to supervised learning with gradients
• Want to take derivatives of expected reward w.r.t. the policy parameters.

28

Policy gradient

• What happens in a rollout? Recall we’re maximizing

= Eτ∼πθ  ∂ ∂θ log(p(τ|θ))R(τ)

• <latexit sha1_base64="YiCLZgx465nwtT5blhotIWcaRk=">ACXHicbVFda9swFJW9bkvTdc02MteLguD5CXY7WB7GZSNwR670bSFyARZuU5EZVtI14Pg+U/urS/7K62cmNGvC4Jz7kf0lFqtHIURVdB+GTn6bPnvd3+3ov9lweDV6/PXFlZiVNZ6tJepMKhVgVOSZHGC2NR5KnG8/TyW6uf/0brVFmc0tpgkotloTIlBXlqPnDwBXguaJWm9fdmXnMSFXCncuBGtekKSTQN15jRDHhmhay5EZaU0M1/BF0dANflcmRG7Zg/W3I8hl+bfAzcquWKkvlgGE2iTcBDEHdgyLo4mQ/+8kUpqxwLklo4N4sjQ0nd7pYamz6vHBohL8USZx4WIkeX1BtzGvjgmQVkpfWnINiwtztqkTu3zlNf2frg7mst+Zg2qyj7nNSqMBVhIbeLskoDldA6DQtlUZJeyCkVf6uIFfCG0j+P/rehPj+kx+Cs8NJfDSJfn4cHn/t7Oixd+w9G7GYfWLH7Ac7YVMm2RW7DnrBbvAv3An3wv1taRh0PW/YnQjf3gAzd7YH</latexit>

Slide adapted from R. Grosse and J. Ba ∂ ∂θ log(p(τ|θ)) = ∂ ∂θ log " p(s0)

T

Y

t=0

πθ(at|st)

T

Y

t=1

p(st|st−1, at−1) # = ∂ ∂θ log

T

Y

t=0

πθ(at|st) =

T

X

t=0

∂ ∂θ log πθ(at|st)

• All actions become more likely if the reward is high.
• Doesn’t do credit assignment.

29

• 1. Sample a rollout, e.g. play the game with current policy

• 2. Compute reward, e.g. what was our game score?

• 3. Do a gradient update:

Policy gradient

Slide adapted from R. Grosse and J. Ba

• Algorithm: do SGD on policy

τ = (s0, a0, s1, a1, . . . , sT , aT )

θ ← θ + αr(τ) ∂ ∂θπθ(at|st)

r(τ) =

T

X

t=0

R(st)

30

x

π

Up Down

+10

Policy output

Up Down

1

Action conditional expected return

• <latexit sha1_base64="efCHYzZUrW5FJbsCkY/M8lzlGQ=">ACP3icfZDPSgMxEMaz/rf+rR69LFZBRMquCHos6sGLqGCt0C0ym05rMJsyaxYlj6DV30cH8Mn8CZevZnWClrFgcCPb74kM1+cSmEpCJ69kdGx8YnJqenCzOzc/MJicenC6sxwrHItbmMwaIUCqskSOJlahCSWGItvjno9Wu3aKzQ6pw6KTYSaCvREhzISdVINzVdLZaCctAv/zeEAyixQZ1eFb21qKl5lqAiLsHaehik1MjBkOASu4Uos5gCv4E21h0qSNA28v60X/dKU2/pY07ivy+v1GDom1nSR2zgTo2g73euJfvXpGrb1GLlSaESr+VErkz5pv7e63xQGOcmOA+BGuFl9fg0GOLmACoXoEN0yBo/dwycpGiBtNvMITDuBu65brh1t9eg/o1BfRkcu13A4xd9wsV0Og3J4tlOq7A8SnmIrbJVtsJDtsgo7YqesyjgT7J49sEfvyXvxXr23T+uIN7izH6U9/4BJ7WvgQ=</latexit><latexit sha1_base64="efCHYzZUrW5FJbsCkY/M8lzlGQ=">ACP3icfZDPSgMxEMaz/rf+rR69LFZBRMquCHos6sGLqGCt0C0ym05rMJsyaxYlj6DV30cH8Mn8CZevZnWClrFgcCPb74kM1+cSmEpCJ69kdGx8YnJqenCzOzc/MJicenC6sxwrHItbmMwaIUCqskSOJlahCSWGItvjno9Wu3aKzQ6pw6KTYSaCvREhzISdVINzVdLZaCctAv/zeEAyixQZ1eFb21qKl5lqAiLsHaehik1MjBkOASu4Uos5gCv4E21h0qSNA28v60X/dKU2/pY07ivy+v1GDom1nSR2zgTo2g73euJfvXpGrb1GLlSaESr+VErkz5pv7e63xQGOcmOA+BGuFl9fg0GOLmACoXoEN0yBo/dwycpGiBtNvMITDuBu65brh1t9eg/o1BfRkcu13A4xd9wsV0Og3J4tlOq7A8SnmIrbJVtsJDtsgo7YqesyjgT7J49sEfvyXvxXr23T+uIN7izH6U9/4BJ7WvgQ=</latexit><latexit sha1_base64="efCHYzZUrW5FJbsCkY/M8lzlGQ=">ACP3icfZDPSgMxEMaz/rf+rR69LFZBRMquCHos6sGLqGCt0C0ym05rMJsyaxYlj6DV30cH8Mn8CZevZnWClrFgcCPb74kM1+cSmEpCJ69kdGx8YnJqenCzOzc/MJicenC6sxwrHItbmMwaIUCqskSOJlahCSWGItvjno9Wu3aKzQ6pw6KTYSaCvREhzISdVINzVdLZaCctAv/zeEAyixQZ1eFb21qKl5lqAiLsHaehik1MjBkOASu4Uos5gCv4E21h0qSNA28v60X/dKU2/pY07ivy+v1GDom1nSR2zgTo2g73euJfvXpGrb1GLlSaESr+VErkz5pv7e63xQGOcmOA+BGuFl9fg0GOLmACoXoEN0yBo/dwycpGiBtNvMITDuBu65brh1t9eg/o1BfRkcu13A4xd9wsV0Og3J4tlOq7A8SnmIrbJVtsJDtsgo7YqesyjgT7J49sEfvyXvxXr23T+uIN7izH6U9/4BJ7WvgQ=</latexit><latexit sha1_base64="efCHYzZUrW5FJbsCkY/M8lzlGQ=">ACP3icfZDPSgMxEMaz/rf+rR69LFZBRMquCHos6sGLqGCt0C0ym05rMJsyaxYlj6DV30cH8Mn8CZevZnWClrFgcCPb74kM1+cSmEpCJ69kdGx8YnJqenCzOzc/MJicenC6sxwrHItbmMwaIUCqskSOJlahCSWGItvjno9Wu3aKzQ6pw6KTYSaCvREhzISdVINzVdLZaCctAv/zeEAyixQZ1eFb21qKl5lqAiLsHaehik1MjBkOASu4Uos5gCv4E21h0qSNA28v60X/dKU2/pY07ivy+v1GDom1nSR2zgTo2g73euJfvXpGrb1GLlSaESr+VErkz5pv7e63xQGOcmOA+BGuFl9fg0GOLmACoXoEN0yBo/dwycpGiBtNvMITDuBu65brh1t9eg/o1BfRkcu13A4xd9wsV0Og3J4tlOq7A8SnmIrbJVtsJDtsgo7YqesyjgT7J49sEfvyXvxXr23T+uIN7izH6U9/4BJ7WvgQ=</latexit>

Σ

+6 Expected return

Estimate gradient using REINFORCE and do gradient descent

rθEτ∼πθ[R(τ)] = Eτ∼πθ[R(τ)rθ log πθ]

Approximated via sampling

31

Source: Isola, Torralba, Freeman

Policy gradients

• 1. Start with an arbitrary initial policy
• 2. Rollout this stochastic policy a bunch of times, sampling different random

actions each time

• 3. Update your policy to place higher probability on actions that led to higher

returns Mathematically, this approximates gradient ascent on policy parameters, so as to maximize reward.

Environment is not differentiable! — How to optimize?

32

Source: Isola, Torralba, Freeman

Agent Environment

Observations Actions

Intelligent agents

Why vision?

33

Source: Isola, Torralba, Freeman

Why vision?

• 1. Human-like intelligence (and animal-like), relies heavily on vision

We already know it works well!

34

Source: Isola, Torralba, Freeman

Why vision?

• 2. Eyes are good sensors

Farther away things look smaller Get details on stuff that we can immediately interact with, rough summary of more distant context

35

Source: Isola, Torralba, Freeman

Why vision?

• 2. Eyes are good sensors

Laser rangefinder

36

Source: Isola, Torralba, Freeman

Why vision?

• 2. Eyes are good sensors

Laser rangefinder

37

Source: Isola, Torralba, Freeman

Why vision?

• 2. Eyes are good sensors

Camera

38

Source: Isola, Torralba, Freeman

Why vision?

• 3. Universal interface

39

Source: Isola, Torralba, Freeman

Model-based intelligence

If vision can give us a good representation/model of the world, then planning and control should be easy.

Yann LeCun’s “cake” Reinforcement learning Representation learning

40

Source: Isola, Torralba, Freeman

21 million games!

Atari Games

~10-50 million interactions!

[Slide adapted from Pulkit Agrawal]

41

Source: Isola, Torralba, Freeman

Next class: more visual reinforcement learning

42