Reinforcement Learning George Konidaris gdk@cs.brown.edu Fall - - PowerPoint PPT Presentation

Reinforcement Learning

George Konidaris gdk@cs.brown.edu

Fall 2019

Machine Learning

Subfield of AI concerned with learning from data. Broadly, using:

• Experience
• To Improve Performance
• On Some Task

(Tom Mitchell, 1997)

vs …

ML

vs

Statistics

vs

Data Mining

Why?

Developing effective learning methods has proved difficult. Why bother? Autonomous discovery

• We don’t know something, want to find out.

Hard to program

• Easier to specify task, collect data.

Adaptive behavior

• Our agents should adapt to new data, unforeseen

circumstances.

Types of Machine Learning

Depends on feedback available: Labeled data:

• Supervised learning

No feedback, just data:

• Unsupervised learning.

Sequential data, weak labels:

• Reinforcement learning

Supervised Learning

Input: X = {x1, …, xn} Y = {y1, …, yn} Learn to predict new labels. Given x: y?

inputs labels training data

Unsupervised Learning

Input: X = {x1, …, xn} Try to understand the structure of the data. E.g., how many types of cars? How can they vary?

inputs

Reinforcement Learning

Learning counterpart of planning. max

π

R =

∞

• t=0

γtrt π : S → A

MDPs

Agent interacts with an environment At each time t:

• Receives sensor signal
• Executes action
• Transition:
• new sensor signal
• reward

st at st+1 rt Goal: find policy that maximizes expected return (sum

• f discounted future rewards):

π max

π

E

• R =

∞

• t=0

γtrt

Markov Decision Processes

: set of states : set of actions : discount factor : reward function is the reward received taking action from state and transitioning to state . : transition function is the probability of transitioning to state after taking action in state . S A R R(s, a, s′) γ a s s′ T T(s′|s, a) s′ a s

< S, A, γ, R, T >

RL vs Planning

In planning:

• Transition function (T) known.
• Reward function (R) known.
• Computation “offline”.

In reinforcement learning:

• One or both of T, R unknown.
• Action in the world only source of data.
• Transitions are executed not simulated.

Reinforcement Learning

RL

This formulation is general enough to encompass a wide variety of learned control problems.

MDPs

As before, our target is a policy: A policy maps states to actions. The optimal policy maximizes: This means that we wish to find a policy that maximizes the return from every state.

π : S → A

max

π

∀s, E

• R(s) =

∞

• t=0

γtrt

• s0 = s

Planning via Policy Iteration

In planning, we used policy iteration to find an optimal policy.

• 1. Start with a policy
• 2. Estimate
• 3. Improve

a. π π

Repeat

V π π(s) = max

a

E [r + γV π(s0)] , ∀s More precisely, we use a value function: … then we would update by computing:

π

π(s) = argmaxa X

s0

T(s, a, s0) [r(s, a, s0) + γV [s0]]

V π(s) = E " ∞ X

i=0

γiri #

can’t do this anymore

Value Functions

For learning, we use a state-action value function as follows: This is the value of executing in state , then following . Note that . π a s

|A| x

V π(s) = Qπ(s, π(s)) Qπ(s, a) = E " ∞ X

i=0

γiri|s0 = s, a0 = a #

Policy Iteration

This leads to a general policy improvement framework:

• 1. Start with a policy
• 2. Learn
• 3. Improve

a. π π π(s) = max

a

Q(s, a), ∀s

Repeat

Steps 2 and 3 can be interleaved as rapidly as you like. Usually, perform 3a every time step. Qπ

Value Function Learning

Learning proceeds by gathering samples of . 
 Methods differ by:

• How you get the samples.
• How you use them to update .

Q Q(s, a)

Monte Carlo

Simplest thing you can do: sample . Do this repeatedly, average values: R(s)

r r r r r r r r

Q(s, a) = R1(s) + R2(s) + ... + Rn(s) n

Temporal Difference Learning

Where can we get more (immediate) samples? Idea: use the Bellman equation.

value of this state reward value

• f next state

Qπ(s, a) = Es0 [r(s, a, s0) + γQπ(s0, π(s0))]

TD Learning

Ideally and in expectation: is correct if this holds in expectation for all states. When it does not: temporal difference error.

ri + γQ(si+1, ai+1) − Q(si, ai) = 0

Q st st+1 at rt Q(st, at) ← rt + γQ(st+1, at+1)

Sarsa

Sarsa: very simple algorithm

• 1. Initialize Q[s][a] = 0
• 2. For n episodes
• observe state s
• select a = argmaxa Q[s][a]
• observe transition
• compute TD error
• update Q:
• if not end of episode, repeat

(s, a, r, s′, a′) δ = r + γQ(s′, a′) − Q(s, a) Q(s, a) = Q(s, a) + αδ

zero by def. if s is absorbing

Sarsa

Sarsa

Exploration vs. Exploitation

Always maxa Q(s, a)?

• Exploit current knowledge.

What if your current knowledge is wrong? How are you going to find out?

• Explore to gain new knowledge.

Exploration is mandatory if you want to find the optimal solution, but every exploratory action may sacrifice reward.

Exploration vs. Exploitation - when to try new things? Consistent theme of RL.

Exploration vs. Exploitation

How to balance? Simplest, most popular approach: Instead of always being greedy:

• maxa Q(s, a)

Explore with probability :

• maxa Q(s, a) with probability .
• random action with probability .

✏ ✏

(1 − ✏)

• greedy exploration
• Very simple
• Ensures asymptotic coverage of state space

✏

(✏ ≈ 0.1)

TD vs. MC

TD and MC two extremes of obtaining samples of Q:

t=1 t=2 t=3 t=4 t=L ...

r + γV r + γV r + γV

t=1 t=2 t=3 t=4 t=L ...

• i

γiri

Generalizing TD

We can generalize this to the idea of an n-step rollout: R(1)

st = rt + γQ(st+1, at+1)

R(2)

st = rt + γrt+1 + γ2Q(st+2, at+2)

Each tells us something about the value function.

• We can combine all n-step rollouts.
• This is known as a complex backup.

R(n)

st = rt + γrt+1 + γ2rt+2 + ... + γn−1rt+n−1 + γnQ(st+n, at+n)

. . .

TD(λ)

Weighted sum: . . . Estimator:

1

λ λn

weights

R(1) = r0 + γQ(s1, a1)

R(2) = r0 + γr1 + γ2Q(s2, a2)

R(n) =

n−1

X

i=0

γiri + γnQ(sn, an)

Sarsa(λ)

This is called the λ-return.

• At λ=0 we get Sarsa, at λ=1 we get MC.
• Intermediate values of λ usually best.
• TD(λ) family of algorithms

Sarsa(λ): Implementation

Each state has eligibility trace e(s, a). At time t: e(st, at) = 1 e(s, a) = γλe(s, a), for all other (s, a) pairs. At end of episode: e(s, a) = 0, for all (s, a) pairs. When updating:

• Compute 𝜀 as before
• Q(s, a) = Q(s, a) + α𝜀e(s, a), for each (s, a) pair.

Sarsa(λ): Implementation

• 1. Initialize Q[s][a] = 0, for all (s, a)
• 2. Initialize e[s][a] = 0, for all (s, a)
• 2. For n episodes
• observe state st
• select at = argmaxa Q(st, a)
• observe transition (st, at, rt, st+1, at+1)
• compute TD error
• e(st, at) = 1; other e(s, a) = γλe(s, a)
• update Q:
• if not end of episode, repeat
• if end of episode, e[s][a] = 0 for all (s,a)

zero by def. if s is absorbing

δ = rt + γQ(st+1, at+1) − Q(st, at)

don’t forget!

Q(s, a) = Q(s, a) + αδe(s, a), ∀s, a

Sarsa(λ)

Sarsa(λ)

Next Week: More Realism