Multi-Arm Bandit Sutton and Barto Sutton slides and Silver 1 - - PowerPoint PPT Presentation

Multi-Arm Bandit Sutton and Barto

1

Sutton slides and Silver

Multi-Arm Bandits

Sutton and Barto, Chapter 2

The simplest reinforcement learning problem

The Exploration/Exploitation Dilemma

295, class 2 3

Online decision-making involves a fundamental choice:

• Exploitation Make the best decision given current information
• Exploration Gather more information

The best long-term strategy may involve short-term sacrifices Gather enough information to make the best overall decisions

Examples

295, class 2 4

Restaurant Selection Exploitation Go to your favourite restaurant Exploration Try a new restaurant Online Banner Advertisements Exploitation Show the most successful advert Exploration Show a different advert Oil Drilling Exploitation Drill at the best known location Exploration Drill at a new location Game Playing Exploitation Play the move you believe is best Exploration Play an experimental move

Multi‐Armed Bandit

295, class 2 6

Regret

295, class 2 10

Counting Regret

295, class 2 11

Multi-Armed Bandits Regret

11 12 13 14 15 16 17 1819

Totalregret ϵ-greedy greedy

1 2 3 4 5 6 7 8 9 10

Time-steps decaying ϵ-greedy

If an algorithm forever explores it will have linear total regret If an algorithm never explores it will have linear total regret Is it possible to achieve sublinear total regret?

Linear or Sublinear Regret

Complexity of regret

295, class 2 13

Overview

• Action‐value methods

– Epsilon‐greedy strategy – Incremental implementation – Stationary vs. non‐stationary environment – Optimistic initial values

• UCB action selection
• Gradient bandit algorithms
• Associative search (contextual bandits)

295, class 2 14

Basics

• Maximize total reward collected

– vs learn (optimal) policy (RL)

• Episode is one step
• Complex function of

– True value – Uncertainty – Number of time steps – Stationary vs non‐stationary?

295, class 2 15

Action‐Value Methods

Greedy Algorithms

295, class 2 17

‐Greedy Algorithm

295, class 2 18

A sim ple bandit algorithm

1 2 3 4 7 8 9 10 1 2 3

• 3
• 2
• 1

q⇤(1) q

⇤(2)

q

⇤(3)

q

⇤(4)

q

⇤(5)

q

⇤(6)

q

⇤(7)

q

⇤(8)

q

⇤(9)

q

⇤(10)

Reward distribution

5 6

Action

• 4

4

One Bandit T askfrom

The 10‐armedTestbed

Run for 1000 steps Repeat the whole thing 2000 times with different bandit tasks

Figure 2.1: An example bandit problem from the 10-armed testbed. The true value q(a) of each of the ten actions was selected according to a normal distribution with mean zero and unit variance, and then the actual rewards were selected according to a mean q(a) unit variance normal distribution, as suggested by these gray distributions.

‐Greedy Methods on the 10‐ArmedTestbed

Averaging ⟶ learning rule

• T
• simplify notation, let us focus on one action
• We consider only its rewards, and its estimate after n+1 rewards:
• How can we do this incrementally (without storing all the rewards)?
• Could store a running sum and count (and divide), or equivalently:

. Qn = R1 + R2 + · · ·+ Rn-1 n - 1

Derivation of incremental update

Tracking a Non‐stationary Problem

Standard stochastic approximation convergence conditions

Optimistic InitialValues

Optimistic InitialValues

So far we have used

• All methods so far depend on Q1(a), i.e.,they are biased.

Q1(a) = 0

• Suppose we initialize the action values optimistically (Q1(a) = 5 ), e.g., on

the 10-armed testbed (with alpha= 0.1 )

20% 0% 40% 60% 80% 100%

% Optimal action

200 400 600 800 1000

Plays

realistic, ε-greedy

Steps

Q1= 0, ε = 0.1

• ptimistic, greedy

Q1= 5, ε = 0

Decaying ε‐greedy

295, class 2 29

Optimism in the face of uncertainty

295, class 2 30

Optimism in the face of uncertainty

295, class 2 31

Upper Confidence Bounds

295, class 2 32

Hoeffding’s Inequality

295, class 2 33

Calculating UCB

295, class 2 34

Upper Confidence Bound (UCB) action selection

• A clever way of reducing exploration over time
• Focus on actions whose estimate has large degree of uncertainty
• Estimate an upper bound on the true action values
• Select the action with the largest (estimated) upper bound

UCB c =2

E-greedy E = 0.1

Average reward Steps

Theorem

t →∞

lim Lt ≤ 8 logt The UCB algorithm achieves logarithmic asymptotic total regret

a

a|∆ > 0

∆ a

Complexity of UCB Algorithm

UCB vs ε‐greedy on 10‐armed bandit

UCB vs ε‐greedy on 10‐armed bandit

Gradient‐Bandit Algorithms

• Let Ht(a) be a learned preference for taking action a

% Optimal action α =0.1

100% 80% 60% 40% 20% 0%

α =0.4 α =0.1 α =0.4

without baseline with baseline

250 500

Steps

750 1000

Derivation of gradient‐bandit algorithm

Summary Comparison of BanditAlgorithms

Conclusions

• These are all simple methods
• but they are complicated enough—we will build on them
• we should understand them completely
• there are still open questions
• Our first algorithms that learn from evaluative feedback
• and thus must balance exploration and exploitation
• Our first algorithms that appear to have a goal

—that learn to maximize reward by trial and error

Our first dimensions!

• Problems vs Solution Methods
• Evaluative vs Instructive
• Associative vs Non-associative

Bandits? Problem orSolution?

Problem space

Single State Associative Instructive feedback Evaluative feedback

Problem space

Single State Associative Instructive feedback Evaluative feedback Bandits

(Function optimization)

Problem space

Single State Associative Instructive feedback Supervised learning Evaluative feedback Bandits

(Function optimization)

Problem space

Single State Associative Instructive feedback Averaging Supervised learning Evaluative feedback Bandits

(Function optimization)

Problem space

Single State Associative Instructive feedback Averaging Supervised learning Evaluative feedback Bandits

(Function optimization)

Associative Search

(Contextual bandits)