Garbage In, Reward Out: Bootstrapping Exploration in Multi-Armed - - PowerPoint PPT Presentation

Garbage In, Reward Out: Bootstrapping Exploration in Multi-Armed Bandits

Branislav Kveton, Google Research Csaba Szepesvári, DeepMind and University of Alberta Sharan Vaswani, Mila, University of Montreal Zheng Wen, Adobe Research Mohammad Ghavamzadeh, Facebook AI Research Tor Lattimore, DeepMind

• Learning agent sequentially pulls K arms in n rounds
• The agent pulls arm It in round t ∈ [n] and observes its reward
• Reward of arm i is in [0, 1] and drawn i.i.d. from a distribution with mean μi
• Goal: Maximize the expected n-round reward
• Challenge: Exploration-exploitation trade-off

Stochastic Multi-Armed Bandit

…

Arm 1 Arm 2 Arm K

Thompson Sampling (Thompson, 1933)

• Sample μi,t from posterior distribution Pi,t and pull arm It = argmaxi μi,t

μ1 P1,t μ2 P2,t Expected reward

Thompson Sampling (Thompson, 1933)

• Sample μi,t from posterior distribution Pi,t and pull arm It = argmaxi μi,t
• Key properties

○ Pi,t concentrates at μi with the number of pulls

μ1 P1,t μ2 P2,t Expected reward

Thompson Sampling (Thompson, 1933)

• Sample μi,t from posterior distribution Pi,t and pull arm It = argmaxi μi,t
• Key properties

○ Pi,t concentrates at μi with the number of pulls ○ μi,t overestimates μi with a sufficient probability

μ1 P1,t μ2 P2,t Expected reward

Thompson Sampling (Thompson, 1933)

• Sample μi,t from posterior distribution Pi,t and pull arm It = argmaxi μi,t
• Key properties

○ Pi,t concentrates at μi with the number of pulls ○ μi,t overestimates μi with a sufficient probability

Bernoulli bandit Pi,t = beta Gaussian bandit Pi,t = normal Neural network Pi,t = ??? μ1 P1,t μ2 P2,t Expected reward

General Randomized Exploration

• Sample μi,t from posterior distribution Pi,t and pull arm It = argmaxi μi,t
• Key properties

○ Pi,t concentrates at (scaled and shifted) μi with the number of pulls ○ μi,t overestimates (scaled and shifted) μi with a sufficient probability

How do we design distribution Pi,t? μ1 P1,t μ2 P2,t Expected reward

• μi,t is the mean of a non-parametric bootstrap sample of the history of arm i

with pseudo-rewards (garbage)

Giro (Garbage In, Reward Out) with [0, 1] Rewards

• μi,t is the mean of a non-parametric bootstrap sample of the history of arm i

with pseudo-rewards (garbage)

Giro (Garbage In, Reward Out) with [0, 1] Rewards

History Arm 1 Arm 2 1 1

• μi,t is the mean of a non-parametric bootstrap sample of the history of arm i

with pseudo-rewards (garbage)

Giro (Garbage In, Reward Out) with [0, 1] Rewards

History Garbage Arm 1 Arm 2 1 1 1 1 1 1 1

• μi,t is the mean of a non-parametric bootstrap sample of the history of arm i

with pseudo-rewards (garbage)

Giro (Garbage In, Reward Out) with [0, 1] Rewards

History Garbage Bootstrap sample Arm 1 Arm 2 μi,t 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 / 3 5 / 9

• μi,t is the mean of a non-parametric bootstrap sample of the history of arm i

with pseudo-rewards (garbage)

• Benefits and challenges of randomized garbage

○ μi,t overestimates scaled and shifted μi with a sufficient probability ○ Bias in the estimate of μi

Giro (Garbage In, Reward Out) with [0, 1] Rewards

History Garbage Bootstrap sample Arm 1 Arm 2 μi,t 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 / 3 5 / 9

Contextual Giro with [0, 1] Rewards

• Straightforward generalization to complex structured problems
• μi,t is the estimated reward of arm i in a model trained on a non-parametric

bootstrap sample of the history with pseudo-rewards (garbage)

• Giro is as general as the ε-greedy policy... but no tuning!

History Garbage Bootstrap sample μi,t (x1, ) Estimate from a learned model (x2, ) 1 (x3, ) (x1, ) (x2, ) (x3, ) (x1, ) 1 (x2, ) 1 (x3, ) 1 (x1, ) (x3, ) (x1, ) 1 (x2, ) 1 (x3, ) 1 (x2, ) (x2, ) 1 (x3, ) 1 (x2, ) 1

See you at poster #125!

How to do bandits with neural networks easily? How does Giro compare to Thompson sampling?