The multi armed-bandit problem (with covariates if we have time) - PowerPoint PPT Presentation

The multi armed-bandit problem (with covariates if we have time) Vianney Perchet & Philippe Rigollet LPMA ORFE Université Paris – Diderot Princeton University Algorithms and Dynamics for Games and Optimization October, 14-18th 2013

Introduction Introduction Boring and useless definitions: Bandits: Optimization of a noisy function. – Observations: f ( x ) + ε x where ε x is random variable – Statistics : lack of information (exploration) – Optimization : maximize f ( · ) (exploitation) – Games : cumulative loss/payoff/reward Covariates: Some additional side observations gathered Start "easy": f is maximized over a finite set Concrete, simple and understandable examples follow.

Introduction Some real world examples

Introduction Some real world examples

Introduction Some real world examples

Introduction Simplified decision problem of Google Different firms go to Google and offer if you put my ad after the keywords "Flat Rental Paris", every time a customer clicks on it, I’ll give you b i ’s euros A given ad i has some exogenous but unknown probability of being clicked p i . Displaying ad i gives in expectation p i . b i to Google. Objective of Google... maximize cumulated payoff as fast as possible.

Introduction Simplified decision problem of Google Different firms go to Google and offer if you put my ad after the keywords "Flat Rental Paris", every time a customer clicks on it, I’ll give you b i ’s euros A given ad i has some exogenous but unknown probability of being clicked p i . Displaying ad i gives in expectation p i . b i to Google. Objective of Google... maximize cumulated payoff as fast as possible. Difficulties: The expected revenue of an ad i is unknown; p i cannot be estimated if ad i is not displayed. Take risk and display new ads (to compute new and maybe high p i ) or be safe and display the best estimated ad ?

Static case Framework Static Case Successive Elimination (SE) Static bandit – No queries Structure of a specific instance Decision set: { 1 , . . . , K } (the set of "arms" ... ads). Expected payoff of arm k : f k ∈ [ 0 , 1 ] . Best ad ⋆ , f ⋆ . Problem difficulty: ∆ k = f ⋆ − f k , ∆ min = min ∆ k > 0 ∆ k Repeated decision problem. At stage t ∈ N , Choose k t ∈ { 1 , . . . , K } , receive Y t ∈ [ 0 , 1 ] i.i.d. expectation f k t Observe only the payoff Y t (and not f k t ) and move to stage t + 1 Objectives: maximize cumulative expected payoff or Minimize regret: R T = T . f ⋆ − � T t = 1 f k t = � T t = 1 ∆ k t Choose the quickest possible the best decision with noise.

Static case Framework Static Case Successive Elimination (SE) Static Case: UCB log ( T ∆ 2 min ) with ∆ min = min f ⋆ − f k Lower bound for K=2: R T ≥ ✷ ∆ min Famous algo: U pper C onfidence B ound (and its variants) log ( T ) ≤ 8 K log ( T ) Using UCB, E [ R T ] ≤ 8 � k ∆ k ∆ min

Static case Framework Static Case Successive Elimination (SE) Static Case: UCB log ( T ∆ 2 min ) with ∆ min = min f ⋆ − f k Lower bound for K=2: R T ≥ ✷ ∆ min Famous algo: U pper C onfidence B ound (and its variants) – Draw each arm 1 , .., K once and observe Y 1 1 , .., Y K K (Round 1) – After stage t , compute the following: t k = ♯ { τ ≤ t ; k τ = k } the number of times arm k was drawn; t = 1 Y k ¯ � Y k τ an estimate of f k t k τ ≤ t ; k τ = k � – Draw the arm k t + 1 = arg max k ¯ 2 log ( t ) Y k t + t k log ( T ) ≤ 8 K log ( T ) Using UCB, E [ R T ] ≤ 8 � k ∆ k ∆ min

Static case Framework Static Case Successive Elimination (SE) Remarks on UCB log ( T ∆ 2 min ) Lower bound for K=2: R T ≥ ✷ , ∆ min = min ∆ k > 0 ∆ k ∆ min UCB algo: – Draw each arm 1 , .., K once and observe Y 1 1 , .., Y K K (Round 1) � – Draw the arm k t + 1 = arg max k ¯ 2 log ( t ) Y k t + t k log ( T ) ≤ 8 K log ( T ) UCB Upper bound: E [ R T ] ≤ 8 � k ∆ k ∆ min Remarks: – Proof based on Hoeffding inequality; – Not intuitive: clearly suboptimal arms keep being drawn – MOSS, a variant of UCB, achieves E [ R T ] ≤ ✷ K log ( T ∆ 2 min / K ) ∆ min – Neither log ( T ) or K log ( T ∆ 2 min / K ) sufficient with covariates.

Static case Framework Static Case Successive Elimination (SE) Successive Elimination (SE) Simple policy based on the intuition: Determine the suboptimal arms, and do not play them. Time is divided in rounds n ∈ N : – after round n : eliminate arms (with great proba.) suboptimal � � i.e., arm k s.t. ¯ 2 log ( T / n ) ≤ ¯ 2 log ( T / n ) Y k Y k ′ n + n − n n – at round n + 1: draw each remaining arm once. Easy to describe, to understand (but not to analyse for K > 2...) , intuitive. Simple confidence term (but requires knowledge of T ). (SE) is a variant of Even-Dar et al. (’06) Auer and Ortner (’10)

Static case Framework Static Case Successive Elimination (SE) Regret of successive elimination Theorem [P. and Rigollet (’13)] Played on K arms, the (SE) policy satisfies �� � log ( T ∆ 2 k ) � E [ R T ] ≤ ✷ min , TK log ( K ) ∆ k k ∆ k , MOSS: K log ( T ∆ 2 min / K ) log ( T ) UCB: � k ∆ min E [ R T ] = � k ∆ k . E [ n k ] with n k the number of draws of arm k Exact bound:

Static case Framework Static Case Successive Elimination (SE) Regret of successive elimination Theorem [P. and Rigollet (’13)] Played on K arms, the (SE) policy satisfies �� � log ( T ∆ 2 k ) � E [ R T ] ≤ ✷ min , TK log ( K ) ∆ k k ∆ k , MOSS: K log ( T ∆ 2 min / K ) log ( T ) UCB: � k ∆ min E [ R T ] = � k ∆ k . E [ n k ] with n k the number of draws of arm k Exact bound: � � � T ∆ 2 1 � �� � k � E [ R T ] ≤ min 646 log max 18 , e , 166 TK log ( K ) ∆ k k

Static case Framework Static Case Successive Elimination (SE) Successive Elimination: Example Two arms A round: a draw of both arms f 1 f 2

Static case Framework Static Case Successive Elimination (SE) Successive Elimination: Example Two arms A round: a draw of both arms f 1 ¯ Y 1 1 � 2 log ( T / 1 ) 1 ¯ Y 2 Round 1: no elimination 1 f 2

Static case Framework Static Case Successive Elimination (SE) Successive Elimination: Example Two arms � 2 log ( T / 2 ) ¯ Y 1 2 A round: a draw of both arms 2 f 1 ¯ Y 2 2 Round 1: no elimination f 2 Round 2: no elimination

Static case Framework Static Case Successive Elimination (SE) Successive Elimination: Example Two arms � 2 log ( T / 3 ) A round: a draw of both arms 3 f 1 ¯ Y 1 3 ¯ Y 2 3 Round 1: no elimination f 2 Round 2: no elimination Round 3: elimination

Static case Framework Static Case Successive Elimination (SE) Sketch of proof with K = 2 Basic idea: arm 2 (subopt.) eliminated at the first round n s.t.: � � 2log ( T / n ) 2log ( T / n ) ¯ ≤ ¯ Y 2 Y 1 n + n − n n

Static case Framework Static Case Successive Elimination (SE) Sketch of proof with K = 2 Basic idea: arm 2 (subopt.) eliminated at the first round n s.t.: � � 2log ( T / n ) 2log ( T / n ) f 2 + ≤ f 1 − n n

Static case Framework Static Case Successive Elimination (SE) Sketch of proof with K = 2 Basic idea: arm 2 (subopt.) eliminated at the first round n s.t.: � 2log ( T / n ) f 1 − f 2 = ∆ 2 ≥ 2 n

Static case Framework Static Case Successive Elimination (SE) Sketch of proof with K = 2 Basic idea: arm 2 (subopt.) eliminated at the first round n s.t.: � n 2 ≤ ✷ log ( T ∆ 2 2log ( T / n ) 2 ) f 1 − f 2 = ∆ 2 ≥ 2 ∆ 2 n 2

Static case Framework Static Case Successive Elimination (SE) Sketch of proof with K = 2 Basic idea: arm 2 (subopt.) eliminated at the first round n s.t.: � n 2 ≤ ✷ log ( T ∆ 2 2log ( T / n ) 2 ) f 1 − f 2 = ∆ 2 ≥ 2 ∆ 2 n 2 What could go wrong: Arm 1 eliminated before round n 2 � � � 2log ( T / n ) ≤ ✷ n 2 ∃ n ≤ n 2 , ¯ n − ¯ Y 1 Y 2 n ≤ − 2 P n T Arm 2 not eliminated at round n 2 .

Static case Framework Static Case Successive Elimination (SE) Sketch of proof with K = 2 Basic idea: arm 2 (subopt.) eliminated at the first round n s.t.: � n 2 ≤ ✷ log ( T ∆ 2 2log ( T / n ) 2 ) f 1 − f 2 = ∆ 2 ≥ 2 ∆ 2 n 2 What could go wrong: Arm 1 eliminated before round n 2 (with proba. ≤ ✷ n 2 T ) � � � 2log ( T / n ) ≤ ✷ n 2 ∃ n ≤ n 2 , ¯ n − ¯ Y 1 Y 2 n ≤ − 2 P n T Arm 2 not eliminated at round n 2 .

Static case Framework Static Case Successive Elimination (SE) Sketch of proof with K = 2 Basic idea: arm 2 (subopt.) eliminated at the first round n s.t.: � n 2 ≤ ✷ log ( T ∆ 2 2log ( T / n ) 2 ) f 1 − f 2 = ∆ 2 ≥ 2 ∆ 2 n 2 What could go wrong: Arm 1 eliminated before round n 2 (with proba. ≤ ✷ n 2 T ) Arm 2 not eliminated at round n 2 . � � � 2log ( T / n ) ∀ n ≤ n 2 , ¯ n − ¯ Y 2 Y 1 n ≥ − 2 P n

Static case Framework Static Case Successive Elimination (SE) Sketch of proof with K = 2 Basic idea: arm 2 (subopt.) eliminated at the first round n s.t.: � n 2 ≤ ✷ log ( T ∆ 2 2log ( T / n ) 2 ) f 1 − f 2 = ∆ 2 ≥ 2 ∆ 2 n 2 What could go wrong: Arm 1 eliminated before round n 2 (with proba. ≤ ✷ n 2 T ) Arm 2 not eliminated at round n 2 .   � 2log ( T / n 2 )  ¯ Y 2 n 2 − ¯ Y 1 P n 2 ≥ − 2  n 2