On conditional versus marginal bias in multi-armed bandits Jaehyeok - - PowerPoint PPT Presentation

On conditional versus marginal bias in multi-armed bandits

Jaehyeok Shin1, Aaditya Ramdas1,2 and Alessandro Rinaldo1

• Dept. of Statistics and Data Science1,

Machine Learning Dept.2, CMU

Stochastic Multi-armed bandits (MABs)

2

μK μ1 μ2

. . .

Y ∼

"Random reward"

μK μ1

. . .

Time

Adaptive sampling scheme to maximize rewards / to identify the best arm

μ2

3

μK μ1

. . .

Time

t = 1

Adaptive sampling scheme to maximize rewards / to identify the best arm

μ2

3

μK μ1

. . .

Time

t = 1

Adaptive sampling scheme to maximize rewards / to identify the best arm

μ2

3

μK μ1

. . .

Y1

Time

t = 1

Adaptive sampling scheme to maximize rewards / to identify the best arm

μ2

3

μK μ1

. . .

t = 2 Y1

Time

t = 1

Adaptive sampling scheme to maximize rewards / to identify the best arm

μ2

3

μK μ1

. . .

t = 2 Y1

Time

t = 1

Adaptive sampling scheme to maximize rewards / to identify the best arm

μ2

3

μK μ1

. . .

Y2 t = 2 Y1

Time

t = 1

Adaptive sampling scheme to maximize rewards / to identify the best arm

μ2

3

μK μ1

. . .

⋮ Y2 t = 2 Y1

Time

t = 1

Adaptive sampling scheme to maximize rewards / to identify the best arm

μ2

3

μK μ1

. . .

⋮ Y2 t = 2 Y1

Time

t = 1

Adaptive sampling scheme to maximize rewards / to identify the best arm

μ2

𝒰

Stopping time

3

μK μ1

. . .

⋮ Y2 t = 2 Y1

Time

t = 1

Collected data can be used to identify an interesting arm...

μ2

𝒰

4

μK μ1

. . .

⋮ Y2 t = 2 Y1

Time

t = 1

Collected data can be used to identify an interesting arm...

μ2

𝒰

e.g., κ = arg max

k

̂ μk(𝒰)

4

μK μ1

. . .

⋮ Y2 t = 2 Y1

Time

t = 1

...and the data can be used to conduct statistical inferences.

μ2

𝒰 ̂ μκ(𝒰)

Sample mean at a stopping time 𝒰

5

• Q. Sign of the bias of sample mean?

Xu et al. [2013] : An informal argument why the sample mean is negatively biased for “optimistic” algorithms. Villar et al. [2015] : Demonstrate this negative bias in a simulation study motivated by using MAB for clinical trials.

𝔽 [ ̂ μκ(𝒰) − μκ] ≤ or ≥ 0?

6

• Q. Sign of the bias of sample mean?

Xu et al. [2013] : An informal argument why the sample mean is negatively biased for “optimistic” algorithms. Villar et al. [2015] : Demonstrate this negative bias in a simulation study motivated by using MAB for clinical trials.

𝔽 [ ̂ μκ(𝒰) − μκ] ≤ or ≥ 0?

6

Shin et al. [2019] Introduced "monotonicity property" characterizing the bias of the sample mean for more general classes of MABs. Chosen Arm Stopping Time

𝔽 [ ̂ μκ(𝒰) − μκ]

7

Nie et al. [2018] Sample mean is negatively biased Fixed Arm Fixed Time

𝔽 [ ̂ μk(t) − μk] ≤ 0

for MABs designed to maximize cumulative reward.

Shin et al. [2019] Introduced "monotonicity property" characterizing the bias of the sample mean for more general classes of MABs. Chosen Arm Stopping Time

𝔽 [ ̂ μκ(𝒰) − μκ]

8

Nie et al. [2018] Sample mean is negatively biased Fixed Arm Fixed Time

𝔽 [ ̂ μk(t) − μk] ≤ 0

for MABs designed to maximize cumulative reward.

However, current understanding

• f bias is limited in two aspects.

9

• 1. Existing results concern the bias of the sample mean only.

However, current understanding

• f bias is limited in two aspects.

9

• 1. Existing results concern the bias of the sample mean only.

We study the bias of monotone functions of the rewards.

9

• 1. Existing results concern the bias of the sample mean only.

We study the bias of monotone functions of the rewards.

• 2. Existing guarantees cover only the marginal bias.

However, current understanding

• f bias is limited in two aspects.

9

• 1. Existing results concern the bias of the sample mean only.

We study the bias of monotone functions of the rewards.

• 2. Existing guarantees cover only the marginal bias.

We extend previous results to cover the conditional bias.

However, current understanding

• f bias is limited in two aspects.

Marginal vs Conditional bias

10

μK μ1 μ2

. . .

• K prototype items

Want to screen out some items by testing for

H0k : μk ≥ c

vs H1k : μk < c

k = 1,…, K .

Marginal vs Conditional bias

11

H0 : μ ≥ c vs H1 : μ < c

̂ μ(t) T

"Keep the item" "Screen out the item at "

𝒰

𝒰 ̂ μ(t)

(Fail to reject the null) (Reject the null)

Marginally, the sample mean is negatively biased.

12

Item 1

. . .

𝔽 [ ̂ μk − μk] ≤ 0, k = 1,…, K

(e.g. Starr & Woodroofe [1968], Shin et al. [2019])

𝒰 ̂ μ(t) ̂ μ(t) T

Item 2 Item K

𝒰 ̂ μ(t)

"Underestimating the true mean revenue."

13

Item 1

. . .

𝒰 ̂ μ(t) ̂ μ(t) T

Item 2 Item K

𝒰 ̂ μ(t)

...however, we usually do not evaluate the sample mean every time.

13

Item 1

. . .

𝒰 ̂ μ(t) ̂ μ(t) T

Item 2 Item K

𝒰 ̂ μ(t)

...however, we usually do not evaluate the sample mean every time.

13

𝔽 [ ̂ μk − μk ∣ item k is active] ≥ 0, k = 1,…, K

Item 1

. . .

𝒰 ̂ μ(t) ̂ μ(t) T

Item 2 Item K

𝒰 ̂ μ(t)

Conditioned on the "active" event, the sample mean is positively biased.

"Overestimating the true mean revenue."

e.g., C = {Reject the null}, {Chosen as the best arm} . . . . ̂ F k,𝒰 : Empirical CDF of arm k at time 𝒰 Fk : True CDF of arm k at time 𝒰 where

14

Conditional bias of the empirical cumulative distribution function (CDF)

For a fixed y ∈ ℝ,

𝔽 [ ̂ F k,𝒰(y) − Fk(y) ∣ C] ≤ or ≥ 0?

Tabular model of MABs

μK μ1 μ2

. . .

X*

1,1

X*

1,2

X*

1,K

. . .

X*

2,1

X*

2,2

X*

2,K

. . .

⋮ ⋮ ⋮ ⋮

}

X*

∞ ∈ ℝℕ×K

:= ∼

i.i.d.

∼

i.i.d.

∼

i.i.d.

"Hypothetical table"

15

μK μ1 μ2

. . .

X*

1,1

X*

1,2

X*

1,K

. . .

X*

2,1

X*

2,2

X*

2,K

. . .

⋮ ⋮ ⋮ ⋮

Tabular model of MABs

Time

16

μK μ1 μ2

. . . Time

X*

1,1

X*

1,2

X*

1,K

. . .

X*

2,1

X*

2,2

X*

2,K

. . .

⋮ ⋮ ⋮ ⋮ t = 1

Tabular model of MABs

16

μK μ1 μ2

. . . Time

X*

1,1

Y1 X*

1,K

. . .

X*

2,1

X*

2,2

X*

2,K

. . .

⋮ ⋮ ⋮ ⋮ t = 1

Tabular model of MABs

16

μK μ1 μ2

. . . Time

X*

1,1

Y1 X*

1,K

. . .

X*

2,1

X*

2,2

X*

2,K

. . .

⋮ ⋮ ⋮ ⋮ t = 1 t = 2

Tabular model of MABs

16

μK μ1 μ2

. . . Time

X*

1,1

Y1 X*

1,K

. . .

X*

2,1

X*

2,2

X*

2,K

. . .

⋮ ⋮ ⋮ ⋮ t = 1 t = 2

Tabular model of MABs

16

μK μ1 μ2

. . . Time

X*

1,1

Y1 X*

1,K

. . .

Y2 X*

2,2

X*

2,K

. . .

⋮ ⋮ ⋮ ⋮ t = 1 t = 2

Tabular model of MABs

16

Hypothetical dataset

𝒠*

∞ = X* ∞ ∪ {Wt}∞ t=1

Hypothetical table Random seeds

17

Hypothetical dataset

, and for each and can be expressed as some functions of .

C 𝒰 Nk(t) t k 𝒠*

∞

Given 𝒠*

∞ = X* ∞ ∪ {Wt}∞ t=1

18

Monotone effect of a sample

(Negative conditional bias of the empirical CDF) 𝔽 [ ̂ F k,𝒰(y) − Fk(y) ∣ C] ≤ 0

19

Theorem Suppose arm has a finite mean. If is an increasing function of each while keeping all other entries in fixed then we have

k

1(C) Nk(𝒰)

X*

i,k

𝒠*

∞

Monotone effect of a sample

(Negative conditional bias of the empirical CDF) 𝔽 [ ̂ F k,𝒰(y) − Fk(y) ∣ C] ≤ 0 (Positive conditional bias of the sample mean) 𝔽 [ ̂ μk(𝒰) − μk ∣ C] ≥ 0

19

Theorem Suppose arm has a finite mean. If is an increasing function of each while keeping all other entries in fixed then we have

k

1(C) Nk(𝒰)

X*

i,k

𝒠*

∞

Monotone effect of a sample

(Positive conditional bias of the empirical CDF) 𝔽 [ ̂ F k,𝒰(y) − Fk(y) ∣ C] ≥ 0

20

Theorem Suppose arm has a finite mean. If is a decreasing function of each while keeping all other entries in fixed then we have

k

1(C) Nk(𝒰)

X*

i,k

𝒠*

∞

Monotone effect of a sample

(Positive conditional bias of the empirical CDF) 𝔽 [ ̂ F k,𝒰(y) − Fk(y) ∣ C] ≥ 0 (Negative conditional bias

• f the sample mean)

𝔽 [ ̂ μk(𝒰) − μk ∣ C] ≤ 0

20

Theorem Suppose arm has a finite mean. If is a decreasing function of each while keeping all other entries in fixed then we have

k

1(C) Nk(𝒰)

X*

i,k

𝒠*

∞

E.g.: Best arm identification

21

μK μ1 μ2

. . .

• K prototype items

Want to figure out which one has the largest revenue.

lil' UCB algorithm

μK μ1

. . .

Y2 t = 2 Y1

Time

t = 1

μ2

22

E.g.: Best arm identification

lil' UCB algorithm

μK μ1

. . .

Y2 t = 2 Y1

Time

t = 1

μ2

22

E.g.: Best arm identification

At = arg max

k

̂ μk(t) + u(Nk(t))

(Upper confidence bound)

lil' UCB algorithm

μK μ1

. . .

Y2 t = 2 Y1

Time

t = 1

μ2

𝒰 = inf t : ∃k, Nk(t) ≥ 1 + λ∑

j≠k

Nj(t)

22

E.g.: Best arm identification

At = arg max

k

̂ μk(t) + u(Nk(t))

(Upper confidence bound)

lil' UCB algorithm

μK μ1

. . .

Y2 t = 2 Y1

Time

t = 1

μ2

𝒰 = inf t : ∃k, Nk(t) ≥ 1 + λ∑

j≠k

Nj(t) κ = arg max

k

Nk(𝒰)

22

E.g.: Best arm identification

At = arg max

k

̂ μk(t) + u(Nk(t))

(Upper confidence bound)

lil' UCB algorithm

μK μ1

. . .

Y2 t = 2 Y1

Time

t = 1

μ2

23

E.g.: Best arm identification

a) Item 1 is chosen as the best. b) Item 1 is NOT chosen as the best.

a) Item 1 is chosen as the best ( ).

κ = 1

Sample from item

1

1(κ = 1) N1(𝒰)

Increasing

μK μ1

. . .

Y2 t = 2 Y1

Time

t = 1

μ2

At = arg max

k

̂ μk(t) + u(Nk(t)) 𝒰 = inf t : ∃k, Nk(t) ≥ 1 + λ∑

j≠k

Nj(t) κ = arg max

k

Nk(𝒰)

24

E.g.: Best arm identification

a) Item 1 is chosen as the best ( ).

κ = 1

Sample from item

1

1(κ = 1) N1(𝒰)

Increasing

Negative conditional bias of the empirical CDF

μK μ1

. . .

Y2 t = 2 Y1

Time

t = 1

μ2

At = arg max

k

̂ μk(t) + u(Nk(t)) 𝒰 = inf t : ∃k, Nk(t) ≥ 1 + λ∑

j≠k

Nj(t) κ = arg max

k

Nk(𝒰)

24

E.g.: Best arm identification

a) Item 1 is chosen as the best ( ).

κ = 1

Sample from item

1

1(κ = 1) N1(𝒰)

Increasing

Negative conditional bias of the empirical CDF Positive conditional bias of the sample mean

μK μ1

. . .

Y2 t = 2 Y1

Time

t = 1

μ2

At = arg max

k

̂ μk(t) + u(Nk(t)) 𝒰 = inf t : ∃k, Nk(t) ≥ 1 + λ∑

j≠k

Nj(t) κ = arg max

k

Nk(𝒰)

24

E.g.: Best arm identification

Sample from item

1

1(κ ≠ 1) N1(𝒰)

Decreasing

μK μ1

. . .

Y2 t = 2 Y1

Time

t = 1

μ2

At = arg max

k

̂ μk(t) + u(Nk(t)) 𝒰 = inf t : ∃k, Nk(t) ≥ 1 + λ∑

j≠k

Nj(t) κ = arg max

k

Nk(𝒰)

b) Item 1 is NOT chosen as the best ( ).

κ ≠ 1

25

E.g.: Best arm identification

Sample from item

1

1(κ ≠ 1) N1(𝒰)

Decreasing

Positive conditional bias of the empirical CDF

μK μ1

. . .

Y2 t = 2 Y1

Time

t = 1

μ2

At = arg max

k

̂ μk(t) + u(Nk(t)) 𝒰 = inf t : ∃k, Nk(t) ≥ 1 + λ∑

j≠k

Nj(t) κ = arg max

k

Nk(𝒰)

b) Item 1 is NOT chosen as the best ( ).

κ ≠ 1

25

E.g.: Best arm identification

Sample from item

1

1(κ ≠ 1) N1(𝒰)

Decreasing

Positive conditional bias of the empirical CDF Negative conditional bias

• f the sample mean

μK μ1

. . .

Y2 t = 2 Y1

Time

t = 1

μ2

At = arg max

k

̂ μk(t) + u(Nk(t)) 𝒰 = inf t : ∃k, Nk(t) ≥ 1 + λ∑

j≠k

Nj(t) κ = arg max

k

Nk(𝒰)

b) Item 1 is NOT chosen as the best ( ).

κ ≠ 1

25

E.g.: Best arm identification

26

Average of the empirical CDF of item 1 conditioned on each event

0.00 0.25 0.50 0.75 1.00 −2 2 4

y CDF of item 1

Item 1 is chosen Item 2 is chosen Item 3 is chosen

Mean bias = (0.2, −0.93, −1.14)

lilʹUCB on 3 items (µ1 = 1)

On conditional versus marginal bias in multi-armed bandits

Jaehyeok Shin, Aaditya Ramdas and Alessandro Rinaldo

Thank you!