Online Learning with Kernel Losses Aldo Pacchiano UC Berkeley - - PowerPoint PPT Presentation

Online Learning with Kernel Losses

Aldo Pacchiano UC Berkeley Joint work with Niladri Chatterji and Peter Bartlett

1

Talk Overview

• Intro to Online Learning
• Linear Bandits
• Kernel Bandits

2

Online Learning

3

Online Learning

3

Learner Adversary

t = 1, · · · , n

Online Learning

Learner chooses an action at ∈ A

3

Learner Adversary

t = 1, · · · , n

Online Learning

Learner chooses an action at ∈ A Adversary reveals loss (or reward) `t ∈ W

3

Learner Adversary

t = 1, · · · , n

Online Learning

Learner chooses an action at ∈ A Adversary reveals loss (or reward)

Can be i.i.d or adversarial

`t ∈ W

3

Learner Adversary

t = 1, · · · , n

Online Learning

Learner chooses an action at ∈ A Adversary reveals loss (or reward)

Can be i.i.d or adversarial

`t ∈ W

3

Learner Adversary

n

X

t=1

`t(at) t = 1, · · · , n

Online Learning

Learner chooses an action at ∈ A Adversary reveals loss (or reward)

Can be i.i.d or adversarial

`t ∈ W

3

Learner Adversary

R(n) =

n

X

t=1

`t(at) − min

a∗∈A n

X

t=1

`t(at) t = 1, · · · , n

Online Learning

Learner chooses an action The learner’s objective is to minimize Regret at ∈ A Adversary reveals loss (or reward)

Can be i.i.d or adversarial

`t ∈ W

3

Learner Adversary

R(n) =

n

X

t=1

`t(at) − min

a∗∈A n

X

t=1

`t(at) t = 1, · · · , n

Full information vs Bandit feedback

4

Full information vs Bandit feedback

Full Information: Learner gets to sees all of

`t(·)

4

Full information vs Bandit feedback

Full Information: Bandit Feedback: Learner gets to sees all of Learner only sees the value

`t(·)

`t(at)

4

Full information vs Bandit feedback

Full Information: Bandit Feedback: Learner gets to sees all of Learner only sees the value

`t(·)

`t(at)

4

Multi Armed Bandits

5

P1 µ1 P2 P3 µ2 µ3

Multi Armed Bandits

at ∈ {1, · · · , K}

5

Learner chooses P1 µ1 P2 P3 µ2 µ3

Multi Armed Bandits

Xat ∼ Pat at ∈ {1, · · · , K}

5

Learner chooses P1 µ1 Gets reward P2 P3 µ2 µ3

Multi Armed Bandits

Xat ∼ Pat at ∈ {1, · · · , K}

5

Learner chooses P1 µ1 Gets reward P2 P3 µ2 µ3 R(n) = max

a∗∈{1,···K} nµa∗ − E

" n X

t=1

Xat #

Multi Armed Bandits

Xat ∼ Pat at ∈ {1, · · · , K}

5

Learner chooses P1 µ1 Gets reward P2 P3 µ2 µ3 R(n) = max

a∗∈{1,···K} nµa∗ − E

" n X

t=1

Xat # R(n) = O( p Kn log(n)) MAB regret

[Auer et al. 2002]

Structured losses

6

Arms = Paths MAB regret Exponential Packet routing Network

(V, E) at ∈ A ⊂ {0, 1}E wt ∈ W = [0, 1]E

Loss = delay

hat, wti

Delay is linear

R(n) = O( p |num paths| · n log(n))

Structured losses

6

Arms = Paths MAB regret Exponential Packet routing Network

(V, E) at ∈ A ⊂ {0, 1}E wt ∈ W = [0, 1]E

Loss = delay

hat, wti

Delay is linear

R(n) = O( p |num paths| · n log(n))

Linear Bandits

7

Linear Bandits

Learner chooses an action at ∈ A ⊂ Rd

7

Linear Bandits

Learner chooses an action at ∈ A ⊂ Rd Adversary’s loss `t(a) = hwt, ai for wt ∈ W ⊂ Rd

7

Linear Bandits

Learner chooses an action at ∈ A ⊂ Rd Adversary’s loss `t(a) = hwt, ai for

Can be i.i.d or adversarial

wt ∈ W ⊂ Rd

7

Linear Bandits

Learner chooses an action at ∈ A ⊂ Rd Adversary’s loss `t(a) = hwt, ai for Learner only experienceshwt, ati

Can be i.i.d or adversarial

wt ∈ W ⊂ Rd

7

Linear Bandits

Learner chooses an action at ∈ A ⊂ Rd Adversary’s loss `t(a) = hwt, ai for Learner only experienceshwt, ati

Can be i.i.d or adversarial

wt ∈ W ⊂ Rd

7

Expected regret: R(n) = E " n X

t=1

hwt, ati inf

a∈A n

X

t=1

hwt, ai #

Linear Bandits

Learner chooses an action at ∈ A ⊂ Rd Adversary’s loss `t(a) = hwt, ai for Learner only experienceshwt, ati

Can be i.i.d or adversarial

wt ∈ W ⊂ Rd

MAB reduces to Linear Bandits

7

A = {e1, · · · , ed} W = [0, 1]d

Expected regret: , R(n) = E " n X

t=1

hwt, ati inf

a∈A n

X

t=1

hwt, ai #

Exponential weights for adversarial linear bandits

8

Exponential weights for adversarial linear bandits

8

For t = 1, · · · , n : Sample mixture at ∼ pt = (1 − γ)qt | {z }

Exploitation

+ γν |{z}

Exploration

Exponential weights for adversarial linear bandits

8

For t = 1, · · · , n : Sample mixture at ∼ pt = (1 − γ)qt | {z }

Exploitation

+ γν |{z}

Exploration

Exponential weights for adversarial linear bandits

8

For t = 1, · · · , n : Sample mixture at ∼ pt = (1 − γ)qt | {z }

Exploitation

+ γν |{z}

Exploration

Exponential weights for adversarial linear bandits

8

For t = 1, · · · , n : See hwt, ati Sample mixture at ∼ pt = (1 − γ)qt | {z }

Exploitation

+ γν |{z}

Exploration

Exponential weights for adversarial linear bandits

8

For t = 1, · · · , n : See hwt, ati Sample mixture at ∼ pt = (1 − γ)qt | {z }

Exploitation

+ γν |{z}

Exploration

Build loss estimator ˆ wt

Exponential weights for adversarial linear bandits

8

For t = 1, · · · , n : See hwt, ati Sample mixture at ∼ pt = (1 − γ)qt | {z }

Exploitation

+ γν |{z}

Exploration

Update Build loss estimator ˆ wt qt(a) / exp(ηh ˆ wt, ai)qt−1(a) | {z }

Exponential weights

Exponential weights

9

qt(a) / exp(ηh

t

X

i=1

ˆ wi, ai)

Exponential weights

9

A

t

X

i=1

ˆ wi

qt(a) / exp(ηh

t

X

i=1

ˆ wi, ai)

Exponential weights

9

A A

t

X

i=1

ˆ wi

qt

qt(a) / exp(ηh

t

X

i=1

ˆ wi, ai)

Unbiased estimator of the loss

Σt = Ea⇠pt ⇥ aa>⇤ ˆ wt = (Σt)−1 athwt, ati Let and set

10

Unbiased estimator of the loss

Σt = Ea⇠pt ⇥ aa>⇤ ˆ wt = (Σt)−1 athwt, ati Let and set

10

Unbiased estimator of the loss

Σt = Ea⇠pt ⇥ aa>⇤ ˆ wt = (Σt)−1 athwt, ati Let and set is an unbiased estimator of :

ˆ wt wt

10

Unbiased estimator of the loss

Σt = Ea⇠pt ⇥ aa>⇤ ˆ wt = (Σt)−1 athwt, ati Let and set is an unbiased estimator of :

Eat⇠pt [ ˆ wt|Ft1] =

• Ea⇠pt

⇥ aaT ⇤1 Eat⇠pt [athwt, ati|Ft1] =

• Ea⇠pt

⇥ aaT ⇤1 Eat⇠pt ⇥ ata>

t |Ft1

⇤ wt = wt ˆ wt wt

10

Linear bandits regret

11

• Theorem. (Linear Bandits Regret).

R(n)  γn + log(|A|) η + η

n

X

t=1

EEa∼pt(h ˆ wt, ai)2

[See for example Bubeck ‘11]

Uniform over , [Cesa-Bianchi, Lugosi, ’12] John’s distribution [Bubeck, Cesa-Bianchi, Kakade ’12]

A

O(d√n)

O( p dn log(|A|)) = O(d√n)

Exploration over Barycentric Spanner, [Dani, Hayes, Kakade ’08]

O(d p n log(|A|)) = O(d3/2√n)

Linear bandits regret Dimension dependence

Variance bound: E ⇥ Eat∼pt ⇥ (h ˆ wt, ai)2⇤⇤  d

12

Linear bandits regret Dimension dependence

Variance bound: E ⇥ Eat∼pt ⇥ (h ˆ wt, ai)2⇤⇤  d Dimension dependence

12

Linear bandits regret Dimension dependence

Variance bound: E ⇥ Eat∼pt ⇥ (h ˆ wt, ai)2⇤⇤  d Dimension dependence

12

≤ ηdn R(n)  γn + log(|A|) η + η

n

X

t=1

EEa∼pt(h ˆ wt, ai)2 | {z }

Recap

13

• Intro to Online Learning
• Linear Bandits
• Kernel Bandits

Online Quadratic losses

Bt `t(a) = hbt, ai + a>Bta

Offline problem has polytime solution

14

min `t(a) a ∈ A

Symmetric and possibly non convex Strong Duality

z = x2 − .5 ∗ y2 + x ∗ y − .5 ∗ x + .5y + 1 Peter Bartlett Niladri Chatterji

at 2 A = {a s.t. kak2  1}

Linearization of Quadratic losses

Quadratic losses are linear in the space of `(a) = hbt, ai + a>Bta `(a) = ⌧✓Bt bt ◆ , ✓aa> a ◆ matrices vector

( )

We can use the linear bandits machinery Exponential weights for quadratic bandits

15

Exponential weights for adversarial quadratic bandits

16

For t = 1, · · · , n : See Sample mixture at ∼ pt = (1 − γ)qt | {z }

Exploitation

+ γν |{z}

Exploration

Update Build loss estimator hbt, ati + a>

t Btat ✓ ˆ Bt ˆ bt ◆

qt(a) / exp(η(hˆ bt, ai + a> ˆ Bta))qt1(a) | {z }

Exponential weights

Sampling is poly time

Beyond “Finite Dimensional” Losses

17

Evasion games: Gaussian kernel - `t(a) = exp(ka wtk2) Infinite dimensional Obstacle avoidance

Space of Quadratics as a Reproducing Kernel Hilbert Space

The Reproducing Kernel Hilbert Space of HK Quadratics losses lie in an RKHS K(x, y) = hx, yi + (hx, yi)2 K Feature map

x → Φ(x) ∈ RD

Dot product

K(x, y) = hΦ(x), Φ(y)i

18

Kernel Bandits

19

Kernel Bandits

If `t ∈ HK can we leverage linearity?

19

Kernel Bandits

If `t ∈ HK can we leverage linearity? Main challenge: Dimension of HK might be infinite. Naive Linear regret

19

O( p dn log(|A|))

Kernel Bandits

If `t ∈ HK can we leverage linearity? Main challenge: Dimension of HK might be infinite. Naive Linear regret Encouraging facts: Kernel spaces are “small”

19

O( p dn log(|A|))

Towards an Algorithm

Algorithm Strategy:

20

Towards an Algorithm

Algorithm Strategy: 1) Construct an dimensional proxy kernel that uniformly approximates the original kernel over .

20

Km(x, y) ≈ K(x, y)

m < ∞

A × A

Towards an Algorithm

Algorithm Strategy: 1) Construct an dimensional proxy kernel that uniformly approximates the original kernel over . II) Exponential weights using the proxy kernel. Control the bias.

20

Km(x, y) ≈ K(x, y)

m < ∞

A × A

K(wt, at)

Towards an Algorithm

Algorithm Strategy: 1) Construct an dimensional proxy kernel that uniformly approximates the original kernel over . II) Exponential weights using the proxy kernel. Control the bias.

20

Km(x, y) ≈ K(x, y) Receive

m < ∞

A × A

K(wt, at)

Towards an Algorithm

Algorithm Strategy: 1) Construct an dimensional proxy kernel that uniformly approximates the original kernel over . II) Exponential weights using the proxy kernel. Control the bias.

20

Km(x, y) ≈ K(x, y) Receive

m < ∞

A × A

K(wt, at)

Towards an Algorithm

Algorithm Strategy: 1) Construct an dimensional proxy kernel that uniformly approximates the original kernel over . II) Exponential weights using the proxy kernel. Control the bias.

20

Km(x, y) ≈ K(x, y) Km(wt, at) Receive Pretend it was

m < ∞

A × A

Kernel functions are “small”

Kernel function evaluations can be uniformly approximated by a small number of basis functions.

21

Kernel functions are “small”

Kernel function evaluations can be uniformly approximated by a small number of basis functions. Mercer’s Theorem

There exist functions

{φi}∞

i=1 and nonnegative values {µi}∞

i=1

K(x, y) =

∞

X

i=1

µiφi(x)φi(y)

such that:

∀x, y ∈ A × A

21

Eigendecay

22

K(x, y) =

∞

X

i=1

µiφi(x)φi(y)

Eigendecay

22

Gaussian Kernel Sobolev Kernel

K(x, y) = exp(kx yk2)

K(x, y) = min(x, y)

K(x, y) =

∞

X

i=1

µiφi(x)φi(y)

Eigendecay

22

Gaussian Kernel Sobolev Kernel

µj ≤ Ce−βj Exponential decay

{φj(x)} = {sin(jπx), cos(jπx)}

µj ≈ e−cj log(j)

K(x, y) = exp(kx yk2)

K(x, y) = min(x, y)

K(x, y) =

∞

X

i=1

µiφi(x)φi(y)

Eigendecay

22

Gaussian Kernel Sobolev Kernel

µj ≤ Ce−βj µj ≤ Cj−β Exponential decay Polynomial decay

{φj(x)} = {sin(jπx), cos(jπx)}

µj ≈ e−cj log(j)

K(x, y) = exp(kx yk2)

K(x, y) = min(x, y)

µj ≈ 1 j2

φj(x) ≈ sin ✓2jπx 2 ◆

K(x, y) =

∞

X

i=1

µiφi(x)φi(y)

Eigendecay

22

Gaussian Kernel Sobolev Kernel

µj ≤ Ce−βj µj ≤ Cj−β Exponential decay Polynomial decay

{φj(x)} = {sin(jπx), cos(jπx)}

µj ≈ e−cj log(j)

K(x, y) = exp(kx yk2)

K(x, y) = min(x, y)

µj ≈ 1 j2

φj(x) ≈ sin ✓2jπx 2 ◆

K(x, y) =

∞

X

i=1

µiφi(x)φi(y)

Construction of a finite dimensional Proxy Kernel

Eigenfunctions

{φi}∞

i=1 with eigenvalues {µi}∞

i=1

K(x, y) =

∞

X

i=1

µiφi(x)φi(y)

23

Construction of a finite dimensional Proxy Kernel

Eigenfunctions

{φi}∞

i=1 with eigenvalues {µi}∞

i=1

K(x, y) =

∞

X

i=1

µiφi(x)φi(y) Truncate at i = m

23

Construction of a finite dimensional Proxy Kernel

Eigenfunctions

{φi}∞

i=1 with eigenvalues {µi}∞

i=1

K(x, y) =

∞

X

i=1

µiφi(x)φi(y) Truncate at i = m Ko(x, y) =

m

X

i=1

µiφi(x)φi(y)

23

Construction of a finite dimensional Proxy Kernel

Eigenfunctions

{φi}∞

i=1 with eigenvalues {µi}∞

i=1

K(x, y) =

∞

X

i=1

µiφi(x)φi(y) Truncate at i = m Deterministic Proxy Kernel Ko(x, y) =

m

X

i=1

µiφi(x)φi(y)

23

Construction of a finite dimensional Proxy Kernel

Eigenfunctions

{φi}∞

i=1 with eigenvalues {µi}∞

i=1

K(x, y) =

∞

X

i=1

µiφi(x)φi(y) Truncate at i = m Deterministic Proxy Kernel Ko(x, y) =

m

X

i=1

µiφi(x)φi(y)

23

Building proxy Kernel with samples Kernel PCA

Exponential weights for adversarial kernel bandits

24

Exponential weights for adversarial kernel bandits

24

ˆ Km(x, y) = hΦm(x), Φm(y)i Build from P

Exponential weights for adversarial kernel bandits

24

For t = 1, · · · , n : Sample mixture at ∼ pt = (1 − γ)qt | {z }

Exploitation

+ γν |{z}

Exploration

ˆ Km(x, y) = hΦm(x), Φm(y)i Build from P

Exponential weights for adversarial kernel bandits

24

For t = 1, · · · , n : See Sample mixture at ∼ pt = (1 − γ)qt | {z }

Exploitation

+ γν |{z}

Exploration

ˆ Km(x, y) = hΦm(x), Φm(y)i Build from P K(wt, at)

Exponential weights for adversarial kernel bandits

24

For t = 1, · · · , n : See Sample mixture at ∼ pt = (1 − γ)qt | {z }

Exploitation

+ γν |{z}

Exploration

Build loss estimator ˆ Km(x, y) = hΦm(x), Φm(y)i Build from P K(wt, at) ˆ wt

Exponential weights for adversarial kernel bandits

24

For t = 1, · · · , n : See Sample mixture at ∼ pt = (1 − γ)qt | {z }

Exploitation

+ γν |{z}

Exploration

Update Build loss estimator ˆ Km(x, y) = hΦm(x), Φm(y)i Build from P K(wt, at) ˆ wt

qt(a) / exp(ηh ˆ wt, Φm(a)i)qt−1(a) | {z }

Exponential weights

Exponential weights for adversarial kernel bandits

24

For t = 1, · · · , n : See Sample mixture at ∼ pt = (1 − γ)qt | {z }

Exploitation

+ γν |{z}

Exploration

Update Build loss estimator

Sampling might not be poly time

ˆ Km(x, y) = hΦm(x), Φm(y)i Build from P K(wt, at) ˆ wt

qt(a) / exp(ηh ˆ wt, Φm(a)i)qt−1(a) | {z }

Exponential weights

Biased loss estimates

is biased:

Eat∼pt [ ˆ wt|Ft−1] = E h K(at, yt) ⇣ (Σ(t)

m )−1Φm(at)

⌘

• Ft−1

i = Φm(yt) + E 2 6 6 4 ⇣ K(at, yt) − ˆ Km(at, yt) ⌘ ⇣ (Σ(t)

m )−1Φm(at)

⌘ | {z }

=:ξt, the bias

• Ft−1

3 7 7 5

25

Let and set

Σ(t)

m = Ea⇠pt

⇥ Φm(a)Φm(a)>⇤

ˆ wt ˆ wt := K(at, yt) ⇣ (Σ(t)

m )−1Φm(at)

⌘

Kernel Bandits Regret

26

Expected regret

game vs

Km K game

ξt bias

R(n) = E " n X

t=1

K(wt, at) − inf

a∈A n

X

t=1

K(wt, a∗) #

Theorem. R(n) ≤ 2n + ⌘mn + 2✏n ⌘ | {z }

Bias variance

+2✏n + 1 ⌘ log(|A|).

Kernel Bandits Regret

26

Expected regret

game vs

Km K game

ξt bias

R(n) = E " n X

t=1

K(wt, at) − inf

a∈A n

X

t=1

K(wt, a∗) #

Theorem. R(n) ≤ 2n + ⌘mn + 2✏n ⌘ | {z }

Bias variance

+2✏n + 1 ⌘ log(|A|).

Kernel Bandits Regret

26

Expected regret

game vs

Km K game

ξt bias

R(n) = E " n X

t=1

K(wt, at) − inf

a∈A n

X

t=1

K(wt, a∗) #

Theorem. R(n) ≤ 2n + ⌘mn + 2✏n ⌘ | {z }

Bias variance

+2✏n + 1 ⌘ log(|A|).

Kernel Bandits Regret

26

Expected regret

game vs

Km K game

ξt bias

R(n) = E " n X

t=1

K(wt, at) − inf

a∈A n

X

t=1

K(wt, a∗) #

Theorem. R(n) ≤ 2n + ⌘mn + 2✏n ⌘ | {z }

Bias variance

+2✏n + 1 ⌘ log(|A|).

Kernel Bandits Regret

27

Corollary. 1 Polynomial Decay. µj ≤ Cj−β and β > 2: R(n) ≤ O ⇣ log(|A|)

β−2 2(β−1) n β 2(β−1)

⌘ 2 Exponential Decay. µj ≤ Ce−βj: R(n) ≤ O ⇣p log(|A|) log(n)n ⌘

Experiments

28

Lower Bound

29

Lower Bound

29

Polynomial decay µj ≤ Cj−β

:

Lower Bound

29

Polynomial decay µj ≤ Cj−β

Rn ≥ Ω ⇣ n

β+1 2β

⌘

:

Lower Bound

29

Polynomial decay µj ≤ Cj−β

Rn ≥ Ω ⇣ n

β+1 2β

⌘

Exponential decay µj ≤ C exp(−βj)

: :

Lower Bound

29

Polynomial decay µj ≤ Cj−β

Rn ≥ Ω ⇣ n

β+1 2β

⌘

Exponential decay µj ≤ C exp(−βj)

Rn ≥ Ω ⇣ n1/2⌘

: :

Lower Bound

29

Polynomial decay µj ≤ Cj−β

Rn ≥ Ω ⇣ n

β+1 2β

⌘

Exponential decay µj ≤ C exp(−βj)

Rn ≥ Ω ⇣ n1/2⌘

: :

Lower Bound

29

Polynomial decay µj ≤ Cj−β

Rn ≥ Ω ⇣ n

β+1 2β

⌘

Exponential decay µj ≤ C exp(−βj)

Rn ≥ Ω ⇣ n1/2⌘

: :

A = {(Aj)∞

j=1 s.t. |Aj| = 1 ∀j}

Lower Bound

29

Polynomial decay µj ≤ Cj−β

Rn ≥ Ω ⇣ n

β+1 2β

⌘

Exponential decay µj ≤ C exp(−βj)

Rn ≥ Ω ⇣ n1/2⌘

: :

A = {(Aj)∞

j=1 s.t. |Aj| = 1 ∀j}

W = {(wj)∞

j=1 s.t. |wj| = µj ∀j}

Final thoughts

30

Thanks for your attention