Advanced Machine Learning CS 7140 - Spring 2019 Lecture 24: - - PowerPoint PPT Presentation

Advanced Machine Learning

CS 7140 - Spring 2019

Lecture 24: Bayesian Optimization

Jan-Willem van de Meent Slide credits: Ryan Adams, Nando de Freitas

Background: Multi-Armed Bandits

• Problem: Which machine has highest rate of payout?
• Trade-off: Exploration (trying a new machine) vs


Exploitation (playing machine with best returns so far)

• Regret: Difference between reward of action, and reward 

• f optimal action (with benefit of hindsight)

Background: Multi-Armed Bandits

• Problem: Which machine has highest rate of payout?
• Trade-off: Exploration (trying a new machine) vs


Exploitation (playing machine with best returns so far)

• Regret: Difference between reward of action, and reward 

• f optimal action (with benefit of hindsight)

Example: Thompson Sampling

Goal: Use A/B testing to optimize button click rate

Bandit Require: ; : hyperparameters of the beta prior 1: Initialize na;0 ¼ na;1 ¼ i ¼ 0 for all a 2: repeat 3: for a ¼ 1; . . . ; K do 4: ~ wa betað þ na;1; þ na;0Þ 5: end for 6: ai ¼ arg maxa ~ wa 7: Observe yi by pulling arm ai 8: if yi ¼ 0 then 9: nai;0 ¼ nai;0 þ 1 10: else 11: nai;1 ¼ nai;1 þ 1 12: end if 13: i ¼ i þ 1 14: until stopping criterion reached

Thompson Sampling

Bayesian Optimization

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −3 −2 −1 1 2 3

current! best Goal: Optimize unknown cost function 
 (continuous version of bandit problem)

Bayesian Optimization

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −3 −2 −1 1 2 3

current! best Goal: Optimize unknown cost function 
 (continuous version of bandit problem)

Bayesian Optimization

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −3 −2 −1 1 2 3

current! best Goal: Optimize unknown cost function 
 (continuous version of bandit problem)

Bayesian Optimization

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −3 −2 −1 1 2 3

current! best Goal: Optimize unknown cost function 
 (continuous version of bandit problem)

Bayesian Optimization

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −3 −2 −1 1 2 3

current! best Goal: Optimize unknown cost function 
 (continuous version of bandit problem)

Bayesian Optimization

Problem: Which point should we evaluate next?

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −3 −2 −1 1 2 3

current! best

Bayesian Optimization

Idea 1: Model uncertainty about objective function

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −3 −2 −1 1 2 3

Bayesian Optimization

Idea 2: Define acquisition function 
 that balances exploration and exploitation

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −3 −2 −1 1 2 3

Bayesian Optimization

Bayesian Optimization

Bayesian Optimization

Bayesian Optimization

Bayesian Optimization

Bayesian Optimization

Bayesian Optimization

Intuition: Why does Bayes Opt work?

Idea: Use confidence bounds to adaptively eliminate regions in search space that are not likely to contain optimum

Modeling Uncertainty

Predictive Posterior over Functions

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −3 −2 −1 1 2 3

Recap: Gaussian Processes

Formal View: Prior over Functions

Practical View: Generalization of Multivariate Normal

Recap: Gaussian Processes

Predictive Posterior: Distribution on f* for a new point x*

Recap: Gaussian Processes

Predictive Posterior: Distribution on f* for a new point x*

Bayesian Optimization with GPs

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −3 −2 −1 1 2 3

Choices of Kernel Functions

C(x, x0) = exp ( −1 2

D

X

d=1

✓xd − x0

d

⇥d ◆2)

C(r) = 21−ν Γ() √ 2 r ⇥ !ν Kν √ 2 r ⇥ !

C(x, x0) = 2 π sin1 8 < : 2xTΣx0 q (1 + 2xTΣx)(1 + 2x0TΣx0) 9 = ;

C(x, x0) = exp ( −2 sin2 1

2(x − x0)

• 2

)

Squared-Exponential Matérn “Neural Network” Periodic

Acquisition Functions

Acquisition Functions

• 1. Exploration: Evaluate highest posterior uncertainty
• 2. Exploitation: Evaluate lowest posterior mean

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −3 −2 −1 1 2 3

Acquisition Functions

Exploration-exploitation trade-off:

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −3 −2 −1 1 2 3

e µ(x) − κe σ(x)

Upper/Lower Confidence Bounds

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −3 −2 −1 1 2 3

Can produce sequence xn with provably optimal 
 regret bounds, but tuning often needed in practice

Probability of Improvement

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −3 −2 −1 1 2 3

Not used often, but works well with a fixed target μ-

Expected Improvement

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −3 −2 −1 1 2 3

EI(x) = Ep(f |y) ⇥ max{0,µ− − f (x)} ⇤ µ− = argmin

n

e µ(xn)

• − − e x

⇥ { − } ⇤ e =

• µ− − e

µ(x)

• Φ(Z) + e

σ(x) N(Z;0,1) Z = µ− − e µ(x) e σ(x)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −3 −2 −1 1 2 3

Idea: model uncertainty about location of optimum

Entropy Search

Choosing Kernel Hyperparameters

Choices for GP Hyperparameters

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −3 −2 −1 1 2 3 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −3 −2 −1 1 2 3 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −3 −2 −1 1 2 3 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −3 −2 −1 1 2 3

C(r) = 21−ν Γ() √ 2 r ⇥ !ν Kν √ 2 r ⇥ !

ν = 1/2 ν = 3/2 ν = 5/2 ν = ∞

Matérn: ν determines how many times differentiable

MCMC for GP Hyperparameters

• Integrated Acquisition Function:!

! ! ! !

• For a theoretical discussion of the implications of inferring

ˆ a(x) = Z a(x ; θ) p(θ | {xn, yn}N

n=1) dθ

≈ 1 K

K

X

k=1

a(x ; θ(k)) θ(k) ∼ p(θ | {xn, yn}N

n=1)

Idea: define prior over kernel parameter, a GP likelihood and take expectation of acusition function over posterior.

MCMC for GP Hyperparameters

Posterior samples ent Length scale specific

• vement

Integrated expected

Posterior samples for 3 different length scales Expected improvement for each length scale Integrated expected improvement

MCMC for GP Hyperparameters

10 20 30 40 50 60 70 80 90 100 0.05 0.1 0.15 0.2 0.25 Classification Error Function Evaluations GP EI MCMC GP EI Conventional Tree Parzen Algorithm

Snoek, Larochelle & RPA, NIPS 2012

[Snoek, Larochelle & Adams, NIPS 2012] Optimizing SGD & Regularization Parameters for Logistic Regression

Use GP to model Computational Cost

[Snoek, Larochelle & Adams, NIPS 2012] Tuning Hyperparmeters for Deep Convolutional Neural Nets (CIFAR10)

5 10 15 20 25 30 0.18 0.2 0.22 0.24 0.26 0.28 0.3 Classification Error Time (Hours) GP EI MCMC GP EI per Second State−of−the−art

Snoek, Larochelle & RPA, NIPS 2012

Open Source Implementations

https://github.com/HIPS/Spearmint

Open Source Implementations

https://github.com/probprog/bopp

[Rainforth, Le, van de Meent, Osborne, Wood, NIPS 2016]