Mismatched Models & Can GP Regression Be Made Robust Against - - PowerPoint PPT Presentation

Gaussian Process Regression with Mismatched Models & Can GP Regression Be Made Robust Against Model Mismatch?

Peter Sollich NEURIPS 2002 & International Workshop on Deterministic and Statistical Methods in Machine Learning (2004)

Learning curve

Ideal learning curve:

• Performance on true

distribution

• Average over multiple

training datasets

What is GP regression?

𝑧 = 𝑔 𝑦 + 𝜗, 𝜗~ 𝑂 0,𝜏2 want to estimate𝑔 Put a GP-prior on 𝑔

• 𝐷𝑝𝑤 𝑔 𝑦𝑗 ,𝑔 𝑦𝑘

= 𝐿(𝑦𝑗,𝑦𝑘)

• 𝐹 𝑔 𝑦

= 0 Why GP-regression?

• Error bars
• Posterior analytically (requires 𝑃(𝑜3))

Mismatched model?

Input to GP: kernel 𝐿, noise level 𝜏

• What if we use the wrong one?

Setting:

• Assume p(x) known: uniform on line or

hypercube

• Theory exact if 𝑒 = ∞, otherwise all kinds
• f approximations
• Assume K x, x′ = g( 𝑦 − 𝑦′ )

Weird learning curves

• Plateaus or arbitrary # overfitting maxima

Hypercube, d=10, noise level too small: 1e-4, 1e-3, … true = 1 Line 1D, noise level too low results in plateau

Asymptotic problems

• If true kernel (OU, MB2) is less smooth than chosen kernel (RBF)
• No asymptotic decay 𝜗 = 𝑃(

1 𝑜) such as for parametric models, much

slower (log. slow)

• Prior cannot be overwhelmed by data (is too strong)

Fix?

• But maybe we just chose very bad hyperparameters?
• A true Bayesian is too expensive… What if evidence maximization?
• Maximize: 𝑄 𝐸 = ∫ 𝑄 𝐸 𝜄 𝑄 𝜄 𝑒𝜄 w.r.t. hyperpar.
• Setting: assume wrong kernel, but we tune 𝜏, 𝑏, 𝑚 using evidence
• All kinds of approximations to make the analysis tractable…

Hypercube analysis

• If we can tune par to get Bayes optimal performance, we will get it.
• No maxima’s
• If we cannot find those par (for example, 𝑚 → ∞), convergence still very slow
• No experiments???

1D case

• True kernel = MB2
• Used kernel = in plot
• No maxima, plateaus
• Optimal rate

achieved