Lecture 6 Jan-Willem van de Meent Regression Curve Fitting - - PowerPoint PPT Presentation

Unsupervised Machine Learning 
 and Data Mining

DS 5230 / DS 4420 - Fall 2018

Lecture 6

Jan-Willem van de Meent

Regression

Curve Fitting (according to XKCD)

https://xkcd.com/2048/

Linear Regression

Goal: Approximate points with a line or hyper-surface

ε ∼ Norm(0,σ2)

Assume f is a linear combination of D features

Linear Regression

Learning: Estimate w For N points we write Prediction: Estimate y’ given x’

Error Measure: Sum of Squares

Mean Squared Error (MSE): E(w) = 1 N

N

X

n=1

(wTxn yn)2 = 1 N k Xw y k2 where X = 2 6 6 4 — x1T — — x2T — . . . — xNT — 3 7 7 5 y = 2 6 6 4 y1T y2T . . . yNT 3 7 7 5

Minimizing the Error

E(w) = 1

N k Xw y k2

5E(w) = 2

NXT(Xw y) = 0

XTXw = XTy w = X†y where X† = (XTX)1XT is the ’pseudo-inverse’ of X

2

E(w) = 1

N k Xw y k2

5E(w) = 2

NXT(Xw y) = 0

XTXw = XTy w = X†y where X† = (XTX)1XT is the ’pseudo-inverse’ of X

2

Minimizing the Error

Matrix Cookbook (on course website)

Ordinary Least Squares

Construct matrix X and the vector y from the dataset {(x1, y1), x2, y2), . . . , (xN, yN)} (each x includes x0 = 1) as follows: X =     — xT

1 —

— xT

2 —

. . . — xT

N —

    y =     y T

1

y T

2

. . . y T

N

    Compute X† = (XTX)−1XT Return w = X†y

Basis function regression

Linear regression Basis function regression Polynomial regression For N samples

Polynomial Regression

x t M = 0 1 −1 1 x t M = 1 1 −1 1 x t M = 3 1 −1 1 x t M = 9 1 −1 1

Polynomial Regression

x t M = 0 1 −1 1 x t M = 1 1 −1 1 x t M = 3 1 −1 1 x t M = 9 1 −1 1

Underfit

Polynomial Regression

x t M = 0 1 −1 1 x t M = 1 1 −1 1 x t M = 3 1 −1 1 x t M = 9 1 −1 1

Overfit

L2 regularization (ridge regression) minimizes: E(w) = 1 N k Xw y k2 + λ k w k2 where λ 0 and k w k2 = wTw

• k

k L1 regularization (LASSO) minimizes: E(w) = 1 N k Xw y k2 + λ|w|1 where λ 0 and |w|1 =

D

P

i=1

|ωi|

Regularization

Regularization

L2: closed form solution w = (XTX + λI)1XTy L1: No closed form solution. Use quadratic programming: minimize k Xw y k2 s.t. k w k1 s

Regularization

Maximum Likelihood

Regression: Probabilistic Interpretation

What is the probability ?

Regression: Probabilistic Interpretation

Least Squares 
 Objective Likelihood

Maximum Likelihood

Least Squares 
 Objective Log-Likelihood Maximizing the likelihood minimizes the sum of squares

Maximum a Posteriori

Regression with Priors

Can we maximize ? (i.e. can we perform MAP estimation?)

Regression with Priors

From Bayes Rule

Maximum a Posteriori

Maximum a Posteriori is Equivalent to Ridge Regression

Closed-Form Solution for Ridge Regression

Objective

X>y =

• X>X + λ
• w

w ⇤ = argmax

w

log p(y, w) = argmax

w

(y X w)> (y X w) λw >w

r = 2

• X>(y X w) + λw
• <latexit sha1_base64="ljQxZ2REXfnHRDbh/EbBWqIKjUY=">AGT3icfZRLb9QwEICzpdtHeG3hyCViJdSKLUpK1faCVBVcCxVX9JmWTnJbGLVcYLj9GX5n/EnOHLkV3BD2NlIbOIR4omM987Jk4yAkuOv+7C09Wu6vrK6t24+fPH32fLDx4qLIShbCeZiRjF0FqACKZxzAlc5QxQGhC4DK4/avlDbACZ/SM3+cwSVFM8QyHiCvVdCBcx37zwfYpCgiaCv9G3Ep+ySLHTvfVJ/3cuRU2i3b98d7OZ9ofntHMTDjm7ayXcmvPs9yx57zrZTKWs3x36rWFVRhBy7Utk+w3HCt6aDofvOrZjCl4tDA+t+TqZbix/96MsLFOgPCSoKMaeqwoSiHEcElCBywJyF6jGMZKpCiFYiKqU5JOw3rmTcQsoxo2HATKC1SxBNDqeGiqQ0TlRhYM2tnAgdJYICx7TpFaTStv0IZqpjVWUiCkgJUpx+OpLCHe29H3k7+7KFMIhqwjtwR+pAzEDoDVysDvy9g5MJi9ZTuAf5GpMV8OAwm2YpSmikZoBCOVYnY8PtCgZ6I0IP0jF0JN6NlrwHFU+lV1N0oLxTgrhNwpUA3An29j9AqY3Wo1RG3roivVgYN+6MJ8nwFH9bybRmUHWxpsaULMgFi7QujMCXmBSUaN/cwW6GpOZmZSsDUPdYh5z+dETFPuvE8wQZ72urMqdTjskgFqdI9dnPcmCIZ0z/dLeYJwSnmBeitkvTC9P/eyl7O9mxbA6lfgeBOJYGQakGszm2ZkTGrKoyeldmAxa2LzxnWAeQusD1iT6r7z2rebKVzsvPOU/GV3eHhU3xr1ivrtbVpeda+dWh9tk6scyu0fvVWe4PeRv9H/3f/z0qNLvVq4aXVWCvrfwGQsklb</latexit><latexit sha1_base64="ljQxZ2REXfnHRDbh/EbBWqIKjUY=">AGT3icfZRLb9QwEICzpdtHeG3hyCViJdSKLUpK1faCVBVcCxVX9JmWTnJbGLVcYLj9GX5n/EnOHLkV3BD2NlIbOIR4omM987Jk4yAkuOv+7C09Wu6vrK6t24+fPH32fLDx4qLIShbCeZiRjF0FqACKZxzAlc5QxQGhC4DK4/avlDbACZ/SM3+cwSVFM8QyHiCvVdCBcx37zwfYpCgiaCv9G3Ep+ySLHTvfVJ/3cuRU2i3b98d7OZ9ofntHMTDjm7ayXcmvPs9yx57zrZTKWs3x36rWFVRhBy7Utk+w3HCt6aDofvOrZjCl4tDA+t+TqZbix/96MsLFOgPCSoKMaeqwoSiHEcElCBywJyF6jGMZKpCiFYiKqU5JOw3rmTcQsoxo2HATKC1SxBNDqeGiqQ0TlRhYM2tnAgdJYICx7TpFaTStv0IZqpjVWUiCkgJUpx+OpLCHe29H3k7+7KFMIhqwjtwR+pAzEDoDVysDvy9g5MJi9ZTuAf5GpMV8OAwm2YpSmikZoBCOVYnY8PtCgZ6I0IP0jF0JN6NlrwHFU+lV1N0oLxTgrhNwpUA3An29j9AqY3Wo1RG3roivVgYN+6MJ8nwFH9bybRmUHWxpsaULMgFi7QujMCXmBSUaN/cwW6GpOZmZSsDUPdYh5z+dETFPuvE8wQZ72urMqdTjskgFqdI9dnPcmCIZ0z/dLeYJwSnmBeitkvTC9P/eyl7O9mxbA6lfgeBOJYGQakGszm2ZkTGrKoyeldmAxa2LzxnWAeQusD1iT6r7z2rebKVzsvPOU/GV3eHhU3xr1ivrtbVpeda+dWh9tk6scyu0fvVWe4PeRv9H/3f/z0qNLvVq4aXVWCvrfwGQsklb</latexit><latexit sha1_base64="ljQxZ2REXfnHRDbh/EbBWqIKjUY=">AGT3icfZRLb9QwEICzpdtHeG3hyCViJdSKLUpK1faCVBVcCxVX9JmWTnJbGLVcYLj9GX5n/EnOHLkV3BD2NlIbOIR4omM987Jk4yAkuOv+7C09Wu6vrK6t24+fPH32fLDx4qLIShbCeZiRjF0FqACKZxzAlc5QxQGhC4DK4/avlDbACZ/SM3+cwSVFM8QyHiCvVdCBcx37zwfYpCgiaCv9G3Ep+ySLHTvfVJ/3cuRU2i3b98d7OZ9ofntHMTDjm7ayXcmvPs9yx57zrZTKWs3x36rWFVRhBy7Utk+w3HCt6aDofvOrZjCl4tDA+t+TqZbix/96MsLFOgPCSoKMaeqwoSiHEcElCBywJyF6jGMZKpCiFYiKqU5JOw3rmTcQsoxo2HATKC1SxBNDqeGiqQ0TlRhYM2tnAgdJYICx7TpFaTStv0IZqpjVWUiCkgJUpx+OpLCHe29H3k7+7KFMIhqwjtwR+pAzEDoDVysDvy9g5MJi9ZTuAf5GpMV8OAwm2YpSmikZoBCOVYnY8PtCgZ6I0IP0jF0JN6NlrwHFU+lV1N0oLxTgrhNwpUA3An29j9AqY3Wo1RG3roivVgYN+6MJ8nwFH9bybRmUHWxpsaULMgFi7QujMCXmBSUaN/cwW6GpOZmZSsDUPdYh5z+dETFPuvE8wQZ72urMqdTjskgFqdI9dnPcmCIZ0z/dLeYJwSnmBeitkvTC9P/eyl7O9mxbA6lfgeBOJYGQakGszm2ZkTGrKoyeldmAxa2LzxnWAeQusD1iT6r7z2rebKVzsvPOU/GV3eHhU3xr1ivrtbVpeda+dWh9tk6scyu0fvVWe4PeRv9H/3f/z0qNLvVq4aXVWCvrfwGQsklb</latexit><latexit sha1_base64="zV1JqbANtDGRl2b7+o3f9eMJxZg=">AGT3icfZRLb9QwEICzpdtHeG3hyCViJdSKLUpK1faCVBVcCxVX9JmWTnJbGLVcYLj9GX5n/EnOHLkV3BD2NlIbOIR4omM987Jk4yAkuOv+7C09Wu6vrK6t24+fPH32fLDx4qLIShbCeZiRjF0FqACKZxzAlc5QxQGhC4DK4/avlDbACZ/SM3+cwSVFM8QyHiCvVdCBcx37zwfYpCgiaCv9G3Ep+ySLHTvfVJ/3cuRU2i3b98d7OZ9ofntHMTDjm7ayXcmvPs9yx57zrZTKWs3x36rWFVRhBy7Utk+w3HCt6aDofvOrZjCl4tDK16nUw3lr/7URaWKVAeElQUY89VBQnEOA4JqMBlATkKr1EMYyVSlEIxEdUpSadhPfMmYpZRDjRsuAmUFiniaHUcNHUholKDKyZtlZOhI4SQYFj2vQKUmnbfgQz1bGqMhEFpAQpTj8dSeGO9t6PvJ192UIYRDXhHbgj9bSBmAHQGjnYHXl7ByaTlywn8A9yNarYUDhNszSFNFIzQCEcqzOxwdalAz0RoQfpGLoST0bLXiOKp/KriZpwXgnhfAbBaoBuJNt7H4B0xutxqgNPXTFejCwb12YzxPgqKN63k2jsoMtDbY0IWZArF0hdOaEvMAko8Z+Zgt0NSczMylZYOoe65Dzn86ImCfdeJ5gz1tdeZU6nFZJBCLU6T67Gc5MQzpn+6W8wTglPMC1HbpemF6f+9lL2d7Fg2h1K/g0AcS4MA1INZvPszAkNWdTk9C47sJg1sXnjOsC8BdYHrEl13nt280ULnbeUr+sjs8PKpvjXrlfXa2rQ8a986tD5bJ9a5FVq/equ9QW+j/6P/u/9npUaXerXw0mqslfW/PuNJGw=</latexit>

w ⇤ =

• X>X + λ

1 X y

Solution Optimum (solve for 0 gradient)

0 = rw log p(y, w)

• <latexit sha1_base64="ljQxZ2REXfnHRDbh/EbBWqIKjUY=">AGT3icfZRLb9QwEICzpdtHeG3hyCViJdSKLUpK1faCVBVcCxVX9JmWTnJbGLVcYLj9GX5n/EnOHLkV3BD2NlIbOIR4omM987Jk4yAkuOv+7C09Wu6vrK6t24+fPH32fLDx4qLIShbCeZiRjF0FqACKZxzAlc5QxQGhC4DK4/avlDbACZ/SM3+cwSVFM8QyHiCvVdCBcx37zwfYpCgiaCv9G3Ep+ySLHTvfVJ/3cuRU2i3b98d7OZ9ofntHMTDjm7ayXcmvPs9yx57zrZTKWs3x36rWFVRhBy7Utk+w3HCt6aDofvOrZjCl4tDA+t+TqZbix/96MsLFOgPCSoKMaeqwoSiHEcElCBywJyF6jGMZKpCiFYiKqU5JOw3rmTcQsoxo2HATKC1SxBNDqeGiqQ0TlRhYM2tnAgdJYICx7TpFaTStv0IZqpjVWUiCkgJUpx+OpLCHe29H3k7+7KFMIhqwjtwR+pAzEDoDVysDvy9g5MJi9ZTuAf5GpMV8OAwm2YpSmikZoBCOVYnY8PtCgZ6I0IP0jF0JN6NlrwHFU+lV1N0oLxTgrhNwpUA3An29j9AqY3Wo1RG3roivVgYN+6MJ8nwFH9bybRmUHWxpsaULMgFi7QujMCXmBSUaN/cwW6GpOZmZSsDUPdYh5z+dETFPuvE8wQZ72urMqdTjskgFqdI9dnPcmCIZ0z/dLeYJwSnmBeitkvTC9P/eyl7O9mxbA6lfgeBOJYGQakGszm2ZkTGrKoyeldmAxa2LzxnWAeQusD1iT6r7z2rebKVzsvPOU/GV3eHhU3xr1ivrtbVpeda+dWh9tk6scyu0fvVWe4PeRv9H/3f/z0qNLvVq4aXVWCvrfwGQsklb</latexit><latexit sha1_base64="ljQxZ2REXfnHRDbh/EbBWqIKjUY=">AGT3icfZRLb9QwEICzpdtHeG3hyCViJdSKLUpK1faCVBVcCxVX9JmWTnJbGLVcYLj9GX5n/EnOHLkV3BD2NlIbOIR4omM987Jk4yAkuOv+7C09Wu6vrK6t24+fPH32fLDx4qLIShbCeZiRjF0FqACKZxzAlc5QxQGhC4DK4/avlDbACZ/SM3+cwSVFM8QyHiCvVdCBcx37zwfYpCgiaCv9G3Ep+ySLHTvfVJ/3cuRU2i3b98d7OZ9ofntHMTDjm7ayXcmvPs9yx57zrZTKWs3x36rWFVRhBy7Utk+w3HCt6aDofvOrZjCl4tDA+t+TqZbix/96MsLFOgPCSoKMaeqwoSiHEcElCBywJyF6jGMZKpCiFYiKqU5JOw3rmTcQsoxo2HATKC1SxBNDqeGiqQ0TlRhYM2tnAgdJYICx7TpFaTStv0IZqpjVWUiCkgJUpx+OpLCHe29H3k7+7KFMIhqwjtwR+pAzEDoDVysDvy9g5MJi9ZTuAf5GpMV8OAwm2YpSmikZoBCOVYnY8PtCgZ6I0IP0jF0JN6NlrwHFU+lV1N0oLxTgrhNwpUA3An29j9AqY3Wo1RG3roivVgYN+6MJ8nwFH9bybRmUHWxpsaULMgFi7QujMCXmBSUaN/cwW6GpOZmZSsDUPdYh5z+dETFPuvE8wQZ72urMqdTjskgFqdI9dnPcmCIZ0z/dLeYJwSnmBeitkvTC9P/eyl7O9mxbA6lfgeBOJYGQakGszm2ZkTGrKoyeldmAxa2LzxnWAeQusD1iT6r7z2rebKVzsvPOU/GV3eHhU3xr1ivrtbVpeda+dWh9tk6scyu0fvVWe4PeRv9H/3f/z0qNLvVq4aXVWCvrfwGQsklb</latexit><latexit sha1_base64="ljQxZ2REXfnHRDbh/EbBWqIKjUY=">AGT3icfZRLb9QwEICzpdtHeG3hyCViJdSKLUpK1faCVBVcCxVX9JmWTnJbGLVcYLj9GX5n/EnOHLkV3BD2NlIbOIR4omM987Jk4yAkuOv+7C09Wu6vrK6t24+fPH32fLDx4qLIShbCeZiRjF0FqACKZxzAlc5QxQGhC4DK4/avlDbACZ/SM3+cwSVFM8QyHiCvVdCBcx37zwfYpCgiaCv9G3Ep+ySLHTvfVJ/3cuRU2i3b98d7OZ9ofntHMTDjm7ayXcmvPs9yx57zrZTKWs3x36rWFVRhBy7Utk+w3HCt6aDofvOrZjCl4tDA+t+TqZbix/96MsLFOgPCSoKMaeqwoSiHEcElCBywJyF6jGMZKpCiFYiKqU5JOw3rmTcQsoxo2HATKC1SxBNDqeGiqQ0TlRhYM2tnAgdJYICx7TpFaTStv0IZqpjVWUiCkgJUpx+OpLCHe29H3k7+7KFMIhqwjtwR+pAzEDoDVysDvy9g5MJi9ZTuAf5GpMV8OAwm2YpSmikZoBCOVYnY8PtCgZ6I0IP0jF0JN6NlrwHFU+lV1N0oLxTgrhNwpUA3An29j9AqY3Wo1RG3roivVgYN+6MJ8nwFH9bybRmUHWxpsaULMgFi7QujMCXmBSUaN/cwW6GpOZmZSsDUPdYh5z+dETFPuvE8wQZ72urMqdTjskgFqdI9dnPcmCIZ0z/dLeYJwSnmBeitkvTC9P/eyl7O9mxbA6lfgeBOJYGQakGszm2ZkTGrKoyeldmAxa2LzxnWAeQusD1iT6r7z2rebKVzsvPOU/GV3eHhU3xr1ivrtbVpeda+dWh9tk6scyu0fvVWe4PeRv9H/3f/z0qNLvVq4aXVWCvrfwGQsklb</latexit><latexit sha1_base64="zV1JqbANtDGRl2b7+o3f9eMJxZg=">AGT3icfZRLb9QwEICzpdtHeG3hyCViJdSKLUpK1faCVBVcCxVX9JmWTnJbGLVcYLj9GX5n/EnOHLkV3BD2NlIbOIR4omM987Jk4yAkuOv+7C09Wu6vrK6t24+fPH32fLDx4qLIShbCeZiRjF0FqACKZxzAlc5QxQGhC4DK4/avlDbACZ/SM3+cwSVFM8QyHiCvVdCBcx37zwfYpCgiaCv9G3Ep+ySLHTvfVJ/3cuRU2i3b98d7OZ9ofntHMTDjm7ayXcmvPs9yx57zrZTKWs3x36rWFVRhBy7Utk+w3HCt6aDofvOrZjCl4tDK16nUw3lr/7URaWKVAeElQUY89VBQnEOA4JqMBlATkKr1EMYyVSlEIxEdUpSadhPfMmYpZRDjRsuAmUFiniaHUcNHUholKDKyZtlZOhI4SQYFj2vQKUmnbfgQz1bGqMhEFpAQpTj8dSeGO9t6PvJ192UIYRDXhHbgj9bSBmAHQGjnYHXl7ByaTlywn8A9yNarYUDhNszSFNFIzQCEcqzOxwdalAz0RoQfpGLoST0bLXiOKp/KriZpwXgnhfAbBaoBuJNt7H4B0xutxqgNPXTFejCwb12YzxPgqKN63k2jsoMtDbY0IWZArF0hdOaEvMAko8Z+Zgt0NSczMylZYOoe65Dzn86ImCfdeJ5gz1tdeZU6nFZJBCLU6T67Gc5MQzpn+6W8wTglPMC1HbpemF6f+9lL2d7Fg2h1K/g0AcS4MA1INZvPszAkNWdTk9C47sJg1sXnjOsC8BdYHrEl13nt280ULnbeUr+sjs8PKpvjXrlfXa2rQ8a986tD5bJ9a5FVq/equ9QW+j/6P/u/9npUaXerXw0mqslfW/PuNJGw=</latexit>

Basis Function Regression

x t M = 3 1 −1 1

Ridge regression, but replace xn with φ(xn)

Predictive Posterior

adapted from: Carl Rasmussen, Probabilistic Machine Learning 4f13, http://mlg.eng.cam.ac.uk/teaching/4f13/1617/

Priors on Functions

−1 1 2 −2 −1 1 2

M=0

−1 1 2 −4 −2 2 4

M=1

−1 1 2 −5 5

M=2

−1 1 2 −10 10

M=3

−1 1 2 −50 50

M=5

−1 1 2 −2 2 x 10

5 M=17

Idea: sampling w ~ p(w) defines a function wTφ(x), so p(w) is equivalent to a prior on functions.

adapted from: Carl Rasmussen, Probabilistic Machine Learning 4f13, http://mlg.eng.cam.ac.uk/teaching/4f13/1617/

Posterior Uncertainty

−6 −4 −2 2 4 6 −2 2 −6 −4 −2 2 4 6 −2 2 −6 −4 −2 2 4 6 −2 2

Can we reason about the posterior on functions? Idea: sample w ~ p(w | X, y) and plot functions Increasing λ

−6 −4 −2 2 4 6 −2 2 −6 −4 −2 2 4 6 −2 2 −6 −4 −2 2 4 6 −2 2

adapted from: Carl Rasmussen, Probabilistic Machine Learning 4f13, http://mlg.eng.cam.ac.uk/teaching/4f13/1617/

The Predictive Distribution

Idea: Average over all possible values of w Increasing λ

Kernel Ridge Regression

Goal: Predict value of function at new point x* One Solution: Normal Ridge Regression

Φ := Φ(X) A := (Φ>Φ + λI) E[w|y] = A1Φ>y

(requires inversion of DxD matrix) Second Solution: Kernel Ridge Regression (replace DxD inversion with NxN inversion)

A1Φ = (Φ>Φ + λI)1Φ = Φ>(ΦΦ> + λI)1

DxN NxD NxD DxN

Ep(y⇤|x ⇤,y,X)[y⇤] = Ep(y⇤|x ⇤,y,X)[f (x⇤)] = Ep(w|y,X)[w]>φ(x ⇤)

Kernel Ridge Regression

−0.5 0.5 1 1.5 −1 −0.5 0.5 1

λ=0.1, σ=0.6

−0.5 0.5 1 1.5 −1 −0.5 0.5 1

λ=10, σ=0.6

−0.5 0.5 1 1.5 −1 −0.5 0.5 1 1.5

λ=1e−07, σ=0.6

Define kernel function

k(x, x 0) := φ(x)>φ(x 0)

Closed form Solution

f (x ⇤) = y>(K + λI)1k

ki = k(xi, x ∗)

adapted from: Carl Rasmussen, Probabilistic Machine Learning 4f13, http://mlg.eng.cam.ac.uk/teaching/4f13/1617/

Intuition: Kernel controls smoothness

• −10

−5 5 10 −0.5 0.0 0.5 1.0 input, x function value, y too long about right too short

The mean posterior predictive function is plotted for 3 different length scales (the

function: k(x, x0) = v2 exp

• − (x − x0)2

2`2

• + 2

noisexx0.

Characteristic Lengthscales

Kernel Functions

source: David Duvenaud (PhD Thesis)

Kernel name: Squared-exp (SE) Periodic (Per) Linear (Lin) k(x, xÕ) = σ2

f exp

1

−(x≠xÕ)2

2¸2

2

σ2

f exp

1

− 2

¸2 sin2 1

π x≠xÕ

p

22

σ2

f(x − c)(xÕ − c)

Plot of k(x, xÕ): x − xÕ x − xÕ x (with xÕ = 1)

↓ ↓ ↓

Functions f(x) sampled from

GP prior:

x x x Type of structure: local variation repeating structure linear functions

Combinations of Kernels

source: David Duvenaud (PhD Thesis)

Lin × Lin SE × Per Lin × SE Lin × Per

x (with xÕ = 1) x − xÕ x (with xÕ = 1) x (with xÕ = 1)

↓ ↓ ↓ ↓

quadratic functions locally periodic increasing variation growing amplitude

Gaussian Processes

adapted from: Carl Rasmussen, Probabilistic Machine Learning 4f13, http://mlg.eng.cam.ac.uk/teaching/4f13/1617/

Gaussian Processes

−5 5 −2 −1 1 2 input, x

• utput, f(x)

−5 5 −2 −1 1 2 input, x

• utput, f(x)

(a.k.a. Kernel Ridge Regression with Variance Estimates) Samples from GP prior Samples from GP posterior

adapted from: Carl Rasmussen, Probabilistic Machine Learning 4f13, http://mlg.eng.cam.ac.uk/teaching/4f13/1617/

Gaussian Process Sample (2D)

−6 −4 −2 2 4 6 −6 −4 −2 2 4 6 −2 −1 1 2 3 4 5 6 7 8

Function drawn at random from a Gaussian Process with Gaussian covariance

adapted from: Carl Rasmussen, Probabilistic Machine Learning 4f13, http://mlg.eng.cam.ac.uk/teaching/4f13/1617/

The Predictive Distribution

Predictive distribution on observations Prior

p(w | y, X) = p(y | w, X)p(w) p(y | X)

Posterior (Gaussian) Likelihood (Gaussian) Prior (Gaussian) Marginal Likelihood (Gaussian)

High-Level Idea: Use Conjugacy (Gaussian is conjugate to itself)

Marginals and Conditionals

Suppose that x and y are jointly Gaussian:

x x y

z = x y

• ∼ N

✓a b

• ,

 A C CT B ◆

x|y ∼ N

• a + CB−1(y − b), A − CB−1CT

y|x ∼ N

• b + CT A−1(x − a), B − CT A−1C
• (from Lecture 2)

x ∼ N (a, A) y ∼ N (b, B)

Gaussian Processes

m(x) = E[f(x)], k(x, x0) = E[(f(x) m(x))(f(x0) m(x0))],

Mean Function Kernel Function

f(x) ⇠ GP

• m(x), k(x, x0)
• Formal Definition: A prior on functions

Gaussian Processes

Practical Definition: Generalization of Multivariate Normal

Gaussian Process Prediction

Joint Distribution on past values f and future values f*

Joint Distribution on past data y and future data y*

ïy∗ y ò ∼ Norm Åïm(X∗) m(X) ò , ïk(X∗, X∗) + σ2I k(X∗, X) k(X∗, X) k(X, X) + σ2I òã

Gaussian Process Prediction

Joint Distribution on past data y and future data y*

ïy∗ y ò ∼ Norm Åïm(X∗) m(X) ò , ïk(X∗, X∗) + σ2I k(X∗, X) k(X∗, X) k(X, X) + σ2I òã

Predictive Distribution (Gaussian due to Conjugacy)

y⇤ | y ⇠ Norm Ä m⇤, K ⇤)

Ä m⇤ = k(X⇤, X)>(k(X, X) + σ2I)1y

K ⇤ = k(X⇤, X⇤) + σ2 k(X⇤, X)> k(X, X) + σ2I 1 k(X⇤, X)

(same as in kernel ridge regression)

adapted from: Carl Rasmussen, Probabilistic Machine Learning 4f13, http://mlg.eng.cam.ac.uk/teaching/4f13/1617/

GP Prior and Posterior

−5 5 −2 −1 1 2 input, x

• utput, f(x)

−5 5 −2 −1 1 2 input, x

• utput, f(x)

p(y⇤ | x ⇤, y, X) ⇠ Norm Ä k(x ⇤, X)>(k(X, X) + σ2I)1y, k(x ⇤, x ⇤) + σ2 k(x ⇤, X)> k(X, X) + σ2I 1 k(x ⇤, X) ä

The Kernel Trick

Cost of Feature Computation

Example: Mapping with linear and quadratic terms

Example: Mapping with linear and quadratic terms 1+d+d2/2 terms

Cost of Feature Computation

Example: Mapping with linear and quadratic terms

• Polynomial

ϕ (x)

Cost 100 features Quadratic > d2/2 terms up to degree 2 d2 N2 /4 2,500 N2 Cubic > d3/6 terms up to degree 3 d3 N2 /12 83,000 N2 Quartic > d4/24 terms up to degree 4 d4 N2 /48 1,960,000 N2

Cost of Feature Computation

The Kernel Trick

Define a kernel function such that k can be cheaper to evaluate than φ!

Define a kernel function such that k can be cheaper to evaluate than φ!

The Kernel Trick

Kernel for polynomials up to degree q

The Kernel Trick

Computational Cost

Kernel for polynomials up to degree q

• Polynomial

ϕ (x)

Cost 100 features Quadratic > d2/2 terms up to degree 2 d2 N2 /4 2,500 N2 Cubic > d3/6 terms up to degree 3 d3 N2 /12 83,000 N2 Quartic > d4/24 terms up to degree 4 d4 N2 /48 1,960,000 N2

Computational Cost

Kernel for polynomials up to degree q

• Polynomial

ϕ (x)

Cost 100 features Quadratic > d2/2 terms up to degree 2 d2 N2 /4 2,500 N2 Cubic > d3/6 terms up to degree 3 d3 N2 /12 83,000 N2 Quartic > d4/24 terms up to degree 4 d4 N2 /48 1,960,000 N2

100 100 100

Computational Cost

Kernel for polynomials up to degree q Implication: Kernel regression is basis function regression 
 with with an unbounded number of basis functions