Linear models Oliver Stegle and Karsten Borgwardt Machine Learning - - PowerPoint PPT Presentation
Linear models Oliver Stegle and Karsten Borgwardt Machine Learning and Computational Biology Research Group, Max Planck Institute for Biological Cybernetics and Max Planck Institute for Developmental Biology, Tbingen Oliver Stegle and
Oliver Stegle and Karsten Borgwardt: Computational Approaches for Analysing Complex Biological Systems, Page 1
Motivation
Curve fitting
Tasks we are interested in:
◮ Making predictions ◮ Comparison of alternative
models
x*
Linear models T¨ ubingen 1
Motivation
Curve fitting
Tasks we are interested in:
◮ Making predictions ◮ Comparison of alternative
models
x*
Linear models T¨ ubingen 1
Motivation
Further reading, useful material
◮ Christopher M. Bishop: Pattern Recognition and Machine learning.
◮ Good background, covers most of the course material and much more! ◮ This lecture is largely inspired by chapter 3 of the book.
Linear models T¨ ubingen 2
Outline
Outline
Linear models T¨ ubingen 3
Linear Regression
Outline
Motivation Linear Regression Bayesian linear regression Model comparison and hypothesis testing Summary
Linear models T¨ ubingen 4
Linear Regression
Regression
Noise model and likelihood
◮ Given a dataset D = {xn, yn}N n=1, where xn = {xn,1, . . . , xn,D} is D
dimensional, fit parameters θ of a regressor f with added Gaussian noise: yn = f(xn; θ) + ǫn where p(ǫ | σ2) = N
.
◮ Equivalent likelihood formulation:
p(y | X) =
N
N
Linear models T¨ ubingen 5
Linear Regression
Regression
Choosing a regressor
◮ Choose f to be linear:
p(y | X) =
N
N
◮ Consider bias free case, c = 0,
column of ones in each xn.
Linear models T¨ ubingen 6
Linear Regression
Regression
Choosing a regressor
◮ Choose f to be linear:
p(y | X) =
N
N
◮ Consider bias free case, c = 0,
column of ones in each xn.
Equivalent graphical model
Linear models T¨ ubingen 6
Linear Regression
Linear Regression
Maximum likelihood
◮ Taking the logarithm, we obtain
ln p(y | w, X, σ2) =
N
ln N
= −N 2 ln 2πσ2 − 1 2σ2
N
(yn − wT · xn)2
◮ The likelihood is maximized when the squared error is minimized. ◮ Least squares and maximum likelihood are equivalent.
Linear models T¨ ubingen 7
Linear Regression
Linear Regression
Maximum likelihood
◮ Taking the logarithm, we obtain
ln p(y | w, X, σ2) =
N
ln N
= −N 2 ln 2πσ2 − 1 2σ2
N
(yn − wT · xn)2
◮ The likelihood is maximized when the squared error is minimized. ◮ Least squares and maximum likelihood are equivalent.
Linear models T¨ ubingen 7
Linear Regression
Linear Regression
Maximum likelihood
◮ Taking the logarithm, we obtain
ln p(y | w, X, σ2) =
N
ln N
= −N 2 ln 2πσ2 − 1 2σ2
N
(yn − wT · xn)2
◮ The likelihood is maximized when the squared error is minimized. ◮ Least squares and maximum likelihood are equivalent.
Linear models T¨ ubingen 7
Linear Regression
Linear Regression and Least Squares
y x f(xn, w) y
n
xn
(C.M. Bishop, Pattern Recognition and Machine Learning)
E(w) = 1 2
N
(yn − wTxn)2
Linear models T¨ ubingen 8
Linear Regression
Linear Regression and Least Squares
◮ Derivative w.r.t a single weight entry wi
d dwi ln p(y | w, σ2) = d dwi
2σ2
N
(yn − w · xn)2
σ2
N
(yn − w · xn)xi
◮ Set gradient w.r.t to w to zero
∇w ln p(y | w, σ2) = 1 σ2
N
(yn − w · xn)xT
n = 0
= ⇒ wML = (XTX)−1XT
y
◮ Here, the matrix X is defined as X =
x1,1 . . . x1, D . . . . . . . . . xN,1 . . . xN,D
Linear models T¨ ubingen 9
Linear Regression
Polynomial Curve Fitting
◮ Use the polynomials up to degree K to construct new features from x
f(x, w) = w0 + w1x + w2x2 + · · · + wKxK = wTφ(x), where we defined φ(x) = (1, x, x2, . . . , xK).
◮ Similarly, φ can be any feature mapping. ◮ Possible to show: the feature map φ can be expressed in terms of
kernels (kernel trick).
Linear models T¨ ubingen 10
Linear Regression
Polynomial Curve Fitting
◮ Use the polynomials up to degree K to construct new features from x
f(x, w) = w0 + w1x + w2x2 + · · · + wKxK = wTφ(x), where we defined φ(x) = (1, x, x2, . . . , xK).
◮ Similarly, φ can be any feature mapping. ◮ Possible to show: the feature map φ can be expressed in terms of
kernels (kernel trick).
Linear models T¨ ubingen 10
Linear Regression
Polynomial Curve Fitting
Overfitting
◮ The degree of the polynomial is crucial to avoid under- and
x t M = 0 1 −1 1
(C.M. Bishop, Pattern Recognition and Machine Learning)
Linear models T¨ ubingen 11
Linear Regression
Polynomial Curve Fitting
Overfitting
◮ The degree of the polynomial is crucial to avoid under- and
x t M = 1 1 −1 1
(C.M. Bishop, Pattern Recognition and Machine Learning)
Linear models T¨ ubingen 11
Linear Regression
Polynomial Curve Fitting
Overfitting
◮ The degree of the polynomial is crucial to avoid under- and
x t M = 3 1 −1 1
(C.M. Bishop, Pattern Recognition and Machine Learning)
Linear models T¨ ubingen 11
Linear Regression
Polynomial Curve Fitting
Overfitting
◮ The degree of the polynomial is crucial to avoid under- and
x t M = 9 1 −1 1
(C.M. Bishop, Pattern Recognition and Machine Learning)
Linear models T¨ ubingen 11
Linear Regression
Regularized Least Squares
◮ Solutions to avoid overfitting:
◮ Intelligently choose K ◮ Regularize the regression weights w
◮ Construct a smoothed error function
E(w) = 1 2
N
2
+ λ 2 wTw
Regularizer
Linear models T¨ ubingen 12
Linear Regression
Regularized Least Squares
◮ Solutions to avoid overfitting:
◮ Intelligently choose K ◮ Regularize the regression weights w
◮ Construct a smoothed error function
E(w) = 1 2
N
2
+ λ 2 wTw
Regularizer
Linear models T¨ ubingen 12
Linear Regression
Regularized Least Squares
More general regularizers
◮ A more general regularization approach:
E(w) = 1 2
N
2
+ λ 2
D
|wd|q
Linear models T¨ ubingen 13
Linear Regression
Regularized Least Squares
More general regularizers
◮ A more general regularization approach:
E(w) = 1 2
N
2
+ λ 2
D
|wd|q
q =0 .5 q =1 q =2 q =4
(C.M. Bishop, Pattern Recognition and Machine Learning)
Linear models T¨ ubingen 13
Linear Regression
Regularized Least Squares
More general regularizers
◮ A more general regularization approach:
E(w) = 1 2
N
2
+ λ 2
D
|wd|q
q =0 .5 q =1 q =2 q =4
Quadratic Lasso sparse
(C.M. Bishop, Pattern Recognition and Machine Learning)
Linear models T¨ ubingen 13
Linear Regression
Loss functions and other methods
◮ Even more general: vary the loss function
E(w) = 1 2
N
L(yn − wTφ(xn))
+ λ 2
D
|wd|q
◮ Many state-of-the-art machine learning methods can be expressed
within this framework.
◮ Linear Regression: squared loss, squared regularizer. ◮ Support Vector Machine: hinge loss, squared regularizer. ◮ Lasso: squared loss, L1 regularizer.
◮ Inference: minimize the cost function E(w), yielding a point estimate
for w.
Linear models T¨ ubingen 14
Linear Regression
Loss functions and other methods
◮ Even more general: vary the loss function
E(w) = 1 2
N
L(yn − wTφ(xn))
+ λ 2
D
|wd|q
◮ Many state-of-the-art machine learning methods can be expressed
within this framework.
◮ Linear Regression: squared loss, squared regularizer. ◮ Support Vector Machine: hinge loss, squared regularizer. ◮ Lasso: squared loss, L1 regularizer.
◮ Inference: minimize the cost function E(w), yielding a point estimate
for w.
Linear models T¨ ubingen 14
Linear Regression
Loss functions and other methods
◮ Even more general: vary the loss function
E(w) = 1 2
N
L(yn − wTφ(xn))
+ λ 2
D
|wd|q
◮ Many state-of-the-art machine learning methods can be expressed
within this framework.
◮ Linear Regression: squared loss, squared regularizer. ◮ Support Vector Machine: hinge loss, squared regularizer. ◮ Lasso: squared loss, L1 regularizer.
◮ Inference: minimize the cost function E(w), yielding a point estimate
for w.
Linear models T¨ ubingen 14
Linear Regression
Regularized Least Squares
Probabilistic equivalent
◮ So far: minimization of error functions. ◮ Back to probabilities?
E(w) = 1 2
N
2
+ λ 2 wTw
Regularizer
◮ Similarly: most other choices of regularizers and loss functions can be
mapped to an equivalent probabilistic representation.
Linear models T¨ ubingen 15
Linear Regression
Regularized Least Squares
Probabilistic equivalent
◮ So far: minimization of error functions. ◮ Back to probabilities?
E(w) = 1 2
N
2
+ λ 2 wTw
Regularizer
= − ln p(y | w, Φ(X), σ2) − ln p(w)
◮ Similarly: most other choices of regularizers and loss functions can be
mapped to an equivalent probabilistic representation.
Linear models T¨ ubingen 15
Linear Regression
Regularized Least Squares
Probabilistic equivalent
◮ So far: minimization of error functions. ◮ Back to probabilities?
E(w) = 1 2
N
2
+ λ 2 wTw
Regularizer
= − ln p(y | w, Φ(X), σ2) − ln p(w) = −
N
ln N
− ln N
λI
mapped to an equivalent probabilistic representation.
Linear models T¨ ubingen 15
Linear Regression
Regularized Least Squares
Probabilistic equivalent
◮ So far: minimization of error functions. ◮ Back to probabilities?
E(w) = 1 2
N
2
+ λ 2 wTw
Regularizer
= − ln p(y | w, Φ(X), σ2) − ln p(w) = −
N
ln N
− ln N
λI
mapped to an equivalent probabilistic representation.
Linear models T¨ ubingen 15
Bayesian linear regression
Outline
Motivation Linear Regression Bayesian linear regression Model comparison and hypothesis testing Summary
Linear models T¨ ubingen 16
Bayesian linear regression
Bayesian linear regression
◮ Likelihood as before
p(y | X, w, σ2) =
N
N
◮ Define a conjugate prior over w
p(w) = N (w | m0, S0)
Linear models T¨ ubingen 17
Bayesian linear regression
Bayesian linear regression
◮ Likelihood as before
p(y | X, w, σ2) =
N
N
◮ Define a conjugate prior over w
p(w) = N (w | m0, S0)
Linear models T¨ ubingen 17
Bayesian linear regression
Bayesian linear regression
◮ Posterior probability of w
p(w | y, X, σ2) ∝
N
N
· N (w | m0, S0) = N
= N (w | µw, Σw)
◮ where
µw = Σw
0 m0 + 1
σ2 Φ(X)Ty
+ 1 σ2 Φ(X)TΦ(X) −1
Linear models T¨ ubingen 18
Bayesian linear regression
Bayesian linear regression
Prior choice
◮ A common choice is a prior that corresponds to regularized regression
p(w) = N
λI
◮ In this case
µw = Σw
0 m0 + 1
σ2 Φ(X)Ty
+ 1 σ2 Φ(X)TΦ(X) −1
Linear models T¨ ubingen 19
Bayesian linear regression
Bayesian linear regression
Prior choice
◮ A common choice is a prior that corresponds to regularized regression
p(w) = N
λI
◮ In this case
µw = Σw
σ2 Φ(X)Ty
σ2 Φ(X)TΦ(X) −1
Linear models T¨ ubingen 19
Bayesian linear regression
Bayesian linear regression
Example
0 Data points
(C.M. Bishop, Pattern Recognition and Machine Learning)
Linear models T¨ ubingen 20
Bayesian linear regression
Bayesian linear regression
Example
1 Data point
(C.M. Bishop, Pattern Recognition and Machine Learning)
Linear models T¨ ubingen 20
Bayesian linear regression
Bayesian linear regression
Example
20 Data points
(C.M. Bishop, Pattern Recognition and Machine Learning)
Linear models T¨ ubingen 20
Bayesian linear regression
Making predictions
◮ Prediction for fixed weight ˆ
w at input x⋆ trivial: p(y⋆ | x⋆, ˆ w, σ2) = N
wTφ(x⋆), σ2
◮ Integrate over w to take the posterior uncertainty into account
p(y⋆ | x⋆, D) =
p(y⋆ | x⋆, w, σ2)p(w | X, y, σ2) =
N
wTφ(x⋆), σ2 N (w | µw, Σw) = N
µT
wφ(x⋆), σ2 + φ(x⋆)TΣwφ(x⋆)
◮ prediction is again Gaussian ◮ Predictive variance is increase due to the posterior uncertainty in w.
Linear models T¨ ubingen 21
Bayesian linear regression
Making predictions
◮ Prediction for fixed weight ˆ
w at input x⋆ trivial: p(y⋆ | x⋆, ˆ w, σ2) = N
wTφ(x⋆), σ2
◮ Integrate over w to take the posterior uncertainty into account
p(y⋆ | x⋆, D) =
p(y⋆ | x⋆, w, σ2)p(w | X, y, σ2) =
N
wTφ(x⋆), σ2 N (w | µw, Σw) = N
µT
wφ(x⋆), σ2 + φ(x⋆)TΣwφ(x⋆)
◮ prediction is again Gaussian ◮ Predictive variance is increase due to the posterior uncertainty in w.
Linear models T¨ ubingen 21
Bayesian linear regression
Making predictions
◮ Prediction for fixed weight ˆ
w at input x⋆ trivial: p(y⋆ | x⋆, ˆ w, σ2) = N
wTφ(x⋆), σ2
◮ Integrate over w to take the posterior uncertainty into account
p(y⋆ | x⋆, D) =
p(y⋆ | x⋆, w, σ2)p(w | X, y, σ2) =
N
wTφ(x⋆), σ2 N (w | µw, Σw) = N
µT
wφ(x⋆), σ2 + φ(x⋆)TΣwφ(x⋆)
◮ prediction is again Gaussian ◮ Predictive variance is increase due to the posterior uncertainty in w.
Linear models T¨ ubingen 21
Model comparison and hypothesis testing
Outline
Motivation Linear Regression Bayesian linear regression Model comparison and hypothesis testing Summary
Linear models T¨ ubingen 22
Model comparison and hypothesis testing
Model comparison
Motivation
◮ What degree of polynomials
describes the data best?
◮ Is the linear model at all
appropriate?
◮ Association testing.
Linear models T¨ ubingen 23
Model comparison and hypothesis testing
Model comparison
Motivation
◮ What degree of polynomials
describes the data best?
◮ Is the linear model at all
appropriate?
◮ Association testing.
?
Phenome Genome
ATGACCTGAAACTGGGGGACTGACGTGGAACGGT ATGACCTGCAACTGGGGGACTGACGTGCAACGGT ATGACCTGCAACTGGGGGACTGACGTGCAACGGT ATGACCTGAAACTGGGGGATTGACGTGGAACGGT ATGACCTGCAACTGGGGGATTGACGTGCAACGGT ATGACCTGCAACTGGGGGATTGACGTGCAACGGT individuals phenotypes SNPs
y y y y y
y1
Linear models T¨ ubingen 23
Model comparison and hypothesis testing
Bayesian model comparison
◮ How do we choose among alternative models? ◮ Assume we want to choose among models H0, . . . , HM for a
dataset D.
◮ Posterior probability for a particular model i
p(Hi | D) ∝ p(D | Hi)
p(Hi)
Prior
Linear models T¨ ubingen 24
Model comparison and hypothesis testing
Bayesian model comparison
◮ How do we choose among alternative models? ◮ Assume we want to choose among models H0, . . . , HM for a
dataset D.
◮ Posterior probability for a particular model i
p(Hi | D) ∝ p(D | Hi)
p(Hi)
Prior
Linear models T¨ ubingen 24
Model comparison and hypothesis testing
Bayesian model comparison
How to calculate the evidence
◮ The evidence is not the model likelihood!
p(D | Hi) =
p(D | θ)p(θ) for model parameters θ.
◮ Remember:
p(θ | Hi, D) = p(D | Hi, θ)p(θ) p(D | Hi)
Linear models T¨ ubingen 25
Model comparison and hypothesis testing
Bayesian model comparison
How to calculate the evidence
◮ The evidence is not the model likelihood!
p(D | Hi) =
p(D | θ)p(θ) for model parameters θ.
◮ Remember:
p(θ | Hi, D) = p(D | Hi, θ)p(θ) p(D | Hi) posterior = likelihood · prior Evidence
Linear models T¨ ubingen 25
Model comparison and hypothesis testing
Bayesian model comparison
Ocam’s razor
◮ The evidence integral penalizes
◮ A model with few parameters
and lower maximum likelihood (H1) may win over a model with a peaked likelihood that requires many more parameters (H2).
wM
A P
w Likelihood
H2 H1 (C.M. Bishop, Pattern Recognition and Machine Learning)
Linear models T¨ ubingen 26
Model comparison and hypothesis testing
Bayesian model comparison
Ocam’s razor
◮ The evidence integral penalizes
◮ A model with few parameters
and lower maximum likelihood (H1) may win over a model with a peaked likelihood that requires many more parameters (H2).
wM
A P
w Likelihood
H2 H1 (C.M. Bishop, Pattern Recognition and Machine Learning)
Linear models T¨ ubingen 26
Model comparison and hypothesis testing
Application to GWA
◮ Consider an association study.
◮ H0: p(y | H0, X, θ) = N
θ = {σ2}
◮ H1: p(y | H1, X, θ) = N
θ = {σ2, w}
◮ Choosing conjugate priors for σ2 and w, the required integrals are
tractable in closed form.
Linear models T¨ ubingen 27
Model comparison and hypothesis testing
Application to GWA
◮ Consider an association study.
◮ H0: p(y | H0, X, θ) = N
θ = {σ2}
◮ H1: p(y | H1, X, θ) = N
θ = {σ2, w}
◮ Choosing conjugate priors for σ2 and w, the required integrals are
tractable in closed form.
Linear models T¨ ubingen 27
Model comparison and hypothesis testing
Application to GWA
◮ Consider an association study.
◮ H0: p(y | H0, X, θ) = N
θ = {σ2}
◮ H1: p(y | H1, X, θ) = N
θ = {σ2, w}
◮ Choosing conjugate priors for σ2 and w, the required integrals are
tractable in closed form.
Linear models T¨ ubingen 27
Model comparison and hypothesis testing
Application to GWA
Scoring models
◮ The ratio of the evidences, the Bayes factor is a common scoring
metric to compare two models: BF = ln p(D | H1) p(D | H0).
Linear models T¨ ubingen 28
Model comparison and hypothesis testing
Application to GWA
Scoring models
◮ The ratio of the evidences, the Bayes factor is a common scoring
metric to compare two models: BF = ln p(D | H1) p(D | H0).
1.3354 1.3356 1.3358 1.336 1.3362 1.3364 1.3366 1.3368 1.337 1.3372 1.3374 x 10
8
5 10 15 LOD/BF Position in chr. 7 SLC35B4
0.01% FPR
0.01% FPR SLC35B4
Linear models T¨ ubingen 28
Model comparison and hypothesis testing
Application to GWA
Posterior probability of an association
◮ Bayes factors are useful, however we would like a probabilistic answer
how certain an association really is.
◮ Posterior probability of H1
p(H1 | D) = p(D | H1)p(H1) p(D) = p(D | H1)p(H1) p(D | H1)p(H1) + p(D | H0)p(H0)
◮ p(H1 | D) + p(H0 | D) = 1, prior probability of observing a real
association.
Linear models T¨ ubingen 29
Model comparison and hypothesis testing
Application to GWA
Posterior probability of an association
◮ Bayes factors are useful, however we would like a probabilistic answer
how certain an association really is.
◮ Posterior probability of H1
p(H1 | D) = p(D | H1)p(H1) p(D) = p(D | H1)p(H1) p(D | H1)p(H1) + p(D | H0)p(H0)
◮ p(H1 | D) + p(H0 | D) = 1, prior probability of observing a real
association.
Linear models T¨ ubingen 29
Model comparison and hypothesis testing
Application to GWA
Posterior probability of an association
◮ Bayes factors are useful, however we would like a probabilistic answer
how certain an association really is.
◮ Posterior probability of H1
p(H1 | D) = p(D | H1)p(H1) p(D) = p(D | H1)p(H1) p(D | H1)p(H1) + p(D | H0)p(H0)
◮ p(H1 | D) + p(H0 | D) = 1, prior probability of observing a real
association.
Linear models T¨ ubingen 29
Summary
Outline
Motivation Linear Regression Bayesian linear regression Model comparison and hypothesis testing Summary
Linear models T¨ ubingen 30
Summary
Summary
◮ Curve fitting and linear regression. ◮ Maximum likelihood and least squares regression are identical. ◮ Construction of features using a mapping φ. ◮ Regularized least squares. ◮ Bayesian linear regression. ◮ Model comparison and ocam’s razor.
Linear models T¨ ubingen 31