Lecture 9: Regularized/penalized regression Felix Held, Mathematical - - PowerPoint PPT Presentation

Lecture 9: Regularized/penalized regression

Felix Held, Mathematical Sciences

MSA220/MVE440 Statistical Learning for Big Data 15th April 2019

Revisited: Expectation-Maximization (I)

New target function: Maximize log(𝑞(𝐘|𝜾)) = 𝔽𝑟(𝐚) [log 𝑞(𝐘, 𝐚|𝜾) 𝑟(𝐚) ] − 𝔽𝑟(𝐚) [log 𝑞(𝐚|𝐘, 𝜾) 𝑟(𝐚) ] with respect to 𝑟(𝐚) and 𝜾 Note:

▶ The left hand side is independent of 𝑟(𝐚) ▶ The difference on the right hand side has always the

same value, irrespective of the chosen 𝑟(𝐚). Choosing 𝑟(𝐚) is therefore a trade-off between 𝔽𝑟(𝐚) [log 𝑞(𝐘, 𝐚|𝜾) 𝑟(𝐚) ] and 𝔽𝑟(𝐚) [log 𝑞(𝐚|𝐘, 𝜾) 𝑟(𝐚) ]

1/24

Revisited: Expectation-Maximization (II)

• 1. Expectation step: For given parameters 𝜾(𝑛) the density

𝑟(𝐚) = 𝑞(𝐚|𝐘, 𝜾(𝑛)) minimizes the second term and thereby maximizes the first one. Set 𝑅(𝜾, 𝜾(𝑛)) = 𝔽𝑞(𝐚|𝐘,𝜾(𝑛)) [log 𝑞(𝐘, 𝐚|𝜾) 𝑞(𝐚|𝐘, 𝜾(𝑛))]

• 2. Maximization step: Maximize the first term with

𝜾(𝑛+1) = arg max

𝜾

𝑅(𝜾, 𝜾(𝑛)) Note: Since 𝔽𝑞(𝐚|𝐘,𝜾(𝑛)) [log 𝑞(𝐚|𝐘, 𝜾(𝑛)) 𝑞(𝐚|𝐘, 𝜾(𝑛))] = 0 it follows that log(𝑞(𝐘|𝜾(𝑛))) = 𝔽𝑞(𝐚|𝐘,𝜾(𝑛)) [log 𝑞(𝐘, 𝐚|𝜾(𝑛)) 𝑞(𝐚|𝐘, 𝜾(𝑛))]

2/24

Regularized/penalized regression

Remember ordinary least-squares (OLS)

Consider the model 𝐳 = 𝐘𝜸 + 𝜻 where

▶ 𝐳 ∈ ℝ𝑜 is the outcome, 𝐘 ∈ ℝ𝑜×(𝑞+1) is the design matrix,

𝜸 ∈ ℝ𝑞+1 are the regression coefficients, and 𝜻 ∈ ℝ𝑜 is the additive error

▶ Five basic assumptions have to be checked

Underlying relationship is linear (1) Zero mean (2), uncorrelated (3) errors with constant variance (4) which are (roughly) normally distributed (5)

▶ Centring (

1 𝑜 ∑ 𝑜 𝑚=1 𝑦𝑚𝑘 = 0) and standardisation

(

1 𝑜 ∑ 𝑜 𝑚=1 𝑦2 𝑚𝑘 = 1) of predictors simplifies interpretation

▶ Centring the outcome (

1 𝑜 ∑ 𝑜 𝑚=1 𝑧𝑚 = 0) and features

removes the need to estimate the intercept

3/24

Feature selection as motivation

Analytical solution exists when 𝐘𝑈𝐘 is invertible ̂ 𝜸OLS = (𝐘𝑈𝐘)−1𝐘𝑈𝐳 This can be unstable or fail in case of

▶ high correlation between predictors, or ▶ if 𝑞 > 𝑜.

Solutions: Regularisation or feature selection

4/24

Filtering for feature selection

▶ Choose features through pre-processing

▶ Features with maximum variance ▶ Use only the first 𝑙 PCA components

▶ Examples of other useful measures

▶ Use a univariate criterion, e.g. F-score: Features that

correlate most with the response

▶ Mutual Information: Reduction in uncertainty about 𝐲

after observing 𝑧

▶ Variable importance: Determine variable importance with

random forests

▶ Summary

▶ Pro: Fast and easy ▶ Con: Filtering mostly operates on single features and is

not geared towards a certain method

▶ Care with cross-validation and multiple testing necessary

▶ Filtering is often more of a pre-processing step and less

• f a proper feature selection step

5/24

Wrapping for feature selection

▶ Idea: Determine the best set of features by fitting models

• f different complexity and comparing their performance

▶ Best subset selection: Try all possible (exponentially

many) subsets of features and compare model performance with e.g. cross-validation

▶ Forward selection: Start with just an intercept and add in

each step the variable that improves fit the most (greedy algorithm)

▶ Backward selection: Start with all variables included and

then remove sequentially the one with the least impact (greedy algorithm)

▶ As discreet procedures, all of these methods exhibit high

variance (small changes could lead to different predictors being selected, resulting in a potentially very different model)

6/24

Embedding for feature selection

▶ Embed/include the feature selection into the model

estimation procedure

▶ Ideally, penalization on the number of included features

̂ 𝜸 = arg min

𝜸

‖𝐳 − 𝐘𝜸‖2

2 + 𝜇 𝑞

∑

𝑘=1

1(𝛾

𝑘 ≠ 0)

However, discrete optimization problems are hard to solve

▶ Softer regularisation methods can help

̂ 𝜸 = arg min

𝜸

‖𝐳 − 𝐘𝜸‖2

2 + 𝜇‖𝜸‖𝑟 𝑟

where 𝜇 is a tuning parameter and 𝑟 ≥ 1 or 𝑟 = ∞.

7/24

Constrained regression

The optimization problem arg min

𝜸

‖𝐳 − 𝐘𝜸‖2

2

subject to ‖𝜸‖𝑟

𝑟 ≤ 𝑢

for 𝑟 > 0 is equivalent to ̂ 𝜸 = arg min

𝜸

‖𝐳 − 𝐘𝜸‖2

2 + 𝜇‖𝜸‖𝑟 𝑟

when 𝑟 ≥ 1. This is the Lagrangian of the constrained problem.

▶ Clear when 𝑟 > 1: Convex constraint + target function and

both are differentiable

▶ Harder to prove for 𝑟 = 1, but possible (e.g. with

subgradients)

8/24

Ridge regression

For 𝑟 = 2 the constrained problem is ridge regression ̂ 𝜸ridge(𝜇) = arg min

𝜸

‖𝐳 − 𝐘𝜸‖2

2 + 𝜇‖𝜸‖2 2

where ‖𝜸‖2

2 = ∑ 𝑞 𝑘=1 𝛾2 𝑘 .

An analytical solution exists if 𝐘𝑈𝐘 + 𝜇𝐉𝑞 is invertible ̂ 𝜸ridge(𝜇) = (𝐘𝑈𝐘 + 𝜇𝐉𝑞)−1𝐘𝑈𝐳 If 𝐘𝑈𝐘 = 𝐉𝑞, then ̂ 𝜸ridge(𝜇) = ̂ 𝜸OLS 1 + 𝜇, i.e. ̂ 𝜸ridge(𝜇) is biased but has lower variance.

9/24

SVD and ridge regression

Recall: The SVD of a matrix 𝐘 ∈ ℝ𝑜×𝑞 was 𝐘 = 𝐕𝐄𝐖𝑈 The analytical solution for ridge regression becomes (𝑜 ≥ 𝑞) ̂ 𝜸ridge(𝜇) = (𝐘𝑈𝐘 + 𝜇𝐉𝑞)−1𝐘𝑈𝐳 = (𝐖𝐄2𝐖𝑈 + 𝜇𝐉𝑞)−1𝐖𝐄𝐕𝑈𝐳 = 𝐖(𝐄2 + 𝜇𝐉𝑞)−1𝐄𝐕𝑈𝐳 =

𝑞

∑

𝑘=1

𝑒

𝑘

𝑒2

𝑘 + 𝜇𝐰 𝑘𝐯𝑈 𝑘 𝐳

Ridge regression acts most on principal components with lower eigenvalues, e.g. in presence of correlation between features.

10/24

Effective degrees of freedom

Recall the hat matrix 𝐈 = 𝐘(𝐘𝑈𝐘)−1𝐘𝑈 in OLS. The trace of 𝐈 tr(𝐼) = tr(𝐘(𝐘𝑈𝐘)−1𝐘𝑈) = tr(𝐘𝑈𝐘(𝐘𝑈𝐘)−1) = tr(𝐉𝑞) = 𝑞 is equal to the trace of ˆ 𝚻 and the degrees of freedom for the regression coefficients. In analogy define for ridge regression 𝐈(𝜇) ∶= 𝐘(𝐘𝑈𝐘 + 𝜇𝐉𝑞)−1𝐘𝑈 and df(𝜇) ∶= tr(𝐈(𝜇)) =

𝑞

∑

𝑘=1

𝑒2

𝑘

𝑒2

𝑘 + 𝜇,

the effective degrees of freedom.

11/24

Lasso regression

For 𝑟 = 1 the constrained problem is known as the lasso ̂ 𝜸ridge(𝜇) = arg min

𝜸

‖𝐳 − 𝐘𝜸‖2

2 + 𝜇‖𝜸‖1

▶ Smallest 𝑟 in penalty such that constraint is still convex ▶ Performs feature selection

12/24

Intuition for the penalties (I)

Assume the OLS solution 𝜸OLS exists and set 𝐬 = 𝐳 − 𝐘𝜸OLS it follows for the residual sum of squares (RSS) that ‖𝐳 − 𝐘𝜸‖2

2 = ‖(𝐘𝜸OLS + 𝐬) − 𝐘𝜸‖2 2

= ‖(𝐘(𝜸 − 𝜸OLS) − 𝐬‖2

2

= (𝜸 − 𝜸OLS)𝑈𝐘𝑈𝐘(𝜸 − 𝜸OLS) − 2𝐬𝑈𝐘(𝜸 − 𝜸OLS) + 𝐬𝑈𝐬 which is an ellipse (at least in 2D) centred on 𝜸OLS.

13/24

Intuition for the penalties (II)

The least squares RSS is minimized for 𝜸OLS. If a constraint is added (‖𝜸‖𝑟

𝑟 ≤ 𝑢) then the RSS is minimized by the closest 𝜸

possible that fulfills the constraint.

β1 β2 βOLS

• βlasso

Lasso

β1 β2 βOLS

• βridge

Ridge

The blue lines are the contour lines for the RSS.

14/24

Intuition for the penalties (III)

Depending on 𝑟 the different constraints lead to different

• solutions. If 𝜸OLS is

in one of the coloured areas or

• n a line, the

constrained solution will be at the corresponding dot.

β1 β2

• β1

β2

• β1

β2

• β1

β2

• q: 2

q: Inf q: 0.7 q: 1

15/24

Computational aspects of the Lasso (I)

What estimates does the lasso produce? Target function arg min

𝜸

1 2‖𝐳 − 𝐘𝜸‖2

2 + 𝜇‖𝜸‖1

Special case: 𝐘𝑈𝐘 = 𝐉𝑞. Then 1 2‖𝐳 − 𝐘𝜸‖2

2 + 𝜇‖𝜸‖1 = 1

2𝐳𝑈𝐳 − 𝐳𝑈𝐘 ⏟

=𝜸𝑈

OLS

𝜸 + 1 2𝜸𝑈𝜸 + 𝜇‖𝜸‖1 = 𝑕(𝜸) How do we find the solution ̂ 𝜸 in presence of the non-differentiable penalisation ‖𝜸‖1?

16/24

Computational aspects of the Lasso (II)

For 𝐘𝑈𝐘 = 𝐉𝑞 the target function can be written as arg min

𝜸 𝑞

∑

𝑘=1

−𝛾OLS,𝑘𝛾

𝑘 + 1

2𝛾2

𝑘 + 𝜇|𝛾 𝑘|

This results in 𝑞 uncoupled optimization problems.

▶ If 𝛾OLS,𝑘 > 0, then 𝛾

𝑘 > 0 to minimize the target

▶ If 𝛾OLS,𝑘 ≤ 0, then 𝛾

𝑘 ≤ 0

Each case results in ˆ 𝛾

𝑘 = sign(𝛾OLS,𝑘)(|𝛾OLS,𝑘| − 𝜇)+ = ST(𝛾OLS,𝑘, 𝜇),

where 𝑦+ = {𝑦 𝑦 > 0

• therwise

and ST is the soft-thresholding operator

17/24

Relation to OLS estimates

Both ridge regression and the lasso estimates can be written as functions of 𝜸OLS if 𝐘𝑈𝐘 = 𝐉𝑞. 𝛾ridge,𝑘 = 𝛾OLS,𝑘 1 + 𝜇 and ˆ 𝛾

𝑘 = sign(𝛾OLS,𝑘)(|𝛾OLS,𝑘| − 𝜇)+

λ

Ridge Lasso

Visualisation of the transformations applied to the OLS estimates.

18/24

Shrinkage

When 𝜇 is fixed, the shrinkage of the lasso estimate 𝜸lasso(𝜇) compared to the OLS estimate 𝜸OLS is defined as 𝑡(𝜇) = ‖𝜸lasso(𝜇)‖1 ‖𝜸OLS‖1 Note: 𝑡(𝜇) ∈ [0, 1] with 𝑡(𝜇) → 0 for increasing 𝜇 and 𝑡(𝜇) = 1 if 𝜇 = 0

19/24

A regularisation path

Prostate cancer dataset (𝑜 = 67, 𝑞 = 8)

Red dashed lines indicate the 𝜇 selected by cross-validation

−0.25 0.00 0.25 0.50 0.75 2 4 6 8 Effective degrees of freedom Coefficient

Ridge

−0.25 0.00 0.25 0.50 0.75 0.00 0.25 0.50 0.75 1.00 Shrinkage Coefficient

Lasso

−0.25 0.00 0.25 0.50 0.75 −5 5 10 log(λ) Coefficient −0.25 0.00 0.25 0.50 0.75 −5 5 10 log(λ) Coefficient

20/24

Notes on the lasso

▶ In the general case, i.e. 𝐘𝑈𝐘 ≠ 𝐉𝑞, there is no explicit

solution.

▶ Numerical solution possible, e.g. with coordinate descent ▶ As for ridge regression, estimates are biased ▶ But

▶ Asymptotic consistency: If 𝜇 = o(𝑜) then 𝜸lasso → 𝜸true for

𝑜 → ∞

▶ Model selection consistency: If 𝜇 ∝ 𝑜1/2, then there is a

non-zero probability of identifying the true model

▶ Degrees of freedom: The degrees of freedom are equal to

the number of non-zero coefficients

21/24

Potential caveats of the lasso (I)

▶ Sparsity of the true model:

▶ The lasso only works if the data is generated from a

sparse process.

▶ However, a dense process with many variables and not

enough data or high correlation between predictors can be unidentifiable either way

▶ Correlations: Many non-relevant variables correlated

with relevant variables can lead to the selection of the wrong model, even for large 𝑜

▶ Irrepresentable condition: Split 𝐘 such that 𝐘1 contains

all relevant variables and 𝐘2 contains all irrelevant

• variables. If

|(𝐘𝑈

2 𝐘1)−1(𝐘𝑈 1 𝐘1)| < 1 − 𝜽

for some 𝜽 > 0 then the lasso is (almost) guaranteed to pick the true model

22/24

Potential caveats of the lasso (II)

In practice, both the sparsity of the true model and the irrepresentable condition cannot be checked.

▶ Assumptions and domain knowledge have to be used

23/24

Take-home message

▶ Filtering and wrapping methods useful for feature

selection in practice but can be unprincipled or have high variance

▶ Penalisation gives stability to regression ▶ The lasso performs variable selection and variance

stabilisation at the same time

24/24