Penalized Linear Regression Prof. Mike Hughes Many slides - - PowerPoint PPT Presentation

Penalized Linear Regression

2

Tufts COMP 135: Introduction to Machine Learning https://www.cs.tufts.edu/comp/135/2020f/

Many slides attributable to: Erik Sudderth (UCI) Finale Doshi-Velez (Harvard) James, Witten, Hastie, Tibshirani (ISL/ESL books)

• Prof. Mike Hughes

Today’s objectives (day 05)

• Recap: Overfitting with high-degree features
• Remedy: Add L2 penalty to the loss (“Ridge”)
• Avoid high magnitude weights
• Remedy: Add L1 penalty to the loss (“Lasso”)
• Avoid high magnitude weights
• Often, some weights exactly zero (feature selection)

3

Mike Hughes - Tufts COMP 135 - Fall 2020

What will we learn?

4

Mike Hughes - Tufts COMP 135 - Fall 2020

Supervised Learning Unsupervised Learning Reinforcement Learning

Data, Label Pairs Performance measure Task data x label y

{xn, yn}N

n=1

Training Prediction Evaluation

5

Mike Hughes - Tufts COMP 135 - Fall 2020

Task: Regression

Supervised Learning Unsupervised Learning Reinforcement Learning

regression

x y y

is a numeric variable e.g. sales in $$

6

Mike Hughes - Tufts COMP 135 - Fall 2020

Review: Linear Regression

Optimization problem: “Least Squares”

Exact formula for estimating optimal parameter vector values:

Can use formula when you observe at least F+1 examples that are linearly independent Otherwise, many theta values yield lowest possible training error (many linear functions make perfect predictions on the training set)

min

θ∈RF +1 N

X

n=1

(yn − ˆ y(xn, θ))2

˜ X =     x11 . . . x1F 1 x21 . . . x2F 1 . . . xN1 . . . xNF 1     y =

     y1 y2 . . . yN     

T = ( ˜

XT ˜ X)−1 ˜ XT y

θ = [

7

Mike Hughes - Tufts COMP 135 - Fall 2020

Review: Linear Regression with Transformed Features

Optimization problem: “Least Squares” Exact solution:

ˆ y(xi) = θT φ(xi) φ(xi) = [1 φ1(xi) φ2(xi) . . . φG−1(xi)] minθ PN

n=1(yn − θT φ(xi))2

θ∗ = (ΦT Φ)−1ΦT y

Φ =      1 φ1(x1) . . . φG−1(x1) 1 φ1(x2) . . . φG−1(x2) . . . ... 1 φ1(xN) . . . φG−1(xN)     

N x G matrix G x 1 vector

0th degree polynomial features

8

Mike Hughes - Tufts COMP 135 - Fall 2020 Credit: Slides from course by Prof. Erik Sudderth (UCI)

• -- true function
• training data
• - predictions from

LR using polynomial features

φ(xi) = [1]

# parameters: G = 1

9

Mike Hughes - Tufts COMP 135 - Fall 2020 Credit: Slides from course by Prof. Erik Sudderth (UCI)

1st degree polynomial features

• -- true function
• training data
• - predictions from

LR using polynomial features

φ(xi) = [1 xi1]

# parameters: G = 2

10

Mike Hughes - Tufts COMP 135 - Fall 2020 Credit: Slides from course by Prof. Erik Sudderth (UCI)

3rd degree polynomial features

• -- true function
• training data
• - predictions from

LR using polynomial features

φ(xi) = [1 xi1 x2

i1 x3 i1]

# parameters: G = 4

9th degree polynomial features

11

Mike Hughes - Tufts COMP 135 - Fall 2020 Credit: Slides from course by Prof. Erik Sudderth (UCI)

• -- true function
• training data
• - predictions from

LR using polynomial features

φ(xi) = [1 xi1 x2

i1 x3 i1 x4 i1 x5 i1 x6 i1 x7 i1 x8 i1 x9 i1]

# parameters: G = 10

Error vs Complexity

12

Mike Hughes - Tufts COMP 135 - Fall 2020

polynomial degree mean squared error

13

Mike Hughes - Tufts COMP 135 - Fall 2020

Polynomial degree

1 3 9

Credit: Slides from course by Prof. Erik Sudderth (UCI)

Weight Values vs Complexity

Estimated Regression Coefficients

WOW! These values are very large.

Idea: Add Penalty Term to Loss

Goal: Avoid finding weights with large magnitude Result: Ridge regression, a method with objective:

14

Mike Hughes - Tufts COMP 135 - Fall 2020

α ≥ 0

Hyperparameter: Penalty strength “alpha”

Alpha = 0 recovers original unpenalized Linear Regression Larger alpha means we prefer smaller magnitude weights

Penalty term: Sum of squares of entries of theta = Square of the “L2 norm” of theta vector Thus, also called “L2-penalized” linear regression

J(θ) =

N

X

n=1

(yn − θT φ(xn))2 + α

G

X

g=1

θ2

g

Rewrite in matrix notation?

15

Mike Hughes - Tufts COMP 135 - Fall 2020

Rewriting, this is equivalent to

Can rewrite sum of squares as an inner product of theta vector with itself

J(θ) =

N

X

n=1

(yn − θT φ(xn))2 + α

G

X

g=1

θ2

g

J(θ) = (y − Φθ)T (y − Φθ) + αθT θ

Φ =      1 φ1(x1) . . . φG−1(x1) 1 φ1(x2) . . . φG−1(x2) . . . ... 1 φ1(xN) . . . φG−1(xN)     

N x G y =      y1 y2 . . . yN     

θ =      θ1 θ2 . . . θG     

N x 1 G x 1

N : num. examples G : num transformed features

Estimating weights for L2 penalized linear regression

16

Mike Hughes - Tufts COMP 135 - Fall 2020

Optimization problem: “Penalized Least Squares” Solution:

If alpha > 0 , the matrix is always invertible! Always one unique optimal theta vector, provided by this formula

min

θ

) = (y − Φθ)T (y − Φθ) + αθT θ

θ∗ = (ΦT Φ + αIG)−1ΦT y

G x G identity matrix

What happens if we rescale a feature in unpenalized LR?

17

Mike Hughes - Tufts COMP 135 - Fall 2020

Suppose we changed units on “volume” of the engine feature, from liters to mL Remember, 1 liter = 1000 mL

Before After

Answer: Just “rescale” the individual weight for that single feature. No other weights will change.

vol_in_L hp mi_per_gal 2.1 115.0 30.0 2.3 150.0 25.0 2.5 193.0 21.2 vol_in_mL hp mi_per_gal 2100 115.0 30.0 2300 150.0 25.0 2500 193.0 21.2

vol [-51.25 ] hp [ 0.15 ] bias [120.375] [ -0.05125 ] [ 0.15 ] [120.375 ]

x:1

x:2

y

What happens if we rescale a feature in penalized LR?

18

Mike Hughes - Tufts COMP 135 - Fall 2020

Suppose we changed units on “volume” of the engine feature, from liters to mL Remember, 1 liter = 1000 mL

Before After

vol_in_L hp mi_per_gal 2.1 115.0 30.0 2.3 150.0 25.0 2.5 193.0 21.2 vol_in_mL hp mi_per_gal 2100 115.0 30.0 2300 150.0 25.0 2500 193.0 21.2

[ 18.428 ] [ -0.199] [ 13.407] [ 0.027 ] [ -0.248 ] [ 1.439 ]

x:1

x:2

y

Answer: Because all weights contribute to the penalty term, ALL learned weights change alpha = 0.01

Ridge Regression is more sensitive to the scale of your features.

19

Mike Hughes - Tufts COMP 135 - Fall 2020

Before feeding data into a Ridge regression model, should standardize the scale

• f all features, so the penalty acts on each feature in more uniform way.
• Rescale each column between 0 and 1
• sklearn’s MinMaxScaler
• https://scikit-learn.org/stable/modules/preprocessing.html#scaling-

features-to-a-range

• Transform each column to have mean 0 and variance 1
• sklearn’s StandardScaler
• https://scikit-

learn.org/stable/modules/generated/sklearn.preprocessing.StandardScale r.html#sklearn.preprocessing.StandardScaler OR, you can impose your own feature-specific penalties if you want.

Lasso Regression (L1 penalty)

20

Mike Hughes - Tufts COMP 135 - Fall 2020

min

θ

(y − Φθ)T (y − Φθ) + α

G

X

g=1

|θg|

Sum of absolute values of entries (aka the L1 norm of the vector theta) N : num. examples G : num transformed features

Like L2 penalty (Ridge), the Lasso objective above encourages small magnitude weights. We’ll see in lab: L1 penalty encourages theta to be a sparse vector At modest alpha values, many entries of theta could be exactly zero. Can use for feature selection (only features with non-zero weights matter)

Today’s objectives (day 05)

• Recap: Overfitting with high-degree features
• Remedy: Add L2 penalty to the loss (“Ridge”)
• Avoid high magnitude weights
• Remedy: Add L1 penalty to the loss (“Lasso”)
• Avoid high magnitude weights
• Often, some weights exactly zero (feature selection)

21

Mike Hughes - Tufts COMP 135 - Fall 2020