Regularization Jia-Bin Huang Virginia Tech Spring 2019 ECE-5424G - - PowerPoint PPT Presentation

Regularization

Jia-Bin Huang Virginia Tech

Spring 2019

ECE-5424G / CS-5824

Administrative

• Women in Data Science Blacksburg
• Location: Holtzman Alumni Center
• Welcome, 3:30 - 3:40, Assembly hall
• Keynote Speaker: Milinda Lakkam,

"Detecting automation on LinkedIn's platform," 3:40 - 4:05, Assembly hall

• Career Panel, 4:05 - 5:00, hall
• Break , 5:00 - 5:20, Grand hallAssembly
• Keynote Speaker: Sally Morton , "Bias," 5:20 - 5:45, Assembly hall
• Dinner with breakout discussion groups, 5:45 - 7:00, Museum
• Introductory track tutorial: Jennifer Van Mullekom, "Data Visualization", 7:00 - 8:15,

Assembly hall

• Advanced track tutorial: Cheryl Danner, "Focal-loss-based Deep Learning for Object

Detection," 7-8:15, 2nd floor board room

k-NN (Classification/Regression)

• Model

𝑦 1 , 𝑧 1 , 𝑦 2 , 𝑧 2 , ⋯ , 𝑦 𝑛 , 𝑧 𝑛

• Cost function

None

• Learning

Do nothing

• Inference

ො 𝑧 = ℎ 𝑦test = 𝑧(𝑙), where 𝑙 = argmin𝑗 𝐸(𝑦test, 𝑦(𝑗))

Linear regression (Regression)

• Model

ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦1 + 𝜄2𝑦2 + ⋯ + 𝜄𝑜𝑦𝑜 = 𝜄⊤𝑦

• Cost function

𝐾 𝜄 = 1 2𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

2

• Learning

1) Gradient descent: Repeat {𝜄

𝑘 ≔ 𝜄 𝑘 − 𝛽 1 𝑛 σ𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗 𝑦𝑘

𝑗 }

2) Solving normal equation 𝜄 = (𝑌⊤𝑌)−1𝑌⊤𝑧

• Inference

ො 𝑧 = ℎ𝜄 𝑦test = 𝜄⊤𝑦test

Naïve Bayes (Classification)

• Model

ℎ𝜄 𝑦 = 𝑄(𝑍|𝑌1, 𝑌2, ⋯ , 𝑌𝑜) ∝ 𝑄 𝑍 Π𝑗𝑄 𝑌𝑗 𝑍)

• Cost function

Maximum likelihood estimation: 𝐾 𝜄 = − log 𝑄 Data 𝜄 Maximum a posteriori estimation :𝐾 𝜄 = − log 𝑄 Data 𝜄 𝑄 𝜄

• Learning

𝜌𝑙 = 𝑄(𝑍 = 𝑧𝑙) (Discrete 𝑌𝑗) 𝜄𝑗𝑘𝑙 = 𝑄(𝑌𝑗 = 𝑦𝑗𝑘𝑙|𝑍 = 𝑧𝑙) (Continuous 𝑌𝑗) mean 𝜈𝑗𝑙, variance 𝜏𝑗𝑙

2 , 𝑄 𝑌𝑗 𝑍 = 𝑧𝑙) = 𝒪(𝑌𝑗|𝜈𝑗𝑙, 𝜏𝑗𝑙

2 )

• Inference

෠ 𝑍 ← argmax

𝑧𝑙

𝑄 𝑍 = 𝑧𝑙 Π𝑗𝑄 𝑌𝑗

test 𝑍 = 𝑧𝑙)

Logistic regression (Classification)

• Model

ℎ𝜄 𝑦 = 𝑄 𝑍 = 1 𝑌1, 𝑌2, ⋯ , 𝑌𝑜 =

1 1+𝑓−𝜄⊤𝑦

• Cost function

𝐾 𝜄 = 1 𝑛 ෍

𝑗=1 𝑛

Cost(ℎ𝜄(𝑦 𝑗 ), 𝑧(𝑗))) Cost(ℎ𝜄 𝑦 , 𝑧) = ൝−log ℎ𝜄 𝑦 if 𝑧 = 1 −log 1 − ℎ𝜄 𝑦 if 𝑧 = 0

• Learning

Gradient descent: Repeat {𝜄

𝑘 ≔ 𝜄 𝑘 − 𝛽 1 𝑛 σ𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗 𝑦𝑘

𝑗 }

• Inference

෠ 𝑍 = ℎ𝜄 𝑦test = 1 1 + 𝑓−𝜄⊤𝑦test

Logistic Regression

• Hypothesis representation
• Cost function
• Logistic regression with gradient descent
• Regularization
• Multi-class classification

ℎ𝜄 𝑦 = 1 1 + 𝑓−𝜄⊤𝑦 Cost(ℎ𝜄 𝑦 , 𝑧) = ൝−log ℎ𝜄 𝑦 if 𝑧 = 1 −log 1 − ℎ𝜄 𝑦 if 𝑧 = 0 𝜄

𝑘 ≔ 𝜄 𝑘 − 𝛽 1

𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧(𝑗) 𝑦𝑘

(𝑗)

How about MAP?

• Maximum conditional likelihood estimate (MCLE)
• Maximum conditional a posterior estimate (MCAP)

𝜄MCLE = argmax

𝜄

ς𝑗=1

𝑛 𝑄𝜄 𝑧(𝑗)|𝑦 𝑗

𝜄MCAP = argmax

𝜄

ς𝑗=1

𝑛 𝑄𝜄 𝑧(𝑗)|𝑦 𝑗

𝑄(𝜄)

Prior 𝑄(𝜄)

• Common choice of 𝑄(𝜄):
• Normal distribution, zero mean, identity covariance
• “Pushes” parameters towards zeros
• Corresponds to Regularization
• Helps avoid very large weights and overfitting

Slide credit: Tom Mitchell

MLE vs. MAP

• Maximum conditional likelihood estimate (MCLE)

𝜄

𝑘 ≔ 𝜄 𝑘 − 𝛽 1

𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧(𝑗) 𝑦𝑘

(𝑗)

• Maximum conditional a posterior estimate (MCAP)

𝜄

𝑘 ≔ 𝜄 𝑘 − 𝛽𝜇𝜄 𝑘 − 𝛽 1

𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧(𝑗) 𝑦𝑘

(𝑗)

Logistic Regression

• Hypothesis representation
• Cost function
• Logistic regression with gradient descent
• Regularization
• Multi-class classification

Multi-class classification

• Email foldering/taggning: Work, Friends, Family, Hobby
• Medical diagrams: Not ill, Cold, Flu
• Weather: Sunny, Cloudy, Rain, Snow

Slide credit: Andrew Ng

Binary classification

𝑦2 𝑦1

Multiclass classification

𝑦2 𝑦1

One-vs-all (one-vs-rest)

𝑦2 𝑦1

Class 1: Class 2: Class 3:

ℎ𝜄

𝑗 𝑦 = 𝑄 𝑧 = 𝑗 𝑦; 𝜄

(𝑗 = 1, 2, 3) 𝑦2 𝑦1 𝑦2 𝑦1 𝑦2 𝑦1

ℎ𝜄

1 𝑦

ℎ𝜄

2 𝑦

ℎ𝜄

3 𝑦

Slide credit: Andrew Ng

One-vs-all

• Train a logistic regression classifier ℎ𝜄

𝑗 𝑦 for

each class 𝑗 to predict the probability that 𝑧 = 𝑗

• Given a new input 𝑦, pick the class 𝑗 that

maximizes max

i

ℎ𝜄

𝑗 𝑦

Slide credit: Andrew Ng

Generative Approach

Ex: Naïve Bayes

Estimate 𝑄(𝑍) and 𝑄(𝑌|𝑍) Prediction

ො 𝑧 = argmax𝑧 𝑄 𝑍 = 𝑧 𝑄(𝑌 = 𝑦|𝑍 = 𝑧)

Discriminative Approach

Ex: Logistic regression

Estimate 𝑄(𝑍|𝑌) directly (Or a discriminant function: e.g., SVM) Prediction

ො 𝑧 = 𝑄(𝑍 = 𝑧|𝑌 = 𝑦)

Further readings

• Tom M. Mitchell

Generative and discriminative classifiers: Naïve Bayes and Logistic Regression http://www.cs.cmu.edu/~tom/mlbook/NBayesLogReg.pdf

• Andrew Ng, Michael Jordan

On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes http://papers.nips.cc/paper/2020-on-discriminative-vs-generative- classifiers-a-comparison-of-logistic-regression-and-naive-bayes.pdf

Regularization

• Overfitting
• Cost function
• Regularized linear regression
• Regularized logistic regression

Regularization

• Overfitting
• Cost function
• Regularized linear regression
• Regularized logistic regression

Example: Linear regression

Price ($) in 1000’s Size in feet^2 Price ($) in 1000’s Size in feet^2 Price ($) in 1000’s Size in feet^2

ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦 ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦 + 𝜄2𝑦2 ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦 + 𝜄2𝑦2 + 𝜄3𝑦3 + 𝜄4𝑦4 + ⋯

Underfitting Overfitting Just right

Slide credit: Andrew Ng

Overfitting

• If we have too many features (i.e. complex model), the

learned hypothesis may fit the training set very well 𝐾 𝜄 = 1 2𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

2 ≈ 0

but fail to generalize to new examples (predict prices on new examples).

Slide credit: Andrew Ng

Example: Linear regression

Price ($) in 1000’s Size in feet^2 Price ($) in 1000’s Size in feet^2 Price ($) in 1000’s Size in feet^2

ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦 ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦 + 𝜄2𝑦2 ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦 + 𝜄2𝑦2 + 𝜄3𝑦3 + 𝜄4𝑦4 + ⋯

Underfitting Overfitting Just right

High bias High variance

Slide credit: Andrew Ng

Bias-Variance Tradeoff

• Bias: difference between

what you expect to learn and truth

• Measures how well you expect to represent true solution
• Decreases with more complex model
• Variance: difference between

what you expect to learn and what you learn from a particular dataset

• Measures how sensitive learner is to specific dataset
• Increases with more complex model

Low variance High variance Low bias High bias

Bias–variance decomposition

• Training set { 𝑦1, 𝑧1 , 𝑦2, 𝑧2 , ⋯ , 𝑦𝑜, 𝑧𝑜 }
• 𝑧 = 𝑔 𝑦 + 𝜁
• We want መ

𝑔 𝑦 that minimizes 𝐹 𝑧 − መ 𝑔 𝑦

2

𝐹 𝑧 − መ 𝑔 𝑦

2

= Bias መ 𝑔 𝑦

2 + Var መ

𝑔 𝑦 + 𝜏2

Bias መ 𝑔 𝑦 = 𝐹 መ 𝑔 𝑦 − 𝑔(𝑦) Var መ 𝑔 𝑦 = 𝐹 መ 𝑔 𝑦 2 − 𝐹 መ 𝑔 𝑦

2

https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff

Overfitting

Tumor Size Age Tumor Size Age Tumor Size Age

ℎ𝜄 𝑦 = 𝑕(𝜄0 + 𝜄1𝑦 + 𝜄2𝑦2) ℎ𝜄 𝑦 = 𝑕(𝜄0 + 𝜄1𝑦 + 𝜄2𝑦2 + 𝜄3𝑦1

2 + 𝜄4𝑦2 2 + 𝜄5𝑦1𝑦2)

ℎ𝜄 𝑦 = 𝑕(𝜄0 + 𝜄1𝑦 + 𝜄2𝑦2 + 𝜄3𝑦1

2 + 𝜄4𝑦2 2 + 𝜄5𝑦1𝑦2 +

𝜄6𝑦1

3𝑦2 + 𝜄7𝑦1𝑦2 3 + ⋯ )

Underfitting Overfitting

Slide credit: Andrew Ng

Addressing overfitting

• 𝑦1 = size of house
• 𝑦2 = no. of bedrooms
• 𝑦3 = no. of floors
• 𝑦4 = age of house
• 𝑦5 = average income in neighborhood
• 𝑦6 = kitchen size
• ⋮
• 𝑦100

Price ($) in 1000’s Size in feet^2

Slide credit: Andrew Ng

Addressing overfitting

• 1. Reduce number of features.
• Manually select which features to keep.
• Model selection algorithm (later in course).
• 2. Regularization.
• Keep all the features, but reduce magnitude/values of parameters

𝜄

𝑘.

• Works well when we have a lot of features, each of which

contributes a bit to predicting 𝑧.

Slide credit: Andrew Ng

Overfitting Thriller

• https://www.youtube.com/watch?v=DQWI1kvmwRg

Regularization

• Overfitting
• Cost function
• Regularized linear regression
• Regularized logistic regression

Intuition

• Suppose we penalize and make 𝜄3, 𝜄4 really small.

min

𝜄 𝐾 𝜄 = 1

2𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

2 + 1000 𝜄3 2 + 1000 𝜄4 2

ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦 + 𝜄2𝑦2 ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦 + 𝜄2𝑦2 + 𝜄3𝑦3 + 𝜄4𝑦4

Price ($) in 1000’s Size in feet^2 Price ($) in 1000’s Size in feet^2

Slide credit: Andrew Ng

Regularization.

• Small values for parameters 𝜄1, 𝜄2, ⋯ , 𝜄𝑜
• “Simpler” hypothesis
• Less prone to overfitting
• Housing:
• Features: 𝑦1, 𝑦2, ⋯ , 𝑦100
• Parameters: 𝜄0, 𝜄1, 𝜄2, ⋯ , 𝜄100

𝐾 𝜄 = 1 2𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

2 + 𝜇 ෍ 𝑘=1 𝑜

𝜄

𝑘 2

Slide credit: Andrew Ng

Regularization

𝐾 𝜄 = 1 2𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

2 + 𝜇 ෍ 𝑘=1 𝑜

𝜄

𝑘 2

min

𝜄 𝐾(𝜄)

Price ($) in 1000’s Size in feet^2

𝜇: Regularization parameter

Slide credit: Andrew Ng

Question

𝐾 𝜄 = 1 2𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

2 + 𝜇 ෍ 𝑘=1 𝑜

𝜄

𝑘 2

What if 𝜇 is set to an extremely large value (say 𝜇 = 1010)?

• 1. Algorithm works fine; setting to be very large can’t hurt it
• 2. Algorithm fails to eliminate overfitting.
• 3. Algorithm results in underfitting. (Fails to fit even training data well).
• 4. Gradient descent will fail to converge.

Slide credit: Andrew Ng

Question

𝐾 𝜄 = 1 2𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

2 + 𝜇 ෍ 𝑘=1 𝑜

𝜄

𝑘 2

What if 𝜇 is set to an extremely large value (say 𝜇 = 1010)?

Price ($) in 1000’s Size in feet^2

ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦1 + 𝜄2𝑦2 + ⋯ + 𝜄𝑜𝑦𝑜 = 𝜄⊤𝑦

Slide credit: Andrew Ng

Regularization

• Overfitting
• Cost function
• Regularized linear regression
• Regularized logistic regression

Regularized linear regression

𝐾 𝜄 = 1 2𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

2 + 𝜇 ෍ 𝑘=1 𝑜

𝜄

𝑘 2

min

𝜄 𝐾(𝜄)

𝑜: Number of features 𝜄0 is not panelized

Slide credit: Andrew Ng

Gradient descent (Previously)

Repeat { 𝜄0 ≔ 𝜄0 − 𝛽 1 𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗 𝜄

𝑘 ≔ 𝜄 𝑘 − 𝛽 1

𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗 𝑦𝑘

𝑗

} (𝑘 = 1, 2, 3, ⋯ , 𝑜)

Slide credit: Andrew Ng

(𝑘 = 0)

Gradient descent (Regularized)

Repeat { 𝜄0 ≔ 𝜄0 − 𝛽 1 𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗 𝜄

𝑘 ≔ 𝜄 𝑘 − 𝛽 1

𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗 𝑦𝑘

𝑗 + 𝜇𝜄 𝑘

}

𝜄

𝑘 ≔ 𝜄 𝑘(1 − 𝛽 𝜇

𝑛) − 𝛽 1 𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗 𝑦𝑘

𝑗

Slide credit: Andrew Ng

Comparison

Regularized linear regression

𝜄

𝑘 ≔ 𝜄 𝑘(1 − 𝛽 𝜇

𝑛) − 𝛽 1 𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗 𝑦𝑘

𝑗

Un-regularized linear regression

𝜄

𝑘 ≔ 𝜄 𝑘

− 𝛽 1 𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗 𝑦𝑘

𝑗

1 − 𝛽

𝜇 𝑛 < 1: Weight decay

Normal equation

• 𝑌 =

𝑦 1 ⊤ 𝑦 2 ⊤ ⋮ 𝑦 𝑛 ⊤ ∈ 𝑆𝑛×(𝑜+1) 𝑧 = 𝑧(1) 𝑧(2) ⋮ 𝑧(𝑛) ∈ 𝑆𝑛

• min

𝜄 𝐾(𝜄)

• 𝜄 =

𝑌⊤𝑌 + 𝜇 ⋯ 1 ⋮ ⋮ ⋱ ⋮ 1

−1

𝑌⊤𝑧

(𝑜 + 1 ) × (𝑜 + 1)

Slide credit: Andrew Ng

Regularization

• Overfitting
• Cost function
• Regularized linear regression
• Regularized logistic regression

Regularized logistic regression

• Cost function:

𝐾 𝜄 = 1 𝑛 ෍

𝑗=1 𝑛

𝑧 𝑗 log ℎ𝜄 𝑦 𝑗 + (1 − 𝑧 𝑗 ) log 1 − ℎ𝜄 𝑦 𝑗 + 𝜇 2 ෍

𝑘=1 𝑜

𝜄

𝑘 2

Tumor Size Age

ℎ𝜄 𝑦 = 𝑕(𝜄0 + 𝜄1𝑦 + 𝜄2𝑦2 + 𝜄3𝑦1

2 + 𝜄4𝑦2 2 + 𝜄5𝑦1𝑦2 +

𝜄6𝑦1

3𝑦2 + 𝜄7𝑦1𝑦2 3 + ⋯ )

Slide credit: Andrew Ng

Gradient descent (Regularized)

Repeat { 𝜄0 ≔ 𝜄0 − 𝛽 1 𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗 𝜄

𝑘 ≔ 𝜄 𝑘 − 𝛽 1

𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗 𝑦𝑘

𝑗 − 𝜇𝜄 𝑘

} ℎ𝜄 𝑦 = 1 1 + 𝑓−𝜄⊤𝑦 𝜖 𝜖𝜄

𝑘

𝐾(𝜄)

Slide credit: Andrew Ng

𝜄 1: Lasso regularization

𝐾 𝜄 = 1 2𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

2 + 𝜇 ෍ 𝑘=1 𝑜

|𝜄

𝑘|

LASSO: Least Absolute Shrinkage and Selection Operator

Single predictor: Soft Thresholding

• minimize𝜄

1 2𝑛 σ𝑗=1 𝑛

𝑦(𝑗)𝜄 − 𝑧 𝑗

2 + 𝜇 𝜄 1

෠ 𝜄 = 1 𝑛 < 𝒚, 𝒛 > −𝜇 if 1 𝑛 < 𝒚, 𝒛 > > 𝜇 if 1 𝑛 | < 𝒚, 𝒛 > | ≤ 𝜇 1 𝑛 < 𝒚, 𝒛 > +𝜇 if 1 𝑛 < 𝒚, 𝒛 > < −𝜇

෠ 𝜄 = 𝑇𝜇(1 𝑛 < 𝒚, 𝒛 >) Soft Thresholding operator 𝑇𝜇 𝑦 = sign 𝑦 𝑦 − 𝜇 +

Multiple predictors: : Cyclic Coordinate Desce

• minimize𝜄

1 2𝑛 σ𝑗=1 𝑛

𝑦𝑘

𝑗 𝜄 𝑘 + σ𝑙≠𝑘 𝑦𝑗𝑘 𝑗 𝜄𝑙 − 𝑧 𝑗 2

+ 𝜇 ෍

𝑙≠𝑘

|𝜄𝑙| + 𝜇 𝜄

𝑘 1

For each 𝑘, update 𝜄

𝑘 with

minimize𝜄 1 2𝑛 ෍

𝑗=1 𝑛

𝑦𝑘

𝑗 𝜄 𝑘 − 𝑠 𝑘 (𝑗) 2

+ 𝜇 𝜄

𝑘 1

where 𝑠

𝑘 (𝑗) = 𝑧 𝑗 − σ𝑙≠𝑘 𝑦𝑗𝑘 𝑗 𝜄𝑙

L1 and L2 balls

Image credit: https://web.stanford.edu/~hastie/StatLearnSparsity_files/SLS.pdf

Terminology

Regularization function Name Solver

𝜄 2

2 = ෍ 𝑘=1 𝑜

𝜄

𝑘 2

Tikhonov regularization Ridge regression Close form 𝜄

1 = ෍ 𝑘=1 𝑜

|𝜄𝑘| LASSO regression Proximal gradient descent, least angle regression 𝛽 𝜄

1 + (1 − 𝛽)

𝜄 2

2

Elastic net regularization Proximal gradient descent

Things to remember

• Overfitting
• Cost function
• Regularized linear regression
• Regularized logistic regression