Linear Regression Jia-Bin Huang Virginia Tech Spring 2019 - - PowerPoint PPT Presentation

Linear Regression

Jia-Bin Huang Virginia Tech

Spring 2019

ECE-5424G / CS-5824

Administrative

• Office hour
• Chen Gao
• Shih-Yang Su
• Feedback (Thanks!)
• Notation?
• More descriptive slides?
• Video/audio recording?
• TA hours (uniformly spread over the week)?

Recap: Machine learning algorithms

Supervised Learning Unsupervised Learning Discrete Classification Clustering Continuous Regression Dimensionality reduction

Recap: Nearest neighbor classifier

• Training data

𝑦 1 , 𝑧 1 , 𝑦 2 , 𝑧 2 , ⋯ , 𝑦 𝑂 , 𝑧 𝑂

• Learning

Do nothing.

• Testing

ℎ 𝑦 = 𝑧(𝑙), where 𝑙 = argmini 𝐸(𝑦, 𝑦(𝑗))

Recap: Instance/Memory-based Learning

• 1. A distance metric
• Continuous? Discrete? PDF? Gene data? Learn the metric?
• 2. How many nearby neighbors to look at?
• 1? 3? 5? 15?
• 3. A weighting function (optional)
• Closer neighbors matter more
• 4. How to fit with the local points?
• Kernel regression

Slide credit: Carlos Guestrin

Validation set

• Spliting training set: A fake test set to tune hyper-parameters

Slide credit: CS231 @ Stanford

Cross-validation

• 5-fold cross-validation -> split the training data into 5 equal folds
• 4 of them for training and 1 for validation

Slide credit: CS231 @ Stanford

Things to remember

• Supervised Learning
• Training/testing data; classification/regression; Hypothesis
• k-NN
• Simplest learning algorithm
• With sufficient data, very hard to beat “strawman” approach
• Kernel regression/classification
• Set k to n (number of data points) and chose kernel width
• Smoother than k-NN
• Problems with k-NN
• Curse of dimensionality
• Not robust to irrelevant features
• Slow NN search: must remember (very large) dataset for prediction

Today’s plan: Linear Regression

• Model representation
• Cost function
• Gradient descent
• Features and polynomial regression
• Normal equation

Linear Regression

• Model representation
• Cost function
• Gradient descent
• Features and polynomial regression
• Normal equation

Training set Learning Algorithm ℎ 𝑦 𝑧

Hypothesis Size of house Estimated price

Regression real-valued output

House pricing prediction

Price ($) in 1000’s 500 1000 1500 2000 2500 100 200 300 400 Size in feet^2

• Notation:
• 𝑛 = Number of training examples
• 𝑦 = Input variable / features
• 𝑧 = Output variable / target variable
• (𝑦, 𝑧) = One training example
• (𝑦(𝑗), 𝑧(𝑗)) = 𝑗𝑢ℎ training example

Training set

Size in feet^2 (x) Price ($) in 1000’s (y) 2104 460 1416 232 1534 315 852 178 … …

𝑛 = 47 Examples: 𝑦(1) = 2104 𝑦(2) = 1416 𝑧(1) = 460

Slide credit: Andrew Ng

Model representation

𝑧 = ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦

Shorthand ℎ 𝑦

Training set Learning Algorithm ℎ 𝑦 𝑧

Hypothesis Size of house Estimated price

Price ($) in 1000’s 500 1000 1500 2000 2500 100 200 300 400 Size in feet^2

Univariate linear regression

Slide credit: Andrew Ng

Linear Regression

• Model representation
• Cost function
• Gradient descent
• Features and polynomial regression
• Normal equation

Training set

• Hypothesis ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦

𝜄0, 𝜄1: parameters/weights How to choose 𝜄𝑗’s?

Size in feet^2 (x) Price ($) in 1000’s (y) 2104 460 1416 232 1534 315 852 178 … …

𝑛 = 47

Slide credit: Andrew Ng

ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦

1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3

𝜄0 = 1.5 𝜄1 = 0 𝜄0 = 0 𝜄1 = 0.5 𝜄0 = 1 𝜄1 = 0.5

𝑦 𝑧 𝑦 𝑧 𝑦 𝑧

Slide credit: Andrew Ng

Cost function

• Idea:

Choose 𝜄0, 𝜄1 so that ℎ𝜄 𝑦 is close to 𝑧 for our training example (𝑦, 𝑧)

Price ($) in 1000’s 500 1000 1500 2000 2500 100 200 300 400 Size in feet^2

𝑦 𝑧

ℎ𝜄 𝑦 𝑗 = 𝜄0 + 𝜄1𝑦(𝑗)

𝐾 𝜄0, 𝜄1 = 1 2𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

2

minimize

1 2𝑛 σ𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

2

𝜄0, 𝜄1

minimize 𝐾 𝜄0, 𝜄1

𝜄0, 𝜄1

Cost function

Slide credit: Andrew Ng

• Hypothesis:

ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦

• Parameters:

𝜄0, 𝜄1

• Cost function:

𝐾 𝜄0, 𝜄1 = 1 2𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

2

• Goal:

minimize 𝐾 𝜄0, 𝜄1

𝜄0, 𝜄1

• Hypothesis:

ℎ𝜄 𝑦 = 𝜄1𝑦 𝜄0 = 0

• Parameters:

𝜄1

• Cost function:

𝐾 𝜄1 = 1 2𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

2

• Goal:

minimize 𝐾 𝜄1

Simplified

𝜄0, 𝜄1

Slide credit: Andrew Ng

ℎ𝜄 𝑦 , function of 𝑦

1 2 3 1 2 3 𝑦 𝑧 1 2 3 1 2 3 𝐾 𝜄1 𝜄1

𝐾 𝜄1 , function of 𝜄1

Slide credit: Andrew Ng

ℎ𝜄 𝑦 , function of 𝑦

1 2 3 1 2 3 𝑦 𝑧 1 2 3 1 2 3 𝐾 𝜄1 𝜄1

𝐾 𝜄1 , function of 𝜄1

Slide credit: Andrew Ng

ℎ𝜄 𝑦 , function of 𝑦

1 2 3 1 2 3 𝑦 𝑧 1 2 3 1 2 3 𝐾 𝜄1 𝜄1

𝐾 𝜄1 , function of 𝜄1

Slide credit: Andrew Ng

ℎ𝜄 𝑦 , function of 𝑦

1 2 3 1 2 3 𝑦 𝑧 1 2 3 1 2 3 𝐾 𝜄1 𝜄1

𝐾 𝜄1 , function of 𝜄1

Slide credit: Andrew Ng

ℎ𝜄 𝑦 , function of 𝑦

1 2 3 1 2 3 𝑦 𝑧 1 2 3 1 2 3 𝐾 𝜄1 𝜄1

𝐾 𝜄1 , function of 𝜄1

Slide credit: Andrew Ng

• Hypothesis: ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦
• Parameters: 𝜄0, 𝜄1
• Cost function: 𝐾 𝜄0, 𝜄1 =

1 2𝑛 σ𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

2

• Goal: minimize 𝐾 𝜄0, 𝜄1

𝜄0, 𝜄1

Slide credit: Andrew Ng

Cost function

Slide credit: Andrew Ng

How do we find good 𝜄0, 𝜄1 that minimize 𝐾 𝜄0, 𝜄1 ?

Slide credit: Andrew Ng

Linear Regression

• Model representation
• Cost function
• Gradient descent
• Features and polynomial regression
• Normal equation

Gradient descent

Have some function 𝐾 𝜄0, 𝜄1 Want argmin 𝐾 𝜄0, 𝜄1 Outline:

• Start with some 𝜄0, 𝜄1
• Keep changing 𝜄0, 𝜄1 to reduce 𝐾 𝜄0, 𝜄1

until we hopefully end up at minimum

𝜄0, 𝜄1

Slide credit: Andrew Ng

Slide credit: Andrew Ng

Gradient descent

Repeat until convergence{ 𝜄

𝑘 ≔ 𝜄 𝑘 − 𝛽 𝜖 𝜖𝜄𝑘 𝐾 𝜄0, 𝜄1

(for 𝑘 = 0 and 𝑘 = 1) } 𝛽: Learning rate (step size)

𝜖 𝜖𝜄𝑘 𝐾 𝜄0, 𝜄1 : derivative (rate of change)

Slide credit: Andrew Ng

Gradient descent

Incorrect: temp0 ≔ 𝜄0 −𝛽 𝜖 𝜖𝜄0 𝐾 𝜄0, 𝜄1 𝜄0 ≔ temp0 temp1 ≔ 𝜄1 −𝛽 𝜖 𝜖𝜄1 𝐾 𝜄0, 𝜄1 𝜄1 ≔ temp1 Correct: simultaneous update temp0 ≔ 𝜄0 −𝛽 𝜖 𝜖𝜄0 𝐾 𝜄0, 𝜄1 temp1 ≔ 𝜄1 −𝛽 𝜖 𝜖𝜄1 𝐾 𝜄0, 𝜄1 𝜄0 ≔ temp0 𝜄1 ≔ temp1

Slide credit: Andrew Ng

𝜄1 ≔ 𝜄1 − 𝛽 𝜖 𝜖𝜄1 𝐾 𝜄1

1 2 3 1 2 3 𝐾 𝜄1 𝜄1

𝜖 𝜖𝜄1 𝐾 𝜄1 > 0 𝜖 𝜖𝜄1 𝐾 𝜄1 < 0

Slide credit: Andrew Ng

Learning rate

Gradient descent for linear regression

Repeat until convergence{ 𝜄

𝑘 ≔ 𝜄 𝑘 − 𝛽 𝜖 𝜖𝜄𝑘 𝐾 𝜄0, 𝜄1

(for 𝑘 = 0 and 𝑘 = 1) }

• Linear regression model

ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦 𝐾 𝜄0, 𝜄1 = 1 2𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

2

Slide credit: Andrew Ng

Computing partial derivative

• 𝜖

𝜖𝜄𝑘 𝐾 𝜄0, 𝜄1 = 𝜖 𝜖𝜄𝑘 1 2𝑛 σ𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

2

=

𝜖 𝜖𝜄𝑘 1 2𝑛 σ𝑗=1 𝑛

𝜄0 + 𝜄1𝑦(𝑗) − 𝑧 𝑗

2

• 𝑘 = 0:

𝜖 𝜖𝜄0 𝐾 𝜄0, 𝜄1 = 1 𝑛 σ𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

• 𝑘 = 1:

𝜖 𝜖𝜄1 𝐾 𝜄0, 𝜄1 = 1 𝑛 σ𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗 𝑦 𝑗

Slide credit: Andrew Ng

Gradient descent for linear regression

Repeat until convergence{ 𝜄0 ≔ 𝜄0 − 𝛽 1 𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗 𝜄1 ≔ 𝜄1 − 𝛽 1 𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗 𝑦 𝑗 } Update 𝜄0 and 𝜄1 simultaneously

Slide credit: Andrew Ng

Batch gradient descent

• “Batch”: Each step of gradient descent uses all the training examples

Repeat until convergence{ 𝜄0 ≔ 𝜄0 − 𝛽 1 𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗 𝜄1 ≔ 𝜄1 − 𝛽 1 𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗 𝑦 𝑗 }

𝑛: Number of training examples

Slide credit: Andrew Ng

Linear Regression

• Model representation
• Cost function
• Gradient descent
• Features and polynomial regression
• Normal equation

Training dataset

Size in feet^2 (x) Price ($) in 1000’s (y) 2104 460 1416 232 1534 315 852 178 … …

ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦

Slide credit: Andrew Ng

Multiple features (input variables)

Size in feet^2 (𝑦1) Number of bedrooms (𝑦2) Number of floors (𝑦3) Age of home (years) (𝑦4) Price ($) in 1000’s (y) 2104 5 1 45 460 1416 3 2 40 232 1534 3 2 30 315 852 2 1 36 178 … … Notation: 𝑜 = Number of features 𝑦(𝑗)= Input features of 𝑗𝑢ℎ training example 𝑦𝑘

(𝑗)= Value of feature 𝑘 in 𝑗𝑢ℎ training example

𝑦3

(2) =?

𝑦3

(4) =?

Slide credit: Andrew Ng

Hypothesis

Previously: ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦 Now: ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦1 + 𝜄2𝑦2 +𝜄3 𝑦3 + 𝜄4𝑦4

Slide credit: Andrew Ng

ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦1 + 𝜄2𝑦2 + ⋯ + 𝜄𝑜𝑦𝑜

• For convenience of notation, define 𝑦0 = 1

(𝑦0

(𝑗) = 1 for all examples)

• 𝒚 =

𝑦0 𝑦1 𝑦2 ⋮ 𝑦𝑜 ∈ 𝑆𝑜+1 𝜾 = 𝜄0 𝜄1 𝜄2 ⋮ 𝜄𝑜 ∈ 𝑆𝑜+1

• ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦1 + 𝜄2𝑦2 + ⋯ + 𝜄𝑜𝑦𝑜

= 𝜾⊤𝒚

Slide credit: Andrew Ng

Gradient descent

• Previously (𝑜 = 1)

Repeat until convergence{ 𝜄0 ≔ 𝜄0 − 𝛽 1 𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗 𝜄1 ≔ 𝜄1 − 𝛽 1 𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗 𝑦 𝑗 }

• New algorithm (𝑜 ≥ 1)

Repeat until convergence{ 𝜄

𝑘 ≔ 𝜄 𝑘 − 𝛽 1

𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗 𝑦𝑘

𝑗

} Simultaneously update 𝜄

𝑘, for 𝑘 = 0, 1, ⋯ , 𝑜

Slide credit: Andrew Ng

Gradient descent in practice: Feature scaling

• Idea: Make sure features are on a similar scale (e.g,. −1 ≤ 𝑦𝑗≤ 1)
• E.g. 𝑦1 = size (0-2000 feat^2)

𝑦2 = number of bedrooms (1-5)

1 2 3 1 2 3 𝜄1 𝜄2 1 2 3 1 2 3 𝜄1 𝜄2

Slide credit: Andrew Ng

Gradient descent in practice: Learning rate

• Automatic convergence test
• 𝛽 too small: slow convergence
• 𝛽 too large: may not converge
• To choose 𝛽, try

0.001, … 0.01, …, 0.1, … , 1

Image credit: CS231n@Stanford

House prices prediction

• ℎ𝜄 𝑦 = 𝜄0 + 𝜄1 × frontage + 𝜄2 × depth
• Area

𝑦 = frontage × depth

• ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦

Slide credit: Andrew Ng

Polynomial regression

• ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦1 + 𝜄2𝑦2 + 𝜄3𝑦3

= 𝜄0 + 𝜄1 𝑡𝑗𝑨𝑓 + 𝜄2 𝑡𝑗𝑨𝑓 2 + 𝜄3 𝑡𝑗𝑨𝑓 3

Price ($) in 1000’s 500 1000 1500 2000 2500 100 200 300 400 Size in feet^2

𝑦1 = (size) 𝑦2 = (size)^2 𝑦3 = (size)^3

Slide credit: Andrew Ng

Linear Regression

• Model representation
• Cost function
• Gradient descent
• Features and polynomial regression
• Normal equation

(𝑦0) Size in feet^2 (𝑦1) Number of bedrooms (𝑦2) Number of floors (𝑦3) Age of home (years) (𝑦4) Price ($) in 1000’s (y) 1 2104 5 1 45 460 1 1416 3 2 40 232 1 1534 3 2 30 315 1 852 2 1 36 178 … …

𝑧 = 460 232 315 178

𝜄 = (𝑌⊤𝑌)−1𝑌⊤𝑧

Slide credit: Andrew Ng

Least square solution

• 𝐾 𝜄 =

1 2𝑛 σ𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

2

=

1 2𝑛 σ𝑗=1 𝑛

𝜄⊤𝑦(𝑗) − 𝑧 𝑗

2

=

1 2𝑛 𝑌𝜄 − 𝑧 2 2

• 𝜖

𝜖𝜄 𝐾 𝜄 = 0

• 𝜄 = (𝑌⊤𝑌)−1𝑌⊤𝑧

Justification/interpretation 1

• Loss minimization

𝐾 𝜄 = 1 2𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

2 = 1

𝑛 ෍

𝑗=1 𝑛

𝑀𝑚𝑡 ℎ𝜄 𝑦 𝑗 , 𝑧 𝑗

• 𝑀𝑚𝑡 𝑧, ො

𝑧 =

1 2 𝑧 − ො

𝑧 2

2: Least squares loss

• Empirical Risk Minimization (ERM)

1 𝑛 ෍

𝑗=1 𝑛

𝑀𝑚𝑡 𝑧 𝑗 , ො 𝑧

Justification/interpretation 2

• Probabilistic model
• Assume linear model with Gaussian errors

𝑞𝜄 𝑧 𝑗 𝑦 𝑗 = 1 2𝜌𝜏2 exp(− 1 2𝜏2 (𝑧 𝑗 − 𝜄⊤𝑦 𝑗 )

• Solving maximum likelihood

argmin

𝜄

ෑ

𝑗=1 𝑛

𝑞𝜄 𝑧 𝑗 𝑦 𝑗 argmin

𝜄

log(ෑ

𝑗=1 𝑛

𝑞 𝑧 𝑗 𝑦 𝑗 ) = argmin

𝜄

1 2𝜏2 ෍

𝑗=1 𝑛 1

2 𝜄⊤𝑦 𝑗 − 𝑧 𝑗

2

Image credit: CS 446@UIUC

Justification/interpretation 3

• Geometric interpretation

𝒀 = 1 ← 𝒚(1) → 1 ⋮ 1 ← 𝒚(2) → ⋮ ← 𝒚(𝑛) → = ↑ 𝒜𝟏 ↑ 𝒜𝟐 ↑ 𝒜𝟑 ⋯ ↑ 𝒜𝒐 ↓ ↓ ↓ ↓

• 𝒀𝜾: column space of 𝒀 or span({𝒜𝟏, 𝒜𝟐, ⋯ , 𝒜𝒐})
• Residual 𝒀𝜾 − 𝐳 is orthogonal to the column space of 𝒀
• 𝒀⊤ 𝒀𝜾 − 𝐳 = 0 → (𝒀⊤𝒀)𝜾 = 𝒀⊤𝒛

𝒛

column space of 𝒀

𝒀𝜾 𝒀𝜾 − 𝒛

𝑛 training examples, 𝑜 features

Gradient Descent

• Need to choose 𝛽
• Need many iterations
• Works well even when

𝑜 is large Normal Equation

• No need to choose 𝛽
• Don’t need to iterate
• Need to compute

(𝑌⊤𝑌)−1

• Slow if 𝑜 is very large

Slide credit: Andrew Ng

Things to remember

• Model representation

ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦1 + 𝜄2𝑦2 + ⋯ + 𝜄𝑜𝑦𝑜 = 𝜄⊤𝑦

• Cost function

𝐾 𝜄 =

1 2𝑛 σ𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

2

• Gradient descent for linear regression

Repeat until convergence {𝜄

𝑘 ≔ 𝜄 𝑘 − 𝛽 1 𝑛 σ𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗 𝑦𝑘

𝑗 }

• Features and polynomial regression

Can combine features; can use different functions to generate features (e.g., polynomial)

• Normal equation 𝜄 = (𝑌⊤𝑌)−1𝑌⊤𝑧

Next

• Naïve Bayes, Logistic regression, Regularization