Linear and Logistic Regression Yingyu Liang Computer Sciences 760 - - PowerPoint PPT Presentation

Linear and Logistic Regression

Yingyu Liang Computer Sciences 760 Fall 2017

http://pages.cs.wisc.edu/~yliang/cs760/

Some of the slides in these lectures have been adapted/borrowed from materials developed by Mark Craven, David Page, Jude Shavlik, Tom Mitchell, Nina Balcan, Matt Gormley, Elad Hazan, Tom Dietterich, and Pedro Domingos.

Goals for the lecture

• understand the concepts
• linear regression
• closed form solution for linear regression
• lasso
• RMSE, MAE, and R-square
• logistic regression for linear classification
• gradient descent for logistic regression
• multiclass logistic regression

Linear regression

• Given training data

𝑦 𝑗 , 𝑧(𝑗) : 1 ≤ 𝑗 ≤ 𝑛 i.i.d. from distribution 𝐸

• Find 𝑔

𝑥 𝑦 = 𝑥𝑈𝑦 that minimizes ෠

𝑀 𝑔

𝑥 = 1 𝑛 σ𝑗=1 𝑛

𝑥𝑈𝑦(𝑗) − 𝑧(𝑗) 2

𝑚2 loss; also called mean squared error

Hypothesis class 𝓘

Linear regression: optimization

• Given training data

𝑦 𝑗 , 𝑧(𝑗) : 1 ≤ 𝑗 ≤ 𝑛 i.i.d. from distribution 𝐸

• Find 𝑔

𝑥 𝑦 = 𝑥𝑈𝑦 that minimizes ෠

𝑀 𝑔

𝑥 = 1 𝑛 σ𝑗=1 𝑛

𝑥𝑈𝑦(𝑗) − 𝑧(𝑗) 2

• Let 𝑌 be a matrix whose 𝑗-th row is 𝑦(𝑗) 𝑈, 𝑧 be the vector

𝑧(1), … , 𝑧(𝑛)

𝑈

෠ 𝑀 𝑔

𝑥 = 1

𝑛 ෍

𝑗=1 𝑛

𝑥𝑈𝑦(𝑗) − 𝑧(𝑗) 2 = 1 𝑛 ⃦𝑌𝑥 − 𝑧 ⃦2

2

Linear regression: optimization

• Set the gradient to 0 to get the minimizer

𝛼

𝑥 ෠

𝑀 𝑔

𝑥 = 𝛼 𝑥

1 𝑛 ⃦𝑌𝑥 − 𝑧 ⃦2

2 = 0

𝛼

𝑥[ 𝑌𝑥 − 𝑧 𝑈(𝑌𝑥 − 𝑧)] = 0

𝛼

𝑥[ 𝑥𝑈𝑌𝑈𝑌𝑥 − 2𝑥𝑈𝑌𝑈𝑧 + 𝑧𝑈𝑧] = 0

2𝑌𝑈𝑌𝑥 − 2𝑌𝑈𝑧 = 0 w = 𝑌𝑈𝑌 −1𝑌𝑈𝑧

Linear regression: optimization

• Algebraic view of the minimizer
• If 𝑌 is invertible, just solve 𝑌𝑥 = 𝑧 and get 𝑥 = 𝑌−1𝑧
• But typically 𝑌 is a tall matrix

𝑌 𝑥 = 𝑧 𝑌𝑈𝑌 𝑥 = 𝑌𝑈𝑧 Normal equation: w = 𝑌𝑈𝑌 −1𝑌𝑈𝑧

Linear regression with bias

• Given training data

𝑦 𝑗 , 𝑧(𝑗) : 1 ≤ 𝑗 ≤ 𝑛 i.i.d. from distribution 𝐸

• Find 𝑔

𝑥,𝑐 𝑦 = 𝑥𝑈𝑦 + 𝑐 to minimize the loss

• Reduce to the case without bias:
• Let 𝑥′ = 𝑥; 𝑐 , 𝑦′ = 𝑦; 1
• Then 𝑔

𝑥,𝑐 𝑦 = 𝑥𝑈𝑦 + 𝑐 = 𝑥′ 𝑈(𝑦′)

Bias term

Linear regression with lasso penalty

• Given training data

𝑦 𝑗 , 𝑧(𝑗) : 1 ≤ 𝑗 ≤ 𝑛 i.i.d. from distribution 𝐸

• Find 𝑔

𝑥 𝑦 = 𝑥𝑈𝑦 that minimizes

෠ 𝑀 𝑔

𝑥 = 1

𝑛 ෍

𝑗=1 𝑛

𝑥𝑈𝑦(𝑗) − 𝑧(𝑗) 2 + 𝜇 𝑥 1

lasso penalty: 𝑚1 norm of the parameter, encourages sparsity

• Root mean squared error (RMSE)
• Mean absolute error (MAE) – average 𝑚1 error
• R-square (R-squared)
• Historically all were computed on training data, and possibly

adjusted after, but really should cross-validate

Evaluation Metrics

• Formulation 1:
• Formulation 2: square of Pearson correlation coefficient r

between the label and the prediction. Recall for x, y:

R-square

  

    

i i i i i i i

y y x x y y x x r

2 2

) ( ) ( ) )( (

Linear classification

𝑥𝑈𝑦 = 0 Class 1 Class 0 𝑥 𝑥𝑈𝑦 > 0 𝑥𝑈𝑦 < 0

Linear classification: natural attempt

• Given training data

𝑦 𝑗 , 𝑧(𝑗) : 1 ≤ 𝑗 ≤ 𝑛 i.i.d. from distribution 𝐸

• Hypothesis 𝑔

𝑥 𝑦 = 𝑥𝑈𝑦

• 𝑧 = 1 if 𝑥𝑈𝑦 > 0
• 𝑧 = 0 if 𝑥𝑈𝑦 < 0
• Prediction: 𝑧 = step(𝑔

𝑥 𝑦 ) = step(𝑥𝑈𝑦)

Linear model 𝓘

Linear classification: natural attempt

• Given training data

𝑦 𝑗 , 𝑧(𝑗) : 1 ≤ 𝑗 ≤ 𝑛 i.i.d. from distribution 𝐸

• Find 𝑔

𝑥 𝑦 = 𝑥𝑈𝑦 to minimize

෠ 𝑀 𝑔

𝑥 = 1

𝑛 ෍

𝑗=1 𝑛

𝕁[step(𝑥𝑈𝑦 𝑗 ) ≠ 𝑧(𝑗)]

• Drawback: difficult to optimize
• NP-hard in the worst case

0-1 loss

Linear classification: simple approach

• Given training data

𝑦 𝑗 , 𝑧(𝑗) : 1 ≤ 𝑗 ≤ 𝑛 i.i.d. from distribution 𝐸

• Find 𝑔

𝑥 𝑦 = 𝑥𝑈𝑦 that minimizes ෠

𝑀 𝑔

𝑥 = 1 𝑛 σ𝑗=1 𝑛

𝑥𝑈𝑦(𝑗) − 𝑧(𝑗) 2

Reduce to linear regression; ignore the fact 𝑧 ∈ {0,1}

Linear classification: simple approach

Figure borrowed from Pattern Recognition and Machine Learning, Bishop

Drawback: not robust to “outliers”

Compare the two

𝑧 = 𝑥𝑈𝑦

𝑥𝑈𝑦 𝑧

𝑧 = step(𝑥𝑈𝑦)

Between the two

• Prediction bounded in [0,1]
• Smooth
• Sigmoid: 𝜏 𝑏 =

1 1+exp(−𝑏) Figure borrowed from Pattern Recognition and Machine Learning, Bishop

Linear classification: sigmoid prediction

• Squash the output of the linear function

Sigmoid 𝑥𝑈𝑦 = 𝜏 𝑥𝑈𝑦 = 1 1 + exp(−𝑥𝑈𝑦)

• Find 𝑥 that minimizes ෠

𝑀 𝑔

𝑥 = 1 𝑛 σ𝑗=1 𝑛

𝜏(𝑥𝑈𝑦 𝑗 ) − 𝑧(𝑗) 2

Linear classification: logistic regression

• Squash the output of the linear function

Sigmoid 𝑥𝑈𝑦 = 𝜏 𝑥𝑈𝑦 = 1 1 + exp(−𝑥𝑈𝑦)

• A better approach: Interpret as a probability

𝑄

𝑥(𝑧 = 1|𝑦) = 𝜏 𝑥𝑈𝑦 =

1 1 + exp(−𝑥𝑈𝑦) 𝑄

𝑥 𝑧 = 0 𝑦 = 1 − 𝑄 𝑥 𝑧 = 1 𝑦 = 1 − 𝜏 𝑥𝑈𝑦

Linear classification: logistic regression

• Find 𝑔

𝑥 𝑦 = 𝑥𝑈𝑦 that minimizes ෠

𝑀 𝑔

𝑥 = 1 𝑛 σ𝑗=1 𝑛

𝑥𝑈𝑦(𝑗) − 𝑧(𝑗) 2

• Find 𝑥 that minimizes

෠ 𝑀 𝑥 = − 1 𝑛 ෍

𝑗=1 𝑛

log 𝑄

𝑥 𝑧(𝑗) 𝑦(𝑗)

෠ 𝑀 𝑥 = − 1 𝑛 ෍

𝑧(𝑗)=1

log𝜏(𝑥𝑈𝑦(𝑗)) − 1 𝑛 ෍

𝑧(𝑗)=0

log[1 − 𝜏 𝑥𝑈𝑦(𝑗) ]

Logistic regression:

MLE with sigmoid

Linear classification: logistic regression

• Given training data

𝑦 𝑗 , 𝑧(𝑗) : 1 ≤ 𝑗 ≤ 𝑛 i.i.d. from distribution 𝐸

• Find 𝑥 that minimizes

෠ 𝑀 𝑥 = − 1 𝑛 ෍

𝑧(𝑗)=1

log𝜏(𝑥𝑈𝑦(𝑗)) − 1 𝑛 ෍

𝑧(𝑗)=0

log[1 − 𝜏 𝑥𝑈𝑦(𝑗) ]

No close form solution; Need to use gradient descent

Properties of sigmoid function

• Bounded

𝜏 𝑏 = 1 1 + exp(−𝑏) ∈ (0,1)

• Symmetric

1 − 𝜏 𝑏 = exp −𝑏 1 + exp −𝑏 = 1 exp 𝑏 + 1 = 𝜏(−𝑏)

• Gradient

𝜏′(𝑏) = exp −𝑏 1 + exp −𝑏

2 = 𝜏(𝑏)(1 − 𝜏 𝑏 )

Review: binary logistic regression

• Sigmoid

𝜏 𝑥𝑈𝑦 + 𝑐 = 1 1 + exp(−(𝑥𝑈𝑦 + 𝑐))

• Interpret as conditional probability

𝑞𝑥 𝑧 = 1 𝑦 = 𝜏 𝑥𝑈𝑦 + 𝑐 𝑞𝑥 𝑧 = 0 𝑦 = 1 − 𝑞𝑥 𝑧 = 1 𝑦 = 1 − 𝜏 𝑥𝑈𝑦 + 𝑐

• How to extend to multiclass?

Review: binary logistic regression

• Suppose we model the class-conditional densities 𝑞 𝑦 𝑧 = 𝑗 and

class probabilities 𝑞 𝑧 = 𝑗

• Conditional probability by Bayesian rule:

𝑞 𝑧 = 1|𝑦 = 𝑞 𝑦|𝑧 = 1 𝑞(𝑧 = 1) 𝑞 𝑦|𝑧 = 1 𝑞 𝑧 = 1 + 𝑞 𝑦|𝑧 = 2 𝑞(𝑧 = 2) = 1 1 + exp(−𝑏) = 𝜏(𝑏)

where we define

𝑏 ≔ ln 𝑞 𝑦|𝑧 = 1 𝑞(𝑧 = 1) 𝑞 𝑦|𝑧 = 2 𝑞(𝑧 = 2) = ln 𝑞 𝑧 = 1|𝑦 𝑞 𝑧 = 2|𝑦

Review: binary logistic regression

• Suppose we model the class-conditional densities 𝑞 𝑦 𝑧 = 𝑗 and

class probabilities 𝑞 𝑧 = 𝑗

• 𝑞 𝑧 = 1|𝑦 = 𝜏 𝑏 = 𝜏(𝑥𝑈𝑦 + 𝑐) is equivalent to setting log odds

to be linear: 𝑏 = ln 𝑞 𝑧 = 1|𝑦 𝑞 𝑧 = 2|𝑦 = 𝑥𝑈𝑦 + 𝑐

• Why linear log odds?

Review: binary logistic regression

• Suppose the class-conditional densities 𝑞 𝑦 𝑧 = 𝑗 is normal

𝑞 𝑦 𝑧 = 𝑗 = 𝑂 𝑦|𝜈𝑗, 𝐽 = 1 2𝜌 𝑒/2 exp{− 1 2 𝑦 − 𝜈𝑗

2}

• log odd is

𝑏 = ln 𝑞 𝑦|𝑧 = 1 𝑞(𝑧 = 1) 𝑞 𝑦|𝑧 = 2 𝑞(𝑧 = 2) = 𝑥𝑈𝑦 + 𝑐 where 𝑥 = 𝜈1 − 𝜈2, 𝑐 = − 1 2 𝜈1

𝑈𝜈1 + 1

2 𝜈2

𝑈𝜈2 + ln 𝑞(𝑧 = 1)

𝑞(𝑧 = 2)

Multiclass logistic regression

• Suppose we model the class-conditional densities 𝑞 𝑦 𝑧 = 𝑗 and

class probabilities 𝑞 𝑧 = 𝑗

• Conditional probability by Bayesian rule:

𝑞 𝑧 = 𝑗|𝑦 = 𝑞 𝑦|𝑧 = 𝑗 𝑞(𝑧 = 𝑗) σ𝑘 𝑞 𝑦|𝑧 = 𝑘 𝑞(𝑧 = 𝑘) = exp(𝑏𝑗) σ𝑘 exp(𝑏𝑘) where we define 𝑏𝑗 ≔ ln [𝑞 𝑦 𝑧 = 𝑗 𝑞 𝑧 = 𝑗 ]

Multiclass logistic regression

• Suppose the class-conditional densities 𝑞 𝑦 𝑧 = 𝑗 is normal

𝑞 𝑦 𝑧 = 𝑗 = 𝑂 𝑦|𝜈𝑗, 𝐽 = 1 2𝜌 𝑒/2 exp{− 1 2 𝑦 − 𝜈𝑗

2}

• Then

𝑏𝑗 ≔ ln 𝑞 𝑦 𝑧 = 𝑗 𝑞 𝑧 = 𝑗 = − 1 2 𝑦𝑈𝑦 + 𝑥𝑗

𝑈

𝑦 + 𝑐𝑗 where 𝑥𝑗 = 𝜈𝑗, 𝑐𝑗 = − 1 2 𝜈𝑗

𝑈𝜈𝑗 + ln 𝑞 𝑧 = 𝑗 + ln

1 2𝜌 𝑒/2

Multiclass logistic regression

• Suppose the class-conditional densities 𝑞 𝑦 𝑧 = 𝑗 is normal

𝑞 𝑦 𝑧 = 𝑗 = 𝑂 𝑦|𝜈𝑗, 𝐽 = 1 2𝜌 𝑒/2 exp{− 1 2 𝑦 − 𝜈𝑗

2}

• Cancel out −

1 2 𝑦𝑈𝑦, we have

𝑞 𝑧 = 𝑗|𝑦 = exp(𝑏𝑗) σ𝑘 exp(𝑏𝑘) , 𝑏𝑗 ≔ 𝑥𝑗 𝑈𝑦 + 𝑐𝑗 where 𝑥𝑗 = 𝜈𝑗, 𝑐𝑗 = − 1 2 𝜈𝑗

𝑈𝜈𝑗 + ln 𝑞 𝑧 = 𝑗 + ln

1 2𝜌 𝑒/2

Multiclass logistic regression: conclusion

• Suppose the class-conditional densities 𝑞 𝑦 𝑧 = 𝑗 is normal

𝑞 𝑦 𝑧 = 𝑗 = 𝑂 𝑦|𝜈𝑗, 𝐽 = 1 2𝜌 𝑒/2 exp{− 1 2 𝑦 − 𝜈𝑗

2}

• Then

𝑞 𝑧 = 𝑗|𝑦 = exp( 𝑥𝑗 𝑈𝑦 + 𝑐𝑗) σ𝑘 exp( 𝑥𝑘 𝑈𝑦 + 𝑐𝑘) which is the hypothesis class for multiclass logistic regression

• It is softmax on linear transformation; it can be used to derive the

negative log-likelihood loss (cross entropy)

Softmax

• A way to squash 𝑏 = (𝑏1, 𝑏2, … , 𝑏𝑗, … ) into probability vector 𝑞

softmax 𝑏 = exp(𝑏1) σ𝑘 exp(𝑏𝑘) , exp(𝑏2) σ𝑘 exp(𝑏𝑘) , … , exp 𝑏𝑗 σ𝑘 exp 𝑏𝑘 , …

• Behave like max: when 𝑏𝑗 ≫ 𝑏𝑘 ∀𝑘 ≠ 𝑗 , 𝑞𝑗 ≅ 1, 𝑞𝑘 ≅ 0

Cross entropy for conditional distribution

• Let 𝑞data(𝑧|𝑦) denote the empirical distribution of the data
• Negative log-likelihood

−

1 𝑛 σ𝑗=1 𝑛 log 𝑞 𝑧 = 𝑧(𝑗) 𝑦(𝑗) = −E𝑞data(𝑧|𝑦) log 𝑞(𝑧|𝑦)

is the cross entropy between 𝑞data and the model output 𝑞

• Information theory viewpoint: KL divergence

𝐸(𝑞data| 𝑞 = E𝑞data[log

𝑞data 𝑞 ] = E𝑞data [log 𝑞data] − E𝑞data[log 𝑞]

Entropy; constant Cross entropy

Cross entropy for full distribution

• Let 𝑞data(𝑦, 𝑧) denote the empirical distribution of the data
• Negative log-likelihood

−

1 𝑛 σ𝑗=1 𝑛 log 𝑞(𝑦 𝑗 , 𝑧(𝑗)) = −E𝑞data(𝑦,𝑧) log 𝑞(𝑦, 𝑧)

is the cross entropy between 𝑞data and the model output 𝑞