Logistic Regression Jia-Bin Huang Virginia Tech Spring 2019 - - PowerPoint PPT Presentation

Logistic Regression

Jia-Bin Huang Virginia Tech

Spring 2019

ECE-5424G / CS-5824

Administrative

• Please start HW 1 early!
• Questions are welcome!

Two principles for estimating parameters

• Maximum Likelihood Estimate (MLE)

Choose 𝜄 that maximizes probability of observed data

෡ 𝜾MLE = argmax

𝜄

𝑄(𝐸𝑏𝑢𝑏|𝜄)

• Maximum a posteriori estimation (MAP)

Choose 𝜄 that is most probable given prior probability and data

෡ 𝜾MAP = argmax

𝜄

𝑄 𝜄 𝐸 = argmax

𝜄

𝑄 𝐸𝑏𝑢𝑏 𝜄 𝑄 𝜄 𝑄(𝐸𝑏𝑢𝑏)

Slide credit: Tom Mitchell

Naïve Bayes classifier

• Want to learn 𝑄 𝑍 𝑌1, ⋯ , 𝑌𝑜)
• But require 𝟑𝒐 parameters...
• How about applying Bayes rule?
• 𝑄 𝑍 𝑌1, ⋯ , 𝑌𝑜) =

𝑄(𝑌1,⋯,𝑌𝑜 𝑍 𝑄 𝑍 𝑄(𝑌1,⋯,𝑌𝑜)

∝ 𝑄(𝑌1, ⋯ , 𝑌𝑜 𝑍 𝑄 𝑍

• 𝑄(𝑌1, ⋯ , 𝑌𝑜 𝑍 : Need (𝟑𝒐−𝟐) × 𝟑 parameters
• 𝑄(𝑍):

Need 1 parameter

• Apply conditional independence assumption
• 𝑄 𝑌1, ⋯ , 𝑌𝑜 𝑍 = ς𝑘=1

𝑜

𝑄(𝑌

𝑘|𝑍): Need 𝐨 × 𝟑 parameters

Naïve Bayes classifier

• Bayes rule:

𝑄 𝑍 = 𝑧𝑙 𝑌1, ⋯ , 𝑌𝑜) = 𝑄(𝑍 = 𝑧𝑙)𝑄(𝑌1, ⋯ , 𝑌𝑜 𝑍 = 𝑧𝑙 σ𝑘 𝑄 𝑍 = 𝑧𝑘 𝑄 𝑌1, ⋯ , 𝑌𝑜 𝑍 = 𝑧𝑘

• Assume conditional independence among 𝑌𝑗’s:

𝑄 𝑍 = 𝑧𝑙 𝑌1, ⋯ , 𝑌𝑜) = 𝑄 𝑍 = 𝑧𝑙 Π𝑗𝑄 𝑌𝑗 𝑍 = 𝑧𝑙) σ𝑘 𝑄 𝑍 = 𝑧𝑘 Π𝑗𝑄 𝑌𝑗 𝑍 = 𝑧𝑘)

• Pick the most probable Y

෠ 𝑍 ← argmax

𝑧𝑙

𝑄 𝑍 = 𝑧𝑙 Π𝑗𝑄 𝑌𝑗 𝑍 = 𝑧𝑙)

Slide credit: Tom Mitchell

Example

• 𝑄 𝑍 𝑌1, 𝑌2 ∝ 𝑄 𝑍 𝑄 𝑌1, 𝑌2 𝑍 = 𝑄 𝑍 𝑄 𝑌1 𝑍 𝑄(𝑌2 𝑍
• Estimating parameters
• Test example: 𝑌1 = 1, 𝑌2 = 0
• 𝑍 = 1: 𝑄 𝑍 = 1 𝑄 𝑌1 = 1|𝑍 = 1 𝑄 𝑌2 = 0|𝑍 = 1 = 0.4 × 0.2 × 0.7 = 0.056
• 𝑍 = 0: 𝑄 𝑍 = 0 𝑄 𝑌1 = 1|𝑍 = 0 𝑄 𝑌2 = 0|𝑍 = 0 = 0.6 × 0.7 × 0.1 = 0.042

Bayes rule Conditional indep. 𝑄 𝑍 = 1 = 0.4 𝑄 𝑌1 = 1|𝑍 = 1 = 0.2 𝑄 𝑌1 = 1|𝑍 = 0 = 0.7 𝑄 𝑌2 = 1|𝑍 = 1 = 0.3 𝑄 𝑌2 = 1|𝑍 = 0 = 0.9 𝑄 𝑍 = 0 = 0.6 𝑄 𝑌1 = 0|𝑍 = 1 = 0.8 𝑄 𝑌1 = 0|𝑍 = 0 = 0.3 𝑄 𝑌2 = 0|𝑍 = 1 = 0.7 𝑄 𝑌2 = 0|𝑍 = 0 = 0.1

Naïve Bayes algorithm – discrete Xi

• For each value yk

Estimate 𝜌𝑙 = 𝑄(𝑍 = 𝑧𝑙) For each value xij of each attribute Xi

Estimate 𝜄𝑗𝑘𝑙 = 𝑄(𝑌𝑗 = 𝑦𝑗𝑘𝑙|𝑍 = 𝑧𝑙)

• Classify Xtest

෠ 𝑍 ← argmax

𝑧𝑙

𝑄 𝑍 = 𝑧𝑙 Π𝑗𝑄 𝑌𝑗

test 𝑍 = 𝑧𝑙)

෠ 𝑍 ← argmax

𝑧𝑙

𝜌𝑙 Π𝑗𝜄𝑗𝑘𝑙

Slide credit: Tom Mitchell

Estimating parameters: discrete 𝑍, 𝑌𝑗

• Maximum likelihood estimates (MLE)

ො 𝜌𝑙 = ෠ 𝑄 𝑍 = 𝑧𝑙 = #𝐸 𝑍 = 𝑧𝑙 𝐸 መ 𝜄𝑗𝑘𝑙 = ෠ 𝑄 𝑌𝑗 = 𝑦𝑗𝑘 𝑍 = 𝑧𝑙 = #𝐸 𝑌𝑗 = 𝑦𝑗𝑘 ^ 𝑍 = 𝑧𝑙 #𝐸{𝑍 = 𝑧𝑙}

Slide credit: Tom Mitchell

• F = 1 iff you live in Fox Ridge
• S = 1 iff you watched the superbowl last night
• D = 1 iff you drive to VT
• G = 1 iff you went to gym in the last month

𝑄 𝐺 = 1 = 𝑄 𝑇 = 1|𝐺 = 1 = 𝑄 𝑇 = 1|𝐺 = 0 = 𝑄 𝐸 = 1|𝐺 = 1 = 𝑄 𝐸 = 1|𝐺 = 0 = 𝑄 𝐻 = 1|𝐺 = 1 = 𝑄 𝐻 = 1|𝐺 = 0 = 𝑄 𝐺 = 0 = 𝑄 𝑇 = 0|𝐺 = 1 = 𝑄 𝑇 = 0|𝐺 = 0 = 𝑄 𝐸 = 0|𝐺 = 1 = 𝑄 𝐸 = 0|𝐺 = 0 = 𝑄 𝐻 = 0|𝐺 = 1 = 𝑄 𝐻 = 0|𝐺 = 0 = 𝑄 𝐺|𝑇, 𝐸, 𝐻 = 𝑄 𝐺 P S F P D F P(G|F)

Naïve Bayes: Subtlety #1

• Often the 𝑌𝑗 are not really conditionally independent
• Naïve Bayes often works pretty well anyway
• Often the right classification, even when not the right probability

[Domingos & Pazzani, 1996])

• What is the effect on estimated P(Y|X)?
• What if we have two copies: 𝑌𝑗 = 𝑌𝑙

𝑄 𝑍 = 𝑧𝑙 𝑌1, ⋯ , 𝑌𝑜) ∝ 𝑄 𝑍 = 𝑧𝑙 Π𝑗𝑄 𝑌𝑗 𝑍 = 𝑧𝑙)

Slide credit: Tom Mitchell

Naïve Bayes: Subtlety #2

MLE estimate for 𝑄 𝑌𝑗 𝑍 = 𝑧𝑙) might be zero. (for example, 𝑌𝑗 = birthdate. 𝑌𝑗 = Feb_4_1995)

• Why worry about just one parameter out of many?

𝑄 𝑍 = 𝑧𝑙 𝑌1, ⋯ , 𝑌𝑜) ∝ 𝑄 𝑍 = 𝑧𝑙 Π𝑗𝑄 𝑌𝑗 𝑍 = 𝑧𝑙)

• What can we do to address this?
• MAP estimates (adding “imaginary” examples)

Slide credit: Tom Mitchell

Estimating parameters: discrete 𝑍, 𝑌𝑗

• Maximum likelihood estimates (MLE)

ො 𝜌𝑙 = ෠ 𝑄 𝑍 = 𝑧𝑙 = #𝐸 𝑍 = 𝑧𝑙 𝐸 መ 𝜄𝑗𝑘𝑙 = ෠ 𝑄 𝑌𝑗 = 𝑦𝑗𝑘 𝑍 = 𝑧𝑙 = #𝐸 𝑌𝑗 = 𝑦𝑗𝑘, 𝑍 = 𝑧𝑙 #𝐸{𝑍 = 𝑧𝑙}

• MAP estimates (Dirichlet priors):

ො 𝜌𝑙 = ෠ 𝑄 𝑍 = 𝑧𝑙 = #𝐸 𝑍 = 𝑧𝑙 + (𝛾𝑙−1) 𝐸 + σ𝑛(𝛾𝑛−1) መ 𝜄𝑗𝑘𝑙 = ෠ 𝑄 𝑌𝑗 = 𝑦𝑗𝑘 𝑍 = 𝑧𝑙 = #𝐸 𝑌𝑗 = 𝑦𝑗𝑘, 𝑍 = 𝑧𝑙 + (𝛾𝑙 −1) #𝐸{𝑍 = 𝑧𝑙} + σ𝑛(𝛾𝑛−1)

Slide credit: Tom Mitchell

What if we have continuous Xi

• Gaussian Naïve Bayes (GNB): assume

𝑄 𝑌𝑗 = 𝑦 𝑍 = 𝑧𝑙 = 1 2𝜌𝜏𝑗𝑙 exp(− 𝑦 − 𝜈𝑗𝑙 2𝜏𝑗𝑙

2 2

)

• Additional assumption on 𝜏𝑗𝑙:
• Is independent of 𝑍 (𝜏𝑗)
• Is independent of 𝑌𝑗 (𝜏𝑙)
• Is independent of 𝑌i and 𝑍 (𝜏)

Slide credit: Tom Mitchell

Naïve Bayes algorithm – continuous Xi

• For each value yk

Estimate 𝜌𝑙 = 𝑄(𝑍 = 𝑧𝑙) For each attribute Xi estimate Class conditional mean 𝜈𝑗𝑙, variance 𝜏𝑗𝑙

• Classify Xtest

෠ 𝑍 ← argmax

𝑧𝑙

𝑄 𝑍 = 𝑧𝑙 Π𝑗𝑄 𝑌𝑗

test 𝑍 = 𝑧𝑙)

෠ 𝑍 ← argmax

𝑧𝑙

𝜌𝑙 Π𝑗 𝑂𝑝𝑠𝑛𝑏𝑚(𝑌𝑗

test, 𝜈𝑗𝑙, 𝜏𝑗𝑙)

Slide credit: Tom Mitchell

Things to remember

• Probability basics
• Conditional probability, joint probability, Bayes rule
• Estimating parameters from data
• Maximum likelihood (ML) maximize 𝑄(Data|𝜄)
• Maximum a posteriori estimation (MAP) maximize 𝑄(𝜄|Data)
• Naive Bayes

𝑄 𝑍 = 𝑧𝑙 𝑌1, ⋯ , 𝑌𝑜) ∝ 𝑄 𝑍 = 𝑧𝑙 Π𝑗𝑄 𝑌𝑗 𝑍 = 𝑧𝑙)

Logistic Regression

• Hypothesis representation
• Cost function
• Logistic regression with gradient descent
• Regularization
• Multi-class classification

Logistic Regression

• Hypothesis representation
• Cost function
• Logistic regression with gradient descent
• Regularization
• Multi-class classification

• Threshold classifier output ℎ𝜄 𝑦 at 0.5
• If ℎ𝜄 𝑦 ≥ 0.5, predict “𝑧 = 1”
• If ℎ𝜄 𝑦 < 0.5, predict “𝑧 = 0”

Malignant? 0 (No) 1 (Yes) Tumor Size ℎ𝜄 𝑦 = 𝜄⊤𝑦

Slide credit: Andrew Ng

Classification: 𝑧 = 1 or 𝑧 = 0 ℎ𝜄 𝑦 = 𝜄⊤𝑦 (from linear regression) can be > 1 or < 0 Logistic regression: 0 ≤ ℎ𝜄 𝑦 ≤ 1 Logistic regression is actually for classification

Slide credit: Andrew Ng

Hypothesis representation

• Want 0 ≤ ℎ𝜄 𝑦 ≤ 1
• ℎ𝜄 𝑦 = 𝑕 𝜄⊤𝑦 ,

where 𝑕 𝑨 =

1 1+𝑓−𝑨

• Sigmoid function
• Logistic function

ℎ𝜄 𝑦 = 1 1 + 𝑓−𝜄⊤𝑦

𝑨 𝑕(𝑨)

Slide credit: Andrew Ng

Interpretation of hypothesis output

• ℎ𝜄 𝑦 = estimated probability that 𝑧 = 1 on input 𝑦
• Example: If 𝑦 = 𝑦0

x1 = 1 tumorSize

• ℎ𝜄 𝑦 = 0.7
• Tell patient that 70% chance of tumor being malignant

Slide credit: Andrew Ng

Logistic regression

ℎ𝜄 𝑦 = 𝑕 𝜄⊤𝑦 𝑕 𝑨 = 1 1 + 𝑓−𝑨 Suppose predict “y = 1” if ℎ𝜄 𝑦 ≥ 0.5 𝑨 = 𝜄⊤𝑦 ≥ 0 predict “y = 0” if ℎ𝜄 𝑦 < 0.5 𝑨 = 𝜄⊤𝑦 < 0

𝑨 = 𝜄⊤𝑦 𝑕(𝑨)

Slide credit: Andrew Ng

Decision boundary

• ℎ𝜄 𝑦 = 𝑕(𝜄0 + 𝜄1𝑦1 + 𝜄2𝑦2)

E.g., 𝜄0 = −3, 𝜄1= 1, 𝜄2 = 1

• Predict “𝑧 = 1” if −3 + 𝑦1 + 𝑦2 ≥ 0

Tumor Size Age

Slide credit: Andrew Ng

• ℎ𝜄 𝑦 = 𝑕(𝜄0 + 𝜄1𝑦1 + 𝜄2𝑦2

+ 𝜄3𝑦1

2 + 𝜄4𝑦2 2)

E.g., 𝜄0 = −1, 𝜄1= 0, 𝜄2 = 0, 𝜄3 = 1, 𝜄4 = 1

• Predict “𝑧 = 1” if −1 + 𝑦1

2 + 𝑦2 2 ≥ 0

• ℎ𝜄 𝑦 = 𝑕(𝜄0 + 𝜄1𝑦1 + 𝜄2𝑦2 + 𝜄3𝑦1

2 +

𝜄4𝑦1

2𝑦2 + 𝜄5𝑦1 2𝑦2 2 + 𝜄6𝑦1 3𝑦2 + ⋯ )

Slide credit: Andrew Ng

Where does the form come from?

• Logistic regression hypothesis representation

ℎ𝜄 𝑦 = 1 1 + 𝑓−𝜄⊤𝑦 = 1 1 + 𝑓−(𝜄0+𝜄1𝑦1+𝜄2𝑦2+⋯+𝜄𝑜𝑦𝑜)

• Consider learning f: 𝑌 → 𝑍, where
• 𝑌 is a vector of real-valued features 𝑌1, ⋯ , 𝑌𝑜 ⊤
• 𝑍 is Boolean
• Assume all 𝑌𝑗 are conditionally independent given 𝑍
• Model 𝑄 𝑌𝑗 𝑍 = 𝑧𝑙 as Gaussian 𝑂 𝜈𝑗𝑙, 𝜏𝑗
• Model 𝑄 𝑍 as Bernoulli 𝜌

What is 𝑄 𝑍|𝑌1, 𝑌2, ⋯ , 𝑌𝑜 ?

Slide credit: Tom Mitchell

• 𝑄 𝑍 = 1|𝑌 =

𝑄 𝑍=1 𝑄(𝑌|𝑍=1) 𝑄 𝑍=1 𝑄(𝑌|𝑍=1) +𝑄(𝑍=0)𝑄(𝑌|𝑍=0)

=

1 1+𝑄 𝑍=0 𝑄(𝑌|𝑍=0)

𝑄 𝑍=1 𝑄(𝑌|𝑍=1)

=

1 1+exp(ln(𝑄 𝑍=0 𝑄(𝑌|𝑍=0)

𝑄 𝑍=1 𝑄(𝑌|𝑍=1) ))

=

1 1+exp(ln 1−𝜌

𝜌

+σ𝑗 𝑚𝑜

𝑄(𝑌𝑗|𝑍=0) 𝑄(𝑌𝑗|𝑍=1))

𝑄 𝑍 = 1|𝑌1, 𝑌2, ⋯ , 𝑌𝑜 = 1 1 + exp(𝜄0 + σ𝑗 𝜄𝑗 𝑌𝑗)

෍

𝑗

(𝜈𝑗0 − 𝜈𝑗1 𝜏𝑗

2

𝑌𝑗 + 𝜈𝑗1

2 − 𝜈𝑗0 2

2𝜏𝑗

2

)

Applying Bayes rule Divide by 𝑄 𝑍 = 1 𝑄(𝑌|𝑍 = 1) Apply exp(ln(⋅)) Plug in 𝑄(𝑌𝑗|𝑍)

𝑄 𝑦|𝑧𝑙 = 1 2𝜌𝜏𝑗 𝑓

− 𝑦−𝜈𝑗𝑙 2 2𝜏𝑗

2 Slide credit: Tom Mitchell

Logistic Regression

• Hypothesis representation
• Cost function
• Logistic regression with gradient descent
• Regularization
• Multi-class classification

Training set with 𝑛 examples { 𝑦 1 , 𝑧 1 , 𝑦 2 , 𝑧 2 , ⋯ , 𝑦 𝑛 , 𝑧 𝑛 𝑦 ∈ 𝑦0 𝑦1 ⋮ 𝑦𝑜 𝑦0 = 1, 𝑧 ∈ {0, 1} ℎ𝜄 𝑦 = 1 1 + 𝑓−𝜄⊤𝑦

How to choose parameters 𝜄?

Slide credit: Andrew Ng

Cost function for Linear Regression

𝐾 𝜄 = 1 2𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

2 = 1

𝑛 ෍

𝑗=1 𝑛

Cost(ℎ𝜄(𝑦 𝑗 ), 𝑧))

Cost(ℎ𝜄 𝑦 , 𝑧) = 1 2 ℎ𝜄 𝑦 − 𝑧 2

Slide credit: Andrew Ng

Cost function for Logistic Regression

Cost(ℎ𝜄 𝑦 , 𝑧) = ൝−log ℎ𝜄 𝑦 if 𝑧 = 1 −log 1 − ℎ𝜄 𝑦 if 𝑧 = 0

1

if 𝑧 = 1

ℎ𝜄 𝑦 1

if 𝑧 = 0

ℎ𝜄 𝑦

Slide credit: Andrew Ng

Logistic regression cost function

• Cost(ℎ𝜄 𝑦 , 𝑧) = ൝−log ℎ𝜄 𝑦

if 𝑧 = 1 −log 1 − ℎ𝜄 𝑦 if 𝑧 = 0

• Cost ℎ𝜄 𝑦 , 𝑧 = −𝑧 log h𝜄 x

− (1 − y) log 1 − ℎ𝜄 𝑦

• If 𝑧 = 1: Cost ℎ𝜄 𝑦 , 𝑧 = −log ℎ𝜄 𝑦
• If 𝑧 = 0: Cost ℎ𝜄 𝑦 , 𝑧 = −log 1 − ℎ𝜄 𝑦

Slide credit: Andrew Ng

Logistic regression

𝐾 𝜄 = 1 𝑛 ෍

𝑗=1 𝑛

Cost(ℎ𝜄(𝑦 𝑗 ), 𝑧(𝑗))) = −

1 𝑛 σ𝑗=1 𝑛 𝑧(𝑗) log ℎ𝜄 𝑦(𝑗)

+ (1 − 𝑧(𝑗)) log 1 − ℎ𝜄 𝑦(𝑗)

Prediction: given new 𝑦 Output ℎ𝜄 𝑦 =

1 1+𝑓−𝜄⊤𝑦

Learning: fit parameter 𝜄 min

𝜄 𝐾(𝜄)

Slide credit: Andrew Ng

Where does the cost come from?

• Training set with 𝑛 examples

𝑦 1 , 𝑧 1 , 𝑦 2 , 𝑧 2 , ⋯ , 𝑦 𝑛 , 𝑧 𝑛

• Maximum likelihood estimate for parameter 𝜄

𝜄MLE = argmax

𝜄

𝑄𝜄 𝑦 1 , 𝑧 1 , 𝑦 2 , 𝑧 2 , ⋯ , 𝑦 𝑛 , 𝑧 𝑛 = argmax

𝜄

ෑ

𝑗=1 𝑛

𝑄𝜄 𝑦 𝑗 , 𝑧 𝑗

• Maximum conditional likelihood estimate for parameter 𝜄

Slide credit: Tom Mitchell

• Goal: choose 𝜄 to maximize conditional likelihood of training data
• 𝑄𝜄 𝑍 = 1 𝑌 = 𝑦 = ℎ𝜄 𝑦 =

1 1+𝑓−𝜄⊤𝑦

• 𝑄𝜄 𝑍 = 0 𝑌 = 𝑦 = 1 − ℎ𝜄 𝑦 =

𝑓−𝜄⊤𝑦 1+𝑓−𝜄⊤𝑦

• Training data D =

𝑦 1 , 𝑧 1 , 𝑦 2 , 𝑧 2 , ⋯ , 𝑦 𝑛 , 𝑧 𝑛

• Data likelihood = ς𝑗=1

𝑛 𝑄𝜄

𝑦 𝑗 , 𝑧 𝑗

• Data conditional likelihood = ς𝑗=1

𝑛 𝑄𝜄 𝑧(𝑗)|𝑦 𝑗

𝜄MCLE = argmax

𝜄

ς𝑗=1

𝑛 𝑄𝜄 𝑧(𝑗)|𝑦 𝑗

Slide credit: Tom Mitchell

Expressing conditional log-likelihood

𝑀 𝜄 = log ෑ

𝑗=1 𝑛

𝑄𝜄 𝑧(𝑗)|𝑦 𝑗 = ෍

𝑗=1 𝑛

log 𝑄𝜄 𝑧(𝑗)|𝑦 𝑗 = ෍

𝑗=1 𝑛

𝑧(𝑗) log 𝑄𝜄 𝑧(𝑗) = 1|𝑦 𝑗 + 1 − 𝑧 𝑗 log 𝑄𝜄 𝑧(𝑗) = 0|𝑦 𝑗 = σ𝑗=1

𝑛 𝑧(𝑗) log (ℎ𝜄(𝑦(𝑗))) + 1 − 𝑧 𝑗

log(1 − ℎ𝜄(𝑦(𝑗)))

Cost(ℎ𝜄 𝑦 , 𝑧) = ൝−log ℎ𝜄 𝑦 if 𝑧 = 1 −log 1 − ℎ𝜄 𝑦 if 𝑧 = 0

Logistic Regression

• Hypothesis representation
• Cost function
• Logistic regression with gradient descent
• Regularization
• Multi-class classification

Gradient descent

𝐾 𝜄 = − 1 𝑛 ෍

𝑗=1 𝑛

𝑧(𝑗) log ℎ𝜄 𝑦(𝑗) + (1 − 𝑧(𝑗)) log 1 − ℎ𝜄 𝑦(𝑗)

Goal: min

𝜄 𝐾(𝜄)

Repeat { 𝜄

𝑘 ≔ 𝜄 𝑘 − 𝛽 𝜖

𝜖𝜄

𝑘

𝐾(𝜄) }

(Simultaneously update all 𝜄

𝑘)

𝜖 𝜖𝜄

𝑘

𝐾 𝜄 = 1 𝑛 ෍

𝑗=1 𝑛

(ℎ𝜄 𝑦 𝑗 − 𝑧(𝑗)) 𝑦𝑘

(𝑗)

Good news: Convex function! Bad news: No analytical solution

Slide credit: Andrew Ng

Gradient descent

𝐾 𝜄 = − 1 𝑛 ෍

𝑗=1 𝑛

𝑧(𝑗) log ℎ𝜄 𝑦(𝑗) + (1 − 𝑧(𝑗)) log 1 − ℎ𝜄 𝑦(𝑗)

Goal: min

𝜄 𝐾(𝜄)

Repeat { 𝜄

𝑘 ≔ 𝜄 𝑘 − 𝛽 1

𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧(𝑗) 𝑦𝑘

(𝑗)

}

(Simultaneously update all 𝜄

𝑘)

Slide credit: Andrew Ng

Gradient descent for Linear Regression

Repeat { 𝜄

𝑘 ≔ 𝜄 𝑘 − 𝛽 1

𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧(𝑗) 𝑦𝑘

(𝑗)

}

Gradient descent for Logistic Regression

Repeat { 𝜄

𝑘 ≔ 𝜄 𝑘 − 𝛽 1

𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧(𝑗) 𝑦𝑘

(𝑗)

}

ℎ𝜄 𝑦 = 𝜄⊤𝑦 ℎ𝜄 𝑦 = 1 1 + 𝑓−𝜄⊤𝑦

Slide credit: Andrew Ng

Logistic Regression

• Hypothesis representation
• Cost function
• Logistic regression with gradient descent
• Regularization
• Multi-class classification

How about MAP?

• Maximum conditional likelihood estimate (MCLE)
• Maximum conditional a posterior estimate (MCAP)

𝜄MCLE = argmax

𝜄

ς𝑗=1

𝑛 𝑄𝜄 𝑧(𝑗)|𝑦 𝑗

𝜄MCAP = argmax

𝜄

ς𝑗=1

𝑛 𝑄𝜄 𝑧(𝑗)|𝑦 𝑗

𝑄(𝜄)

Prior 𝑄(𝜄)

• Common choice of 𝑄(𝜄):
• Normal distribution, zero mean, identity covariance
• “Pushes” parameters towards zeros
• Corresponds to Regularization
• Helps avoid very large weights and overfitting

Slide credit: Tom Mitchell

MLE vs. MAP

• Maximum conditional likelihood estimate (MCLE)

𝜄

𝑘 ≔ 𝜄 𝑘 − 𝛽 1

𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧(𝑗) 𝑦𝑘

(𝑗)

• Maximum conditional a posterior estimate (MCAP)

𝜄

𝑘 ≔ 𝜄 𝑘 − 𝛽𝜇𝜄 𝑘 − 𝛽 1

𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧(𝑗) 𝑦𝑘

(𝑗)

Logistic Regression

• Hypothesis representation
• Cost function
• Logistic regression with gradient descent
• Regularization
• Multi-class classification

Multi-class classification

• Email foldering/taggning: Work, Friends, Family, Hobby
• Medical diagrams: Not ill, Cold, Flu
• Weather: Sunny, Cloudy, Rain, Snow

Slide credit: Andrew Ng

Binary classification

𝑦2 𝑦1

Multiclass classification

𝑦2 𝑦1

One-vs-all (one-vs-rest)

𝑦2 𝑦1

Class 1: Class 2: Class 3:

ℎ𝜄

𝑗 𝑦 = 𝑄 𝑧 = 𝑗 𝑦; 𝜄

(𝑗 = 1, 2, 3) 𝑦2 𝑦1 𝑦2 𝑦1 𝑦2 𝑦1

ℎ𝜄

1 𝑦

ℎ𝜄

2 𝑦

ℎ𝜄

3 𝑦

Slide credit: Andrew Ng

One-vs-all

• Train a logistic regression classifier ℎ𝜄

𝑗 𝑦 for

each class 𝑗 to predict the probability that 𝑧 = 𝑗

• Given a new input 𝑦, pick the class 𝑗 that

maximizes max

i

ℎ𝜄

𝑗 𝑦

Slide credit: Andrew Ng

Generative Approach

Ex: Naïve Bayes

Estimate 𝑄(𝑍) and 𝑄(𝑌|𝑍) Prediction

ො 𝑧 = argmax𝑧 𝑄 𝑍 = 𝑧 𝑄(𝑌 = 𝑦|𝑍 = 𝑧)

Discriminative Approach

Ex: Logistic regression

Estimate 𝑄(𝑍|𝑌) directly (Or a discriminant function: e.g., SVM) Prediction

ො 𝑧 = 𝑄(𝑍 = 𝑧|𝑌 = 𝑦)

Further readings

• Tom M. Mitchell

Generative and discriminative classifiers: Naïve Bayes and Logistic Regression http://www.cs.cmu.edu/~tom/mlbook/NBayesLogReg.pdf

• Andrew Ng, Michael Jordan

On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes http://papers.nips.cc/paper/2020-on-discriminative-vs-generative- classifiers-a-comparison-of-logistic-regression-and-naive-bayes.pdf

Things to remember

• Hypothesis representation
• Cost function
• Logistic regression with gradient descent
• Regularization
• Multi-class classification

ℎ𝜄 𝑦 = 1 1 + 𝑓−𝜄⊤𝑦 Cost(ℎ𝜄 𝑦 , 𝑧) = ൝−log ℎ𝜄 𝑦 if 𝑧 = 1 −log 1 − ℎ𝜄 𝑦 if 𝑧 = 0 𝜄

𝑘 ≔ 𝜄 𝑘 − 𝛽 1

𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧(𝑗) 𝑦𝑘

(𝑗)

𝜄

𝑘 ≔ 𝜄 𝑘 − 𝛽𝜇𝜄 𝑘 − 𝛽 1

𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧(𝑗) 𝑦𝑘

(𝑗)

max

i

ℎ𝜄

𝑗 𝑦

Coming up…

• Regularization
• Support Vector Machine