Probabilistic classification CE-717: Machine Learning Sharif - - PowerPoint PPT Presentation

Probabilistic classification

CE-717: Machine Learning

Sharif University of Technology

• M. Soleymani

Fall 2016

Topics

 Probabilistic approach

 Bayes decision theory  Generative models

 Gaussian Bayes classifier  Naïve Bayes

 Discriminative models

 Logistic regression

2

Classification problem: probabilistic view

3

 Each feature as a random variable  Class label also as a random variable  We observe the feature values for a random sample and

we intend to find its class label

 Evidence: feature vector 𝒚  Query: class label

Definitions

4

 Posterior probability: 𝑞 𝒟𝑙 𝒚  Likelihood or class conditional probability: 𝑞 𝒚|𝒟𝑙  Prior probability: 𝑞(𝒟𝑙)

𝑞(𝒚): pdf of feature vector 𝒚 (𝑞 𝒚 = 𝑙=1

𝐿

𝑞 𝒚 𝒟𝑙 𝑞(𝒟𝑙)) 𝑞(𝒚|𝒟𝑙): pdf of feature vector 𝒚 for samples of class 𝒟𝑙 𝑞(𝒟𝑙): probability of the label be 𝒟𝑙

Bayes decision rule

5

𝑞 𝑓𝑠𝑠𝑝𝑠 𝒚 = 𝑞(𝐷2|𝒚) if we decide 𝒟1 𝑄(𝐷1|𝒚) if we decide 𝒟2

 If we use Bayes decision rule:

𝑄 𝑓𝑠𝑠𝑝𝑠 𝒚 = min{𝑄 𝒟1 𝒚 , 𝑄(𝒟2|𝒚)}

𝐿 = 2

If 𝑄 𝒟1|𝒚 > 𝑄(𝒟2|𝒚) decide 𝒟1

• therwise decide 𝒟2

Using Bayes rule, for each 𝒚, 𝑄 𝑓𝑠𝑠𝑝𝑠 𝒚 is as small as possible and thus this rule minimizes the probability of error

Optimal classifier

6

 The optimal decision is the one that minimizes the

expected number of mistakes

 We show that Bayes classifier is an optimal classifier

Bayes decision rule Minimizing misclassification rate

7

 Decision regions: ℛ𝑙 = {𝒚|𝛽 𝒚 = 𝑙}

 All points in ℛ𝑙 are assigned to class 𝒟𝑙

Choose class with highest 𝑞 𝒟𝑙 𝒚 as 𝛽 𝒚 𝑞 𝑓𝑠𝑠𝑝𝑠 = 𝐹𝒚,𝑧 𝐽(𝛽(𝒚) ≠ 𝑧) = 𝑞 𝒚 ∈ ℛ1, 𝒟2 + 𝑞 𝒚 ∈ ℛ2, 𝒟1 =

ℛ1

𝑞 𝒚, 𝒟2 𝑒𝒚 +

ℛ2

𝑞 𝒚, 𝒟1 𝑒𝒚 =

ℛ1

𝑞 𝒟2|𝒚 𝑞 𝒚 𝑒𝒚 +

ℛ2

𝑞 𝒟1|𝒚 𝑞 𝒚 𝑒𝒚

𝐿 = 2

Bayes minimum error

8

 Bayes minimum error classifier:

min

𝛽(.) 𝐹𝒚,𝑧 𝐽(𝛽(𝒚) ≠ 𝑧)

 If we know the probabilities in advance then the above

• ptimization problem will be solved easily.

 𝛽 𝒚 = argmax

𝑧

𝑞(𝑧|𝒚)  In practice, we can estimate 𝑞(𝑧|𝒚) based on a set of

training samples 𝒠

Zero-one loss

Bayes theorem

9

 Bayes’ theorem

𝑞 𝒟𝑙 𝒚 =

𝑞 𝒚|𝒟𝑙 𝑞(𝒟𝑙) 𝑞(𝒚)

 Posterior probability: 𝑞 𝒟𝑙 𝒚  Likelihood or class conditional probability: 𝑞 𝒚|𝒟𝑙  Prior probability: 𝑞(𝒟𝑙)

Likelihood Prior Posterior 𝑞(𝒚): pdf of feature vector 𝒚 (𝑞 𝒚 = 𝑙=1

𝐿

𝑞 𝒚 𝒟𝑙 𝑞(𝒟𝑙)) 𝑞(𝒚|𝒟𝑙): pdf of feature vector 𝒚 for samples of class 𝒟𝑙 𝑞(𝒟𝑙): probability of the label be 𝒟𝑙

Bayes decision rule: example

10

 Bayes decision: Choose the class with highest 𝑞 𝒟𝑙 𝒚

𝑞(𝑦|𝒟1) 𝑞(𝑦|𝒟2) 𝑞 𝒟1 = 2 3 𝑞 𝒟2 = 1 3 𝑞(𝒟1|𝑦) 𝑞(𝒟2|𝑦) ℛ2 ℛ2 𝑞 𝒟𝑙 𝒚 = 𝑞 𝒚|𝒟𝑙 𝑞(𝒟𝑙) 𝑞(𝒚) 𝑞 𝒚 = 𝑞 𝒟1 𝑞 𝒚 𝒟1 + 𝑞 𝒟2 𝑞 𝒚 𝒟2

Bayesian decision rule

11

 If 𝑄 𝒟1|𝒚 > 𝑄(𝒟2|𝒚) decide 𝒟1  otherwise decide 𝒟2  If

𝑞 𝒚|𝒟1 𝑄(𝒟1) 𝑞(𝒚)

>

𝑞 𝒚|𝒟2 𝑄(𝒟2) 𝑞(𝑦)

decide 𝒟1

 otherwise decide 𝒟2  If 𝑞 𝒚|𝒟1 𝑄(𝒟1) > 𝑞 𝒚|𝒟2 𝑄(𝒟2) decide 𝒟1  otherwise decide 𝒟2

Equivalent Equivalent

Bayes decision rule: example

12

 Bayes decision: Choose the class with highest 𝑞 𝒟𝑙 𝒚

𝑞(𝑦|𝒟1) 𝑞(𝑦|𝒟2) 𝑞 𝒟1 = 2 3 𝑞 𝒟2 = 1 3 𝑞(𝒟1|𝑦) 𝑞(𝒟2|𝑦) ℛ2 ℛ2 ℛ2 ℛ2 2 × 𝑞(𝑦|𝒟1)

Bayes Classier

13

 Simple Bayes classifier: estimate posterior probability of

each class

 What should the decision criterion be?

 Choose class with highest 𝑞 𝒟𝑙 𝒚

 The optimal decision is the one that minimizes the

expected number of mistakes

Diabetes example

14

 white blood cell count

This example has been adopted from Sanja Fidler’s slides, University of Toronto, CSC411

Diabetes example

15

 Doctor has a prior 𝑞 𝑧 = 1 = 0.2

 Prior: In the absence of any observation, what do I know about

the probability of the classes?

 A patient comes in with white blood cell count 𝑦  Does the patient have diabetes 𝑞 𝑧 = 1|𝑦 ?

 given a new observation, we still need to compute the

posterior

Diabetes example

16

This example has been adopted from Sanja Fidler’s slides, University of Toronto, CSC411

𝑞 𝑦 𝑧 = 0 𝑞 𝑦 𝑧 = 1

Estimate probability densities from data

17

 If we assume Gaussian distributions for 𝑞(𝑦|𝒟1) and

𝑞(𝑦|𝒟2)

 Recall that for samples {𝑦 1 , … , 𝑦 𝑂 }, if we assume a

Gaussian distribution, the MLE estimates will be

Diabetes example

18

This example has been adopted from Sanja Fidler’s slides, University of Toronto, CSC411

𝑞 𝑦 𝑧 = 1 = 𝑂 𝜈1, 𝜏1

2

𝜈1 = 𝑜: 𝑧(𝑜)=1 𝑦(𝑜) 𝑜: 𝑧(𝑜)=1 1 = 𝑜: 𝑧(𝑜)=1 𝑦(𝑜) 𝑂1 𝜏1

2 = 𝑜: 𝑧(𝑜)=1 𝑦 𝑜 −𝜈1

2

𝑂1

𝑞 𝑦 𝑧 = 0 𝑞 𝑦 𝑧 = 1

Diabetes example

19

 Add a second observation: Plasma glucose value

This example has been adopted from Sanja Fidler’s slides, University of Toronto, CSC411

Generative approach for this example

20

 Multivariate Gaussian distributions for 𝑞(𝑦|𝒟𝑙):

𝑞 𝒚 𝑧 = 𝑙 = 1 2𝜌 𝑒/2 Σ 1/2 exp{− 1 2 𝒚 − 𝝂𝑙 𝑈𝜯𝑙

−1 𝒚 − 𝝂𝑙 }

𝑙 = 1,2

 Prior distribution 𝑞(𝑦|𝒟𝑙):

 𝑞 𝑧 = 1 = 𝜌,

𝑞 𝑧 = 0 = 1 − 𝜌

MLE for multivariate Gaussian

21

 For samples {𝑦 1 , … , 𝑦 𝑂 }, if we assume a multivariate

Gaussian distribution, the MLE estimates will be: 𝝂 = 𝑜=1

𝑂

𝒚(𝑜) 𝑂 𝜯 = 1 𝑂

𝑜=1 𝑂

𝒚(𝑜) − 𝝂 𝒚(𝑜) − 𝝂

𝑈

Generative approach: example

22

Maximum likelihood estimation (𝐸 = 𝒚 𝑜 , 𝑧 𝑜

𝑜=1 𝑂

):

 𝜌 =

𝑂1 𝑂

 𝝂1 =

𝑜=1

𝑂

𝑧(𝑜)𝒚(𝑜) 𝑂1

, 𝝂2 =

𝑜=1

𝑂

(1−𝑧(𝑜))𝒚(𝑜) 𝑂2

 𝜯1 =

1 𝑂1 𝑜=1 𝑂

𝑧(𝑜) 𝒚(𝑜) − 𝝂 𝒚(𝑜) − 𝝂

𝑈

 𝜯2 =

1 𝑂2 𝑜=1 𝑂

(1 − 𝑧 𝑜 ) 𝒚(𝑜) − 𝝂 𝒚(𝑜) − 𝝂

𝑈

𝑂1 =

𝑜=1 𝑂

𝑧(𝑜) 𝑂2 = 𝑂 − 𝑂1

Decision boundary for Gaussian Bayes classifier

23

𝑞 𝒟1 𝒚 = 𝑞(𝒟2|𝒚) ln 𝑞(𝒟1|𝒚) = ln 𝑞(𝒟2|𝒚) ln 𝑞(𝒚|𝒟1) + ln 𝑞(𝒟1) − ln 𝑞(𝒚) = ln 𝑞(𝒚|𝒟2) + ln 𝑞(𝒟2) − ln 𝑞(𝒚) ln 𝑞(𝒚|𝒟1) + ln 𝑞(𝒟1) = ln 𝑞(𝒚|𝒟2) + ln 𝑞(𝒟2) ln 𝑞(𝒚|𝒟𝑙) = − 𝑒 2 ln 2𝜌 − 1 2 ln 𝜯𝑙

−1 − 1

2 𝒚 − 𝝂𝑙 𝑈𝜯𝑙

−1 𝒚 − 𝝂𝑙 𝑞 𝒟𝑙 𝒚 = 𝑞 𝒚|𝒟𝑙 𝑞(𝒟𝑙)

𝑞(𝒚)

Decision boundary

24

𝑞(𝒚|𝐷1) 𝑞(𝒚|𝐷2) 𝑞(𝐷1|𝒚) 𝑞(𝐷1|𝒚)=𝑞(𝐷2|𝒚)

Shared covariance matrix

26

 When classes share a single covariance matrix 𝜯 = 𝜯1

= 𝜯2 𝑞 𝒚 𝐷𝑙 = 1 2𝜌 𝑒/2 Σ 1/2 exp{− 1 2 𝒚 − 𝝂𝑙 𝑈𝜯−1 𝒚 − 𝝂𝑙 }

𝑙 = 1,2

 𝑞 𝐷1 = 𝜌,

𝑞 𝐷2 = 1 − 𝜌

Likelihood

27

𝑜=1 𝑂

𝑞(𝒚 𝑜 , 𝑧(𝑜)|𝜌, 𝝂1, 𝝂2, 𝜯) =

𝑜=1 𝑂

𝑞(𝒚 𝑜 |𝑧 𝑜 , 𝝂1, 𝝂2, 𝜯)𝑞(𝑧 𝑜 |𝜌)

Shared covariance matrix

28

 Maximum likelihood estimation (𝐸 =

𝒚 𝑗 , 𝑧 𝑗

𝑗=1 𝑜

):

𝜌 = 𝑂1 𝑂 𝝂1 = 𝑜=1

𝑂

𝑧(𝑜)𝒚(𝑜) 𝑂1 𝝂2 = 𝑜=1

𝑂

(1 − 𝑧(𝑜))𝒚(𝑜) 𝑂2 𝜯 = 1 𝑂

𝑜∈𝐷1

𝒚(𝑜) − 𝝂1 𝒚(𝑜) − 𝝂1

𝑈 + 𝑜∈𝐷2

𝒚(𝑜) − 𝝂2 𝒚(𝑜) − 𝝂2

𝑈

Decision boundary when shared covariance matrix

29

ln 𝑞(𝒚|𝒟1) + ln 𝑞(𝒟1) = ln 𝑞(𝒚|𝒟2) + ln 𝑞(𝒟2) ln 𝑞(𝒚|𝒟𝑙) = − 𝑒 2 ln 2𝜌 − 1 2 ln 𝜯𝑙

−1 − 1

2 𝒚 − 𝝂𝑙 𝑈𝜯−1 𝒚 − 𝝂𝑙

Bayes decision rule Multi-class misclassification rate

31

 Multi-class problem: Probability of error of Bayesian

decision rule

 Simpler to compute the probability of correct decision

𝑄 𝑓𝑠𝑠𝑝𝑠 = 1 − 𝑄(𝑑𝑝𝑠𝑠𝑓𝑑𝑢)

ℛ𝑗: the subset of feature space assigned to the class 𝒟𝑗 using the classifier 𝑄 𝐷𝑝𝑠𝑠𝑓𝑑𝑢 =

𝑗=1 𝐿 ℛ𝑗

𝑞(𝒚, 𝒟𝑗) 𝑒𝒚 =

𝑗=1 𝐿 ℛ𝑗

𝑞 𝒟𝑗 𝒚 𝑞(𝒚) 𝑒𝒚

Bayes minimum error

32

 Bayes minimum error classifier:

min

𝛽(.) 𝐹𝒚,𝑧 𝐽(𝛽(𝒚) ≠ 𝑧)

𝛽 𝒚 = argmax

𝑧

𝑞(𝑧|𝒚)

Zero-one loss

Minimizing Bayes risk (expected loss)

33

for each 𝒚 minimize it that is called conditional risk

𝐹𝒚,𝑧 𝑀 𝛽 𝒚 , 𝑧 =

𝑘=1 𝐿

𝑀 𝛽 𝒚 , 𝒟

𝑘 𝑞 𝒚, 𝒟 𝑘 𝑒𝒚

= 𝑞 𝒚

𝑘=1 𝐿

𝑀 𝛽 𝒚 , 𝒟

𝑘 𝑞 𝒟 𝑘|𝒚 𝑒𝒚

 Bayes minimum loss (risk) decision rule:

𝛽(𝒚) 𝛽(𝒚) = argmin

𝑗=1,…,𝐿 𝑘=1 𝐿

𝑀𝑗𝑘𝑞 𝒟

𝑘|𝒚

The loss of assigning a sample to 𝒟𝑗 where the correct class is 𝒟

𝑘

Minimizing expected loss: special case (loss = misclassification rate)

 Problem definition for this special case:

 If action 𝛽 𝒚 = 𝑗 is taken and the true category is 𝒟𝑘, then the

decision is correct if 𝑗 = 𝑘 and otherwise it is incorrect.

 Zero-one loss function:

𝑀𝑗𝑘 = 1 − 𝜀𝑗𝑘 = 0 𝑗 = 𝑘 1 𝑝. 𝑥.

𝛽 𝒚 = argmin

𝑗=1,…,𝐿 𝑘=1 𝐿

𝑀𝑗𝑘𝑞 𝒟

𝑘|𝒚

= argmin

𝑗=1,…,𝐿

0 × 𝑞 𝒟𝑗|𝒚 +

𝑘≠𝑗

𝑞 𝒟𝑘|𝒚

= argmin

𝑗=1,…,𝐿 1 − 𝑞 𝒟𝑗|𝒚 = argmax 𝑗=1,…,𝐿 𝑞 𝒟𝑗|𝒚

35

Probabilistic discriminant functions

36

 Discriminant functions: A popular way of representing

a classifier

 A discriminant function 𝑔

𝑗 𝒚 for each class 𝒟𝑗 (𝑗 = 1, … , 𝐿):

 𝒚 is assigned to class 𝒟𝑗 if:

𝑔𝑗(𝒚) > 𝑔𝑘(𝒚) 𝑘  𝑗  Representing

Bayesian classifier using discriminant functions:

 Classifier minimizing error rate: 𝑔𝑗 𝒚 = 𝑄(𝒟𝑗|𝒚)  Classifier minimizing risk: 𝑔𝑗 𝒚 = − 𝑘=1

𝐿

𝑀𝑗𝑘𝑞 𝒟

𝑘|𝒚

Naïve Bayes classifier

37

 Generative methods

 High number of parameters

 Assumption: Conditional independence

𝑞 𝒚 𝐷𝑙 = 𝑞 𝑦1 𝐷𝑙 × 𝑞 𝑦2 𝐷𝑙 × ⋯ × 𝑞 𝑦𝑒 𝐷𝑙

Naïve Bayes classifier

38

 In the decision phase, it finds the label of 𝒚 according to:

argmax

𝑙=1,…,𝐿

𝑞 𝐷𝑙 𝒚 argmax

𝑙=1,…,𝐿

𝑞(𝐷𝑙)

𝑗=1 𝑜

𝑞(𝑦𝑗|𝐷𝑙)

𝑞 𝒚 𝐷𝑙 = 𝑞 𝑦1 𝐷𝑙 × 𝑞 𝑦2 𝐷𝑙 × ⋯ × 𝑞 𝑦𝑒 𝐷𝑙 𝑞 𝐷𝑙 𝒚 ∝ 𝑞(𝐷𝑙)

𝑗=1 𝑜

𝑞(𝑦𝑗|𝐷𝑙)

Naïve Bayes classifier

39

 Finds 𝑒 univariate distributions 𝑞 𝑦1 𝐷𝑙 , ⋯ , 𝑞 𝑦𝑒 𝐷𝑙

instead

• f finding one multi-variate distribution 𝑞 𝒚 𝐷𝑙

 Example 1: For Gaussian class-conditional density 𝑞 𝒚 𝐷𝑙 , it finds 𝑒 + 𝑒 (mean

and sigma parameters on different dimensions) instead of 𝑒 + 𝑒(𝑒+1)

2

parameters

 Example 2: For Bernoulli class-conditional density 𝑞 𝒚 𝐷𝑙 , it finds 𝑒 (mean

parameters on different dimensions) instead of 2d − 1 parameters  It

first estimates the class conditional densities 𝑞 𝑦1 𝐷𝑙 , ⋯ , 𝑞 𝑦𝑒 𝐷𝑙 and the prior probability 𝑞(𝐷𝑙) for each class (𝑙 = 1, … , 𝐿) based on the training set.

Naïve Bayes: discrete example

40

 𝑞 𝐼 = 𝑍𝑓𝑡 = 0.3  𝑞 𝐸 = 𝑍𝑓𝑡 𝐼 = 𝑍𝑓𝑡 =

1 3

 𝑞 𝑇 = 𝑍𝑓𝑡 𝐼 = 𝑍𝑓𝑡 =

2 3

 𝑞 𝐸 = 𝑍𝑓𝑡 𝐼 = 𝑂𝑝 =

2 7

 𝑞 𝑇 = 𝑍𝑓𝑡 𝐼 = 𝑂𝑝 =

2 7

 Decision on 𝒚 = [𝑍𝑓𝑡, 𝑍𝑓𝑡] (a person that has diabetes and also smokes):



𝑞 𝐼 = 𝑍𝑓𝑡 𝒚 ∝ 𝑞 𝐼 = 𝑍𝑓𝑡 𝑞 𝐸 = 𝑧𝑓𝑡 𝐼 = 𝑍𝑓𝑡 𝑞 𝑇 = 𝑧𝑓𝑡 𝐼 = 𝑍𝑓𝑡 = 0.066



𝑞 𝐼 = 𝑂𝑝 𝒚 ∝ 𝑞 𝐼 = 𝑂𝑝 𝑞 𝐸 = 𝑧𝑓𝑡 𝐼 = 𝑂𝑝 𝑞 𝑇 = 𝑧𝑓𝑡 𝐼 = 𝑂𝑝 = 0.057



Thus decide 𝐼 = 𝑧𝑓𝑡

Diabetes (D) Smoke (S) Heart Disease (H) Y N Y Y N N N Y N N Y N N N N N Y Y N N N N Y Y N N N Y N N

Probabilistic classifiers

41

 How can we find the probabilities required in the Bayes

decision rule?

 Probabilistic classification approaches can be divided in

two main categories:

 Generative

 Estimate pdf 𝑞(𝒚, 𝒟𝑙) for each class 𝒟𝑙 and then use it to find

𝑞(𝒟𝑙|𝒚)

 or alternatively estimate both pdf 𝑞(𝒚|𝒟𝑙) and 𝑞 𝒟𝑙 to find 𝑞(𝒟𝑙|𝒚)

 Discriminative

 Directly estimate 𝑞(𝒟𝑙|𝒚) for each class 𝒟𝑙

Generative approach

42

 Inference stage

 Determine class conditional densities 𝑞(𝒚|𝒟𝑙)

and priors 𝑞(𝒟𝑙)

 Use the Bayes theorem to find 𝑞(𝒟𝑙|𝒚)

 Decision stage: After learning the model (inference stage),

make optimal class assignment for new input

 if 𝑞 𝒟𝑗 𝒚 > 𝑞 𝒟

𝑘 𝒚

∀𝑘 ≠ 𝑗 then decide 𝒟𝑗

Discriminative vs. generative approach

43

[Bishop]

Class conditional densities vs. posterior

44

𝑞 𝒚 𝐷1 𝑞 𝒚 𝐷2 𝑞 𝐷1 𝒚 𝑞 𝒟1 𝒚 = 𝜏(𝒙𝑈𝒚 + 𝑥0) 𝒙 = 𝜯−1 𝝂1 − 𝝂2 𝑥0 = − 1 2 𝝂1

𝑈𝜯−1𝝂1 + 1

2 𝝂2

𝑈𝜯−1𝝂2 + ln 𝑞(𝒟1)

𝑞(𝒟2) [Bishop]

Discriminative approach

45

 Inference stage

 Determine the posterior class probabilities 𝑄(𝒟𝑙|𝒚) directly

 Decision stage: After learning the model (inference stage),

make optimal class assignment for new input

 if 𝑄 𝒟𝑗 𝒚 > 𝑄 𝒟

𝑘 𝒚

∀𝑘 ≠ 𝑗 then decide 𝒟𝑗

Posterior probabilities

46

 Two-class: 𝑞 𝒟𝑙 𝒚 can be written as a logistic sigmoid for

a wide choice of 𝑞 𝒚 𝒟𝑙 distributions 𝑞 𝒟1 𝒚 = 𝜏 𝑏(𝒚) = 1 1 + exp(−𝑏(𝒚))

 Multi-class: 𝑞 𝒟𝑙 𝒚

can be written as a soft-max for a wide choice of 𝑞 𝒚 𝒟𝑙 𝑞 𝒟𝑙 𝒚 = exp(𝑏𝑙(𝒚)) 𝑘=1

𝐿

exp(𝑏𝑘(𝒚))

Discriminative approach: logistic regression

47

 More general than discriminant functions:

 𝑔 𝒚; 𝒙 predicts posterior probabilities 𝑄 𝑧 = 1 𝒚

𝑔 𝒚; 𝒙 = 𝜏(𝒙𝑈𝒚)

𝜏 . is an activation function

 Sigmoid (logistic) function

 Activation function

𝜏 𝑨 = 1 1 + 𝑓−𝑨

𝐿 = 2 𝒚 = 1, 𝑦1, … , 𝑦𝑒 𝒙 = 𝑥0, 𝑥1, … , 𝑥𝑒

Logistic regression

48

 𝑔 𝒚; 𝒙 : probability that 𝑧 = 1 given 𝒚 (parameterized by 𝒙)

𝑄 𝑧 = 1 𝒚, 𝒙 = 𝑔 𝒚; 𝒙 𝑄 𝑧 = 0 𝒚, 𝒙 = 1 − 𝑔 𝒚; 𝒙

 Example: Cancer (Malignant, Benign)

 𝑔 𝒚; 𝒙 = 0.7  70% chance of tumor being malignant

𝐿 = 2 𝑧 ∈ {0,1}

𝑔 𝒚; 𝒙 = 𝜏(𝒙𝑈𝒚) 0 ≤ 𝑔 𝒚; 𝒙 ≤ 1 estimated probability of 𝑧 = 1 on input 𝑦

Logistic regression: Decision surface

49

 Decision surface 𝑔 𝒚; 𝒙 = constant  𝑔 𝒚; 𝒙 = 𝜏 𝒙𝑈𝒚 =

1 1+𝑓−(𝒙𝑈𝒚) = 0.5

 Decision surfaces are linear functions of 𝒚

 if 𝑔 𝒚; 𝒙 ≥ 0.5 then 𝑧 = 1  else 𝑧 = 0

Equivalent to

 if 𝒙𝑈𝒚 + 𝑥0 ≥ 0 then 𝑧 = 1  else 𝑧 = 0

Logistic regression: ML estimation

50

 Maximum (conditional) log likelihood:

𝒙 = argmax

𝒙

log

𝑗=1 𝑜

𝑞 𝑧(𝑗) 𝒙, 𝒚(𝑗) 𝑞 𝑧(𝑗) 𝒙, 𝒚(𝑗) = 𝑔 𝒚(𝑗); 𝒙

𝑧(𝑗)

1 − 𝑔 𝒚(𝑗); 𝒙

(1−𝑧(𝑗))

log 𝑞 𝒛 𝒀, 𝒙 =

𝑗=1 𝑜

𝑧(𝑗)log 𝑔 𝒚(𝑗); 𝒙 + (1 − 𝑧(𝑗))log 1 − 𝑔 𝒚(𝑗); 𝒙

Logistic regression: cost function

51

𝒙 = argmin

𝒙

𝐾(𝒙) 𝐾 𝒙 = −

𝑗=1 𝑜

log 𝑞 𝑧(𝑗) 𝒙, 𝒚(𝒋) =

𝑗=1 𝑜

−𝑧(𝑗)log 𝑔 𝒚(𝑗); 𝒙 − (1 − 𝑧(𝑗))log 1 − 𝑔 𝒚(𝑗); 𝒙

 No closed form solution for

𝛂𝒙 𝐾(𝒙) = 0

 However 𝐾(𝒙) is convex.

Logistic regression: Gradient descent

52

𝒙𝑢+1 = 𝒙𝑢 − 𝜃𝛼

𝒙𝐾(𝒙𝑢)

𝛼

𝒙𝐾 𝒙 = 𝑗=1 𝑜

𝑔 𝒚 𝑗 ; 𝒙 − 𝑧 𝑗 𝒚 𝑗

 Is it similar to gradient of SSE for linear regression?

𝛼

𝒙𝐾 𝒙 = 𝑗=1 𝑜

𝒙𝑈𝒚 𝑗 − 𝑧 𝑗 𝒚 𝑗

Logistic regression: loss function

53

Loss 𝑧, 𝑔 𝒚; 𝒙 = −𝑧 × log 𝑔 𝒚; 𝒙 − (1 − 𝑧) × log(1 − 𝑔 𝒚; 𝒙 ) Loss 𝑧, 𝑔 𝒚; 𝒙 = −log(𝑔(𝒚; 𝒙)) if 𝑧 = 1 −log(1 − 𝑔 𝒚; 𝒙 ) if 𝑧 = 0 How is it related to zero-one loss? Loss 𝑧, 𝑧 = 1 𝑧 ≠ 𝑧 𝑧 = 𝑧

𝑔 𝒚; 𝒙 = 1 1 + 𝑓𝑦𝑞(−𝒙𝑈𝒚) Since 𝑧 = 1 or 𝑧 = 0 ⇒

Logistic regression: cost function (summary)

54

 Logistic Regression (LR) has a more proper cost function for

classification than SSE and Perceptron

 Why is the cost function of LR also more suitable than?

𝐾 𝒙 = 1 𝑜

𝑗=1 𝑜

𝑧 𝑗 − 𝑔 𝒚 𝑗 ; 𝒙

2

 where 𝑔 𝒚; 𝒙 = 𝜏(𝒙𝑈𝒚)

 The conditional distribution 𝑞 𝑧|𝒚, 𝒙

in the classification problem is not Gaussian (it is Bernoulli)

 The cost function of LR is also convex

Multi-class logistic regression

55

 For each class 𝑙, 𝑔

𝑙 𝒚; 𝑿 predicts the probability of 𝑧 = 𝑙

 i.e., 𝑄(𝑧 = 𝑙|𝒚, 𝑿)

 On a new input 𝒚, to make a prediction, pick the class that

maximizes 𝑔

𝑙 𝒚; 𝑿 :

𝛽 𝒚 = argmax

𝑙=1,…,𝐿

𝑔

𝑙 𝒚 if 𝑔

𝑙 𝒚 > 𝑔 𝑘 𝒚

∀𝑘 ≠ 𝑙 then decide 𝐷𝑙

Multi-class logistic regression

56

𝐿 > 2 𝑧 ∈ {1,2, … , 𝐿}

𝑔

𝑙 𝒚; 𝑿 = 𝑞 𝑧 = 𝑙 𝒚 =

exp(𝒙𝑙

𝑈𝒚 )

𝑘=1

𝐿

exp(𝒙𝑘

𝑈𝒚 )

 Normalized exponential (aka softmax)

 If 𝒙𝑙

𝑈𝒚 ≫ 𝒙𝑘 𝑈𝒚 for all 𝑘 ≠ 𝑙 then 𝑞(𝐷𝑙|𝒚) ≃ 1, 𝑞(𝐷 𝑘|𝒚) ≃ 0

𝑞 𝐷𝑙 𝒚 = 𝑞 𝒚 𝐷𝑙 𝑞(𝐷𝑙) 𝑘=1

𝐿

𝑞 𝒚 𝐷

𝑘 𝑞(𝐷 𝑘)

Logistic regression: multi-class

57

𝑿 = argmin

𝑿

𝐾(𝑿) 𝐾 𝑿 = − log

𝑗=1 𝑜

𝑞 𝒛 𝑗 𝒚 𝑗 , 𝑿 = − log

𝑗=1 𝑜 𝑙=1 𝐿

𝑔

𝑙 𝒚 𝑗 ; 𝑿 𝑧𝑙

𝑗

= −

𝑗=1 𝑜 𝑙=1 𝐿

𝑧𝑙

𝑗 log 𝑔 𝑙 𝒚(𝑗); 𝑿 𝑿 = 𝒙1 ⋯ 𝒙𝐿 𝒁 =

𝒛(1) ⋮ 𝒛(𝑜) = 𝑧1

(1)

⋯ 𝑧𝐿

(1)

⋮ ⋱ ⋮ 𝑧1

(𝑜)

⋯ 𝑧𝐿

(𝑜)

𝒛 is a vector of length 𝐿 (1-of-K coding) e.g., 𝒛 = 0,0,1,0 𝑈 when the target class is 𝐷3

Logistic regression: multi-class

58

𝒙𝑘

𝑢+1 = 𝒙𝑘 𝑢 − 𝜃𝛼𝑿𝐾(𝑿𝑢)

𝛼

𝒙𝑘𝐾 𝑿 = 𝑗=1 𝑜

𝑔

𝑘 𝒚 𝑗 ; 𝑿 − 𝑧𝑘 𝑗

𝒚 𝑗

Logistic Regression (LR): summary

59

 LR is a linear classifier  LR

• ptimization

problem is

• btained

by maximum likelihood

 when

assuming Bernoulli distribution for conditional probabilities whose mean is

1 1+𝑓−(𝒙𝑈𝒚)

 No closed-form solution for its optimization problem

 But convex cost function and global optimum can be found by

gradient ascent

Discriminative vs. generative: number of parameters

60

 𝑒-dimensional feature space  Logistic regression: 𝑒 + 1 parameters

 𝒙 = (𝑥0, 𝑥1 , . . , 𝑥𝑒)

 Generative approach:

 Gaussian class-conditionals with shared covariance matrix

 2𝑒 parameters for means  𝑒(𝑒 + 1)/2 parameters for shared covariance matrix  one parameter for class prior 𝑞(𝐷1).

 But LR is more robust, less sensitive to incorrect modeling

assumptions

Summary of alternatives

61

 Generative

 Most demanding, because it finds the joint distribution 𝑞(𝒚, 𝒟𝑙)  Usually needs a large training set to find 𝑞(𝒚|𝒟𝑙)  Can find 𝑞(𝒚) ⇒ Outlier or novelty detection

 Discriminative

 Specifies what is really needed (i.e., 𝑞(𝒟𝑙|𝒚))  More computationally efficient

Resources

62

 C. Bishop, “Pattern Recognition and Machine Learning”,

Chapter 4.2-4.3.