Lecture 2: Model-based classification Felix Held, Mathematical - - PowerPoint PPT Presentation

Lecture 2: Model-based classification

Felix Held, Mathematical Sciences

MSA220/MVE440 Statistical Learning for Big Data 28th March 2019

Reprise: Statistical Learning (I)

Regression

▶ Theoretically best regression function for squared error

loss ˆ 𝑔(𝐲) = 𝔽𝑞(𝑧|𝐲)[𝑧]

▶ Approximate (1) or make model-assumptions (2)

• 1. k-nearest neighbour regression

𝔽𝑞(𝑧|𝐲)[𝑧] ≈ 1 𝑙 ∑

𝐲𝑗𝑚∈𝑂𝑙(𝐲)

𝑧𝑗𝑚

• 2. linear regression (viewpoint: generalized linear models

(GLM)) 𝔽𝑞(𝑧|𝐲)[𝑧] ≈ 𝐲𝑈𝜸

1/25

Reprise: Statistical Learning (II)

Classification

▶ Theoretically best classification rule for 0-1 loss and 𝐿

possible classes ̂ 𝑑(𝐲) = arg max

1≤𝑗≤𝐿

𝑞(𝑗|𝐲)

▶ Approximate (1) or make model-assumptions (2)

• 1. k-nearest neighbour classification

𝑞(𝑗|𝐲) ≈ 1 𝑙 ∑

𝐲𝑚∈𝑂𝑙(𝐲)

1(𝑗𝑚 = 𝑗)

• 2. Instead of approximating 𝑞(𝑗|𝐲) from data, can we make

sensible model assumptions instead?

2/25

Amendment: kNN methods

There are two choices to make when implementing a kNN method

• 1. The metric to determine a neighbourhood

▶ e.g. Euclidean/ℓ2 norm, Manhattan/ℓ1 norm, max norm, …

• 2. The number of neighbours, i.e. 𝑙

The choice of metric changes the underlying local model of the method while 𝑙 is a tuning parameter.

3/25

Model-based classification

Classification as regression

▶ Consider a two-class problem, with 𝑗𝑚 = 0 or 𝑗𝑚 = 1 ▶ Instead of 0-1 loss, use square error loss, i.e.

𝔽𝑞(𝑗|𝐲)[𝑗] = 0 ⋅ 𝑞(0|𝐲) + 1 ⋅ 𝑞(1|𝐲) = 𝑞(1|𝐲) Note that 𝑗 has a discrete distribution.

▶ Linear regression model assumption

𝑞(1|𝐲) = 𝔽𝑞(𝑗|𝐲)[𝑗] ≈ 𝐲𝑈𝜸

▶ Since we are approximating 𝑞(1|𝐲) and

𝑞(0|𝐲) = 1 − 𝑞(1|𝐲) ≈ 1 − 𝐲𝑈𝜸, we indirectly specified a model approximation for Bayes’ rule as well 𝑑(𝐲) = { 𝐲𝑈𝜸 ≤

1 2

1

• therwise

Note that 𝐲𝑈𝜸 =

1 2 defines the decision boundary 4/25

0-1 regression

• ●
• ●
• ●
• ●
• 0.0

0.5 1.0 5 6 7

Sepal Length Coding

• ●
• 2.0

2.5 3.0 3.5 4.0 4.5 5 6 7

Sepal Length Sepal Width Species

• setosa

versicolor

The solid black lines show the decision boundary.

5/25

0-1 regressions and outliers

• −5

5 5 10

x1 x2 Case

No outlier With Outlier

6/25

Dummy encoding for categorical variables

In regression, when a predictor 𝑦 is categorical, i.e. takes one

• f 𝐿 values, it is common to use a dummy encoding.

Example: 𝑦 = 1 → 𝑨 = (1, 0, 0) 𝑦 = 2 → 𝑨 = (0, 1, 0) 𝑦 = 3 → 𝑨 = (0, 0, 1) Idea Turn a classification problem into a regression problem by representing the class outcomes 𝑗𝑚 in the training data (𝑗𝑚, 𝐲𝑚) as vectors in dummy encoding.

7/25

Multiple classes

▶ This creates a sequence of 0-1 regressions (see

blackboard). If there are 𝐿 classes then 𝑨(1)

𝑚

∶= 1(𝑗𝑚 = 1) → 𝑞(𝑨(1) = 1|𝐲) ≈ 𝐲𝑈𝜸(1) ⋮ 𝑨(𝐿)

𝑚

∶= 1(𝑗𝑚 = 𝐿) → 𝑞(𝑨(𝐿) = 1|𝐲) ≈ 𝐲𝑈𝜸(𝐿)

▶ Note that

𝑞(𝑗|𝐲) = 𝑞(𝑨(𝑗) = 1|𝐲) ≈ 𝐲𝑈𝜸(𝑗)

▶ Classification rule

𝑑(𝑦) = arg max

1≤𝑗≤𝐿

𝑞(𝑗|𝐲) ≈ arg max

1≤𝑗≤𝐿

𝐲𝑈𝜸(𝑗) Decision boundaries are defined by 𝑑(𝑦) = 𝐲𝑈𝛾(𝑗) = 𝐲𝑈𝛾(𝑘) for 𝑗 ≠ 𝑘

8/25

Multiple 0-1 regressions

• ●
• 1

Coding

• ●
• 1

5 10

Predictor Coding Class

• 1

2 3

9/25

Problems with 0-1 regression

Observations:

• 1. 𝐲𝑈𝜸 is unbounded but models a probability 𝑞(𝑗|𝐲) ∈ [0, 1]
• 2. Only values of 𝐲𝑈𝜸 around 0.5 (for binary classification) or

close to the maximal value (for multiple classes) are really of interest.

• 3. Sensitive to points far away from the boundary (outliers)
• 4. Masking: Classes can get buried among other classes

(adding polynomial predictors can sometimes help, but this is arbitrary and data dependent) Inspiration from GLM Can we transform 𝐲𝑈𝜸 such that the transformed values are in [0, 1], are similar to the original values when close to 0.5 and insensitive outliers far away from the boundary?

10/25

Logistic function and Normal Distribution CDF

0.00 0.25 0.50 0.75 1.00 −4 −2 2 4

x y Type

Logistic Function Standard Normal CDF

Logistic (sigmoid) function 𝜏(𝑦) = exp(𝑦) 1 + exp(𝑦) Standard Normal CDF Φ(𝑦) = ∫

𝑦 −∞

1 √2𝜌 exp (−𝑨2 2 ) d𝑨

11/25

Logistic and probit regression

▶ We arrive at logistic regression when assuming

𝑞(1|𝐲) = 𝔽𝑞(𝑗|𝐲)[𝑗] = 𝜏−1 (𝐲𝑈𝜸)

• r probit regression when assuming

𝑞(1|𝐲) = 𝔽𝑞(𝑗|𝐲)[𝑗] = Φ−1 (𝐲𝑈𝜸)

▶ Parameters can be estimated by iteratively reweighted

least squares (Details in ESL Ch. 4.4.1)

▶ A warning: Problematic situation in two-class case

(occurs seldom in practice)

▶ Assume two classes can be separated perfectly in one or

more predictors

▶ Logistic regression tries to fit a step-like function, which

forces the intercept to −∞ and the corresponding predictor coefficient to +∞.

12/25

Logistic regression and outliers

• −5

5 5 10

x1 x2 Case

0−1: no outlier 0−1: with outlier Logistic: with outlier

13/25

Multi-class logistic regression

▶ In case of 𝐿 > 2 classes, using dummy encoding for the

• utcome leads again to a series of regression problems.

▶ Requirement: Probabilities should be modelled, i.e. in

𝑞(𝑗|𝐲) ∈ [0, 1] for each class and ∑𝑗 𝑞(𝑗|𝐲) = 1

▶ Softmax function: 𝝉 ∶ ℝ𝐿 ↦ [0, 1]𝐿

𝝉

𝑘(𝐴) =

𝑓𝑨𝑘 ∑

𝐿 𝑚=1 𝑓𝑨𝑚

⇔ 𝝉

𝑘(𝐴) =

𝑓(𝑨𝑘−𝑨𝐿) 1 + ∑

𝐿−1 𝑚=1 𝑓(𝑨𝑚−𝑨𝐿)

▶ Model now:

𝑞(𝑗|𝐲) = 𝑓𝐲𝑈𝜸(𝑗) ∑

𝐿 𝑚=1 𝑓𝐲𝑈𝜸(𝑗)

• r

𝑞(𝑗|𝐲) = 𝑓𝐲𝑈(𝜸(𝑚)−𝜸(𝐿)) 1 + ∑

𝐿−1 𝑚=1 𝑓𝐲𝑈(𝜸(𝑚)−𝜸(𝐿))

▶ This method has many names: softmax regression,

multinomial logistic regression, maximum entropy classifier, …

14/25

Multi-class logistic regression: An example

0.0 0.5 1.0

Probability

• ●
• ●
• ●
• ●
• setosa

versicolor virginica 5 6 7 8

Sepal Length Species

15/25

Classification with focus on the feature/predictor space

Motivation for a different viewpoint: Nearest centroids

• ●
• 2

3 4 4 5 6 7 8

Sepal Length Sepal Width Species

• setosa

versicolor virginica

Determine mean predictor vector per class ˆ 𝝂𝑗 = 1 𝑜𝑗 ∑

𝑗𝑚=𝑗

𝐲𝑚 where 𝑜𝑗 =

𝑜

∑

𝑚=1

1(𝑗𝑚 = 𝑗) and classify points to the class who’s mean is closest.

16/25

A change of scenery

Summary

▶ Classification can be approached through regression and

approximation of 𝔽𝑞(𝑗|𝐲)[𝑗]

▶ Indirectly we approximated 𝑞(𝑗|𝐲) and were able to use

Bayes’ rule Observation: Good predictors group by class in feature space Change of focus: Let’s model the density of 𝐲 conditionally on 𝑗 instead! How? Bayes’ law

17/25

The setting of Discriminant Analysis

Apply Bayes’ law 𝑞(𝑗|𝐲) = 𝑞(𝐲|𝑗)𝑞(𝑗) ∑

𝐿 𝑘=1 𝑞(𝐲|𝑘)𝑞(𝑘)

Instead of specifying 𝑞(𝑗|𝐲) we can specify 𝑞(𝐲|𝑗) and 𝑞(𝑗) The main assumption of Discriminant Analysis (DA) is 𝑞(𝐲|𝑗) ∼ 𝑂(𝝂𝑗, 𝚻𝑗) where 𝝂𝑗 ∈ ℝ𝑞 is the mean vector for class 𝑗 and 𝚻𝑗 ∈ ℝ𝑞×𝑞 the corresponding covariance matrix.

18/25

Finding the parameters of DA

▶ Notation: Write 𝑞(𝑗) = 𝜌𝑗 and consider them as unknown

parameters

▶ Given data (𝑗𝑚, 𝐲𝑚) the likelihood maximization problem is

arg max

𝝂,𝚻,𝝆 𝑜

∏

𝑚=1

𝑂(𝐲𝑚|𝝂𝑗𝑚, 𝚻𝑗𝑚)𝜌𝑗𝑚 subject to

𝐿

∑

𝑗=1

𝜌𝑗 = 1.

▶ Can be solved using a Lagrange multiplier (try it!) and

leads to ˆ 𝜌𝑗 = 𝑜𝑗 𝑜 , with 𝑜𝑗 =

𝑜

∑

𝑚=1

1(𝑗𝑚 = 𝑗) ˆ 𝝂𝑗 = 1 𝑜𝑗 ∑

𝑗𝑚=𝑗

𝑦𝑚 ˆ 𝚻𝑗 = 1 𝑜𝑗 − 1 ∑

𝑗𝑚=𝑗

(𝑦𝑚 − ˆ 𝝂𝑗)(𝑦𝑚 − ˆ 𝝂𝑗)𝑈

19/25

Performing classification in DA

Bayes’ rule implies the classification rule 𝑑(𝐲) = arg max

1≤𝑗≤𝐿

𝑂(𝐲|𝝂𝑗, 𝚻𝑗)𝜌𝑗 Note that since log is strictly increasing this is equivalent to 𝑑(𝐲) = arg max

1≤𝑗≤𝐿

𝜀𝑗(𝐲) where 𝜀𝑗(𝐲) = log 𝑂(𝐲|𝝂𝑗, 𝚻𝑗) + log 𝜌𝑗 = log 𝜌𝑗 − 1 2(𝐲 − 𝝂𝑗)𝑈𝚻−1

𝑗 (𝐲 − 𝝂𝑗) − 1

2 log |𝚻𝑗| (+ 𝐷) This is a quadratic function in 𝐲.

20/25

Different levels of complexity

▶ This method is called Quadratic Discriminant Analysis

(QDA)

▶ Problem: Many parameters that grow quickly with

dimension

▶ 𝐿 − 1 for all 𝜌𝑗 ▶ 𝑞 ⋅ 𝐿 for all 𝝂𝑗 ▶ 𝑞(𝑞 + 1)/2 ⋅ 𝐿 for all 𝚻𝑗 (most costly)

▶ Solution: Replace covariance matrices 𝚻𝑗 by a pooled

estimate ˆ 𝚻 =

𝐿

∑

𝑗=1

ˆ 𝚻𝑗 𝑜𝑗 − 1 𝑜 − 𝐿 = 1 𝑜 − 𝐿

𝐿

∑

𝑗=1

∑

𝑗𝑚=𝑗

(𝑦𝑚 − ˆ 𝝂𝑗)(𝑦𝑚 − ˆ 𝝂𝑗)𝑈

▶ Simpler correlation and variance structure: All classes

are assumed to have the same correlation structure between features

21/25

Performing classification in the simplified case

As before, consider 𝑑(𝐲) = arg max

1≤𝑗≤𝐿

𝜀𝑗(𝐲) where 𝜀𝑗(𝐲) = log 𝜌𝑗 + 𝐲𝑈𝚻−1𝝂𝑗 − 1 2𝝂𝑈

𝑗 𝚻−1𝝂𝑗

(+ 𝐷) This is a linear function in 𝐲. The method is therefore called Linear Discriminant Analysis (LDA).

22/25

Even more simplifications

Other simplifications of the correlation structure are possible

▶ Ignore all correlations between features but allow

different variances, i.e. 𝚻𝑗 = 𝚳𝑗 for a diagonal matrix 𝚳𝑗 (Diagonal QDA or Naive Bayes’ Classifier)

▶ Ignore all correlations and make feature variances equal,

i.e. 𝚻𝑗 = 𝚳 for a diagonal matrix 𝚳 (Diagonal LDA)

▶ Ignore correlations and variances, i.e. 𝚻𝑗 = 𝜏2𝐉𝑞×𝑞

(Nearest Centroids adjusted for class frequencies 𝜌𝑗 )

23/25

Examples of LDA and QDA

• 2

3 4 4 5 6 7 8

Sepal Length Sepal Width

Nearest Centroids

• ●
• 2

3 4 4 5 6 7 8

Sepal Length Sepal Width

LDA

• ●
• 2

3 4 4 5 6 7 8

Sepal Length Sepal Width

QDA

Species

• setosa

versicolor virginica

Decision boundaries can be found with 𝑂(𝐲|𝝂𝑗, 𝚻𝑗)𝜌𝑗 = 𝑂(𝐲|𝝂

𝑘, 𝚻 𝑘)𝜌 𝑘

for 𝑗 ≠ 𝑘 and 𝚻𝑗 = 𝚻 for LDA and 𝚻𝑗 = 𝜏2𝐉𝑞×𝑞 for Nearest Centroids.

24/25

Take-home message

▶ Classification can be achieved through the point-of-view

• f regression

▶ Modelling the conditional densities of features instead of

classes leads to Discriminant Analysis (DA)

▶ There is a range of assumptions in DA about the

correlation structure in feature space → trade-off between stability and flexibility

25/25