CS480/680 Lecture 7: May 29, 2019 Classification with Mixture of - - PowerPoint PPT Presentation

▶

Feb 10, 2024 372 likes •560 views

CS480/680 Lecture 7: May 29, 2019 Classification with Mixture of Gaussians [B] Sections 4.2, [M] Section 4.2 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Linear Models Probabilistic Generative Models Regression

SLIDE 1

CS480/680 Lecture 7: May 29, 2019

Classification with Mixture of Gaussians [B] Sections 4.2, [M] Section 4.2

CS480/680 Spring 2019 Pascal Poupart 1 University of Waterloo

SLIDE 2

Linear Models

Probabilistic Generative Models

Regression Classification

CS480/680 Spring 2019 Pascal Poupart 2 University of Waterloo

SLIDE 3

Probabilistic Generative Model

Pr(𝐷): prior probability of class 𝐷
Pr 𝒚 𝐷 : class conditional distribution of 𝒚
Classification: compute posterior Pr(𝐷|𝒚) according

to Bayes’ theorem

Pr 𝐷 𝒚 = Pr 𝒚 𝐷 Pr 𝐷 ∑! Pr 𝒚 𝐷 Pr 𝐷 = 𝑙𝑄𝑠 𝒚 𝐷 Pr(𝐷)

CS480/680 Spring 2019 Pascal Poupart 3 University of Waterloo

SLIDE 4

Assumptions

In classification, the number of classes is finite, so a

natural prior Pr(𝐷) is the multinomial Pr 𝐷 = 𝑑! = 𝜌!

When 𝒚 ∈ ℜ", then it is often OK to assume that

Pr(𝒚|𝐷) is Gaussian.

Furthermore, assume that the same covariance

matrix 𝚻 is used for each class. Pr 𝒚 𝑑! ∝ 𝑓#$

% 𝒚#𝝂𝒍 "𝚻#$ 𝒚#𝝂𝒍

CS480/680 Spring 2019 Pascal Poupart 4 University of Waterloo

SLIDE 5

Posterior Distribution

Pr 𝑑! 𝒚 =

""##$

% 𝒚#𝝂𝒍 )𝚻#$ 𝒚#𝝂𝒍

∑" ""##$

% 𝒚#𝝂𝒍 )𝚻#$ 𝒚#𝝂𝒍

=

""##$

% 𝒚)𝚻#$𝒚#%𝝂" )𝚻#$+,𝝂" )𝚻#$𝝂"

∑" ""##$

% 𝒚)𝚻#$𝒚#%𝝂" )𝚻#$𝒚,𝝂" )𝚻#$𝒗"

Consider two classes 𝑑! and 𝑑

%

=

& &'

./0𝝂/ )𝚻#$𝒚#$ %𝝂/ )𝚻#$𝝂/ ."0𝝂" )𝚻#$𝒚#$ %𝝂" )𝚻#$𝝂"

CS480/680 Spring 2019 Pascal Poupart 5 University of Waterloo

SLIDE 6

Posterior Distribution

=

$ $)*

# 𝝂& "#𝝂' " 𝚻#$𝒚*$ +𝝂& "𝚻#$𝝂&#$ +𝝂' "𝚻#$𝝂'#,-.& .'

=

$ $)*# 𝒙"𝒚*01

where 𝒙 = 𝚻#$(𝝂! − 𝝂+) and 𝑥, = −

$ % 𝝂!

𝚻#$𝝂! +

$ % 𝝂+

𝚻#$𝝂+ + ln

.& .'

CS480/680 Spring 2019 Pascal Poupart 6 University of Waterloo

SLIDE 7

Logistic Sigmoid

Let 𝜏 𝑏 =

$ $)*#2

Then Pr 𝑑! 𝒚 = 𝜏(𝒙-𝒚 + 𝑥,)
Picture:

Logistic sigmoid

CS480/680 Spring 2019 Pascal Poupart 7 University of Waterloo

SLIDE 8

Logistic Sigmoid

class conditionals posterior

CS480/680 Spring 2019 Pascal Poupart 8 University of Waterloo

SLIDE 9

Prediction

𝑐𝑓𝑡𝑢 𝑑𝑚𝑏𝑡𝑡 = 𝑏𝑠𝑕𝑛𝑏𝑦! Pr 𝑑! 𝒚 = A𝑑$ 𝜏 𝒙-𝒚 + 𝑥, ≥ 0.5 𝑑%

therwise

Class boundary: 𝜏 𝒙!

𝒚 = 0.5 ⟹

$ $)*# 𝒙&

"3 𝒚 = 0.5

⟹ 𝒙!

𝒚 = 0 ∴ linear separator

CS480/680 Spring 2019 Pascal Poupart 9 University of Waterloo

SLIDE 10

Multi-class Problems

Consider Gaussian conditional distributions with identical Σ

Pr 𝑑1 𝒚 =

23 4! 23 𝒚 𝑑1 ∑" 23 4" 23 𝒚 𝑑 6

=

7!8#$

% 𝒚#𝝂! (𝚻#$ 𝒚#𝝂𝒍

∑" 7"8#$

% 𝒚#𝝂" ( 𝚻#$ 𝒚#𝝂"

=

7!8#$

% #%𝝂! (𝚻#$𝒚+𝝂! (𝚻#$𝝂!

∑" 7"8#$

% #%𝝂" (𝚻#$𝒚+𝝂" (𝚻#$𝝂"

=

8𝝂!

(𝚻#$𝒚#$ %𝝂! (𝚻#$𝒗!+-. /!

∑" 8

𝝂" (𝚻#$𝒚#$ %𝝂" (𝚻#$𝝂"+-. /" =

8𝒙!

(1 𝒚

∑" 8

𝒙" (1 𝒚 ⟹ softmax

where 𝒙2

3 = (− 4 5 𝝂2 3𝚻64𝝂2 + ln 𝜌2 , 𝝂2 3𝚻64)

CS480/680 Spring 2019 Pascal Poupart 10 University of Waterloo

SLIDE 11

Softmax

When there are several classes, the posterior is a softmax

(generalization of the sigmoid)

Softmax distribution: Pr 𝑑1 𝒚 =

8!" 𝒚 ∑$ 8!$ 𝒚

Argmax distribution:

Pr 𝑑1 𝒚 = -1 if 𝑙 = 𝑏𝑠𝑕𝑛𝑏𝑦6 𝑔

6(𝑦)

therwise

= lim

9:;8→= 9:;8!" % ∑$ 9:;8!$ %

≈

8!" % ∑$ 8!$(%)

(softmax approximation)

CS480/680 Spring 2019 Pascal Poupart 11 University of Waterloo

SLIDE 12

Softmax

class conditionals posterior

CS480/680 Spring 2019 Pascal Poupart 12 University of Waterloo

SLIDE 13

Parameter Estimation

Where do Pr(𝑑!) and Pr(𝒚|𝑑!) come from?
Parameters: 𝜌, 𝝂𝟐, 𝝂𝟑, 𝚻

Pr 𝑑! = 𝜌, Pr 𝒚 𝑑! = 𝑙𝚻 𝑓#!

" 𝒚#𝝂𝟐 $𝚻%! 𝒚#𝝂!

Pr 𝑑& = 1 − 𝜌, Pr 𝒚 𝑑& = 𝑙𝚻 𝑓#!

" 𝒚#𝝂𝟑 $𝚻%! 𝒚#𝝂"

where 𝑙𝚻 is the normalization constant that depends on 𝚻

Estimate parameters by

– Maximum likelihood – Maximum a posteriori – Bayesian learning

CS480/680 Spring 2019 Pascal Poupart 13 University of Waterloo

SLIDE 14

Maximum Likelihood Solution

Likelihood:

L 𝐘, 𝐳 = Pr 𝒀, 𝒛 𝜌, 𝝂', 𝝂(, 𝚻 = 6

)

𝜌𝑂 𝒚) 𝝂', 𝚻

1 − 𝜌 𝑂 𝒚) 𝝂(, 𝚻

'+*!

ML hypothesis:

< 𝜌∗, 𝝂'

∗, 𝝂( ∗, 𝚻∗ > =

𝑏𝑠𝑕𝑛𝑏𝑦-,𝝂",𝝂",𝚻 ∑) 𝑧) ln 𝜌 + ln 𝑙𝚻 − '

( 𝒚) − 𝝂' 1𝚻+' 𝒚) − 𝝂'

+ 1 − 𝑧) ln 1 − 𝜌 + ln 𝑙𝚻 − '

( 𝒚) − 𝝂( 1𝚻+' 𝒚) − 𝝂(

𝑧> ∈ {0,1}

CS480/680 Spring 2019 Pascal Poupart 14 University of Waterloo

SLIDE 15

Maximum Likelihood Solution

Set derivative to 0

0 = G HI J 𝒀,𝒛

G"

⟹ 0 = ∑K 𝑧K

& " + 1 − 𝑧K

−

& &L"

⟹ 0 = ∑K 𝑧K 1 − 𝜌 + (1 − 𝑧K)(−𝜌) ⟹ ∑K 𝑧K = 𝜌 ∑K 𝑧K + ∑K 1 − 𝑧K ⟹ ∑K 𝑧K = 𝜌𝑂 (where 𝑂 is the # of training points) ∴

∑? M? N

= 𝜌

CS480/680 Spring 2019 Pascal Poupart 15 University of Waterloo

SLIDE 16

Maximum Likelihood Solution

0 = 𝜖 ln 𝑀 𝒀, 𝒛 /𝜖𝝂! ⟹ 0 = ∑" 𝑧"[−𝚻#! 𝒚" − 𝝂! ] ⟹ ∑" 𝑧"𝒚" = ∑" 𝑧"𝝂! ⟹ ∑" 𝑧"𝒚" = 𝑂

!𝝂!

∴

∑( %(𝒚( ')

= 𝝂! Similarly:

∑((!#%()𝒚( '*

= 𝝂*

where 𝑂

$ is the # of data points in class 1

𝑂% is the # of data points in class 2

CS480/680 Spring 2019 Pascal Poupart 16 University of Waterloo