Lecture 13 Jan-Willem van de Meent Four Types of Clustering 1. - - PowerPoint PPT Presentation

Unsupervised Machine Learning 
 and Data Mining

DS 5230 / DS 4420 - Fall 2018

Lecture 13

Jan-Willem van de Meent

Four Types of Clustering

• 1. Centroid-based (K-means, K-medoids)

Notion of Clusters: Voronoi tesselation

Four Types of Clustering

• 4. Distribution-based (Mixture Models)

Notion of Clusters: Distributions on features (not on midterm)

Review: K-means Clustering

Objective: Sum of Squares μ1 μ2 μ3 Alternate between two steps
 


• 1. Minimize SSE w.r.t. zn
• 2. Minimize SSE w.r.t. μk

µk

One-hot assignment Center for cluster k

SSE(z,µ) =

N

X

n=1 K

X

k=1

I[zn = k]|xn − µk|2

1 2 3 4 5 1 2 3 4 5

μ1 μ2 μ3

K-means Clustering

Assign each point to closest centroid, then update centroids to average of points

1 2 3 4 5 1 2 3 4 5

μ1 μ2 μ3

K-means Clustering

Assign each point to closest centroid, then update centroids to average of points

1 2 3 4 5 1 2 3 4 5

μ1 μ2 μ3

K-means Clustering

Repeat until convergence 
 (no points reassigned, means unchanged)

1 2 3 4 5 1 2 3 4 5

μ1 μ2 μ3

K-means Clustering

Repeat until convergence 
 (no points reassigned, means unchanged)

K-Means vs Gaussian Mixture Models

μ1 μ2 μ3 Idea1: Learn both means μk and covariances Σk μ3 Σ3 μ2 Σ2 μ1 Σ1 Don’t just learn where the center of the cluster is, but also how big it is, and what shape it has.

μ1 μ2 μ3

γnk = I[zn = k]

Hard assignments to clusters (one-hot vector)

Idea 2: Replace hard assignments with soft assignments

Soft assignments to clusters (posterior probability)

γnk = p(zn=k | xn)

μ3 Σ3 μ2 Σ2 μ1 Σ1

K-Means vs Gaussian Mixture Models

EM for Gaussian Mixtures

Credit: Andrew Moore

EM for Gaussian Mixtures

Credit: Andrew Moore

EM for Gaussian Mixtures

Credit: Andrew Moore

EM for Gaussian Mixtures

Credit: Andrew Moore

EM for Gaussian Mixtures

Credit: Andrew Moore

EM for Gaussian Mixtures

Credit: Andrew Moore

EM for Gaussian Mixtures

Credit: Andrew Moore

Gaussian Mixture Models

μ3 Σ3 μ2 Σ2 μ1 Σ1 Idea 1: Points in each cluster 
 are sampled for a Gaussian Idea 2: Compute probability that point belongs to each cluster

xn | zn=k ∼ Norm(µk,Σk)

γnk = p(zn=k | xn)

K

X

k=1

γnk = 1

Weights sum to 1:

Idea 2: Learn Soft Assignments to Clusters

Posterior on Cluster Assignments (from Bayes’ Rule)

γnk = p(zn=k | xn) = p(xn | zn=k)p(zn=k) p(xn)

Likelihood Prior Marginal Likelihood Posterior

Generative Model (defines prior and likelihood)

zn ∼ Discrete(π1,...,πK) xn | zn=k ∼ Norm(µk,Σk)

Idea 2: Learn Soft Assignments to Clusters

Posterior on Cluster Assignments (from Bayes’ Rule)

γnk = p(zn=k | xn) = p(xn | zn=k)p(zn=k) p(xn)

Likelihood Prior Marginal Likelihood Posterior

p(zn=k) =

) = πk

p(xn | zn=k) =

) = 1 p 2π|Σ| e 1

2 (xnµk)>Σ1(xnµk)

p(xn) =

p | | ) =

K

X

k=1

p(xn | zn=k)p(zn=k)

Prior Likelihood Marginal Likelihood

Idea 1: Learn Gaussian for Each Cluster

Maximum Likelihood Estimation

µ∗,Σ∗,π∗ = argmax

µ,Σ,π

log p(x1,..., xN | µ,Σ,π)

Nk = X

n

γnk

µk = 1 Nk X

n

γnkxn

πk = Nk N

Cluster Mean Cluster Covariance Fraction of points in each cluster Idea: Use weights γnk = p(zn=k | xn) to compute estimates

Σk = 1 Nk X

n

γnk(xn µk)(xn µk)>

Idea 1: Learn Gaussian for Each Cluster

Maximum Likelihood Estimation

µ∗,Σ∗,π∗ = argmax

µ,Σ,π

log p(x1,..., xN | µ,Σ,π)

Idea: Use weights γnk = p(zn=k | xn) to compute estimates

Nk = X

n

γnk

µk = 1 Nk X

n

γnkxn

Σk = X

n

γnk(xn µk)(xn µk)>

πk = Nk N

(same as in k-means) (same as in scatter criteria) (same as in Naive Bayes)

Review: K-means Clustering

Objective: Sum of Squares μ1 μ2 μ3 Alternate between two steps
 


• 1. Minimize SSE w.r.t. zn
• 2. Minimize SSE w.r.t. μk

µk

One-hot assignment Center for cluster k

SSE(z,µ) =

N

X

n=1 K

X

k=1

I[zn = k]|xn − µk|2

Algorithm Initialize parameters to Repeat until convergence

• 1. Update cluster assignments
• 2. Update parameters

θ := {µ1:K,Σ1:K,π}

Intuition: Gaussian K-means

(not a real algorithm that people actually use)

Parameter Updates

µk =

1 Nk

PN

n=1 znk xn

PN π = (N1/N,..., NK/N)

1 P

P Σk =

1 Nk

PN

n=1 znk (xn µk)(xn µk)>

Nk := PN

n=1 znk

znk := I[zn = k]

Assignment Update

How can we deal with 


• verlapping clusters 


in a better way?

Intuition: Gaussian K-means

(not a real algorithm that people actually use)

Parameter Updates

µk =

1 Nk

PN

n=1 znk xn

PN π = (N1/N,..., NK/N)

1 P

P Σk =

1 Nk

PN

n=1 znk (xn µk)(xn µk)>

Assignment Update

Idea: Replace hard 
 assignments with soft assignments

Nk := PN

n=1 znk

znk := I[zn = k]

Intuition: Gaussian K-means

(not a real algorithm that people actually use)

Nk := PN

n=1 γnk

P µk =

1 Nk

PN

n=1 γnk xn

P P Σk =

1 Nk

PN

n=1 γnk (xn µk)(xn µk)>

Parameter Updates

Gaussian Mixture Models

π = (N1/N,..., NK/N)

1 P

Soft Assignment Update

Idea: Replace hard 
 assignments with soft assignments

(a) 0.5 1 0.5 1 (b) 0.5 1 0.5 1 (c) 0.5 1 0.5 1

+ Works with overlapping clusters + Works with clusters of different densities + Same complexity as K-means

• Can get stuck in local maximum
• Need to set number of components

GMM Advantages / Disadvantages

GMM Advantages / Disadvantages

+ Works with overlapping clusters + Works with clusters of different densities + Same complexity as K-means

• Can get stuck in local maximum
• Need to set number of components

Gaussian Mixture Models (formal view)

Generative Model Expectation Maximization Initialize θ Repeat until convergence

• 1. Expectation Step
• 2. Maximization Step

xn | zn = k ∼ Norm(µk,Σk) zn ∼ Discrete(π)

γi = argmax

γ

L

• q(z | γ),θ i1

θ i = argmax

γ

L

• q(z | γi),θ
• <latexit sha1_base64="bo7AJDkLGaSDErHBQdUDu7bS7yQ=">AGOnicfZRLb9QwEIDd0oUSXi0cuazYSyutqRUbS9IFaiCA4dS9SXV28pxZrNWnUcdpy/L/L+BP4AV26IG+IGB+zdoG7iCEeKRp5vXp6xw5yzQvr+15nZe3Od+w/mH3qPHj95+mxh8flBkZWCwj7NeCaOQlIAZynsSyY5HOUCSBJyOAzP3ln94QWIgmXpnrzOYZCQOGVDRok0W6cLQ3yhsByBJPqEd90PUxEnJCrU4VjkiREd73PHk6IHFHC1UftYQ5DueSdLxnDG93FCYu61seEPmHL/e6dSw8LFo/ksne60PNX/PHqukJQCb0tNFk7p4tztzjKaJlAKiknRXEc+LkcKCIkoxyM57KAnNAzEsOxEVOSQDFQ4wMxOU9r94KBGmaphJTWzBRJCluYs2nhor5LRyYwiHrYanOgrJcIChandasw0Z6HIxia5owzU1HIS9Bq9/1brfz+ut+sLqhG4iAqCKCTb9viYQC4C0QjbX+sH6psvkpcg53EG+xWw2AlK4pJlpVhopfAFUH5vzwZAWpQBbiMJhonqB1tqBJ6ixGes9PK280krhWoJmDK50E7uewmyhBrp2oJs2XzcOdt6GVZPnZi/baVK2sKXDli4kHEg0M4TWmJAXjGepU89wih7PydANyqeYqsfWJTc3PiKOx3zUjucj5rC7jc7sajsu08TkZdAKZzkIjNhL90lkyPOEiYLVem1a8XS/1sZfTPYtq4Ppf2HodrWDklDPh7M+tm5E0pFVOdslS1YLOrYpHEtYN4AqwO2pHnvgubr5goHqyuBkT+t9b+vXz6CV6hZQgDbQFvqAdtA+ougL+ol+oz+d2863zvfOjwk6O1PZvEC1fn1F+81Sj4=</latexit><latexit sha1_base64="bo7AJDkLGaSDErHBQdUDu7bS7yQ=">AGOnicfZRLb9QwEIDd0oUSXi0cuazYSyutqRUbS9IFaiCA4dS9SXV28pxZrNWnUcdpy/L/L+BP4AV26IG+IGB+zdoG7iCEeKRp5vXp6xw5yzQvr+15nZe3Od+w/mH3qPHj95+mxh8flBkZWCwj7NeCaOQlIAZynsSyY5HOUCSBJyOAzP3ln94QWIgmXpnrzOYZCQOGVDRok0W6cLQ3yhsByBJPqEd90PUxEnJCrU4VjkiREd73PHk6IHFHC1UftYQ5DueSdLxnDG93FCYu61seEPmHL/e6dSw8LFo/ksne60PNX/PHqukJQCb0tNFk7p4tztzjKaJlAKiknRXEc+LkcKCIkoxyM57KAnNAzEsOxEVOSQDFQ4wMxOU9r94KBGmaphJTWzBRJCluYs2nhor5LRyYwiHrYanOgrJcIChandasw0Z6HIxia5owzU1HIS9Bq9/1brfz+ut+sLqhG4iAqCKCTb9viYQC4C0QjbX+sH6psvkpcg53EG+xWw2AlK4pJlpVhopfAFUH5vzwZAWpQBbiMJhonqB1tqBJ6ixGes9PK280krhWoJmDK50E7uewmyhBrp2oJs2XzcOdt6GVZPnZi/baVK2sKXDli4kHEg0M4TWmJAXjGepU89wih7PydANyqeYqsfWJTc3PiKOx3zUjucj5rC7jc7sajsu08TkZdAKZzkIjNhL90lkyPOEiYLVem1a8XS/1sZfTPYtq4Ppf2HodrWDklDPh7M+tm5E0pFVOdslS1YLOrYpHEtYN4AqwO2pHnvgubr5goHqyuBkT+t9b+vXz6CV6hZQgDbQFvqAdtA+ougL+ol+oz+d2863zvfOjwk6O1PZvEC1fn1F+81Sj4=</latexit><latexit sha1_base64="bo7AJDkLGaSDErHBQdUDu7bS7yQ=">AGOnicfZRLb9QwEIDd0oUSXi0cuazYSyutqRUbS9IFaiCA4dS9SXV28pxZrNWnUcdpy/L/L+BP4AV26IG+IGB+zdoG7iCEeKRp5vXp6xw5yzQvr+15nZe3Od+w/mH3qPHj95+mxh8flBkZWCwj7NeCaOQlIAZynsSyY5HOUCSBJyOAzP3ln94QWIgmXpnrzOYZCQOGVDRok0W6cLQ3yhsByBJPqEd90PUxEnJCrU4VjkiREd73PHk6IHFHC1UftYQ5DueSdLxnDG93FCYu61seEPmHL/e6dSw8LFo/ksne60PNX/PHqukJQCb0tNFk7p4tztzjKaJlAKiknRXEc+LkcKCIkoxyM57KAnNAzEsOxEVOSQDFQ4wMxOU9r94KBGmaphJTWzBRJCluYs2nhor5LRyYwiHrYanOgrJcIChandasw0Z6HIxia5owzU1HIS9Bq9/1brfz+ut+sLqhG4iAqCKCTb9viYQC4C0QjbX+sH6psvkpcg53EG+xWw2AlK4pJlpVhopfAFUH5vzwZAWpQBbiMJhonqB1tqBJ6ixGes9PK280krhWoJmDK50E7uewmyhBrp2oJs2XzcOdt6GVZPnZi/baVK2sKXDli4kHEg0M4TWmJAXjGepU89wih7PydANyqeYqsfWJTc3PiKOx3zUjucj5rC7jc7sajsu08TkZdAKZzkIjNhL90lkyPOEiYLVem1a8XS/1sZfTPYtq4Ppf2HodrWDklDPh7M+tm5E0pFVOdslS1YLOrYpHEtYN4AqwO2pHnvgubr5goHqyuBkT+t9b+vXz6CV6hZQgDbQFvqAdtA+ougL+ol+oz+d2863zvfOjwk6O1PZvEC1fn1F+81Sj4=</latexit><latexit sha1_base64="6S7qD18mUN87SE1VsNH4zveIAU=">AGOnicfZRLb9QwEIBd6EIJrxaOXFbspZVWVKqthekClTBgUOp+pLq7cpxZrNWnUcdpy/L/L+BP4AV26IG+IGB+zdSN3EY4UjTzfvDxjhzlnhfT9b3P37s93HjxceOQ9fvL02fPFpReHRVYKCgc045k4DkBnKVwIJnkcJwLIEnI4Sg8e2/1RxcgCpal+/I6h0FC4pSNGCXSbA0XR/hCYTkGSfQp67tepiIOCFXQ4VjkiREd70vHk6IHFPC1SftYQ4jueydLxvDG93FCYu61seUPmUr/e6dSw8LFo/lijdc7Pmr/mR1XSGohB6q1u5waf4WRxktE0gl5aQoTgI/lwNFhGSUg/FcFpATekZiODFiShIoBmpyICbnWe1+MFCjLJWQ0pqZIklhC3M2LVzUd+nYBAZRD1tDpT1EkHB4rRuFSba83AEI9OcSWYqCnkJWu19eKeV3940w/WNnUDERBVRLDl983XBGIBkFbI1no/2NhymbwUOYc7yLeYzUZACpc0M81KI4UvgOoTcz4Y0qIUYAtROExUL9BaO/AUNTYTvYdnlVdaKVxL0IzBlW5i1zOYLdRA1w50+brxsHO27Bq8tzsZTtNyha2dNjShYQDiWaG0BoT8oLxLHXqGc3QkzkZuUH5DFP12Lrk5sZHxPGYj9vxfMwcdq/RmT1tx2WmL4MWuEsB0FkJuylu2RyzFnCZKEqvXatWPp/K6NvBtvR9aG0/zBUO9ohacgng1k/O3dCqYjqnK2yBYtFHZs2rgXMG2B1wJY0713QfN1c4XBtNTDy5/Xe9nb18i2gV+g1WkYB2kTb6CPaRQeIoq/oF/qD/nZuO987Pzo/p+i9ucrmJaqtzu9/nWZJ/g=</latexit>

Objective

L (q(z | γ),θ) = Eq(z|γ) ï log p(x, z | θ) q(z | γ) ò

Unsupervised (e.g. GMM) Supervised (e.g. regression) Solve for zero
 gradient to find
 maximum

Maximum Likelihood Estimation

w ∗ = argmax

w

log p(y | X, w) = argmax

θ

log P

z p(X, z |θ)

= argmax

θ

PN

n=1 log

PK

zn=1 p(xn,zn |θ)

| = argmax

w N

X

n=1

log p(yn|xn, w)

Not so easy here, because of sum inside logarithm

Lower Bound on Log Likelihood

(multiplication by 1)

Lower Bound on Log Likelihood

(multiplication by 1) (multiplication by 1)

Lower Bound on Log Likelihood

(multiplication by 1) (multiplication by 1) (Bayes rule)

Lower Bound on Log Likelihood

(multiplication by 1) (multiplication by 1) (Bayes rule)

Lower Bound on Log Likelihood

Lower Bound on Log Likelihood

Claim:

Model Selection

Need to specify two components

• 1. Likelihood
• 2. Mixture distribution

How do we know that we have made “good” choices?

Model Selection

Strategy 1: Cross-validation Split data in to K folds. For each fold k

• Perform EM to learn θ from 


training set Xtrain

• Calculate test set likelihood


p(Xtest | θ)

Model Selection

Strategy 2: Model Evidence Define a prior p(θ) and evaluate 
 the marginal likelihood

com- data, symbols i- y that suboptimal hap- K p(D|K) 1 2 3 4 5 6

Two families of methods

• Variational Inference
• Importance Sampling

Variational Inference (Sketch)

Lower bound on Log Evidence Variational E-step Variational M-step

Variational Inference (Sketch)

15 60 120

Can use lower bound

• n evidence to select

best model

com- data, symbols i- y that suboptimal hap- K p(D|K) 1 2 3 4 5 6

Variational inference for


• ften assigns zero weight

to superfluous components