Midterm Review Jan-Willem van de Meent Review: Frequent Itemsets - - PowerPoint PPT Presentation

Unsupervised Machine Learning 
 and Data Mining

DS 5230 / DS 4420 - Fall 2018

Midterm Review

Jan-Willem van de Meent

Review: 
 Frequent Itemsets

Frequent Itemsets

• Items = {milk, coke, pepsi, beer, juice}
• Baskets

B1 = {m, c, b} B2 = {m, p, j} B3 = {m, b} B4 = {c, j} B5 = {m, c, b} B6 = {m, c, b, j} B7 = {c, b, j} B8 = {b, c}

• Frequent itemsets (σ(X) ≥ 3):

{m}:5, {c}:6, {b}:6, {j}:4, {m,c}: 3, 
 {m,b}:4, {c,b}:5, {c,j}:3, {m,c,b}:3

Example: Confidence and Interest

B1 = {m, c, b} B2 = {m, p, j} B3 = {m, b} B4 = {c, j} B5 = {m, c, b} B6 = {m, c, b, j} B7 = {c, b, j} B8 = {b, c}

• Association rule: {m} →b
• Confidence = 4/5
• Interest Factor = 1/6 4/5 = 4/30
• Item b appears in 6/8 of the baskets
• Rule is not very interesting!

adapted from: J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

Lift = c(A − → B) s(B) ,

Apriori – Overview

C1 L1 C2 L2 C3 Filter Filter Construct Construct

All items All pairs

• f items

from L1 Count the pairs All pairs of sets
 that differ by 
 1 element Count the items

• 1. Set k = 0
• 2. Define C1 as all size 1 item sets
• 3. While Ck+1 is not empty
• 4. Set k = k + 1
• 5. Scan DB to determine subset Lk ⊆Ck


with support ≥ s

• 6. Construct candidates Ck+1 by combining


sets in Lk that differ by 1 element (I/O limited) (Memory
 limited)

FP-Growth – Intuition

• Apriori requires one pass for each k
• Can we find all frequent item sets


in fewer passes over the data? FP-Growth Algorithm:

• Pass 1: Count items with support ≥ s
• Sort frequent items in descending 

• rder according to count
• Pass 2: Store all frequent itemsets 


in a frequent pattern tree (FP-tree)

• Mine patterns from FP-Tree

FP-Growth vs Apriori

Advantages of FP-Growth

• Only 2 passes over dataset
• Stores “compact” version of dataset
• No candidate generation
• Faster than A-priori

Disadvantages of FP-Growth

• The FP-Tree may not be “compact”


enough to fit in memory

• Used in practice: PFP (a distributed


version of FP-growth)

Review: ML Basics

What is Similarity?

Can be hard to define, but we know it when we see it.

Regression: Similar points x and x’ should 
 have similar function values f(x) and f(x’)

Similarity Metrics in Machine Learning

Clustering: Similar points x and x’ should 
 have the same cluster assignments z and z’ Dimensionality Reduction: Reduce dimension of points x and x’ in a manner that preserves similarities

Distance Norms

Euclidean Distance s (

k

P

i=1

(xi − yi)2) Mahattan Distance

k

P

i=1

|xi − yi| Minkowski Distance ✓ k P

i=1

(|xi − yi|)q ◆ 1

q

Sensitivity to Scaling

Normalization Strategies

Min-Max: X−Xmin Xmax−Xmin

a + (X−Xmin)(b−a)

Xmax−Xmin Z-score: X ∼ N(µ, σ): X−µ σ Scaling: X Xmax

Curse of Dimensionality

Fraction of Volume Distance 0.0 0.2 0.4 0.6 0.0 0.2 0.4 0.6 0.8 1.0 p=1 p=2 p=3 p=10

30% 9% 3%

D=10 D=3 D=2 D=1

Implication: Estimating similarities 
 is difficult for high-dimensional data

Review: Probability

Bayes' Rule

Posterior Likelihood Prior

Sum Rule: Product Rule:

Expected Values

Statistics Machine Learning

(explicitly define distribution) (distribution implied by X) X is a random variable with density p(x)

Mean, Variance, Covariance

Mean

Variance Covariance

ΣX,Y = Cov[X, Y ] = E[(X − µX)(Y − µY )]

Objectives in Unsupervised Learning

Model parameters (e.g. neural network weights) Unknown Variables Known Variables Quantity

• f Interest

Self-driving Cars Medical Diagnosis Pedestrian motion Will pedestrian cross road? Chance of accident Symptoms / Test results Condition of patient Treatment outcome Setting Model for pedestrian behavior Model for diseases / symptoms

ML and MAP estimation

Maximum A Posterior Estimation

ML Objective (same as minimizing a loss function) Regularization

Maximum Likelihood Estimation

Maximum Likelihood Estimation

Likelihood of N independent events:

pθ(xn) =

K

Y

k=1

θ

xn,k k

Maximum likelihood estimation

Nk =

N

X

n=1

xn,k

(known as cross-entropy loss in neural net libraries)

Maximum Likelihood Estimation

Likelihood of N independent events:

pθ(xn) =

K

Y

k=1

θ

xn,k k

Maximum likelihood estimation

k = Nk

N

Nk =

N

X

n=1

xn,k

Intuition: Bayesian Posterior

Posterior Likelihood Prior

MAP estimate after 50 trials

Properties of Conjugate Priors

Bern(x | µ) = µN1(1 − µ)N0

Likelihood Sufficient Statistics Conjugate Prior

Beta(µ | a, b) = 1 B(a, b)µa−1(1 − µ)b−1

Posterior

N1 =

N

X

n=1

xn, N0 = N − N1

Marginal Likelihood

Intuition: Bayesian Posterior

• Sufficient Statistics N1 and N0 are counts (knowledge of the data)
• Parameters a and b are “pseudocounts” (knowledge of the prior)
• Posterior combines our knowledge of the prior and data.

Review: Exponential Families

Exponential Family Distributions

p(x | η) = h(x)exp ñX

i

ηiti(x) − a(η) ô

Base Measure Natural Parameters Sufficient Statistics Log Normalizer Product of terms: x-dependent, xη-dependent, η-dependent

Conjugate Priors

p(x | η) = h(x)exp ñX

i

ηiti(x) − a(η) ô

Likelihood (exponential family)

Conjugate Prior (also exponential family)

Joint Distribution x-dep, η-dep, ηxλ-dep, ηλ-dep, λ-dep

Conjugate Priors

p(x | η) = h(x)exp ñX

i

ηiti(x) − a(η) ô

Likelihood (exponential family)

Conjugate Prior (also exponential family)

Posterior Distribution (same family as prior)

Review: Information Theory

Information Theory

KL Divergence Entropy Mutual Information Intuition: Dissimilarity 
 between distributions (assymetric) Intuition: Log number

• f distinct choices

Example Question

(a) (b) (c)

• a. (easy) Rank the 3 probability distributions 


from highest to lowest entropy

p(x) x x x

• b. (more difficult) Which of these distributions


has an entropy H[x] > log 5 (b) > (a) > (c) (b)

Review: Naive Bayes

Motvation: Spam Filtering

n

x =            1 . . . 1 . . .            a aardvark aardwolf . . . buy . . . zygmurgy

Features: Words in E-mail Labels: Spam or not Spam

Input: Labeled Data Goal: Predict Unlabeled Data

Graphical Model: Naive Bayes

Parameters Observed Variables (value known) Unobserved Variables (value unknown)

Maximum Likelihood Estimation

argmax

µ,φ N

X

n=1

log p(yn, xn | µ,φ)

Objective

Interpretation: Fraction


• f Spam in training set

Interpretation: Fraction


• f non-spam (k=0) and

spam (k=1) messages that contain term. Optimum Parameters Training Data (Labeled)

Maximum Likelihood Estimation

argmax

µ,φ N

X

n=1

log p(yn, xn | µ,φ)

Objective Training Data (Labeled)

Optimum Parameters Problem: What do you do for words
 not found in training set?

X

n

I[xnd = 1] = 0

Maximum Likelihood Estimation

argmax

µ,φ N

X

n=1

log p(yn, xn | µ,φ)

Objective Training Data (Labeled)

Optimum Parameters Problem: What do you do for words
 not found in training set?

X

n

I[xnd = 1] = 0

µ = N y

1 +1

N+2 φkd = N x

kd+1

N y

k +D

(with Laplace smoothing)

Naive Bayes with Priors

Generative Model

Graphical Model Predictive Distribution (Online)

(same as using Laplace smoothing)

Review: Gaussians

Central Limit Theorem

N = 1 0.5 1 1 2 3 N = 2 0.5 1 1 2 3 N = 10 0.5 1 1 2 3

If X1, …, XN are 1. Independent identically distributed (i.i.d.) 2. Have finite variance 0 < σX 2 < ∞ Then, as N approaches ∞, the mean is distributed as

¯ X = 1 N

N

X

n=1

Xn

Multivariate Normal

• 1. CRIM: per capita crime rate by town

Density: p(x;µ,Σ) =

1 p (2π)D|Σ| exp 1

2 (xµ)>Σ1(xµ)

Vectors (x1, …, xD) (μ1, …, μD) Covariance Matrix Mean:

Covariance: Cov[Xd, Xe] = Σde

Covariance Matrices

Density:

 1.0 0.0 0.0 1.0

•  2.0

0.0 0.0 0.5

•  1.0

0.5 0.5 1.0

• 

1.0 −0.5 −0.5 1.0

• Question: Which covariance matrix Σ 


corresponds to which plot?

Marginals and Conditionals

Suppose that x and y are jointly Gaussian:

x x y

Question: What are the marginal
 distributions p(x) and p(y)?

z = x y

• ∼ N

✓a b

• ,

 A C CT B ◆

x ∼ N (a, A) y ∼ N (b, B)

Marginals and Conditionals

Suppose that x and y are jointly Gaussian:

x x y

Question: What are the conditional
 distributions p(x | y) and p(y | x)?

z = x y

• ∼ N

✓a b

• ,

 A C CT B ◆

x|y ∼ N

• a + CB−1(y − b), A − CB−1CT

y|x ∼ N

• b + CT A−1(x − a), B − CT A−1C

Empirical Estimates

Assumption: Data are samples from unknown distribution

{x1,..., xN}

Data (given)

xn ∼ pDATA(x)

Unknown Distribution

Empirical Estimates: Average over Samples

xn ∈ RD

Covariance

Σd,e = EpDATA(x)[(xd µd)(xe µe)]

Mean

µd = EpDATA(x)[xd]

N

X

µd ' 1 N

N

X

n=1

xn,d

• ˆ

Σd,e ' 1 N

N

X

n=1

(xn,d µd)(xn,e µe)

Maximum Likelihood for Gaussians

Assumption: Data sampled from Gaussian Maximum Likelihood: Same as empirical estimate

xn ∼ Normal(µ,Σ)

xn ∈ RD

µ∗,Σ∗ = argmax

µ,Σ

log p(x1,..., xN | µ,Σ)

µ⇤

d = ˆ

µd ' 1 N

N

X

n=1

xn,d

ˆ Σ⇤ = ˆ Σd,e ' 1 N

N

X

n=1

(xn,d µd)(xn,e µe)

Note: Covariance as a Matrix Product

X =     — x>

1

— — x>

2

— ... — x>

N

—    

Original Data (as matrix)

˜ X =     — (x1 ˆ µ)> — — (x2 ˆ µ)> — ... — (xN ˆ µ)> —    

Centered Data (subtract mean)

N x D N x D

(remember: X is size D x N, not N x D, in PCA slides) Covariance

D x N N x D D x D

ˆ Σ = 1 N ˜ X > ˜ X

Review: PCA

Dimensionality Reduction

Original Data (4 dims) Projection with PCA (2 dims) Goal: Map high dimensional data onto lower-dimensional data in a manner that preserves distances/similarities Objective: projection should “preserve” relative distances

Problem Setup

Given n data points in d dimensions: x1, . . . , xn ∈ Rd X =( x1 · · · · · · xn) ∈ Rd⇥n Want to reduce dimensionality from d to k Choose k directions u1, . . . , uk U =( u1 ·· uk ) ∈ Rd⇥k For each uj, compute “similarity” zj = u>

j x

Transpose of X
 used elsewhere!

Equivalence of two objectives

Key intuition: variance of data | {z }

fixed

= captured variance | {z }

want large

+ reconstruction error | {z }

want small

Pythagorean decomposition: x = UU>x + (I UU>)x kUU>xk k(I UU>)xk kxk Take expectations; note rotation U doesn’t affect length: ˆ E[kxk2] = ˆ E[kU>xk2] + ˆ E[kx UU>xk2] Minimize reconstruction error $ Maximize captured variance

Data

X =( x1 · · · · · · xn) ∈ Rd⇥n

Eigenvectors of Covariance

Principal Component Analysis

= ˆ Σ

Intuition: PCA fits a Gaussian to data, and then projects this Gaussian to the largest principal components

U =( u1 ·· uk ) ∈ Rd⇥k

Principal Components Truncated decomposition

Λ(k) =    λ1 λ2 ... λk   

Review: Regression

ε ∼ Norm(0,σ2)

Assume f is a linear combination of D features

Linear Regression

Learning: Estimate w For N points we write Prediction: Estimate y’ given x’

Basis function regression

Linear regression Basis function regression Polynomial regression For N samples

Polynomial Regression

x t M = 0 1 −1 1 x t M = 1 1 −1 1 x t M = 3 1 −1 1 x t M = 9 1 −1 1

Polynomial Regression

x t M = 0 1 −1 1 x t M = 1 1 −1 1 x t M = 3 1 −1 1 x t M = 9 1 −1 1

Underfit

Polynomial Regression

x t M = 0 1 −1 1 x t M = 1 1 −1 1 x t M = 3 1 −1 1 x t M = 9 1 −1 1

Overfit

L2 regularization (ridge regression) minimizes: E(w) = 1 N k Xw y k2 + λ k w k2 where λ 0 and k w k2 = wTw

• k

k L1 regularization (LASSO) minimizes: E(w) = 1 N k Xw y k2 + λ|w|1 where λ 0 and |w|1 =

D

P

i=1

|ωi|

Regularization

Regularization

Regression with Priors

From Bayes Rule

Maximum a Posteriori

Maximum a Posteriori is Equivalent to Ridge Regression

−6 −4 −2 2 4 6 −2 2 −6 −4 −2 2 4 6 −2 2 −6 −4 −2 2 4 6 −2 2

adapted from: Carl Rasmussen, Probabilistic Machine Learning 4f13, http://mlg.eng.cam.ac.uk/teaching/4f13/1617/

The Predictive Distribution

Idea: Average over all possible values of w Increasing λ

Predictive Distribution for Regression

     y∗

1

y∗

2

y1 ... yN      ∼ Norm           E[y∗

1]

E[y∗

2]

E[y1] ... E[yN]     ,      Cov(y∗

1, y∗ 1)

Cov(y∗

1, y∗ 2)

Cov(y∗

1, y1)

... Cov(y∗

1, yN)

Cov(y∗

2, y∗ 1)

Cov(y∗

2, y∗ 2)

... ... Cov(y1, y∗

1)

... Cov(y1, y1) ... Cov(y1, yN) ... Cov(yN, y∗

1)

... ... Cov(yN, yN)          

Implication: Can predict (y*1, y*2) from (y1, …, yN)

y⇤ | y ⇠ Norm

• a + CB1(y b), A CB1C >

Key Idea: y* and y are jointly Gaussian

˜ y = ïy⇤ y ò ⇠ Norm Åïa b ò , ï A C C > B òã

Predictive Distribution for Regression

y⇤ | y ⇠ Norm

• a + CB1(y b), A CB1C >

Key Idea: y* and y are jointly Gaussian

˜ y = ïy⇤ y ò ⇠ Norm Åïa b ò , ï A C C > B òã

ïa b ò = ïE[y∗] E[y] ò = ïX∗ E[w] X E[w] ò = ïX∗ m0 X m0 ò

ï A C C > B ò = ïCov(y⇤, y⇤) Cov(y⇤, y) Cov(y, y⇤) Cov(y, y) ò =  X⇤ S0 X⇤ >+σ2I X⇤ S0 X> X S0 X⇤ > X S0 X>+σ2I

• <latexit sha1_base64="xnupYX1ofYthUTfpRT8K6W81Uqw=">AAAIFXiclZRbb9MwFICzMugItw0eecmoNI1RTcmYtgkJqWyagLcxdpPmrnLS09ZabjhOd7H8O/gJ/Al4Q7wh3kD8G+w0sCbONhFL0dE5n8/Nx3ZjnyTMtn9P1G5M3rxVn7pt3rl77/6D6ZmHe0mUUg92vciP6IGLE/BJCLuMMB8OYgo4cH3Yd483lH1/CDQhUbjDzmJoB7gfkh7xMJOqzkythXzosUPLRC70ScgDzCg5FSYa8lfCmrPQ7N9lSdWGsBAyM+EIsSjWgHW5E8LuhRtK+gPWNl+al8SxLMTglPGNaCjmpYMzcbTQtHLhqSXtc4oZBbiKzliZnI5c6k4mU41K8L+rGPIDGUIKmY/3omPnolTzBQu9sFTDZP9mn6lWJaQf4KMl620hJbWudZb5+Ves0l0b9dIgV4XQU61qSme6YS/a2WfpgpMLjZYx+rY6M5OfUTfy0gBC5vk4SQ4dO2Ztjikjng/ScZpAjL1j3IdDKYY4gKTNs1kXVsG647R5LwoZhF5hG8dBItMcaEoFJ0WtN5CBgRbD5sq2KnbQBdmBsLjLDYRpoi705L3LMuNd109B8O3X64LbzZXnTWdpVZQQCt2ccNbsplxloE8BwhxZW246K2s6E6c09uECshWmsqEQwokXBQGWp4SG4IlD2R95ZklKQRXCkRvwhiOE0OARKvdkdhONG08F56iQoJySU1HGzsYwVWh2k8rQeZWvcw37UIUhNgCGK7Jn1TROK9hUY1MdohpEyxlCZUyIE+JHoVZPb4zO5qSnB/XHmPyMlUtfPuZdrHmMB9V4PCAau106mW2hxmWcwFRecXnOKIqBYhZRdelOCBv4JCAs4bld6LtIePUuaS8H2xTFoVR/1+WbQiM9188Gs9g7fUI92i1yqsoKrE+L2OjgKsC4BOYNVqR875zy66YLe0uLjpTfLTda6/nLN2U8Np4Y84ZjrBot442xZewaXu1T7XvtZ+1X/WP9S/1r/dsIrU3kex4Zha/+4w8tF9or</latexit>

w ∼ Norm(m0,S0)

Assume prior:

Relationship to Gaussian Processes

Linear Regression

E[y] X E[w] Cov[y, y] X Cov[w, w] X> + σ2I

Basis Function Regression

φ(X) E[w] φ(X)Cov[w, w]φ(X)> + σ2I k

Gaussian Process

µ(X) I k(X, X) + σ2I

(high-level)

Cov[f (x), f (x 0)] := k(x, x 0) = φ(x)>Cov[w, w]φ(x 0)

Kernel Function: Defines covariance between function values Intuition: Gaussian Process Regression is the same as
 Bayesian Regression relative to some feature map φ

Kernel Functions

source: David Duvenaud (PhD Thesis)

Kernel name: Squared-exp (SE) Periodic (Per) Linear (Lin) k(x, xÕ) = σ2

f exp

1

−(x≠xÕ)2

2¸2

2

σ2

f exp

1

− 2

¸2 sin2 1

π x≠xÕ

p

22

σ2

f(x − c)(xÕ − c)

Plot of k(x, xÕ): x − xÕ x − xÕ x (with xÕ = 1)

↓ ↓ ↓

Functions f(x) sampled from

GP prior:

x x x Type of structure: local variation repeating structure linear functions

Combinations of Kernels

source: David Duvenaud (PhD Thesis)

Lin × Lin SE × Per Lin × SE Lin × Per

x (with xÕ = 1) x − xÕ x (with xÕ = 1) x (with xÕ = 1)

↓ ↓ ↓ ↓

quadratic functions locally periodic increasing variation growing amplitude

Review: Clustering

Four Types of Clustering

• 1. Centroid-based (K-means, K-medoids)

Notion of Clusters: Voronoi tesselation

Four Types of Clustering

• 2. Connectivity-based (Hierarchical)

Notion of Clusters: Cut off dendrogram at some depth

Four Types of Clustering

• 3. Density-based (DBSCAN, OPTICS)

Notion of Clusters: Connected regions of high density

Four Types of Clustering

• 4. Distribution-based (Mixture Models)

Notion of Clusters: Distributions on features (not on midterm)

Comparing Clustering Methods

Agglomerative
 (average) MiniBatch
 KMeans DBSCAN Agglomative (ward) http://scikit-learn.org/stable/auto_examples/cluster/plot_cluster_comparison.html GMM (not on midterm)