Lecture 5 Jan-Willem van de Meent Conjugate Priors <latexit - - PowerPoint PPT Presentation

Unsupervised Machine Learning 
 and Data Mining

DS 5230 / DS 4420 - Fall 2018

Lecture 5

Jan-Willem van de Meent

Conjugate Priors

ML and MAP estimation

Maximum A Posterior Estimation

ML Objective (same as minimizing a loss function) Regularization

Maximum Likelihood Estimation

Intuition: Bayesian Posterior

Posterior Likelihood Prior

MAP estimate after 50 trials

Bern(x|µ) = µx(1 − µ)1−x E[x] = µ var[x] = µ(1 − µ) mode[x] =

• 1

if µ 0.5,

• therwise

H[ ] = ln (1 ) ln

Likelihood: Bernoulli

µ ∈ [0, 1] that ariable x ∈ {0, 1} by a single continuous

(a discrete distribution with outcomes x=0 and x=1)

Prior: Beta Distribution

Beta(µ|a, b) = Γ(a + b) Γ(a)Γ(b)µa−1(1 − µ)b−1 E[µ] = a a + b var[µ] = ab (a + b)2(a + b + 1) mode[µ] = a − 1 a + b − 2.

(a distribution on values in the range 0 to 1)

Posterior after Trials

Beta(µ | a, b) = 1 B(a, b)µa−1(1 − µ)b−1

Bern(x | µ) = µN1(1 − µ)N0

Properties of Conjugate Priors

Bern(x | µ) = µN1(1 − µ)N0

Likelihood Sufficient Statistics Conjugate Prior

Beta(µ | a, b) = 1 B(a, b)µa−1(1 − µ)b−1

Posterior

N1 =

N

X

n=1

xn, N0 = N − N1

Marginal Likelihood

Intuition: Bayesian Posterior

• Sufficient Statistics N1 and N0 are counts (knowledge of the data)
• Parameters a and b are “pseudocounts” (knowledge of the prior)
• Posterior combines our knowledge of the prior and data.

MAP Estimate After Trials

Beta(µ|a, b) = Γ(a + b) Γ(a)Γ(b)µa−1(1 − µ)b−1 a

Question: After 1 trial with outcome x = 1, what is
 the ML estimate? If we use a prior Beta(μ | 2, 2) then what is the MAP estimate?

mode[µ] = a − 1 a + b − 2.

θ ML =

= N1 N = 1

= N1 + a − 1 N + a + b − 2 = 2 3

θ MAP =

p(µ | a, b, x) = Beta(µ | N1+a, N0+b)

The Marginal Likelihood

“Average” likelihood of data (under prior)

p(x | a, b) = Z p(x | µ) p(µ | a, b) dµ

Marginal likelihood (a.k.a. the Evidence) Prior Likelihood

Z = Ep(µ|a,b)[p(x | µ)]

Can calculate in closed form for conjugate priors

Question: Suppose we have a = 1, b = 1, and an 


• utcome x = 1, then what is the predictive probability?

The Predictive Distribution

Future data Past data

Graphical Models

Motvation: Spam Filtering

n

x =            1 . . . 1 . . .            a aardvark aardwolf . . . buy . . . zygmurgy

Features: Words in E-mail Labels: Spam or not Spam

Input: Labeled Data Goal: Predict Unlabeled Data

Graphical Model: Naive Bayes

Parameters Observed Variables (value known) Unobserved Variables (value unknown)

Graphical Model: Naive Bayes

Generative Model Graphical Model Joint Distribution Goal: Predict Labels

Prediction with Bayesian Posterior

Posterior on Label (Spam=1, Not Spam=0)

Marginal Likelihood Joint

Maximum Likelihood Estimation

argmax

µ,φ N

X

n=1

log p(yn, xn | µ,φ)

Objective

Interpretation: Fraction


• f Spam in training set

Interpretation: Fraction


• f non-spam (k=0) and

spam (k=1) messages that contain term. Optimum Parameters Training Data (Labeled)

Maximum Likelihood Estimation

argmax

µ,φ N

X

n=1

log p(yn, xn | µ,φ)

Objective Training Data (Labeled)

Optimum Parameters Problem: What do you do for words
 not found in training set?

X

n

I[xnd = 1] = 0

Maximum Likelihood Estimation

argmax

µ,φ N

X

n=1

log p(yn, xn | µ,φ)

Objective Training Data (Labeled)

Optimum Parameters Problem: What do you do for words
 not found in training set?

X

n

I[xnd = 1] = 0

µ = N y

1 +1

N+2 φkd = N x

kd+1

N y

k +D

(with Laplace smoothing)

Naive Bayes with Priors

Generative Model

Graphical Model Maximum A Posteriori (Offline)

Naive Bayes with Priors

Generative Model

Graphical Model Predictive Distribution (Online)

(same as using Laplace smoothing)

Exponential Families

Exponential Family Distributions

p(x | η) = h(x)exp ñX

i

ηiti(x) − a(η) ô

Base Measure Natural Parameters Sufficient Statistics Log Normalizer Product of terms: x-dependent, xη-dependent, η-dependent

Example: Discrete Distribution

p(x | η) = h(x)exp ñX

i

ηiti(x) − a(η) ô

Example: Gaussian Distribution

p(x | η) = h(x)exp ñX

i

ηiti(x) − a(η) ô

h(x) = 1 p 2π η = Å µ σ2 , 1 σ2 ã t(x) = (x, x2) a(η) = µ2 σ2 + logσ

Exponential Families

https://en.wikipedia.org/wiki/Exponential_family Most of the “normally” used distributions: Normal, Gamma, Dirichlet, Discrete, Poisson, Cauchy, etc.

Conjugate Priors

p(x | η) = h(x)exp ñX

i

ηiti(x) − a(η) ô

Likelihood (exponential family)

Conjugate Prior (also exponential family)

Joint Distribution x-dep, η-dep, ηxλ-dep, ηλ-dep, λ-dep

Conjugate Priors

p(x | η) = h(x)exp ñX

i

ηiti(x) − a(η) ô

Likelihood (exponential family)

Conjugate Prior (also exponential family)

Posterior Distribution (same family as prior)

Example: Dirichlet Distribution

(Conjugate to the Discrete Distribution)

Example: Dirichlet Distribution

Conjugate Priors

https://en.wikipedia.org/wiki/Conjugate_prior