Lecture 4 Jan-Willem van de Meent Homework 1 Counting with Spark - - PowerPoint PPT Presentation

Unsupervised Machine Learning 
 and Data Mining

DS 5230 / DS 4420 - Fall 2018

Lecture 4

Jan-Willem van de Meent

Homework 1

Counting with Spark

Probabilistic Prediction

Objectives in Unsupervised Learning

Model parameters (e.g. neural network weights) Unknown Variables Known Variables Quantity

• f Interest

Self-driving Cars Medical Diagnosis Pedestrian motion Will pedestrian cross road? Chance of accident Symptoms / Test results Condition of patient Treatment outcome Setting Model for pedestrian behavior Model for diseases / symptoms

Probability Spaces

Definition: A probability space (Ω, F, P) consists of

• A sample space Ω (i.e. the set of outcomes)
• A set of events F (i.e. the set possible sets)
• A probability measure P (maps events to probabilities)

P : F → R

Axioms of Probability

Conditional Probabilities

• Definition: Joint Probability

Events (i.e. sets of outcomes) Outcomes in both A and B

• Definition: Conditional Probability

Probability Density Functions

• Problem: If X is a continuous variable, then


P(X=x) is 0 for any outcome x

• Solution: Define a density function as a derivative

Event

Capital P for probability Small p for density Single Outcome

Association Rules

I(A, B) = s(A, B) s(A) × s(B) Lift = c(A − → B) s(B) ,

Interest Factor (or Lift) of a rule A → B: The Confidence of an association rule is the 
 probability of B={b} given A = {a1,…,ak}

Support, s(X − → Y ) = σ(X ∪ Y ) N ; ( Confidence, c(X − → Y ) = σ(X ∪ Y ) σ(X) .

Probability of an event A

Joint probability

Distributions over Baskets

Distribution/density over baskets
 x is a outcome, i.e. a basket

Question: Is Ω finite or infinite in size?

P(A, B)/P(A) = P(B | A)

Confidence

Lift

Independent vs Dependent Events

Joint probability of independent events

Lift for independent events

Question: What is the largest lift you can get?

Implicit Objective in Market Basket Analysis

Market Basket Analysis Items already in basket Items that customer might also buy Total profit for sale Setting Model for customer sales

Question: Are we estimating any parameters in this model? Unknown Variables Known Variables Quantity

• f Interest

Maximum Likelihood Estimation

Discrete/Categorical Distribution

Example: Loaded Dice

Equivalent Notation: Indicator variables

Question: If you perform 1000 rolls and get 200 outcomes x=6, then how would you estimate θ6?

Maximum Likelihood Estimation

Likelihood of N independent events:

pθ(xn) =

K

Y

k=1

θ

xn,k k

Maximum likelihood estimation

Problem: Express θ* in terms of {x1, …, xN} hint: solve for

Maximum Likelihood Estimation

Likelihood of N independent events:

pθ(xn) =

K

Y

k=1

θ

xn,k k

Maximum likelihood estimation

Nk =

N

X

n=1

xn,k

(known as cross-entropy loss in neural net libraries)

Lagrangian Optimization

Constrained Optimization Problem

Optimum

Lagrangian

Lagrangian for MLE

Nk =

N

X

n=1

xn,k

Solution Same as naive guess!

Maximum Likelihood Estimation

Likelihood of N independent events:

pθ(xn) =

K

Y

k=1

θ

xn,k k

Maximum likelihood estimation

k = Nk

N

Nk =

N

X

n=1

xn,k

Question: Suppose we perform 3 trials with outcomes x1=2, x2=6, x3=5. What is θ*4? Αnswer: θ*4 = 0

Maximum A Posteriori (MAP)

Problem: Maximum likelihood can overfit when data is limited One Solution: Place a prior over parameters

ML Objective (same as minimizing CE loss) Regularization

Intuition: Bayesian Posterior

Posterior Likelihood Prior Observed data (flip outcomes) Unknown variable (coin bias) Example: Biased Coin

θ = 0.5: No bias θ = 1.0: Always heads θ = 0.0: Always tails x = 1: Heads x = 0: Tails

Intuition: Bayesian Posterior

Posterior Likelihood Prior

Uninformative Prior

Intuition: Bayesian Posterior

Posterior Likelihood Prior

Posterior after 10 trials

Intuition: Bayesian Posterior

Posterior Likelihood Prior

Posterior after 30 trials

Intuition: Bayesian Posterior

Posterior Likelihood Prior

Posterior after 50 trials

Intuition: Bayesian Posterior

Posterior Likelihood Prior

Posterior after 50 trials

Intuition: Bayesian Posterior

Posterior Likelihood Prior

MAP estimate after 50 trials

Bern(x|µ) = µx(1 − µ)1−x E[x] = µ var[x] = µ(1 − µ) mode[x] =

• 1

if µ 0.5,

• therwise

H[ ] = ln (1 ) ln

Likelihood: Bernoulli

µ ∈ [0, 1] that ariable x ∈ {0, 1} by a single continuous

(a discrete distribution with outcomes x=0 and x=1)

Bern(x|µ) = µx(1 − µ)1−x

Likelihood: Bernoulli

(a discrete distribution with outcomes x=0 and x=1) Question: What is the likelihood of N trials? What is ML estimate for μ?

Prior: Beta Distribution

Beta(µ|a, b) = Γ(a + b) Γ(a)Γ(b)µa−1(1 − µ)b−1 E[µ] = a a + b var[µ] = ab (a + b)2(a + b + 1) mode[µ] = a − 1 a + b − 2.

(a distribution on values in the range 0 to 1)

Joint Density on Trials

Beta(µ|a, b) = Γ(a + b) Γ(a)Γ(b)µa−1(1 − µ)b−1

Bern(x|µ) = µx(1 − µ)1−x [ ] = Question: For N trials, what is the form the joint?

Posterior after Trials

Bern(x | µ) = µN1(1 − µ)N0

Beta(µ | a, b) = 1 B(a, b)µa−1(1 − µ)b−1

Properties of Conjugate Priors

Bern(x | µ) = µN1(1 − µ)N0

Likelihood Sufficient Statistics Conjugate Prior

Beta(µ | a, b) = 1 B(a, b)µa−1(1 − µ)b−1

Posterior

N1 =

N

X

n=1

xn, N0 = N − N1

Marginal Likelihood

MAP Estimate After Trials

p(µ | a, b, x) = Beta(µ | N1 + a, N2 + b)

Beta(µ|a, b) = Γ(a + b) Γ(a)Γ(b)µa−1(1 − µ)b−1 a

Question: After 1 trial with outcome x = 1, what is
 the ML estimate? If we use a prior Beta(μ | 2, 2) then what is the MAP estimate?

mode[µ] = a − 1 a + b − 2.

θ ML =

= N1 N = 1

= N1 + a − 1 N + a + b − 2 = 2 3

θ MAP =