Nave Bayes Jia-Bin Huang Virginia Tech Spring 2019 ECE-5424G / - - PowerPoint PPT Presentation

Naïve Bayes

Jia-Bin Huang Virginia Tech

Spring 2019

ECE-5424G / CS-5824

Administrative

• HW 1 out today. Please start early!
• Office hours
• Chen: Wed 4pm-5pm
• Shih-Yang: Fri 3pm-4pm
• Location: Whittemore 266

Linear Regression

• Model representation

ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦1 + 𝜄2𝑦2 + ⋯ + 𝜄𝑜𝑦𝑜 = 𝜄⊤𝑦

• Cost function 𝐾 𝜄 =

1 2𝑛 σ𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

2

• Gradient descent for linear regression

Repeat until convergence {𝜄

𝑘 ≔ 𝜄 𝑘 − 𝛽 1 𝑛 σ𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗 𝑦𝑘

𝑗 }

• Features and polynomial regression

Can combine features; can use different functions to generate features (e.g., polynomial)

• Normal equation 𝜄 = (𝑌⊤𝑌)−1𝑌⊤𝑧

(𝑦0) Size in feet^2 (𝑦1) Number of bedrooms (𝑦2) Number of floors (𝑦3) Age of home (years) (𝑦4) Price ($) in 1000’s (y) 1 2104 5 1 45 460 1 1416 3 2 40 232 1 1534 3 2 30 315 1 852 2 1 36 178 … …

𝑧 = 460 232 315 178

𝜄 = (𝑌⊤𝑌)−1𝑌⊤𝑧

Slide credit: Andrew Ng

Least square solution

• 𝐾 𝜄 =

1 2𝑛 σ𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

2

=

1 2𝑛 σ𝑗=1 𝑛

𝜄⊤𝑦(𝑗) − 𝑧 𝑗

2

=

1 2𝑛 𝑌𝜄 − 𝑧 2 2

• 𝜖

𝜖𝜄 𝐾 𝜄 = 0

• 𝜄 = (𝑌⊤𝑌)−1𝑌⊤𝑧

Justification/interpretation 1

• Geometric interpretation

𝒀 = 1 ← 𝒚(1) → 1 ⋮ 1 ← 𝒚(2) → ⋮ ← 𝒚(𝑛) → = ↑ 𝒜𝟏 ↑ 𝒜𝟐 ↑ 𝒜𝟑 ⋯ ↑ 𝒜𝒐 ↓ ↓ ↓ ↓

• 𝒀𝜾: column space of 𝒀 or span({𝒜𝟏, 𝒜𝟐, ⋯ , 𝒜𝒐})
• Residual 𝒀𝜾 − 𝐳 is orthogonal to the column space of 𝒀
• 𝒀⊤ 𝒀𝜾 − 𝐳 = 0 → (𝒀⊤𝒀)𝜾 = 𝒀⊤𝒛

𝒛

column space of 𝒀

𝒀𝜾 𝒀𝜾 − 𝒛

Justification/interpretation 2

• Probabilistic model
• Assume linear model with Gaussian errors

𝑞𝜄 𝑧 𝑗 𝑦 𝑗 = 1 2𝜌𝜏2 exp(− 1 2𝜏2 (𝑧 𝑗 − 𝜄⊤𝑦 𝑗 )

• Solving maximum likelihood

argmin

𝜄

ෑ

𝑗=1 𝑛

𝑞𝜄 𝑧 𝑗 𝑦 𝑗 argmin

𝜄

log(ෑ

𝑗=1 𝑛

𝑞 𝑧 𝑗 𝑦 𝑗 ) = argmin

𝜄

1 2𝜏2 ෍

𝑗=1 𝑛 1

2 𝜄⊤𝑦 𝑗 − 𝑧 𝑗

2

Image credit: CS 446@UIUC

Justification/interpretation 3

• Loss minimization

𝐾 𝜄 = 1 2𝑛 ෍

𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

2 = 1

𝑛 ෍

𝑗=1 𝑛

𝑀𝑚𝑡 ℎ𝜄 𝑦 𝑗 , 𝑧 𝑗

• 𝑀𝑚𝑡 𝑧, ො

𝑧 =

1 2 𝑧 − ො

𝑧 2

2: Least squares loss

• Empirical Risk Minimization (ERM)

1 𝑛 ෍

𝑗=1 𝑛

𝑀𝑚𝑡 𝑧 𝑗 , ො 𝑧

𝑛 training examples, 𝑜 features

Gradient Descent

• Need to choose 𝛽
• Need many iterations
• Works well even when

𝑜 is large Normal Equation

• No need to choose 𝛽
• Don’t need to iterate
• Need to compute

(𝑌⊤𝑌)−1

• Slow if 𝑜 is very large

Slide credit: Andrew Ng

Things to remember

• Model representation

ℎ𝜄 𝑦 = 𝜄0 + 𝜄1𝑦1 + 𝜄2𝑦2 + ⋯ + 𝜄𝑜𝑦𝑜 = 𝜄⊤𝑦

• Cost function 𝐾 𝜄 =

1 2𝑛 σ𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗

2

• Gradient descent for linear regression

Repeat until convergence {𝜄

𝑘 ≔ 𝜄 𝑘 − 𝛽 1 𝑛 σ𝑗=1 𝑛

ℎ𝜄 𝑦 𝑗 − 𝑧 𝑗 𝑦𝑘

𝑗 }

• Features and polynomial regression

Can combine features; can use different functions to generate features (e.g., polynomial)

• Normal equation 𝜄 = (𝑌⊤𝑌)−1𝑌⊤𝑧

Today’s plan

• Probability basics
• Estimating parameters from data
• Maximum likelihood (ML)
• Maximum a posteriori estimation (MAP)
• Naïve Bayes

Today’s plan

• Probability basics
• Estimating parameters from data
• Maximum likelihood (ML)
• Maximum a posteriori estimation (MAP)
• Naive Bayes

Random variables

• Outcome space S
• Space of possible outcomes
• Random variables
• Functions that map outcomes to real numbers
• Event E
• Subset of S

Visualizing probability 𝑄(𝐵)

A is true

Sample space Area = 1

A is false

𝑄 𝐵 = Area of the blue circle

Visualizing probability 𝑄 𝐵 + P ~A

A is true A is false

𝑄 𝐵 + P ~A = 1

Visualizing probability 𝑄 𝐵

A^~B

𝑄 𝐵 = P(A, B) + P A, ~𝐶

B A^B

Visualizing conditional probability

A

𝑄 𝐵|𝐶 = 𝑄 𝐵, 𝐶 /𝑄(𝐶)

B A^B

Corollary: The chain rule

𝑄 𝐵, 𝐶 = 𝑄 𝐵|𝐶 𝑄(𝐶)

Bayes rule

A

𝑄 𝐵|𝐶 = 𝑄 𝐵, 𝐶 𝑄 𝐶 = 𝑄 𝐶 𝐵 𝑄 𝐵 𝑄(𝐶)

B A^B

Corollary: The chain rule

𝑄 𝐵, 𝐶 = 𝑄 𝐵|𝐶 𝑄 𝐶 = P B P A B

Thomas Bayes

Other forms of Bayes rule

𝑄 𝐵|𝐶 = 𝑄 𝐶 𝐵 𝑄 𝐵 𝑄(𝐶) 𝑄 𝐵|𝐶, 𝑌 = 𝑄 𝐶 𝐵, 𝑌 𝑄 𝐵, 𝑌 𝑄(𝐶, 𝑌) 𝑄 𝐵|𝐶 = 𝑄 𝐶 𝐵 𝑄 𝐵 𝑄 𝐶 𝐵 𝑄 𝐵 + 𝑄 𝐶 ~𝐵 𝑄(~𝐵)

Applying Bayes rule

𝑄 𝐵|𝐶 = 𝑄 𝐶 𝐵 𝑄 𝐵 𝑄 𝐶 𝐵 𝑄 𝐵 + 𝑄 𝐶 ~𝐵 𝑄(~𝐵)

• A = you have the flu

B = you just coughed

• Assume:
• 𝑄 𝐵 = 0.05
• 𝑄 𝐶 𝐵 = 0.8
• 𝑄 𝐶 ~𝐵 = 0.2
• What is P(flu | cough) = P(A|B)?

𝑄 𝐵|𝐶 = 0.8 × 0.05 0.8 × 0.05 + 0.2 × 0.95 ~0.17

Slide credit: Tom Mitchell

Why we are learning this? Learn 𝑄 𝑍|𝑌

ℎ 𝑦 𝑧

Hypothesis

Joint distribution

• Making a joint distribution of M variables
• 1. Make a truth table listing all combinations
• 2. For each combination of values, say how probable it is
• 3. Probability must sum to 1

A B C Prob 0.30 1 0.05 1 0.10 1 1 0.05 1 0.05 1 1 0.10 1 1 0.25 1 1 1 0.10

Slide credit: Tom Mitchell

Using joint distribution

• Can ask for any logical expression

involving these variables

• 𝑄 𝐹 = σrows matching E 𝑄(row)
• 𝑄 𝐹1|𝐹2 =

σrows matching E1 and 𝐹2 𝑄 row σrows matching 𝐹2 𝑄 row

Slide credit: Tom Mitchell

The solution to learn 𝑄 𝑍|𝑌 ?

• Main problem: learning 𝑄 𝑍|𝑌 may require more data than we have
• Say, learning a joint distribution with 100 attributes
• # of rows in this table?
• # of people on earth?

2100 ≥ 1030 109

Slide credit: Tom Mitchell

What should we do?

• 1. Be smart about

how we estimate probabilities from sparse data

• Maximum likelihood estimates (ML)
• Maximum a posteriori estimates (MAP)
• 2. Be smart about

how to represent joint distributions

• Bayes network, graphical models (more on this later)

Slide credit: Tom Mitchell

Today’s plan

• Probability basics
• Estimating parameters from data
• Maximum likelihood (ML)
• Maximum a posteriori (MAP)
• Naive Bayes

Estimating the probability

• Flip the coin repeatedly, observing
• It turns heads 𝛽1 times
• It turns tails 𝛽0 times
• Your estimate for 𝑄 𝑌 = 1 is?
• Case A: 100 flips: 51 Heads (𝑌 = 1), 49 Tails (𝑌 = 0)

𝑄 𝑌 = 1 = ?

• Case B: 3 flips: 2 Heads (𝑌 = 1), 1 Tails (𝑌 = 0)

𝑄 𝑌 = 1 = ?

𝑌 = 1 𝑌 =0

Slide credit: Tom Mitchell

Two principles for estimating parameters

• Maximum Likelihood Estimate (MLE)

Choose 𝜄 that maximizes probability of observed data

෡ 𝜾MLE = argmax

𝜄

𝑄(𝐸𝑏𝑢𝑏|𝜄)

• Maximum a posteriori estimation (MAP)

Choose 𝜄 that is most probable given prior probability and data

෡ 𝜾MAP = argmax

𝜄

𝑄 𝜄 𝐸 = argmax

𝜄

𝑄 𝐸𝑏𝑢𝑏 𝜄 𝑄 𝜄 𝑄(𝐸𝑏𝑢𝑏)

Slide credit: Tom Mitchell

Two principles for estimating parameters

• Maximum Likelihood Estimate (MLE)

Choose 𝜄 that maximizes 𝑄 𝐸𝑏𝑢𝑏 𝜄 ෡ 𝜾MLE = 𝛽1 𝛽1 + 𝛽0

• Maximum a posteriori estimation (MAP)

Choose 𝜄 that maximize 𝑄 𝜄 𝐸𝑏𝑢𝑏 ෡ 𝜾MAP = (𝛽1 + #halluciated 1s) (𝛽1+#halluciated 1𝑡) + (𝛽0 + #halluciated 0s)

Slide credit: Tom Mitchell

Maximum likelihood estimate

• Each flip yields Boolean value for 𝑌

𝑌 ∼ Bernoulli: 𝑄 𝑌 = 𝜄𝑌 1 − 𝜄 1−𝑌

• Data set 𝐸 of independent, identically distributed (iid)

flips, produces 𝛽1 ones, 𝛽0 zeros 𝑄 𝐸 𝜄 = 𝑄 𝛽1, 𝛽0 𝜄 = 𝜄𝛽1 1 − 𝜄 𝛽0 ෡ 𝜾 = argmax

𝜄

𝑄(𝐸|𝜄) = 𝛽1 𝛽1 + 𝛽0

𝑌 = 1 𝑌 =0 𝑄 𝑌 = 1 = 𝜄 𝑄 𝑌 = 0 = 1 − 𝜄

Slide credit: Tom Mitchell

Beta prior distribution 𝑄 𝜄

• 𝑄 𝜄 = 𝐶𝑓𝑢𝑏 𝛾1, 𝛾0 =

1 𝐶(𝛾1,𝛾0) 𝜄𝛾1−1 1 − 𝜄 𝛾0−1

Slide credit: Tom Mitchell

Maximum likelihood estimate

• Data set 𝐸 of iid flips,

produces 𝛽1 ones, 𝛽0 zeros 𝑄 𝐸 𝜄 = 𝑄 𝛽1, 𝛽0 𝜄 = 𝜄𝛽1 1 − 𝜄 𝛽0

• Assume prior (Conjugate prior: Closed form representation of posterior)

𝑄 𝜄 = 𝐶𝑓𝑢𝑏 𝛾1, 𝛾0 = 1 𝐶(𝛾1, 𝛾0) 𝜄𝛾1−1 1 − 𝜄 𝛾0−1 ෡ 𝜾 = argmax

𝜄

𝑄 𝐸 𝜄 P(𝜄) = 𝛽1 + 𝛾1 − 1 (𝛽1 + 𝛾1 − 1) + (𝛽0 + 𝛾0 − 1)

𝑌 = 1 𝑌 =0

Slide credit: Tom Mitchell

Some terminology

• Likelihood function 𝑄 𝐸𝑏𝑢𝑏 𝜄
• Prior 𝑄 𝜄
• Posterior 𝑄 𝜄 𝐸𝑏𝑢𝑏
• Conjugate prior:

Prior 𝑄 𝜄 is the conjugate prior for a likelihood function𝑄 𝐸𝑏𝑢𝑏 𝜄 if the prior 𝑄 𝜄 and the posterior 𝑄 𝜄 𝐸𝑏𝑢𝑏 have the same form.

• Example (coin flip problem)
• Prior 𝑄 𝜄 : 𝐶𝑓𝑢𝑏 𝛾1, 𝛾0

Likelihood 𝑄 𝐸𝑏𝑢𝑏 𝜄 : Binomial 𝜄𝛽1 1 − 𝜄 𝛽0

• Posterior 𝑄 𝜄 𝐸𝑏𝑢𝑏 : 𝐶𝑓𝑢𝑏 𝛽1 + 𝛾1, 𝛽0 + 𝛾0

Slide credit: Tom Mitchell

How many parameters?

• Suppose 𝑌 = [𝑌1, ⋯ , 𝑌𝑜], where

𝑌𝑗 and 𝑍 are Boolean random variables To estimate 𝑄 𝑍 𝑌1, ⋯ , 𝑌𝑜) When 𝑜 = 2 (Gender, Hours-worked)? When 𝑜 = 30?

Slide credit: Tom Mitchell

Can we reduce paras using Bayes rule?

𝑄 𝑍|𝑌 = 𝑄 𝑌 𝑍 𝑄 𝑍 𝑄(𝑌)

• How many parameters for 𝑄(𝑌1, ⋯ , 𝑌𝑜|𝑍)?

2𝑜 − 1 × 2

• How many parameters for 𝑄 𝑍 ?

1

Slide credit: Tom Mitchell

Today’s plan

• Probability basics
• Estimating parameters from data
• Maximum likelihood (ML)
• Maximum a posteriori estimation (MAP)
• Naive Bayes

Naïve Bayes

• Assumption:

𝑄 𝑌1, ⋯ , 𝑌𝑜 𝑍 = ෑ

𝑘=1 𝑜

𝑄(𝑌

𝑘|𝑍)

• i.e., 𝑌𝑗 and 𝑌

𝑘 are conditionally independent

given 𝑍 for 𝑗 ≠ 𝑘

Slide credit: Tom Mitchell

Conditional independence

• Definition: 𝑌 is conditionally independent of 𝑍 given 𝑎, if the

probability distribution governing 𝑌 is independent of the value of 𝑍, given the value of 𝑎 ∀𝑗, 𝑘, 𝑙 𝑄 𝑌 = 𝑦𝑗 𝑍 = 𝑧𝑘, 𝑎 = 𝑨𝑙) = 𝑄(𝑌 = 𝑦𝑗|𝑎𝑙) 𝑄 𝑌 𝑍, 𝑎 = 𝑄(𝑌|𝑎) Example: 𝑄 Thunder Rain, Lightning = 𝑄(Thunder|Lightning)

Slide credit: Tom Mitchell

Applying conditional independence

• Naïve Bayes assumes 𝑌𝑗 are conditionally independent given 𝑍

e.g., 𝑄 𝑌1 𝑌2, 𝑍 = 𝑄(𝑌1|𝑍) 𝑄 𝑌1, 𝑌2 𝑍 = 𝑄 𝑌1 𝑌2, 𝑍 𝑄 𝑌2 𝑍 = 𝑄 𝑌1 𝑍 𝑄(𝑌2|𝑍) General form: 𝑄 𝑌1, ⋯ , 𝑌𝑜 𝑍 = ς𝑘=1

𝑜

𝑄(𝑌

𝑘|𝑍)

How many parameters to describe 𝑄 𝑌1, ⋯ , 𝑌𝑜 𝑍 ? 𝑄(Y)?

• Without conditional indep assumption?
• With conditional indep assumption?

Slide credit: Tom Mitchell

Naïve Bayes classifier

• Bayes rule:

𝑄 𝑍 = 𝑧𝑙 𝑌1, ⋯ , 𝑌𝑜) = 𝑄(𝑍 = 𝑧𝑙)𝑄(𝑌1, ⋯ , 𝑌𝑜 𝑍 = 𝑧𝑙 σ𝑘 𝑄 𝑍 = 𝑧𝑘 𝑄 𝑌1, ⋯ , 𝑌𝑜 𝑍 = 𝑧𝑘

• Assume conditional independence among 𝑌𝑗’s:

𝑄 𝑍 = 𝑧𝑙 𝑌1, ⋯ , 𝑌𝑜) = 𝑄 𝑍 = 𝑧𝑙 Π𝑗𝑄 𝑌𝑗 𝑍 = 𝑧𝑙) σ𝑘 𝑄 𝑍 = 𝑧𝑘 Π𝑗𝑄 𝑌𝑗 𝑍 = 𝑧𝑘)

• Pick the most probable Y

෠ 𝑍 ← argmax

𝑧𝑙

𝑄 𝑍 = 𝑧𝑙 Π𝑗𝑄 𝑌𝑗 𝑍 = 𝑧𝑙)

Slide credit: Tom Mitchell

Naïve Bayes algorithm – discrete Xi

• For each value yk

Estimate 𝜌𝑙 = 𝑄(𝑍 = 𝑧𝑙) For each value xij of each attribute Xi

Estimate 𝜄𝑗𝑘𝑙 = 𝑄(𝑌𝑗 = 𝑦𝑗𝑘𝑙|𝑍 = 𝑧𝑙)

• Classify Xtest

෠ 𝑍 ← argmax

𝑧𝑙

𝑄 𝑍 = 𝑧𝑙 Π𝑗𝑄 𝑌𝑗

test 𝑍 = 𝑧𝑙)

෠ 𝑍 ← argmax

𝑧𝑙

𝜌𝑙 Π𝑗𝜄𝑗𝑘𝑙

Estimating parameters: discrete 𝑍, 𝑌𝑗

• Maximum likelihood estimates (MLE)

ො 𝜌𝑙 = ෠ 𝑄 𝑍 = 𝑧𝑙 = #𝐸 𝑍 = 𝑧𝑙 𝐸 መ 𝜄𝑗𝑘𝑙 = ෠ 𝑄 𝑌𝑗 = 𝑦𝑗𝑘 𝑍 = 𝑧𝑙 = #𝐸 𝑌𝑗 = 𝑦𝑗𝑘 ^ 𝑍 = 𝑧𝑙 #𝐸{𝑍 = 𝑧𝑙}

Slide credit: Tom Mitchell

• F = 1 iff you live in Fox Ridge
• S = 1 iff you watched the superbowl last night
• D = 1 iff you drive to VT
• G = 1 iff you went to gym in the last month

𝑄 𝐺 = 1 = 𝑄 𝑇 = 1|𝐺 = 1 = 𝑄 𝑇 = 1|𝐺 = 0 = 𝑄 𝐸 = 1|𝐺 = 1 = 𝑄 𝐸 = 1|𝐺 = 0 = 𝑄 𝐻 = 1|𝐺 = 1 = 𝑄 𝐻 = 1|𝐺 = 0 = 𝑄 𝐺 = 0 = 𝑄 𝑇 = 0|𝐺 = 1 = 𝑄 𝑇 = 0|𝐺 = 0 = 𝑄 𝐸 = 0|𝐺 = 1 = 𝑄 𝐸 = 0|𝐺 = 0 = 𝑄 𝐻 = 0|𝐺 = 1 = 𝑄 𝐻 = 0|𝐺 = 0 = 𝑄 𝐺|𝑇, 𝐸, 𝐻 = 𝑄 𝐺 P S F P D F P(G|F)

Naïve Bayes: Subtlety #1

• Often the 𝑌𝑗 are not really conditionally independent
• Naïve Bayes often works pretty well anyway
• Often the right classification, even when not the right probability [Domingos

& Pazzani, 1996])

• What is the effect on estimated P(Y|X)?
• What if we have two copies: 𝑌𝑗 = 𝑌𝑙

𝑄 𝑍 = 𝑧𝑙 𝑌1, ⋯ , 𝑌𝑜) ∝ 𝑄 𝑍 = 𝑧𝑙 Π𝑗𝑄 𝑌𝑗 𝑍 = 𝑧𝑙)

Slide credit: Tom Mitchell

Naïve Bayes: Subtlety #2

MLE estimate for 𝑄 𝑌𝑗 𝑍 = 𝑧𝑙) might be zero. (for example, 𝑌𝑗 = birthdate. 𝑌𝑗 = Feb_4_1995)

• Why worry about just one parameter out of many?

𝑄 𝑍 = 𝑧𝑙 𝑌1, ⋯ , 𝑌𝑜) ∝ 𝑄 𝑍 = 𝑧𝑙 Π𝑗𝑄 𝑌𝑗 𝑍 = 𝑧𝑙)

• What can we do to address this?
• MAP estimates (adding “imaginary” examples)

Slide credit: Tom Mitchell

Estimating parameters: discrete 𝑍, 𝑌𝑗

• Maximum likelihood estimates (MLE)

ො 𝜌𝑙 = ෠ 𝑄 𝑍 = 𝑧𝑙 = #𝐸 𝑍 = 𝑧𝑙 𝐸 መ 𝜄𝑗𝑘𝑙 = ෠ 𝑄 𝑌𝑗 = 𝑦𝑗𝑘 𝑍 = 𝑧𝑙 = #𝐸 𝑌𝑗 = 𝑦𝑗𝑘, 𝑍 = 𝑧𝑙 #𝐸{𝑍 = 𝑧𝑙}

• MAP estimates (Dirichlet priors):

ො 𝜌𝑙 = ෠ 𝑄 𝑍 = 𝑧𝑙 = #𝐸 𝑍 = 𝑧𝑙 + (𝛾𝑙−1) 𝐸 + σ𝑛(𝛾𝑛−1) መ 𝜄𝑗𝑘𝑙 = ෠ 𝑄 𝑌𝑗 = 𝑦𝑗𝑘 𝑍 = 𝑧𝑙 = #𝐸 𝑌𝑗 = 𝑦𝑗𝑘, 𝑍 = 𝑧𝑙 + (𝛾𝑙 −1) #𝐸{𝑍 = 𝑧𝑙} + σ𝑛(𝛾𝑛−1)

Slide credit: Tom Mitchell

What if we have continuous Xi

• Gaussian Naïve Bayes (GNB): assume

𝑄 𝑌𝑗 = 𝑦 𝑍 = 𝑧𝑙 = 1 2𝜌𝜏𝑗𝑙 exp(− 𝑦 − 𝜈𝑗𝑙 2𝜏𝑗𝑙

2 2

)

• Additional assumption on 𝜏𝑗𝑙:
• Is independent of 𝑍 (𝜏𝑗)
• Is independent of 𝑌𝑗 (𝜏𝑙)
• Is independent of 𝑌i and 𝑍 (𝜏𝑙)

Slide credit: Tom Mitchell

Naïve Bayes algorithm – continuous Xi

• For each value yk

Estimate 𝜌𝑙 = 𝑄(𝑍 = 𝑧𝑙) For each attribute Xi estimate Class conditional mean 𝜈𝑗𝑙, variance 𝜏𝑗𝑙

• Classify Xtest

෠ 𝑍 ← argmax

𝑧𝑙

𝑄 𝑍 = 𝑧𝑙 Π𝑗𝑄 𝑌𝑗

test 𝑍 = 𝑧𝑙)

෠ 𝑍 ← argmax

𝑧𝑙

𝜌𝑙 Π𝑗 𝑂𝑝𝑠𝑛𝑏𝑚(𝑌𝑗

test, 𝜈𝑗𝑙, 𝜏𝑗𝑙)

Slide credit: Tom Mitchell

Things to remember

• Probability basics
• Estimating parameters from data
• Maximum likelihood (ML) maximize 𝑄(Data|𝜄)
• Maximum a posteriori estimation (MAP) maximize 𝑄(𝜄|Data)
• Naive Bayes

𝑄 𝑍 = 𝑧𝑙 𝑌1, ⋯ , 𝑌𝑜) ∝ 𝑄 𝑍 = 𝑧𝑙 Π𝑗𝑄 𝑌𝑗 𝑍 = 𝑧𝑙)

Next class

• Logistic regression