Presentation of a Scientific Paper Naive Bayes Models for - - PowerPoint PPT Presentation

Motivation The Contribution Summary

Presentation of a Scientific Paper

Naive Bayes Models for Probability Estimation

Daniel Lowd and Pedro Domingos

Bertrand Dechoux

Aalborg University

Bertrand Dechoux Naive Bayes Estimation

Motivation The Contribution Summary Probability Estimation Bayesian Network

Probabilistic Inference Over a Set of Variables

One Solution : Bayesian Network visual representation of causality compact representation of the joint probability distribution Issues probabilistic inference is NP-hard approximate methods are not predictable enough focus : Naive Bayes Models

Bertrand Dechoux Naive Bayes Estimation

Motivation The Contribution Summary Probability Estimation Bayesian Network

Probabilistic Inference Over a Set of Variables

One Solution : Bayesian Network visual representation of causality compact representation of the joint probability distribution Issues probabilistic inference is NP-hard approximate methods are not predictable enough focus : Naive Bayes Models

Bertrand Dechoux Naive Bayes Estimation

Motivation The Contribution Summary Probability Estimation Bayesian Network

Probabilistic Inference Over a Set of Variables

One Solution : Bayesian Network visual representation of causality compact representation of the joint probability distribution Issues probabilistic inference is NP-hard approximate methods are not predictable enough focus : Naive Bayes Models

Bertrand Dechoux Naive Bayes Estimation

Motivation The Contribution Summary Probability Estimation Bayesian Network

Generalities

DAG(nodes, edges, conditional probabilities) chain rule P(X1, . . . , Xn) =

• Xi

P(Xi|pa(Xi)) learning : structure and parameters

Bertrand Dechoux Naive Bayes Estimation

Motivation The Contribution Summary Probability Estimation Bayesian Network

Naive Bayes Models

a simple (equivalent?) BN inference P(x1, x2) =

C

• P(c)P(x1|c)P(x2|c)

Z

• Z
• P(z|c)

inference : O(k|Xq|) learning : state space of C and parameters

Bertrand Dechoux Naive Bayes Estimation

Motivation The Contribution Summary Probability Estimation Bayesian Network

Naive Bayes Models

a simple (equivalent?) BN inference P(x1, x2) =

C

• P(c)P(x1|c)P(x2|c)

Z

• Z
• P(z|c)

inference : O(k|Xq|) learning : state space of C and parameters

Bertrand Dechoux Naive Bayes Estimation

Motivation The Contribution Summary the NBE algorithm benchmark

Adding New Components

Prior P(C) is updated ∼ uniformly previous P(C) → updated P(C)

Bertrand Dechoux Naive Bayes Estimation

Motivation The Contribution Summary the NBE algorithm benchmark

Adding New Components

Posterior d is a case in your database and introduces a bias P(Xi|c) ∝ ˜ P(Xi = xi|d) + λ × ˜ P(Xi)

Bertrand Dechoux Naive Bayes Estimation

Motivation The Contribution Summary the NBE algorithm benchmark

Pruning Existing Components

keep only components responsible for 99,9% of P(C) P(X = x) = k

c=1 P(c) |X| i=1 P(xi|c)

i.e. remove low-weight components

Bertrand Dechoux Naive Bayes Estimation

Motivation The Contribution Summary the NBE algorithm benchmark

A wrapper around Expectation-Maximisation

input : P(C), P(Xi|C) convergence : optimize L(M|D) up to a threshold σ

• uput : local optimum for P(C), P(Xi|C)

Bertrand Dechoux Naive Bayes Estimation

Motivation The Contribution Summary the NBE algorithm benchmark

input training set T and hold-out set H Initialize M with one component while L(M|H) improves

add k new components to M while L(M|H) improves

EM step if L(M|H) improves Mbest ← M every 5 cycles, prune components of M

M ← Mbest prune components of M k ← 2k

two EM steps on Mbest using H and T

• uput Mbest

Bertrand Dechoux Naive Bayes Estimation

Motivation The Contribution Summary the NBE algorithm benchmark

input training set T and hold-out set H Initialize M with one component while L(M|H) improves

add k new components to M while L(M|H) improves

EM step if L(M|H) improves Mbest ← M every 5 cycles, prune components of M

M ← Mbest prune components of M k ← 2k

two EM steps on Mbest using H and T

• uput Mbest

Bertrand Dechoux Naive Bayes Estimation

Motivation The Contribution Summary the NBE algorithm benchmark

benchmarks

Dataset 47 from the UCI repository 2 for for the collaborative filtering Compared Algorithms BNE, the proposed one WinMine, doing structure search a baseline : a naive bayes model with one component

Bertrand Dechoux Naive Bayes Estimation

Motivation The Contribution Summary the NBE algorithm benchmark

results

learning time equivalent

EM (BNE) versus structure search (WinMine)

modeling accuracy equivalent

for a random Bayesian Network, an ’equivalent’ Naive Bayes Network can be found

query speed and accuracy not equivalent

BNE is most of the time at least as accurate as Gibbs Sampling and Belief Propagation, and definitively quicker

Bertrand Dechoux Naive Bayes Estimation

Motivation The Contribution Summary the NBE algorithm benchmark

results

learning time equivalent

EM (BNE) versus structure search (WinMine)

modeling accuracy equivalent

for a random Bayesian Network, an ’equivalent’ Naive Bayes Network can be found

query speed and accuracy not equivalent

BNE is most of the time at least as accurate as Gibbs Sampling and Belief Propagation, and definitively quicker

Bertrand Dechoux Naive Bayes Estimation

Motivation The Contribution Summary

Summary

Naive Bayes Network are -arguably- better than random Bayesian Networks for probability estimation BNE is an algorithm to find such Naive Bayes Networks

• pen discussion : knowledge extraction

Open issues

theory i.e. the proof is empirical only the influence of hidden variables the importance of used memory space

Bertrand Dechoux Naive Bayes Estimation

Motivation The Contribution Summary

Summary

Naive Bayes Network are -arguably- better than random Bayesian Networks for probability estimation BNE is an algorithm to find such Naive Bayes Networks

• pen discussion : knowledge extraction

Open issues

theory i.e. the proof is empirical only the influence of hidden variables the importance of used memory space

Bertrand Dechoux Naive Bayes Estimation