Learning Bayesian Networks: Learning Bayesian Networks: Na ve and - - PowerPoint PPT Presentation

▶

Jun 14, 2023 181 likes •380 views

Learning Bayesian Networks: Learning Bayesian Networks: Na ve and non ve and non- -Na Na ve Bayes ve Bayes Na Hypothesis Space Hypothesis Space fixed size fixed size stochastic stochastic continuous

SLIDE 1

Learning Bayesian Networks: Learning Bayesian Networks: Na Naï ïve and non ve and non-

Naï ïve Bayes ve Bayes

Hypothesis Space Hypothesis Space

– – fixed size fixed size – – stochastic stochastic – – continuous parameters continuous parameters

Learning Algorithm Learning Algorithm

– – direct computation direct computation – – eager eager – – batch batch

SLIDE 2

Multivariate Gaussian Classifier Multivariate Gaussian Classifier

The multivariate Gaussian Classifier is The multivariate Gaussian Classifier is equivalent to a simple Bayesian network equivalent to a simple Bayesian network This models the joint distribution P( This models the joint distribution P(x x,y) under ,y) under the assumption that the class conditional the assumption that the class conditional distributions P( distributions P(x x|y) are multivariate gaussians |y) are multivariate gaussians

– – P(y): multinomial random variable (K P(y): multinomial random variable (K-

sided coin)

sided coin) – – P( P(x x|y): multivariate gaussian mean |y): multivariate gaussian mean µ µk

k covariance

covariance matrix matrix Σ Σk

y x

SLIDE 3

Na Naï ïve Bayes Model ve Bayes Model

Each node contains a probability table Each node contains a probability table

– – y y: P( : P(y y = = k k) ) – – x xj

j: P(

: P(x xj

j =

= v v | | y y = = k k) ) “ “class conditional probability class conditional probability” ”

Interpret as a generative model Interpret as a generative model

– – Choose the class Choose the class k k according to P( according to P(y y = = k k) ) – – Generate each feature Generate each feature independently independently according to P(x according to P(xj

=v v | | y y= =k k) ) – – The feature values are The feature values are conditionally independent conditionally independent P( P(x xi

,x xj

j |

| y y) = P( ) = P(x xi

i |

| y y) ) · · P( P(x xj

j |

| y y) )

y x3 x2 x1 xn

…

SLIDE 4

Representing P( Representing P(x xj

j|

|y y) )

Many representations are possible Many representations are possible

– – Univariate Gaussian Univariate Gaussian

if if x xj

j is a continuous random variable, then we can use a normal

is a continuous random variable, then we can use a normal distribution and learn the mean distribution and learn the mean µ µ and variance and variance σ σ2

– – Multinomial Multinomial

if if x xj

j is a discrete random variable,

is a discrete random variable, x xj

j ∈

∈ { {v v1

, … …, , v vm

m}, then we construct

}, then we construct the conditional probability table the conditional probability table

– – Discretization Discretization

convert continuous convert continuous x xj

j into a discrete variable

into a discrete variable

– – Kernel Density Estimates Kernel Density Estimates

apply a kind of nearest apply a kind of nearest-

neighbor algorithm to compute P(

neighbor algorithm to compute P(x xj

j |

| y y) in ) in neighborhood of query point neighborhood of query point

P(x P(xj

=v vm

m |

| y = y = K) K) … … P(x P(xj

=v vm

m |

| y = y = 2) 2) P(x P(xj

=v vm

m |

| y = y = 1) 1) x xj

=v vm

… … … … … … … … … … P(x P(xj

=v vm

m |

| y = y = K) K) … … P(x P(xj

=v v2

2 |

| y = y = 2) 2) P(x P(xj

=v v2

2 |

| y = y = 1) 1) x xj

=v v2

P(x P(xj

=v vm

m |

| y = y = K) K) … … P(x P(xj

=v v1

1 |

| y = y = 2) 2) P(x P(xj

=v v1

1 |

| y = y = 1) 1) x xj

=v v1

y y=K =K … … y y=2 =2 y y = 1 = 1

SLIDE 5

Discretization via Mutual Information Discretization via Mutual Information

Many discretization algorithms have been studied. One Many discretization algorithms have been studied. One

f the best is mutual information discretization
f the best is mutual information discretization

– – To discretize feature x To discretize feature xj

j, grow a decision tree considering only

, grow a decision tree considering only splits on splits on x xj

j. Each leaf of the resulting tree will correspond to a

. Each leaf of the resulting tree will correspond to a single value of the discretized single value of the discretized x xj

. – – Stopping rule (applied at each node). Stop when Stopping rule (applied at each node). Stop when – – where where S S is the training data in the parent node; is the training data in the parent node; S Sl

l and

and S Sr

r are the

are the examples in the left and right child. K, K examples in the left and right child. K, Kl

l, and K

, and Kr

r are the

are the corresponding number of classes present in these examples. corresponding number of classes present in these examples. I I is the mutual information, is the mutual information, H H is the entropy, and is the entropy, and N N is the number is the number

f examples in the node.
f examples in the node.

I(xj; y) < log2(N − 1) N + ∆ N ∆ = log2(3K−2)−[K·H(S)−Kl·H(Sl)−Kr·H(Sr)]

SLIDE 6

Kernel Density Estimators Kernel Density Estimators

Define to Define to be the Gaussian Kernel with parameter be the Gaussian Kernel with parameter σ σ Estimate Estimate where N where Nk

k is the number of training

is the number of training examples in class examples in class k k. .

P(xj|y = k) =

P {i|y=k} K(xj, xi,j)

Nk

K(xj, xi,j) = 1 √ 2πσ exp −

µxj − xi,j

σ

¶2

SLIDE 7

Kernel Density Estimators (2) Kernel Density Estimators (2)

This is equivalent to placing a Gaussian This is equivalent to placing a Gaussian “ “bump bump” ” of height 1/

f height 1/N

Nk

k on each trianing

n each trianing

data point from class data point from class k k and then adding and then adding them up them up

xj P(xj|y)

SLIDE 8

Kernel Density Estimators Kernel Density Estimators

Resulting probability density Resulting probability density

P(xj|y) xj

SLIDE 9

The value chosen for The value chosen for σ σ is critical is critical

σ=0.15 σ=0.50

SLIDE 10

Na Naï ïve Bayes learns a ve Bayes learns a Linear Threshold Unit Linear Threshold Unit

For multinomial and discretized attributes (but For multinomial and discretized attributes (but not not Gaussian), Na Gaussian), Naï ïve Bayes gives a linear ve Bayes gives a linear decision boundary decision boundary Define a discriminant function for class 1 versus Define a discriminant function for class 1 versus class K class K

P (x|Y = y) = P(x1 = v1|Y = y)·P(x2 = v2|Y = y) · · · P(xn = vn|Y = y)

SLIDE 11

Log of Odds Ratio Log of Odds Ratio

Suppose each xj is binary and define

SLIDE 12

Log Odds (2) Log Odds (2)

Now rewrite as Now rewrite as

log P(y = 1|x) P (y = K|x) =

X j

(αj,1 − αj,0)xj + αj,0 + log P(y = 1) P (y = K) log P(y = 1|x) P (y = K|x) =

X j

(αj,1 − αj,0)xj +

⎛ ⎝X j

αj,0 + log P(y = 1) P(y = K)

⎞ ⎠

We classify into class 1 if this is We classify into class 1 if this is ≥ ≥ 0 and 0 and into class K otherwise into class K otherwise

SLIDE 13

Learning the Probability Learning the Probability Distributions by Direct Computation Distributions by Direct Computation

P( P(y y= =k k) is just the fraction of training examples ) is just the fraction of training examples belonging to class belonging to class k. k. For multinomial variables, P( For multinomial variables, P(x xj

j =

= v v | | y y = = k k) is the ) is the fraction of training examples in class fraction of training examples in class k k where where x xj

= = v v For Gaussian variables, is the average For Gaussian variables, is the average value of value of x xj

j for training examples in class

for training examples in class k k. . is the sample standard deviation of those points: is the sample standard deviation of those points:

ˆ µjk

ˆ σjk =

v u u t 1

Nk

X {i|yi=k}

(xi,j − ˆ µjk)2

ˆ σjk

SLIDE 14

Improved Probability Estimates via Improved Probability Estimates via Laplace Corrections Laplace Corrections

When we have very little training data, direct probability compu When we have very little training data, direct probability computation tation can give probabilities of 0 or 1. Such extreme probabilities ar can give probabilities of 0 or 1. Such extreme probabilities are e “ “too too strong strong” ” and cause problems and cause problems Suppose we are estimate a probability P(z) and we have n Suppose we are estimate a probability P(z) and we have n0 examples where z is false and n examples where z is false and n1

1 examples where z is true. Our

examples where z is true. Our direct estimate is direct estimate is Laplace Estimate. Add 1 to the numerator and 2 to the denominat Laplace Estimate. Add 1 to the numerator and 2 to the denominator

This says that in the absence of any evidence, we expect P(z This says that in the absence of any evidence, we expect P(z) = 0.5, ) = 0.5, but our belief is weak (equivalent to 1 example for each outcome but our belief is weak (equivalent to 1 example for each outcome). ). Generalized Laplace Estimate. If z has K different outcomes, th Generalized Laplace Estimate. If z has K different outcomes, then en we estimate it as we estimate it as

P(z = 1) = n1 + 1 n0 + n1 + 2 P(z = 1) = n1 n0 + n1 P (z = 1) = n1 + 1 n0 + · · · + nK−1 + K

SLIDE 15

Na Naï ïve Bayes Applied to Diabetes Diagnosis ve Bayes Applied to Diabetes Diagnosis

Bayes nets and causality Bayes nets and causality

– – Bayes nets work best when arrows follow the direction of Bayes nets work best when arrows follow the direction of causality causality

two things with a common cause are likely to be conditionally two things with a common cause are likely to be conditionally independent given the cause; arrows in the causal direction capt independent given the cause; arrows in the causal direction capture ure this independence this independence

– – In a Na In a Naï ïve Bayes network, arrows are often ve Bayes network, arrows are often not not in the causal in the causal direction direction

diabetes does not cause pregnancies diabetes does not cause pregnancies diabetes does not cause age diabetes does not cause age

– – But some arrows are correct But some arrows are correct

diabetes does cause the level of blood insulin and blood glucose diabetes does cause the level of blood insulin and blood glucose

SLIDE 16

Non Non-

Naï ïve Bayes ve Bayes

Manually construct a graph in which Manually construct a graph in which all arcs are causal all arcs are causal Learning the probability tables is still Learning the probability tables is still

easy. For example, P(Mass | Age,
easy. For example, P(Mass | Age,

Preg) involves counting the number Preg) involves counting the number

f patients of a given age and
f patients of a given age and

number of pregnancies that have a number of pregnancies that have a given body mass given body mass Classification: Classification:

P (D = d|A, P, M, I, G) = P(I|D = d)P(G|I, D = d)P(D = d|A, M, P) P (I, G)

SLIDE 17

Evaluation of Na Evaluation of Naï ïve Bayes ve Bayes

yes yes some some yes yes some some no no no no yes yes no no no no SVM SVM yes yes yes yes yes yes some some yes yes disc disc disc disc yes yes yes yes NB NB no no no no some some no no no no no no yes yes some some no no NNbr NNbr yes yes no no yes yes no no yes yes some some yes yes no no no no Nets Nets no no yes yes no no some some yes yes yes yes yes yes yes yes yes yes Trees Trees yes yes yes yes yes yes no no yes yes no no no no yes yes no no LDA LDA yes yes yes yes Accurate Accurate yes yes yes yes Interpretable Interpretable yes yes yes yes Linear combinations Linear combinations no no no no Irrelevant inputs Irrelevant inputs yes yes yes yes Scalability Scalability no no no no Monotone transformations Monotone transformations yes yes no no Outliers Outliers no no no no Missing values Missing values no no no no Mixed data Mixed data Logistic Logistic LMS LMS Criterion Criterion

Naïve Bayes is very popular, particularly in natural language processing

and information retrieval where there are many features compared to the number of examples

In applications with lots of data, Naïve Bayes does not usually perform

as well as more sophisticated methods

SLIDE 18

Na Naï ïve Bayes Summary ve Bayes Summary

Advantages of Bayesian networks Advantages of Bayesian networks

– – Produces stochastic classifiers Produces stochastic classifiers

can be combined with utility functions to make optimal decisions can be combined with utility functions to make optimal decisions

– – Easy to incorporate causal knowledge Easy to incorporate causal knowledge

resulting probabilities are easy to interpret resulting probabilities are easy to interpret

– – Very simple learning algorithms Very simple learning algorithms

if all variables are observed in training data if all variables are observed in training data

Disadvantages of Bayesian networks Disadvantages of Bayesian networks

– – Fixed sized hypothesis space Fixed sized hypothesis space

may underfit or overfit the data may underfit or overfit the data may not contain may not contain any any good classifiers if prior knowledge is wrong good classifiers if prior knowledge is wrong

– – Harder to handle continuous features Harder to handle continuous features