Lecture 20 A Bayes Net defines a joint distribution P(X 1 X n ) over - - PowerPoint PPT Presentation

▶

Sep 09, 2023 321 likes •365 views

CS440/ECE448: Intro to Artificial Intelligence Bayes Nets Lecture 20 A Bayes Net defines a joint distribution P(X 1 X n ) over a set of random variables X 1 X n More on learning Using the chain rule, we can factor P(X 1

SLIDE 1

Lecture 20  More on learning graphical models

Prof. Julia Hockenmaier

juliahmr@illinois.edu

http://cs.illinois.edu/fa11/cs440
CS440/ECE448: Intro to Artificial Intelligence

Bayes Nets

A Bayes Net defines a joint distribution P(X1…Xn)

ver a set of random variables X1…Xn
Using the chain rule, we can factor P(X1…Xn) into

a product of n conditional distributions: P(X1…Xn) = !j P(Xi | X1…Xi-1).

A Bayes Net makes a number of (conditional)

independence assumptions: P(X1…Xn) =def !j P(Xi | Parents(Xi)⊆ {X1…Xi-1})

Learning Bayes Nets

Parameter estimation: Given some data D over a set of random variables X and a Bayes Net (with empty CPTs) estimate the parameters (= fill in the CPTs) of the Bayes Net.

Structure learning: Given some data D over a set
f random variables X, find a Bayes Net (define its

CPTs) and estimate its parameters. (This is much harder… we wonʼt deal with it here)

Bayes Rule
P(h): prior probability of hypothesis

P(h | D): posterior probability of hypothesis. P(D | h): likelihood of data, given hypothesis

Prior ∝ posterior × likelihood

CS440/ECE448: Intro AI

P(h | D)! P(D | h)P(h) P(h | D) = P(D | h)P(h) P(D)

SLIDE 2

Three kinds of estimation techniques

Bayes optimal: Marginalize out the hypotheses P(X | D) = !i P(X | hi )P(hi |D) MAP (maximum a posteriori): Pick the hypothesis with the highest posterior hMAP = argmaxh P(h|D) ML (maximum likelihood): Pick the hypothesis that assigns highest likelihood hML = argmaxh P(D|h)

CS440/ECE448: Intro AI

Maximum likelihood learning

Given data D, we want to find the parameters that maximize P(D | θ).

We have a data set with N candies.

c are cherry. l = (N-c), are lime. Parameter θ = probability of cherry

Maximum likelihood estimate: " = c/N
6

CS440/ECE448: Intro AI

A more complex model

Now the candy has two kinds of wrappers (red or green). The wrapper is chosen probabilistically, depending

n the flavor of the candy.

CS440/ECE448: Intro AI

flavor

cherry: !

wrapper

F P(red | F) cherry !1 lime !2

Out of N candies, c are cherry. rc are cherry with a red wrapper, rl are lime with a red wrapper

The likelihood of this data set:

P(d | ", "1, "2) = " c(1-")N-c "1 rc (1-"1)c-rc"2 rl (1-"1)(N-c)-rl

The log likelihood of this data set:

L(d | ", "1, "2) = [c log" + (N-c)log(1-")]

+[rc log"1 + (c-rc)log(1-"1)] +[lc log"2 + (N-c-lc)log(1-"2)]

The ML parameter estimates: " = c/N "1 = rc/c "2 = rl /(N-c)

CS440/ECE448: Intro AI

SLIDE 3

Medical diagnosis

Patients see a doctor and complain about a number of symptoms (headache, 100F fever, …).

What is the most likely disease di, given the

set of symptoms S the patient has?

CS440/ECE448: Intro AI

( )

S d P

i di

| max arg

The Naïve Bayes classifier

Assume the items in your data set have a number of attribute A1…An.

Each item also belongs to one of a number of

given classes C1…Ck.

Which attributes an item has depends on its

class.

If you only observe the attributes of an item,

can you predict the class?

CS440/ECE448: Intro AI

The Naïve Bayes classifier

CS440/ECE448: Intro AI

C A1 A2 An …

Naïve Bayes

Symptom1 (T/F) Disease (1,2,3,…) Symptom2 (T/F) Symptom3 (T/F)

• •

P(d1) P(d2) P(d3)

• •

P(s1|d1) P(s1|d2) P(s1|d3)

• •

P(s2|d1) P(s2|d2) P(s2|d3)

• •

P(s3|d1) P(s3|d2) P(s3|d3)

• •

SLIDE 4

Naïve Bayes

argmaxC P(C| A1…An) = = argmaxC P(A1…An | C) P(C) = argmaxC !j P(Aj | C) P(C) We need to estimate:

– the multinomial P( C ) – for each attribute Aj and class c P(Aj | c)

CS440/ECE448: Intro AI

Maximum likelihood estimation

If we have a set of training data where the class of each item is given:

– the multinomial P( C=c ) = freq(c)/N – for each attribute Aj and class c:  P(Aj = a| c) = freq(a, c)/freq( c ) where freq(c ) = the number of items in the training data that have class c freq(a, c ) = the number of items in the training data that have attribute a and class c.

CS440/ECE448: Intro AI

Lecture 20 More on learning graphical models

juliahmr@illinois.edu

Bayes Nets

A Bayes Net defines a joint distribution P(X1…Xn)

a product of n conditional distributions: P(X1…Xn) = !j P(Xi | X1…Xi-1).

independence assumptions: P(X1…Xn) =def !j P(Xi | Parents(Xi)⊆ {X1…Xi-1})

Parameter estimation: Given some data D over a set of random variables X and a Bayes Net (with empty CPTs) estimate the parameters (= fill in the CPTs) of the Bayes Net.

CPTs) and estimate its parameters. (This is much harder… we wonʼt deal with it here)

P(h | D): posterior probability of hypothesis. P(D | h): likelihood of data, given hypothesis

P(h | D)! P(D | h)P(h) P(h | D) = P(D | h)P(h) P(D)

Three kinds of estimation techniques

Bayes optimal: Marginalize out the hypotheses P(X | D) = !i P(X | hi )P(hi |D) MAP (maximum a posteriori): Pick the hypothesis with the highest posterior hMAP = argmaxh P(h|D) ML (maximum likelihood): Pick the hypothesis that assigns highest likelihood hML = argmaxh P(D|h)

Maximum likelihood learning

Given data D, we want to find the parameters that maximize P(D | θ).

c are cherry. l = (N-c), are lime. Parameter θ = probability of cherry

A more complex model

Now the candy has two kinds of wrappers (red or green). The wrapper is chosen probabilistically, depending

flavor

wrapper

Out of N candies, c are cherry. rc are cherry with a red wrapper, rl are lime with a red wrapper

P(d | ", "1, "2) = " c(1-")N-c "1 rc (1-"1)c-rc"2 rl (1-"1)(N-c)-rl

L(d | ", "1, "2) = [c log" + (N-c)log(1-")]

The ML parameter estimates: " = c/N "1 = rc/c "2 = rl /(N-c)

Medical diagnosis

Patients see a doctor and complain about a number of symptoms (headache, 100F fever, …).

set of symptoms S the patient has?

( )

S d P

i di

| max arg

The Naïve Bayes classifier

Assume the items in your data set have a number of attribute A1…An.

given classes C1…Ck.

class.

can you predict the class?

The Naïve Bayes classifier

C A1 A2 An …

Naïve Bayes

Naïve Bayes

argmaxC P(C| A1…An) = = argmaxC P(A1…An | C) P(C) = argmaxC !j P(Aj | C) P(C) We need to estimate:

– the multinomial P( C ) – for each attribute Aj and class c P(Aj | c)

Maximum likelihood estimation

If we have a set of training data where the class of each item is given:

– the multinomial P( C=c ) = freq(c)/N – for each attribute Aj and class c: P(Aj = a| c) = freq(a, c)/freq( c ) where freq(c ) = the number of items in the training data that have class c freq(a, c ) = the number of items in the training data that have attribute a and class c.

Lecture 20  More on learning graphical models

– the multinomial P( C=c ) = freq(c)/N – for each attribute Aj and class c:  P(Aj = a| c) = freq(a, c)/freq( c ) where freq(c ) = the number of items in the training data that have class c freq(a, c ) = the number of items in the training data that have attribute a and class c.