Bayesian Networks George Konidaris gdk@cs.brown.edu Fall 2019 - - PowerPoint PPT Presentation

▶
bayesian networks
SMART_READER_LITE
LIVE PREVIEW

Bayesian Networks George Konidaris gdk@cs.brown.edu Fall 2019 - - PowerPoint PPT Presentation

Jun 21, 2023 486 likes •882 views

Bayesian Networks George Konidaris gdk@cs.brown.edu Fall 2019 Recall Joint distributions: P(X 1 , , X n ). All you (statistically) need to know about X 1 X n . From it you can infer P(X 1 ), P(X 1 | Xs), etc. Raining Cold

slide-1
SLIDE 1

Bayesian Networks

George Konidaris gdk@cs.brown.edu

Fall 2019

slide-2
SLIDE 2

Recall

Joint distributions:

  • P(X1, …, Xn).
  • All you (statistically) need to know about X1 … Xn.
  • From it you can infer P(X1), P(X1 | Xs), etc.

Raining Cold Prob. True True 0.3 True False 0.1 False True 0.4 False False 0.2

slide-3
SLIDE 3

Joint Distributions Are Useful

Classification

  • P(X1 | X2 … Xn)

Co-occurrence

  • P(Xa, Xb)

Rare event detection

  • P(X1, …, Xn)

thing you want to know things you know how likely are these two things together?

slide-4
SLIDE 4

Independence

If independent, can break JPD into separate tables.

Raining Prob. True 0.6 False 0.4 Cold Prob. True 0.75 False 0.25

Raining Cold Prob. True True 0.45 True False 0.15 False True 0.3 False False 0.1

X

P(A, B) = P(A)P(B)

slide-5
SLIDE 5

Conditional Independence

A and B are conditionally independent given C if:

  • P(A | B, C) = P(A | C)
  • P(A, B | C) = P(A | C) P(B | C)

(recall independence: P(A, B) = P(A)P(B)) This means that, if we know C, we can treat A and B as if they were independent. A and B might not be independent otherwise!

slide-6
SLIDE 6

Example

Consider 3 RVs:

  • Temperature
  • Humidity
  • Season

Temperature and humidity are not independent. But, they might be, given the season: the season explains both, and they become independent of each other.

slide-7
SLIDE 7

Bayes Nets

A particular type of graphical model:

  • A directed, acyclic graph.
  • A node for each RV.

Given parents, each RV independent of non- descendants.

T H S

slide-8
SLIDE 8

Bayes Net

JPD decomposes: So for each node, store conditional probability table (CPT):

P(x1, ..., xn) = Y

i

P(xi|parents(xi))

T H S

P(xi|parents(xi))

slide-9
SLIDE 9

CPTs

Conditional Probability Table

  • Probability distribution over variable given parents.
  • One distribution per setting of parents.

X Y Z P True True True 0.7 False True True 0.3 True True False 0.2 False True False 0.8 True False True 0.5 False False True 0.5 True False False 0.4 False False False 0.6

variable of interest conditioning variables distributions (sum to 1)

slide-10
SLIDE 10

Example

Suppose we know:

  • The flu causes sinus inflammation.
  • Allergies cause sinus inflammation.
  • Sinus inflammation causes a runny nose.
  • Sinus inflammation causes headaches.
slide-11
SLIDE 11

Example

Sinus Flu Allergy Nose Headache

slide-12
SLIDE 12

Example

Sinus Flu Allergy Nose Headache

Flu P True 0.6 False 0.4

Allergy P True 0.2 False 0.8

Sinus Flu Allergy P True True True 0.9 False True True 0.1 True True False 0.6 False True False 0.4 True False False 0.2 False False False 0.8 True False True 0.4 False False True 0.6

joint: 32 (31) entries

Nose Sinus P True True 0.8 False True 0.2 True False 0.3 False False 0.7

Headache Sinus P True True 0.6 False True 0.4 True False 0.5 False False 0.5 Headache Sinus P

slide-13
SLIDE 13

Uses

Things you can do with a Bayes Net:

  • Inference: given some variables, posterior?
  • (might be intractable: NP-hard)
  • Learning (fill in CPTs)
  • Structure Learning (fill in edges)

Generally:

  • Often few parents.
  • Inference cost often reasonable.
  • Can include domain knowledge.
slide-14
SLIDE 14

Inference

Sinus Flu Allergy Nose Headache

What is: P(f | h)?

slide-15
SLIDE 15

Inference

Given A compute P(B | A).

Sinus Flu Allergy Nose Headache

slide-16
SLIDE 16

Inference

Sinus Flu Allergy Nose Headache

What is: P(F=True | H=True)?

slide-17
SLIDE 17

Inference

P(f|h) = P(f, h) P(h) = P

SAN P(f, h, S, A, N)

P

SANF P(h, S, A, N, F)

identity

P(a) = X

B=T,F

P(a, B) P(a) = X

B=T,F

X

C=T,F

P(a, B, C)

slide-18
SLIDE 18

Inference

We know from definition of Bayes net: P(f|h) = P(f, h) P(h) = P

SAN P(f, h, S, A, N)

P

SANF P(h, S, A, N, F)

P(h) = X

SANF

P(h, S, A, N, F) P(h) = X

SANF

P(h|S)P(N|S)P(S|A, F)P(F)P(A)

slide-19
SLIDE 19

Variable Elimination

So we have: … we can eliminate variables one at a time: (distributive law) P(h) = X

SANF

P(h|S)P(N|S)P(S|A, F)P(F)P(A) P(h) = X

SN

P(h|S)P(N|S) X

AF

P(S|A, F)P(F)P(A) P(h) = X

S

P(h|S) X

N

P(N|S) X

AF

P(S|A, F)P(F)P(A)

slide-20
SLIDE 20

Variable Elimination

P(h) = X

S

P(h|S) X

N

P(N|S) X

AF

P(S|A, F)P(F)P(A)

0.6 × X

N

P(N|S = True) X

AF

P(S = True|A, F)P(F)P(A)+ 0.5 × X

N

P(N|S = False) X

AF

P(S = False|A, F)P(F)P(A)

sinus = true sinus = false

Headache Sinus P True True 0.6 False True 0.4 True False 0.5 False False 0.5 Headache Sinus P True True 0.6

slide-21
SLIDE 21

Variable Elimination

P(h) = X

S

P(h|S) X

N

P(N|S) X

AF

P(S|A, F)P(F)P(A) 0.6×[0.8 × X

AF

P(S = True|A, F)P(F)P(A)+ 0.2 × X

AF

P(S = True|A, F)P(F)P(A)]+ 0.5×[0.3 × X

AF

P(S = False|A, F)P(F)P(A)+ 0.7 × X

AF

P(S = False|A, F)P(F)P(A)]

Nose Sinus P True True 0.8 False True 0.2 True False 0.3 False False 0.7 Nose Sinus P True True 0.8

slide-22
SLIDE 22

Variable Elimination

Downsides:

  • How to simplify? (Hard in general.)
  • Computational complexity
  • Hard to parallelize
slide-23
SLIDE 23

Alternative

Sampling approaches

  • Based on drawing random numbers
  • Computationally expensive, but easy to code!
  • Easy to parallelize
slide-24
SLIDE 24

Sampling

What’s a sample? From a distribution:

x x x x x x x x x x x x x xx x x x

From a CPT:

Flu P True 0.6 False 0.4

F=True F=True F=False F=False F=True

0.6 1

slide-25
SLIDE 25

Generative Models

How do we sample from a Bayes Net? A Bayes Net is known as a generative model. Describe a generative process for the data.

  • Each variable is generated by a distribution.
  • Describes the structure of that generation.
  • Can generate more data.

Natural way to include domain knowledge via causality.

slide-26
SLIDE 26

Sampling the Joint

Algorithm for generating samples drawn from the joint distribution: For each node with no parents:

  • Draw sample from marginal distribution.
  • Condition children on choice (removes edge)
  • Repeat.

Results in artificial data set. Probability values - literally just count.

slide-27
SLIDE 27

Generative Models

Sinus Flu Allergy Nose Headache

Flu P True 0.6 False 0.4

Allergy P True 0.2 False 0.8

Sinus Flu Allergy P True True True 0.9 False True True 0.1 True True False 0.6 False True False 0.4 True False False 0.2 False False False 0.8 True False True 0.4 False False True 0.6

Nose Sinus P True True 0.8 False True 0.2 True False 0.3 False False 0.7 Nose Sinus P True True 0.8

Headache Sinus P True True 0.6 False True 0.4 True False 0.5 False False 0.5 Headache Sinus P True True 0.6

slide-28
SLIDE 28

Generative Models

Sinus Allergy Nose Headache

Allergy P True 0.2 False 0.8

Sinus Flu Allergy P True True True 0.9 False True True 0.1 True True False 0.6 False True False 0.4 True False False 0.2 False False False 0.8 True False True 0.4 False False True 0.6

Flue = True

Nose Sinus P True True 0.8 False True 0.2 True False 0.3 False False 0.7 Nose Sinus P True True 0.8

Headache Sinus P True True 0.6 False True 0.4 True False 0.5 False False 0.5 Headache Sinus P True True 0.6

slide-29
SLIDE 29

Generative Models

Sinus Nose Headache

Sinus Flu Allergy P True True True 0.9 False True True 0.1 True True False 0.6 False True False 0.4 True False False 0.2 False False False 0.8 True False True 0.4 False False True 0.6

Flue = True Allergy = False

Nose Sinus P True True 0.8 False True 0.2 True False 0.3 False False 0.7 Nose Sinus P True True 0.8

Headache Sinus P True True 0.6 False True 0.4 True False 0.5 False False 0.5 Headache Sinus P True True 0.6

slide-30
SLIDE 30

Generative Models

Nose Headache

Flue = True Allergy = False Sinus = True Nose = True Headache = False

Nose Sinus P True True 0.8 False True 0.2 True False 0.3 False False 0.7 Nose Sinus P True True 0.8

Headache Sinus P True True 0.6 False True 0.4 True False 0.5 False False 0.5 Headache Sinus P True True 0.6

slide-31
SLIDE 31

Sampling the Conditional

What if we want to know P(A | B)? We could use the previous procedure, and just divide the data up based on B. What if we want P(A | b)?

  • Could do the same, just use data with B=b.
  • Throw away the rest of the data.
  • Rejection sampling.
slide-32
SLIDE 32

Sampling the Conditional

What if b is uncommon? What if b involves many variables? Importance sampling:

  • Bias the sampling process to get more “hits”.
  • New distribution, Q.
  • Use a reweighing trick to unbias probabilities.
  • Multiply by P/Q to get probability of sample.
slide-33
SLIDE 33

Sampling

Properties of sampling:

  • Slow.
  • Always works.
  • Always applicable.
  • Easy to parallelize.
  • Computers are getting faster.
slide-34
SLIDE 34

Independence

What does this look like with a Bayes Net?

Raining Prob. True 0.6 False 0.4 Cold Prob. True 0.75 False 0.25 Raining Cold Prob. True True 0.45 True False 0.15 False True 0.3 False False 0.1

X

Raining Cold

slide-35
SLIDE 35

Naive Bayes

S W1 W2 W3 Wn

…

P(S) P(W1|S) P(W2|S) P(W3|S) P(Wn|S)

slide-36
SLIDE 36

Spam Filter (Naive Bayes)

S W1 W2 W3 Wn

…

P(S) P(W1|S) P(W2|S) P(W3|S) P(Wn|S) Want P(S | W1 … Wn)

slide-37
SLIDE 37

Naive Bayes

P(S|W1, ..., Wn) = P(W1, ..., Wn|S)P(S) P(W1, ..., Wn)

given

P(W1, ..., Wn|S) = Y

i

P(Wi|S)

(from the Bayes Net)

slide-38
SLIDE 38

Bayes Nets

Bayes Nets are a type of representation. Multiple inference algorithms; you can choose!

  • AI researchers talk about models more than

algorithms. Potentially very compressed but exact.

  • Requires careful construction!

VS Approximate representation.

  • Hope you’re not too wrong!

Many, many applications in all areas.