Probabilistic Graphical Models Probabilistic Graphical Models - - PowerPoint PPT Presentation

Probabilistic Graphical Models Probabilistic Graphical Models

introduction to learning

Siamak Ravanbakhsh Fall 2019

Learning objectives Learning objectives

different goals of learning a graphical model effect of goals on the learning setup

Where does a graphical model come from? Where does a graphical model come from?

image: http://blog.londolozi.com/

Where does a graphical model come from? Where does a graphical model come from?

designed by domain experts: more suitable for directed models

• cond. probabilities are more intuitive than unnormalized factors

no need to estimate the partition function

image: http://blog.londolozi.com/

Where does a graphical model come from? Where does a graphical model come from?

designed by domain experts: more suitable for directed models

• cond. probabilities are more intuitive than unnormalized factors

no need to estimate the partition function

image: http://blog.londolozi.com/

learning from data: fixed structure: easy for directed models unknown structure fully or partially observed data, hidden variables

Goals of learning: Goals of learning: density estimation density estimation

assumption: data is IID sample from a P ∗

D = {X , … , X } X ∼

(1) (M) (m)

P ∗

empirical distribution:

P

D

∣D∣ 1

D)

Goals of learning: Goals of learning: density estimation density estimation

assumption: data is IID sample from a P ∗

D = {X , … , X } X ∼

(1) (M) (m)

P ∗

• bjective: learn a close to P ∗

∈ P ^ P

= P ^ arg min

P KL ∗ empirical distribution:

P

D

∣D∣ 1

D)

Goals of learning: Goals of learning: density estimation density estimation

assumption: data is IID sample from a P ∗

D = {X , … , X } X ∼

(1) (M) (m)

P ∗

• bjective: learn a close to P ∗

∈ P ^ P

= P ^ arg min

P KL ∗

= E

P ∗ ∗

E

P ∗ empirical distribution:

P

D

∣D∣ 1

D)

Goals of learning: Goals of learning: density estimation density estimation

assumption: data is IID sample from a P ∗

D = {X , … , X } X ∼

(1) (M) (m)

P ∗

• bjective: learn a close to P ∗

∈ P ^ P

= P ^ arg min

P KL ∗

= E

P ∗ ∗

E

P ∗

negative Entropy of P* (does not depend on P)

empirical distribution:

P

D

∣D∣ 1

D)

Goals of learning: Goals of learning: density estimation density estimation

assumption: data is IID sample from a P ∗

D = {X , … , X } X ∼

(1) (M) (m)

P ∗

• bjective: learn a close to P ∗

∈ P ^ P

= P ^ arg min

P KL ∗

= E

P ∗ ∗

E

P ∗

negative Entropy of P* (does not depend on P)

substitute with : P ∗

empirical distribution:

P

D

∣D∣ 1

D)

P

D

= P ^ arg max

P ∑x∈D log-likelihood

how to compare two log-likelihood values?

its negative is called the log loss

Goals of learning: Goals of learning: prediction prediction

given D = {(X

, Y )}

(m) (m)

the output in our prediction is structured

interested in learning

(X ∣ P ^ Y )

e.g. in image segmentation

making prediction:

(Y ) = X ^ arg max (x ∣

x P

^ Y )

Goals of learning: Goals of learning: prediction prediction

given D = {(X

, Y )}

(m) (m)

the output in our prediction is structured

interested in learning

(X ∣ P ^ Y )

e.g. in image segmentation

error measures:

0/1 loss (unforgiving): Hamming loss: conditional log-likelihood:

takes prediction uncertainty into account

making prediction:

(Y ) = X ^ arg max (x ∣

x P

^ Y )

E

(X,Y )∼P ∗

(Y )) X ^ E

(X,Y )∼P ∗ ∑i i

(Y )

X ^

i

E

(X ∣

(X,Y )∼P ∗

P ^ Y )

Goals of learning: Goals of learning: knowledge discovery knowledge discovery

given D = {(X

)}

(m)

finding conditional independencies or causal relationships

interested in learning G or H

E.g. in gene regulatory network

image credit: Chen et al., 2014

Goals of learning: Goals of learning: knowledge discovery knowledge discovery

given D = {(X

)}

(m)

finding conditional independencies or causal relationships same undirected skeleton same immoralities

interested in learning G or H not always uniquely identifiable

two DAGs are I-equivalent if

I(G) = I(G )

′ E.g. in gene regulatory network

image credit: Chen et al., 2014

Recall

bias-variance trade-off bias-variance trade-off

learning ideally minimizes some risk (expected loss) in reality we use empirical risk

image: http://ipython-books.github.io

E

X∼P ∗

E

x∈D

if our model is expressive we can overfit

low empirical risk does not translate to low risk

• ur model does not generalize to samples outside

different choices of produce very different models

bias-variance trade-off bias-variance trade-off

learning ideally minimizes some risk (expected loss) in reality we use empirical risk

image: http://ipython-books.github.io

E

X∼P ∗

E

x∈D

• verfitting in density estimation

D D ∼ P ∗ P ^

high variance as measured by a validation set

if our model is expressive we can overfit

low empirical risk does not translate to low risk

• ur model does not generalize to samples outside

different choices of produce very different models

bias-variance trade-off bias-variance trade-off

learning ideally minimizes some risk (expected loss) in reality we use empirical risk

image: http://ipython-books.github.io

E

X∼P ∗

E

x∈D

• verfitting in density estimation

simple models cannot fit the data

the model has a bias, and even large dataset cannot help D D ∼ P ∗ P ^

high variance

D

high bias as measured by a validation set

if our model is expressive we can overfit

low empirical risk does not translate to low risk

• ur model does not generalize to samples outside

different choices of produce very different models

bias-variance trade-off bias-variance trade-off

learning ideally minimizes some risk (expected loss) in reality we use empirical risk

image: http://ipython-books.github.io

E

X∼P ∗

E

x∈D

• verfitting in density estimation

simple models cannot fit the data

the model has a bias, and even large dataset cannot help D D ∼ P ∗ P ^

high variance

D

high bias as measured by a validation set

a solution: penalize model complexity

regularization

Discreminative vs generative Discreminative vs generative

if the goal is prediction: Generative: learn and condition on Y (e.g., MRF) Discriminative: directly learn (e.g., CRF)

(X ∣ P ^ Y ) (X, Y ) P ^ (X ∣ P ^ Y )

training

Discreminative vs generative Discreminative vs generative

if the goal is prediction: Generative: learn and condition on Y (e.g., MRF) Discriminative: directly learn (e.g., CRF)

(X ∣ P ^ Y ) (X, Y ) P ^ (X ∣ P ^ Y )

Example

training

X Y

naive Bayes vs logistic regression

Naive Bayes P(X ∣ Y ) ∝ P(X)P(Y ∣ X)

trained generatively (log-likelihood) works better on small datasets (higher bias) unnecessary cond. ind. assumptions about Y can deal with missing values & learn from unlabeled data

Discreminative vs generative Discreminative vs generative

if the goal is prediction: Generative: learn and condition on Y (e.g., MRF) Discriminative: directly learn (e.g., CRF)

(X ∣ P ^ Y ) (X, Y ) P ^ (X ∣ P ^ Y )

Example

training

X X Y Y

naive Bayes vs logistic regression

Naive Bayes logistic regression P(X = 1∣Y ) = σ(W Y +

T

b) P(X ∣ Y ) ∝ P(X)P(Y ∣ X)

trained generatively (log-likelihood) works better on small datasets (higher bias) unnecessary cond. ind. assumptions about Y can deal with missing values & learn from unlabeled data trained discriminatively (cond. log-likelihood) works better on large datasets no assumptions about cond. ind. in Y

Discreminative vs generative Discreminative vs generative

Example

training

naive Bayes vs logistic regression on UCI dataset

naive Bayes logistic regression from: Ng & Jordan 2001

summary summary

learning can have different objectives: density estimation

calculating P(x) sampling from P (generative modeling)

prediction (conditional density estimation)

discriminative and generative modeling

knowledge discovery expressed as empirical risk minimization

bias-variance trade-off regularize the model