Outline Latent Variable Generative Models Cooperative Vector - - PDF document

SLIDE 1

2/4/2020 1

CS 3750 Advanced Machine Learning

Latent Variable Generative Models II

Ahmad Diab AHD23@cs.pitt.edu Feb 4, 2020

Based on slides of Professor Milos Hauskrecht

Outline

• Latent Variable Generative Models
• Cooperative Vector Quantizer Model
• Model Formulation
• Expectation Maximization (EM)
• Variational Approximation
• Noisy-OR Component Analyzer
• Model Formulation
• Variational EM for NOCA
• References

SLIDE 2

2/4/2020 2

Latent Variable Generative Models

• Generative Models: Unsupervised learning models that study the underlying

structure (e.g. interesting patterns) and causal structures of data to generate data like it.

• Latent (hidden) variables are random variables that are hard to observe. (ex.

Length is measured, but intelligence is not), and is assumed to affect the response variable.

• The idea: introduce an unobserved latent variable, S, and use it to generate a

traceable, less complex distribution. p(x)

Complex Distribution

p(x, s) = p(x | s) p(s)

Simpler Distribution

Latent Variable Generative Models

• Assumption: Observable variables are independent given latent

variables.

. . . . . .

S1 S2 Sq

x1 x2 xd-1 xd

SLIDE 3

2/4/2020 3

Cooperative Vector Quantizer (CVQ)

• Latent variables (s): Binary vars with Dimensionality k
• Observed variables (x): real valued vars Dimensionality d

. . . . . .

S1 S2 Sk

x1 x2 xd-1 xd

CVQ – Model Description

• Model
• 𝑦 = σ𝑙=1

𝐿

𝑡𝑙𝑥𝑙 + 𝜗

• Latent variables 𝑡𝑗
• ~ Bernoulli distribution parameter: 𝜌𝑗
• 𝑄(𝑡𝑗 | 𝜌𝑗) = 𝜌𝑗𝑡𝑗 (1 − 𝜌𝑗)1−𝑡𝑗
• 𝑥𝑙 is the weight output by source 𝑡𝑙
• Observable variables 𝑦
• ~ Normal distributions parameters: W, Σ
• 𝑄(𝑦| 𝑡) = N(Ws, Σ),
• we assume Σ = 𝜏𝐽
• Joint for one instance of s and x
• 𝑞 𝑦, 𝑡 Θ) = 2−𝑒/2𝜏−𝑒/2exp{−

1 2𝜏2 (𝑦 − 𝑋𝑡)𝑈 (𝑦 − 𝑋𝑡) ς𝑗=1 𝑙

𝜌𝑗𝑡𝑗 (1 − 𝜌𝑗)1−𝑡𝑗

6

X: d real valued vars

…

S: k binary vars

              

dk d k

w w w w w w .. .. .. ..

1 21 1 12 11

W

SLIDE 4

2/4/2020 4

CVQ – Model Description

• Objective: to learn parameters of the model: W, π, σ
• If both x and s are observable,
• Use loglikelihood:

෍

𝑜=1 𝑂

𝑚𝑝𝑕𝑄(𝑦 𝑜 , 𝑡 𝑜 |Θ) = σ𝑜=1

𝑂

−𝑒 𝑚𝑝𝑕 𝜏 −

1 2𝜏2 (𝑦 𝑜 − 𝑋𝑡 𝑜 )𝑈(𝑦 𝑜 − 𝑋𝑡 𝑜 ) + σ𝑗=1 𝑙

𝑡𝑗

(𝑜) 𝑚𝑝𝑕 𝜌𝑗

(1 − 𝑡𝑗

(𝑜))log(1 − 𝜌𝑗) + c

• Solution is nice and easy

7

CVQ – Model Description

• Objective: to learn parameters of the model: W, π, σ
• If only x are observable
• Log likelihood of data:

𝑚𝑝𝑕𝑄 𝐸 Θ = σ𝑜=1

𝑂

𝑚𝑝𝑕𝑄(𝑦 𝑜 |Θ) = σ𝑜=1

𝑂

𝑚𝑝𝑕 σ{𝑡𝑜} 𝑄 (𝑦 𝑜 , 𝑡 𝑜 |Θ)

• Solution is hard, we can no longer benefit from the decomposition.
• Use Expectation Maximization (EM).

8

SLIDE 5

2/4/2020 5

Expectation Maximization (EM)

• Let H be a set of all variables with hidden or missing values
• 𝑄(𝐼, 𝐸|Θ, 𝜊) = 𝑄(𝐼 |𝐸, Θ, 𝜊)𝑄(𝐸|Θ, 𝜊)
• log 𝑄 ( 𝐼 , 𝐸 | Θ , 𝜊 ) = log 𝑄 ( 𝐼 | 𝐸 , Θ , 𝜊 ) + log 𝑄 ( 𝐸 | Θ , 𝜊 )
• log 𝑄 ( 𝐸 | Θ , 𝜊 ) = log 𝑄 ( 𝐼 , 𝐸 | Θ , 𝜊 ) − log 𝑄 ( 𝐼 | 𝐸 , Θ , 𝜊 )
• Average both sides with 𝑄(𝐼 |𝐸, Θ′, 𝜊) for Θ′
• 𝐹𝐼|𝐸,Θ′ 𝑚𝑝𝑕𝑄(𝐸|Θ, 𝜊) = 𝐹𝐼|𝐸,Θ′𝑚𝑝𝑕𝑄(𝐼, 𝐸|Θ, 𝜊) − 𝐹𝐼|𝐸,Θ′𝑚𝑝𝑕𝑄(𝐼|𝐸, Θ, 𝜊)
• log 𝑄(𝐸 | Θ, 𝜊 ) = 𝐺 (Θ | Θ′) = 𝐹(Θ | Θ′) + 𝐼 (Θ | Θ′)
• EM uses the true posterior.

) 𝑄(𝐼|𝐸, 𝛪′, 𝜊

9

Log-likelihood

• f data

Expectation Maximization (EM)

• General EM Algorithm:
• Initialize parameters Θ
• Set Θ'=Θ
• Expectation step
• 𝐹(Θ|Θ′) =

𝑚𝑝𝑕𝑄(𝐼, 𝐸|Θ, 𝜊) 𝑄(𝐼|𝐸,Θ′)

• Maximization step
• Θ = argmax 𝐹(Θ|Θ′)
• Repeat until no or small improvement in Θ ((Θ = Θ')
• Problem
• 𝑄 𝐼 𝐸, Θ′ = ς𝑜=1

𝑂

𝑄(𝑦 𝑜 , 𝑡 𝑜 |Θ′)

• Each data point requires us to calculate 2𝑙 probabilities
• If k is large, then this is a bottleneck

10

SLIDE 6

2/4/2020 6

Variational Approximation

• An alternative method to approximate inference based on stochastic sampling.
• Let H be a set of all variables with hidden or missing values
• log 𝑄 ( 𝐸 | Θ , 𝜊 ) = log 𝑄 ( 𝐼 , 𝐸 | Θ , 𝜊 ) − log 𝑄 ( 𝐼 | 𝐸 , Θ , 𝜊 )
• Average both sides using a distribution 𝑅(𝐼 | 𝜇) [surrogate posterior]

𝐹𝐼|𝜇𝑚𝑝𝑕𝑄 𝐸 Θ, 𝜊 = 𝐹𝐼|𝜇𝑚𝑝𝑕𝑄(𝐼, 𝐸|Θ, 𝜊) − 𝐹𝐼|𝜇𝑚𝑝𝑕𝑅(𝐼 |𝜇) +𝐹𝐼|𝜇𝑚𝑝𝑕𝑅(𝐼 |𝜇) − 𝐹𝐼|𝜇𝑚𝑝𝑕𝑄(𝐼|Θ, 𝜊) ) log𝑄(𝐸|𝛪, 𝜊) = 𝐺(𝑅, 𝛪) + 𝐿𝑀(𝑅, 𝑄

𝐺(𝑅, Θ) = Σ{𝐼}𝑅(𝐼 |𝜇)𝑚𝑝𝑕𝑄(𝐼, 𝐸|Θ, 𝜊) − Σ{𝐼}𝑅(𝐼 |𝜇)𝑚𝑝𝑕𝑅(𝐼 |𝜇) 𝐿𝑀(𝑅, 𝑄) = Σ{𝐼}𝑅(𝐼 |𝜇)[𝑚𝑝𝑕𝑅(𝐼 |𝜇) − 𝑚𝑝𝑕𝑄(𝐼 |𝐸, Θ)]

11

Variational Approximation

) log𝑄(𝐸|𝛪, 𝜊) = 𝐺(𝑅, 𝛪) + 𝐿𝑀(𝑅, 𝑄

𝐺(𝑅, Θ) = Σ{𝐼}𝑅(𝐼 |𝜇)𝑚𝑝𝑕𝑄(𝐼, 𝐸|Θ, 𝜊) − Σ{𝐼}𝑅(𝐼 |𝜇)𝑚𝑝𝑕𝑅(𝐼 |𝜇) 𝐿𝑀(𝑅, 𝑄) = Σ{𝐼}𝑅(𝐼 |𝜇)[𝑚𝑝𝑕𝑅(𝐼 |𝜇) − 𝑚𝑝𝑕𝑄(𝐼 |𝐸, Θ)]

• Approximation: maximize 𝐺(𝑅, Θ)
• Parameters: Θ, 𝜇
• Maximization of F pushes up the lower bound on the log-likelihood

log 𝑄 𝐸 Θ, 𝜊 ≥ 𝐺 𝑅, Θ .

SLIDE 7

2/4/2020 7

Kullback-Leibler (KL) divergence

• A method to measure the difference between two probability distributions
• ver the same variable x
• 𝐿𝑀(𝑄 || 𝑅)
• Where the “||” operator indicates “divergence” or P’s divergence from Q
• Entropy: the average amount of information for a probability distribution
• 𝐼 𝑄 = 𝐹𝑄 𝐽𝑄 𝑌

= − σ𝑗=1

𝑜

𝑄 𝑗 log(𝑄 𝑗 )

• 𝐿𝑀 𝑄 ||𝑅 = 𝐼 𝑄, 𝑅 − 𝐼 𝑄 = − σ𝑗=1

𝑜

𝑄 𝑗 log 𝑅 𝑗 + σ𝑗=1

𝑜

𝑄 𝑗 log(𝑄 𝑗 ) = σ𝑗=1

𝑜

𝑄 𝑗 log(

𝑄(𝑗) 𝑅(𝑗))

• If we have some theoretic minimal distribution P, we want to try to find an

approximation Q that tries to get as close as possible by minimizing the KL divergence

13

Variational EM

• To use Variational EM, we hope if we choose 𝑅(𝐼 | 𝜇) well, the optimization of

both 𝜇 and Θ will become easy.

• A well-behaved choice for 𝑅(𝐼 | 𝜇) is the mean field approximation.
• Let H – be a set of all variables with hidden or missing values:
• E-step: Compute expectation over hidden variables
• Optimize: 𝐺(𝑅, Θ) with respect to 𝜇 while keeping Θ fixed.
• M-step: Maximize expected loglikelihood
• Optimize: 𝐺 (𝑅 , Θ) with respect to Θ while keeping 𝜇𝑡 fixed.

14

SLIDE 8

2/4/2020 8

Mean Field Approximation

• To find the distribution Q, we use Mean Field Approximation
• Assumption:
• 𝑅(𝐼|𝜇) is the mean field approximation
• Variables in the 𝑅(𝐼) distribution are independent variables 𝐼𝑗
• Q is completely factorized

𝑅(𝐼|𝜇) = ς𝑅𝑗(𝐼𝑗|𝜇𝑗)

• For our CVQ model
• Hidden variables are binary sources

𝑅(𝐼|𝜇) = ෑ

𝑜=1…𝑂

𝑅(𝑡 𝑜 |𝜇𝑜) 𝑅(𝑡 𝑜 |𝜇𝑜) = ς𝑗=1…𝑙 𝑅(𝑡𝑗

𝑜 |𝜇𝑗 (𝑜))

𝑅 𝑡𝑗

𝑜 𝜇𝑗 𝑜

= 𝜇𝑗

𝑜 𝑡𝑗 𝑜

(1 − 𝜇𝑗

𝑜 )1 −𝑡𝑗 𝑜

15

Mean Field Approximation

• Functional F for the mean field:

𝐺(𝑅, 𝛪) = ෍

𝐼

) 𝑅(𝐼|𝜇 log𝑄(𝐼, 𝐸|𝛪, 𝜊) − ෍

𝐼

) 𝑅(𝐼|𝜇 log𝑅(𝐼|𝜇)

• Assume just one data point x and corresponding s :

𝐺 𝑅, Θ = ෍

𝑜=1 𝑂

𝑚𝑝𝑕𝑄((𝑦 𝑜 , 𝑡 𝑜 |Θ) 𝑅(𝑡 𝑜 |𝜇 𝑜 ) − 𝑚𝑝𝑕𝑅 𝑡 𝑜 𝜇 𝑜

𝑅 𝑡 𝑜 𝜇 𝑜

= ቁ −𝑒log𝜏 −

1 2𝜏2 𝐲 − 𝐗𝐭 𝑈(𝐲 − 𝐗𝐭 ) 𝑅(𝑡|𝜇

+ ෌𝑗=1

𝑙

) 𝑡𝑗log𝜌𝑗 + (1 − 𝑡𝑗)log(1 − 𝜌𝑗

) 𝑅(𝑡|𝜇

− ෌𝑗=1

𝑙

) 𝑡𝑗log𝜇𝑗 + (1 − 𝑡𝑗)log(1 − 𝜇𝑗

) 𝑅(𝑡|𝜇

16

(1) (2) (3)

SLIDE 9

2/4/2020 9

Mean Field Approximation

• Functional F. Part (1)

ቇ −𝑒log𝜏 − 1 2𝜏2 𝐲 − 𝐗𝐭 𝑈(𝐲 − 𝐗𝐭

) 𝑅(𝑡|𝜇

= ቁ −𝑒log𝜏 −

1 2𝜏2 𝐲 − σ𝑗=1 𝑙

𝑡𝑗 𝐱𝑗

𝑈(𝐲 − σ𝑗=1 𝑙

𝑡𝑗 𝐱𝑗

) 𝑅(𝑡|𝜇

= −𝑒log𝜏 −

1 2𝜏2 𝐲𝑈𝐲 − 2 ෌𝑗=1 𝑙

൫𝑡𝑗 𝐱𝑗)𝐲 + ෍

𝑗=1 𝑙

σ𝑘=1

𝑙

𝑡𝑗 𝑡

𝑘𝐱𝑗𝑈𝐱 𝑘 ) 𝑅(𝑡|𝜇

= −𝑒log𝜏 − 1 2𝜏2 𝐲𝑈𝐲 − 2 ෍

𝑗=1 𝑙

𝑡𝑗

) 𝑅(𝑡𝑗|𝜇𝑗 𝐱𝑗)𝐲 + ා 𝑗=1 𝑙

෍

𝑘=1 𝑙

𝑡𝑗𝑡

𝑘 ) 𝑅(𝑡|𝜇 𝐱𝑗𝑈𝐱 𝑘

𝑡𝑗

) 𝑅(𝑡𝑗|𝜇𝑗 = 𝜇𝑗

൯ 𝑡𝑗𝑡

𝑘 ) 𝑅(𝑡|𝜇 = 𝜇𝑗𝜇𝑘 + 𝜀𝑗𝑘(𝜇𝑗 − 𝜇𝑗 2

Mean Field Approximation

• Functional F. Part (2)
• Functional F. part (3)

෍

𝑗=1 𝑙

) 𝑡𝑗log𝜌𝑗 + (1 − 𝑡𝑗)log(1 − 𝜌𝑗

) 𝑅(𝑡|𝜇 = ෍ 𝑗=1 𝑙

൯ 𝑡𝑗

) 𝑅(𝑡𝑗|𝜇𝑗 log𝜌𝑗 + (1 − 𝑡𝑗 ) 𝑅(𝑡𝑗|𝜇𝑗 )log(1 − 𝜌𝑗

= ෍

𝑗=1 𝑙

) 𝜇𝑗log𝜌𝑗 + (1 − 𝜇𝑗)log(1 − 𝜌𝑗 ෍

𝑗=1 𝑙

) 𝑡𝑗log𝜇𝑗 + (1 − 𝑡𝑗)log(1 − 𝜇𝑗

) 𝑅(𝑡|𝜇 = ෍ 𝑗=1 𝑙

) 𝜇𝑗log𝜇𝑗 + (1 − 𝜇𝑗)log(1 − 𝜇𝑗

SLIDE 10

2/4/2020 10

Mean Field Approximation

Functional F:

= −𝑒log𝜏 − 1 2𝜏2 𝐲𝑈𝐲 − 2 ෍

𝑗=1 𝑙

𝑡𝑗

) 𝑅(𝑡𝑗|𝜇𝑗 𝐱𝑗)𝐲 + ා 𝑗=1 𝑙

෍

𝑘=1 𝑙

𝑡𝑗𝑡𝑘

) 𝑅(𝑡|𝜇 𝐱𝑗𝑈𝐱 𝑘

Parameters: 𝑋, 𝜌, 𝜏 Mean field parameters: 𝜇

+ ෍

𝑗=1 𝑙

) 𝜇𝑗log𝜌𝑗 + (1 − 𝜇𝑗)log(1 − 𝜌𝑗 + ෍

𝑗=1 𝑙

) 𝜇𝑗log𝜇𝑗 + (1 − 𝜇𝑗)log(1 − 𝜇𝑗

Mean Field Approximation

Functional F (for all data points):

𝐺(𝑅, 𝛪) = ෍

𝑜=1 𝑂

൯ log𝑄(𝐲 𝑜 , 𝐭 𝑜 |𝛪

൯ 𝑅(𝑡 𝑜 |𝜇 𝑜

− ൯ log𝑅(𝐭 𝑜 |𝜇 𝑜

൯ 𝑅(𝑡 𝑜 |𝜇 𝑜

= −𝑒log𝜏 − 1 2𝜏2 𝐲 𝑜 𝑈𝐲 𝑜 − 2 ෍

𝑗=1 𝑙

𝜇𝑗

𝑜 𝐱𝑗)𝐲 𝑜 + ා 𝑗=1 𝑙

෍

𝑘=1 𝑙

ቁ 𝜇𝑗

𝑜 𝜇𝑘 𝑜 + 𝜀𝑗𝑘(𝜇𝑗 𝑜 − 𝜇𝑗 𝑜 2

𝐱𝑗𝑈𝐱

𝑘

+ ෍

𝑗=1 𝑙

൯ 𝜇𝑗

𝑜 log𝜌𝑗 + (1 − 𝜇𝑗 𝑜 )log(1 − 𝜌𝑗

+ ෍

𝑗=1 𝑙

൯ 𝜇𝑗

𝑜 log𝜇𝑗 𝑜 + (1 − 𝜇𝑗 𝑜 )log(1 − 𝜇𝑗 𝑜

Parameters: 𝑋, 𝜌, 𝜏 Mean field parameters: 𝜇 = 𝜇1, 𝜇2, … , 𝜇𝑜

SLIDE 11

2/4/2020 11

Variational EM

• E-step
• Optimize 𝐺(𝑅, Θ) with respect to 𝜇 while keeping Θ fixed

𝜖 𝜖𝜇𝑣 𝐺 = 1 𝜏2 (𝐲 − ෍

𝑘≠𝑣

𝜇𝑘𝐱

𝑘)𝑈 𝐱𝑣 − 1

2𝜏2 𝐱𝑣𝑈𝐱𝑣 + log 𝜌𝑣 1 − 𝜌𝑣 − log 𝜇𝑣 1 − 𝜇𝑣 Set

𝜖 𝜖𝜇𝑣 𝐺 = 0

𝜇𝑣 = g(

1 𝜏2 (𝑦 − σ𝑘≠𝑣 𝜇𝑘𝑥 𝑘)𝑈𝑥𝑣 − 1 2𝜏2 𝑥𝑣𝑈𝑥𝑣 + log π𝑣 1 −π𝑣) , g(x) = 1 1+ 𝑓−𝑦

21

Variational EM

• M-step
• Optimize 𝐺 (𝑅 , Θ ) with respect to Θ while keeping 𝜇𝑡

Start with : For N data points 𝜖 𝜖𝜌𝑣 𝐺 = ෎

𝑜=1 𝑂

𝜇𝑣

𝑜log 1

𝜌𝑣 − (1 − 𝜇𝑣

𝑜)log

1 1 − 𝜌𝑣 Set

𝜖 𝜖π𝑣 𝐺 = 0,

𝜌𝑣 =

෍

𝑜=1 𝑂

𝜇𝑣

𝑜

𝑂

(closed form solution)

22

SLIDE 12

2/4/2020 12

Variational EM

And for parameter w:

𝜖 𝜖𝑥𝑣𝑤 𝐺 = ෍

𝑜=1 𝑂

− 1 2𝜏2 𝜇𝑤

𝑜 𝑦𝑣 𝑜 + 2 ෍ 𝑘≠𝑤

𝜇𝑤

𝑜 𝜇𝑘 𝑜 𝑥𝑣𝑘 + 2𝜇𝑤 𝑜 𝑥𝑣𝑤 = 0

• For each variable v:

The equations define a set of k linear equations that can be solved

23

              

dk d k

w w w w w w .. .. .. ..

1 21 1 12 11

W ) w w (w W

k 2 1 



Image Separation Experiment

Source images associated with latent variables

SLIDE 13

2/4/2020 13

Mixed images

• Images generated by the model.
• Some of the images are noise.
• Generating enough samples; the

model can retrieve the original source.

Recovered sources

SLIDE 14

2/4/2020 14

Modeling High-Dimensional Data

• Definition: Number of dimensions are high that makes calculations extremely

difficult (number of features exceed number of observations).

• Examples of domains with High-Dimensional Data:
• Sensor Networks
• Document Repositories.
• Typically, variables are dependent.
• How to model dependencies?
• Full model (intractable, overfitting)
• All-independent (unrealistic)
• Middle-of-the-road approaches
• Captures dependencies in an efficient way (representation, reasoning, learning).

Noisy-OR Component Analyzer

• Objective: Capture dependencies via latent factors and combinations.
• The dependencies between observables are represented using a smaller number of

hidden binary factors.

• NOCA model has binary nodes:
• k parameters for each observed node, p1j, …., pkj
• pij is interpreted as “strength of influence” of Si on
• bservable variable xj

𝑄(𝑌

𝑘 = 0|s) = ෑ 𝑗=1 𝐿

1 − 𝑞𝑗𝑘

𝑡𝑗

𝑄(𝑌

𝑘 = 1|𝐭) = 1 − 𝑄(𝑌 𝑘 = 0|𝐭) = 1 − ෑ 𝑗=1 𝐿

1 − 𝑞𝑗𝑘

𝑡𝑗

… … 𝑡1 𝑡2 𝑡𝑙 𝑦1 𝑦2 𝑦𝑒−1 𝑦𝑒

πk π2 π1

… p

SLIDE 15

2/4/2020 15

Noisy-OR Component Analyzer

Assumptions:

• All possible causes 𝑉𝑗 for an event 𝑌 are modeled using nodes (random

variables) and their values, with T (or 1) reflecting the presence of the cause , and F (or 0) its absence

• If one needs to represent unknown causes one can add a leak node
• Parameters: For each cause 𝑉𝑗 define an (independent) probability qi that

represents the probability with which the cause does not lead to X = T (or 1), or in other words, it represents the probability that the positive value of variable 𝑌 is inhibited when 𝑉𝑗 is present Note: The negated causes ¬𝑉𝑗 (reflecting the absence of the cause) do not have any influence on 𝑌.Why?

29

• A generalization of the logical OR

p | 𝑦 = 1 𝑉1, … , 𝑉j, ¬𝑉

𝑘+1, … , ¬𝑉𝑙 = 1 − ς𝑗=1 𝑘

𝑟𝑗 𝑞 | 𝑦 = 0 𝑉1, … , 𝑉j, ¬𝑉

𝑘+1, … , ¬𝑉𝑙 = ෑ 𝑗=1 𝑘

𝑟𝑗

Noisy-OR Example

𝜈 | 𝑦 = 1 𝑉1, … , 𝑉j, ¬𝑉

𝑘+1, … , ¬𝑉𝑙 = 1 − ෑ 𝑗=1 𝑘

𝑟𝑗 𝜈 | 𝑦 = 0 𝑉1, … , 𝑉j, ¬𝑉

𝑘+1, … , ¬𝑉𝑙 = ෑ 𝑗=1 𝑘

𝑟𝑗

30

Cold Flu Malaria 𝜈(Fever ) 𝜈 ¬𝐺𝑓𝑤𝑓𝑠 F F F 1 F F T 0.9 0.1 F T F 0.8 0.2 F T T 0.98 0.02 = 0.2 × 0.1 T F F 0.4 0.6 T F T 0.94 0.06 = 0.6 × 0.1 T T F 0.88 0.12 = 0.6 × 0.2 T T T 0.988 0.012 = 0.6 × 0.2 × 0.1 Flue Cold Malaria Fever q_mal=0.1 q_fl=0.2 q_cold=0.6

SLIDE 16

2/4/2020 16

Noisy-OR parameter reduction

31

Flue Cold Mala

• ria

Fever q_mal=0.1 q_fl=0.2 q_cold=0.6

• Please note that in general the number of entries defining the

CPT (conditional probability table) grows exponentially with the number of parents;

• for q binary parents the number is : 2𝑟
• For the noisy-or CPT the number of parameters is q + 1

Example: CPT: 8 different combination

• f values for 3 binary parents

Noisy-or: 4 parameters

Noisy-OR Component Analyzer (NOCA)

32

… … Observed variables 𝐲 : 𝒆-dimensions 𝑦 ∈ 0,1 𝑒

𝑄 𝑦 = ා

𝑡

ෑ

𝑘=1 𝑒

𝑄 ห 𝑦𝑘 𝑡 ෑ

𝑗=1 𝑟

𝑄 𝑡𝑗

Latent variables 𝒕: (𝒓 + 𝟐)-dimensions s ∈ 0,1 𝑟, 𝑄 | 𝑡𝑗 𝜌𝑗 = 𝜌𝑗

𝑡𝑗 1 − 𝜌𝑗 1−𝑡𝑗

Loading Matrix: 𝒒 = {𝑞𝑗𝑘}𝑘=1,…,𝑒

𝑗=1,…,𝑟

𝑟 < 𝑒 𝑡1 𝑡2 𝑡𝑟 𝑦1 𝑦2 𝑦𝑒−1 𝑦𝑒

SLIDE 17

2/4/2020 17

Why EM won’t work?

• Take N iid samples (D-dimensional binary vectors)
• We will need:
• The joint distribution

𝑄(𝐲, 𝐭) = 𝑄(𝐲|𝐭)𝑄(𝐭) = 𝑄(𝐭) ී

𝑘

1 − ෑ

𝑗=1 𝐿

1 − 𝑞𝑗𝑘

𝑡𝑗 𝑦𝑘

ෑ

𝑗=1 𝐿

1 − 𝑞𝑗𝑘

𝑡𝑗 1−𝑦𝑘

• Joint over observables

𝑄(𝐲) = ෍

𝐭

) 𝑄(𝐲, 𝐭 = ා

𝐭

൱ ෑ

𝑘

൯ 𝑄(𝑦𝑘|𝐭 𝑄(𝐭 Problem1: not a product Problem 2: summation over 2K terms

Variational EM for NOCA

• Similar to what we did for CVQ, we simplify the distribution with a

decomposable Q(s)

log 𝑄 | 𝑦 𝜄 = log ෑ

𝑜=1 𝑂

𝑄 | 𝑦𝑜 𝜄 = ා

𝑜=1 𝑂

log ෍

𝑡

𝑄 𝑦𝑜, | 𝑡𝑜 𝜄 = ා

𝑜=1 𝑂

log ෎

𝑡

𝑄 𝑦𝑜, | 𝑡𝑜 𝜄, 𝑟𝑜 𝑅 𝑡𝑜 𝑅 𝑡𝑜 ≥ ා

𝑜=1 𝑂

෍

𝑡𝑜

𝐹𝑡𝑜 log 𝑄 𝑦𝑜, | 𝑇𝑜 𝜄 − 𝐹𝑡𝑜 log 𝑅 𝑇𝑜

• log(𝑄 𝑦𝑜,

| 𝑡𝑜 𝜄, 𝑟𝑜 still can not be solved easily

• Noisy-Or is not in exponential family

34

SLIDE 18

2/4/2020 18

Variational EM for NOCA

A further lower bound is required

• Jensen’s inequality: 𝒈(𝒃 + ෍

𝒌

𝒓𝒌𝒚𝒌) ≥ ෍

𝒌 𝒓𝒌𝒈 𝒃 + 𝐲𝒌

𝑄 ห 𝑦𝑘 𝑡 = 1 − 1 − 𝑞0𝑘 ෑ

𝑗=1 𝑟

1 − 𝑞𝑗𝑘

𝑡𝑗 𝑦𝑘

1 − 𝑞0𝑘 ෑ

𝑗=1 𝑟

1 − 𝑞𝑗𝑘

𝑡𝑗 1−𝑦𝑘

𝐓𝐟𝐮 𝜾𝒋𝒌 = − 𝐦𝐩𝐡 𝟐 − 𝒒𝒋𝒌 𝑄 ห 𝑦𝑘 𝑡 = exp 𝑦𝑘 log 1 − exp −𝜄0𝑘 − ෍

𝑗=1 𝑟

𝜄𝑗𝑘𝑡𝑗 + 1 − 𝑦𝑘 −𝜄0𝑘 − ෍

𝑗=1 𝑟

𝜄𝑗𝑘𝑡𝑗

𝑸 ห 𝒚𝒌 𝒕 does not factorize for 𝒚𝒌 = 𝟐

𝑄 ห 𝑦𝑘 = 1 𝑡 = exp[log 1 − exp −𝜄0𝑘 − ෍

𝑗=1 𝑟

𝜄𝑗𝑘𝑡𝑗 ] = exp log 1 − exp −𝜄0𝑘 − ෎

𝑗=1 𝑟

𝜄𝑗𝑘𝑡𝑗 𝑟𝑘 𝑗 𝑟𝑘 𝑗 ≥ exp[෍

𝑗=1 𝑟

𝑟𝑘(𝑗) log 1 − exp −𝜄0𝑘 − 𝜄𝑗𝑘𝑡𝑗 𝑟𝑘(𝑗) ] = exp[෍

𝑗=1 𝑟

𝑟𝑘 𝑗 [𝑡𝑗log 1 − exp −𝜄0𝑘 − 𝜄𝑗𝑘 𝑟𝑘 𝑗 + (1 − 𝑡𝑗)log 1 − exp −𝜄0𝑘 ]] = ෑ

𝑗=1 𝑟

exp[𝑟𝑘 𝑗 𝑡𝑗[log 1 − exp −𝜄0𝑘 − 𝜄𝑗𝑘 𝑟𝑘 𝑗 − log( 1 − exp −𝜄0𝑘 )] + 𝑟𝑘 𝑗 log( 1 − exp −𝜄0𝑘 )]] 35

Variational EM for NOCA

A further lower bound is required

log 𝑄 | 𝑦 𝜄

≥ ෎

𝑜=1 𝑂

෍

𝑡𝑜

𝐹𝑡𝑜 log 𝑄 𝑦𝑜, | 𝑡𝑜 𝜄 − 𝐹𝑡𝑜 log 𝑅 𝑡𝑜 ≥ ෎

𝑜=1 𝑂

෍

𝑡𝑜

𝐹𝑡𝑜 log ෨ 𝑄 𝑦𝑜, | 𝑡𝑜 𝜄, 𝑟𝑜 − 𝐹𝑡𝑜 log 𝑅 𝑡𝑜

= ෎

𝑜=1 𝑂

෍

𝑡𝑜

𝐹𝑡𝑜 log ෨ 𝑄 𝑦𝑜|𝑡𝑜, 𝜄, 𝑟𝑜 𝑄

| 𝑡𝑜 𝜄

− 𝐹𝑡𝑜 log 𝑅 𝑡𝑜

= ෌𝑜=1

𝑂

ℱ

𝑜 𝑦𝑜, 𝑅 𝑡𝑜

= ℱ(𝑦, 𝑅(𝑡))

36

SLIDE 19

2/4/2020 19

Variational EM for NOCA

37

Parameters: 𝒓𝒐, 𝜾𝒋𝒌, 𝜾𝟏𝒌

• E-step: update 𝒓𝒐 to optimize 𝑮𝒐
• 𝑟𝑜𝑘 ⅈ ⇐ 𝑡𝑜𝑗 𝑅 𝑇𝑜

𝑟𝑜𝑘 ⅈ log 1−ⅇ−𝜄0𝑘 ൤log 1 − 𝐵𝑜 ⅈ, 𝑘

−

𝜄𝑗𝑘 𝑟𝑜𝑘 ⅈ 𝐵𝑜 ⅈ,𝑘 1−𝐵𝑜 ⅈ,𝑘 −

Structure Recovery Experiment

a) Image patterns associated with hidden sources. b) Example images generated by the NOCA model c) Images recovered from source input.

(a) (b) (c)

SLIDE 20

2/4/2020 20

References

• X. Lu, M. Hauskrecht, R.S. Day. Variational Bayesian learning of the cooperative vector quantizer

(CVQ) model. Part I: The Theory, Technical Report, Computer Science Department, University of Pittsburgh, 2002

• Singliar, Hauskrecht. Noisy-or Component Analysis and its Application to Link Analysis. Journal
• f Machine Learning Research 2006
• https://people.cs.pitt.edu/~milos/courses/cs3750-Fall2007/lectures/class20.pdf
• http://people.cs.pitt.edu/~milos/courses/cs3750/lectures/class8.pdf
• http://www.blutner.de/Intension/Noisy%20OR.pdf
• Jaakkola, Tommi S., and Michael I. Jordan. "Variational probabilistic inference and the QMR-DT

network." Journal of artificial intelligence research 10 (1999): 291-322.

• http://bjlkeng.github.io/posts/variational-bayes-and-the-mean-field-approximation/
• https://machinelearningmastery.com/divergence-between-probability-distributions/
• https://towardsdatascience.com/deep-latent-variable-models-unravel-hidden-structures-

a5df0fd32ae2