CSci 8980: Advanced Topics in Graphical Models Mixture Models, EM, - PowerPoint PPT Presentation

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net CSci 8980: Advanced Topics in Graphical Models Mixture Models, EM, Exponential Families Instructor: Arindam Banerjee September 11, 2007

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Incremental EM p ( Z ) = � Since z i are independent, optimal ˜ i ˜ p ( z i )

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Incremental EM p ( Z ) = � Since z i are independent, optimal ˜ i ˜ p ( z i ) Sufficient to work with such ˜ p in F (˜ p , θ )

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Incremental EM p ( Z ) = � Since z i are independent, optimal ˜ i ˜ p ( z i ) Sufficient to work with such ˜ p in F (˜ p , θ ) Then F (˜ p , θ ) = � i F i (˜ p i , θ ) where F i (˜ p i , θ ) = E ˜ p i [log p ( x i , z i | θ )] + H (˜ p i )

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Incremental EM p ( Z ) = � Since z i are independent, optimal ˜ i ˜ p ( z i ) Sufficient to work with such ˜ p in F (˜ p , θ ) Then F (˜ p , θ ) = � i F i (˜ p i , θ ) where F i (˜ p i , θ ) = E ˜ p i [log p ( x i , z i | θ )] + H (˜ p i ) Incremental algorithm that works one point at a time

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Incremental EM (Contd.) Basic Incremental EM

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Incremental EM (Contd.) Basic Incremental EM E-step: Choose a data item i to be updated p ( t ) p ( t − 1) Set ˜ = ˜ for j � = i j j p ( t ) = p ( z i | x i , θ ( t ) ) Set ˜ i

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Incremental EM (Contd.) Basic Incremental EM E-step: Choose a data item i to be updated p ( t ) p ( t − 1) Set ˜ = ˜ for j � = i j j p ( t ) = p ( z i | x i , θ ( t ) ) Set ˜ i M-step: Set θ ( t ) to argmax θ E ˜ p ( t ) [log p ( x , z | θ )]

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Incremental EM (Contd.) Basic Incremental EM E-step: Choose a data item i to be updated p ( t ) p ( t − 1) Set ˜ = ˜ for j � = i j j p ( t ) = p ( z i | x i , θ ( t ) ) Set ˜ i M-step: Set θ ( t ) to argmax θ E ˜ p ( t ) [log p ( x , z | θ )] M-step needs to look at all components of ˜ p

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Incremental EM (Contd.) Basic Incremental EM E-step: Choose a data item i to be updated p ( t ) p ( t − 1) Set ˜ = ˜ for j � = i j j p ( t ) = p ( z i | x i , θ ( t ) ) Set ˜ i M-step: Set θ ( t ) to argmax θ E ˜ p ( t ) [log p ( x , z | θ )] M-step needs to look at all components of ˜ p Can be simplified by using sufficient statistics

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Incremental EM (Contd.) Basic Incremental EM E-step: Choose a data item i to be updated p ( t ) p ( t − 1) Set ˜ = ˜ for j � = i j j p ( t ) = p ( z i | x i , θ ( t ) ) Set ˜ i M-step: Set θ ( t ) to argmax θ E ˜ p ( t ) [log p ( x , z | θ )] M-step needs to look at all components of ˜ p Can be simplified by using sufficient statistics For a distribution p ( x | θ ), s ( x ) is a sufficient statistic if p ( x | s ( x ) , θ ) = p ( x | s ( x )) = ⇒ p ( x | θ ) = h ( x ) q ( s ( x ) | θ )

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Incremental EM with Sufficient Statistics EM with sufficient statistics

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Incremental EM with Sufficient Statistics EM with sufficient statistics s ( t ) = E ˜ p ( z ) = p ( z | x , θ ( t − 1) ) E-step: Set ˜ p [ s ( x , z )] where ˜

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Incremental EM with Sufficient Statistics EM with sufficient statistics s ( t ) = E ˜ p ( z ) = p ( z | x , θ ( t − 1) ) E-step: Set ˜ p [ s ( x , z )] where ˜ M-step: Set θ ( t ) to θ , the max likelihood given ˜ s ( t )

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Incremental EM with Sufficient Statistics EM with sufficient statistics s ( t ) = E ˜ p ( z ) = p ( z | x , θ ( t − 1) ) E-step: Set ˜ p [ s ( x , z )] where ˜ M-step: Set θ ( t ) to θ , the max likelihood given ˜ s ( t ) Incremental EM with sufficient statistics

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Incremental EM with Sufficient Statistics EM with sufficient statistics s ( t ) = E ˜ p ( z ) = p ( z | x , θ ( t − 1) ) E-step: Set ˜ p [ s ( x , z )] where ˜ M-step: Set θ ( t ) to θ , the max likelihood given ˜ s ( t ) Incremental EM with sufficient statistics E-step: Choose a data item i to be updated s ( t ) s ( t − 1) Set ˜ = ˜ , for j � = i j j s ( t ) p i ( z i ) = p ( z i | x i , θ ( t − 1) ) Set ˜ = E ˜ p i [ s i ( x i , z i )], where ˜ i s ( t ) = ˜ s ( t − 1) − ˜ s ( t − 1) s ( t ) Set ˜ + ˜ i i

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Incremental EM with Sufficient Statistics EM with sufficient statistics s ( t ) = E ˜ p ( z ) = p ( z | x , θ ( t − 1) ) E-step: Set ˜ p [ s ( x , z )] where ˜ M-step: Set θ ( t ) to θ , the max likelihood given ˜ s ( t ) Incremental EM with sufficient statistics E-step: Choose a data item i to be updated s ( t ) s ( t − 1) Set ˜ = ˜ , for j � = i j j s ( t ) p i ( z i ) = p ( z i | x i , θ ( t − 1) ) Set ˜ = E ˜ p i [ s i ( x i , z i )], where ˜ i s ( t ) = ˜ s ( t − 1) − ˜ s ( t − 1) s ( t ) Set ˜ + ˜ i i M-step: Set θ ( t ) to θ , the max likelihood given ˜ s ( t )

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Example Consider a mixture of 2 univariate Gaussians

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Example Consider a mixture of 2 univariate Gaussians Parameter set θ = ( α, µ 1 , σ 1 , µ 2 , σ 2 )

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Example Consider a mixture of 2 univariate Gaussians Parameter set θ = ( α, µ 1 , σ 1 , µ 2 , σ 2 ) Sufficient statistics s i ( x i , z i ) = [ z i (1 − z i ) z i x i (1 − z i ) x i z i x 2 i (1 − z i ) x 2 i ]

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Example Consider a mixture of 2 univariate Gaussians Parameter set θ = ( α, µ 1 , σ 1 , µ 2 , σ 2 ) Sufficient statistics s i ( x i , z i ) = [ z i (1 − z i ) z i x i (1 − z i ) x i z i x 2 i (1 − z i ) x 2 i ] Given s ( x , z ) = � i s ( x i , z i ) = ( n 1 , n 2 , m 1 , m 2 , q 1 , q 2 ) � 2 n 1 µ h = m h h = q h � m h σ 2 α = , , − n 1 + n 2 n h n h n h

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Sparse EM Consider a mixture model with many components

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Sparse EM Consider a mixture model with many components Most p ( z | x , θ ) will be negligibly small

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Sparse EM Consider a mixture model with many components Most p ( z | x , θ ) will be negligibly small Computation can be saved by freezing these

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Sparse EM Consider a mixture model with many components Most p ( z | x , θ ) will be negligibly small Computation can be saved by freezing these Only a small set of component posteriors need to be updated � q ( t ) z , if z �∈ S t p ( t ) ( z ) = ˜ Q ( t ) r ( t ) if z ∈ S t z

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net Sparse EM Consider a mixture model with many components Most p ( z | x , θ ) will be negligibly small Computation can be saved by freezing these Only a small set of component posteriors need to be updated � q ( t ) z , if z �∈ S t p ( t ) ( z ) = ˜ Q ( t ) r ( t ) if z ∈ S t z S t = set of plausible values