. Summary April 7th, 2011 Biostatistics 615/815 - Lecture 22 Hyun Min Kang April 7th, 2011 Hyun Min Kang Gibbs Sampling Biostatistics 615/815 Lecture 22: . . . . . . . . . Metropolis-Hastings Inference Implementation Gibbs Sampler Introduction . . . . . . . . . 1 / 34 . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . conditional distribution of latent variable z . distribution of z can be obtained . Maximization step (M-step) . . . . . . . . Hyun Min Kang Biostatistics 615/815 - Lecture 22 April 7th, 2011 . . . Inference . . . . . . . . . Introduction Gibbs Sampler . Implementation 2 / 34 . Expectation step (E-step) . Recap: The E-M algorithm Summary Metropolis-Hastings . . . . . . . . . . . . . . . . . . . . . . . . . . . • Given the current estimates of parameters λ ( t ) , calculate the • Then the expected log-likelihood of data given the conditional Q ( λ | λ ( t ) ) = E z | x ,λ ( t ) [ log p ( x , z | λ )] • Find the parameter that maximize the expected log-likelihood λ ( t +1) = arg max Q ( λ | λ t ) λ
. Inference April 7th, 2011 Biostatistics 615/815 - Lecture 22 Hyun Min Kang http://maxdama.blogspot.com/2008/07/trading-optimization-simulated.html Images by Max Dama from Recap - Local minimization methods Summary . Metropolis-Hastings 3 / 34 . Implementation Gibbs Sampler Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. Inference April 7th, 2011 Biostatistics 615/815 - Lecture 22 Hyun Min Kang http://maxdama.blogspot.com/2008/07/trading-optimization-simulated.html Images by Max Dama from Recap - Global minimization with Simulated Annealing Summary . Metropolis-Hastings 4 / 34 . Implementation Gibbs Sampler Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . 2 Randomly select a new point in the neighborhood of the original . . 3 Compare the two points using the Metropolis criterion . 4 Repeat steps 2 and 3 until system reaches equilibrium state . . . 5 Decrease temperature and repeat the above steps, stop when system reaches frozen state Hyun Min Kang Biostatistics 615/815 - Lecture 22 April 7th, 2011 1 Select starting temperature and initial parameter values . . Recap - Simulated Annealing Recipes . . . . . . . . . Introduction Gibbs Sampler Implementation Inference Metropolis-Hastings . Summary 5 / 34 . . . . . . . . . . . . . . . . . . . . . . . . . . . • In practice, repeat the process N times for large N .
. Inference April 7th, 2011 Biostatistics 615/815 - Lecture 22 Hyun Min Kang Gibbs Sampling Optimization Streategies Summary . . Metropolis-Hastings Implementation . . . . . . . . . 6 / 34 Introduction Gibbs Sampler . . . . . . . . . . . . . . . . . . . . . . . . . . . • Single Dimension • Golden Search • Parabolic Approximations • Multiple Dimensions • Simplex Method • E-M Algorithm • Simulated Annealing
. Implementation April 7th, 2011 Biostatistics 615/815 - Lecture 22 Hyun Min Kang Optimization Streategies Summary . . Inference Metropolis-Hastings 6 / 34 . Gibbs Sampler . Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Single Dimension • Golden Search • Parabolic Approximations • Multiple Dimensions • Simplex Method • E-M Algorithm • Simulated Annealing • Gibbs Sampling
. Implementation April 7th, 2011 Biostatistics 615/815 - Lecture 22 Hyun Min Kang Gibbs Sampler Summary . Metropolis-Hastings . Inference Gibbs Sampler Introduction . . . . . . . . . 7 / 34 . . . . . . . . . . . . . . . . . . . . . . . . . . . • Another MCMC Method • Update a single parameter at a time • Sample from conditional distribution when other parameters are fixed
. Summary April 7th, 2011 Biostatistics 615/815 - Lecture 22 Hyun Min Kang 3 Increment t and repeat the previous steps. . . i 2 Define the next set of parameter values by . . . . . Gibbs Sampler Algorithm . Metropolis-Hastings . . . . . . . . . Introduction 8 / 34 Gibbs Sampler Implementation Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Consider a particular choice of parameter values λ ( t ) . • Selecting a component to update, say i . • Sample value for λ ( t +1) , from p ( λ i | x , λ 1 , · · · , λ i − 1 , λ i +1 , · · · , λ k ) .
. Summary April 7th, 2011 Biostatistics 615/815 - Lecture 22 Hyun Min Kang 3 Increment t and repeat the previous steps. . . k 2 Define the next set of parameter values by . . . . . An alternative Gibbs Sampler Algorithm 9 / 34 . Inference Implementation Gibbs Sampler . . . . Metropolis-Hastings . . . . . Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Consider a particular choice of parameter values λ ( t ) . • Sample value for λ ( t +1) , from p ( λ 1 | x , λ 2 , λ 3 , · · · , λ k ) . 1 • Sample value for λ ( t +1) , from p ( λ 1 | x , λ 1 , λ 3 , · · · , λ k ) . 2 • · · · • Sample value for λ ( t +1) , from p ( λ k | x , λ 1 , λ 2 , · · · , λ k − 1 ) .
. . . . . . straightforward . Using source of each observations . . . . . . . . Hyun Min Kang Biostatistics 615/815 - Lecture 22 April 7th, 2011 . . . Inference . . . . . . . . . Introduction Gibbs Sampler Implementation . 10 / 34 Metropolis-Hastings Summary Using conditional distributions . Gibbs Sampling for Gaussian Mixture . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Observed data : x = ( x 1 , · · · , x n ) • Parameters : λ = ( π 1 , · · · , π k , µ 1 , · · · , µ k , σ 2 1 , · · · , σ 2 k ) . • Sample each λ i from conditional distribution - not very • Observed data : x = ( x 1 , · · · , x n ) • Parameters : z = ( z 1 , · · · , z n ) where z i ∈ { 1 , · · · , k } . • Sample each z i conditioned by all the other z .
. Inference April 7th, 2011 Biostatistics 615/815 - Lecture 22 Hyun Min Kang estimates of mixture parameters. specific component Update procedure in Gibbs sampler Summary . Metropolis-Hastings . 11 / 34 . . . . . . Gibbs Sampler Introduction . . . Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . π i N ( x j | µ i , σ 2 i ) Pr ( z j = i | x j , λ ) = l π l N ( x j | µ l , σ 2 ∑ l ) • Calculate the probability that the observation is originated from a • For a random j ∈ { 1 , · · · , n } , sample z j based on the current
. . April 7th, 2011 Biostatistics 615/815 - Lecture 22 Hyun Min Kang probabilities observed data point Initialization Summary . Metropolis-Hastings Inference Implementation Gibbs Sampler Introduction . . . . . . . . . 12 / 34 . . . . . . . . . . . . . . . . . . . . . . . . . . . • Must start with an initial assignment of component labels for each • A simple choice is to start with random assignment with equal
. Implementation April 7th, 2011 Biostatistics 615/815 - Lecture 22 Hyun Min Kang The Gibbs Sampler Summary . Metropolis-Hastings . Inference Gibbs Sampler Introduction . . . . . . . . . 13 / 34 . . . . . . . . . . . . . . . . . . . . . . . . . . . • Select initial parameters • Repeat a large number of times • Select an element • Update conditional on other elements
. Inference April 7th, 2011 Biostatistics 615/815 - Lecture 22 Hyun Min Kang . Implementing Gaussian Mixture Gibbs Sampler Summary . Metropolis-Hastings 14 / 34 Implementation . . . . . . . . . Gibbs Sampler Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . class normMixGibbs { public: int k; // # of components int n; // # of data std::vector<double> data; // size n : observed data std::vector<double> labels; // size n : label assignment for each observations std::vector<double> pis; // size k : pis std::vector<double> means; // size k : means std::vector<double> sigmas; // size k : sds std::vector<int> counts; // size k : # of elements in each labels std::vector<double> sums; // size k : sum across each label std::vector<double> sumsqs; // size k : squared sum across each label normMixGibbs(std::vector<double>& _data, int _k); // constructor void initParams(); // initialize parameters void updateParams(int numObs); // update parameters void remove(int i); // remove elements void add(int i, int label); // add an element with new label int sampleLabel(double x); // sample the label of an element void runGibbs(); // run Gibbs sampler };
. Inference April 7th, 2011 Biostatistics 615/815 - Lecture 22 Hyun Min Kang estimates of mixture parameters. specific component Update procedure in Gibbs sampler Summary . Metropolis-Hastings . 15 / 34 . . . . . . Gibbs Sampler Introduction . . . Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . π i N ( x j | µ i , σ 2 i ) Pr ( z j = i | x j , π, λ ) = l π l N ( x j | µ l , σ 2 ∑ l ) • Calculate the probability that the observation is originated from a • For a random j ∈ { 1 , · · · , n } , sample z j based on the current
Recommend
More recommend
