VCMC: Variational Consensus Monte Carlo Maxim Rabinovich, Elaine - - PowerPoint PPT Presentation
VCMC: Variational Consensus Monte Carlo Maxim Rabinovich, Elaine Angelino, Michael I. Jordan Berkeley Vision and Learning Center September 22, 2015 probabilistic models! sky fog bridge water grass object tracking & recognition
Maxim Rabinovich, Elaine Angelino, Michael I. Jordan Berkeley Vision and Learning Center September 22, 2015
small molecule discovery
fog sky bridge grass water
genomics & phylogenetics personalized recommendations
Bayesian inference and Markov chain Monte Carlo MCMC is hard → New data–parallel algorithms VCMC: Our approach and theoretical results Empirical evaluation
x y
Probability distribution
π(α, β, σ | x, y) A model is a probabilistic description of data yi ∼ N(αxi + β, σ2)
Model parameters θ = (α, β, σ) Data x = {(x1, y1), (x2, y2), . . . , (x10, y10)} Probabilistic model of data yi ∼ N(αxi + β, σ2)
Normalizing involves an integral that is often intractable π(θ | x) = π(θ)π(x | θ)
Normalizing involves an integral that is often intractable π(θ | x) = π(θ)π(x | θ)
Expectations w.r.t. the posterior = More intractable integrals Eπ[f ] =
f (θ)π(θ | x)dθ (These are statistics that distill information about the posterior)
Given a finite set of samples θ1, θ2, . . . , θT ∼ π(θ | x)
x y x y
Given a finite set of samples θ1, θ2, . . . , θT ∼ π(θ | x) Estimate an intractable expectation as a sum: Eπ[f ] =
f (θ)π(θ | x)dθ ≈ 1 T
T
f (θt)
x y x y
Given a finite set of samples θ1, θ2, . . . , θT ∼ π(θ | x) Estimate an intractable expectation as a sum: Eπ[f ] =
f (θ)π(θ | x)dθ ≈ 1 T
T
f (θt) i.e., replace a distribution with samples from it:
x y x y
Widely used class of sampling algorithms Sample by simulating a Markov chain (biased random walk) whose stationary distribution (after convergence) is the posterior θ1, θ2, . . . , θT ∼ π(θ | x) Use samples for Monte Carlo integration Eπ[f ] =
f (θ)π(θ | x)dθ ≈ 1 T
T
f (θt)
Bayesian inference and Markov chain Monte Carlo MCMC is hard → New data–parallel algorithms VCMC: Our approach and theoretical results Empirical evaluation
◮ Serial, iterative algorithm for generating samples ◮ Slow for two reasons:
(1) Large number of iterations required to converge (2) Each iteration depends on the entire dataset
◮ Most innovation in MCMC has targeted (1) ◮ Recent threads of work target (2)
posterior
likelihood
J
posterior
likelihood
J
◮ Partition the data as x(1), . . . , x(J) across J cores ◮ The jth core samples from a distribution proportional to the
jth sub-posterior (a ‘piece’ of the full posterior)
◮ Aggregate the sub-posterior samples to form approximate full
posterior samples
posterior
likelihood
J
Sub-posterior density estimation (Neiswanger et al, UAI 2014) Weierstrass samplers (Wang & Dunson, 2013) Weighted averaging of sub-posterior samples
◮ Consensus Monte Carlo (Scott et al, Bayes 250, 2013) ◮ Variational Consensus Monte Carlo (Rabinovich et al, NIPS 2015)
0.5
0.5 Aggregate( , )
0.58
0.42 Aggregate( , )
Aggregate( , )
◮ Weights are inverse covariance matrices ◮ Motivated by Gaussian assumptions ◮ Designed at Google for the MapReduce framework
Bayesian inference and Markov chain Monte Carlo MCMC is hard → New data–parallel algorithms VCMC: Our approach and theoretical results Empirical evaluation
Goal: Choose the aggregation function to best approximate the target distribution Method: Convex optimization via variational Bayes
Goal: Choose the aggregation function to best approximate the target distribution Method: Convex optimization via variational Bayes F = aggregation function qF = approximate distribution L (F)
= EqF [log π (X, θ)]
+ H [qF]
entropy
Goal: Choose the aggregation function to best approximate the target distribution Method: Convex optimization via variational Bayes F = aggregation function qF = approximate distribution ˜ L (F)
= EqF [log π (X, θ)]
+ ˜ H [qF]
relaxed entropy
Goal: Choose the aggregation function to best approximate the target distribution Method: Convex optimization via variational Bayes F = aggregation function qF = approximate distribution ˜ L (F)
= EqF [log π (X, θ)]
+ ˜ H [qF]
relaxed entropy
No mean field assumption
Aggregate( , )
◮ Optimize over weight matrices (⋆) ◮ Restrict to valid solutions when parameter vectors constrained
Under mild structural assumptions, we can choose ˜ H [qF] = c0 + 1 K
K
hk (F) , with each hk a concave function of F such that H [qF] ≥ ˜ H [qF] . We therefore have L (F) ≥ ˜ L (F) .
Under mild structural assumptions, the relaxed variational Bayes
˜ L (F) = EqF [log π (X, θ)] + ˜ H [qF] is concave in F.
Bayesian inference and Markov chain Monte Carlo MCMC is hard → New data–parallel algorithms VCMC: Our approach and theoretical results Empirical evaluation
◮ Compare 3 aggregation strategies:
◮ Uniform average ◮ Gaussian-motivated weighted average (CMC) ◮ Optimized weighted average (VCMC)
◮ For each algorithm A, report approximation error of some
expectation Eπ[f ], relative to serial MCMC ǫA (f ) = |EA [f ] − EMCMC [f ]| |EMCMC [f ]|
◮ Preliminary speedup results
#data = 100, 000, d = 300 First moment estimation error, relative to serial MCMC (Error truncated at 2.0)
Normal-inverse Wishart model #data = 100, 000, #dim = 100 = ⇒ 5, 050 parameters (L) First moment estimation error (R) Eigenvalue estimation error
Error relative to serial MCMC, for cluster comembership probabilities of pairs of test data points
Initialize VCMC with CMC weights (inverse covariance matrices)
VCMC speedup is approximately linear
Contributions
◮ Convex optimization framework for Consensus Monte Carlo ◮ Structured aggregation accounting for constrained parameters ◮ Entropy relaxation ◮ Empirical evaluation
Future work
◮ More structured and complex (latent variable) models ◮ Alternate posterior factorizations and aggregation schemes
We’d love to hear about your Bayesian inference problems!
5 10 25 50 100 Number of cores 0.0 0.5 1.0 1.5 2.0 Error First Uniform Gaussian VCMC 5 10 25 50 100 Number of cores 0.0 0.5 1.0 1.5 2.0 Second (Mixed) Subposteriors 5 10 25 50 100 Number of cores 0.0 0.5 1.0 1.5 2.0 Error First Uniform Gaussian VCMC 5 10 25 50 100 Number of cores 0.0 0.5 1.0 1.5 2.0 Second (Mixed) Partial posteriors
Figure: High-dimensional probit regression (d = 300). Moment approximation error for the uniform and Gaussian averaging baselines and VCMC, relative to serial MCMC, for (left) subposteriors and (right) partial posteriors. We assessed three groups of functions: first moments, with f (β) = βj for 1 ≤ j ≤ d; pure second moments, with f (β) = β2
j for
1 ≤ j ≤ d; and mixed second moments, with f (β) = βiβj for 1 ≤ i < j ≤ d. For brevity, results for pure second moments are relegated to the supplement. Relative errors are truncated to 2 in all cases.
25 50 100 Number of cores 0.0 0.1 0.2 0.3 0.4 Error First
Uniform Gaussian VCMC
25 50 100 Number of cores 0.0 0.1 0.2 0.3 0.4 Second (Pure) 25 50 100 Number of cores 0.0 0.1 0.2 0.3 0.4 Second (Mixed)
Figure: High-dimensional normal-inverse Wishart model (d = 100). (Far left, left, right) Moment approximation error for the uniform and Gaussian averaging baselines and VCMC, relative to serial MCMC. Letting ρj denote the jth largest eigenvalue of Λ−1, we assessed three groups of functions: first moments, with f (Λ) = ρj for 1 ≤ j ≤ d; pure second moments, with f (Λ) = ρ2
j for 1 ≤ j ≤ d; and mixed second
moments, with f (Λ) = ρiρj for 1 ≤ i < j ≤ d. (Far right) Graph of error in estimating E [ρj] as a function of j (where ρ1 ≥ ρ2 ≥ · · · ≥ ρd).
Figure: Mixture of Gaussians (d = 8, L = 8). Expectation approximation error for the uniform and Gaussian baselines and VCMC. We report the median error, relative to serial MCMC, for cluster comembership probabilities of pairs of test data points, for (left) σ = 1 and (right) σ = 2, where we run the VCMC optimization procedure for 50 and 200 iterations, respectively. When σ = 2, some comembership probabilities are estimated poorly by all methods; we therefore only use the 70% of comembership probabilities with the smallest errors across all the methods.
Figure: Error versus timing and speedup measurements. (Left) VCMC error as a function of number of seconds of optimization. The cost of
MCMC—particularly since our optimization scheme only needs small batches of samples and can therefore operate concurrently with the
CMC with Gaussian averaging (small markers) and VCMC (large markers). In this case, the cost of optimization is small enough that a near linear speedup is achieved.