Sequential Monte Carlo Algorithms for Bayesian Sequential Design Dr - PowerPoint PPT Presentation

Introduction SMC A1 - Comparing Utilities A2 - Mutual Information Utility A3 - GPU References Sequential Monte Carlo Algorithms for Bayesian Sequential Design Dr Chris Drovandi Queensland University of Technology c.drovandi@qut.edu.au Collaborators: James McGree, Tony Pettitt, Gentry White Acknowledgements: Australian Research Council Discovery Grant and organisers of MCMski January 6, 2014 Chris Drovandi MCMski 2014, Chamonix, France

Introduction SMC A1 - Comparing Utilities A2 - Mutual Information Utility A3 - GPU References Sequential Experimental Design y 1: d 1: y 1: Adaptive decisions as new data are collected t , t ). t d 1: t . U is utility function More robust to parameter and model uncertainty Natural to use Bayesian framework. Posterior becomes new prior Next decision obtained by looking forward to all future decisions (backward induction) Simplified by myopic design (one-at-a-time) Next design point d t +1 = arg max U ( d | collected data at design Chris Drovandi MCMski 2014, Chamonix, France

Introduction SMC A1 - Comparing Utilities A2 - Mutual Information Utility A3 - GPU References Why SMC for Bayesian sequential design? Much more efficient than MCMC. Simple re-weighting step to incorporate new information. Parallel implementation possible (e.g. GPU) Increase in efficiency allows comparisons of utility functions Convenient estimation of important Bayesian utility functions (e.g. mutual information) Decisions in real time? Chris Drovandi MCMski 2014, Chamonix, France

y 1: d 1: y 1: d 1: m | t , t ) = f ( t | θ m , t ) π ( θ m ) / Z m , t , for t = 1 , . . . , T . Introduction SMC A1 - Comparing Utilities A2 - Mutual Information Utility A3 - GPU References y 1: d 1: t (independent) data up to t , t design points up to t , θ m SMC For One static Model m y 1: d 1: y 1: d 1: t | m , t ) = Z m , t = t | θ m , t ) π ( θ m ) d θ m . Sample from sequence of targets m Data annealing here π t ( θ parameter for model m . f is likelihood, π prior, π t posterior � f ( f ( θ SMC: Generate a weighted sample (particles) for each target in the sequence via steps Reweight: particles as data comes in (efficient) Resample: when ESS small Mutation: diversify duplicated particles (can be efficient) Chris Drovandi MCMski 2014, Chamonix, France

i y 1: t t } N i Introduction SMC A1 - Comparing Utilities A2 - Mutual Information Utility A3 - GPU References t , d t +1 ) . SMC For One STATIC Model m (Algorithm) Chopin (2002) Have current particles { W i t , θ i =1 based on data Re-weight step to included y t +1 W i t +1 ∝ W i t f ( y t +1 | θ Check effective sample size: ESS = 1 / � N t +1 ) 2 i =1 ( W i If ESS > E (e.g. E = N / 2) go back to re-weight step for next observation If ESS < E do the following Resample proportional to weights. Duplicates good particles Mutation: Move all particles via MCMC kernel say R times (adaptive proposal) Chris Drovandi MCMski 2014, Chamonix, France

y 1: y 1: d 1: t , d t +1 ) = t , t ) d θ . Introduction SMC A1 - Comparing Utilities A2 - Mutual Information Utility A3 - GPU References SMC Estimate of Evidence Del Moral et al (2006) i t , d t +1 ) . It can be shown � Z t +1 / Z t = f ( y t +1 | f ( y t +1 | θ , d t +1 ) π ( θ | θ Using SMC particles to approximate posterior at t gives estimator N � W i Z t +1 / Z t ≈ t f ( y t +1 | θ i =1 Can then obtain approximation of Z t +1 through Z t +1 = Z t +1 Z t · · · Z 1 . Z 0 Z t Z t − 1 Z 0 Also gives estimate of posterior predictive probability of y t +1 Chris Drovandi MCMski 2014, Chamonix, France

Introduction SMC y 1: A1 - Comparing Utilities d 1: A2 - Mutual Information Utility y 1: d 1: y 1: A3 - GPU d 1: References t , t ) = t , t , d ) U ( d , z | t , t ) . Advantage 1: Efficiently comparing utilities (Drovandi et al 2013) Discrete data (binary) example Need to compute utility for all possible d (then for all t = 1 , . . . , T ) � U ( d | f ( z | z ∈{ 0 , 1 } The whole process requires the computation (sampling) of many many posterior distributions SMC (IS) to rescue. Pretend z is the ‘next’ observation collected at design d . Simple re-weight to incorporate this observation. Use weighted sample to estimate U ( d , z ). SMC also provides estimate of posterior predictive. There may be different choices for U ( d , z ), want to compare. Need to simulate the design many times. Chris Drovandi MCMski 2014, Chamonix, France

Introduction SMC A1 - Comparing Utilities A2 - Mutual Information Utility A3 - GPU References The Design Problem Estimating Maximum tolerated dose, minimum effective dose (clinical trials) E [ Y t ] = g − 1 ( η t ) where d λ t − 1 η t = θ 0 + θ 1 , λ where d t is the dose assigned to the t th subject. Y t ∼ Binary ( E [ Y t ]). Uninformative prior λη ∗ − θ 0 � � π ( θ 0 , θ 1 , λ ) ∝ N ( θ 0 ; 0 , 100) N ( θ 1 ; 0 , 100) U ( λ ; 0 , 1)1 + 1 > 0 , θ 1 Objective is precise estimation of � 1 /λ T λ T η ∗ − θ T � d ∗ = 0 + 1 . θ T 1 Let θ = ( θ 0 , θ 1 , λ ). Chris Drovandi MCMski 2014, Chamonix, France

y 1: d 1: t , t ) Introduction SMC A1 - Comparing Utilities A2 - Mutual Information Utility A3 - GPU References The Utility Functions Possible choices for U ( d , z | 1 Posterior precision of d ∗ (Natural choice, but posterior of d ∗ can be unstable when little information is available) 2 Kullback-Leibler Divergence between prior and posterior for θ 3 Determinant of Posterior Covariance matrix of θ 4 Hybrid utility. Utility 3 for 10 subjects. Utility 1 thereafter. Chris Drovandi MCMski 2014, Chamonix, France

Introduction SMC A1 - Comparing Utilities A2 - Mutual Information Utility A3 - GPU References Some Results 1.5 prec kld D−post hybrid T = (0,3,1) producing 1 * D 0.5 0 20 40 60 80 100 Subject Figure: Distributions of the estimated target stimulus over 10 to 100 subjects for the true parameter configuration of θ d ∗ = 0 . 538. Solid horizontal line is the true d ∗ . Shown are the 2.5%, 50% and 97.5% quantiles over the 500 runs for each utility function. Chris Drovandi MCMski 2014, Chamonix, France

Introduction SMC A1 - Comparing Utilities A2 - Mutual Information Utility A3 - GPU References Advantage 2: Estimating Difficult Utilities y 1: y 1: y 1: t , d ) = H ( M | t ) − H ( M | Z ; t , d ) . E.g. Mutual Information for Model Discrimination (Drovandi et al 2014) y 1: y 1: Have set of K proposed models M = 1 , . . . , K . Select design t ) = − H ( M | Z ; t , d ) which is equal to to maximise ability to discriminate between models Consider mutual information between model indicator M and y 1: y 1: y 1: y 1: t ) = t ) t , d ) log π ( m | t , z , d ) d µ ( z ) , predicted observation Z for y t +1 Box and Hill (1967). I ( M ; Z | Therefore U ( d | K � � U ( d | π ( m | f ( z | m , m =1 Chris Drovandi MCMski 2014, Chamonix, France

Introduction SMC A1 - Comparing Utilities A2 - Mutual Information Utility A3 - GPU References i m , t } N SMC for multiple models y 1: t , and particles of all models Effectively run an SMC algorithm for each model m = 1 , . . . , K Have set of N particles for each model { W i m , t , θ i =1 . ESS for each model m resampling and within-model updates when required Design part: use data up to t , to compute the next design d t +1 Chris Drovandi MCMski 2014, Chamonix, France

y 1: y 1: y 1: y 1: t ) = t ) t , d ) log π ( m | t , z , d ) d µ ( z ) , Introduction SMC A1 - Comparing Utilities A2 - Mutual Information Utility A3 - GPU References Estimating the Utility K � � U ( d | π ( m | f ( z | m , m =1 Borth (1975) notes difficult computation SMC to the rescue. Potential observation z at potential design point d . Pretend this the observation for y t +1 . Chris Drovandi MCMski 2014, Chamonix, France

i m , t , d ) , Introduction SMC A1 - Comparing Utilities A2 - Mutual Information Utility A3 - GPU References y 1: d 1: t , t , d ) = Estimating the Utility (cont...) Estimate predictive probability using weights w i m , t ( d , z ) = W i m , t f ( z | θ y 1: d 1: t , t , d ) . N ˆ � w i f ( z | m , m , t ( d , z ) . y 1: i =1 t , d , z ) Z m , t denotes current evidence for model m , which integrates out posterior of θ at t . Estimate evidence including ( d , z ) Z m , t ( d , z ) using log ˆ Z m , t ( d , z ) = log ˆ Z m , t + log ˆ f ( z | m , Convert to ˆ π ( m | Chris Drovandi MCMski 2014, Chamonix, France

y 1: y 1: y 1: y 1: t ) = t ) t , d ) log ˆ t , z , d ) . Introduction SMC A1 - Comparing Utilities A2 - Mutual Information Utility A3 - GPU References Estimating the Utility (cont...) Therefore estimate of utility for discrete z is K ˆ � � ˆ U ( d | π ( m | ˆ f ( z | m , π ( m | m =1 z ∈S In continuous case approximate integral via MC integration. Draw z i m , t ∼ f ( z | m , θ i m , t , d ) for i = 1 , . . . , N . Weighted sample { W i m , t , θ i m , t , z i m , t } from joint p ( z , θ | d , m , y 1 : t , d 1 : t ). Therefore estimate of utility for continuous z is ˆ U ( d | y 1 : t , d 1 : t ) = K N � � W i π ( m | y 1 : t , d 1 : t , z i π ( m | y 1 : t , d 1 : t ) ˆ m , t log ˆ m , t , d ) . m =1 i =1 Could also be used for large counts. Chris Drovandi MCMski 2014, Chamonix, France