Bayesian Experimental Design for Large Scale Signal Acquisition - - PowerPoint PPT Presentation

Bayesian Experimental Design for Large Scale Signal Acquisition Optimization

Matthias Seeger

Laboratory for Probabilistic Machine Learning Ecole Polytechnique Fédérale de Lausanne http://lapmal.epfl.ch/

9/12/2013

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 1 / 20

Motivation

Magnetic Resonance Imaging

⊕ Extremely versatile ⊕ Noninvasive, no ionizing radiation ⊖ Very expensive ⊖ Long scan times: Major limiting factor

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 2 / 20

Motivation

Magnetic Resonance Imaging

Faster scans by undersampled reconstruction Which fast designs give best images?

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 2 / 20

Motivation

Magnetic Resonance Imaging

Faster scans by undersampled reconstruction Which fast designs give best images?

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 2 / 20

Motivation

Image Reconstruction

Ideal Image u Reconstruction Measurement Design

y X u

Data y

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 3 / 20

Motivation

Sampling Optimization

Reconstruction

y X u

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 4 / 20

Bayesian Experimental Design

Bayesian Experimental Design

Posterior: Uncertainty in reconstruction Experimental design: Find poorly determined directions Sequential search with interjacent partial measurements

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 5 / 20

Bayesian Experimental Design

Bayesian Experimental Design

Posterior: Uncertainty in reconstruction Experimental design: Find poorly determined directions Sequential search with interjacent partial measurements

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 5 / 20

Bayesian Experimental Design

Maximizing Information Gain

Score design extension X ∗ by information gain: I(X ∗) = I(y∗, u|y) = H[P(u|y)]

• Before

− H[P(u|y∗, y)]

• After

Most work: Combinatorial aspects

Assume: I(X ∗) tractable to compute Assume: I(X ∗) cheap to compute (many X ∗) Simple greedy forward works well in practice . . .

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 6 / 20

Bayesian Experimental Design

Maximizing Information Gain

Score design extension X ∗ by information gain: I(X ∗) = I(y∗, u|y) = H[P(u|y)]

• Before

− H[P(u|y∗, y)]

• After

Most work: Combinatorial aspects

Assume: I(X ∗) tractable to compute Assume: I(X ∗) cheap to compute (many X ∗) Simple greedy forward works well in practice . . .

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 6 / 20

Bayesian Experimental Design

Maximizing Information Gain

Score design extension X ∗ by information gain: I(X ∗) = I(y∗, u|y) = H[P(u|y)]

• Before

− H[P(u|y∗, y)]

• After

Most work: Combinatorial aspects

Assume: I(X ∗) tractable to compute Assume: I(X ∗) cheap to compute (many X ∗) Simple greedy forward works well in practice . . .

So is it . . . ?

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 6 / 20

Bayesian Experimental Design

Challenges

I(X ∗) = H[P(u|y)] − H[P(u|y∗, y) Assume: I(X ∗) tractable to compute. Only if P(u|y) Gaussian . . .

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 7 / 20

Bayesian Experimental Design

Image Statistics

Whatever images are . . . they are not Gaussian!

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 8 / 20

Bayesian Experimental Design

Challenges

I(X ∗) = H[P(u|y)] − H[P(u|y∗, y) Assume: I(X ∗) tractable to compute? No: Needs approximate inference Assume: I(X ∗) cheap to compute (many X ∗).

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 9 / 20

Bayesian Experimental Design

Size Does Matter

1

Global covariances Scores I(X ∗) need full CovP[u|y]

2

Massive scale R131072 (just one slice).

3

Many times Posterior after each design extension

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 10 / 20

Bayesian Experimental Design

Challenges

I(X ∗) = H[P(u|y)] − H[P(u|y∗, y) Assume: I(X ∗) tractable to compute? No: Needs approximate inference Assume: I(X ∗) cheap to compute (many X ∗)? No: Needs new algorithms and high performance computing

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 11 / 20

Variational Bayesian Inference

Variational Bayesian Inference

Approximate inference for non-Gaussian models Computations driven by Gaussian inference

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 12 / 20

Variational Bayesian Inference

Variational Bayesian Inference

P(u|y) = P(y|u) × P(u) P(y) Variational Inference Approximation Write intractable integration as optimization Relax to tractable optimization problem

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 13 / 20

Variational Bayesian Inference

Variational Bayesian Inference

P(u|y) = P(y|u) × P(u) P(y) Variational Relaxation: Bound the master function − log P(y) = − log

• P(u, y) du ≤ 1

2 min

γ min u∗ φ(u∗, γ)

Approximate posterior P(u|y) by Gaussian Integration ⇒ Convex optimization

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 13 / 20

Variational Bayesian Inference

Variational Bayesian Inference

P(u|y) = P(y|u) × P(u) P(y) Variational Relaxation: Bound the master function − log P(y) = − log

• P(u, y) du ≤ 1

2 min

γ min u∗ φ(u∗, γ)

Approximate posterior P(u|y) by Gaussian Integration ⇒ Convex optimization

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 13 / 20

Variational Bayesian Inference

No Inference Without . . .

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 14 / 20

Variational Bayesian Inference

Double Loop Algorithm

Double loop algorithm Inner loop optimization: Standard MAP Estimation Outer loop update: Gaussian Variances

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 15 / 20

Variational Bayesian Inference

Double Loop Algorithm

Double loop algorithm Inner loop optimization: Standard MAP Estimation Outer loop update: Gaussian Variances

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 15 / 20

Variational Bayesian Inference

Tricks of the Trade

Monte Carlo Gaussian variances: Perturb&MAP

Papandreou, Yuille, NIPS 2010

Inner loop: Fast first-order MAP solvers Warmstarting variational optimization: Small changes after each (X ∗, y∗)

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 16 / 20

Variational Bayesian Inference

Tricks of the Trade

Monte Carlo Gaussian variances: Perturb&MAP

Papandreou, Yuille, NIPS 2010

Inner loop: Fast first-order MAP solvers Warmstarting variational optimization: Small changes after each (X ∗, y∗)

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 16 / 20

Experimental Results

Optimizing Cartesian MRI

Bayes Optim. VD Random Low Pass

Seeger et.al., MRM 63(1), 2010 Lustig, Donoho, Pauli, MRM 58(6), 2007 Common MRI practice Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 17 / 20

Experimental Results

Experimental Results: Test Set Errors

80 100 120 140 160 1 2 3 4 5 6 7 Number phase encodes L2 reconstruction error axial short TE 80 100 120 140 160 Number phase encodes axial long TE 1 2 3 4 5 6 7 L2 reconstruction error sagittal short TE Low Pass VD Random Bayes Optim sagittal long TE

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 18 / 20

Outlook

Large Scale Bayesian Inference

Advanced Bayesian experimental design

Signal acquisition optimization

Computer Vision

Hierarchically structured image priors

Ko, Seeger, ICML 2012

Learning Image Models (fields of experts, . . . ) Bayesian dictionary learning Intelligent user interfaces (Bayesian active learning)

Advanced variational inference

Speeding up expectation propagation

Seeger, Nickisch, AISTATS 2011

Generic framework

You can do MAP estimation efficiently? You can do variational Bayesian inference!

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 19 / 20

Outlook

Large Scale Bayesian Inference

Advanced Bayesian experimental design

Signal acquisition optimization

Computer Vision

Hierarchically structured image priors

Ko, Seeger, ICML 2012

Learning Image Models (fields of experts, . . . ) Bayesian dictionary learning Intelligent user interfaces (Bayesian active learning)

Advanced variational inference

Speeding up expectation propagation

Seeger, Nickisch, AISTATS 2011

Generic framework

You can do MAP estimation efficiently? You can do variational Bayesian inference!

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 19 / 20

Outlook

People& Code

glm-ie: Toolbox by Hannes Nickisch mloss.org/software/view/269/ Generalized sparse linear models MAP reconstruction and variational Bayesian inference (double loop algorithm for super-Gaussian bounding) Matlab 7.x, GNU Octave 3.2.x Hannes Nickisch (now Philips Research, Hamburg) Rolf Pohmann, Bernhard Schölkopf (MPI Tübingen) Young Jun Ko Emtiyaz Khan

Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 20 / 20