Approximate Inference Part 1 of 2 Tom Minka Microsoft Research, - PowerPoint PPT Presentation

Approximate Inference Part 1 of 2 Tom Minka Microsoft Research, Cambridge, UK Machine Learning Summer School 2009 http://mlg.eng.cam.ac.uk/mlss09/ 1

Bayesian paradigm • Consistent use of probability theory for representing unknowns (parameters, latent variables, missing data) 2

Bayesian paradigm • Bayesian posterior distribution summarizes what we’ve learned from training data and prior knowledge • Can use posterior to: • Can use posterior to: – Describe training data – Make predictions on test data – Incorporate new data (online learning) • Today’s question: How to efficiently represent and compute posteriors? 3

Factor graphs • Shows how a function of several variables can be factored into a product of simpler functions • f(x,y,z) = (x+y)(y+z)(x+z) • f(x,y,z) = (x+y)(y+z)(x+z) • Very useful for representing posteriors 4

Example factor graph = p ( x | m ) N ( x ; m , 1 ) i i 5

Two tasks • Modeling – What graph should I use for this data? • Inference – Given the graph and data, what is the mean – Given the graph and data, what is the mean of x (for example)? – Algorithms: • Sampling • Variable elimination • Message-passing (Expectation Propagation, Variational Bayes, …) 6

A (seemingly) intractable problem A (seemingly) intractable problem 7

Clutter problem • Want to estimate x given multiple y’s 8

Exact posterior exact ,D) p(x,D -1 0 1 2 3 4 x 9

Representing posterior distributions Sampling Deterministic approximation Good for complex, Good for simple, multi-modal distributions smooth distributions Fast, but unpredictable accuracy 10 Slow, but predictable accuracy

Deterministic approximation Laplace’s method • Bayesian curve fitting, neural networks (MacKay) • Bayesian PCA (Minka) Variational bounds • Bayesian mixture of experts (Waterhouse) • Mixtures of PCA (Tipping, Bishop) • Factorial/coupled Markov models (Ghahramani, Jordan, Williams) 11

Moment matching Another way to perform deterministic approximation • Much higher accuracy on some problems Assumed-density filtering (1984) Loopy belief propagation (1997) Expectation Propagation (2001) 12

Today • Moment matching (Expectation Propagation) Tomorrow • Variational bounds (Variational Message Passing) 13

Best Gaussian by moment matching exact bestGaussian ) p(x,D) -1 0 1 2 3 4 x 14

Strategy • Approximate each factor by a Gaussian in x 15

Approximating a single factor 16

(naïve) f i ( x ) × × \ x \ x q i q i ( ( ) ) = p ( x ) 17

(informed) f i ( x ) × × \ x \ x q i q i ( ( ) ) = p ( x ) 18

Single factor with Gaussian context 19

Gaussian multiplication formula 20

Approximation with narrow context 21

Approximation with medium context 22

Approximation with wide context 23

Two factors x x Message passing 24

Three factors x x Message passing 25

Message Passing = Distributed Optimization • Messages represent a simpler distribution � � � � that approximates � � � � – A distributed representation • Message passing = optimizing � to fit � • Message passing = optimizing � to fit � – � stands in for � when answering queries • Choices: – What type of distribution to construct (approximating family) – What cost to minimize (divergence measure) 26

Distributed divergence minimization • Write p as product of factors: • Approximate factors one by one: • Multiply to get the approximation: 27

Gaussian found by EP ep exact bestGaussian p(x,D) p( -1 0 1 2 3 4 x 28

Other methods vb laplace exact p(x,D) p( -1 0 1 2 3 4 x 29

Accuracy Posterior mean: exact = 1.64864 ep = 1.64514 laplace = 1.61946 vb = 1.61834 vb = 1.61834 Posterior variance: exact = 0.359673 ep = 0.311474 laplace = 0.234616 vb = 0.171155 30

Cost vs. accuracy 200 points 20 points Deterministic methods improve with more data (posterior is more Gaussian) Sampling methods do not 31

Censoring example • Want to estimate x given multiple y’s = = p ( x ) N ( x ; 0 , 100 ) p ( y | x ) N ( y ; x , 1 ) i − − ∞ ∞ ∫ t ∫ > = + p (| y | t | x ) N ( y ; x , 1 ) dy N ( y ; x , 1 ) dy i − ∞ t 32

Time series problems Time series problems 33

Example: Tracking Guess the position of an object given noisy measurements y 2 x 2 x x 1 3 y 4 y y 1 3 x 4 Object 34

Factor graph x x x x 3 1 2 4 y y y y 1 2 3 4 = + ν x x e.g. (random walk) − 1 t t t = + y x noise t t want distribution of x’s given y’s 35

Approximate factor graph x x x x 3 1 2 4 36

Splitting a pairwise factor x x 1 2 x x 1 2 37

Splitting in context x x 3 2 x x 3 2 38

Sweeping through the graph x x x x 3 1 2 4 39

Sweeping through the graph x x x x 3 1 2 4 40

Sweeping through the graph x x x x 3 1 2 4 41

Sweeping through the graph x x x x 3 1 2 4 42

Example: Poisson tracking • y t is a Poisson-distributed integer with mean exp(x t ) 43

Poisson tracking model p ( 1 x ) ~ N ( 0 , 100 ) p p ( ( x x | | x x ) ) ~ ~ N N ( ( x x , , 0 0 . . 01 01 ) ) − − t t 1 t 1 = − x p ( y | x ) exp( y x e ) / y ! t t t t t t 44

Factor graph x x x x 3 1 2 4 y y y y 1 2 3 4 x x x x 3 1 2 4 45

Approximating a measurement factor x 1 y 1 x 1 46

47

Posterior for the last state 48

49

50

EP for signal detection (Qi and Minka, 2003) • Wireless communication problem ω + φ a sin( t ) • Transmitted signal = φ • vary to encode each symbol ( a , ) φ φ i • In complex numbers: ae Im a φ Re 51

Binary symbols, Gaussian noise 1 = = − 0 • Symbols are and s 1 s 1 (in complex plane) = ω t + φ + y t a sin( ) noise • Received signal • Optimal detection is easy in this case • Optimal detection is easy in this case y t 0 1 s s 52

Fading channel • Channel systematically changes amplitude and phase: = + y x s noise t t t • • = transmitted symbol = transmitted symbol s s t x • = channel multiplier (complex number) t x • changes over time y 1 t x t s t 0 x t s 53

Differential detection • Use last measurement to estimate state: ≈ x y 1 / s − − t t t 1 • State estimate is noisy – can we do better? better? y y t − t 1 − y − 54 t 1

Factor graph s s s s 3 1 2 4 y y y y y y y y 1 2 4 3 x x x x 3 1 2 4 Symbols can also be correlated (e.g. error-correcting code) Channel dynamics are learned from training data (all 1’s) 55

56

57

Splitting a transition factor 58

Splitting a measurement factor 59

On-line implementation • Iterate over the last δ measurements • Previous measurements act as prior • Results comparable to particle filtering, but much faster 60

61

Classification problems Classification problems 62

Spam filtering by linear separation Not spam Spam Choose a boundary that will generalize to new data 63

Linear separation Minimum training error solution (Perceptron) Too arbitrary – won’t generalize well 64

Linear separation Maximum-margin solution (SVM) 65 Ignores information in the vertical direction

Linear separation Bayesian solution (via averaging) 66 Has a margin, and uses information in all dimensions

Geometry of linear separation Separator is any vector w such that: > T x w 0 (class 1) i < T x w 0 (class 2) i = = (sphere) (sphere) w w 1 1 This set has an unusual shape SVM: Optimize over it Bayes: Average over it 67

Performance on linear separation EP Gaussian approximation to posterior 68

Factor graph = p ( w ) N ( w ; 0 , I ) 69

Computing moments w = \ i \ i \ i q ( ) N ( w ; m , V ) 70

Computing moments 71

Time vs. accuracy A typical run on the 3-point problem Error = distance to true mean of w Billiard = Monte Carlo sampling (Herbrich et al, 2001) Opper&Winther’s algorithms: MF = mean-field theory TAP = cavity method (equiv to Gaussian EP for this problem) 72

Gaussian kernels • Map data into high-dimensional space so that 73

Bayesian model comparison • Multiple models M i with prior probabilities p(M i ) • Posterior probabilities: • For equal priors, models are compared using model evidence: 74

Highest-probability kernel 75

Margin-maximizing kernel 76