Approximate Inference Part 1 of 2 Tom Minka Microsoft Research, - - PowerPoint PPT Presentation

Approximate Inference

Part 1 of 2 Tom Minka

Microsoft Research, Cambridge, UK Machine Learning Summer School 2009 http://mlg.eng.cam.ac.uk/mlss09/

1

Bayesian paradigm

• Consistent use of probability theory for

representing unknowns (parameters, latent variables, missing data)

2

Bayesian paradigm

• Bayesian posterior distribution

summarizes what we’ve learned from training data and prior knowledge

• Can use posterior to:
• Can use posterior to:

– Describe training data – Make predictions on test data – Incorporate new data (online learning)

• Today’s question: How to efficiently

represent and compute posteriors?

3

Factor graphs

• Shows how a function of several variables

can be factored into a product of simpler functions

• f(x,y,z) = (x+y)(y+z)(x+z)
• f(x,y,z) = (x+y)(y+z)(x+z)
• Very useful for representing posteriors

4

Example factor graph

) 1 , ; ( ) | ( m x N m x p

i i

=

5

Two tasks

• Modeling

– What graph should I use for this data?

• Inference

– Given the graph and data, what is the mean – Given the graph and data, what is the mean

• f x (for example)?

– Algorithms:

• Sampling
• Variable elimination
• Message-passing (Expectation Propagation,

Variational Bayes, …)

6

A (seemingly) intractable problem A (seemingly) intractable problem

7

Clutter problem

• Want to estimate x given multiple y’s

8

Exact posterior

,D) exact

• 1

1 2 3 4 x p(x,D

9

Representing posterior distributions

Sampling Deterministic approximation

Good for complex, multi-modal distributions Slow, but predictable accuracy Good for simple, smooth distributions Fast, but unpredictable accuracy 10

Deterministic approximation

Laplace’s method

• Bayesian curve fitting, neural

networks (MacKay)

• Bayesian PCA (Minka)

Variational bounds

• Bayesian mixture of experts (Waterhouse)
• Mixtures of PCA (Tipping, Bishop)
• Factorial/coupled Markov models

(Ghahramani, Jordan, Williams)

11

Moment matching

Another way to perform deterministic approximation

• Much higher accuracy on some

problems

Expectation Propagation Assumed-density filtering Loopy belief propagation

(1997) (1984) (2001)

12

Today

• Moment matching

(Expectation Propagation) Tomorrow

• Variational bounds

(Variational Message Passing)

13

Best Gaussian by moment matching

) exact bestGaussian

• 1

1 2 3 4 x p(x,D)

14

Strategy

• Approximate each factor by a Gaussian in

x

15

Approximating a single factor

16

×

) (x fi

) (

\ x

q i

(naïve)

× =

) (

\ x

q i

) (x p

17

×

(informed)

) (x fi

) (

\ x

q i

× =

) (x p

) (

\ x

q i

18

Single factor with Gaussian context

19

Gaussian multiplication formula

20

Approximation with narrow context

21

Approximation with medium context

22

Approximation with wide context

23

Two factors

x x

Message passing

24

Three factors

x x

Message passing

25

Message Passing = Distributed Optimization

• Messages represent a simpler distribution

that approximates

– A distributed representation

• Message passing = optimizing to fit
• Message passing = optimizing to fit

– stands in for when answering queries

• Choices:

– What type of distribution to construct (approximating family) – What cost to minimize (divergence measure)

26

• Write p as product of factors:

Distributed divergence minimization

• Approximate factors one by one:
• Multiply to get the approximation:

27

Gaussian found by EP

p(x,D) ep exact bestGaussian

• 1

1 2 3 4 x p(

28

Other methods

p(x,D) vb laplace exact

• 1

1 2 3 4 x p(

29

Accuracy

Posterior mean: exact = 1.64864 ep = 1.64514 laplace = 1.61946 vb = 1.61834 vb = 1.61834 Posterior variance: exact = 0.359673 ep = 0.311474 laplace = 0.234616 vb = 0.171155

30

Cost vs. accuracy

20 points 200 points Deterministic methods improve with more data (posterior is more Gaussian) Sampling methods do not

31

Censoring example

• Want to estimate x given multiple y’s

) 1 , ; ( ) | ( x y N x y p

i

= ) 100 , ; ( ) ( x N x p =

− ∞

∫ ∫

− ∞ − ∞

+ = >

t t i

dy x y N dy x y N x t y p ) 1 , ; ( ) 1 , ; ( ) | | (|

32

Time series problems Time series problems

33

Example: Tracking

Guess the position of an object given noisy measurements

1

y

4

y

Object

1

x

2

x

3

x

4

x

2

y

3

y

34

Factor graph

1

x

2

x

3

x

4

x

1

y

2

y

3

y

4

y

t t t

ν x x + =

−1

noise + =

t t

x y

(random walk) e.g. want distribution of x’s given y’s

35

Approximate factor graph

1

x

2

x

3

x

4

x

36

Splitting a pairwise factor

1

x

2

x

1

x

2

x

37

Splitting in context

2

x

3

x

2

x

3

x

38

Sweeping through the graph

1

x

2

x

3

x

4

x

39

Sweeping through the graph

1

x

2

x

3

x

4

x

40

Sweeping through the graph

1

x

2

x

3

x

4

x

41

Sweeping through the graph

1

x

2

x

3

x

4

x

42

Example: Poisson tracking

• yt is a Poisson-distributed integer with

mean exp(xt)

43

Poisson tracking model

) 01 . , ( ~ ) | ( x N x x p ) 100 , ( ~ ) ( 1 N x p ) 01 . , ( ~ ) | (

1 1 − − t t t

x N x x p

! / ) exp( ) | (

t x t t t t

y e x y x y p

t

− =

44

Factor graph

1

x

2

x

3

x

4

x

1

y

2

y

3

y

4

y

1

x

2

x

3

x

4

x

45

Approximating a measurement factor

1

x

1

y

1

x

46

47

Posterior for the last state

48

49

50

EP for signal detection

• Wireless communication problem
• Transmitted signal =
• vary to encode each symbol

) sin( φ ω + t a

φ

) , ( φ a

(Qi and Minka, 2003)

• In complex numbers:

φ i

ae

Re Im

φ

a

51

Binary symbols, Gaussian noise

• Symbols are and

(in complex plane)

• Received signal
• Optimal detection is easy in this case

noise ) sin( + + = φ ωt a yt 1

1 =

s 1 − = s

• Optimal detection is easy in this case

t

y

s

1

s

52

Fading channel

• Channel systematically changes amplitude

and phase:

• = transmitted symbol

noise + =

t t t

s x y

s

• = transmitted symbol
• = channel multiplier (complex number)
• changes over time

t

x

t

y

s xt

1

s xt

t

x

t

s

53

Differential detection

• Use last measurement to estimate state:
• State estimate is noisy – can we do

better?

1 1 / − −

≈

t t t

s y x

better?

t

y

1 −

−

t

y

1 − t

y

54

Factor graph

y y

y

y

1

s

2

s

3

s

4

s

1

y

2

y

3

y

4

y

1

x

2

x

3

x

4

x

Channel dynamics are learned from training data (all 1’s) Symbols can also be correlated (e.g. error-correcting code)

55

56

57

Splitting a transition factor

58

Splitting a measurement factor

59

On-line implementation

• Iterate over the last δ measurements
• Previous measurements act as prior
• Results comparable to particle filtering, but

much faster

60

61

Classification problems Classification problems

62

Spam filtering by linear separation

Spam Not spam Choose a boundary that will generalize to new data

63

Linear separation

Minimum training error solution (Perceptron)

Too arbitrary – won’t generalize well

64

Linear separation

Maximum-margin solution (SVM)

Ignores information in the vertical direction

65

Linear separation

Bayesian solution (via averaging)

Has a margin, and uses information in all dimensions

66

Geometry of linear separation

Separator is any vector w such that:

>

i Tx

w

(class 1)

<

i Tx

w

(class 2)

1 = w

(sphere)

1 = w

(sphere) This set has an unusual shape SVM: Optimize over it Bayes: Average over it

67

Performance on linear separation

EP Gaussian approximation to posterior

68

Factor graph

) , ; ( ) ( I w w N p =

69

Computing moments

) , ; ( ) (

\ \ \ i i i

N q V m w w =

70

Computing moments

71

Time vs. accuracy

A typical run on the 3-point problem Error = distance to true mean of w Billiard = Monte Carlo sampling (Herbrich et al, 2001) Opper&Winther’s algorithms: MF = mean-field theory TAP = cavity method (equiv to Gaussian EP for this problem)

72

Gaussian kernels

• Map data into high-dimensional space so

that

73

Bayesian model comparison

• Multiple models Mi with prior probabilities

p(Mi)

• Posterior probabilities:
• For equal priors, models are compared

using model evidence:

74

Highest-probability kernel

75

Margin-maximizing kernel

76

Bayesian feature selection

Synthetic data where 6 features are relevant (out of 20) Bayes picks 6 Margin picks 13

77

EP versus Monte Carlo

• Monte Carlo is general but expensive

– A sledgehammer

• EP exploits underlying simplicity of the

problem (if it exists) problem (if it exists)

• Monte Carlo is still needed for complex

problems (e.g. large isolated peaks)

• Trick is to know what problem you have

78

Software for EP

• Bayes Point Machine toolbox

http://research.microsoft.com/~minka/papers/ep/bpm/

• Sparse Online Gaussian Process toolbox

http://www.kyb.tuebingen.mpg.de/bs/people/csatol/ogp/index.html

• Infer.NET

http://research.microsoft.com/infernet

79

Further reading

• EP bibliography
• EP bibliography

http://research.microsoft.com/~minka/papers/ep/roadmap.html

• EP quick reference

http://research.microsoft.com/~minka/papers/ep/minka-ep- quickref.pdf

80

Tomorrow

• Variational Message Passing
• Divergence measures
• Comparisons to EP

81