A Fresh Look at the Bayes Theorem from Information Theory Tan - - PowerPoint PPT Presentation

A Fresh Look at the Bayes’ Theorem from Information Theory

Tan Bui-Thanh

Computational Engineering and Optimization (CEO) Group Department of Aerospace Engineering and Engineering Mechanics Institute for Computational Engineering and Sciences (ICES) The University of Texas at Austin

Babuska Series, ICES Sep 9, 2016

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 1 / 28

Outline

1

Bayesian Inversion Framework

2

Entropy

3

Relative Entropy

4

Bayes’ Theorem and Information Theory

5

Conclusions

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 2 / 28

Large-scale computation under uncertainty

Inverse electromagnetic scattering Randomness Random errors in measurements are unavoidable Inadequacy of the mathematical model (Maxwell equations)

Challenge

How to invert for the invisible shape/medium using computational electromagnetics with O

• 106

degree of freedoms?

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 3 / 28

Large-scale computation under uncertainty

Full wave form seismic inversion Randomness Random errors in seismometer measurements are unavoidable Inadequacy of the mathematical model (elastodynamics)

Challenge

How to image the earth interior using forward computational model with with O

• 109

degree of freedoms?

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 4 / 28

Inverse Shape Electromagnetic Scattering Problem

Maxwell Equations: r ⇥ E = µ@H @t , (Faraday) r ⇥ H = ✏@E @t , (Ampere) E: Electric field, H: Magnetic field, µ: permeability, ✏: permittivity

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 5 / 28

Inverse Shape Electromagnetic Scattering Problem

Maxwell Equations: r ⇥ E = µ@H @t , (Faraday) r ⇥ H = ✏@E @t , (Ampere) E: Electric field, H: Magnetic field, µ: permeability, ✏: permittivity

Forward problem (discontinuous Galerkin discretization)

d = G(x) where G maps shape parameters x to electric/magnetic field d at the measurement points

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 5 / 28

Inverse Shape Electromagnetic Scattering Problem

Maxwell Equations: r ⇥ E = µ@H @t , (Faraday) r ⇥ H = ✏@E @t , (Ampere) E: Electric field, H: Magnetic field, µ: permeability, ✏: permittivity

Forward problem (discontinuous Galerkin discretization)

d = G(x) where G maps shape parameters x to electric/magnetic field d at the measurement points

Inverse Problem

Given (possibly noise-corrupted) measurements on d, infer x?

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 5 / 28

The Bayesian Statistical Inversion Framework

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 6 / 28

The Bayesian Statistical Inversion Framework

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 6 / 28

The Bayesian Statistical Inversion Framework

Bayes Theorem

⇡post (x|d) / ⇡like (d|x) ⇥ ⇡prior (x)

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 6 / 28

Bayes theorem for inverse electromagnetic scattering

Prior knowledge: The obstacle is smooth: ⇡pr(x) / exp ✓

• Z 2π

r00(x)d✓ ◆

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 7 / 28

Bayes theorem for inverse electromagnetic scattering

Prior knowledge: The obstacle is smooth: ⇡pr(x) / exp ✓

• Z 2π

r00(x)d✓ ◆ Likelihood: Additive Gaussian noise, for example, ⇡like(d|x) / exp ✓ 1 2 kG(x) dk2

Cnoise

◆

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 7 / 28

Outline

1

Bayesian Inversion Framework

2

Entropy

3

Relative Entropy

4

Bayes’ Theorem and Information Theory

5

Conclusions

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 8 / 28

Entropy

Definition

We define the uncertainty in a random variable X distributed by 0  ⇡ (x)  1 as H (X) = Z ⇡ (x) log ⇡ (x) dx 0

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 9 / 28

Entropy

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 10 / 28

Entropy

Wiener and Shannon Kolmogorov

Copied from Sergio Verdu

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 10 / 28

Entropy

Wiener and Shannon Kolmogorov

Copied from Sergio Verdu

Wiener: “...for it belongs to the two of us equally”

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 10 / 28

Entropy

Wiener and Shannon Kolmogorov

Copied from Sergio Verdu

Wiener: “...for it belongs to the two of us equally” Shannon: “...a mathematical pun”

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 10 / 28

Entropy

Wiener and Shannon Kolmogorov

Copied from Sergio Verdu

Wiener: “...for it belongs to the two of us equally” Shannon: “...a mathematical pun” Kolmogorov: “...has no physical interpretation”

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 10 / 28

Entropy

Entropy of uniform distribution

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 11 / 28

Entropy

Entropy of uniform distribution Let U be a uniform random variable with values in X, and |X| < 1

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 11 / 28

Entropy

Entropy of uniform distribution Let U be a uniform random variable with values in X, and |X| < 1 ⇡ (u) := 1 |X| ) H (U) = log (|X|)

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 11 / 28

Entropy

Entropy of uniform distribution Let U be a uniform random variable with values in X, and |X| < 1 ⇡ (u) := 1 |X| ) H (U) = log (|X|) How uncertain is the uniform random variable?

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 11 / 28

Entropy

Entropy of uniform distribution Let U be a uniform random variable with values in X, and |X| < 1 ⇡ (u) := 1 |X| ) H (U) = log (|X|) How uncertain is the uniform random variable? H (X)  H (U)

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 11 / 28

100 years of uniform distribution

source: Christoph Aistleitner

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 12 / 28

100 years of uniform distribution

source: Christoph Aistleitner

Hermann Weyl

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 12 / 28

and Maximum entropy

Maximum entropy distribution X with known mean and variance

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 13 / 28

and Maximum entropy

Maximum entropy distribution X with known mean and variance ⇡ (x)? with maximum entropy

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 13 / 28

and Maximum entropy

Maximum entropy distribution X with known mean and variance ⇡ (x)? with maximum entropy max

π(x) H(X) =

Z ⇡(x) log(⇡(x)) dx subject to Z x⇡(x) dx = µ Z (x µ)2⇡(x) dx = 2 Z ⇡(x) dx = 1

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 13 / 28

Gaussian and Maximum entropy

Maximum entropy distribution X with known mean and variance ⇡ (x)? with maximum entropy max

π(x) H(X) =

Z ⇡(x) log(⇡(x)) dx subject to Z x⇡(x) dx = µ Z (x µ)2⇡(x) dx = 2 Z ⇡(x) dx = 1 Gaussian distribution: ⇡ (x) = N

• µ, 2

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 13 / 28

Outline

1

Bayesian Inversion Framework

2

Entropy

3

Relative Entropy

4

Bayes’ Theorem and Information Theory

5

Conclusions

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 14 / 28

Relative Entropy

Abraham Wald (1945) Harold Jeffreys (1945) D (⇡||q) := Z ⇡(x) log ✓⇡(x) q(x) ◆ dx

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 15 / 28

Kullback-Leibler divergence = Relative Entropy

Solomon Kullback (1951) Richard Leibler (1951) D (⇡||q) := Z ⇡(x) log ✓⇡(x) q(x) ◆ dx

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 16 / 28

Kullback-Leibler divergence = Relative Entropy

Solomon Kullback (1951) Richard Leibler (1951) D (⇡||q) := Z ⇡(x) log ✓⇡(x) q(x) ◆ dxdiscrete = X ⇡i log ✓⇡i qi ◆

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 16 / 28

Information Inequality

The most important inequality in information theory

D (⇡||q) 0 Can we see it easily?

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 17 / 28

Information Inequality

The most important inequality in information theory

D (⇡||q) 0 Can we see it easily?

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 17 / 28

Outline

1

Bayesian Inversion Framework

2

Entropy

3

Relative Entropy

4

Bayes’ Theorem and Information Theory

5

Conclusions

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 18 / 28

From Relative Entropy to Bayes’ Theorem

Toss n times an kth dimensional dice with the prior distribution of each face {pi}k

i=1: k

X

i=1

pi = 1

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 19 / 28

From Relative Entropy to Bayes’ Theorem

Toss n times an kth dimensional dice with the prior distribution of each face {pi}k

i=1: k

X

i=1

pi = 1 Let ni is the number of times we see face i: ni n ! pi

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 19 / 28

From Relative Entropy to Bayes’ Theorem

Toss n times an kth dimensional dice with the prior distribution of each face {pi}k

i=1: k

X

i=1

pi = 1 Let ni is the number of times we see face i: ni n ! pi What is the likelihood that these n faces also distributed by the posterior distrubtion qi:

k

X

i=1

qi = 1?

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 19 / 28

From Relative Entropy to Bayes’ Theorem

Toss n times an kth dimensional dice with the prior distribution of each face {pi}k

i=1: k

X

i=1

pi = 1 Let ni is the number of times we see face i: ni n ! pi What is the likelihood that these n faces also distributed by the posterior distrubtion qi:

k

X

i=1

qi = 1? The likelihood of {ni}k

i=1 distributed by {qi}k i=1

Πk

i=1qni i

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 19 / 28

From Relative Entropy to Bayes’ Theorem

Toss n times an kth dimensional dice with the prior distribution of each face {pi}k

i=1: k

X

i=1

pi = 1 Let ni is the number of times we see face i: ni n ! pi What is the likelihood that these n faces also distributed by the posterior distrubtion qi:

k

X

i=1

qi = 1? The likelihood of {ni}k

i=1 distributed by {qi}k i=1 (Multinomial

distribution) L := n! Πk

i=1ni!Πk i=1qni i

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 19 / 28

From Relative Entropy to Bayes’ Theorem

The likelihood of {ni}k

i=1 distributed by {qi}k i=1

L := n! Πk

i=1ni!Πk i=1qni i

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 20 / 28

From Relative Entropy to Bayes’ Theorem

The likelihood of {ni}k

i=1 distributed by {qi}k i=1

L := n! Πk

i=1ni!Πk i=1qni i

Take the log likelihood log L = log(n!) X log(ni!) + X ni log(qi)

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 20 / 28

From Relative Entropy to Bayes’ Theorem

The likelihood of {ni}k

i=1 distributed by {qi}k i=1

L := n! Πk

i=1ni!Πk i=1qni i

Take the log likelihood log L = log(n!) X log(ni!) + X ni log(qi) Stirling’s approximation log n! ⇡ n log(n) n log L = n log(n) X ni log(ni) + X ni log(qi) + X ni n | {z }

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 20 / 28

From Relative Entropy to Bayes’ Theorem

The likelihood of {ni}k

i=1 distributed by {qi}k i=1

L := n! Πk

i=1ni!Πk i=1qni i

Take the log likelihood log L = log(n!) X log(ni!) + X ni log(qi) Stirling’s approximation log n! ⇡ n log(n) n log L = n log(n) X ni log(ni) + X ni log(qi) + X ni n | {z } 1 n ( log L) = X ni n log ✓ni/n qi ◆

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 20 / 28

From Relative Entropy to Bayes’ Theorem

The likelihood of {ni}k

i=1 distributed by {qi}k i=1

L := n! Πk

i=1ni!Πk i=1qni i

Take the log likelihood log L = log(n!) X log(ni!) + X ni log(qi) Stirling’s approximation log n! ⇡ n log(n) n log L = n log(n) X ni log(ni) + X ni log(qi) + X ni n | {z } 1 n ( log L) = X ni n log ✓ni/n qi ◆ = X pi log ✓pi qi ◆

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 20 / 28

From Relative Entropy to Bayes’ Theorem

The likelihood of {ni}k

i=1 distributed by {qi}k i=1

L := n! Πk

i=1ni!Πk i=1qni i

Take the log likelihood log L = log(n!) X log(ni!) + X ni log(qi) Stirling’s approximation log n! ⇡ n log(n) n log L = n log(n) X ni log(ni) + X ni log(qi) + X ni n | {z } Relative entropy = average likelihood 1 n ( log L) = X ni n log ✓ni/n qi ◆ = X pi log ✓pi qi ◆ = D (p||q)

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 20 / 28

From Relative Entropy to Bayes’ Theorem

Relative entropy = average likelihood

1 n ( log L) = D (p||q)

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 21 / 28

From Relative Entropy to Bayes’ Theorem

Relative entropy = average likelihood

1 n ( log L) = D (p||q) Write X ! Z

• Z

log(L)p(x) dx = Z log ✓p q ◆ p(x) dx

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 21 / 28

From Relative Entropy to Bayes’ Theorem

Relative entropy = average likelihood

1 n ( log L) = D (p||q) Write X ! Z

• Z

log(L)p(x) dx = Z log ✓p q ◆ p(x) dx q(x) = L(x)p(x)

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 21 / 28

From Relative Entropy to Bayes’ Theorem

Relative entropy = average likelihood ! Bayes

1 n ( log L) = D (p||q) Write X ! Z

• Z

log(L)p(x) dx = Z log ✓p q ◆ p(x) dx Bayes’ theorem q(x) = L(x)p(x)

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 21 / 28

From Optimization to Bayes’ Theorem

Inverse Problem Given observation model d = G (x) + "

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 22 / 28

From Optimization to Bayes’ Theorem

Inverse Problem Given observation model d = G (x) + " Inverse task: given d, infer x

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 22 / 28

From Optimization to Bayes’ Theorem

Inverse Problem Given observation model d = G (x) + " Inverse task: given d, infer x Statistical inversion: Prior knowledge: X ⇠ ⇡prior(x). Look for the posterior distribution ⇡post(x) that combines prior information and information from the data.

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 22 / 28

From Optimization to Bayes’ Theorem

Inverse Problem Given observation model d = G (x) + " Inverse task: given d, infer x Statistical inversion: Prior knowledge: X ⇠ ⇡prior(x). Look for the posterior distribution ⇡post(x) that combines prior information and information from the data. The likelihood: assume " ⇠ N (0, C) ⇡like(x) = exp ✓ 1 2 kd G(x)k2

C

◆

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 22 / 28

From Optimization to Bayes’ Theorem

Prior Elicitation Try to get the best prior information = discrepancy relative to the posterior is minimized

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 23 / 28

From Optimization to Bayes’ Theorem

Prior Elicitation Try to get the best prior information = discrepancy relative to the posterior is minimized Conversely, best prior ! the information gained in the posterior should not be large

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 23 / 28

From Optimization to Bayes’ Theorem

Prior Elicitation Try to get the best prior information = discrepancy relative to the posterior is minimized Conversely, best prior ! the information gained in the posterior should not be large Equivalently, ⇡post = arg min

π(x)

D (⇡||⇡prior) = Z ⇡(x) log ✓ ⇡(x) ⇡prior(x) ◆ dx

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 23 / 28

From Optimization to Bayes’ Theorem

How about information from the data? Want to find x to match the data as well as we can

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 24 / 28

From Optimization to Bayes’ Theorem

How about information from the data? Want to find x to match the data as well as we can “Equivalently”: want to find the posterior distribution such that kd G(x)k2

C is minimized!

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 24 / 28

From Optimization to Bayes’ Theorem

How about information from the data? Want to find x to match the data as well as we can “Equivalently”: want to find the posterior distribution such that kd G(x)k2

C is minimized!

One approach: minimize the mean squared error ⇡post = arg min

π(x)

Z ⇡(x) kd G(x)k2

C dx

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 24 / 28

From Optimization to Bayes’ Theorem

How about information from the data? Want to find x to match the data as well as we can “Equivalently”: want to find the posterior distribution such that kd G(x)k2

C is minimized!

One approach: minimize the mean squared error ⇡post = arg min

π(x)

Z ⇡(x) kd G(x)k2

C dx=

Z ⇡(x) log (⇡like(x)) dx

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 24 / 28

From Optimization to Bayes’ Theorem

Prior + data information From prior ⇡post = arg min

π(x)

D (⇡||⇡prior) = Z ⇡(x) log ✓ ⇡(x) ⇡prior(x) ◆ dx

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 25 / 28

From Optimization to Bayes’ Theorem

Prior + data information From prior ⇡post = arg min

π(x)

D (⇡||⇡prior) = Z ⇡(x) log ✓ ⇡(x) ⇡prior(x) ◆ dx From likelihood ⇡post = arg min

π(x)

• Z

⇡(x) log (⇡like(x)) dx

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 25 / 28

From Optimization to Bayes’ Theorem

Prior + data information From prior ⇡post = arg min

π(x)

D (⇡||⇡prior) = Z ⇡(x) log ✓ ⇡(x) ⇡prior(x) ◆ dx From likelihood ⇡post = arg min

π(x)

• Z

⇡(x) log (⇡like(x)) dx A Compromise ⇡post = arg min

π(x)

• Z

⇡(x) log (⇡like(x)) dx+ Z ⇡(x) log ✓ ⇡(x) ⇡prior(x) ◆ dx subject to Z ⇡(x) dx = 1, and ⇡(x) 0.

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 25 / 28

From Optimization to Bayes’ Theorem

Prior + data information A Compromise ⇡post = arg min

π(x)

• Z

⇡(x) log (⇡like(x)) dx+ Z ⇡(x) log ✓ ⇡(x) ⇡prior(x) ◆ dx subject to Z ⇡(x) dx = 1, and ⇡(x) 0.

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 26 / 28

From Optimization to Bayes’ Theorem

Prior + data information A Compromise ⇡post = arg min

π(x)

• Z

⇡(x) log (⇡like(x)) dx+ Z ⇡(x) log ✓ ⇡(x) ⇡prior(x) ◆ dx subject to Z ⇡(x) dx = 1, and ⇡(x) 0. Does it have a solution ⇡post(x)? is it unique?

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 26 / 28

From Optimization to Bayes’ Theorem

Prior + data information A Compromise ⇡post = arg min

π(x)

• Z

⇡(x) log (⇡like(x)) dx+ Z ⇡(x) log ✓ ⇡(x) ⇡prior(x) ◆ dx subject to Z ⇡(x) dx = 1, and ⇡(x) 0. Does it have a solution ⇡post(x)? is it unique? How to solve?

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 26 / 28

From Optimization to Bayes’ Theorem

Prior + data information A Compromise ⇡post = arg min

π(x)

• Z

⇡(x) log (⇡like(x)) dx+ Z ⇡(x) log ✓ ⇡(x) ⇡prior(x) ◆ dx subject to Z ⇡(x) dx = 1, and ⇡(x) 0. Does it have a solution ⇡post(x)? is it unique? How to solve? Lagrangian + calculus of variation

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 26 / 28

From Optimization to Bayes’ Theorem

Prior + data information A Compromise ⇡post = arg min

π(x)

• Z

⇡(x) log (⇡like(x)) dx+ Z ⇡(x) log ✓ ⇡(x) ⇡prior(x) ◆ dx subject to Z ⇡(x) dx = 1, and ⇡(x) 0. Does it have a solution ⇡post(x)? is it unique? How to solve? Lagrangian + calculus of variation Solution = Bayes’ theorem ⇡post (x|d) = ⇡like (d|x) ⇥ ⇡prior (x) R ⇡like (d|x) ⇥ ⇡prior (x) dx

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 26 / 28

Outline

1

Bayesian Inversion Framework

2

Entropy

3

Relative Entropy

4

Bayes’ Theorem and Information Theory

5

Conclusions

(CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 27 / 28

Conclusions

1 Information provide an intuitive and fresh view of Bayes’ theorem (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 28 / 28

Conclusions

1 Information provide an intuitive and fresh view of Bayes’ theorem 2 Relative entropy ! Bayes’ theorem (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 28 / 28

Conclusions

1 Information provide an intuitive and fresh view of Bayes’ theorem 2 Relative entropy ! Bayes’ theorem 3 Optimization + information ! Bayes’ theorem (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 28 / 28