a fresh look at the bayes theorem from information theory
play

A Fresh Look at the Bayes Theorem from Information Theory Tan - PowerPoint PPT Presentation

Sep 30, 2022 •362 likes •1.13k views

A Fresh Look at the Bayes Theorem from Information Theory Tan Bui-Thanh Computational Engineering and Optimization (CEO) Group Department of Aerospace Engineering and Engineering Mechanics Institute for Computational Engineering and Sciences

  1. A Fresh Look at the Bayes’ Theorem from Information Theory Tan Bui-Thanh Computational Engineering and Optimization (CEO) Group Department of Aerospace Engineering and Engineering Mechanics Institute for Computational Engineering and Sciences (ICES) The University of Texas at Austin Babuska Series, ICES Sep 9, 2016 (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 1 / 28

  2. Outline Bayesian Inversion Framework 1 Entropy 2 Relative Entropy 3 Bayes’ Theorem and Information Theory 4 Conclusions 5 (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 2 / 28

  3. Large-scale computation under uncertainty Inverse electromagnetic scattering Randomness Random errors in measurements are unavoidable Inadequacy of the mathematical model (Maxwell equations) Challenge How to invert for the invisible shape/medium using computational � 10 6 � electromagnetics with O degree of freedoms? (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 3 / 28

  4. Large-scale computation under uncertainty Full wave form seismic inversion Randomness Random errors in seismometer measurements are unavoidable Inadequacy of the mathematical model (elastodynamics) Challenge How to image the earth interior using forward computational model with � 10 9 � with O degree of freedoms? (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 4 / 28

  5. Inverse Shape Electromagnetic Scattering Problem Maxwell Equations: r ⇥ E = � µ @ H @ t , (Faraday) r ⇥ H = ✏@ E @ t , (Ampere) E : Electric field, H : Magnetic field, µ : permeability, ✏ : permittivity (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 5 / 28

  6. Inverse Shape Electromagnetic Scattering Problem Maxwell Equations: r ⇥ E = � µ @ H @ t , (Faraday) r ⇥ H = ✏@ E @ t , (Ampere) E : Electric field, H : Magnetic field, µ : permeability, ✏ : permittivity Forward problem (discontinuous Galerkin discretization) d = G ( x ) where G maps shape parameters x to electric/magnetic field d at the measurement points (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 5 / 28

  7. Inverse Shape Electromagnetic Scattering Problem Maxwell Equations: r ⇥ E = � µ @ H @ t , (Faraday) r ⇥ H = ✏@ E @ t , (Ampere) E : Electric field, H : Magnetic field, µ : permeability, ✏ : permittivity Forward problem (discontinuous Galerkin discretization) d = G ( x ) where G maps shape parameters x to electric/magnetic field d at the measurement points Inverse Problem Given (possibly noise-corrupted) measurements on d , infer x ? (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 5 / 28

  8. The Bayesian Statistical Inversion Framework (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 6 / 28

  9. The Bayesian Statistical Inversion Framework (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 6 / 28

  10. The Bayesian Statistical Inversion Framework Bayes Theorem ⇡ post ( x | d ) / ⇡ like ( d | x ) ⇥ ⇡ prior ( x ) (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 6 / 28

  11. Bayes theorem for inverse electromagnetic scattering Prior knowledge: The obstacle is smooth: Z 2 π ✓ ◆ ⇡ pr ( x ) / exp � � r 00 ( x ) d ✓ 0 (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 7 / 28

  12. Bayes theorem for inverse electromagnetic scattering Prior knowledge: The obstacle is smooth: Z 2 π ✓ ◆ ⇡ pr ( x ) / exp � � r 00 ( x ) d ✓ 0 Likelihood: Additive Gaussian noise, for example, ✓ ◆ � 1 2 k G ( x ) � d k 2 ⇡ like ( d | x ) / exp C noise (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 7 / 28

  13. Outline Bayesian Inversion Framework 1 Entropy 2 Relative Entropy 3 Bayes’ Theorem and Information Theory 4 Conclusions 5 (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 8 / 28

  14. Entropy Definition We define the uncertainty in a random variable X distributed by 0  ⇡ ( x )  1 as Z H ( X ) = � ⇡ ( x ) log ⇡ ( x ) dx � 0 (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 9 / 28

  15. Entropy (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 10 / 28

  16. Entropy Wiener and Shannon Kolmogorov Copied from Sergio Verdu (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 10 / 28

  17. Entropy Wiener and Shannon Kolmogorov Copied from Sergio Verdu Wiener: “...for it belongs to the two of us equally” (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 10 / 28

  18. Entropy Wiener and Shannon Kolmogorov Copied from Sergio Verdu Wiener: “...for it belongs to the two of us equally” Shannon: “...a mathematical pun” (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 10 / 28

  19. Entropy Wiener and Shannon Kolmogorov Copied from Sergio Verdu Wiener: “...for it belongs to the two of us equally” Shannon: “...a mathematical pun” Kolmogorov: “...has no physical interpretation” (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 10 / 28

  20. Entropy Entropy of uniform distribution (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 11 / 28

  21. Entropy Entropy of uniform distribution Let U be a uniform random variable with values in X , and | X | < 1 (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 11 / 28

  22. Entropy Entropy of uniform distribution Let U be a uniform random variable with values in X , and | X | < 1 ⇡ ( u ) := 1 | X | ) H ( U ) = log ( | X | ) (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 11 / 28

  23. Entropy Entropy of uniform distribution Let U be a uniform random variable with values in X , and | X | < 1 ⇡ ( u ) := 1 | X | ) H ( U ) = log ( | X | ) How uncertain is the uniform random variable? (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 11 / 28

  24. Entropy Entropy of uniform distribution Let U be a uniform random variable with values in X , and | X | < 1 ⇡ ( u ) := 1 | X | ) H ( U ) = log ( | X | ) How uncertain is the uniform random variable? H ( X )  H ( U ) (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 11 / 28

  25. 100 years of uniform distribution source: Christoph Aistleitner (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 12 / 28

  26. 100 years of uniform distribution source: Christoph Aistleitner Hermann Weyl (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 12 / 28

  27. and Maximum entropy Maximum entropy distribution X with known mean and variance (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 13 / 28

  28. and Maximum entropy Maximum entropy distribution X with known mean and variance ⇡ ( x )? with maximum entropy (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 13 / 28

  29. and Maximum entropy Maximum entropy distribution X with known mean and variance ⇡ ( x )? with maximum entropy Z max π ( x ) H ( X ) = � ⇡ ( x ) log( ⇡ ( x )) dx subject to Z x ⇡ ( x ) dx = µ Z ( x � µ ) 2 ⇡ ( x ) dx = � 2 Z ⇡ ( x ) dx = 1 (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 13 / 28

  30. Gaussian and Maximum entropy Maximum entropy distribution X with known mean and variance ⇡ ( x )? with maximum entropy Z max π ( x ) H ( X ) = � ⇡ ( x ) log( ⇡ ( x )) dx subject to Z x ⇡ ( x ) dx = µ Z ( x � µ ) 2 ⇡ ( x ) dx = � 2 Z ⇡ ( x ) dx = 1 � µ, � 2 � Gaussian distribution: ⇡ ( x ) = N (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 13 / 28

  31. Outline Bayesian Inversion Framework 1 Entropy 2 Relative Entropy 3 Bayes’ Theorem and Information Theory 4 Conclusions 5 (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 14 / 28

  32. Relative Entropy Abraham Wald (1945) Harold Je ff reys (1945) Z ✓ ⇡ ( x ) ◆ D ( ⇡ || q ) := ⇡ ( x ) log dx q ( x ) (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 15 / 28

  33. Kullback-Leibler divergence = Relative Entropy Solomon Kullback Richard Leibler (1951) (1951) ✓ ⇡ ( x ) ◆ Z D ( ⇡ || q ) := ⇡ ( x ) log dx q ( x ) (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 16 / 28

  34. Kullback-Leibler divergence = Relative Entropy Solomon Kullback Richard Leibler (1951) (1951) ✓ ⇡ ( x ) ◆ ✓ ⇡ i ◆ Z X dx discrete D ( ⇡ || q ) := ⇡ ( x ) log = ⇡ i log q ( x ) q i (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 16 / 28

  35. Information Inequality The most important inequality in information theory D ( ⇡ || q ) � 0 Can we see it easily? (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 17 / 28

  36. Information Inequality The most important inequality in information theory D ( ⇡ || q ) � 0 Can we see it easily? (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 17 / 28

  37. Outline Bayesian Inversion Framework 1 Entropy 2 Relative Entropy 3 Bayes’ Theorem and Information Theory 4 Conclusions 5 (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 18 / 28

  38. From Relative Entropy to Bayes’ Theorem Toss n times an k th dimensional dice with the prior distribution of k X each face { p i } k i =1 : p i = 1 i =1 (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 19 / 28

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Bayes Theorem Thomas Bayes (1701-1761) Simple form of Bayes Theorem, for

Bayes Theorem Thomas Bayes (1701-1761) Simple form of Bayes Theorem, for

Nave Bayes Classifica/on Debapriyo Majumdar Data Mining Fall 2014 Indian Statistical Institute Kolkata August 14, 2014 Bayes Theorem Thomas Bayes (1701-1761) Simple form of Bayes Theorem, for two

1.04k views • 20 slides
Statistical Reasoning Probability and Bayes' Theorem Certainty Factors and Rule- Based

Statistical Reasoning Probability and Bayes' Theorem Certainty Factors and Rule- Based

Statistical Reasoning Probability and Bayes' Theorem Certainty Factors and Rule- Based Systems Bayesian Networks Dempster-Shafer Theory Fuzzy Logic Chapter 8 1 Probability and Bayes' Theorem P(H i|E) = the probability that

281 views • 13 slides
Chapter 3: Basics from Probability Theory and Statistics 3.1 Probability Theory Events,

Chapter 3: Basics from Probability Theory and Statistics 3.1 Probability Theory Events,

Chapter 3: Basics from Probability Theory and Statistics 3.1 Probability Theory Events, Probabilities, Bayes Theorem, Random Variables, Distributions, Moments, Tail Bounds, Central Limit Theorem, Entropy Measures 3.2 Statistical Inference

845 views • 45 slides
Formal Modeling in Cognitive Science Independence Lecture 23: Conditional Probability; Bayes

Formal Modeling in Cognitive Science Independence Lecture 23: Conditional Probability; Bayes

Conditional Probability and Independence Conditional Probability and Independence Bayes Theorem Bayes Theorem An Application of Bayes Theorem An Application of Bayes Theorem 1 Conditional Probability and Independence Conditional

247 views • 6 slides
Frequentist example An entomologist spots what might be a rare subspecies of beetle, due to the

Frequentist example An entomologist spots what might be a rare subspecies of beetle, due to the

CS201 Bayes Theorem Excerpts from WikiPedia http://en.wikipedia.org/wiki/Bayes%27_theorem http://en.wikipedia.org/wiki/Bayesian_inference Bayes's theorem is stated mathematically as the following simple form: [1] For an epistemological

333 views • 5 slides
1 Problem with Brute force Nave Bayes ( ) ( ) ( ) It cannot generalize to unseen

1 Problem with Brute force Nave Bayes ( ) ( ) ( ) It cannot generalize to unseen

Outline Bayes theorem Discrete Bayesian classifiers Maximum likelihood classification Brute force Bayesian learning Nave Bayes Lecture 5 Bayesian Belief Networks Bayes theorem Maximum likelihood ( ) ( ) ( ) P x

72 views • 4 slides
Spam Filtering with Naive Bayes Classifier Yuriy Arabskyy June 6, 2017 Table of contents What

Spam Filtering with Naive Bayes Classifier Yuriy Arabskyy June 6, 2017 Table of contents What

Spam Filtering with Naive Bayes Classifier Yuriy Arabskyy June 6, 2017 Table of contents What is spam? Different spam types Anti-Spam Techniques Probability theory basics Conditional probability Bayes Theorem Naive Bayes Theorem Spam

1.04k views • 27 slides
Bayesian Learning Bayes Theorem MAP, ML hypotheses MAP learners Minimum description

Bayesian Learning Bayes Theorem MAP, ML hypotheses MAP learners Minimum description

Bayesian Learning Bayes Theorem MAP, ML hypotheses MAP learners Minimum description length principle Bayes optimal classifier Naive Bayes learner Example: Learning over text data Bayesian belief networks

1.11k views • 51 slides
Which probability Which probability Which probability Which probability theory for cosmology?

Which probability Which probability Which probability Which probability theory for cosmology?

Which probability Which probability Which probability Which probability theory for cosmology? theory for cosmology? theory for cosmology? theory for cosmology? From Bayes theorem to the anthropic principle From Bayes theorem to the

537 views • 35 slides
Conditional and Small Sample Probability August 6, 2019 August 6, 2019 1 / 63 Bayes Theorem

Conditional and Small Sample Probability August 6, 2019 August 6, 2019 1 / 63 Bayes Theorem

Conditional and Small Sample Probability August 6, 2019 August 6, 2019 1 / 63 Bayes Theorem Bayes Theorem will help us more easily calculate P (statement about variable 1 | statement about variable 2) when we have information about P

759 views • 63 slides
Formal Modeling in Cognitive Science Lecture 18: Conditional Probability; Bayes Theorem Steve

Formal Modeling in Cognitive Science Lecture 18: Conditional Probability; Bayes Theorem Steve

Conditional Probability and Independence Bayes Theorem Formal Modeling in Cognitive Science Lecture 18: Conditional Probability; Bayes Theorem Steve Renals (notes by Frank Keller) School of Informatics University of Edinburgh

169 views • 13 slides
Data Mining Lecture 06: Bayes Theorem Theses slides are based on the slides by Tan,

Data Mining Lecture 06: Bayes Theorem Theses slides are based on the slides by Tan,

CISC 4631 Data Mining Lecture 06: Bayes Theorem Theses slides are based on the slides by Tan, Steinbach and Kumar (textbook authors) Eamonn Koegh (UC Riverside) Andrew Moore (CMU/Google) 1 Nave Bayes Classifier Thomas

657 views • 36 slides
Bayes Theorem Bayess theorem Pr [ A | B ] = Pr [ B | A ] Pr [ A ] Pr [ B ] Note: This

Bayes Theorem Bayess theorem Pr [ A | B ] = Pr [ B | A ] Pr [ A ] Pr [ B ] Note: This

Bayes Theorem Bayess theorem Pr [ A | B ] = Pr [ B | A ] Pr [ A ] Pr [ B ] Note: This is very useful in both this course and in life. Example of Application of Bayess theorem Pr [ A | B ] = Pr [ B | A ] Pr [ A ] Pr [ B ] . There

86 views • 8 slides
1 Bayes Theorem Bayesian Categorization Determine category of x k by determining for each y i

1 Bayes Theorem Bayesian Categorization Determine category of x k by determining for each y i

Axioms of Probability Theory All probabilities between 0 and 1 P A 0 ( ) 1 CS 391L: Machine Learning: True proposition has probability 1, false has Bayesian Learning: probability 0. Nave Bayes P(true) = 1

765 views • 4 slides
PCP Theorem [PCP Theorem is] the most important result in complexity theory since Cooks

PCP Theorem [PCP Theorem is] the most important result in complexity theory since Cooks

PCP Theorem [PCP Theorem is] the most important result in complexity theory since Cooks Theorem. Ingo Wegener, 2005 Computational Complexity, by Fu Yuxi PCP Theorem 1 / 119 S. Arora, C. Lund, R. Motwani, M. Sudan, and M. Szegedy. Proof

1.53k views • 120 slides
Independence, Variance, Bayes Theorem Russell Impagliazzo and Miles Jones Thanks to Janine

Independence, Variance, Bayes Theorem Russell Impagliazzo and Miles Jones Thanks to Janine

Independence, Variance, Bayes Theorem Russell Impagliazzo and Miles Jones Thanks to Janine Tiefenbruck http://cseweb.ucsd.edu/classes/sp16/cse21-bd/ June 1, 2016 Independence Rosen p. 486 Theorem : If X and Y are independent random

1.26k views • 92 slides
Introduction to Bayesian Analysis in Stata The Method Bayes rule Fundamental equation MCMC

Introduction to Bayesian Analysis in Stata The Method Bayes rule Fundamental equation MCMC

Bayesian analysis in Stata Outline The general idea Introduction to Bayesian Analysis in Stata The Method Bayes rule Fundamental equation MCMC Gustavo Snchez Stata tools bayesmh bayesstats ess StataCorp LLC Blocking bayesgraph

1.37k views • 73 slides
Review of Conditional Probability and Independence Definition L7.3 (Def 1.3.2 on p.20): If A, B

Review of Conditional Probability and Independence Definition L7.3 (Def 1.3.2 on p.20): If A, B

Review of Conditional Probability and Independence Definition L7.3 (Def 1.3.2 on p.20): If A, B S and P ( B ) &gt; 0 , then P ( A | B ) = P ( A B ) . P ( B ) Bayes Rule Theorem L7.2 (Thm 1.3.5 on p.23): Let A 1 , A 2 , . . . be a

305 views • 11 slides
Hierarchical Methods for Bayesian Inverse Problems Optimization and Inversion under Uncertainty,

Hierarchical Methods for Bayesian Inverse Problems Optimization and Inversion under Uncertainty,

Hierarchical Methods for Bayesian Inverse Problems Optimization and Inversion under Uncertainty, RICAM, November 12 th , 2019 Matt Dunlop (Courant) Tapio Helin (Lappeenranta) Andrew Stuart (Caltech) Outline 1. Introduction 2. Hierarchical

820 views • 60 slides
Bayesian Inference Harvard Math Camp - Econometrics Ashesh Rambachan Summer 2018 Outline What

Bayesian Inference Harvard Math Camp - Econometrics Ashesh Rambachan Summer 2018 Outline What

Bayesian Inference Harvard Math Camp - Econometrics Ashesh Rambachan Summer 2018 Outline What is Bayesian Inference? Inference Frequentists vs. Bayesians Conjugate Priors Normal-Normal Beta-Bernoulli Multinomial-Dirichlet Exchangeability

702 views • 51 slides
On Some Geometrical Aspects of Bayesian Inference Miguel de Carvalho Joint with B. J.

On Some Geometrical Aspects of Bayesian Inference Miguel de Carvalho Joint with B. J.

On Some Geometrical Aspects of Bayesian Inference Miguel de Carvalho Joint with B. J. Barney and G. L. Page; Brigham Young University, US School of Mathematics M. de Carvalho On the Geometry of Bayesian Inference 1 / 37 ISBA 2018:

724 views • 56 slides
Generative Learning INFO-4604, Applied Machine Learning University of Colorado Boulder November

Generative Learning INFO-4604, Applied Machine Learning University of Colorado Boulder November

Generative Learning INFO-4604, Applied Machine Learning University of Colorado Boulder November 29, 2018 Prof. Michael Paul Generative vs Discriminative The classification algorithms we have seen so far are called discriminative algorithms

1.04k views • 44 slides
2. Naive Bayes Classification Machine Learning and Real-world Data (MLRD) Paula Buttery (based

2. Naive Bayes Classification Machine Learning and Real-world Data (MLRD) Paula Buttery (based

2. Naive Bayes Classification Machine Learning and Real-world Data (MLRD) Paula Buttery (based on slides created by Simone Teufel) Lent 2018 Last session: we used a sentiment lexicon for sentiment classification Movie review sentiment

446 views • 15 slides
CSC 411 Lecture 19: Bayesian Linear Regression Roger Grosse, Amir-massoud Farahmand, and Juan

CSC 411 Lecture 19: Bayesian Linear Regression Roger Grosse, Amir-massoud Farahmand, and Juan

CSC 411 Lecture 19: Bayesian Linear Regression Roger Grosse, Amir-massoud Farahmand, and Juan Carrasquilla University of Toronto UofT CSC 411: 19-Bayesian Linear Regression 1 / 36 Overview Weve covered both parametric and nonparametric

543 views • 36 slides

More recommend