information theory and statistical inference
play

Information Theory and Statistical Inference Samuel Cheng School of - PowerPoint PPT Presentation

Sep 27, 2023 •202 likes •1.1k views

Information Theory and Statistical Inference Samuel Cheng School of ECE University of Oklahoma August 23, 2018 S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 1 / 45 Lecture 2 Introduction to probabilistic

  1. Information Theory and Statistical Inference Samuel Cheng School of ECE University of Oklahoma August 23, 2018 S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 1 / 45

  2. Lecture 2 Introduction to probabilistic inference Inference o : (Observed) evidence, θ : Parameter, x : prediction Maximum Likelihood (ML) x = arg max x p ( x | ˆ θ ) , ˆ ˆ θ = arg max θ p ( o | θ ) Maximum A Posteriori (MAP) x = arg max x p ( x | ˆ θ ) , ˆ ˆ θ = arg max θ p ( θ | o ) Bayesian � x = � ˆ x x p ( x | θ ) p ( θ | o ) θ � �� � p ( x | o ) where p ( θ | o ) = p ( o | θ ) p ( θ ) ∝ p ( o | θ ) p ( θ ) p ( o ) ���� prior S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 2 / 45

  3. Lecture 2 Introduction to probabilistic inference Coin Flip C 1 C 2 C 3 P(H|C 1 ) = 0.1 P(H|C 3 ) = 0.9 P(H|C 2 ) = 0.5 Which coin will I use? P(C 1 ) = 1/3 P(C 2 ) = 1/3 P(C 3 ) = 1/3 Prior: Probability of a hypothesis before we make any observations (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 3 / 45

  4. Lecture 2 Introduction to probabilistic inference Coin Flip C 1 C 2 C 3 P(H|C 1 ) = 0.1 P(H|C 3 ) = 0.9 P(H|C 2 ) = 0.5 Which coin will I use? P(C 1 ) = 1/3 P(C 2 ) = 1/3 P(C 3 ) = 1/3 Uniform Prior: All hypothesis are equally likely before we make any observations (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 4 / 45

  5. Lecture 2 Introduction to probabilistic inference Experiment 1: Heads Which coin did I use? P(C 1 |H) = ? P(C 2 |H) = ? P(C 3 |H) = ? 3 P ( C 1 | H ) = P ( H | C 1 ) P ( C 1 ) � P ( H ) = P ( H | C i ) P ( C i ) P ( H ) i =1 C 2 C 3 C 1 P(H|C 1 ) = 0.1 P(H|C 2 ) = 0.5 P(H|C 3 ) = 0.9 P(C 1 ) = 1/3 P(C 2 ) = 1/3 P(C 3 ) = 1/3 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 5 / 45

  6. Lecture 2 Introduction to probabilistic inference Experiment 1: Heads Which coin did I use? P(C 1 |H) = 0.066 P(C 2 |H) = 0.333 P(C 3 |H) = 0.6 Posterior: Probability of a hypothesis given data C 2 C 3 C 1 P(H|C 1 ) = 0.1 P(H|C 2 ) = 0.5 P(H|C 3 ) = 0.9 P(C 1 ) = 1/3 P(C 2 ) = 1/3 P(C 3 ) = 1/3 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 6 / 45

  7. Lecture 2 Introduction to probabilistic inference Experiment 2: Tails Which coin did I use? P(C 1 |HT) = ? P(C 2 |HT) = ? P(C 3 |HT) = ? P ( C 1 | HT ) = α P ( HT | C 1 ) P ( C 1 ) = α P ( H | C 1 ) P ( T | C 1 ) P ( C 1 ) C 1 C 2 C 3 P(H|C 1 ) = 0.1 P(H|C 2 ) = 0.5 P(H|C 3 ) = 0.9 P(C 1 ) = 1/3 P(C 2 ) = 1/3 P(C 3 ) = 1/3 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 7 / 45

  8. Lecture 2 Introduction to probabilistic inference Experiment 2: Tails Which coin did I use? P(C 1 |HT) = 0.21 P(C 2 |HT) = 0.58 P(C 3 |HT) = 0.21 P ( C 1 | HT ) = α P ( HT | C 1 ) P ( C 1 ) = α P ( H | C 1 ) P ( T | C 1 ) P ( C 1 ) C 1 C 2 C 3 P(H|C 1 ) = 0.1 P(H|C 2 ) = 0.5 P(H|C 3 ) = 0.9 P(C 1 ) = 1/3 P(C 2 ) = 1/3 P(C 3 ) = 1/3 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 8 / 45

  9. Lecture 2 Introduction to probabilistic inference Experiment 2: Tails Which coin did I use? P(C 1 |HT) = 0.21 P(C 2 |HT) = 0.58 P(C 3 |HT) = 0.21 C 1 C 2 C 3 P(H|C 1 ) = 0.1 P(H|C 3 ) = 0.9 P(H|C 2 ) = 0.5 P(C 1 ) = 1/3 P(C 2 ) = 1/3 P(C 3 ) = 1/3 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 9 / 45

  10. Lecture 2 Introduction to probabilistic inference Your Estimate? What is the probability of heads after two experiments? Most likely coin: Best estimate for P(H) C 2 P(H|C 2 ) = 0.5 C 2 C 3 C 1 P(H|C 1 ) = 0.1 P(H|C 2 ) = 0.5 P(H|C 3 ) = 0.9 P(C 1 ) = 1/3 P(C 2 ) = 1/3 P(C 3 ) = 1/3 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 10 / 45

  11. Lecture 2 Introduction to probabilistic inference Your Estimate? Maximum Likelihood Estimate: The best hypothesis that fits observed data assuming uniform prior Most likely coin: Best estimate for P(H) C 2 P(H|C 2 ) = 0.5 C 2 P(H|C 2 ) = 0.5 P(C 2 ) = 1/3 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 11 / 45

  12. Lecture 2 Introduction to probabilistic inference Using Prior Knowledge • Should we always use Uniform Prior? • Background knowledge: • Heads => you go first in Abalone against TA • TAs are nice people • => TA is more likely to use a coin biased in your favor C 1 C 2 C 3 P(H|C 1 ) = 0.1 P(H|C 2 ) = 0.5 P(H|C 3 ) = 0.9 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 12 / 45

  13. Lecture 2 Introduction to probabilistic inference Using Prior Knowledge We can encode it in the prior: P(C 1 ) = 0.05 P(C 2 ) = 0.25 P(C 3 ) = 0.70 C 1 C 2 C 3 P(H|C 1 ) = 0.1 P(H|C 2 ) = 0.5 P(H|C 3 ) = 0.9 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 13 / 45

  14. Lecture 2 Introduction to probabilistic inference Experiment 1: Heads Which coin did I use? P(C 1 |H) = ? P(C 2 |H) = ? P(C 3 |H) = ? P ( C 1 | H ) = α P ( H | C 1 ) P ( C 1 ) C 2 C 3 C 1 P(H|C 1 ) = 0.1 P(H|C 2 ) = 0.5 P(H|C 3 ) = 0.9 P(C 1 ) = 0.05 P(C 2 ) = 0.25 P(C 3 ) = 0.70 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 14 / 45

  15. Lecture 2 Introduction to probabilistic inference Experiment 1: Heads Which coin did I use? P(C 1 |H) = 0.006 P(C 2 |H) = 0.165 P(C 3 |H) = 0.829 ML posterior after Exp 1: P(C 1 |H) = 0.066 P(C 2 |H) = 0.333 P(C 3 |H) = 0.600 C 1 C 2 C 3 P(H|C 1 ) = 0.1 P(H|C 2 ) = 0.5 P(H|C 3 ) = 0.9 P(C 1 ) = 0.05 P(C 2 ) = 0.25 P(C 3 ) = 0.70 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 15 / 45

  16. Lecture 2 Introduction to probabilistic inference Experiment 2: Tails Which coin did I use? P(C 1 |HT) = ? P(C 2 |HT) = ? P(C 3 |HT) = ? P ( C 1 | HT ) = α P ( HT | C 1 ) P ( C 1 ) = α P ( H | C 1 ) P ( T | C 1 ) P ( C 1 ) C 2 C 3 C 1 P(H|C 1 ) = 0.1 P(H|C 2 ) = 0.5 P(H|C 3 ) = 0.9 P(C 1 ) = 0.05 P(C 2 ) = 0.25 P(C 3 ) = 0.70 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 16 / 45

  17. Lecture 2 Introduction to probabilistic inference Experiment 2: Tails Which coin did I use? P(C 1 |HT) = 0.035 P(C 2 |HT) = 0.481 P(C 3 |HT) = 0.485 P ( C 1 | HT ) = α P ( HT | C 1 ) P ( C 1 ) = α P ( H | C 1 ) P ( T | C 1 ) P ( C 1 ) C 2 C 3 C 1 P(H|C 1 ) = 0.1 P(H|C 2 ) = 0.5 P(H|C 3 ) = 0.9 P(C 1 ) = 0.05 P(C 2 ) = 0.25 P(C 3 ) = 0.70 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 17 / 45

  18. Lecture 2 Introduction to probabilistic inference Experiment 2: Tails Which coin did I use? P(C 1 |HT) = 0.035 P(C 2 |HT) = 0.481 P(C 3 |HT) = 0.485 C 1 C 2 C 3 P(H|C 1 ) = 0.1 P(H|C 3 ) = 0.9 P(H|C 2 ) = 0.5 P(C 1 ) = 0.05 P(C 2 ) = 0.25 P(C 3 ) = 0.70 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 18 / 45

  19. Lecture 2 Introduction to probabilistic inference Your Estimate? What is the probability of heads after two experiments? Most likely coin: Best estimate for P(H) C 3 P(H|C 3 ) = 0.9 C 1 C 2 C 3 P(H|C 1 ) = 0.1 P(H|C 2 ) = 0.5 P(H|C 3 ) = 0.9 P(C 1 ) = 0.05 P(C 2 ) = 0.25 P(C 3 ) = 0.70 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 19 / 45

  20. Lecture 2 Introduction to probabilistic inference Your Estimate? Maximum A Posteriori (MAP) Estimate: The best hypothesis that fits observed data assuming a non-uniform prior Most likely coin: Best estimate for P(H) C 3 P(H|C 3 ) = 0.9 C 3 P(H|C 3 ) = 0.9 P(C 3 ) = 0.70 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 20 / 45

  21. Lecture 2 Introduction to probabilistic inference Did We Do The Right Thing? P(C 1 |HT) = 0.035 P(C 2 |HT) = 0.481 P(C 3 |HT) = 0.485 C 1 C 2 C 3 P(H|C 1 ) = 0.1 P(H|C 3 ) = 0.9 P(H|C 2 ) = 0.5 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 21 / 45

  22. Lecture 2 Introduction to probabilistic inference Did We Do The Right Thing? P(C 1 |HT) = 0.035 P(C 2 |HT) = 0.481 P(C 3 |HT) = 0.485 C 2 and C 3 are almost equally likely C 1 C 2 C 3 P(H|C 1 ) = 0.1 P(H|C 3 ) = 0.9 P(H|C 2 ) = 0.5 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 22 / 45

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Chapter 1: Introduction to Information Theory Book: Information Theory, Inference, and

Chapter 1: Introduction to Information Theory Book: Information Theory, Inference, and

Chapter 1: Introduction to Information Theory Book: Information Theory, Inference, and Learning Algorithms from David MacKay Noisy Channels - Error correcting Codes Examples - System solution - Channel models - Binary symmetric Channel

571 views • 22 slides
Chapter 8: Hypothesis Testing STK4011/9011: Statistical Inference Theory Johan Pensar

Chapter 8: Hypothesis Testing STK4011/9011: Statistical Inference Theory Johan Pensar

Chapter 8: Hypothesis Testing STK4011/9011: Statistical Inference Theory Johan Pensar STK4011/9011: Statistical Inference Theory Chapter 8: Hypothesis Testing 1 / 36 Overview Methods of Finding Tests 1 Likelihood Ratio Tests

672 views • 36 slides
Chapter 1: Probability Theory (a recap) STK4011/9011: Statistical Inference Theory Johan Pensar

Chapter 1: Probability Theory (a recap) STK4011/9011: Statistical Inference Theory Johan Pensar

Chapter 1: Probability Theory (a recap) STK4011/9011: Statistical Inference Theory Johan Pensar STK4011/9011: Statistical Inference Theory Chapter 1: Probability Theory (a recap) 1 / 11 Overview Set Theory 1 Probability Function 2

369 views • 11 slides
Chapter 4: Multiple Random Variables STK4011/9011: Statistical Inference Theory Johan Pensar

Chapter 4: Multiple Random Variables STK4011/9011: Statistical Inference Theory Johan Pensar

Chapter 4: Multiple Random Variables STK4011/9011: Statistical Inference Theory Johan Pensar STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 1 / 20 Overview Joint and Marginal Distributions 1 Conditional

604 views • 25 slides
Chapter 2: Transformations and Expectations (a recap) STK4011/9011: Statistical Inference Theory

Chapter 2: Transformations and Expectations (a recap) STK4011/9011: Statistical Inference Theory

Chapter 2: Transformations and Expectations (a recap) STK4011/9011: Statistical Inference Theory Johan Pensar STK4011/9011: Statistical Inference Theory Chapter 2: Transformations and Expectations (a recap) 1 / 18 Overview Distributions of

833 views • 18 slides
Review of Estimation Theory Berlin 2003 References: 1. X. Huang et. al., Spoken Language

Review of Estimation Theory Berlin 2003 References: 1. X. Huang et. al., Spoken Language

Review of Estimation Theory Berlin 2003 References: 1. X. Huang et. al., Spoken Language Processing, Chapter 3 Introduction Estimation theory is the most important theory and method in statistical inference Statistical inference

300 views • 17 slides
UQ, STAT2201, 2017, Lecture 6 Unit 6 Statistical Inference Ideas. 1 Statistical Inference is

UQ, STAT2201, 2017, Lecture 6 Unit 6 Statistical Inference Ideas. 1 Statistical Inference is

UQ, STAT2201, 2017, Lecture 6 Unit 6 Statistical Inference Ideas. 1 Statistical Inference is the process of forming judgements about the parameters of a population typically on the basis of random sampling . 2 The random variables X 1 , X 2

670 views • 32 slides
Theory of statistical inference: a lazy approach to obtaining asymptotic results in parametric

Theory of statistical inference: a lazy approach to obtaining asymptotic results in parametric

Theory of statistical inference: a lazy approach to obtaining asymptotic results in parametric models Hien D. Nguyen 1 , 2 1 DECRA Research Fellow, Australian Research Council. 2 Lecturer, Department of Mathematics and Statistics, La Trobe

834 views • 67 slides
Chapter 3: Common Families of Distributions STK4011/9011: Statistical Inference Theory Johan

Chapter 3: Common Families of Distributions STK4011/9011: Statistical Inference Theory Johan

Chapter 3: Common Families of Distributions STK4011/9011: Statistical Inference Theory Johan Pensar STK4011/9011: Statistical Inference Theory Chapter 3: Common Families of Distributions 1 / 8 Overview Exponential Families 1 Location and

194 views • 8 slides
Foundations for Inference I Dajiang Liu @PHS525 Feb-09-2016 Statistical Inference

Foundations for Inference I Dajiang Liu @PHS525 Feb-09-2016 Statistical Inference

Foundations for Inference I Dajiang Liu @PHS525 Feb-09-2016 Statistical Inference Statistical inference is usually performed using a randomly selected sample from the population The estimates obtained from the sample may not actually

168 views • 14 slides
Statistical Natural Language Processing A refresher on information theory ar ltekin

Statistical Natural Language Processing A refresher on information theory ar ltekin

Statistical Natural Language Processing A refresher on information theory ar ltekin University of Tbingen Seminar fr Sprachwissenschaft Summer Semester 2017 Information theory Information theory storage and transmission of

346 views • 34 slides
Theory of Statistical Inference Dajiang Liu @PHS 525 Feb-11, 2016 Sampling Distribution for

Theory of Statistical Inference Dajiang Liu @PHS 525 Feb-11, 2016 Sampling Distribution for

Theory of Statistical Inference Dajiang Liu @PHS 525 Feb-11, 2016 Sampling Distribution for the Mean can be calculated For each sample, a mean value What is the distribution like? Normal distribution For a typical

581 views • 15 slides
Statistical Natural Language Processing Statistical models: learning, inference, estimation,

Statistical Natural Language Processing Statistical models: learning, inference, estimation,

Statistical Natural Language Processing Statistical models: learning, inference, estimation, prediction ar ltekin University of Tbingen Seminar fr Sprachwissenschaft Summer Semester 2017 Statistical models: learning, inference,

497 views • 25 slides
Hypothesis testing and statistical decision theory Lirong Xia March 25, 2016 Schedule

Hypothesis testing and statistical decision theory Lirong Xia March 25, 2016 Schedule

Hypothesis testing and statistical decision theory Lirong Xia March 25, 2016 Schedule Hypothesis testing Statistical decision theory a more general framework for statistical inference try to explain the scene behind tests

478 views • 28 slides
Hypothesis testing and statistical decision theory Lirong Xia Fall, 2016 Schedule

Hypothesis testing and statistical decision theory Lirong Xia Fall, 2016 Schedule

Hypothesis testing and statistical decision theory Lirong Xia Fall, 2016 Schedule Hypothesis testing Statistical decision theory a more general framework for statistical inference try to explain the scene behind tests

635 views • 28 slides
Introduction to information theory and empirical statistical theory Nan Chen and Andrew J. Majda

Introduction to information theory and empirical statistical theory Nan Chen and Andrew J. Majda

Introduction to information theory and empirical statistical theory Nan Chen and Andrew J. Majda Center for Atmosphere Ocean Science Courant Institute of Mathematical Sciences New York University October 13, 2016 6.1 Introduction Goal of

819 views • 36 slides
Introduction to Gaussian Processes Iain Murray School of Informatics, University of Edinburgh

Introduction to Gaussian Processes Iain Murray School of Informatics, University of Edinburgh

Introduction to Gaussian Processes Iain Murray School of Informatics, University of Edinburgh The problem Learn scalar function of vector values f ( x ) 1 f(x) 0.5 y i 5 0 0 f 0.5 5 0 1 1 0.5 1.5 0.5 0 0.2 0.4 0.6

767 views • 42 slides
Voyaging around nacre with the x-ray shuttle from biomineralisation to prosthetics via mollusc

Voyaging around nacre with the x-ray shuttle from biomineralisation to prosthetics via mollusc

Voyaging around nacre with the x-ray shuttle from biomineralisation to prosthetics via mollusc phylogeny Daniel Chateigner, Magali Morales, Luca Lutterotti CRISMAT-ENSICAEN, IUT-Caen, France TMS 2010, X-ray and neutron symposium calcite -

298 views • 13 slides
Event Calendar SHIMA Daio,

Event Calendar SHIMA Daio,

Event Calendar SHIMA Daio, Funakoshi Funakoshi Atarashiki Maehamashou Park Jan

244 views • 11 slides
First International Workshop on Learning over Multiple Contexts LMCE 2014 Nancy, 19 September

First International Workshop on Learning over Multiple Contexts LMCE 2014 Nancy, 19 September

C. Ferri, J. Hernndez-Orallo, A. Martnez-Us and M.J. Ramrez-Quintana DSIC, UPV, DSIC, UPV, Valncia Valncia, Spain , Spain {cferri, jorallo, admarus, mramirez}@dsic.upv.es First International Workshop on Learning over Multiple

173 views • 14 slides
Teaching VDM & Teaching Formal Methods Ana Paiva apaiva@fe.up.pt www.fe.up.pt/~apaiva OVT

Teaching VDM & Teaching Formal Methods Ana Paiva apaiva@fe.up.pt www.fe.up.pt/~apaiva OVT

Teaching VDM & Teaching Formal Methods Ana Paiva apaiva@fe.up.pt www.fe.up.pt/~apaiva OVT 17: 17TH OVERTURE WORKSHOP -- Ana Paiva 1 Agenda Our semester is structured in 13 lecture So, This talk has 13 sections OVT 17: 17TH OVERTURE

651 views • 42 slides
Infinitely many corks with shadow complexity one . . . . . Hironobu Naoe (Tohoku University)

Infinitely many corks with shadow complexity one . . . . . Hironobu Naoe (Tohoku University)

4-manifolds and its exotic pairs Polyhedron and reconstruction of 4-manifold Main result . . Infinitely many corks with shadow complexity one . . . . . Hironobu Naoe (Tohoku University) October 28, 2015 Topology and Geometry of

593 views • 58 slides
MLSS 06 - Canberra Elements Hierarchical Basis Sparse Grids Sparse Grids Combination

MLSS 06 - Canberra Elements Hierarchical Basis Sparse Grids Sparse Grids Combination

MLSS 06 - Canberra Sparse Grids Jochen Garcke (Very) Short Course on Finite MLSS 06 - Canberra Elements Hierarchical Basis Sparse Grids Sparse Grids Combination Technique Jochen Garcke Regression / Classification via Function

977 views • 68 slides
Wentworth Lawyers COMMERCIAL LAWYERS D IRECTORS D UTIES : L ESSONS FROM THE PAST TO GUIDE

Wentworth Lawyers COMMERCIAL LAWYERS D IRECTORS D UTIES : L ESSONS FROM THE PAST TO GUIDE

Wentworth Lawyers COMMERCIAL LAWYERS D IRECTORS D UTIES : L ESSONS FROM THE PAST TO GUIDE FUTURE CONDUCT FOR D IRECTORS SOME PRACTICAL TIPS FOR NFP S On 30 November 2017, we presented a webinar in partnership with Better Boards Australasia

621 views • 42 slides

More recommend