bayesian neural network foundation and practice
play

Bayesian Neural Network: Foundation and Practice Tianyu Cui, Yi - PowerPoint PPT Presentation

Jul 14, 2023 •232 likes •986 views

Bayesian Neural Network: Foundation and Practice Tianyu Cui, Yi Zhao Department of Computer Science Aalto University May 2, 2019 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  1. Bayesian Neural Network: Foundation and Practice Tianyu Cui, Yi Zhao Department of Computer Science Aalto University May 2, 2019 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  2. Outline Introduction to Bayesian Neural Network Dropout as Bayesian Approximation Concrete Dropout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  3. Introduction to Bayesian Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  4. What’s a Neural Network? Figure: A simple NN (left) and a BNN (right)[Blundell, 2015]. Probabilistic interpretation of NN: ▶ Model: y = f ( x ; w ) + ϵ , ϵ ∼ N (0 , σ 2 ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  5. What’s a Neural Network? Figure: A simple NN (left) and a BNN (right)[Blundell, 2015]. Probabilistic interpretation of NN: ▶ Model: y = f ( x ; w ) + ϵ , ϵ ∼ N (0 , σ 2 ) ▶ Likelihood: P ( y | x , w ) = N ( y ; f ( x ; w ) , σ 2 ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  6. What’s a Neural Network? Figure: A simple NN (left) and a BNN (right)[Blundell, 2015]. Probabilistic interpretation of NN: ▶ Model: y = f ( x ; w ) + ϵ , ϵ ∼ N (0 , σ 2 ) ▶ Likelihood: P ( y | x , w ) = N ( y ; f ( x ; w ) , σ 2 ) ▶ Prior: P ( w ) = N ( w ; 0 , σ 2 w I ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  7. What’s a Neural Network? Figure: A simple NN (left) and a BNN (right)[Blundell, 2015]. Probabilistic interpretation of NN: ▶ Model: y = f ( x ; w ) + ϵ , ϵ ∼ N (0 , σ 2 ) ▶ Likelihood: P ( y | x , w ) = N ( y ; f ( x ; w ) , σ 2 ) ▶ Prior: P ( w ) = N ( w ; 0 , σ 2 w I ) ▶ Posterior: P ( w | y , x ) ∝ P ( y | x , w ) P ( w ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  8. What’s a Neural Network? Figure: A simple NN (left) and a BNN (right)[Blundell, 2015]. Probabilistic interpretation of NN: ▶ Model: y = f ( x ; w ) + ϵ , ϵ ∼ N (0 , σ 2 ) ▶ Likelihood: P ( y | x , w ) = N ( y ; f ( x ; w ) , σ 2 ) ▶ Prior: P ( w ) = N ( w ; 0 , σ 2 w I ) ▶ Posterior: P ( w | y , x ) ∝ P ( y | x , w ) P ( w ) ▶ MAP: w ⋆ = argmax w P ( w | y , x ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  9. What’s a Neural Network? Figure: A simple NN (left) and a BNN (right)[Blundell, 2015]. Probabilistic interpretation of NN: ▶ Model: y = f ( x ; w ) + ϵ , ϵ ∼ N (0 , σ 2 ) ▶ Likelihood: P ( y | x , w ) = N ( y ; f ( x ; w ) , σ 2 ) ▶ Prior: P ( w ) = N ( w ; 0 , σ 2 w I ) ▶ Posterior: P ( w | y , x ) ∝ P ( y | x , w ) P ( w ) ▶ MAP: w ⋆ = argmax w P ( w | y , x ) ▶ Prediction: y ′ = f ( x ′ ; w ⋆ ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  10. What’s a Bayesian Neural Network? Figure: A simple NN (left) and a BNN (right)[Blundell, 2015]. What do I mean by being Bayesian? ▶ Model: y = f ( x ; w ) + ϵ , ϵ ∼ N (0 , σ 2 ) ▶ Likelihood: P ( y | x , w ) = N ( y ; f ( x ; w ) , σ 2 ) ▶ Prior: P ( w ) = N ( w ; 0 , σ 2 w I ) ▶ Posterior: P ( w | y , x ) ∝ P ( y | x , w ) P ( w ) ▶ MAP: w ⋆ = argmax w P ( w | y , x ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  11. What’s a Bayesian Neural Network? Figure: A simple NN (left) and a BNN (right)[Blundell, 2015]. What do I mean by being Bayesian? ▶ Model: y = f ( x ; w ) + ϵ , ϵ ∼ N (0 , σ 2 ) ▶ Likelihood: P ( y | x , w ) = N ( y ; f ( x ; w ) , σ 2 ) ▶ Prior: P ( w ) = N ( w ; 0 , σ 2 w I ) ▶ Posterior: P ( w | y , x ) ∝ P ( y | x , w ) P ( w ) ▶ MAP: w ⋆ = argmax w P ( w | y , x ) ▶ Prediction: y ′ = f ( x ′ ; w ), w ∼ P ( w | y , x ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  12. Why Should We Care? Calibrated prediction uncertainty : The models should know what they don’t know. One Example: [Gal, 2017] ▶ We train a model to recognise dog breeds. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  13. Why Should We Care? Calibrated prediction uncertainty : The models should know what they don’t know. One Example: [Gal, 2017] ▶ We train a model to recognise dog breeds. ▶ What would you want your model to do when a cat are given? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  14. Why Should We Care? Calibrated prediction uncertainty : The models should know what they don’t know. One Example: [Gal, 2017] ▶ We train a model to recognise dog breeds. ▶ What would you want your model to do when a cat are given? ▶ A prediction with high uncertainty. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  15. Why Should We Care? Calibrated prediction uncertainty : The models should know what they don’t know. One Example: [Gal, 2017] ▶ We train a model to recognise dog breeds. ▶ What would you want your model to do when a cat are given? ▶ A prediction with high uncertainty. buffer Successful Applications: ▶ Identify adversarial examples [Smith, 2018]. ▶ Adapted exploration rate in RL [Gal, 2016]. ▶ Self-driving car [McAllister, 2017, Michelmore, 2018] and medican analysis [Gal, 2017]. buffer Self-driving car and medican analysis. buffer Self-driving car and medican analysis. buffer Self-driving car and medican analysis. buffer Self-driving car and medican analysis. buffer Self-driving car and medican analysis. . . . . . . . . . . . . . . . . . . . . One simple algorhthm: dropout as Bayesian approximation. . . . . . . . . . . . . . . . . . . . .

  16. How To Learn a Bayesian Neural Network? What’s the difficult part? ▶ P ( w | y , x ) is generally intractable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  17. How To Learn a Bayesian Neural Network? What’s the difficult part? ▶ P ( w | y , x ) is generally intractable ▶ Standard approximate inference (difficult): . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  18. How To Learn a Bayesian Neural Network? What’s the difficult part? ▶ P ( w | y , x ) is generally intractable ▶ Standard approximate inference (difficult): ▶ Laplace Approximation [MacKay, 1992]; . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  19. How To Learn a Bayesian Neural Network? What’s the difficult part? ▶ P ( w | y , x ) is generally intractable ▶ Standard approximate inference (difficult): ▶ Laplace Approximation [MacKay, 1992]; ▶ Hamiltonian Monte Carlo [Neal, 1995]; . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  20. How To Learn a Bayesian Neural Network? What’s the difficult part? ▶ P ( w | y , x ) is generally intractable ▶ Standard approximate inference (difficult): ▶ Laplace Approximation [MacKay, 1992]; ▶ Hamiltonian Monte Carlo [Neal, 1995]; ▶ (Stochastic) Variational Inference [Blundell, 2015]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  21. How To Learn a Bayesian Neural Network? What’s the difficult part? ▶ P ( w | y , x ) is generally intractable ▶ Standard approximate inference (difficult): ▶ Laplace Approximation [MacKay, 1992]; ▶ Hamiltonian Monte Carlo [Neal, 1995]; ▶ (Stochastic) Variational Inference [Blundell, 2015]. ▶ Most of the algorithms above are complicated both in theory and in practice. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  22. How To Learn a Bayesian Neural Network? What’s the difficult part? ▶ P ( w | y , x ) is generally intractable ▶ Standard approximate inference (difficult): ▶ Laplace Approximation [MacKay, 1992]; ▶ Hamiltonian Monte Carlo [Neal, 1995]; ▶ (Stochastic) Variational Inference [Blundell, 2015]. ▶ Most of the algorithms above are complicated both in theory and in practice. ▶ A simple and pratical Bayesian neural network: dropout [Gal, 2016]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  23. Dropout as Bayesian Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  24. Dropout as Bayesian Approximation Dropout works by randomly setting network units to zero. We can obtain the distribution of prediction by repeating forward passing several times. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Bayesian and Neural Network Approaches to PDF Reconstruction Joe Karpie (Columbia University)

Bayesian and Neural Network Approaches to PDF Reconstruction Joe Karpie (Columbia University)

Bayesian and Neural Network Approaches to PDF Reconstruction Joe Karpie (Columbia University) As part of the HadStruc Collaboration Along with C. Carlson, C. Egerer, C. Kallidonis, T. Khan, C. Monahan, K. Orginos, R. Sufian (W&M / JLab)

629 views • 20 slides
Some Bayesian extensions of neural network-based graphon approximations Creighton Heaukulani

Some Bayesian extensions of neural network-based graphon approximations Creighton Heaukulani

Some Bayesian extensions of neural network-based graphon approximations Creighton Heaukulani Joint work with Onno Kampman (Hong Kong) EcoSta 2018, Hong Kong June 2018 Overview 1. Review neural network graphon approximation and its

796 views • 50 slides
Building a Bayesian Network 223 / 385 The construction of a Bayesian network Construction of a

Building a Bayesian Network 223 / 385 The construction of a Bayesian network Construction of a

Chapter 5: Building a Bayesian Network 223 / 385 The construction of a Bayesian network Construction of a Bayesian network for an application domain involves three different tasks: to identify the ( random ) variables and their values;

2.11k views • 162 slides
The Bayesian Network Framework 89 / 384 The network formalism, informal A Bayesian network

The Bayesian Network Framework 89 / 384 The network formalism, informal A Bayesian network

Chapter 4: The Bayesian Network Framework 89 / 384 The network formalism, informal A Bayesian network combines two types of domain knowledge to represent a joint probability distribution: qualitative knowledge: a (minimal) directed I-map

1.63k views • 134 slides
Exact inference (Ch. 14) Bayesian Network A Bayesian network (Bayes net) is: (1) a directed

Exact inference (Ch. 14) Bayesian Network A Bayesian network (Bayes net) is: (1) a directed

Exact inference (Ch. 14) Bayesian Network A Bayesian network (Bayes net) is: (1) a directed graph (2) acyclic Additionally, Bayesian networks are assumed to be defined by conditional probability tables (3) P(x | Parents(x) ) We have actually

673 views • 28 slides
Bayesian Methods for Neural Networks Readings: Bishop, Neural Networks for Pattern Recognition .

Bayesian Methods for Neural Networks Readings: Bishop, Neural Networks for Pattern Recognition .

Bayesian Methods for Neural Networks Readings: Bishop, Neural Networks for Pattern Recognition . Chapter 10. Aaron Courville Bayesian Methods for Neural Networks p.1/29 Bayesian Inference Weve seen Bayesian inference before, remember

542 views • 29 slides
Lecture 18: Concluding Convolutional Neural Networks, Graphical Models as Foundation for Recurrent

Lecture 18: Concluding Convolutional Neural Networks, Graphical Models as Foundation for Recurrent

Lecture 18: Concluding Convolutional Neural Networks, Graphical Models as Foundation for Recurrent Neural Networks and Bayesian Networks Reference: We will be referring to sections etc of Deep Learning by Yoshua Bengio, Ian J. Goodfellow

974 views • 54 slides
Bayes Nets (Ch. 14) Announcements Homework 1 posted Bayesian Network A Bayesian network (Bayes

Bayes Nets (Ch. 14) Announcements Homework 1 posted Bayesian Network A Bayesian network (Bayes

Bayes Nets (Ch. 14) Announcements Homework 1 posted Bayesian Network A Bayesian network (Bayes net) is: (1) a directed graph (2) acyclic Additionally, Bayesian networks are assumed to be defined by conditional probability tables (3) P(x |

463 views • 32 slides
Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural

Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural

Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural IR tasks Neural IR architecture Feature Representations Neural IR query auto completion Neural IR query suggestion Neural IR document

1.48k views • 18 slides
Bayesian networks (2) Lirong Xia Last class Bayesian networks compact, graphical

Bayesian networks (2) Lirong Xia Last class Bayesian networks compact, graphical

Bayesian networks (2) Lirong Xia Last class Bayesian networks compact, graphical representation of a joint probability distribution conditional independence 2 Bayesian network Definition of Bayesian network (Bayes net or

674 views • 23 slides
Classification for High Dimensional Problems Using Bayesian Neural Networks and Dirichlet

Classification for High Dimensional Problems Using Bayesian Neural Networks and Dirichlet

Classification for High Dimensional Problems Using Bayesian Neural Networks and Dirichlet Diffusion Trees Rafdord M. Neal and Jianguo Zhang Presented by Jiwen Li Feb 2, 2006 Outline Bayesian view of feature selection The approach

866 views • 30 slides
Bayesian feedforward Neural networks Seung-Hoon Na Chonbuk National University Neural networks

Bayesian feedforward Neural networks Seung-Hoon Na Chonbuk National University Neural networks

Bayesian feedforward Neural networks Seung-Hoon Na Chonbuk National University Neural networks compared to GPs Neural networks: Nonlinear generalization of GLMs Here, defined by a logistic regression model applied to a logistic

357 views • 24 slides
Bayesian Neural Networks - Presenters Group 1: A Practical Bayesian Framework for Backpropagation

Bayesian Neural Networks - Presenters Group 1: A Practical Bayesian Framework for Backpropagation

Bayesian Neural Networks - Presenters Group 1: A Practical Bayesian Framework for Backpropagation Networks - Slides 2-40 Paul Vicol Shane Baccas George Alexandru Adam Group 2: Priors for Infinite Networks - Slides 41-64

1.37k views • 110 slides
Beyond Uniform Priors in Bayesian Network Structure Learning (for Discrete Bayesian Networks)

Beyond Uniform Priors in Bayesian Network Structure Learning (for Discrete Bayesian Networks)

Beyond Uniform Priors in Bayesian Network Structure Learning (for Discrete Bayesian Networks) Marco Scutari scutari@stats.ox.ac.uk Department of Statistics University of Oxford April 5, 2017 Bayesian Network Structure Learning Learning a BN

536 views • 26 slides
Bayesian method probabilities Application of Bayesian methods Demo: McRobot (P . Lewis)

Bayesian method probabilities Application of Bayesian methods Demo: McRobot (P . Lewis)

N. Salamin c Sept 2007 Lecture outline Bayesian setting Definition Simple example Phylogeny and bioinformatics for evolution MCMC Normalizing constant MCMC in practice Proposal distribution Prior distribution Posterior Bayesian

367 views • 12 slides
Artificial Neural Networks By: Kodi Neumiller Overview What is an artificial neural network

Artificial Neural Networks By: Kodi Neumiller Overview What is an artificial neural network

Artificial Neural Networks By: Kodi Neumiller Overview What is an artificial neural network Perceptron Feed forward Backpropagation Learning rate/momentum What is a neural network? Artificial Human Neurons Each

888 views • 12 slides
LS-SVMlab & Large scale modeling Kristiaan Pelckmans, ESAT- SCD/SISTA J.A.K. Suykens, B. De

LS-SVMlab & Large scale modeling Kristiaan Pelckmans, ESAT- SCD/SISTA J.A.K. Suykens, B. De

LS-SVMlab & Large scale modeling Kristiaan Pelckmans, ESAT- SCD/SISTA J.A.K. Suykens, B. De Moor Content Content I. Overview II. Classification III. Regression IV. Unsupervised Learning V. Time-series VI.

474 views • 35 slides
APPLICATION OF THE BAYESIAN APPROACH AND INVERSE DISPERSION MODELLING TO SOURCE TERM ESTIMATES

APPLICATION OF THE BAYESIAN APPROACH AND INVERSE DISPERSION MODELLING TO SOURCE TERM ESTIMATES

APPLICATION OF THE BAYESIAN APPROACH AND INVERSE DISPERSION MODELLING TO SOURCE TERM ESTIMATES IN BUILT-UP ENVIRONMENTS Franois Septier 1 , Christophe Duchenne 2 , and Patrick Armand 1 1 Universit Bretagne Sud / Lab-STICC DE LA RECHERCHE

294 views • 17 slides
Automated Bayesian Inference for PDE-constrained Inverse Problems Ivan Yashchuk VTT Technical

Automated Bayesian Inference for PDE-constrained Inverse Problems Ivan Yashchuk VTT Technical

Automated Bayesian Inference for PDE-constrained Inverse Problems Ivan Yashchuk VTT Technical Research Centre of Finland, Materials Modeling group Aalto University, Probabilistic Machine Learning group July 9, 2019 1 Model + Simulation + Data

474 views • 8 slides
Sequential Optimal Inference for Experiments with Bayesian Particle Filters Remi Daviet Wharton

Sequential Optimal Inference for Experiments with Bayesian Particle Filters Remi Daviet Wharton

Sequential Optimal Inference for Experiments with Bayesian Particle Filters Remi Daviet Wharton Marketing Department, University of Pennsylvania 1/17 Remi Daviet Sequential Optimal Inference (SOI) 1 / 17 Introduction Behavioral experiments

319 views • 17 slides
Introduction to stochastic dynamical modelling Gavin J Gibson Maxwell Institute for Mathematical

Introduction to stochastic dynamical modelling Gavin J Gibson Maxwell Institute for Mathematical

Introduction to stochastic dynamical modelling Gavin J Gibson Maxwell Institute for Mathematical Sciences Heriot-Watt University Outline Motivation for stochastic modelling Structure of stochastic models Simulation of stochastic

553 views • 30 slides
Inference and Evidence Danil Lakens Eindhoven University of Technology @Lakens /

Inference and Evidence Danil Lakens Eindhoven University of Technology @Lakens /

Inference and Evidence Danil Lakens Eindhoven University of Technology @Lakens / Human-Technology Interaction 1-2-2016 PAGE 1 Why is this interesting? The first principle is that you must not fool yourself - and you are the easiest

1.28k views • 85 slides
Defining the firing rate for a non-Poissonian spike train --- a nerdish study --- Shigeru

Defining the firing rate for a non-Poissonian spike train --- a nerdish study --- Shigeru

German-Japanese program in computational neuroscience OIST, March 2-5, 2010 Defining the firing rate for a non-Poissonian spike train --- a nerdish study --- Shigeru Shinomoto Kyoto Univ., Japan A message from a neuron We have established

374 views • 22 slides
Bayesian networks in Mastermind Ji r Vomlel http://www.utia.cas.cz/vomlel/ 1 Contents

Bayesian networks in Mastermind Ji r Vomlel http://www.utia.cas.cz/vomlel/ 1 Contents

Bayesian networks in Mastermind Ji r Vomlel http://www.utia.cas.cz/vomlel/ 1 Contents Bayesian networks, tasks solved by them, and the junction tree propagation. An application of BNs - adaptive testing The game of

345 views • 15 slides

More recommend