Bayesian leave-one-out cross-validation for large data Mns Magnusson - PowerPoint PPT Presentation

Sep 15, 2023 •354 likes •421 views

Bayesian leave-one-out cross-validation for large data Mns Magnusson (Aalto University) Michael Riis Andersen (Technical University of Denmark) Johan Jonasson (University of Gothenburg) Aki Vehtari (Aalto University) Motivation: Model

Bayesian leave-one-out cross-validation for large data Måns Magnusson (Aalto University) Michael Riis Andersen (Technical University of Denmark) Johan Jonasson (University of Gothenburg) Aki Vehtari (Aalto University)
Motivation: Model selection for large data • Bigger data sets and more complex models • We still need to evaluate and compare models • elpd M quantifies how model M generalizes to unseen data ˜ y i � elpd M = log p M (˜ y i | y ) p t (˜ y i ) d ˜ y i , True data generating process Posterior predictive distribution Expected log predictive density 2 DTU Compute Bayesian leave-one-out cross-validation for large data 7.6.2019
Leave-one-out cross-validation • Basic idea: Hold out observation i and predict y i based on y − i • Estimate elpd M using leave-one-out cross-validation (loo) � � n � n elpd loo = 1 log p M ( y i | y − i ) = 1 log p M ( y i | θ ) p M ( θ | y − i ) dθ n n i =1 i =1 • Desirable properties + almost unbiased for large n + straight-forward handling of hierarchical structures • Two major problems - Need to fit the model n times - Need to evaluate predictive densities n times 3 DTU Compute Bayesian leave-one-out cross-validation for large data 7.6.2019
Our contributions: Method � n elpd loo = 1 log p M ( y i | y − i ) n i =1 • We propose a fast approximation for elpd loo 1 Approximate full data posterior q M ( θ | y ) using Variational Bayes/Laplace 2 Compute p M ( y i | y − i ) using importance sampling with q M as proposal 3 Subsample the sum over n using the Hansen-Hurwitz estimator • Solves both problems with leave-one-out CV 1 Only need to fit the model once on the full data set 2 Predictive distributions p M ( y i | y − i ) are only needed for a small subset 4 DTU Compute Bayesian leave-one-out cross-validation for large data 7.6.2019
Our contributions: Results • Theoretical results (under regularity conditions) � p elpd loo → elpd loo for n → ∞ • Extensive empirical results 1 Variational Bayes, Laplace approx., MCMC 2 Bayesian linear regression 3 Hierarchical models • For more details, come see us at poster #231 • Thank you for listening! 5 DTU Compute Bayesian leave-one-out cross-validation for large data 7.6.2019

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Introduction to Data Science: Classifier n 1 n 1 k k Suppose you want to compare two

Classier evaluation Classier evaluation Leave-one-out Cross-Validation Leave-one-out Cross-Validation Leave-one-out Cross-Validation Classier evaluation Classier evaluation Leave-one-out Cross-Validation Resampled validation set

984 views • 51 slides

Occams Razor Sampling Bias : generate the data carefuly Data Snooping : handle the data

recap: Validation and Cross Validation Validation Cross Validation D ( N ) D 1 D 2 D N D train Learning From Data D ( N K ) g 1 g 2 g N Lecture 14 ( x 1 , y 1 ) ( x 2 , y 2 ) ( x N , y N ) Three Learning Principles D

486 views • 15 slides

Bayesian Cross Validation in Regular Asymptotic Theory Information Geometry and Its Applications

Higher Order Analysis of Bayesian Cross Validation in Regular Asymptotic Theory Information Geometry and Its Applications IV In honor of Prof. Amaris 80 th birthday June 12-17, 2016, Liblice, Czech Republic Sumio Watanabe Tokyo Institute of

486 views • 30 slides

Data Mining II Model Validation Heiko Paulheim Why Model Validation? We have seen so far

Data Mining II Model Validation Heiko Paulheim Why Model Validation? We have seen so far Various metrics (e.g., accuracy, F-measure, RMSE, ) Evaluation protocol setups Split Validation Cross Validation Special

1.04k views • 55 slides

Learning From Data Lecture 14 Three Learning Principles Occams Razor Sampling Bias Data

Learning From Data Lecture 14 Three Learning Principles Occams Razor Sampling Bias Data Snooping M. Magdon-Ismail CSCI 4100/6100 recap: Validation and Cross Validation Validation Cross Validation D ( N ) D 1 D 2 D N D train D

1.01k views • 58 slides

15-388/688 - Practical Data Science: Nonlinear modeling, cross-validation, and regularization J.

15-388/688 - Practical Data Science: Nonlinear modeling, cross-validation, and regularization J. Zico Kolter Carnegie Mellon University Fall 2019 1 Outline Example: return to peak demand prediction Overfitting, generalization, and cross

903 views • 67 slides

in Spark Using GPU Minsik Cho, Rajesh Bordawekar IBM TJW Research 1 Cross-Validation 101

Accelerating Cross-Validation in Spark Using GPU Minsik Cho, Rajesh Bordawekar IBM TJW Research 1 Cross-Validation 101 [Wikipedia] Popular Model Validation Technique to avoid overfitting, for better generalization useful when not

430 views • 20 slides

Learning From Data Lecture 13 Validation and Model Selection The Validation Set Model Selection

Learning From Data Lecture 13 Validation and Model Selection The Validation Set Model Selection Cross Validation M. Magdon-Ismail CSCI 4100/6100 recap: Regularization Regularization combats the effects of noise by putting a leash on the

349 views • 31 slides

Importance-Weighted Cross- Importance-Weighted Cross- Validation for Covariate Shift Validation

Importance-Weighted Cross- Importance-Weighted Cross- Validation for Covariate Shift Validation for Covariate Shift (1) (2) Masashi Sugiyama , Benjamin Blankertz , (2,3) (2) Matthias Krauledat , Guido Dornhege , (3,2) Klaus-Robert

623 views • 26 slides

Stratified Cross-Validation in Multi-Label Classification Using Genetic Algorithms 7-8/02/2013

Stratified Cross-Validation in Multi-Label Classification Using Genetic Algorithms 7-8/02/2013 TIN2010-20900-C04 Albacete Index Introduction Multilabel Classification Cross-Validation and Stratified Cross-Validation Methods and

682 views • 43 slides

Progress to Date in A3: Method Transfer, Partial Validation and Cross validation A3: Method

Progress to Date in A3: Method Transfer, Partial Validation and Cross validation A3: Method Transfer, partial and cross validation Team members: In scope Life cycle of a method after first full validation or relation Team

252 views • 10 slides

5: Overtraining and Cross-validation Machine Learning and Real-world Data Helen Yannakoudakis 1

5: Overtraining and Cross-validation Machine Learning and Real-world Data Helen Yannakoudakis 1 Computer Laboratory University of Cambridge Lent 2018 1 Based on slides created by Simone Teufel Last session: smoothing and significance testing

1.56k views • 15 slides

5: Overtraining and Cross-validation Machine Learning and Real-world Data Simone Teufel and Ann

5: Overtraining and Cross-validation Machine Learning and Real-world Data Simone Teufel and Ann Copestake Computer Laboratory University of Cambridge Lent 2017 Last session: smoothing and significance testing You looked at various possible

208 views • 16 slides

From Transfac to HOCOMOCO: using cross-validation and human curation to take most from the high

From Transfac to HOCOMOCO: using cross-validation and human curation to take most from the high throughput data compiling a complete collection of transcription factor binding motifs Vsevolod J. Makeev Vavilov Institute of General Genetics,

721 views • 37 slides

Cross-validation and the Bootstrap In the section we discuss two resampling methods:

Cross-validation and the Bootstrap In the section we discuss two resampling methods: cross-validation and the bootstrap. These methods refit a model of interest to samples formed from the training set, in order to obtain additional

688 views • 44 slides

5: Overtraining and Cross-validation Machine Learning and Real-world Data Paula Buttery (based

5: Overtraining and Cross-validation Machine Learning and Real-world Data Paula Buttery (based on slides created by Simone Teufel) Computer Laboratory University of Cambridge Lent 2019 Last session: smoothing and significance testing You

580 views • 15 slides

Machine Learning July 20, 2016 Basic Concepts: Review Example machine learning problem: Decide

Machine Learning July 20, 2016 Basic Concepts: Review Example machine learning problem: Decide whether to play tennis at a given day. Basic Concepts: Review Example machine learning problem: Decide whether to play tennis at a given day. Input

536 views • 36 slides

Cross-Validation Machine Learning 1 Model selection Very broadly: Choosing the best model using

Cross-Validation Machine Learning 1 Model selection Very broadly: Choosing the best model using given data What makes a model Features Hyper-parameters that control the hypothesis space Example: depth of a decision tree, neural

328 views • 12 slides

Seismic landslide hazard zonation By: M.T.J. Terlien Department of Earth Resources Surveys,

CHAPTER 7 Seismic landslide hazard zonation By: M.T.J. Terlien Department of Earth Resources Surveys, International Institute for Aerospace Survey and Earth Sciences (ITC) Kanaalweg 3, 2628 EB Delft, The Netherlands Tel: +31 15 2748832, Fax:

620 views • 10 slides

Summary of Last Chapter Principles of Knowledge Discovery in Data What is the motivation for

Summary of Last Chapter Principles of Knowledge Discovery in Data What is the motivation for ad-hoc mining process? What defines a data mining task? Fall 2004 Chapter 5: Data Summarization Can we define an ad-hoc mining language?

505 views • 8 slides

CS 6316 Machine Learning Model Selection and Validation Yangfeng Ji Department of Computer

CS 6316 Machine Learning Model Selection and Validation Yangfeng Ji Department of Computer Science University of Virginia Overview Polynominals Polynomial regression (a) d 1 (b) d 3 (c) d 15 2 Boosting Adaboost combines T weak

839 views • 42 slides

Lecture 5: Regularization ML Methodology Aykut Erdem February 2016 Hacettepe University

Lecture 5: Regularization ML Methodology Aykut Erdem February 2016 Hacettepe University Recall from last time Linear Regression y ( x ) = w 0 + w 1 x w = ( w 0 , w 1 ) N i 2 h X t ( n ) ( w 0 + w 1 x ( n ) ) ` ( w ) = n =1

739 views • 48 slides

Time - dela y ed feat u res and a u to - regressi v e models MAC H IN E L E AR N IN G FOR TIME

Time - dela y ed feat u res and a u to - regressi v e models MAC H IN E L E AR N IN G FOR TIME SE R IE S DATA IN P YTH ON Chris Holdgraf Fello w, Berkele y Instit u te for Data Science The past is u sef u l Timeseries data almost al w a y s

624 views • 48 slides

Week 2 Video 5 Cross-Validation and Over-Fitting Over-Fitting Ive mentioned over-fitting a

Week 2 Video 5 Cross-Validation and Over-Fitting Over-Fitting Ive mentioned over-fitting a few times during the last few weeks Fitting to the noise as well as the signal Over-Fitting 25 25 20 20 15 15 10 10 5 5 0 0 0 5 10

796 views • 25 slides