cross validation ensembling
play

Cross Validation & Ensembling Shan-Hung Wu shwu@cs.nthu.edu.tw - PowerPoint PPT Presentation

Jul 16, 2023 •226 likes •1.18k views

Cross Validation & Ensembling Shan-Hung Wu shwu@cs.nthu.edu.tw Department of Computer Science, National Tsing Hua University, Taiwan Machine Learning Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 1 / 34 Outline Cross

  1. Cross Validation & Ensembling Shan-Hung Wu shwu@cs.nthu.edu.tw Department of Computer Science, National Tsing Hua University, Taiwan Machine Learning Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 1 / 34

  2. Outline Cross Validation 1 How Many Folds? Ensemble Methods 2 Voting Bagging Boosting Why AdaBoost Works? Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 2 / 34

  3. Outline Cross Validation 1 How Many Folds? Ensemble Methods 2 Voting Bagging Boosting Why AdaBoost Works? Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 3 / 34

  4. Cross Validation So far, we use the hold out method for: Hyperparameter tuning: validation set Performance reporting: testing set What if we get an “unfortunate” split? Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 4 / 34

  5. Cross Validation So far, we use the hold out method for: Hyperparameter tuning: validation set Performance reporting: testing set What if we get an “unfortunate” split? K -fold cross validation : Split the data set X evenly into K subsets X ( i ) (called folds ) 1 For i = 1 , ··· , K , train f � N ( i ) using all data but the i -th fold ( X \ X ( i ) ) 2 Report the cross-validation error C CV by averaging all testing errors 3 C [ f � N ( i ) ] ’s on X ( i ) Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 4 / 34

  6. Nested Cross Validation Cross validation (CV) can be applied to both hyperparameter tuning and performance reporting E.g, 5 ⇥ 2 nested CV Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 5 / 34

  7. Nested Cross Validation Cross validation (CV) can be applied to both hyperparameter tuning and performance reporting E.g, 5 ⇥ 2 nested CV Inner (2 folds): select 1 hyperparameters giving lowest C CV Can be wrapped by grid search Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 5 / 34

  8. Nested Cross Validation Cross validation (CV) can be applied to both hyperparameter tuning and performance reporting E.g, 5 ⇥ 2 nested CV Inner (2 folds): select 1 hyperparameters giving lowest C CV Can be wrapped by grid search Train final model using both 2 training and validation sets with the selected hyperparameters Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 5 / 34

  9. Nested Cross Validation Cross validation (CV) can be applied to both hyperparameter tuning and performance reporting E.g, 5 ⇥ 2 nested CV Inner (2 folds): select 1 hyperparameters giving lowest C CV Can be wrapped by grid search Train final model using both 2 training and validation sets with the selected hyperparameters Outer (5 folds): report C CV as 3 test error Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 5 / 34

  10. Outline Cross Validation 1 How Many Folds? Ensemble Methods 2 Voting Bagging Boosting Why AdaBoost Works? Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 6 / 34

  11. How Many Folds K ? I Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 7 / 34

  12. How Many Folds K ? I The cross-validation error C CV is an average of C [ f � N ( i ) ] ’s Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 7 / 34

  13. How Many Folds K ? I The cross-validation error C CV is an average of C [ f � N ( i ) ] ’s Regard each C [ f � N ( i ) ] as an estimator of the expected generalization error E X ( C [ f N ]) Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 7 / 34

  14. How Many Folds K ? I The cross-validation error C CV is an average of C [ f � N ( i ) ] ’s Regard each C [ f � N ( i ) ] as an estimator of the expected generalization error E X ( C [ f N ]) C CV is an estimator too, and we have MSE ( C CV ) = E X [( C CV � E X ( C [ f N ])) 2 ] = Var X ( C CV )+ bias ( C CV ) 2 Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 7 / 34

  15. Point Estimation Revisited: Mean Square Error Let ˆ θ n be an estimator of quantity θ related to random variable x mapped from n i.i.d samples of x Mean square error of ˆ θ n : MSE ( ˆ ( ˆ θ n � θ ) 2 ⇤ ⇥ θ n ) = E X Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 8 / 34

  16. Point Estimation Revisited: Mean Square Error Let ˆ θ n be an estimator of quantity θ related to random variable x mapped from n i.i.d samples of x Mean square error of ˆ θ n : MSE ( ˆ ( ˆ θ n � θ ) 2 ⇤ ⇥ θ n ) = E X Can be decomposed into the bias and variance: ( ˆ ( ˆ θ n � E [ ˆ θ n ]+ E [ ˆ ⇥ θ n � θ ) 2 ⇤ ⇥ θ n ] � θ ) 2 ⇤ = E E X Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 8 / 34

  17. Point Estimation Revisited: Mean Square Error Let ˆ θ n be an estimator of quantity θ related to random variable x mapped from n i.i.d samples of x Mean square error of ˆ θ n : MSE ( ˆ ( ˆ θ n � θ ) 2 ⇤ ⇥ θ n ) = E X Can be decomposed into the bias and variance: ( ˆ ( ˆ θ n � E [ ˆ θ n ]+ E [ ˆ ⇥ θ n � θ ) 2 ⇤ ⇥ θ n ] � θ ) 2 ⇤ = E E X θ n ]) 2 +( E [ ˆ θ n ] � θ ) 2 + 2 ( ˆ ( ˆ θ n � E [ ˆ θ n � E [ ˆ θ n ])( E [ ˆ ⇥ ⇤ = E θ n ] � θ ) Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 8 / 34

  18. Point Estimation Revisited: Mean Square Error Let ˆ θ n be an estimator of quantity θ related to random variable x mapped from n i.i.d samples of x Mean square error of ˆ θ n : MSE ( ˆ ( ˆ θ n � θ ) 2 ⇤ ⇥ θ n ) = E X Can be decomposed into the bias and variance: ( ˆ ( ˆ θ n � E [ ˆ θ n ]+ E [ ˆ ⇥ θ n � θ ) 2 ⇤ ⇥ θ n ] � θ ) 2 ⇤ = E E X θ n ]) 2 +( E [ ˆ θ n ] � θ ) 2 + 2 ( ˆ ( ˆ θ n � E [ ˆ θ n � E [ ˆ θ n ])( E [ ˆ ⇥ ⇤ = E θ n ] � θ ) � ˆ ( ˆ θ n � E [ ˆ ( E [ ˆ θ n � E [ ˆ ( E [ ˆ ⇥ θ n ]) 2 ⇤ ⇥ θ n ] � θ ) 2 ⇤ � = E + E + 2E θ n ] θ n ] � θ ) Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 8 / 34

  19. Point Estimation Revisited: Mean Square Error Let ˆ θ n be an estimator of quantity θ related to random variable x mapped from n i.i.d samples of x Mean square error of ˆ θ n : MSE ( ˆ ( ˆ θ n � θ ) 2 ⇤ ⇥ θ n ) = E X Can be decomposed into the bias and variance: ( ˆ ( ˆ θ n � E [ ˆ θ n ]+ E [ ˆ ⇥ θ n � θ ) 2 ⇤ ⇥ θ n ] � θ ) 2 ⇤ = E E X θ n ]) 2 +( E [ ˆ θ n ] � θ ) 2 + 2 ( ˆ ( ˆ θ n � E [ ˆ θ n � E [ ˆ θ n ])( E [ ˆ ⇥ ⇤ = E θ n ] � θ ) � ˆ ( ˆ θ n � E [ ˆ ( E [ ˆ θ n � E [ ˆ ( E [ ˆ ⇥ θ n ]) 2 ⇤ ⇥ θ n ] � θ ) 2 ⇤ � = E + E + 2E θ n ] θ n ] � θ ) � 2 + 2 · 0 · ( E [ ˆ ( ˆ θ n � E [ ˆ E [ ˆ ⇥ θ n ]) 2 ⇤ � = E + θ n ] � θ θ n ] � θ ) Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 8 / 34

  20. Point Estimation Revisited: Mean Square Error Let ˆ θ n be an estimator of quantity θ related to random variable x mapped from n i.i.d samples of x Mean square error of ˆ θ n : MSE ( ˆ ( ˆ θ n � θ ) 2 ⇤ ⇥ θ n ) = E X Can be decomposed into the bias and variance: ( ˆ ( ˆ θ n � E [ ˆ θ n ]+ E [ ˆ ⇥ θ n � θ ) 2 ⇤ ⇥ θ n ] � θ ) 2 ⇤ = E E X θ n ]) 2 +( E [ ˆ θ n ] � θ ) 2 + 2 ( ˆ ( ˆ θ n � E [ ˆ θ n � E [ ˆ θ n ])( E [ ˆ ⇥ ⇤ = E θ n ] � θ ) � ˆ ( ˆ θ n � E [ ˆ ( E [ ˆ θ n � E [ ˆ ( E [ ˆ ⇥ θ n ]) 2 ⇤ ⇥ θ n ] � θ ) 2 ⇤ � = E + E + 2E θ n ] θ n ] � θ ) � 2 + 2 · 0 · ( E [ ˆ ( ˆ θ n � E [ ˆ E [ ˆ ⇥ θ n ]) 2 ⇤ � = E + θ n ] � θ θ n ] � θ ) = Var X ( ˆ θ n )+ bias ( ˆ θ n ) 2 MSE of an unbiased estimator is its variance Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 8 / 34

  21. Example: 5-Fold vs. 10-Fold CV MSE ( C CV ) = E X [( C CV � E X ( C [ f N ])) 2 ] = Var X ( C CV )+ bias ( C CV ) 2 Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 9 / 34

  22. Example: 5-Fold vs. 10-Fold CV MSE ( C CV ) = E X [( C CV � E X ( C [ f N ])) 2 ] = Var X ( C CV )+ bias ( C CV ) 2 Consider polynomial regression where P ( y | x ) = sin ( x )+ ε , ε ⇠ N ( 0 , σ 2 ) Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 9 / 34

  23. Example: 5-Fold vs. 10-Fold CV MSE ( C CV ) = E X [( C CV � E X ( C [ f N ])) 2 ] = Var X ( C CV )+ bias ( C CV ) 2 Consider polynomial regression where P ( y | x ) = sin ( x )+ ε , ε ⇠ N ( 0 , σ 2 ) Let C [ · ] be the MSE of predictions (made by a function) to true labels Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 9 / 34

  24. Example: 5-Fold vs. 10-Fold CV MSE ( C CV ) = E X [( C CV � E X ( C [ f N ])) 2 ] = Var X ( C CV )+ bias ( C CV ) 2 Consider polynomial regression where P ( y | x ) = sin ( x )+ ε , ε ⇠ N ( 0 , σ 2 ) Let C [ · ] be the MSE of predictions (made by a function) to true labels E X ( C [ f N ]) : read line Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 9 / 34

  25. Example: 5-Fold vs. 10-Fold CV MSE ( C CV ) = E X [( C CV � E X ( C [ f N ])) 2 ] = Var X ( C CV )+ bias ( C CV ) 2 Consider polynomial regression where P ( y | x ) = sin ( x )+ ε , ε ⇠ N ( 0 , σ 2 ) Let C [ · ] be the MSE of predictions (made by a function) to true labels E X ( C [ f N ]) : read line bias ( C CV ) : gaps between the red and other solid lines ( E X [ C CV ] ) Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 9 / 34

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Occams Razor Sampling Bias : generate the data carefuly Data Snooping : handle the data

Occams Razor Sampling Bias : generate the data carefuly Data Snooping : handle the data

recap: Validation and Cross Validation Validation Cross Validation D ( N ) D 1 D 2 D N D train Learning From Data D ( N K ) g 1 g 2 g N Lecture 14 ( x 1 , y 1 ) ( x 2 , y 2 ) ( x N , y N ) Three Learning Principles D

486 views • 15 slides
Introduction to Data Science: Classifier n 1 n 1 k k Suppose you want to compare two

Introduction to Data Science: Classifier n 1 n 1 k k Suppose you want to compare two

Classier evaluation Classier evaluation Leave-one-out Cross-Validation Leave-one-out Cross-Validation Leave-one-out Cross-Validation Classier evaluation Classier evaluation Leave-one-out Cross-Validation Resampled validation set

984 views • 51 slides
in Spark Using GPU Minsik Cho, Rajesh Bordawekar IBM TJW Research 1 Cross-Validation 101

in Spark Using GPU Minsik Cho, Rajesh Bordawekar IBM TJW Research 1 Cross-Validation 101

Accelerating Cross-Validation in Spark Using GPU Minsik Cho, Rajesh Bordawekar IBM TJW Research 1 Cross-Validation 101 [Wikipedia] Popular Model Validation Technique to avoid overfitting, for better generalization useful when not

430 views • 20 slides
Importance-Weighted Cross- Importance-Weighted Cross- Validation for Covariate Shift Validation

Importance-Weighted Cross- Importance-Weighted Cross- Validation for Covariate Shift Validation

Importance-Weighted Cross- Importance-Weighted Cross- Validation for Covariate Shift Validation for Covariate Shift (1) (2) Masashi Sugiyama , Benjamin Blankertz , (2,3) (2) Matthias Krauledat , Guido Dornhege , (3,2) Klaus-Robert

623 views • 26 slides
Stratified Cross-Validation in Multi-Label Classification Using Genetic Algorithms 7-8/02/2013

Stratified Cross-Validation in Multi-Label Classification Using Genetic Algorithms 7-8/02/2013

Stratified Cross-Validation in Multi-Label Classification Using Genetic Algorithms 7-8/02/2013 TIN2010-20900-C04 Albacete Index Introduction Multilabel Classification Cross-Validation and Stratified Cross-Validation Methods and

682 views • 43 slides
Progress to Date in A3: Method Transfer, Partial Validation and Cross validation A3: Method

Progress to Date in A3: Method Transfer, Partial Validation and Cross validation A3: Method

Progress to Date in A3: Method Transfer, Partial Validation and Cross validation A3: Method Transfer, partial and cross validation Team members: In scope Life cycle of a method after first full validation or relation Team

252 views • 10 slides
Cross-validation and the Bootstrap In the section we discuss two resampling methods:

Cross-validation and the Bootstrap In the section we discuss two resampling methods:

Cross-validation and the Bootstrap In the section we discuss two resampling methods: cross-validation and the bootstrap. These methods refit a model of interest to samples formed from the training set, in order to obtain additional

688 views • 44 slides
Maximum Entropy Classifier Ensembling using Ge- netic Algorithm for NER in Bengali Asif Ekbal 1

Maximum Entropy Classifier Ensembling using Ge- netic Algorithm for NER in Bengali Asif Ekbal 1

Outline Background and Motivation Classifier Ensembling Genetic Algorithms Proposed Method of Classifier Ensemble Feature Set Used Experimental Results Conclusions Future Works Maximum Entropy Classifier Ensembling using Ge- netic

749 views • 36 slides
STAT 213 Cross-Validation (and Multifactor ANOVA?) Colin Reimer Dawson Oberlin College 12

STAT 213 Cross-Validation (and Multifactor ANOVA?) Colin Reimer Dawson Oberlin College 12

Outline Last Time Cross-Validation STAT 213 Cross-Validation (and Multifactor ANOVA?) Colin Reimer Dawson Oberlin College 12 April 2016 Outline Last Time Cross-Validation Outline Last Time Cross-Validation Outline Last Time

381 views • 25 slides
Criticality experiments and benchmarks for for validation of cross validation of cross sections:

Criticality experiments and benchmarks for for validation of cross validation of cross sections:

Criticality experiments and benchmarks Criticality experiments and benchmarks for for validation of cross validation of cross sections: the neptunium sections: the neptunium case case L.S.Leong, L.Tassan-Got, L.Audouin, C. Paradela,

464 views • 21 slides
CSC411 Tutorial #3 Cross-Validation and Decision Trees February 3, 2016 Boris Ivanovic*

CSC411 Tutorial #3 Cross-Validation and Decision Trees February 3, 2016 Boris Ivanovic*

CSC411 Tutorial #3 Cross-Validation and Decision Trees February 3, 2016 Boris Ivanovic* csc411ta@cs.toronto.edu *Based on the tutorial given by Erin Grant, Ziyu Zhang, and Ali Punjani in previous years. Outline for Today Cross-Validation

355 views • 24 slides
Data Mining II Model Validation Heiko Paulheim Why Model Validation? We have seen so far

Data Mining II Model Validation Heiko Paulheim Why Model Validation? We have seen so far

Data Mining II Model Validation Heiko Paulheim Why Model Validation? We have seen so far Various metrics (e.g., accuracy, F-measure, RMSE, ) Evaluation protocol setups Split Validation Cross Validation Special

1.04k views • 55 slides
15-388/688 - Practical Data Science: Nonlinear modeling, cross-validation, and regularization J.

15-388/688 - Practical Data Science: Nonlinear modeling, cross-validation, and regularization J.

15-388/688 - Practical Data Science: Nonlinear modeling, cross-validation, and regularization J. Zico Kolter Carnegie Mellon University Fall 2019 1 Outline Example: return to peak demand prediction Overfitting, generalization, and cross

903 views • 67 slides
Learning From Data Lecture 14 Three Learning Principles Occams Razor Sampling Bias Data

Learning From Data Lecture 14 Three Learning Principles Occams Razor Sampling Bias Data

Learning From Data Lecture 14 Three Learning Principles Occams Razor Sampling Bias Data Snooping M. Magdon-Ismail CSCI 4100/6100 recap: Validation and Cross Validation Validation Cross Validation D ( N ) D 1 D 2 D N D train D

1.01k views • 58 slides
Learning From Data Lecture 13 Validation and Model Selection The Validation Set Model Selection

Learning From Data Lecture 13 Validation and Model Selection The Validation Set Model Selection

Learning From Data Lecture 13 Validation and Model Selection The Validation Set Model Selection Cross Validation M. Magdon-Ismail CSCI 4100/6100 recap: Regularization Regularization combats the effects of noise by putting a leash on the

349 views • 31 slides
From Transfac to HOCOMOCO: using cross-validation and human curation to take most from the high

From Transfac to HOCOMOCO: using cross-validation and human curation to take most from the high

From Transfac to HOCOMOCO: using cross-validation and human curation to take most from the high throughput data compiling a complete collection of transcription factor binding motifs Vsevolod J. Makeev Vavilov Institute of General Genetics,

721 views • 37 slides
The Relational Algebra and Relational Calculus 5DV119 Introduction to Database Management

The Relational Algebra and Relational Calculus 5DV119 Introduction to Database Management

The Relational Algebra and Relational Calculus 5DV119 Introduction to Database Management Ume a University Department of Computing Science Stephen J. Hegner hegner@cs.umu.se http://www.cs.umu.se/~hegner The Relational Algebra and

511 views • 27 slides
Ashequl Qadir University of Wolverhampton, UK ashequl.qadir@wlv.ac.uk Outline Introduction

Ashequl Qadir University of Wolverhampton, UK ashequl.qadir@wlv.ac.uk Outline Introduction

Ashequl Qadir University of Wolverhampton, UK ashequl.qadir@wlv.ac.uk Outline Introduction Related Works Methodology Results Improvement Challenges Conclusion and Future works Introduction Review Examples Introduction

511 views • 26 slides
cccccc BIBFRAME Ray Denenberg BIBFRAME Pilot Ray Denenberg / Nate Trail / Sally McCallum

cccccc BIBFRAME Ray Denenberg BIBFRAME Pilot Ray Denenberg / Nate Trail / Sally McCallum

cccccc BIBFRAME Ray Denenberg BIBFRAME Pilot Ray Denenberg / Nate Trail / Sally McCallum Library of Congress, United States of America December 5, 2017 Extensions ArtFrame Columbia University. Art objects - paintings, photographs, sculptures,

679 views • 51 slides
Lecture 3: Policy Evaluation Without Knowing How the World Works / Model Free Policy Evaluation

Lecture 3: Policy Evaluation Without Knowing How the World Works / Model Free Policy Evaluation

Lecture 3: Policy Evaluation Without Knowing How the World Works / Model Free Policy Evaluation CS234: RL Emma Brunskill Winter 2018 Material builds on structure from David SIlvers Lecture 4: Model-Free Prediction:

796 views • 60 slides
Emerging Technology Community of Interest February 19, 2019 Advancing Government through

Emerging Technology Community of Interest February 19, 2019 Advancing Government through

Emerging Technology Community of Interest February 19, 2019 Advancing Government through Collaboration, Education and Action Agenda Introductions and Overview ACT-IAC Update Committee Team Updates Featured Topic: 5G Enablement

643 views • 33 slides
A Scalable, Portable, and Memory-Effjcient Lock-Free FIFO Queue Ruslan Nikolaev Systems

A Scalable, Portable, and Memory-Effjcient Lock-Free FIFO Queue Ruslan Nikolaev Systems

A Scalable, Portable, and Memory-Effjcient Lock-Free FIFO Queue Ruslan Nikolaev Systems Software Research Group Virginia Tech, USA Motivation Effjcient concurrent FIFO queues are hard Elimination techniques and relaxed FIFO queues are

178 views • 17 slides
Parallel Algorithms and CS26 S260 Algor gorit ithmic mic Engin ginee eerin ing Yihan

Parallel Algorithms and CS26 S260 Algor gorit ithmic mic Engin ginee eerin ing Yihan

Parallel Algorithms and CS26 S260 Algor gorit ithmic mic Engin ginee eerin ing Yihan han Sun un Implementations * Some of the slides are from MIT 6.712, 6.886 and CMU 15-853. Last Lecture Sc Schedu dule ler: Help you map

818 views • 34 slides
ACLU of Oregon Legislative Advocacy Bill of Rights Action Network Webinar March 15, 2013 1

ACLU of Oregon Legislative Advocacy Bill of Rights Action Network Webinar March 15, 2013 1

ACLU of Oregon Legislative Advocacy Bill of Rights Action Network Webinar March 15, 2013 1 ACLU Legislative Advocacy Tracking almost 1,000 bills Monitor closely all bills that could impact civil liberties in Oregon Today we will

522 views • 17 slides

More recommend