ensemble methods
play

Ensemble methods CS 446 Why ensembles? Standard machine learning - PowerPoint PPT Presentation

Oct 31, 2023 •37 likes •967 views

Ensemble methods CS 446 Why ensembles? Standard machine learning setup: We have some data. We train 10 predictors (3-nn, least squares, SVM, ResNet, . . . ). We output the best on a validation set. 1 / 27 Why ensembles? Standard

  1. Ensemble methods CS 446

  2. Why ensembles? Standard machine learning setup: ◮ We have some data. ◮ We train 10 predictors (3-nn, least squares, SVM, ResNet, . . . ). ◮ We output the best on a validation set. 1 / 27

  3. Why ensembles? Standard machine learning setup: ◮ We have some data. ◮ We train 10 predictors (3-nn, least squares, SVM, ResNet, . . . ). ◮ We output the best on a validation set. Question: can we do better than the best? 1 / 27

  4. Why ensembles? Standard machine learning setup: ◮ We have some data. ◮ We train 10 predictors (3-nn, least squares, SVM, ResNet, . . . ). ◮ We output the best on a validation set. Question: can we do better than the best? What if we use an ensemble/aggregate/combination ? 1 / 27

  5. Why ensembles? Standard machine learning setup: ◮ We have some data. ◮ We train 10 predictors (3-nn, least squares, SVM, ResNet, . . . ). ◮ We output the best on a validation set. Question: can we do better than the best? What if we use an ensemble/aggregate/combination ? We’ll consider two approaches: boosting and bagging . 1 / 27

  6. Bagging 2 / 27

  7. Bagging? This first approach is based upon a simple idea: ◮ If the predictors have indepedent errors , a majority vote of their outputs should be good. Let’s first check this. 3 / 27

  8. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). 4 / 27

  9. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . 4 / 27

  10. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . #classifiers = n = 10 0.25 0.20 0.15 0.10 0.05 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  11. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . #classifiers = n = 20 0.175 0.150 0.125 0.100 0.075 0.050 0.025 0.000 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  12. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . #classifiers = n = 30 0.14 0.12 0.10 0.08 0.06 0.04 0.02 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  13. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . #classifiers = n = 40 0.12 0.10 0.08 0.06 0.04 0.02 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  14. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . #classifiers = n = 50 0.12 0.10 0.08 0.06 0.04 0.02 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  15. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . #classifiers = n = 60 0.10 0.08 0.06 0.04 0.02 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  16. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . Red: all classifiers wrong. #classifiers = n = 2, fraction red = 0.16 0.5 0.4 0.3 0.2 0.1 0.0 0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 4 / 27

  17. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . Red: all classifiers wrong. #classifiers = n = 3, fraction red = 0.064 0.4 0.3 0.2 0.1 0.0 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  18. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . Red: all classifiers wrong. #classifiers = n = 4, fraction red = 0.0256 0.35 0.30 0.25 0.20 0.15 0.10 0.05 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  19. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . Red: all classifiers wrong. #classifiers = n = 5, fraction red = 0.01024 0.35 0.30 0.25 0.20 0.15 0.10 0.05 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  20. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . Red: all classifiers wrong. #classifiers = n = 6, fraction red = 0.004096 0.30 0.25 0.20 0.15 0.10 0.05 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  21. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . Red: all classifiers wrong. #classifiers = n = 7, fraction red = 0.0016384 0.30 0.25 0.20 0.15 0.10 0.05 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  22. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . Green: at least half classifiers wrong. #classifiers = n = 10, fraction green = 0.366897 0.25 0.20 0.15 0.10 0.05 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  23. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . Green: at least half classifiers wrong. #classifiers = n = 20, fraction green = 0.244663 0.175 0.150 0.125 0.100 0.075 0.050 0.025 0.000 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  24. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . Green: at least half classifiers wrong. #classifiers = n = 30, fraction green = 0.175369 0.14 0.12 0.10 0.08 0.06 0.04 0.02 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  25. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . Green: at least half classifiers wrong. #classifiers = n = 40, fraction green = 0.129766 0.12 0.10 0.08 0.06 0.04 0.02 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  26. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . Green: at least half classifiers wrong. #classifiers = n = 50, fraction green = 0.0978074 0.12 0.10 0.08 0.06 0.04 0.02 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  27. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . Green: at least half classifiers wrong. #classifiers = n = 60, fraction green = 0.0746237 0.10 0.08 0.06 0.04 0.02 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Introduction to ensemble methods EN S EMBLE METH ODS IN P YTH ON Romn de las Heras Data

Introduction to ensemble methods EN S EMBLE METH ODS IN P YTH ON Romn de las Heras Data

Introduction to ensemble methods EN S EMBLE METH ODS IN P YTH ON Romn de las Heras Data Scientist, SAP / Agile Solutions Choosing the best model ENSEMBLE METHODS IN PYTHON Surveys ENSEMBLE METHODS IN PYTHON Prerequisite knowledge

582 views • 18 slides
Ensemble Methods Albert Bifet May 2012 COMP423A/COMP523A Data Stream Mining Outline 1.

Ensemble Methods Albert Bifet May 2012 COMP423A/COMP523A Data Stream Mining Outline 1.

Ensemble Methods Albert Bifet May 2012 COMP423A/COMP523A Data Stream Mining Outline 1. Introduction 2. Stream Algorithmics 3. Concept drift 4. Evaluation 5. Classification 6. Ensemble Methods 7. Regression 8. Clustering 9. Frequent

696 views • 23 slides
Decision trees and Ensemble methods Camilo Fosco CS109A Introduction to Data Science Pavlos

Decision trees and Ensemble methods Camilo Fosco CS109A Introduction to Data Science Pavlos

Advanced Section #7: Decision trees and Ensemble methods Camilo Fosco CS109A Introduction to Data Science Pavlos Protopapas and Kevin Rader 1 Outline Decision trees Metrics Tree-building algorithms Ensemble methods

710 views • 58 slides
CSC 411: Lecture 17: Ensemble Methods I Class based on Raquel Urtasun & Rich Zemels

CSC 411: Lecture 17: Ensemble Methods I Class based on Raquel Urtasun & Rich Zemels

CSC 411: Lecture 17: Ensemble Methods I Class based on Raquel Urtasun & Rich Zemels lectures Sanja Fidler University of Toronto March 23, 2016 Urtasun, Zemel, Fidler (UofT) CSC 411: 17-Ensemble Methods I March 23, 2016 1 / 34 Today

751 views • 34 slides
What is it? Instrument or ensemble? Lars Bo Andersen, Humans and IT research seminar, 13/5-2015

What is it? Instrument or ensemble? Lars Bo Andersen, Humans and IT research seminar, 13/5-2015

What is it? Instrument or ensemble? Lars Bo Andersen, Humans and IT research seminar, 13/5-2015 Instrument <> Ensemble Our instruments Concepts and methods Empirical stuff Ensembles From French ensemble "all the parts of a thing

285 views • 12 slides
Overview of Decision Trees, Ensemble Methods and Reinforcement Learning CMSC 678 UMBC Outline

Overview of Decision Trees, Ensemble Methods and Reinforcement Learning CMSC 678 UMBC Outline

Overview of Decision Trees, Ensemble Methods and Reinforcement Learning CMSC 678 UMBC Outline Decision Trees Ensemble Methods Bagging Random Forests Reinforcement Learning Decision Trees 20 Questions: http://20q.net/ Goals: 1.

771 views • 62 slides
Variable selection bias Bias in Ensemble Bias in Ensemble Methods Methods Variable selection

Variable selection bias Bias in Ensemble Bias in Ensemble Methods Methods Variable selection

Variable Selection Variable Selection Variable selection bias Bias in Ensemble Bias in Ensemble Methods Methods Variable selection Variable selection Variable Selection Bias in Classification bias bias rpart Trees and Ensemble Methods

183 views • 6 slides
Chapter 6. Ensemble Methods Wei Pan Division of Biostatistics, School of Public Health,

Chapter 6. Ensemble Methods Wei Pan Division of Biostatistics, School of Public Health,

Chapter 6. Ensemble Methods Wei Pan Division of Biostatistics, School of Public Health, University of Minnesota, Minneapolis, MN 55455 Email: weip@biostat.umn.edu PubH 7475/8475 Wei Pan c Introduction Have a base learner/algorithm;

497 views • 32 slides
Ensemble Methods Roman Kern KDDM2 Roman Kern, ISDS, TU Graz 1 > www.tugraz.at KDDM2 >

Ensemble Methods Roman Kern KDDM2 Roman Kern, ISDS, TU Graz 1 > www.tugraz.at KDDM2 >

www.tugraz.at > Motivation : Consider Kaggle, routinely the winners employ Ensemble Methods ensembles to gain an advantage. > Goal : In this lecture, the main approaches for ensembles will SCIENCE be presented and their main assumptions.

2.16k views • 12 slides
CS570 Data Mining Classification: Ensemble Methods Cengiz Gnay Dept. Math & CS, Emory

CS570 Data Mining Classification: Ensemble Methods Cengiz Gnay Dept. Math & CS, Emory

CS570 Data Mining Classification: Ensemble Methods Cengiz Gnay Dept. Math & CS, Emory University Fall 2013 Some slides courtesy of Han-Kamber-Pei, Tan et al. , and Li Xiong Gnay (Emory) Classification: Ensemble Methods Fall 2013 1 /

580 views • 23 slides
Ensemble Methods Yingyu Liang Computer Sciences 760 Fall 2017

Ensemble Methods Yingyu Liang Computer Sciences 760 Fall 2017

Ensemble Methods Yingyu Liang Computer Sciences 760 Fall 2017 http://pages.cs.wisc.edu/~yliang/cs760/ Some of the slides in these lectures have been adapted/borrowed from materials developed by Mark Craven, David Page, Jude Shavlik, Tom

701 views • 23 slides
Boosting (ensemble) Module 4 - Ensemble classifiers - Objectives module 4: boosting (ensemble

Boosting (ensemble) Module 4 - Ensemble classifiers - Objectives module 4: boosting (ensemble

Boosting (ensemble) Module 4 - Ensemble classifiers - Objectives module 4: boosting (ensemble models) LEARNING PERFORMANCE REPRESENTATION DATA PROBLEM RAW DATA CLUSTERING EVALUATION FEATURES UCI datasets unigrams 20newsgroups SUPERVISED

926 views • 34 slides
A comparison of two ensemble generation methods based on oceanic singular vectors and atmospheric

A comparison of two ensemble generation methods based on oceanic singular vectors and atmospheric

A comparison of two ensemble generation methods based on oceanic singular vectors and atmospheric lagged initialization for decadal predictions MiKlip Project Module A (WP: A Coordination) Camille Marini, Yulia Polkova, Armin K hl and

601 views • 13 slides
Derivative-Free Methods for Machine Learning Tasks Inverse Problem Formulations The Ensemble

Derivative-Free Methods for Machine Learning Tasks Inverse Problem Formulations The Ensemble

EnKF Kovachki & Stuart Derivative-Free Methods for Machine Learning Tasks Inverse Problem Formulations The Ensemble Kalman Filter Ensemble Kalman Filter Numerics Nikola B. Kovachki 1 Andrew M. Stuart 1 1 Computational and Mathematical

353 views • 32 slides
Using Bayesian Model Probability with ensemble methods to quantify uncertainty in reservoir

Using Bayesian Model Probability with ensemble methods to quantify uncertainty in reservoir

Using Bayesian Model Probability with ensemble methods to quantify uncertainty in reservoir modelling with multiple prior scenarios Sigurd Ivar Aanonsen, NORCE Energy Svenn Tveit, NORCE Energy Mathias Alerini, Equinor ASA EnKF Workshop, Voss,

473 views • 43 slides
Earthquake Forecasting Ensemble Methods for Merging Models Alexander K. Christensen Dr.

Earthquake Forecasting Ensemble Methods for Merging Models Alexander K. Christensen Dr.

Earthquake Forecasting Ensemble Methods for Merging Models Alexander K. Christensen Dr. Maximilian J. Werner Motivation What is forecasting? What can we actually forecast? What can we do with this? We developed a new

389 views • 18 slides
Takahito Aoto Nao Hirokawa Naoki Nishida Julian Nagele Overview CoCo 2016 Tools Cops

Takahito Aoto Nao Hirokawa Naoki Nishida Julian Nagele Overview CoCo 2016 Tools Cops

Takahito Aoto Nao Hirokawa Naoki Nishida Julian Nagele Overview CoCo 2016 Tools Cops Scoring StarExec Results TA+NH+JN+NN (CoCo OC) CoCo 2016 IWC 2016 2/10 CoCo 2016 CoCo 2016 5 th International Confluence Competition live

150 views • 10 slides
Human-Oriented Robotics Supervised Learning Part 3/3 Kai Arras Social Robotics Lab, University

Human-Oriented Robotics Supervised Learning Part 3/3 Kai Arras Social Robotics Lab, University

Human-Oriented Robotics Prof. Kai Arras Social Robotics Lab Human-Oriented Robotics Supervised Learning Part 3/3 Kai Arras Social Robotics Lab, University of Freiburg 1 Human-Oriented Robotics Supervised Learning Prof. Kai Arras Social

829 views • 82 slides
Hereditary Substitution for the -Calculus Harley Eades and Aaron Stump Computer Science

Hereditary Substitution for the -Calculus Harley Eades and Aaron Stump Computer Science

Hereditary Substitution for the -Calculus Harley Eades and Aaron Stump Computer Science Saturday, June 22, 13 1 The Big Picture Goal: Prove weak normalization of the -calculus. Tool of choice: hereditary substitution. Novelty:

407 views • 38 slides
Frdric Mothe, Gilles Le Mogudec, Grard Nepveu, Emmanuel Bucket LERFOB : Laboratoire

Frdric Mothe, Gilles Le Mogudec, Grard Nepveu, Emmanuel Bucket LERFOB : Laboratoire

A logging module for oak connected to the Fagaces model Simulation du billonnage de chne en sortie du modle de croissance Fagaces Frdric Mothe, Gilles Le Mogudec, Grard Nepveu, Emmanuel Bucket LERFOB : Laboratoire d'Etude des

369 views • 15 slides
CSC 411 Lecture 5: Ensembles II Roger Grosse, Amir-massoud Farahmand, and Juan Carrasquilla

CSC 411 Lecture 5: Ensembles II Roger Grosse, Amir-massoud Farahmand, and Juan Carrasquilla

CSC 411 Lecture 5: Ensembles II Roger Grosse, Amir-massoud Farahmand, and Juan Carrasquilla University of Toronto UofT CSC 411: 05-Ensembles II 1 / 22 Boosting Recall that an ensemble is a set of predictors whose individual decisions are

663 views • 22 slides
Department of Computer Science CSCI 5622: Machine Learning Chenhao Tan Lecture 13: Boosting

Department of Computer Science CSCI 5622: Machine Learning Chenhao Tan Lecture 13: Boosting

Department of Computer Science CSCI 5622: Machine Learning Chenhao Tan Lecture 13: Boosting Slides adapted from Jordan Boyd-Graber, Chris Ketelsen 1 Learning objectives Understand the general idea behind ensembling Learn about

537 views • 39 slides
2 3 4 Extension of the presentation at ARMS@AAMAS 2015 5 6 7 8 Compute new deployment

2 3 4 Extension of the presentation at ARMS@AAMAS 2015 5 6 7 8 Compute new deployment

Source: robocup.org 2 3 4 Extension of the presentation at ARMS@AAMAS 2015 5 6 7 8 Compute new deployment t 1 t 1 on graph G ( V , C ) iff agent a in vertex v, z 1 av in this case collect

402 views • 19 slides
343H: Honors AI Lecture 24: ML: Decision trees and neural networks 4/22/2014 Kristen Grauman

343H: Honors AI Lecture 24: ML: Decision trees and neural networks 4/22/2014 Kristen Grauman

343H: Honors AI Lecture 24: ML: Decision trees and neural networks 4/22/2014 Kristen Grauman UT Austin Slides courtesy of Dan Klein, UC Berkeley Last time Perceptrons MIRA Dual/kernelized perceptron Support vector machines

450 views • 42 slides

More recommend