intelligible models for classification and regression
play

Intelligible Models for Classification and Regression Yin Lou 1 Rich - PowerPoint PPT Presentation

Jun 20, 2023 •36 likes •436 views

Intelligible Models for Classification and Regression Yin Lou 1 Rich Caruana 2 Johannes Gehrke 1 Department of Computer Science 1 Microsoft Research 2 Cornell University Microsoft Corporation Aug. 13, 2012 Yin Lou (Cornell University)

  1. Intelligible Models for Classification and Regression Yin Lou 1 Rich Caruana 2 Johannes Gehrke 1 Department of Computer Science 1 Microsoft Research 2 Cornell University Microsoft Corporation Aug. 13, 2012 Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 1 / 31

  2. Motivation Simple Model Linear regression, logistic regression Regression: y = β 0 + β 1 x 1 + ... + β n x n Classification: logit ( y ) = β 0 + β 1 x 1 + ... + β n x n Linear Regression Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 2 / 31

  3. Motivation Simple Model Linear regression, logistic regression Regression: y = β 0 + β 1 x 1 + ... + β n x n Classification: logit ( y ) = β 0 + β 1 x 1 + ... + β n x n Linear Regression Intelligible but usually less accurate Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 2 / 31

  4. Motivation Complex Model Random forest, SVMs with RBF kernel, etc. y = f ( x 1 , ..., x n ) Random Forest Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 3 / 31

  5. Motivation Complex Model Random forest, SVMs with RBF kernel, etc. y = f ( x 1 , ..., x n ) Random Forest Unintelligible but usually more accurate Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 3 / 31

  6. Motivation The tradeoff SVMs with RBF Kernel Random Forest Complexity ? Linear Regression Logistic Regression Intelligibility Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 4 / 31

  7. Motivation The tradeoff SVMs with RBF Kernel Random Forest Complexity ? Linear Regression Logistic Regression Intelligibility Intelligibility is important Medical applications Domains where we want scientific understanding Efficient model engineering Impact of features in a ranker Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 4 / 31

  8. Outline Motivation 1 Towards More Accurate Models 2 Algorithms 3 Experiments 4 Discussion 5 Conclusion 6 Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 5 / 31

  9. Outline Motivation 1 Towards More Accurate Models 2 Algorithms 3 Experiments 4 Discussion 5 Conclusion 6 Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 6 / 31

  10. Generalized Additive Models Developed by Hastie and Tibshirani Regression: y = f 1 ( x 1 ) + ... + f n ( x n ) Classification: logit ( y ) = f 1 ( x 1 ) + ... + f n ( x n ) Each feature is “shaped” by shape function f i Intelligible and accurate T. Hastie and R. Tibshirani. Generalized additive models . Chapman & Hall/CRC, 1990. Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 7 / 31

  11. Example 2 + √ x 3 + log x 4 + e x 5 + 2 sin x 6 + ǫ y = x 1 + x 2 2 2 2 1 1 1 0 0 0 −1 −1 −1 −2 −2 −2 0 1 2 3 4 0.0 0.5 1.0 1.5 2.0 0 5 10 15 f 1 ( x 1 ) f 2 ( x 2 ) f 3 ( x 3 ) 2 2 2 1 1 1 0 0 0 −1 −1 −1 −2 −2 −2 0 10 20 30 40 50 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 0 1 2 3 4 5 6 f 4 ( x 4 ) f 5 ( x 5 ) f 6 ( x 6 ) Figure: Shape Functions for Synthetic Dataset. Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 8 / 31

  12. Model Space Model Form Intelligibility Accuracy Linear Model y = β 0 + β 1 x 1 + ... + β n x n +++ + Generalized Linear Model g ( y ) = β 0 + β 1 x 1 + ... + β n x n +++ + Additive Model y = f 1 ( x 1 ) + ... + f n ( x n ) ++ ++ Generalized Additive Model g ( y ) = f 1 ( x 1 ) + ... + f n ( x n ) ++ ++ Full Complexity Model y = f ( x 1 , ..., x n ) + +++ Table: From Linear to Additive Models. Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 9 / 31

  13. Outline Motivation 1 Towards More Accurate Models 2 Algorithms 3 Experiments 4 Discussion 5 Conclusion 6 Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 10 / 31

  14. Fitting GAMs g ( y ) = f 1 ( x 1 ) + ... + f n ( x n ) Shape Functions Splines (SP) Single Tree (TR) Bagged Trees (bagTR) Boosted Trees (bstTR) Boosted Bagged Trees (bbTR) Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 11 / 31

  15. Fitting GAMs g ( y ) = f 1 ( x 1 ) + ... + f n ( x n ) Shape Functions Splines (SP) Single Tree (TR) Bagged Trees (bagTR) Boosted Trees (bstTR) Boosted Bagged Trees (bbTR) Learning Methods Penalized Least Squares (P-LS/P-IRLS) Backfitting (BF) Gradient Boosting (BST) Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 11 / 31

  16. Fitting GAMs g ( y ) = f 1 ( x 1 ) + ... + f n ( x n ) Shape Function: Splines (SP) f i ( x i ) = � d k =1 β k b k ( x i ) 40 30 20 10 0 −10 −20 100 200 300 400 500 Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 12 / 31

  17. Fitting GAMs g ( y ) = f 1 ( x 1 ) + ... + f n ( x n ) Shape Function: Single Tree (TR) f i ( x i ) = RegressionTree ( x i , response ) Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 13 / 31

  18. Fitting GAMs g ( y ) = f 1 ( x 1 ) + ... + f n ( x n ) Shape Function: Bagged Trees (bagTR) f i ( x i ) = 1 � B j =1 RegressionTree ( x i , bootstrap sample j ) B 1 ( + ... + ) B Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 14 / 31

  19. Fitting GAMs g ( y ) = f 1 ( x 1 ) + ... + f n ( x n ) Shape Function: Boosted Trees (bstTR) f i ( x i ) = � B j =1 RegressionTree ( x i , residual j ) + ... + Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 15 / 31

  20. Fitting GAMs g ( y ) = f 1 ( x 1 ) + ... + f n ( x n ) Shape Function: Boosted Bagged Trees (bbTR) f i ( x i ) = � B j =1 BaggedRegressionTree ( x i , residual j ) Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 16 / 31

  21. Fitting GAMs g ( y ) = f 1 ( x 1 ) + ... + f n ( x n ) Learning Method: Penalized Least Squares (P-LS/P-IRLS) Works only on Splines ( f i ( x i ) = � d k =1 β k b k ( x i )) Converts the optimization problem to fitting linear regression/logistic regression with different basis S. Wood. Generalized additive models: an introduction with R . CRC Press, 2006. Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 17 / 31

  22. Fitting GAMs g ( y ) = f 1 ( x 1 ) + ... + f n ( x n ) Learning Method: Backfitting (BF) 1: f j ← 0 2: for m = 1 to M do for j = 1 to n do 3: k � = j f k } N R ← { x ij , y i − � 4: 1 Learn shaping function S : x j → y using R as training dataset 5: f j ← S 6: end for 7: 8: end for T. Hastie and R. Tibshirani. Generalized additive models . Chapman & Hall/CRC, 1990. Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 18 / 31

  23. Fitting GAMs g ( y ) = f 1 ( x 1 ) + ... + f n ( x n ) Learning Method: Gradient Boosting (BST) 1: f j ← 0 2: for m = 1 to M do for j = 1 to n do 3: k f k } N R ← { x ij , y i − � 4: 1 Learn shaping function S : x j → y using R as training dataset 5: f j ← f j + S 6: end for 7: 8: end for J. Friedman. Greedy function approximation: a gradient boosting machine. Annals of Statistics , 29:1189–1232, 2001. Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 19 / 31

  24. Contributions First large-scale study that uses trees as shape function for GAMs Novel methods for using trees as shape functions Largest empirical study of fitting GAMs Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 20 / 31

  25. Outline Motivation 1 Towards More Accurate Models 2 Algorithms 3 Experiments 4 Discussion 5 Conclusion 6 Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 21 / 31

  26. Datasets Dataset Size Attributes %Pos Concrete 1030 9 - Regression Wine 4898 12 - Delta 7192 6 - CompAct 8192 22 - Music 50000 90 - Synthetic 10000 6 - Spambase 4601 58 39.40 Classification Insurance 9823 86 5.97 Magic 19020 11 64.84 Letter 20000 17 49.70 Adult 46033 9/43 16.62 Physics 50000 79 49.72 Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 22 / 31

  27. Methods Shape Least Gradient Backfitting Function Squares Boosting Splines P-LS/P-IRLS BST-SP BF-SP Single Tree N/A BST-TR x BF-TR Bagged Trees N/A BST-bagTR x BF-bagTR Boosted Trees N/A BST-TR x BF-bstTR x Boosted N/A BST-bagTR x BF-bbTR x Bagged Trees Table: Notation for learning methods and shape functions. 9 different methods 5-fold cross validation for each method Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 23 / 31

  28. Results Model Regression Classification Mean Linear/Logistic P-LS/P-IRLS BST-SP BF-SP BST-bagTR2 BST-bagTR3 BST-bagTR4 BST-bagTRX Random Forest Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 24 / 31

  29. Results Model Regression Classification Mean Linear/Logistic 1.68 1.22 1.45 P-LS/P-IRLS BST-SP BF-SP BST-bagTR2 BST-bagTR3 BST-bagTR4 BST-bagTRX Random Forest Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 24 / 31

  30. Results Model Regression Classification Mean Linear/Logistic 1.68 1.22 1.45 P-LS/P-IRLS BST-SP BF-SP BST-bagTR2 BST-bagTR3 BST-bagTR4 BST-bagTRX Random Forest 0.88 0.80 0.84 Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 24 / 31

  31. Results Model Regression Classification Mean Linear/Logistic 1.68 1.22 1.45 P-LS/P-IRLS 1.00 1.00 1.00 BST-SP 1.04 1.00 1.02 BF-SP 1.00 1.00 1.00 BST-bagTR2 BST-bagTR3 BST-bagTR4 BST-bagTRX Random Forest 0.88 0.80 0.84 Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 24 / 31

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Regression l For classification the output(s) is nominal l In regression the output is continuous

Regression l For classification the output(s) is nominal l In regression the output is continuous

Regression l For classification the output(s) is nominal l In regression the output is continuous Function Approximation l Many models could be used Simplest is linear regression Fit data with the best hyper-plane which "goes

592 views • 35 slides
Linear models for classification. Perceptron. Logistic regression. Petr Po s k P. Po

Linear models for classification. Perceptron. Logistic regression. Petr Po s k P. Po

CZECH TECHNICAL UNIVERSITY IN PRAGUE Faculty of Electrical Engineering Department of Cybernetics Linear models for classification. Perceptron. Logistic regression. Petr Po s k P. Po s k c 2015 Artificial Intelligence 1

484 views • 17 slides
Classification or Regression? Regression Classification: want to learn a discrete target

Classification or Regression? Regression Classification: want to learn a discrete target

Classification or Regression? Regression Classification: want to learn a discrete target variable Machine Learning and Pattern Recognition Regression: want to learn a continuous target variable Linear regression,

398 views • 6 slides
Logistic Regression Lecture 6 Logistic Regression Classification Model CS 335

Logistic Regression Lecture 6 Logistic Regression Classification Model CS 335

Logistic Regression Lecture 6 Logistic Regression Classification Model CS 335 Cost function Gradient descent Linear classifiers and decision boundaries Dan Sheldon Classification Example: Hand-Written Digits Input: 20

541 views • 5 slides
10-601 Machine Learning Regression Outline Regression vs Classification Linear regression

10-601 Machine Learning Regression Outline Regression vs Classification Linear regression

10-601 Machine Learning Regression Outline Regression vs Classification Linear regression another discriminative learning method As optimization Gradient descent As matrix inversion (Ordinary Least Squares) Overfitting

1.52k views • 97 slides
Decision Trees TJ Machine Learning Club Classification vs. Regression Classification

Decision Trees TJ Machine Learning Club Classification vs. Regression Classification

Decision Trees TJ Machine Learning Club Classification vs. Regression Classification Classifying photos of fruits Determining whether tumor is benign or malignant Regression Predicting COVID-19 cases given demographic data

505 views • 26 slides
Multiclass Classification Machine Learning So far: Binary Classification We have seen linear

Multiclass Classification Machine Learning So far: Binary Classification We have seen linear

Multiclass Classification Machine Learning So far: Binary Classification We have seen linear models Learning algorithms for linear models Perceptron, Winnow, Adaboost, SVM We will see more soon: Nave Bayes, Logistic Regression

635 views • 24 slides
Theory for Minimum Norm Interpolation: Regression and Classification in High Dimensions Tengyuan

Theory for Minimum Norm Interpolation: Regression and Classification in High Dimensions Tengyuan

Intro. Min-norm Interpolant Regression Classification Theory for Minimum Norm Interpolation: Regression and Classification in High Dimensions Tengyuan Liang Classification: with Pragya Sur (Harvard) Regression: with Sasha Rakhlin (MIT), Xiyu

799 views • 68 slides
Classification with generative models 2 DSE 210 Classification with parametrized models

Classification with generative models 2 DSE 210 Classification with parametrized models

Classification with generative models 2 DSE 210 Classification with parametrized models Classifiers with a fixed number of parameters can represent a limited set of functions. Learning a model is about picking a good approximation. Typically the

427 views • 39 slides
Analysis of variance and regression Other types of regression models Other types of regression

Analysis of variance and regression Other types of regression models Other types of regression

Analysis of variance and regression Other types of regression models Other types of regression models Counts: Poisson models Ordinal data: Proportional odds models Survival analysis (censored, time-to-event data): Cox proportional

1.02k views • 58 slides
Learning From Data Lecture 8 Linear Classification and Regression Linear Classification Linear

Learning From Data Lecture 8 Linear Classification and Regression Linear Classification Linear

Learning From Data Lecture 8 Linear Classification and Regression Linear Classification Linear Regression M. Magdon-Ismail CSCI 4100/6100 recap: Approximation Versus Generalization VC Analysis Bias-Variance Analysis E out E in + ( d

390 views • 23 slides
Logistic regression on Sonar Machine Learning Toolbox Classification models Categorical

Logistic regression on Sonar Machine Learning Toolbox Classification models Categorical

MACHINE LEARNING TOOLBOX Logistic regression on Sonar Machine Learning Toolbox Classification models Categorical (i.e. qualitative) target variable Example: will a loan default? Still a form of supervised learning Use a

315 views • 27 slides
E9 205 Machine Learning for Signal Processing Linear Models for Regression and 23-09-2019

E9 205 Machine Learning for Signal Processing Linear Models for Regression and 23-09-2019

E9 205 Machine Learning for Signal Processing Linear Models for Regression and 23-09-2019 Classification Linear Regression Solution to Maximum Likelihood problem is the least squares solution Pseudo Inverse Based Solution Bishop - PRML

84 views • 7 slides
Linear Models Continued: Perceptron & Logistic Regression CMSC 723 / LING 723 / INST 725

Linear Models Continued: Perceptron & Logistic Regression CMSC 723 / LING 723 / INST 725

Linear Models Continued: Perceptron & Logistic Regression CMSC 723 / LING 723 / INST 725 Marine Carpuat Slides credit: Graham Neubig, Jacob Eisenstein Linear Models for Classification Feature function representation Weights Nave

483 views • 25 slides
Introduction to Machine Learning Classification and Regression Trees (CART): Basics

Introduction to Machine Learning Classification and Regression Trees (CART): Basics

Introduction to Machine Learning Classification and Regression Trees (CART): Basics compstat-lmu.github.io/lecture_i2ml TREE MODEL AND PREDICTION Classification and Regression Trees, introduced by Breiman Binary splits are constructed top-down

415 views • 6 slides
Data Mining II Regression Heiko Paulheim Regression Classification covered in Data

Data Mining II Regression Heiko Paulheim Regression Classification covered in Data

Data Mining II Regression Heiko Paulheim Regression Classification covered in Data Mining I predict a label from a finite collection e.g., true/false, low/medium/high, ... Regression predict a numerical value

831 views • 67 slides
CS 310 Advanced Data Structures and Algorithms Binary Search Tree June 26, 2017 Tong Wang

CS 310 Advanced Data Structures and Algorithms Binary Search Tree June 26, 2017 Tong Wang

CS 310 Advanced Data Structures and Algorithms Binary Search Tree June 26, 2017 Tong Wang UMass Boston CS 310 June 26, 2017 1 / 30 Binary Search Tree A binary search tree is a binary tree: If y is a node in the left subtree of x, then

343 views • 30 slides
Definitions Binary Search Tree: n Tree: hierarchical structure made of nodes with father-son

Definitions Binary Search Tree: n Tree: hierarchical structure made of nodes with father-son

Advanced Programming Binary Search Tree Binary Search Tree Introduction Binary Search Tree, o BST implement dictionaries or ordered queues elements stored are ordered according to a key Efficient (log n) operations S EARCH , M INIMUM , M

362 views • 32 slides
Massive Data Algorithmics Lecture 3: External Search Trees Massive Data Algorithmics Lecture 3:

Massive Data Algorithmics Lecture 3: External Search Trees Massive Data Algorithmics Lecture 3:

BST B-trees Summary Massive Data Algorithmics Lecture 3: External Search Trees Massive Data Algorithmics Lecture 3: External Search Trees BST Definition B-trees Blocking Summary Dynamic Binary search tree Standard method for search

528 views • 21 slides
Trees CoSc 450: Programming Paradigms 08 The definition of a tree CoSc 450: Programming

Trees CoSc 450: Programming Paradigms 08 The definition of a tree CoSc 450: Programming

CoSc 450: Programming Paradigms 08 Trees CoSc 450: Programming Paradigms 08 The definition of a tree CoSc 450: Programming Paradigms 08 The definition of a tree The empty tree is a tree. A nonempty tree tree has three parts. root

348 views • 23 slides
Introduction CSCE423/823 CSCE423/823 Computer Science & Engineering 423/823 Dynamic

Introduction CSCE423/823 CSCE423/823 Computer Science & Engineering 423/823 Dynamic

Introduction CSCE423/823 CSCE423/823 Computer Science & Engineering 423/823 Dynamic programming is a technique for solving optimization Introduction Design and Analysis of Algorithms Introduction problems Rod Cutting Rod Cutting Key

796 views • 7 slides
Binary Search Trees get(), put() and remove() Dictionary Operations: Data Structure Worst

Binary Search Trees get(), put() and remove() Dictionary Operations: Data Structure Worst

Complexity Of Dictionary Operations Binary Search Trees get(), put() and remove() Dictionary Operations: Data Structure Worst Case Expected get(key) Hash Table O(n) O(1) put(key, value) remove(key) Binary Search O(n) O(log

150 views • 10 slides
BST property: for each node v with key k , nodes in left subtree have keys k nodes in

BST property: for each node v with key k , nodes in left subtree have keys k nodes in

BST property: for each node v with key k , nodes in left subtree have keys k nodes in right subtree have keys k Have seen simple BST operations : Search Minimum Maximum Predecessor Successor Insert Delete

325 views • 18 slides
Programming with dependent types: passing fad or useful tool? Xavier Leroy INRIA

Programming with dependent types: passing fad or useful tool? Xavier Leroy INRIA

Programming with dependent types: passing fad or useful tool? Xavier Leroy INRIA Paris-Rocquencourt IFIP WG 2.8, 2009-06 X. Leroy (INRIA) Dependently-typed programming 2009-06 1 / 22 Dependent types In a very general sense: all frameworks

585 views • 22 slides

More recommend