AdaBoost MACH IN E LEARN IN G W ITH TREE-BAS ED MODELS IN P YTH - - PowerPoint PPT Presentation
AdaBoost MACH IN E LEARN IN G W ITH TREE-BAS ED MODELS IN P YTH ON Elie Kawerk Data Scientist Boosting Boosting : Ensemble method combining several weak learners to form a strong learner. Weak learner : Model doing slightly better than random
MACH IN E LEARN IN G W ITH TREE-BAS ED MODELS IN P YTH ON
Elie Kawerk
Data Scientist
MACHINE LEARNING WITH TREE-BASED MODELS IN PYTHON
Boosting: Ensemble method combining several weak learners to form a strong learner. Weak learner: Model doing slightly better than random guessing. Example of weak learner: Decision stump (CART whose maximum depth is 1).
MACHINE LEARNING WITH TREE-BASED MODELS IN PYTHON
Train an ensemble of predictors sequentially. Each predictor tries to correct its predecessor. Most popular boosting methods: AdaBoost, Gradient Boosting.
MACHINE LEARNING WITH TREE-BASED MODELS IN PYTHON
Stands for Adaptive Boosting. Each predictor pays more attention to the instances wrongly predicted by its predecessor. Achieved by changing the weights of training instances. Each predictor is assigned a coefcient α.
α depends on the predictor's training error.
MACHINE LEARNING WITH TREE-BASED MODELS IN PYTHON
MACHINE LEARNING WITH TREE-BASED MODELS IN PYTHON
Learning rate: 0 < η ≤ 1
MACHINE LEARNING WITH TREE-BASED MODELS IN PYTHON
Classication: Weighted majority voting. In sklearn: AdaBoostClassifier . Regression: Weighted average. In sklearn: AdaBoostRegressor .
MACHINE LEARNING WITH TREE-BASED MODELS IN PYTHON
# Import models and utility functions from sklearn.ensemble import AdaBoostClassifier from sklearn.tree import DecisionTreeClassifier from sklearn.metrics import roc_auc_score from sklearn.model_selection import train_test_split # Set seed for reproducibility SEED = 1 # Split data into 70% train and 30% test X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=SEED)
MACHINE LEARNING WITH TREE-BASED MODELS IN PYTHON
# Instantiate a classification-tree 'dt' dt = DecisionTreeClassifier(max_depth=1, random_state=SEED) # Instantiate an AdaBoost classifier 'adab_clf' adb_clf = AdaBoostClassifier(base_estimator=dt, n_estimators=100) # Fit 'adb_clf' to the training set adb_clf.fit(X_train, y_train) # Predict the test set probabilities of positive class y_pred_proba = adb_clf.predict_proba(X_test)[:,1] # Evaluate test-set roc_auc_score adb_clf_roc_auc_score = roc_auc_score(y_test, y_pred_proba)
MACHINE LEARNING WITH TREE-BASED MODELS IN PYTHON
# Print adb_clf_roc_auc_score print('ROC AUC score: {:.2f}'.format(adb_clf_roc_auc_score)) ROC AUC score: 0.99
MACH IN E LEARN IN G W ITH TREE-BAS ED MODELS IN P YTH ON
MACH IN E LEARN IN G W ITH TREE-BAS ED MODELS IN P YTH ON
Elie Kawerk
Data Scientist
MACHINE LEARNING WITH TREE-BASED MODELS IN PYTHON
Sequential correction of predecessor's errors. Does not tweak the weights of training instances. Fit each predictor is trained using its predecessor's residual errors as labels. Gradient Boosted Trees: a CART is used as a base learner.
MACHINE LEARNING WITH TREE-BASED MODELS IN PYTHON
MACHINE LEARNING WITH TREE-BASED MODELS IN PYTHON
MACHINE LEARNING WITH TREE-BASED MODELS IN PYTHON
Regression:
y = y + ηr + ... + ηr
In sklearn: GradientBoostingRegressor . Classication: In sklearn: GradientBoostingClassifier .
pred 1 1 N
MACHINE LEARNING WITH TREE-BASED MODELS IN PYTHON
# Import models and utility functions from sklearn.ensemble import GradientBoostingRegressor from sklearn.model_selection import train_test_split from sklearn.metrics import mean_squared_error as MSE # Set seed for reproducibility SEED = 1 # Split dataset into 70% train and 30% test X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.3, random_state=SEED)
MACHINE LEARNING WITH TREE-BASED MODELS IN PYTHON # Instantiate a GradientBoostingRegressor 'gbt' gbt = GradientBoostingRegressor(n_estimators=300, max_depth=1, random_state=SEED) # Fit 'gbt' to the training set gbt.fit(X_train, y_train) # Predict the test set labels y_pred = gbt.predict(X_test) # Evaluate the test set RMSE rmse_test = MSE(y_test, y_pred)**(1/2) # Print the test set RMSE print('Test set RMSE: {:.2f}'.format(rmse_test)) Test set RMSE: 4.01
MACH IN E LEARN IN G W ITH TREE-BAS ED MODELS IN P YTH ON
MACH IN E LEARN IN G W ITH TREE-BAS ED MODELS IN P YTH ON
Elie Kawerk
Data Scientist
MACHINE LEARNING WITH TREE-BASED MODELS IN PYTHON
GB involves an exhaustive search procedure. Each CART is trained to nd the best split points and features. May lead to CARTs using the same split points and maybe the same features.
MACHINE LEARNING WITH TREE-BASED MODELS IN PYTHON
Each tree is trained on a random subset of rows of the training data. The sampled instances (40%-80% of the training set) are sampled without replacement. Features are sampled (without replacement) when choosing split points. Result: further ensemble diversity. Effect: adding further variance to the ensemble of trees.
MACHINE LEARNING WITH TREE-BASED MODELS IN PYTHON
MACHINE LEARNING WITH TREE-BASED MODELS IN PYTHON
# Import models and utility functions from sklearn.ensemble import GradientBoostingRegressor from sklearn.model_selection import train_test_split from sklearn.metrics import mean_squared_error as MSE # Set seed for reproducibility SEED = 1 # Split dataset into 70% train and 30% test X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.3, random_state=SEED)
MACHINE LEARNING WITH TREE-BASED MODELS IN PYTHON
# Instantiate a stochastic GradientBoostingRegressor 'sgbt' sgbt = GradientBoostingRegressor(max_depth=1, subsample=0.8, max_features=0.2, n_estimators=300, random_state=SEED) # Fit 'sgbt' to the training set sgbt.fit(X_train, y_train) # Predict the test set labels y_pred = sgbt.predict(X_test)
MACHINE LEARNING WITH TREE-BASED MODELS IN PYTHON
# Evaluate test set RMSE 'rmse_test' rmse_test = MSE(y_test, y_pred)**(1/2) # Print 'rmse_test' print('Test set RMSE: {:.2f}'.format(rmse_test)) Test set RMSE: 3.95
MACH IN E LEARN IN G W ITH TREE-BAS ED MODELS IN P YTH ON