Introduction to Machine Learning Random Forest: Introduction compstat-lmu.github.io/lecture_i2ml
RANDOM FORESTS Modification of bagging for trees proposed by Breiman (2001): Tree baselearners on bootstrap samples of the data Uses decorrelated trees by randomizing splits (see below) Tree baselearners are usually fully expanded, without aggressive early stopping or pruning, to increase variance of the ensemble � c Introduction to Machine Learning – 1 / 7
RANDOM FEATURE SAMPLING From our analysis of bagging risk we can see that decorrelating trees improves the ensemble Simple randomized approach: At each node of each tree, randomly draw mtry ≤ p candidate features to consider for splitting. Recommended values: Classification: mtry = ⌊√ p ⌋ Regression: mtry = ⌊ p / 3 ⌋ � c Introduction to Machine Learning – 2 / 7
EFFECT OF ENSEMBLE SIZE 1 Tree for Iris Dataset 4.5 ● ● ● 4.0 ● ● ● ● ● ● ● ● ● ● ● ● ● 3.5 ● ● ● ● ● ● Species Sepal.Width ● ● ● ● ● ● ● ● ● ● ● ● setosa ● ● ● ● ● versicolor ● ● ● ● 3.0 ● ● ● ● ● ● virginica ● ● ● 2.5 ● 2.0 5 6 7 8 Sepal.Length � c Introduction to Machine Learning – 3 / 7
EFFECT OF ENSEMBLE SIZE 10 Trees for Iris Dataset 4.5 ● ● ● 4.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 3.5 ● ● ● ● ● ● Species Sepal.Width ● ● ● ● ● ● ● ● ● ● ● ● setosa ● ● ● ● ● versicolor ● ● ● ● 3.0 ● ● ● ● ● ● virginica ● 2.5 ● 2.0 5 6 7 8 Sepal.Length � c Introduction to Machine Learning – 4 / 7
EFFECT OF ENSEMBLE SIZE 500 Trees for Iris Dataset 4.5 ● ● ● 4.0 ● ● ● ● ● ● ● ● ● ● ● ● ● 3.5 ● ● ● ● ● ● Species Sepal.Width ● ● ● ● ● ● ● ● ● ● ● ● setosa ● ● ● ● ● versicolor ● ● ● ● 3.0 ● ● ● ● ● ● virginica ● 2.5 ● 2.0 5 6 7 8 Sepal.Length � c Introduction to Machine Learning – 5 / 7
OUT-OF-BAG ERROR ESTIMATE With the RF it is possible to obtain unbiased estimates of generalization error directly during training, based on the out-of-bag observations for each tree: 0.12 0.10 nonspam MCE 0.08 OOB spam 0.06 0.04 0 50 100 150 Number of Trees � c Introduction to Machine Learning – 6 / 7
OUT-OF-BAG ERROR ESTIMATE Tree 1 Tree 2 Tree 3 Tree M ... ... ... ... 1 1 1 1 2 2 2 2 …. 3 3 3 3 4 4 4 4 ... ... ... ... n n n n In-bag observations, used to build the trees {Remember: the same observation can enter the in-bag sample more than once} out-of-bag observations( ), used to evaluate prediction performance ( ) � n n →∞ 1 − 1 1 � OOB size: P ( not drawn ) = − → e ≈ 0 . 37 n Predict all observations with trees that didn’t use it for training and compute average loss of these predictions Similar to 3-CV, can be used for a quick model selection � c Introduction to Machine Learning – 7 / 7
Recommend
More recommend
