Introduction to Machine Learning Random Forest: Introduction - - PowerPoint PPT Presentation

Introduction to Machine Learning Random Forest: Introduction

compstat-lmu.github.io/lecture_i2ml

RANDOM FORESTS

Modification of bagging for trees proposed by Breiman (2001): Tree baselearners on bootstrap samples of the data Uses decorrelated trees by randomizing splits (see below) Tree baselearners are usually fully expanded, without aggressive early stopping or pruning, to increase variance of the ensemble

c

• Introduction to Machine Learning – 1 / 7

RANDOM FEATURE SAMPLING

From our analysis of bagging risk we can see that decorrelating trees improves the ensemble Simple randomized approach: At each node of each tree, randomly draw mtry ≤ p candidate features to consider for splitting. Recommended values: Classification: mtry = ⌊√p⌋ Regression: mtry = ⌊p/3⌋

c

• Introduction to Machine Learning – 2 / 7

EFFECT OF ENSEMBLE SIZE

• 2.0

2.5 3.0 3.5 4.0 4.5 5 6 7 8

Sepal.Length Sepal.Width Species

• setosa

versicolor virginica

1 Tree for Iris Dataset

c

• Introduction to Machine Learning – 3 / 7

EFFECT OF ENSEMBLE SIZE

• 2.0

2.5 3.0 3.5 4.0 4.5 5 6 7 8

Sepal.Length Sepal.Width Species

• setosa

versicolor virginica

10 Trees for Iris Dataset

c

• Introduction to Machine Learning – 4 / 7

EFFECT OF ENSEMBLE SIZE

• 2.0

2.5 3.0 3.5 4.0 4.5 5 6 7 8

Sepal.Length Sepal.Width Species

• setosa

versicolor virginica

500 Trees for Iris Dataset

c

• Introduction to Machine Learning – 5 / 7

OUT-OF-BAG ERROR ESTIMATE

With the RF it is possible to obtain unbiased estimates of generalization error directly during training, based on the out-of-bag observations for each tree:

0.04 0.06 0.08 0.10 0.12 50 100 150

Number of Trees MCE

nonspam OOB spam

c

• Introduction to Machine Learning – 6 / 7

OUT-OF-BAG ERROR ESTIMATE

...

1 2 3 4 ... n

...

1 2 3 4 ... n

...

1 2 3 4 ... n

...

1 2 3 4 ... n In-bag observations, used to build the trees {Remember: the same observation can enter the in-bag sample more than once}

• ut-of-bag observations( ), used to evaluate prediction performance ( )

….

Tree 1 Tree 2 Tree 3 Tree M

OOB size: P(not drawn) =

• 1 − 1

n

n

n→∞

− →

1 e ≈ 0.37

Predict all observations with trees that didn’t use it for training and compute average loss of these predictions Similar to 3-CV, can be used for a quick model selection

c

• Introduction to Machine Learning – 7 / 7