Measuring variable importance in random forests Variable Variable - - PowerPoint PPT Presentation

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

A Comparison of Different Variable Importance Measures

Carolin Strobl (LMU M¨ unchen)

Wien, J¨ anner 2009

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Measuring variable importance in random forests

Start p < 0.001 1 ≤ ≤ 8 > > 8 n = 15 y = (0.4, 0.6) 2 Start p < 0.001 3 ≤ ≤ 14 > > 14 n = 34 y = (0.882, 0.118) 4 n = 32 y = (1, 0) 5 Start p < 0.001 1 ≤ ≤ 12 > > 12 n = 38 y = (0.711, 0.289) 2 Number p < 0.001 3 ≤ ≤ 3 > > 3 n = 25 y = (1, 0) 4 n = 18 y = (0.889, 0.111) 5 Start p < 0.001 1 ≤ ≤ 12 > > 12 Age p < 0.001 2 ≤ ≤ 27 > > 27 n = 10 y = (1, 0) 3 Number p < 0.001 4 ≤ ≤ 4 > > 4 n = 14 y = (0.357, 0.643) 5 n = 9 y = (0.111, 0.889) 6 Start p < 0.001 7 ≤ ≤ 13 > 13 n = 11 y = (0.818, 0.182) 8 n = 37 y = (1, 0) 9 Start p < 0.001 1 ≤ ≤ 8 > > 8 Start p < 0.001 2 ≤ ≤ 1 > > 1 n = 9 y = (0.778, 0.222) 3 n = 13 y = (0.154, 0.846) 4 Start p < 0.001 5 ≤ ≤ 12 > > 12 n = 12 y = (0.833, 0.167) 6 n = 47 y = (1, 0) 7 Start p < 0.001 1 ≤ 8 > 8 n = 13 y = (0.308, 0.692) 2 Age p < 0.001 3 ≤ 87 > 87 n = 36 y = (1, 0) 4 Start p < 0.001 5 ≤ 13 > 13 n = 16 y = (0.75, 0.25) 6 n = 16 y = (1, 0) 7 Number p < 0.001 1 ≤ 5 > 5 Age p < 0.001 2 ≤ 81 > 81 n = 33 y = (1, 0) 3 Start p < 0.001 4 ≤ 12 > 12 n = 13 y = (0.385, 0.615) 5 Start p < 0.001 6 ≤ 15 > 15 n = 12 y = (0.833, 0.167) 7 n = 12 y = (1, 0) 8 n = 11 y = (0.364, 0.636) 9 Start p < 0.001 1 ≤ 12 > 12 Age p < 0.001 2 ≤ 81 > 81 n = 20 y = (0.85, 0.15) 3 n = 16 y = (0.188, 0.812) 4 Start p < 0.001 5 ≤ 13 > 13 n = 11 y = (0.818, 0.182) 6 n = 34 y = (1, 0) 7 Start p < 0.001 1 ≤ 12 > 12 Age p < 0.001 2 ≤ 71 > 71 n = 15 y = (0.667, 0.333) 3 n = 17 y = (0.235, 0.765) 4 Start p < 0.001 5 ≤ 14 > 14 n = 17 y = (0.882, 0.118) 6 n = 32 y = (1, 0) 7 Start p < 0.001 1 ≤ 12 > 12 Age p < 0.001 2 ≤ 68 > 68 Number p < 0.001 3 ≤ 4 > 4 n = 11 y = (1, 0) 4 n = 9 y = (0.556, 0.444) 5 n = 12 y = (0.25, 0.75) 6 n = 49 y = (1, 0) 7 Start p < 0.001 1 ≤ 12 > 12 Age p < 0.001 2 ≤ 18 > 18 n = 10 y = (0.9, 0.1) 3 Number p < 0.001 4 ≤ 4 > 4 n = 12 y = (0.417, 0.583) 5 n = 10 y = (0.2, 0.8) 6 Number p < 0.001 7 ≤ 3 > 3 n = 28 y = (1, 0) 8 n = 21 y = (0.952, 0.048) 9 Start p < 0.001 1 ≤ 8 > 8 Start p < 0.001 2 ≤ 3 > 3 n = 12 y = (0.667, 0.333) 3 n = 14 y = (0.143, 0.857) 4 Age p < 0.001 5 ≤ 136 > 136 n = 47 y = (1, 0) 6 n = 8 y = (0.75, 0.25) 7 Start p < 0.001 1 ≤ 12 > 12 n = 28 y = (0.607, 0.393) 2 Start p < 0.001 3 ≤ 14 > 14 n = 21 y = (0.905, 0.095) 4 n = 32 y = (1, 0) 5 Start p < 0.001 1 ≤ 1 > 1 n = 8 y = (0.375, 0.625) 2 Number p < 0.001 3 ≤ 4 > 4 Age p < 0.001 4 ≤ 125 > 125 n = 31 y = (1, 0) 5 n = 11 y = (0.818, 0.182) 6 n = 31 y = (0.806, 0.194) 7 Start p < 0.001 1 ≤ 14 > 14 Age p < 0.001 2 ≤ 71 > 71 n = 15 y = (0.933, 0.067) 3 Start p < 0.001 4 ≤ 12 > 12 n = 16 y = (0.375, 0.625) 5 n = 15 y = (0.733, 0.267) 6 n = 35 y = (1, 0) 7 Number p < 0.001 1 ≤ 6 > 6 Number p < 0.001 2 ≤ 3 > 3 Start p < 0.001 3 ≤ 13 > 13 n = 10 y = (0.8, 0.2) 4 n = 24 y = (1, 0) 5 n = 37 y = (0.865, 0.135) 6 n = 10 y = (0.5, 0.5) 7 Start p < 0.001 1 ≤ 8 > 8 n = 18 y = (0.5, 0.5) 2 Start p < 0.001 3 ≤ 12 > 12 n = 18 y = (0.833, 0.167) 4 Number p < 0.001 5 ≤ 3 > 3 n = 30 y = (1, 0) 6 n = 15 y = (0.933, 0.067) 7

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Measuring variable importance in random forests

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Measuring variable importance in random forests

◮ Gini importance

mean Gini gain produced by Xj over all trees (can be severely biased due to estimation bias and mutiple testing; Strobl et al., 2007)

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Measuring variable importance in random forests

◮ Gini importance

mean Gini gain produced by Xj over all trees (can be severely biased due to estimation bias and mutiple testing; Strobl et al., 2007)

◮ permutation importance

mean decrease in classification accuracy after permuting Xj over all trees (unbiased when subsampling is used; Strobl et al., 2007)

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

The permutation importance

within each tree t VI (t)(xj) =

• i∈B

(t) I

• yi = ˆ

y(t)

i

• B

(t)

• −
• i∈B

(t) I

• yi = ˆ

y(t)

i,πj

• B

(t)

• ˆ

y(t)

i

= f (t)(xi) = predicted class before permuting ˆ y(t)

i,πj = f (t)(xi,πj) = predicted class after permuting Xj

xi,πj = (xi,1, . . . , xi,j−1, xπj(i),j, xi,j+1, . . . , xi,p

• Note: VI (t)(xj) = 0 by definition, if Xj is not in tree t

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

The permutation importance

• ver all trees:

VI(xj) = ntree

t=1 VI (t)(xj)

ntree

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

What kind of independence corresponds to this kind of permutation?

• bs

Y Xj Z 1 y1 xπj(1),j z1 . . . . . . . . . . . . i yi xπj(i),j zi . . . . . . . . . . . . n yn xπj(n),j zn H0 : Xj ⊥ Y , Z or Xj ⊥ Y ∧ Xj ⊥ Z P(Y , Xj, Z)

H0

= P(Y , Z) · P(Xj)

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

What kind of independence corresponds to this kind of permutation?

the original permutation scheme reflects independence of Xj from both Y and the remaining predictor variables Z

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

What kind of independence corresponds to this kind of permutation?

the original permutation scheme reflects independence of Xj from both Y and the remaining predictor variables Z ⇒ a high variable importance can result from violation of either one!

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Suggestion: Conditional permutation scheme

• bs

Y Xj Z 1 y1 xπj|Z=a(1),j z1 = a 3 y3 xπj|Z=a(3),j z3 = a 27 y27 xπj|Z=a(27),j z27 = a 6 y6 xπj|Z=b(6),j z6 = b 14 y14 xπj|Z=b(14),j z14 = b 33 y33 xπj|Z=b(33),j z33 = b . . . . . . . . . . . . H0 : Xj ⊥ Y |Z P(Y , Xj|Z)

H0

= P(Y |Z) · P(Xj|Z)

• r P(Y |Xj, Z)

H0

= P(Y |Z)

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Technically

◮ use any partition of the feature space for conditioning

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Technically

◮ use any partition of the feature space for conditioning ◮ here: use binary partition already learned by tree

for each tree

◮ determine variables to condition on (via threshold) ◮ extract their cutpoints ◮ generate partition using cutpoints as bisectors

Strobl et al. (2008)

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Toy example

spurious correlation between shoe size and reading skills in school-children

> mycf <- cforest(score ~ ., data = readingSkills, + control = cforest_unbiased(mtry = 2)) > varimp(mycf) nativeSpeaker age shoeSize 12.62926 74.89542 20.01108 > varimp(mycf, conditional = TRUE) nativeSpeaker age shoeSize 11.808192 46.995336 2.092454 from party 0.9-991

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Simulation results

mtry = 1

• 5

15 25 mtry = 3

• 10

30 50 mtry = 8

• 1

2 3 4 5 6 7 8 9 10 11 12 20 40 60 80

variable

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Peptide-binding data

0.005 unconditional 0.005 conditional h2y8 flex8 pol3 *

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Other variable importance measures

◮ partial correlation, standardized beta

conditional effect of Xj given all other variables in the model

◮ “averaging over orderings”

◮ for linear models (relaimpo, Gr¨

• mping, 2006)

LMG Lindeman, Merenda, and Gold (1980), ≈ “dominance analysis” Azen and Budescu (2003) PMVD Feldman (2005)

◮ for GLMs (hier.part, Walsh and Nally, 2008)

“hierarchical partitioning” Chevan and Sutherland (1991)

R2 decomposition

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Other variable importance measures

◮ random forest permutation importance

≈ “averaging over trees” unconditional varimp (randomForest, party, Breiman et al., 2006; Hothorn et al., 2008) conditional varimp (party, Hothorn et al., 0089)

◮ elastic net (elasticnet, caret, Zou and Hastie,

2008; Kuhn, 2008) grouping property: correlated predictors get similar (largest) score

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Desirable (?) properties

◮ proper decomposition: scores sum up to model R2 ◮ non-negativity ◮ exclusion: βj = 0 ⇒ score = 0 ◮ inclusion: βj = 0 ⇒ score = 0

Gr¨

• mping (2007)

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Desirable (?) properties

◮ proper decomposition: scores sum up to model R2

LMG, PMVD

◮ non-negativity ◮ exclusion: βj = 0 ⇒ score = 0 ◮ inclusion: βj = 0 ⇒ score = 0

Gr¨

• mping (2007)

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Desirable (?) properties

◮ proper decomposition: scores sum up to model R2

LMG, PMVD

◮ non-negativity

LMG, PMVD, RF varimp (in principle)

◮ exclusion: βj = 0 ⇒ score = 0 ◮ inclusion: βj = 0 ⇒ score = 0

Gr¨

• mping (2007)

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Desirable (?) properties

◮ proper decomposition: scores sum up to model R2

LMG, PMVD

◮ non-negativity

LMG, PMVD, RF varimp (in principle)

◮ exclusion: βj = 0 ⇒ score = 0

partial correlation, standardized betas, PMVD, RF conditional varimp (in principle), elasticnet?

◮ inclusion: βj = 0 ⇒ score = 0

Gr¨

• mping (2007)

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Desirable (?) properties

◮ proper decomposition: scores sum up to model R2

LMG, PMVD

◮ non-negativity

LMG, PMVD, RF varimp (in principle)

◮ exclusion: βj = 0 ⇒ score = 0

partial correlation, standardized betas, PMVD, RF conditional varimp (in principle), elasticnet?

◮ inclusion: βj = 0 ⇒ score = 0

all Gr¨

• mping (2007)

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Simulation study

dgp: yi = β1 · xi,1 + · · · + β12 · xi,12 + εi, εi

i.i.d.

∼ N(0, 1) X1, . . . , X12 ∼ N(0, Σ) Σ =                     1 0.9 0.9 0.9 0.9 0.9 · · · 0.9 1 0.9 0.9 0.9 0.9 · · · 0.9 0.9 1 0.9 0.9 0.9 · · · 0.9 0.9 0.9 1 0.9 0.9 · · · 0.9 0.9 0.9 0.9 1 0.9 · · · 0.9 0.9 0.9 0.9 0.9 1 · · · 1 · · · . . . . . . . . . . . . . . . . . . ... 1                     Xj X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 βj 10 10 7 7 10 10 7 7

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Linear model

• 10

10 7 7 10 10 7 7 2 4 6 8 10

LiMo

coefficient in dgp (standardized) coefficient

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Linear model

2 4 6 8 10 12 0.5 0.6 0.7 0.8 0.9 1.0

LiMo

best k included R2

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

LMG

• 10

10 7 7 10 10 7 7 0.00 0.05 0.10 0.15

LMG

coefficient in dgp importance

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

LMG

2 4 6 8 10 12 0.75 0.80 0.85 0.90 0.95 1.00

LMG mtry = 12

best k included R2

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

PMVD

• 10

10 7 7 10 10 7 7 0.0 0.1 0.2 0.3 0.4

PMVD

coefficient in dgp importance

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

PMVD

2 4 6 8 10 12 0.75 0.80 0.85 0.90 0.95 1.00

PMVD mtry = 12

best k included R2

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

RF unconditional importance

• 10

10 7 7 10 10 7 7 100 200 300

RF variable importance mtry = 2

coefficient in dgp importance

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

RF unconditional importance

• 10

10 7 7 10 10 7 7 100 200 300 400 500 600

RF variable importance mtry = 4

coefficient in dgp importance

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

RF unconditional importance

• 10

10 7 7 10 10 7 7 200 400 600 800 1000

RF variable importance mtry = 8

coefficient in dgp importance

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

RF unconditional importance

• 10

10 7 7 10 10 7 7 200 400 600 800 1000 1200 1400

RF variable importance mtry = 12

coefficient in dgp importance

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

RF unconditional importance

2 4 6 8 10 12 0.75 0.80 0.85 0.90 0.95 1.00

RF variable importance mtry = 2

best k included R2

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

RF unconditional importance

2 4 6 8 10 12 0.75 0.80 0.85 0.90 0.95 1.00

RF variable importance mtry = 4

best k included R2

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

RF unconditional importance

2 4 6 8 10 12 0.75 0.80 0.85 0.90 0.95 1.00

RF variable importance mtry = 8

best k included R2

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

RF unconditional importance

2 4 6 8 10 12 0.75 0.80 0.85 0.90 0.95 1.00

RF variable importance mtry = 12

best k included R2

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

RF conditional importance

• 10

10 7 7 10 10 7 7 20 40 60 80 100

RF conditional variable importance mtry = 2

coefficient in dgp importance

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

RF conditional importance

• 10

10 7 7 10 10 7 7 50 100 150

RF conditional variable importance mtry = 4

coefficient in dgp importance

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

RF conditional importance

• 10

10 7 7 10 10 7 7 50 100 150 200 250

RF conditional variable importance mtry = 8

coefficient in dgp importance

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

RF conditional importance

• 10

10 7 7 10 10 7 7 50 100 150 200 250 300 350

RF conditional variable importance mtry = 12

coefficient in dgp importance

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

RF conditional importance

2 4 6 8 10 12 0.75 0.80 0.85 0.90 0.95 1.00

RF conditional variable importance mtry = 2

best k included R2

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

RF conditional importance

2 4 6 8 10 12 0.75 0.80 0.85 0.90 0.95 1.00

RF conditional variable importance mtry = 4

best k included R2

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

RF conditional importance

2 4 6 8 10 12 0.75 0.80 0.85 0.90 0.95 1.00

RF conditional variable importance mtry = 8

best k included R2

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

RF conditional importance

2 4 6 8 10 12 0.75 0.80 0.85 0.90 0.95 1.00

RF conditional variable importance mtry = 12

best k included R2

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Elastic net

• 10

10 7 7 10 10 7 7 50 100 150

enet

coefficient in dgp (standardized) coefficient

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Elastic net

2 4 6 8 10 12 0.4 0.5 0.6 0.7 0.8 0.9 1.0

elastic net

best k included R2

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Now wait a second...

what about elastic net’s grouping property?

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Elastic net

0.0 0.2 0.4 0.6 0.8 1.0 100 200 300 400 500

elastic net lambda = 10

fraction Standardized Coefficients 10 10 7 7 10 10 7 7

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Elastic net

0.0 0.2 0.4 0.6 0.8 1.0 50 100 150 200

elastic net lambda = 1

fraction Standardized Coefficients 10 10 7 7 10 10 7 7

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Elastic net

0.0 0.2 0.4 0.6 0.8 1.0 50 100 150 200

elastic net lambda = 0.1

fraction Standardized Coefficients 10 10 7 7 10 10 7 7

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Elastic net

0.0 0.2 0.4 0.6 0.8 1.0 50 100 150 200

elastic net lambda = 0

fraction Standardized Coefficients 10 10 7 7 10 10 7 7

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Summary

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Summary

◮ w.r.t. prediction accuracy: importance measures

following the exclusion principle rule

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Summary

◮ w.r.t. prediction accuracy: importance measures

following the exclusion principle rule standardized betas, PMVD (not quite), RF conditional importance (especially with large mtry) and elastic net (tuned!)

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Summary

◮ w.r.t. prediction accuracy: importance measures

following the exclusion principle rule standardized betas, PMVD (not quite), RF conditional importance (especially with large mtry) and elastic net (tuned!)

◮ RF: not limited to linear model, interactions included,

applicable even if p > 30

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Summary

◮ w.r.t. prediction accuracy: importance measures

following the exclusion principle rule standardized betas, PMVD (not quite), RF conditional importance (especially with large mtry) and elastic net (tuned!)

◮ RF: not limited to linear model, interactions included,

applicable even if p > 30

◮ if you want elastic net to group: don’t tune!?

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Azen, R. and D. V. Budescu (2003). The dominance analysis approach for comparing predictors in multiple regression. Psychological Methods 8(2), 129–48. Breiman, L., A. Cutler, A. Liaw, and M. Wiener (2006). Breiman and Cutler’s Random Forests for Classification and Regression. R package version 4.5-16. Chevan, A. and M. Sutherland (1991). Hierarchical partitioning. The American Statistician 45(2), 90–96. Feldman, B. (2005). Relative importance and value. Technical report. Gr¨

• mping, U. (2006). relaimpo: Relative Importance of Regressors

in Linear Models. R package version 2.1. Gr¨

• mping, U. (2007). Estimators of relative importance for linear

regression based on variance decomposition. The American Statistician 61(2), 139–147.

Variable importance in RF Conditional variable importance in RF Other variable importance measures Summary References

Kuhn, M. (2008). caret: Classification and Regression Training. R package version 3.51. Lindeman, R., P. Merenda, and R. Gold (1980). Introduction to Bivariate and Multivariate Analysis. Glenview: Scott Foresman & Co. Strobl, C., A.-L. Boulesteix, T. Kneib, T. Augustin, and A. Zeileis (2008). Conditional variable importance for random forests. BMC Bioinformatics 9:307. Strobl, C., A.-L. Boulesteix, A. Zeileis, and T. Hothorn (2007). Bias in random forest variable importance measures: Illustrations, sources and a solution. BMC Bioinformatics 8:25. Walsh, C. and R. M. Nally (2008). hier.part: Hierarchical

• Partitioning. R package version 1.0-3.

Zou, H. and T. Hastie (2008). elasticnet: Elastic-Net for Sparse Estimation and Sparse PCA. R package version 1.0-5.