Stability Assessment of Tree Ensembles and Psychotrees Using the - - PowerPoint PPT Presentation
Stability Assessment of Tree Ensembles and Psychotrees Using the stablelearner 1 package Lennart Schneider 12 Achim Zeileis 3 Carolin Strobl 2 Ludwig Maximilian University of Munich1 University of Zurich2 University of Innsbruck3 28.02.2020 1
Using the stablelearner1 package Lennart Schneider12 Achim Zeileis3 Carolin Strobl2
Ludwig Maximilian University of Munich1 University of Zurich2 University of Innsbruck3
28.02.2020
1Philipp, Zeileis, and Strobl (2016) and Philipp et al. (2018)
Decision Trees stablelearner stablelearner and Tree Ensembles stablelearner and psychotrees
Decision trees are supervised learners that predict the value of a target variable based on several input variables:
simulated_gene_1 p = 0.015 1 ≤ 8.592 > 8.592 gene_3207 p = 0.028 2 ≤ 8.453 > 8.453 Node 3 (n = 7) Healthy control Bipolar disorder 0.2 0.4 0.6 0.8 1 Node 4 (n = 21) Healthy control Bipolar disorder 0.2 0.4 0.6 0.8 1 Node 5 (n = 33) Healthy control Bipolar disorder 0.2 0.4 0.6 0.8 1
In R, e.g., party or partykit (Hothorn, Hornik, and Zeileis 2006; Zeileis, Hothorn, and Hornik 2008)
◮ Easy to understand and interpret ◮ Handles both numerical and categorical data ◮ But: A single tree can be very non-robust
gender p = 0.01 1 Female Male Node 2 (n = 25) Healthy control Bipolar disorder 0.2 0.4 0.6 0.8 1 Node 3 (n = 36) Healthy control Bipolar disorder 0.2 0.4 0.6 0.8 1
stablelearner (Philipp, Zeileis, and Strobl 2016; Philipp et al. 2018): ◮ A toolkit of descriptive measures and graphical illustrations based on resampling and refitting ◮ Can be used to assess the stability of the variable and cutpoint selection in recursive partitioning
Single Tree Tree Ensemble 1. Original Tree 2. Resampling & Refitting 3. Aggregating & Visualizing
library("partykit") library("stablelearner") data("Bipolar2009", package = "stablelearner") Bipolar2009$simulated_gene_2 <- cut(Bipolar2009$simulated_gene_2, breaks = 3,
str(Bipolar2009, list.len = 6) ## 'data.frame': 61 obs. of 106 variables: ## $ age : int 41 51 29 45 45 29 33 56 48 42 ... ## $ brain_pH : num 6.6 6.67 6.7 6.03 6.35 6.39 6.51 6.07 6.5 6.65 ... ## $ status : Factor w/ 2 levels "Bipolar disorder",..: 1 1 1 1 1 1 1 1 1 1 ... ## $ gender : Factor w/ 2 levels "Female","Male": 2 2 2 2 2 2 1 2 1 2 ... ## $ gene_921 : num 8.33 7.99 8.01 7.83 8.51 ... ## $ gene_4211 : num 6.25 7.02 6.54 6.14 6.65 ... ## [list output truncated] ct <- ctree(status ~ ., data = Bipolar2009) ct_stable <- stabletree(ct)
summary(ct_stable) ## ## Call: ## partykit::ctree(formula = status ~ ., data = Bipolar2009) ## ## Sampler: ## B = 500 ## Method = Bootstrap sampling with 100.0% data ## ## Variable selection overview: ## ## freq * mean * ## simulated_gene_1 0.514 1 0.514 1 ## simulated_gene_2 0.178 0 0.178 0 ## gene_4318 0.128 0 0.128 0 ## gene_3069 0.104 0 0.104 0 ## gene_3207 0.094 1 0.094 1 ## gene_31 0.062 0 0.062 0 ## gene_1440 0.060 0 0.060 0 ## gene_6935 0.046 0 0.048 0 ## gene_9850 0.046 0 0.046 0 ...
Variable selection frequencies
Relative frequency (in %) 20 40 60 80 100
s i m u l a t e d _ g e n e _ 1 s i m u l a t e d _ g e n e _ 2 g e n e _ 4 3 1 8 g e n e _ 3 6 9 g e n e _ 3 2 7 g e n e _ 3 1 g e n e _ 1 4 4 g e n e _ 6 9 3 5 g e n e _ 9 8 5 g e n e _ 1 7 6 7 8 g e n e _ 1 2 2 9 g e n e _ 6 6 9 g e n e _ 2 6 g e n e _ 1 1 7 6 g e n e _ 1 5 3 8 7 g e n e _ 8 4 6 2 b r a i n _ p H g e n e _ 1 3 6 6 9 g e n e _ 2 8 7 g e n e _ 4 4 7 6 g e n e _ 6 1 6 g e n e _ 1 5 1 8 4 g e n e _ 1 1 8 1 g e n d e r g e n e _ 5 2 3 6 g e n e _ 6 9 g e n e _ 9 4 8 9 g e n e _ 1 9 4 5 3 g e n e _ 9 6 7 g e n e _ 9 3 4 g e n e _ 2 1 5 4 5 g e n e _ 1 7 7 2 g e n e _ 1 2 5 8 8 g e n e _ 5 4 9 1 g e n e _ 1 1 6 6 4 g e n e _ 1 7 8 7 g e n e _ 1 9 2 8 g e n e _ 5 6 6 g e n e _ 6 8 8 8 g e n e _ 2 2 1 2 2 g e n e _ 1 1 5 9 5 g e n e _ 2 1 1 8 a g e g e n e _ 9 2 1 g e n e _ 6 4 3 4 g e n e _ 1 4 8 5 7 g e n e _ 7 7 1 9 g e n e _ 1 3 5 3 3 g e n e _ 1 7 2 4 8 g e n e _ 8 8 2 g e n e _ 8 7 1 7 g e n e _ 1 7 3 8 1 g e n e _ 2 6 2 7 g e n e _ 1 2 5 6 g e n e _ 1 7 9 1 8 g e n e _ 1 5 1 1 4 g e n e _ 1 6 7 6 1 g e n e _ 1 5 3 8 3 g e n e _ 7 1 9 3 g e n e _ 7 2 6 g e n e _ 4 2 1 1 g e n e _ 1 9 3 7 8 g e n e _ 1 1 1 8 4 g e n e _ 1 9 6 2 g e n e _ 1 9 8 g e n e _ 2 8 3 8 g e n e _ 2 3 9 g e n e _ 9 4 9 g e n e _ 7 6 2 7 g e n e _ 1 2 8 5 6 g e n e _ 2 1 4 4 2 g e n e _ 9 3 3 g e n e _ 6 7 6 g e n e _ 5 7 6 8 g e n e _ 1 4 6 9 6 g e n e _ 4 8 7 g e n e _ 4 6 1 g e n e _ 1 1 8 5 g e n e _ 1 8 7 7 g e n e _ 1 1 9 8 6 g e n e _ 9 3 3 5 g e n e _ 9 9 7 g e n e _ 7 2 2 6 g e n e _ 4 8 4 g e n e _ 2 9 6 5 g e n e _ 4 8 1 6 g e n e _ 5 6 2 g e n e _ 1 1 9 1 2 g e n e _ 1 2 5 6 g e n e _ 1 8 3 3 9 g e n e _ 1 9 1 6 4 g e n e _ 1 5 5 9 6 g e n e _ 4 8 9 5 g e n e _ 1 2 4 7 1 g e n e _ 2 1 6 2 2 g e n e _ 4 9 6 g e n e _ 1 9 9 7 7 g e n e _ 1 8 1 3 g e n e _ 5 4 1 3 g e n e _ 1 4 3 g e n e _ 9 5 6 9 g e n e _ 1 1 6 7 6 g e n e _ 8 6 7 4 g e n e _ 1 9 2 8 9 g e n e _ 1 5 4 2 2 Counts 8 9 10 11 12 13 100 200 300 400 500
1
simulated_gene_1
x 0.5 1 f(x) 1 x 0.5 1 f(x) 1
Counts 0.0 0.2 0.4 0.6 0.8 1.0 100 200 300 400 500 2 1 Counts 0.0 0.2 0.4 0.6 0.8 1.0 100 200 300 400 500 1
e.g., random forests? Single Tree Tree Ensemble 1. Original Tree Base Learner 2. Resampling & Refitting Resampling & Refitting 3. Aggregating & Visualizing Aggregating & Visualizing Two possibilities:
learner
randomForest package (Liaw and Wiener 2002), or the cforest function (of the party or partykit package) and coerce the forest to a stabletree object using the as.stabletree function
Possibility 1: Use an appropriately specified ctree as a base learner and mimic a cforest of the partykit package:
ct_base <- ctree(status ~ ., data = Bipolar2009, control = ctree_control(mtry = 11, teststat = "quadratic", testtype = "Univariate", mincriterion = 0, saveinfo = FALSE)) cf_stable <- stabletree(ct_base, sampler = subsampling, savetrees = TRUE, B = 500, v = 0.632)
Note that this allows for custom builds, e.g., with respect to the resampling method (bootstrap, subsampling, samplesplitting, jackknife, splithalf or own sampling functions).
summary(cf_stable, original = FALSE) ## ## Call: ## ctree(formula = status ~ ., data = Bipolar2009, control = ctree_control(mtry = 11, ## teststat = "quadratic", testtype = "Univariate", mincriterion = 0, ## saveinfo = FALSE)) ## ## Sampler: ## B = 500 ## Method = Subsampling with 63.2% data ## ## Variable selection overview: ## ## freq mean ## simulated_gene_1 0.152 0.152 ## simulated_gene_2 0.132 0.134 ## gene_4318 0.118 0.118 ## gene_3069 0.098 0.098 ## gene_2807 0.072 0.072 ## gene_1440 0.068 0.068 ## gene_12029 0.052 0.052 ...
Variable selection frequencies
Relative frequency (in %) 20 40 60 80 100
s i m u l a t e d _ g e n e _ 1 s i m u l a t e d _ g e n e _ 2 g e n e _ 4 3 1 8 g e n e _ 3 6 9 g e n e _ 2 8 7 g e n e _ 1 4 4 g e n e _ 1 2 2 9 g e n e _ 6 6 9 g e n e _ 9 4 8 9 b r a i n _ p H g e n e _ 1 3 6 6 9 g e n e _ 1 7 6 7 8 g e n e _ 9 3 3 g e n e _ 7 6 2 7 g e n e _ 1 5 1 8 4 g e n e _ 5 4 9 1 g e n d e r g e n e _ 4 4 7 6 g e n e _ 2 6 2 7 g e n e _ 1 7 9 1 8 g e n e _ 1 7 8 7 g e n e _ 1 4 6 9 6 g e n e _ 1 1 1 8 4 g e n e _ 1 5 3 8 7 g e n e _ 9 8 5 g e n e _ 8 8 2 g e n e _ 6 1 6 g e n e _ 6 9 g e n e _ 1 3 5 3 3 g e n e _ 1 1 6 6 4 g e n e _ 5 2 3 6 g e n e _ 6 7 6 g e n e _ 6 8 8 8 g e n e _ 2 1 1 8 g e n e _ 9 6 7 g e n e _ 1 7 7 2 g e n e _ 5 7 6 8 g e n e _ 9 2 1 g e n e _ 6 4 3 4 g e n e _ 1 9 6 2 g e n e _ 1 9 8 g e n e _ 1 9 4 5 3 g e n e _ 1 5 5 9 6 g e n e _ 7 1 9 3 a g e g e n e _ 9 3 3 5 g e n e _ 1 4 8 5 7 g e n e _ 1 9 2 8 g e n e _ 2 2 1 2 2 g e n e _ 2 6 g e n e _ 1 2 5 8 8 g e n e _ 9 9 7 g e n e _ 9 4 9 g e n e _ 3 2 7 g e n e _ 9 3 4 g e n e _ 8 7 1 7 g e n e _ 1 2 5 6 g e n e _ 8 4 6 2 g e n e _ 1 8 1 3 g e n e _ 1 1 6 7 6 g e n e _ 7 2 6 g e n e _ 6 9 3 5 g e n e _ 7 7 1 9 g e n e _ 4 8 4 g e n e _ 2 9 6 5 g e n e _ 2 3 9 g e n e _ 1 1 9 1 2 g e n e _ 1 8 3 3 9 g e n e _ 1 2 4 7 1 g e n e _ 2 1 6 2 2 g e n e _ 4 9 6 g e n e _ 1 5 1 1 4 g e n e _ 1 9 2 8 9 g e n e _ 4 2 1 1 g e n e _ 1 9 3 7 8 g e n e _ 3 1 g e n e _ 1 1 7 6 g e n e _ 2 8 3 8 g e n e _ 4 8 1 6 g e n e _ 1 2 8 5 6 g e n e _ 1 7 3 8 1 g e n e _ 1 9 1 6 4 g e n e _ 5 6 6 g e n e _ 2 1 4 4 2 g e n e _ 4 8 9 5 g e n e _ 1 1 8 1 g e n e _ 1 6 7 6 1 g e n e _ 1 4 3 g e n e _ 9 5 6 9 g e n e _ 1 1 8 5 g e n e _ 1 1 9 8 6 g e n e _ 1 2 5 6 g e n e _ 2 1 5 4 5 g e n e _ 1 9 9 7 7 g e n e _ 1 5 3 8 3 g e n e _ 5 4 1 3 g e n e _ 4 8 7 g e n e _ 4 6 1 g e n e _ 1 8 7 7 g e n e _ 7 2 2 6 g e n e _ 5 6 2 g e n e _ 1 1 5 9 5 g e n e _ 1 5 4 2 2 g e n e _ 1 7 2 4 8 g e n e _ 8 6 7 4 Counts 8 9 10 11 12 13 100 200 300 400 500
simulated_gene_1
Possibility 2: Fit a random forest externally, e.g., using the cforest function of the partykit package and coerce the forest.
cf_partykit <- cforest(status ~ ., data = Bipolar2009, mtry = 11) cf_partykit_stable <- as.stabletree(cf_partykit) summary(cf_partykit_stable) #. #. #.
The psychotree package provides functionality for model based trees of, e.g., the Rasch model (Strobl, Kopf, and Zeileis 2015). This allows for a global test of Differential Item Functioning (DIF).
library("psychotree") data("VerbalAggression", package = "psychotools") str(VerbalAggression) ## 'data.frame': 316 obs. of 4 variables: ## $ resp : num [1:316, 1:24] 0 0 1 1 1 2 2 0 0 2 ... ## ..- attr(*, "dimnames")=List of 2 ## .. ..$ : NULL ## .. ..$ : chr "S1WantCurse" "S1DoCurse" "S1WantScold" "S1DoScold" ... ## $ resp2 : num [1:316, 1:24] 0 0 1 1 1 1 1 0 0 1 ... ## ..- attr(*, "dimnames")=List of 2 ## .. ..$ : NULL ## .. ..$ : chr "S1WantCurse" "S1DoCurse" "S1WantScold" "S1DoScold" ... ## $ gender: Factor w/ 2 levels "female","male": 2 2 1 1 1 1 1 1 1 1 ... ## $ anger : int 20 11 17 21 17 21 39 21 24 16 ... rt <- raschtree(resp2 ~ gender + anger, data = VerbalAggression, minsize = 30)
gender p = 0.02 1 female male Node 2 (n = 243) 1 5 10 15 20 24 −2.59 3.49 Node 3 (n = 73) 1 5 10 15 20 24 −2.59 3.49
rt_stable <- stabletree(rt, sampler = subsampling, v = 0.632) summary(rt_stable) ## ## Call: ## raschtree(formula = resp2 ~ gender + anger, data = VerbalAggression, ## minsize = 30L) ## ## Sampler: ## B = 500 ## Method = Subsampling with 63.2% data ## ## Variable selection overview: ## ## freq * mean * ## gender 0.270 1 0.270 1 ## anger 0.004 0 0.004 0 ## (* = original tree)
◮ Setting minsize too small results in very unstable item parameter estimates of the Rasch models that are fitted during fitting of the raschtree ◮ Stability results do vary strongly with respect to the sampler (bootstrap vs. subsampling vs. strata sampling), subsampling appears to perform well, see also Strobl et al. (2007)
◮ stablelearner can now be used to assess the stability of tree ensembles by either growing the ensemble using a base learner or by coercing an externally fitted ensemble ◮ stability assessment of psychotrees is technically straightforward
If anyone has experience with IRT resampling, please share your knowledge with me!
Hothorn, T., K. Hornik, and A. Zeileis. 2006. “Unbiased Recursive Partitioning: A Conditional Inference Framework.” Journal of Computational and Graphical Statistics 15 (3): 651–74. https://doi.org/10.1198/106186006x133933. Liaw, A., and M. Wiener. 2002. “Classification and Regression by randomForest.” R News 2 (3): 18–22. Philipp, M., T. Rusch, K. Hornik, and C. Strobl. 2018. “Measuring the Stability of Results from Supervised Statistical Learning.” Journal of Computational and Graphical Statistics 27 (4): 685–700. https://doi.org/10.1080/10618600.2018.1473779. Philipp, M., A. Zeileis, and C. Strobl. 2016. “A Toolkit for Stability Assessment of Tree-Based Learners.” In Proceedings of COMPSTAT 2016 – 22nd International Conference on Computational Statistics, edited by A. Colubi, A. Blanco, and C. Gatu, 315–25. The International Statistical Institute/International Association for Statistical Computing.
Strobl, C., A.-L. Boulesteix, A. Zeileis, and T. Hothorn. 2007. “Bias in Random Forest Variable Importance Measures: Illustrations, Sources and a Solution.” BMC Bioinformatics 8 (25). https://doi.org/10.1186/1471-2105-8-25. Strobl, C., J. Kopf, and A. Zeileis. 2015. “Rasch Trees: A New Method for Detecting Differential Item Functioning in the Rasch Model.” Psychometrika 80 (2): 289–316. https://doi.org/10.1007/s11336-013-9388-3. Zeileis, A., T. Hothorn, and K. Hornik. 2008. “Model-Based Recursive Partitioning.” Journal of Computational and Graphical Statistics 17 (2): 492–514. https://doi.org/10.1198/106186008x319331.