Feature Selection ZHI LI Fenys Lab October 3, 2019 What is - - PowerPoint PPT Presentation

Feature Selection

ZHI LI Fenyö’s Lab October 3, 2019

What is Feature?

X (Independent) Variable Predictor Covariate Face; leg; tail; Hair texture/style Impeachment Trade war Whatever you can think of..

What methods are out there?

Filter methods: Correlation Embedded methods: Regularization Wrapper methods: Forward selection Depend on the way to combine selection algorithm and the model building

Filter methods

• Numeric Outcomes:
• numeric predictors:
• correlation (linear, nonlinear); distance
• mutual information
• categorical predictors:
• t-statistics
• ANOVA (multiple predictors)
• mutual information
• Categorical Outcomes:
• numeric predictors:
• ROC
• Relief
• mutual information
• categorical predictors:
• Chi-square
• Fisher’s exact
• Odds ratio

5 5

Minimizing the loss function, L (sum of squared errors/MSR): ˆ w0 = 1 n X yi − ˆ w1 1 n X xi

Slide courtesy of Wilson with minor adaption

ˆ w1 = n P xiyi − (P xi) (P yi) n (P x2

i ) − (P xi)2

= cov(X, Y ) var(X)

n2 n2

Pearson’s correlation

Feature Importance Measures

Correlation Based Feature selection

Spearman’s Correlation

Spearman’s Correlation

Denominator: Numerator: Courtesy of Glen_b from stackexchange

Other non-linear Measures

Other non-linear Measures

• MIC

Other non-linear Measures

Filter methods

Other non-linear Measures

t-statistics (categorical Predictor)

Categorical Outcome

ROC

18

Evaluation of Binary Classification Models

18

Actual Predicted 0 1 1 True Negative False Negative True Positive False Positive

• False Positive Rate = FP/(FP+TN) – fraction of label 0 predicted to be label 1
• Accuracy = (TP+TN)/total - fraction of correct prediction
• Precision = TP/(TP+FP) – fraction of correct among positive predictions
• Sensitivity = TP/(TP+FN) – fraction of correct predictions among label 1. Also

called true positive rate and recall.

• Specificity = TN/(TN+FP) – fraction of correct predictions among label 0

Slide courtesy of David

Relief algorithm

Relief algorithm

More on categorical outcome with numeric predictors

Categorical Outcome with Categorical Variable

Categorical Outcome with Categorical Variable

Chi-square test: Large number Easy to calculate Approximation Fisher’s exact: 2 x 2 chi-squared test Small number Hard to calculate Exact

Fisher’s exact

Consequences of Using Non-informative Predictors

Embedded methods

!"##$ = &'' + ) *

+,- .

/+ &0123 &3243##0$5 = &'' + ) *

+,- .

/+

6 Slide courtesy of Anna

Wrapper methods Stepwise Regression Forward selection Backward elimination Bidirectional elimination

Forward selection

Applied Predictive Modeling Chapter 19

Backward Selection

Filter Methods

Akaike Information Criterion Bayesian information criterion CV (Covered Earlier by Anna)

!"# = % log (*

+,- .

(/0 − 2 /0)4 ) + 6 ∗ 89:(%)

Feature Importance Measures

Courtesy of Scott Lundberg

Often model dependent

A comparison of two methods for determining variable importance.

Feature Importance Measures

1.Tree SHAP. A new proposed method 2.Saabas. An individualized heuristic feature attribution method. 3.mean(|Tree SHAP|). A global attribution method based on the average magnitude of the individualized Tree SHAP attributions. 4.Gain. The same method used above in XGBoost, and also equivalent to the Gini importance measure used in scikit-learn tree models. 5.Split count. Represents both the closely related “weight” and “cover” methods in XGBoost, but is computed using the “weight” method. 6.Permutation. The resulting drop in accuracy of the model when a single feature is randomly permuted in the test data set. Courtesy of Scott Lundberg

Controlling Procedure (FWER and FDR)

Summary

Various tools to dissect the relationships of predictor and outcome are available and the selection of them are subject to the question in mind. Feature selection is mostly subject to model selection except for filtering method. Most of the time, simple plotting (scatterplot, boxplot, pca) can save you a ton of time to figure out what is relevant/robust or dubious/misleading. Low complexity is mostly preferred when the primary goal is interpreting the contribution of predictor to the outcome. Sometimes, multiplicity should be controlled for multiple hypothesis test.

Thank You