SLIDE 1
Outline
} Dimensionality reduction } Filter univariate methods } Multi-variate filter & wrapper methods } Search strategies } Embedded approach
2
Feature Selection CE-725: Statistical Pattern Recognition Sharif - - PowerPoint PPT Presentation
Feature Selection CE-725: Statistical Pattern Recognition Sharif - - PowerPoint PPT Presentation
Feature Selection CE-725: Statistical Pattern Recognition Sharif University of Technology Soleymani Fall 2018 Some slides are based on Isabelle Guyons slides : Introduction to feature selection, 2007. Outline } Dimensionality
SLIDE 2
SLIDE 3
Dimensionality reduction: Feature selection vs. feature extraction
} Feature selection
} Select a subset of a given feature set
} Feature extraction (e.g., PCA, LDA)
} A linear or non-linear transform on the original feature space
3
𝑦" ⋮ 𝑦$ → 𝑦&' ⋮ 𝑦&() Feature Selection (𝑒+ < 𝑒) 𝑦" ⋮ 𝑦$ → 𝑧" ⋮ 𝑧$) = 𝑔 𝑦" ⋮ 𝑦$ Feature Extraction
SLIDE 4
Feature selection
} Data may contain many irrelevant and redundant variables and
- ften comparably few training examples
} Consider supervised learning problems where the number of
features 𝑒 is very large (perhaps 𝑒 ≫ 𝑜)
} E.g., datasets with tens or hundreds of thousands of features and
(much) smaller number of data samples (text or document processing,
gene expression array analysis)
4
⋮ ⋮ … ⋮ 𝒚(") 𝒚(5) 1 2 𝑒 𝑗" 3 4 𝑒 − 1 𝑗< 𝑗$) …
SLIDE 5
Why feature selection?
5
} FS is a way to find more accurate, faster, and easier to
understand classifiers.
} Performance: enhancing generalization ability
} alleviating the effect of the curse of dimensionality } the higher the ratio of the no. of training patterns 𝑂 to the number of free
classifier parameters, the better the generalization of the learned classifier
} Efficiency: speeding up the learning process } Interpretability: resulting a model that is easier to understand by human
Feature Selection 𝒀 = 𝑦"
(")
⋯ 𝑦$
(")
⋮ ⋱ ⋮ 𝑦"
(5)
⋯ 𝑦$
(5)
, 𝑍 = 𝑧(") ⋮ 𝑧(5) 𝑗", 𝑗<, … , 𝑗$) The selected features Supervised feature selection: Given a labeled set of data points, select a subset of features for data representation
SLIDE 6
Noise (or irrelevant) features
6
} Eliminating
irrelevant features can decrease the classification error on test data
SVM Decision Boundary Noise feature 𝑦" 𝑦" 𝑦< SVM Decision Boundary
SLIDE 7
Drug Screening
[Weston et al, Bioinformatics, 2002]
Number of features
𝑂 = 1909 compounds 𝑒 = 139,351 binary features three-dimensional properties of the molecule.
7
SLIDE 8
Text Filtering
[Bekkerman et al, JMLR, 2003] Top 3 words of some categories:
} Alt.atheism: atheism, atheists, morality } Comp.graphics: image, jpeg, graphics } Sci.space: space, nasa, orbit } Soc.religion.christian: god, church, sin } Talk.politics.mideast: israel, armenian, turkish
Reuters: 21578 news wire, 114 semantic categories. 20 newsgroups: 19997 articles, 20 categories. WebKB: 8282 web pages, 7 categories. Bag-of-words: >100000 features.
8
SLIDE 9
Face Male/Female Classification
Relief: Simba:
100 500 1000
[Navot, Bachrach, and Tishby, ICML, 2004]
𝑂 = 1000 training images 𝑒 = 60×85 = 5100 features
9
SLIDE 10
Some definitions
10
} One categorization of feature selection methods:
} Univariate method (variable ranking): considers one variable
(feature) at a time.
} Multivariate method: considers subsets of features together.
} Another categorization:
} Filter method: ranks features or feature subsets independent of
the classifier as a preprocessing step.
} Wrapper method: uses a classifier to evaluate the score of
features or feature subsets.
} Embedded method: Feature selection is done during the training
- f a classifier
} E.g., Adding a regularization term
𝒙 " in the cost function of linear classifiers
SLIDE 11
Filter: univariate
11
} Univariate filter method
} Score each feature 𝑙 based on the 𝑙-th column of the data
matrix and the label vector
} Relevance
- f
the feature to predict labels: Can the feature discriminate the patterns of different classes?
} Rank features according to their score values and select the
- nes with the highest scores.
} How do you decide how many features k to choose? e.g., using cross-
validation to select among the possible values of k } Advantage: computational and statistical scalability
SLIDE 12
Pearson Correlation Criteria
12
𝑆 𝑙 = 𝑑𝑝𝑤(𝑌R, 𝑍) 𝑤𝑏𝑠(𝑌R)
- 𝑤𝑏𝑠(𝑍)
- ≈
∑ 𝑦R
(&) − 𝑦R
𝑧(&) − 𝑧 X
5 &Y"
∑ 𝑦R
& − 𝑦R < 5 &Y"
∑ 𝑧 & − 𝑧 X <
5 &Y"
- } can only detect linear dependencies between feature and target.
𝑦" 𝑦< 𝑆(1) ≫ 𝑆(2)
SLIDE 13
Single Variable Classifier ROC Curve
1- Specificity=FPR Sensitivity=TPR
13
s- s+
1
x Y=1 Y=-1 µ- µ+ density Y=1 Y=-1
2
x
1
x
2
x density µ- µ+ s- s+
Area Under Curve (AUC)
𝑦< 𝑦"
SLIDE 14
Univariate Mutual Information
14
} Independence:
𝑄(𝑌, 𝑍) = 𝑄(𝑌)𝑄(𝑍)
} Mutual information as a measure of dependence:
𝑁𝐽 𝑌, 𝑍 = 𝐹^,_ log 𝑄(𝑌, 𝑍) 𝑄 𝑌 𝑄(𝑍)
} Score of 𝑌R based on MI with 𝑍:
} 𝐽 𝑙 = 𝑁𝐽 𝑌R, 𝑍
SLIDE 15
Filter – univariate: Disadvantage
15
} Univariate methods may fail:
} a feature may be important in combination with other features.
} Redundant features:
} They can select a group of dependent variables that carry
similar information about the output, i.e. it is sufficient to use
- nly one (or a few) of these variables.
SLIDE 16
Univariate methods: Failure
16
} Samples on which univariate feature analysis and scoring
fails:
[Guyon-Elisseeff, JMLR 2004; Springer 2006]
SLIDE 17
Redundant features
17
What is the relation between redundancy and correlation: Are highly correlated features necessarily redundant? What about completely correlated ones?
SLIDE 18
Multi-variate feature selection
18
} Search in the space of all possible combinations of features.
} all feature subsets: For 𝑒 features, 2$ possible subsets. } high computational and statistical complexity.
SLIDE 19
Search space for feature selection (𝑒 = 4)
19
0,0,0,0 1,0,0,0 0,1,0,0 0,0,1,0 0,0,0,1 1,1,0,0 1,0,1,0 0,1,1,0 1,0,0,1 0,1,0,1 0,0,1,1 1,1,1,0 1,1,0,1 1,0,1,1 0,1,1,1 1,1,1,1 [Kohavi-John,1997]
SLIDE 20
Multivariate methods: General procedure
Subset Generation: select a candidate feature subset for evaluation Subset Evaluation: compute the score (relevancy value) of the subset Stopping criterion: when stopping the search in the space of feature subsets Validation: verify that the selected subset is valid
20
Subset generation Subset evaluation Stopping criterion Validation Original feature set Subset No Yes
SLIDE 21
Stopping criteria
} Predefined number of features is selected } Predefined number of iterations is reached } Addition (or deletion) of any feature does not result in a
better subset
} An optimal subset (according to the evaluation criterion)
is obtained.
21
SLIDE 22
Filter and wrapper methods
22
} Wrappers use the classifier performance to evaluate the
feature subset utilized in the classifier.
} Training 2$ classifiers is infeasible for large 𝑒. } Most wrapper algorithms use a heuristic search.
} Filters use an evaluation function that is cheaper to compute
than the performance of the classifier
} e.g. correlation coefficient
SLIDE 23
Filters vs. wrappers
rank subsets of useful features
Filter Feature subset Classifier Original feature set Wrapper Multiple feature subsets Classifier
take classifier into account to rank feature subsets (e.g., using cross validation to evaluate features)
23
Original feature set
SLIDE 24
Wrapper methods: Performance assessment
24
} For each feature subset, train classifier on training data
and assess its performance using evaluation techniques like cross-validation
SLIDE 25
Filter methods: Evaluation criteria
} Distance (Eulidean distance)
} Class separability: Features supporting instances of the same class to be
closer in terms of distance than those from different classes
} Information (Information Gain)
} Select S1 if IG(S1,Y)>IG(S2,Y)
} Dependency (correlation coefficient)
} good feature subsets contain features highly correlated with the class,
yet uncorrelated with each other
} Consistency (min-features bias)
} Selects features that guarantee no inconsistency in data
} inconsistent instances have the same feature vector but different class labels
} Prefers smaller subset with consistency (min-feature) 25
f1 f2 class instance 1 a b c1 instance 2 a b c2
inconsistent
SLIDE 26
How to search the space of feature subsets?
26
} NP-hard problem. } Complete search is possible only for small number of
features.
} Greedy search is often used (forward selection or backward
elimination).
SLIDE 27
Subset selection or generation
} Search direction
} Forward } Backward } Random
} Search strategies
} Exhaustive - Complete
} Branch & Bound } Best first search
} Heuristic or greedy
} Sequential forward selection } Sequential backward elimination } Plus-l Minus-r Selection } Bidirectional Search } Sequential floating Selection
} Non-deterministic
} Simulated annealing } Genetic algorithm
27
SLIDE 28
Search strategies
} Complete: Examine all combinations of feature subset
} Optimal subset is achievable } Too expensive if 𝑒 is large
} Heuristic: Selection is directed under certain guidelines
} Incremental generation of subsets } Smaller search space and thus faster search } May miss out feature sets of high importance
} Non-deterministic or random: No predefined way to
select feature candidate (i.e., probabilistic approach)
} Optimal subset depends on the number of trials } Need more user-defined parameters
28
SLIDE 29
Filters vs. Wrappers
} Filters
þ Fast execution: evaluation function computation is faster than a classifier training þ Generality: Evaluate intrinsic properties of the data, rather than their interactions with a
particular classifier (“good” for a larger family of classifiers)
ý T
endency to select large subsets: Their objective functions are generally monotonic (so tending to select the full feature set).
§
a cutoff is required on the number of features
} Wrappers
ý Slow execution: must train a classifier for each feature subset (or several trainings if cross-
validation is used)
ý Lack of generality: the solution lacks generality since it is tied to the bias of the classifier
used in the evaluation function.
þ Ability to generalize: Since they typically use cross-validation measures to evaluate
classification accuracy, they have a mechanism to avoid overfitting.
þ Accuracy: Generally achieve better accuracy than filters since they find a proper feature set
for the intended classifier.
29
SLIDE 30
Examples of embedded methods
30
} Decision trees have a built-in mechanism to perform
variable selection
} Nested subset methods
} (input) node pruning techniques in neural networks are feature
selection algorithms.
} Direct objective optimization
} Combines goodness-of-fit and the number of variables in
the objective function
SLIDE 31
Direct objective optimization: example
31
} Performs feature selection as part of learning procedure:
𝑔 𝒚; 𝒙 = 𝒙d𝒚 + 𝑥g 𝐾 𝒙 = 1 𝑂 i 𝑚 𝑔 𝒚 k ; 𝒙 , 𝑧(k)
5 kY"
+ 𝜇 𝒙 m
} In the limit as p → 0, the
𝒙 m is just the number of non-zero weights, i.e., the number of selected features.
} For some application, the L1-norm minimization suffices to
drive enough weights to zero.
} Lasso: 𝑥 " as regularization term
SLIDE 32
Lp-Norms
32
[wikipedia] 𝒚 = 𝑦", … , 𝑦$ 𝒚+ = (𝑦"
+, … , 𝑦$ + )
𝐸 𝒚, 𝒚+ = i 𝑦& − 𝑦&
+ m $ &Y" "/m
Minkowski distance:
SLIDE 33
Example
33
} L1 regularization: the number of zero weights increases
and thus shows feature selection property
[Bishop]
SLIDE 34
References
34