Feature Selection Pattern Recognition: The Early Days Pattern - - PowerPoint PPT Presentation

SLIDE 1

Feature Selection

SLIDE 2

Pattern Recognition: The Early Days

SLIDE 3

Pattern Recognition: The Early Days

Only 200 papers in the world! - I wish!

SLIDE 4

Pattern Recognition: The Early Days

“Using eight very simple measurements [...] a recognition rate of 95 per cent

• n sampled and fresh material (using 50 specimens of each of the hand-printed

letters A, B and C, and a self-organizing computer program based on the above considerations).” [Rutovitz, 1966]

→

508 RUTOVITZ

• Pattern Recognition

[Part 4, This and other similar ad hoc procedures are useful in the contexts discussed by their authors but their applicability depends very much on the properties of the particular patterns involved.

0 0 03a *1 0)60 5 I

1 ft EXi a0 00 0 oo 00 6t 0 90 00 a0 j

• 1.N...N*

0u

* *a *o *o0a 1

50999a

III. N

a 03 00 a 100000 00 000 3

~~~~~

~~00a

a0

• 0a0

00 00 010 O U ~~~~~~~~~~~a0 0 0000 00990000000000

• ~~~~0

cao so @0000 I 00 C 00

00~~~~~~ ~~~~~~0

300 0a I00 0000 00

~~~~~~~~~~~0

00 @ 500 0000 009a000 000 00010000 000

1

*

0~~~~~~~~~~ C)~~~~~~~~~~~~~~00

*to tp @50t t Ot 111 IN%. *0 * , ,~ t+l , 1.t#t tF*+4t*N * X t 0 ~~~~

~~~~~~~

*

~

W~

5

* A ** t*0

I

• 3c*

0 0 1 X X*

2:

@000 3

• f

t t N0

I

b*0 a :r 00

• ll

* Z

I Xz:

~

nN

I * .E X

N:E0 @0 0 ~ ~ ~ jNi0

N~~O

3 I0 001X2

• a0

1 5 .00Nat 1, l X t 40 * z1 I

1 0-

• X~

U los 1 O0 41: * X 000 Z*. * 0N N. V* 00 X:

*:~~~*-*..~~:~* *0: *** *I

~ ~

*0N.N0 *o Du

I~

n I0. Q I 0090 @0 900

~~~~~~~~~~00060

~ ~

3 90 00300 0000 ~0003 3)

• FIG. 2. Line-printer

reconstruction

• f a portion of a digitized

image (BBC Test Card C). Each line-printer symbol corresponds to a 0-06 x 0-06 mm. square area of the original 35 mm. transparency. An arbitrary seven-level grey scale is used and is printed

• ut accord-

ing to the conventions: 0, space; 1, ; 2, -; 3, * ; 4, /; 5, $; 6, W. (Input device: Medical Research Council's FIDAC, built by National Biomedical Research Foundation, Silver Spring, Maryland. Computer: IBM 7090 at Imperial College, London.) In fact, the whole recognition process can be expressed in terms of transformations

• f one type or another. In order to recognize a pattern a machine must first carry out

a prescribed set of measurements of its features or characteristics. On the basis of these measurements the pattern must be categorized (in our present conteXt) into just

• ne of a finite number of "ideal" non-overlapping pattern classes, Fl, F2,

. .., Fm, say.

Now suppose that the pattern presented to the transducer is represented, as before, by a function c1f+ ns, where ns is the specimen noise. Then the pattern to be analysed will be g= Rob= Ro10+nR,

BBC = British Broadcasting Corporation

SLIDE 5

Let’s review...

“Supervised” learning. Computer is presented with a situation, described as a vector x. Required to recognize the situation and ‘classify’ it into one of a finite number of possible categories.

e.g. x : real valued numbers giving a person’s height, weight, body mass, age, blood sugar, etc. Task: classify yes/no for risk of heart disease. e.g. x : binary values of pixels in 8 × 8 image, so |x| = 64. Task: classify as handwritten digit from set [0...9].

SLIDE 6

Pattern Recognition: Then and Now

Image recognition still a major issue. But we’ve gone beyond 8 × 8 characters and dot-matrix printers!

Then.... Now!

SLIDE 7

Pattern Recognition: Then and Now

Predicting recurrence of cancer from gene profiles: Only a subset of features actually influence the phenotype.

SLIDE 8

Pattern Recognition: Then and Now

“Typically, the number of [features] considered is not higher than tens of thousands with sample sizes of about 100.” Saeys et al, Bioinformatics (2007) 23 (19): 2507-2517

Small sample problem! We need subsets of features for interpretability. A lab analyst needs simple biomarkers to indicate diagnosis.

SLIDE 9

Pattern Recognition: Then and Now

Face detection in images (e.g. used on Google Street View)

SLIDE 10

Pattern Recognition: Then and Now

Face detection in images (e.g. used on Google Street View) 28 × 28 pixels × 8 orientations × 7 thresholds = 43, 904 features If using a 256 × 256 image... 3, 670, 016 features! We now deal in petabytes — fewer features = FAST algorithms!

SLIDE 11

Pattern Recognition: Then and Now

Text classification.... is this news story “interesting”? “Bag-of-Words” representation: x = {0, 3, 0, 0, 1, ..., 2, 3, 0, 0, 0, 1} ←

• ne entry per word!

Easily 50,000 words! Very sparse - easy to overfit! Need accuracy, otherwise we lose visitors to our news website!

SLIDE 12

Our High-Dimensional World

The world has gone high dimensional. Biometric authentication, Pharmaceutical industries, Systems biology, Geo-spatial data, Cancer diagnosis, Handwriting recognition. etc, etc, etc... Modern domains may have many thousands/millions of features!

SLIDE 13

Feature Extraction (a.k.a. dimensionality reduction)

Original features Ω. Reduced feature space X = f(Ω, θ) , such that |X| < |Ω| Combines dimensions by some function.

SLIDE 14

Feature Extraction (a.k.a. dimensionality reduction)

Original features Ω. Reduced feature space X = f(Ω, θ) , such that |X| < |Ω| Combines dimensions by some function. Can be linear, e.g. Principal Components Analysis:

Gene 1

• riginal data space

Gene 2 Gene 3 component space PC 1 PC 2

PCA

PC 1 PC 2

SLIDE 15

Feature Extraction (a.k.a. dimensionality reduction)

Or non-linear: No linear function (rotation) exists such that it separates in 2d. Non-linear function easily finds underlying manifold.

Roweis & Saul, “Local Linear Embedding”, Science, vol.290 no.5500 (2000)

SLIDE 16

Feature Selection

Original features Ω. Reduced feature space X ⊂ Ω , such that |X| < |Ω| Selects dimensions by some function. Useful to retain meaningful features, e.g. gene selection:

SLIDE 17

Why select/extract features?

◮ To improve accuracy? ◮ Reduce computation? ◮ Reduce space? ◮ Reduce cost of future measurements? ◮ Improved data/model understanding?

SLIDE 18

Why select/extract features?

◮ To improve accuracy? ◮ Reduce computation? ◮ Reduce space? ◮ Reduce cost of future measurements? ◮ Improved data/model understanding?

Surprisingly... FS is rarely needed to improve accuracy. Overfitting well managed by modern classifiers, e.g. SVMs, Boosting, Bayesian methods.

SLIDE 19

Feature Selection: The ‘Wrapper’ Method

Input: large feature set Ω 10 Identify candidate subset S ⊆ Ω 20 While !stop criterion() Evaluate error of a classifier using S. Adapt subset S. 30 Return S.

◮ Pros: excellent performance for the chosen classifier ◮ Cons: computationally and memory-intense

SLIDE 20

Why can’t we get a bigger computer?

With M features → 2M possible feature subsets. Exhaustive enumeration feasible only for small (M ≈ 20) domains. Could use clever search (Genetic Algs, Simulated Annealing, etc). but ultimately... NP-hard problem!

SLIDE 21

What’s wrong here?

GET DATA Data set, D SELECT SOME FEATURES Using D, try many feature subsets with a classifier. Return the subset θ that has lowest error on D. LEARN A CLASSIFIER Make a new dataset D′ with only features θ. Repeat 50 times

• Split D′ into train/test sets.
• Train a classifier, and record its error on test set.

Report average testing error over 50 repeats.

SLIDE 22

What’s wrong here?

GET DATA Data set, D SELECT SOME FEATURES Using D, try many feature subsets with a classifier. Return the subset θ that has lowest error on D. LEARN A CLASSIFIER Make a new dataset D′ with only features θ. Repeat 50 times

• Split D′ into train/test sets.
• Train a classifier, and record its error on test set.

Report average testing error over 50 repeats. OVERFITTING! - We used our ‘test’ data to pick features!

SLIDE 23

Feature Selection is part of the Learning Process

!"#$%#&'() *+#&%,+-.%/0+& 1+)+,#&'() 2,#')')3 4#&# .&(5 6,'&+,'() 78

*+#&%,+-.+$+9&'()

2,#')')3 :+#,)')3-;(<+$ =+0 >+0&-.%/0+& 2+0&-:+#,)')3 ;(<+$ 2+0&-4#&# ?66

;(<+$-*'&&')3@A+,B(,C#)9+-!"#$%#&'()

5D#0+-E 5D#0+-EE

Liu & Motoda, “Feature Selection: An Ever Evolving Frontier”, Intl Workshop Feature Selection in Data Mining 2010

SLIDE 24

A better way.... (but not the only way)

GET DATA Data set, D LEARN A CLASSIFIER - WITH FEATURE SELECTION Repeat 50 times

• Split D into train/validation/test sets : Tr, V a, Te
• For each feature subset, train a classifier using Tr
• Pick subset θ with lowest error on V a
• Re-train using Tr ∪ V a ... [optional]
• Record test error (using θ) on Te.

Report average testing error over 50 repeats. That’s more like it! :-) ... But still computationally intense!

SLIDE 25

Searching Efficiently: “Forward Selection”

Start with no features. Try each feature not used so far in the classifier. Keep the one that improves training accuracy most. Repeat this greedy search until all features are used. You now have a ranking of the M features (and M classifiers) Test each of the M classifiers on a validation set. Return the feature subset corresponding to the classifier with lowest validation error.

SLIDE 26

Searching Efficiently: “Backward Elimination”

Start with ALL features. Try discarding each feature currently in the classifier. Discard the one that causes LEAST decrease in training accuracy. Repeat this until only one feature remains. You now have a ranking of the M features (and M classifiers) Test each of the M classifiers on a validation set. Return the feature subset corresponding to the classifier with lowest validation error.

SLIDE 27

The Feature Selection Search Space

With M features → 2M possible feature subsets.

0,0,1,1 0,0,0,0 1,0,0,0 0,1,0,0 0,0,1,0 0,0,0,1 1,1,0,0 1,0,1,0 0,1,1,0 1,0,0,1 0,1,0,1 1,1,1,0 1,1,0,1 1,0,1,1 0,1,1,1 1,1,1,1

Forward Selection starts at node {0, 0, 0, 0}. Backward Elimination starts at node {1, 1, 1, 1}.

SLIDE 28

Search Space : Wrappers

Evaluates M(M+1)

2

feature subsets.

SLIDE 29

Complexity of Forward/Backward Heuristic

With forward/backward, we only evaluate M(M+1)

2

subsets.

2 4 6 8 10 200 400 600 800 1000 1200

Number of Features Number of Evaluations Necesary

Exhaustive Forward/Backward

But we can do better... only M subsets (next lecture...)

SLIDE 30

Feature Selection: Wrappers

Input: large feature set Ω 10 Identify candidate subset S ⊆ Ω 20 While !stop criterion() Evaluate error of a classifier using S. Adapt subset S. 30 Return S.

◮ Pros: excellent performance for the chosen classifier ◮ Cons: computationally and memory-intense

SLIDE 31

Feature Selection: Filters

Input: large feature set Ω 10 Identify candidate subset S ⊆ Ω 20 While !stop criterion() Evaluate utility function J using S. Adapt subset S. 30 Return S.

◮ Pros: fast, provides generically useful feature set ◮ Cons: generally higher error than wrappers

SLIDE 32

Types of Filters

A filter evaluates statistics of the data Univariate filters evaluate each feature independently. Multivariate filters evaluate features in context of others.

SLIDE 33

Types of Filters

A filter evaluates statistics of the data Univariate filters evaluate each feature independently. Multivariate filters evaluate features in context of others. also... Some data is ordered. e.g. 1,2,3 Some is not, e.g. dog, cat, sheep (i.e. categorical) A filter statistic must take this into account. Today we mostly look at numerical (ordered) data.

SLIDE 34

How “useful” is a single feature? : Univariate filters

Trying to predict someone’s Biology exam grade from various possible indicators (a.k.a. features): (1)Chemistry grade, (2)History grade, (3)Biology Mock exam grade, or (4)Height ... Which one would you pick?

SLIDE 35

Pearson’s Correlation Coefficient

Feature : xk = {x(1)

k , ..., x(N) k

}T Target : y = {y(1), ..., y(N)}T r(x, y) = N

i=1(x(i) − ¯

x)(y(i) − ¯ y) N

i=1(x(i) − ¯

x)2 N

i=1(y(i) − ¯

y)2 r = +0.5 r = 0.0 r = −0.5 Both positive and negative correlation is useful!

SLIDE 36

Pearson’s Correlation Coefficient

xk = {x(1)

k , ..., x(N) k

} k = 1..M y = {y(1), ..., y(N)} The estimated ‘utility’ for feature Xk is: J(Xk) = |r(xk, y)| (i.e. absolute correlation with target) Algorithm 10. Rank features in descending order by J. 20. Evaluate predictor on M nested subsets. 30. Choose subset with lowest validation error. Features are ranked by their ‘score’ J.

SLIDE 37

Ranking with Filter Criteria

Rank features Xi, ∀i by their values of J(Xk). Retain the highest ranked features, discard the lowest ranked. k J(Xk) 35 0.846 42 0.811 10 0.810 654 0.611 22 0.443 59 0.388 ... ... 212 0.09 39 0.05 Cut-off point decided by user, e.g. |S| = 5, so S = {35, 42, 10, 654, 22}. Or by cross-validation.

SLIDE 38

Limitations...

Pearson assumes all features are INDEPENDENT ! and... only detects LINEAR correlations...

SLIDE 39

Pearson’s Correlation Coefficient

With binary y, Pearson corresponds to linear separability.

−4 −3 −2 −1 1 2 3 4 −0.2 0.2 0.4 0.6 0.8 1

Feature Value Class Label

r = 0.15256

−4 −3 −2 −1 1 2 3 4 −0.2 0.2 0.4 0.6 0.8 1

Feature Value Class Label

r = 0.86652

SLIDE 40

Pearson’s Correlation Coefficient

And....

−4 −3 −2 −1 1 2 3 4 −0.2 0.2 0.4 0.6 0.8 1

Feature Value Class Label

r = 0.99357

−4 −3 −2 −1 1 2 3 4 −0.2 0.2 0.4 0.6 0.8 1

Feature Value Class Label

r = 0.10948

Beware multi-class problems! ... Why?

SLIDE 41

Fisher Score

Something a little more sensible for classification problems: J(Xk) = (µ(y+) − µ(y−))2 σ(y+)2 + σ(y−)2 Maximum between class variance (difference of means). Minimum within class variance (sum of variances).

SLIDE 42

Mutual Information

What if we have categorical variables? X is relevant to Y if they are dependent, i.e. p(xy) = p(x)p(y) So let’s measure the KL-divergence between these distributions: J(Xk) = I(Xk; Y ) =

• x∈Xk
• y∈Y

p(xy) log p(xy) p(x)p(y) Again, RANK features by their score J. We will see more of this in the next lecture.

SLIDE 43

There are LOTS of ranking criteria...

Many produce very similar rankings....

W.Duch, “Filter Methods”, ch2, Feature Extraction: Foundations and Applications

SLIDE 44

There are LOTS of ranking criteria...

Pearson, Fisher, Mutual Info, Jeffreys-Matsusita, Gini Index, AUC, F-measure, Kolmogorov distance, Chi-squared, CFS, Alpha-divergence, Symmetrical Uncertainty,.... etc, etc

How do I pick!? Unfortunately, quite complex.... depends on:

• type of variables/targets (continuous, discrete, categorical).
• class distribution
• degree of nonlinearity/feature interaction
• amount of available data

SLIDE 45

There are LOTS of ranking criteria...

Pearson, Fisher, Mutual Info, Jeffreys-Matsusita, Gini Index, AUC, F-measure, Kolmogorov distance, Chi-squared, CFS, Alpha-divergence, Symmetrical Uncertainty,.... etc, etc

How do I pick!? Unfortunately, quite complex.... depends on:

• type of variables/targets (continuous, discrete, categorical).
• class distribution
• degree of nonlinearity/feature interaction
• amount of available data

And ultimately... there’s no magic bullet...

“There are no relevancy definitions independent of the learner

• r error measure that solve the feature selection problem”

Tsamardinos et al, “Towards Principled Feature Selection: Relevancy, Filters and Wrappers”, AISTATS 2003

SLIDE 46

Search Space : Wrappers

Evaluates M(M+1)

2

feature subsets.

SLIDE 47

Search Space : Filter Ranking Methods

Ranking provided by criterion, hence no need to search.

SLIDE 48

Things to Remember

In general, features work in combination... It doesn’t look like either the X or Y axis here is very useful. But if we have both together.... perfect separation...

I.Guyon et al, “An Introduction to Variable and Feature Selection”, JMLR 2004.

SLIDE 49

Things to Remember

Features can be individually completely irrelevant, and only useful when combined with others

I.Guyon et al, “An Introduction to Variable and Feature Selection”, JMLR 2004.

SLIDE 50

Key Point

The relevance of a feature can only be fairly assessed in the context of other features. Independent ranking criteria are FAST, but naive, being uni-variate. Not all filter methods are naive. Some use context. These are multi-variate filters.

SLIDE 51

RELIEF (Kira & Rendell, 1992)

Classic filter method, very popular. If Dhit ≫ Dmiss.... BAD feature!

SLIDE 52

RELIEF algorithm

10. Set all weights w(i) := 0 20. For t := 1 to T 30. Randomly select an instance 40. Find nearest hit H and nearest miss M 50. For each feature i, 60. w(i) ← w(i) + Dmiss − Dhit 70. End 80. End Dmiss =

(xi−x(M)

i

)2 max(xi)−min(xi)

Dhit =

(xi−x(H)

i

)2 max(xi)−min(xi)

Stochastic! Can be made deterministic by T = |D|. RELIEF is computationally more expensive than Pearson.

SLIDE 53

Pearson versus Relief

Breast Cancer data : 20 bootstraps, 1-NN classifier. Data rescaled to mean zero, variance one.

5 10 15 20 25 30 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 Num features OOB error Pearson Relief

Pearson statistically insignificant after ∼ 26 features. Notice Pearson beats Relief in early stages. Why?

SLIDE 54

Pearson versus Relief - The Effect of Feature Scaling.

Scaling of features affects the outcome of RELIEF!

5 10 15 20 25 30 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 Num features OOB error Pearson Relief

5 10 15 20 25 30 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2

Num features OOB error

Pearson Relief

Scaled (left) versus unscaled data (right). NOTE: Pearson is not affected by scaling (correlation is scale-invariant) ...but the subsequent K-NN is affected. Relief IS affected by scaling, and also the K-NN, hence much larger variance.

SLIDE 55

The Pattern Recognition Pipeline

Coupling at all stages. Data→FS : Relief does not cope well with unscaled data. FS→Classifier : If classifier cannot make use of features, no hope. Data→Classifier : Rescaling affects many classifiers. Even coupling at error stage - what about class imbalance?

SLIDE 56

Modified RELIEF

Use ratio instead of difference. Sum over ALL patterns. 50. For each feature i, 60. w(i) ← w(i) +

x∈D( |x−x(M)| |x−x(H)| )

70. End Avoids scaling issue, but does behave differently than the original. Also loses strong theory links to margin maximisation.

SLIDE 57

Conclusion

Feature Selection is an important part of the data mining process. Wrappers are slow. Univariate filters are fast. Multivariate filters are fast (sometimes), and use context. Which ones you need depends on the problem.

SLIDE 58

Conclusion

Feature Selection is an important part of the data mining process. Wrappers are slow. Univariate filters are fast. Multivariate filters are fast (sometimes), and use context. Which ones you need depends on the problem. Now... lunch... then lab session. We will be marking the SVM lab for the first hour ONLY.