Feature Extraction Aleix M. Martinez aleix@ece.osu.edu Continuous - - PDF document

1

Aleix M. Martinez aleix@ece.osu.edu Machine Learning & Pattern Recognition

Feature Extraction Continuous Feature Space

• Let us now look at the case where we

represent the data in a feature space of p dimensions in the real domain,

• We will later see how our results extend to

the other feature representations.

• Thus, unless otherwise noted, we will

assume

.

p

 Πx .

p

 Πx

Feature Extraction

• In our feature representation, we are

describing the features xi and xj as

• rthogonal dimensions.
• This allows us to determine the correlation

between each pair of features.

,

p

 Πx

2

Correlations

• Linear correlations (or co-relations)

translates to a linear relationship between variables.

• If xi1 and xi2 are linearly dependent, we can

write xi2=f(xi1), where f(.) is linear.

• Ex:
• If they are not 100% correlated, we have

Error function.

Linear least-squares

• Our error function is a set of homogenous

equations: with n>p.

• We can rewrite these as Xa=0.
• If rank(X)=p, there is a unique solution.
• When rank(X)>p, R2>0. Then, to minimize

R2, we need to minimize (Xa)2= aTXTXa. That is: we want to find the dimension in Rp where the data has largest variance.

• Let Q=XTX. And assume . We

can then write

• Now, let We have:

. l = Qa aT

symmetric, positive semidefinite matrix

( ).

, ,

1 p

a a A ! =

. L = QA AT

Schur decomposition

3

Eigenvalue decomposition

• Since the columns of A define an orthogonal

bases, i.e., ATA=I, we can write

• This is the well-known eigenvalue

decomposition equation.

• In the equations above

and we assume

. L = A QA

( ),

, , diag

1 p

l l ! = L .

1

³ ³ ³

p

l l !