[PPT] - http://cs246.stanford.edu High dimensional == many features Find PowerPoint Presentation

SLIDE 1

CS246: Mining Massive Datasets Jure Leskovec, Stanford University

http://cs246.stanford.edu

SLIDE 2

 High‐dimensional == many features  Find concepts/topics/genres:

Documents:
Features: Thousands of words, millions of word pairs
Surveys – Netflix: 480k users x 177k movies

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 2

SLIDE 3

 Compress / reduce dimensionality:

106 rows; 103 columns; no updates
random access to any cell(s); small error: OK

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 3

SLIDE 4

 Assumption: Data lies on or near a low

d‐dimensional subspace

 Axes of this subspace are effective

representation of the data

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 4

SLIDE 5

Why reduce dimensions?

 Discover hidden correlations/topics

Words that occur commonly together

 Remove redundant and noisy features

Not all words are useful

 Interpretation and visualization  Easier storage and processing of the data

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 5

SLIDE 6

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 6

A[m x n] = U[m x r]  r x r] (V[n x r])T

 A: Input data matrix

m x n matrix (e.g., m documents, n terms)

 U: Left singular vectors

m x r matrix (m documents, r concepts)

 : Singular values

r x r diagonal matrix (strength of each ‘concept’)

(r : rank of the matrix A)

 V: Right singular vectors

n x r matrix (n terms, r concepts)

SLIDE 7

7

A

m n



m n

U VT



1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets

SLIDE 8

8

A

m n



+

1u1v1 2u2v2

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets

σi … scalar ui … vector vi … vector

SLIDE 9

It is always possible to decompose a real matrix A into A = U  VT , where

 U, , V: unique  U, V: column orthonormal:

UT U = I; VT V = I (I: identity matrix)
(Cols. are orthogonal unit vectors)

 : diagonal

Entries (singular values) are positive,

and sorted in decreasing order (σ1  σ2  ...  0)

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 9

SLIDE 10

 A = U  VT ‐ example:

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 10

1 1 1 0 0 2 2 2 0 0 1 1 1 0 0 5 5 5 0 0 0 0 0 2 2 0 0 0 3 3 0 0 0 1 1

0.18 0 0.36 0 0.18 0 0.90 0 0.53 0.80 0.27

= SciFi Romnce

9.64 0 5.29

x

0.58 0.58 0.58 0 0.71 0.71

x Matrix Alien Serenity Casablanca Amelie

SLIDE 11

 A = U  VT ‐ example:

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 11

1 1 1 2 2 2 1 1 1 5 5 5 0 0 2 2 0 0 3 3 0 0 1 1

0.18 0 0.36 0 0.18 0 0.90 0 0.53 0.80 0.27

=

9.64 0 5.29

x

0.58 0.58 0.58 0 0.71 0.71

x SciFi‐concept Romance‐concept SciFi Romnce Matrix Alien Serenity Casablanca Amelie

SLIDE 12

 A = U  VT ‐ example:

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 12

1 1 1 2 2 2 1 1 1 5 5 5 0 0 2 2 0 0 3 3 0 0 1 1

0.18 0 0.36 0 0.18 0 0.90 0 0.53 0.80 0.27

=

9.64 0 5.29

x

0.58 0.58 0.58 0 0.71 0.71

x SciFi‐concept Romance‐concept

U is “user‐to‐concept” similarity matrix

SciFi Romnce Matrix Alien Serenity Casablanca Amelie

SLIDE 13

 A = U  VT ‐ example:

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 13

1 1 1 2 2 2 1 1 1 5 5 5 0 0 2 2 0 0 3 3 0 0 1 1

0.18 0 0.36 0 0.18 0 0.90 0 0.53 0.80 0.27

=

9.64 0 5.29

x

0.58 0.58 0.58 0 0.71 0.71

x ‘strength’ of SciFi‐concept SciFi Romnce Matrix Alien Serenity Casablanca Amelie

SLIDE 14

 A = U  VT ‐ example:

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 14

1 1 1 2 2 2 1 1 1 5 5 5 0 0 2 2 0 0 3 3 0 0 1 1

0.18 0 0.36 0 0.18 0 0.90 0 0.53 0.80 0.27

=

9.64 0 5.29

x

0.58 0.58 0.58 0 0.71 0.71

x

V is “movie‐to‐concept” similarity matrix

SciFi‐concept SciFi Romnce Matrix Alien Serenity Casablanca Amelie

SLIDE 15

 A = U  VT ‐ example:

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 15

1 1 1 2 2 2 1 1 1 5 5 5 0 0 2 2 0 0 3 3 0 0 1 1

0.18 0 0.36 0 0.18 0 0.90 0 0.53 0.80 0.27

=

9.64 0 5.29

x

0.58 0.58 0.58 0 0.71 0.71

x SciFi‐concept SciFi Romnce Matrix Alien Serenity Casablanca Amelie

V is “movie‐to‐concept” similarity matrix

SLIDE 16

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 16

‘movies’, ‘users’ and ‘concepts’:

 U: user‐to‐concept similarity matrix  V: movie‐to‐concept sim. matrix  : its diagonal elements:

‘strength’ of each concept

SLIDE 17

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 17

SVD gives best axis to project on:

 ‘best’ = min sum

f squares of

projection errors

 minimum

reconstruction error

v1 first singular vector Movie 1 rating Movie 2 rating

SLIDE 18

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 18

 A = U  VT ‐ example:

1 1 1 2 2 2 1 1 1 5 5 5 0 0 2 2 0 0 3 3 0 0 1 1

0.18 0 0.36 0 0.18 0 0.90 0 0.53 0.80 0.27 9.64 0 5.29

x

0.58 0.58 0.58 0 0.71 0.71

x v1

=

SLIDE 19

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 19

 A = U  VT ‐ example:

1 1 1 2 2 2 1 1 1 5 5 5 0 0 2 2 0 0 3 3 0 0 1 1

0.18 0 0.36 0 0.18 0 0.90 0 0.53 0.80 0.27 9.64 0 5.29

x

0.58 0.58 0.58 0 0.71 0.71

x variance (‘spread’)

n the v1 axis

=

SLIDE 20

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 20

 A = U  VT ‐ example:

U  Gives the coordinates of the

points in the projection axis

1 1 1 2 2 2 1 1 1 5 5 5 0 0 2 2 0 0 3 3 0 0 1 1

0.18 0 0.36 0 0.18 0 0.90 0 0.53 0.80 0.27 9.64 0 5.29

x

0.58 0.58 0.58 0 0.71 0.71

x

=

SLIDE 21

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 21

More details

 Q: How exactly is dim. reduction done?

1 1 1 2 2 2 1 1 1 5 5 5 0 0 2 2 0 0 3 3 0 0 1 1

0.18 0 0.36 0 0.18 0 0.90 0 0.53 0.80 0.27 9.64 0 5.29

SLIDE 25

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 25

More details

 Q: How exactly is dim. reduction done?  A: Set the smallest singular values to zero:

1 1 1 2 2 2 1 1 1 5 5 5 0 0 2 2 0 0 3 3 0 0 1 1

0.18 0.36 0.18 0.90

9.64

x

0.58 0.58 0.58 0

x A=

~

SLIDE 26

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 26

More details

 Q: How exactly is dim. reduction done?  A: Set the smallest singular values to zero

1 1 1 2 2 2 1 1 1 5 5 5 0 0 2 2 0 0 3 3 0 0 1 1

~

1 1 1 2 2 2 1 1 1 5 5 5 0 0 0 0 0 0

A= B=

Frobenius norm:

ǁMǁF = Σij Mij

2

ǁA-BǁF = Σij (Aij-Bij)2

is “small”

SLIDE 27

 Theorem: Let A = U  VT

(σ1σ2…, rank(A)=r)

then B = U S VT

S = diagonal nxn matrix where si=σi (i=1…k) else si=0

is a best rank‐k approximation to A:

B is solution to minB ǁA-BǁF where rank(B)=k

 We will need 2 facts:

where M = P Q R is SVD of M
U  VT ‐ U S VT = U ( ‐ S) VT

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 27

Σ

SLIDE 28

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 28

 We will need 2 facts:

where M = P Q R is SVD of M
U  VT ‐ U S VT = U ( ‐ S) VT

We apply:

- P column orthonormal
- R row orthonormal
- Q is diagonal

SLIDE 29

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 29

 A = U  VT , B = U S VT (σ1σ2…  0, rank(A)=r)

S = diagonal nxn matrix where si=σi (i=1…k) else si=0

then B is solution to minB ǁA-BǁF , rank(B)=k

 Why?  We want to choose si to minimize

we set si=σi (i=1…k) else si=0

  

    

   

r k i i r k i i k i i i s

s

i

1 2 1 2 1 2

) ( min   



 

     

r i i i s F F k B rank B

s S B A

i

1 2 ) ( ,

) ( min min min 

U  VT - U S VT = U ( - S) VT

SLIDE 30

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 30

Equivalent: ‘spectral decomposition’ of the matrix:

1 1 1 2 2 2 1 1 1 5 5 5 0 0 2 2 0 0 3 3 0 0 1 1

= x x u1 u2 σ1 σ2 v1 v2

SLIDE 31

Equivalent: ‘spectral decomposition’ of the matrix

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 31

1 1 1 2 2 2 1 1 1 5 5 5 0 0 2 2 0 0 3 3 0 0 1 1

=

u1 σ1 vT

1

u2 σ2 vT

2

+ +... n m

n x 1 1 x m

k terms Assume: σ1  σ2  σ3  ...  0

Why is setting small σs the thing to do? Vectors ui and vi are unit length, so σi scales them. So, zeroing small σs introduces less error.

SLIDE 32

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 32

Q: How many σs to keep? A: Rule‐of‐a thumb: keep 80‐90% of ‘energy’ (=σi

2)

1 1 1 2 2 2 1 1 1 5 5 5 0 0 2 2 0 0 3 3 0 0 1 1

= u1 σ1 vT

1

u2 σ2 vT

2

+ +... n m assume: σ1  σ2  σ3  ...

SLIDE 33

 To compute SVD:

O(nm2) or O(n2m) (whichever is less)

 But:

Less work, if we just want singular values
or if we want first k singular vectors
or if the matrix is sparse

 Implemented in linear algebra packages like

LINPACK, Matlab, SPlus, Mathematica ...

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 33

SLIDE 34

 SVD: A= U  VT: unique

U: user‐to‐concept similarities
V: movie‐to‐concept similarities
 : strength of each concept

 Dimensionality reduction:

keep the few largest singular values

(80‐90% of ‘energy’)

SVD: picks up linear correlations

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 34

SLIDE 35

 SVD gives us:

A = U  VT

 Eigen‐decomposition:

A = X  XT
A is symmetric
U, V, X are orthonormal (UTU=I),
 are diagonal

 What is:

AAT= U VT(U VT)T = U VT(VTUT) = UT UT
ATA = V T UT (U VT) = V T VT

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 35

X  XT So, λi = σi

2

X  XT

Shows how to compute SVD using eigenvalue decomposition!

SLIDE 36

 A AT = U 2 UT  ATA = V 2 VT  (ATA) k = V 2k VT

E.g.: (ATA)2 = V 2 VT V 2 VT = V 4 VT

 (ATA) k ~ v1 σ1

2k v1 T

for k>>1

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 36

SLIDE 37

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 37

Q: Find users that like ‘Matrix’ and ‘Alien’

1 1 1 0 0 2 2 2 0 0 1 1 1 0 0 5 5 5 0 0 0 0 0 2 2 0 0 0 3 3 0 0 0 1 1

0.18 0 0.36 0 0.18 0 0.90 0 0.53 0.80 0.27

= SciFi Romnce

9.64 0 5.29

x

0.58 0.58 0.58 0 0.71 0.71

x Matrix Alien Serenity Casablanca Amelie

SLIDE 38

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 38

Q: Find users that like ‘Matrix’ and ‘Alien’ A: Map query into a ‘concept space’ – how?

1 1 1 0 0 2 2 2 0 0 1 1 1 0 0 5 5 5 0 0 0 0 0 2 2 0 0 0 3 3 0 0 0 1 1

0.18 0 0.36 0 0.18 0 0.90 0 0.53 0.80 0.27

= SciFi Romnce

9.64 0 5.29

x

0.58 0.58 0.58 0 0.71 0.71

x Matrix Alien Serenity Casablanca Amelie

SLIDE 39

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 39

Q: Find users that like ‘Matrix’ A: map query vectors into ‘concept space’ – how?

5 0

q= Matrix Alien v1 q v2 Matrix Alien Serenity Casablanca Amelie Project into concept space: Inner product with each ‘concept’ vector vi

SLIDE 40

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 40

Q: Find users that like ‘Matrix’ A: map the vector into ‘concept space’ – how?

v1 q q*v1

5 0

q= Matrix Alien Serenity Casablanca Amelie v2 Matrix Alien Project into concept space: Inner product with each ‘concept’ vector vi

SLIDE 41

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 41

Compactly, we have: qconcept = q V E.g.:

0.58 0 0.58 0 0.58 0 0.71 0.71

movie‐to‐concept similarities =

2.9

SciFi‐concept

5 0

q= Matrix Alien Serenity Casablanca Amelie

SLIDE 42

How would the user d that rated (‘Alien’, ‘Serenity’) be handled? dconcept = d V E.g.:

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 42

0.58 0 0.58 0 0.58 0 0.71 0.71

movie‐to‐concept similarities =

5.22 0

SciFi‐concept

0 4 5

d= Matrix Alien Serenity Casablanca Amelie

SLIDE 43

Observation: User d that rated (‘Alien’, ‘Serenity’) will be similar to query “user” q that rated (‘Matrix’), although d did not rate ‘Matrix’!

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 43

0 4 5

d=

1.16 0

SciFi‐concept

5 0

0 .5 8

q= Matrix Alien Serenity Casablanca Amelie Similarity = 0 Similarity ≠ 0

SLIDE 44

+ Optimal low‐rank approximation:

in Frobenius norm
Interpretability problem:
A singular vector specifies a linear

combination of all input columns or rows

Lack of sparsity:
Singular vectors are dense!

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 44

=

U  VT

SLIDE 45

 Goal:

Make ǁA‐CURǁF small

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 45

A C U R

Frobenius norm:

ǁXǁF = Σij Xij

2

SLIDE 46

 Goal:

Make ǁA‐CURǁF small

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 46

Pseudo‐inverse of the intersection of C and R

A C U R

Frobenius norm:

ǁXǁF = Σij Xij

2

SLIDE 47

 Let:

Ak be the “best” rank k approximation to A (that is, Ak is SVD of A) Theorem [Drineas et al.] CUR in O(m∙n) time achieves

ǁA‐CURǁF  ǁA‐AkǁF + ǁAǁF

with probability at least 1‐, by picking

O(k log(1/)/2) columns, and
O(k2 log3(1/)/6) rows

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 47

In practice: Pick 4k cols/rows

SLIDE 48

 Sample columns (similarly for rows):

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 48

SLIDE 49

 Let W be the “intersection” of sampled

columns C and rows R

Let SVD of W = X Z YT

 Then: U = W+ = Y Z+ XT

+: reciprocals of non‐zero

singular values: +

ii  ii

W+ is the “pseudoinverse”

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 49

A C R U = W+ W



Why pseudoinverse works? W = X Z Y then W-1 = X-1 Z-1 Y-1 Due to orthonomality X-1=XT and Y-1=YT Since Z is diagonal Z-1 = 1/Zii Thus, if W is nonsingular, pseudoinverse is the true inverse

SLIDE 50

+ Easy interpretation

Since the basis vectors are actual

columns and rows

+ Sparse basis

Since the basis vectors are actual

columns and rows

Duplicate columns and rows
Columns of large norms will be sampled many

times

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 50

Singular vector Actual column

SLIDE 51

 If we want to get rid of the duplicates:

Throw them away
Scale (multiply) the columns/rows by the

square root of the number of duplicates

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 51

A Cd Rd Cs Rs

Construct a small U

SLIDE 52

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 52

SVD: A = U  VT

Huge but sparse Big and dense

CUR: A = C U R

Huge but sparse Big but sparse dense but small sparse and small

SLIDE 53

 DBLP bibliographic data

Author‐to‐conference big sparse matrix
Aij: Number of papers published by author i at

conference j

428K authors (rows), 3659 conferences (columns)
Very sparse

 Want to reduce dimensionality

How much time does it take?
What is the reconstruction error?
How much space do we need?

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 53

SLIDE 54

 Accuracy:

1 – relative sum squared errors

 Space ratio:

#output matrix entries / #input matrix entries

 CPU time

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 54

0.2 0.4 0.6 0.8 1 10

1

10

2

space ratio accuracy

SVD CUR CMD 0.2 0.4 0.6 0.8 1 10

1

10

2

10

3

time(sec) accuracy

SVD CUR CMD

SVD CUR CUR no duplicates SVD CUR CUR no dup

Sun,Faloutsos: Less is More: Compact Matrix Decomposition for Large Sparse Graphs, SDM ’07.

SLIDE 55

 SVD is limited to linear projections:

Lower‐dimensional linear projection

that preserves Euclidean distances

 Non‐linear methods: Isomap

Data lies on a nonlinear low‐dim curve aka manifold
Use the distance as measured along the manifold
How?
Build adjacency graph
Geodesic distance is

graph distance

SVD/PCA the graph

pairwise distance matrix

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 55

SLIDE 56

 Drineas et al., Fast Monte Carlo Algorithms for Matrices III:

Computing a Compressed Approximate Matrix Decomposition, SIAM Journal on Computing, 2006.

 J. Sun, Y. Xie, H. Zhang, C. Faloutsos: Less is More:

Compact Matrix Decomposition for Large Sparse Graphs, SDM 2007

 Intra‐ and interpopulation genotype reconstruction from

tagging SNPs, P. Paschou, M. W. Mahoney, A. Javed, J. R. Kidd, A. J. Pakstis, S. Gu, K. K. Kidd, and P. Drineas, Genome Research, 17(1), 96‐107 (2007)

 Tensor‐CUR Decompositions For Tensor‐Based Data, M. W.

Mahoney, M. Maggioni, and P. Drineas, Proc. 12‐th Annual SIGKDD, 327‐336 (2006)

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 56

SLIDE 57

1/25/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 57