Multidimensional Scaling Applied Multivariate Statistics Spring 2012 - PowerPoint PPT Presentation

Multidimensional Scaling Applied Multivariate Statistics – Spring 2012

Outline  Fundamental Idea  Classical Multidimensional Scaling  Non-metric Multidimensional Scaling Appl. Multivariate Statistics - Spring 2012 2

How to represent in two dimensions? Basic Idea Appl. Multivariate Statistics - Spring 2012 3

Idea 1: Projection Appl. Multivariate Statistics - Spring 2012 4

Idea 2: Squeeze on table Close points stay close Appl. Multivariate Statistics - Spring 2012 5

Which idea is better? Appl. Multivariate Statistics - Spring 2012 6

Idea of MDS  Represent high-dimensional point cloud in few (usually 2) dimensions keeping distances between points similar  Classical/Metric MDS: Use a clever projection R: cmdscale  Non-metric MDS: Squeeze data on table R: isoMDS Appl. Multivariate Statistics - Spring 2012 7

Classical MDS  Problem: Given euclidean distances among points, recover the position of the points!  Example: Road distance between 21 European cities (almost euclidean, but not quite) … Appl. Multivariate Statistics - Spring 2012 8

Classical MDS  First try: Appl. Multivariate Statistics - Spring 2012 9

Can identify points up to - shift Classical MDS - rotation - reflection  Flip axes: Appl. Multivariate Statistics - Spring 2012 10

Classical MDS  Another example: Airpollution in US cities  Range of manu and popul is much bigger than range of wind  Need to standardize to give every variable equal weight Appl. Multivariate Statistics - Spring 2012 11

Classical MDS Appl. Multivariate Statistics - Spring 2012 12

Classical MDS: Theory  Input: Euclidean distances between n objects in p dimensions  Output: Position of points up to rotation, reflection, shift  Two steps: - Compute inner products matrix B from distance - Compute positions from B Appl. Multivariate Statistics - Spring 2012 13

Classical MDS: Theory – Step 1  Inner products matrix B = XX T  Connect to distance: d 2 ij = b ii + b jj ¡ 2 b ij  Center points to avoid shift invariance  Invert realtionship: b ij = ¡ 1 2 ( d 2 ij ¡ d 2 i: ¡ d 2 :j + d 2 :: ) “doubly centered” Appl. Multivariate Statistics - Spring 2012 14

Classical MDS: Theory – Step 2  Since B = XX T , we need the “square root” of B  B is a symmetric and positive definite n*n matrix  Thus, B can be diagonalized: B = V ¤ V T D is a diagonal matrix with on diagonal ¸ 1 ¸ ¸ 2 ¸ ::: ¸ ¸ n (“eigenvalues”) V contains as columns normalized eigenvectors  Some eigenvalues will be zero; drop them: B = V 1 ¤ 1 V T 1 ¡ 1  Take “square root”: X = V 1 ¤ 2 1 Appl. Multivariate Statistics - Spring 2012 15

Classical MDS: Low-dim representation  Keep only few (e.g. 2) largest eigenvalues and corresponding eigenvectors  The resulting X will be the low-dimensional representation we were looking for  Goodness of fit (GOF) if we reduce to m dimensions: P m (should be at least 0.8) i =1 ¸ i P n GOF = i =1 ¸ i  Finds “optimal” low -dim representation: Minimizes ³ ij ) 2 ´ S = P n P n ij ¡ ( d ( m ) d 2 i =1 j =1 Appl. Multivariate Statistics - Spring 2012 16

Classical MDS: Pros and Cons + Optimal for euclidean input data + Still optimal, if B has non-negative eigenvalues (pos. semidefinite) + Very fast - No guarantees if B has negative eigenvalues However, in practice, it is still used then. New measures for Goodness of fit: P m P m P m i =1 ¸ 2 i =1 j ¸ i j i =1 max (0 ;¸ i ) P n P n P n i GOF = GOF = GOF = i =1 ¸ 2 i =1 j ¸ i j i =1 max (0 ;¸ i ) i Used in R function “ cmdscale ” Appl. Multivariate Statistics - Spring 2012 17

Non-metric MDS: Idea  Sometimes, there is no strict metric on original points  Example: How much do you like the portraits? (1: Not at all, 10: Very much) 9 2 6 10 ?? 5 1 OR Appl. Multivariate Statistics - Spring 2012 18

Non-metric MDS: Idea  Absolute values are not > that meaningful  Ranking is important  Non-metric MDS finds a low-dimensional representation, which respects the ranking of distances > Appl. Multivariate Statistics - Spring 2012 19

Non-metric MDS: Theory  is the true dissimilarity, d ij is the distance of representation ± ij  Minimize STRESS ( is an increasing function): µ P i<j ( µ ( ± ij ) ¡ d ij ) 2 P S = i<j d 2 ij  Optimize over both position of points and µ  is called “disparity” ^ d ij = µ ( ± ij )  Solved numerically (isotonic regression); Classical MDS as starting value; very time consuming Appl. Multivariate Statistics - Spring 2012 20

Non-metric MDS: Example for intuition (only) True points in high dimensional space C Compute best representation 5 3 STRESS = 19.7 A 2 B d AB < d BC < d AC Appl. Multivariate Statistics - Spring 2012 21

Non-metric MDS: Example for intuition (only) True points in high dimensional space C Compute best representation 4.8 2.7 STRESS = 20.1 A 2 B d AB < d BC < d AC Appl. Multivariate Statistics - Spring 2012 22

Non-metric MDS: Example for intuition (only) True points in high dimensional space C Compute best representation 5.2 STRESS = 18.9 2.9 A 2 B Stop if minimal STRESS is found. We will finally represent the distances d AB = 2, d BC = 2.9, d AC = 5.2 d AB < d BC < d AC Appl. Multivariate Statistics - Spring 2012 23

Non-metric MDS: Pros and Cons + Fulfills a clear objective without many assumptions (minimize STRESS) + Results don’t change with rescaling or monotonic variable transformation + Works even if you only have rank information - Slow in large problems - Usually only local (not global) optimum found - Only gets ranks of distances right Appl. Multivariate Statistics - Spring 2012 24

Non-metric MDS: Example  Do people in the same party vote alike?  Agreement of 15 congressman in 19 votes … Appl. Multivariate Statistics - Spring 2012 25

Non-metric MDS: Example Appl. Multivariate Statistics - Spring 2012 26

Concepts to know  Classical MDS: - Finds low-dim projection that respects distances - Optimal for euclidean distances - No clear guarantees for other distances - fast  Non-metric MDS: - Squeezes data points on table - respects only rankings of distances - (locally) solves clear objective - slow Appl. Multivariate Statistics - Spring 2012 27

R commands to know  cmdscale included in standard R distribution  isoMDS from package “MASS” Appl. Multivariate Statistics - Spring 2012 28