Nonlinear Dimensionality Visualize all dimensions Visualize the - PowerPoint PPT Presentation

Overview Direct visualization Dimensionality reduction Nonlinear Dimensionality  Visualize all dimensions  Visualize the intrinsic low-dimensional structure Reduction  Direct visualization vs. within a high-dimensional data space dimensionality reduction  Ideally 2 or 3 dimensions so data can be displayed with a single scatterplot  Nonlinear dimensionality reduction Donovan Parks techniques:  ISOMAP, LLE, Charting Dimensionality Reduction  A fun example that uses non- metric, replicated MDS Sources: Chuah (1998), Wegman (1990) Why do we need nonlinear When to use: Nonlinear dimensionality reduction ISOMAP dimensionality reduction?  Extension of multidimensional Y  Isometric mapping (ISOMAP)  Direct visualization: scaling (MDS)  Mapping a Manifold of Perceptual Observations .  Interested in relationships between Joshua B. Tenenbaum. Neural Information Processing Systems, 1998. attributes (dimensions) of the data  Considers geodesic instead of  Locally Linear Embedding (LLE) Euclidean distances  Think Globally, Fit Locally: Unsupervised Learning of Nonlinear Manifolds . Lawrence K.  Dimensionality reduction: Saul & Sam T. Roweis. University of X Pennsylvania Technical Report MS-CIS-02-18,  Interested in geometric relationships 2002. Linear DR (PCA, Classic MDS, ...) between data points  Charting  Charting a Manifold . Matthew Brand, NIPS Nonlinear DR (Metric MDS , ISOMAP, LLE, ...) 2003. Geodesic vs. Euclidean distance Calculating geodesic distances ISOMAP Algorithm Example: ISOMAP vs. MDS  Q: How do we calculate geodesic Geodesic distance? Distance 1 2 Matrix 3 Observations in Neighborhood High -D space Graph  Construct neighborhood graph   Compute geodesic distance matrix  Apply favorite MDS algorithm ISOMAP Embedding Source: Tenenbaum, 1998 Modified from: Tenenbaum, 1998 Are local constraints sufficient? Example: Punctured sphere +/-’s of ISOMAP Locally Linear Embedding (LLE) A Geometric Interpretation  Advantages:  Maintains approximate global structure  ISOMAP generally fails for manifolds  Forget about global constraints, just since local patches overlap  Easy to understand and implement with holes fit locally extension of MDS  Preserves “true” relationship between data points  Why? Removes the need to estimate distances between widely separated points  Disadvantages:  ISOMAP approximates such distances  Computationally expensive with an expensive shortest path search  Known to have difficulties with “holes”

Are local constraints sufficient? LLE Algorithm Example: Synthetic manifolds Example: Real face images A Geometric Interpretation  Maintains approximate global structure 2 since local patches overlap ( W ) � X � W X � = � i ij j i j 2 ( ) Y Y W Y � = � � � i ij j i j Source: Saul, 2002 Modified from: Saul, 2002 Source: Roweis, 2000 +/-’s of LLE Charting Charting the data Find local coordinate systems  Advantages:  Use PCA in each chart to determine local  Place Gaussian at each point and estimate  Similar to LLE in that it considers coordinate system covariance over local neighborhood  More accurate in preserving local overlapping “locally linear patches” structure than ISOMAP (called charts in this paper)  Less computationally expensive than ISOMAP  Based on a statistical framework  Brand derives method for determining optimal covariances in instead of geometric arguments  Disadvantages: Local the MAP sense Coordinate Systems  Less accurate in preserving global  Enforces certain constraints to ensure structure than ISOMAP nearby Gaussians (charts) have similar covariance matrices  Known to have difficulty on non-convex manifolds (not true of ISOMAP) Conclusion: Connecting the charts Example: Noisy synthetic data +/-’s of Charting +/-’s of dimensionality reduction  Advantage:  Exploit overlap of each  Advantages: neighborhood to determine  More robust to noise than LLE or  Excellent visualization of relationship how to connect the charts ISOMAP between data points  Disadvantage:  Limitations:  More testing needed to demonstrate  Brand suggest a Embedded robustness to noise  Computationally expensive Charts weighted least squares  Unclear computational complexity  Need many observations problem to minimize  Final step is quadratic in the number of  Do not work on all manifolds error in the projection of charts common points Source: Brand, 2003 Action Synopsis: Aspects of motion Dimensionality reduction Pose selection A fun example  Input: pose of person at each frame  Action Synopsis: Pose Selection and Illustration .  Problem: How can these aspects of motion  Problem: how do you select Jackie Assa, Yaron Caspi, Daniel Cohen-Or. ACM be combined? interesting poses from the Transactions on Graphics, 2005. “motion curve”?  Typically 5-9 dimensions  Solution: non-metric, replicated MDS  distance matrix for each aspect of motion  Assa et al. argue that  best preserves rank order of distances across several distance matrices interesting poses occur at “locally extreme points”  Aspects of motion:  Joint position  Essentially NM-RMDS implicitly weights  Joint angle each distance matrix  Joint velocity  Joint angular velocity Source: Assa, 2005 Source: Assa, 2005 Source: Assa, 2005 Source: Assa, 2005

Finding locally extreme points Example: Monkey bars Example: Potential application Do you need dimensionality reduction? Source: Assa, 2005 Source: Assa, 2005 Source: Assa, 2005 Source: Assa, 2005 Critique of Action Synopsis Literature Papers covered:  Pros: Mapping a Manifold of Perceptual Observations . Joshua B. Tenenbaum.  Neural Information Processing Systems, 1998. + Results are convincing Think Globally, Fit Locally: Unsupervised Learning of Nonlinear  Manifolds . Lawrence Saul & Sam Roweis. University of Pennsylvania + Justified algorithm with user study Technical Report MS-CIS-02-18, 2002. Charting a Manifold . Matthew Brand, NIPS 2003.  Action Synopsis: Pose Selection and Illustration . Jackie Assa, Yaron  Caspi, Daniel Cohen-Or. ACM Transactions on Graphics, 2005. Cons: Additional reading:  - Little justification for selected aspects of Multidimensional scaling . Forrest W. Young.  Forrest.psych.unc.edu/teaching/p208a/mds/mds.html motion A Global Geometric Framework for Nonlinear Dimensionality  Reduction. Joshua B. Tenenbaum, Vin de Silva, John C. Langford, - Requiring pose information as input is Science, v. 290 no.5500, 2000. restrictive Nonlinear dimensionality reduction by locally linear embedding. Sam  Roweis & Lawrence Saul. Science v.290 no.5500, 2000. - Unclear that having RMDS implicitly Further citations: weight aspects of motion is a good idea  Information Rich Glyphs for Software Management . M.C. Chuah and  S.G. Eick, IEEE CG&A 18:4 1998. Hyperdimensional Data Analysis Using Parallel Coordinates . Edward J.  Wegman. Journal of the American Statistical Association, Vol. 85, No. 411. (Sep., 1990), pp. 664-675 .