Dimensionality Reduction for Visualization Lecture 13 April 8, - PowerPoint PPT Presentation

CS53000-Spring 2020 Introduction to Scientific Visualization Dimensionality Reduction for Visualization Lecture 13 April 8, 2020

Outline High-dimensional data Dimensionality reduction Manifold learning Topological data analysis 2 CS530 / Spring 2020 : Introduction to Scientific Visualization. 04/02/2020 Dimensionality Reduction for Visualization

<latexit sha1_base64="BdLZ/k2vmHhlTSaAe3o9KOYjGqA=">AC43icbVLNi9NAFJ/Er7V+VT16GSzCSrEkXUEvQsE9eFzB7i40tb5MJ+2wk5kw8yKUIVcvHhTx6j/lzX/Fk5MmW7K7Pg8fh/zvpIWUliMoj9BeO36jZu39m737ty9d/9B/+GjY6tLw/iUanNaQqWS6H4FAVKfloYDnkq+Ul69rbmTz5zY4VWH3BT8HkOKyUywQA9tOj/Tcxa0yQzwNxhkgOu08yVeUOsaJvaMNKnuF+K0oKMChA0q54B3rTsMPQhC010kRBKuECbsRqjc9hRc7NgXjisZftuDHcdc0bFuIK3dQdVRNf+fPNBV3rvNKw3aWlhVdNEfRKNoG/RqErfJgLRxtOj/TpalTlXyCRYO4ujAueuHp1JXvWS0vIC2Bms+MynCnJu5257o4o+8iSZtr4TyHdol2Hg9zaTZ56Zd2ivczV4P+4WYnZ67kTqiRK9YUykp/Ck3rg9OlMJyh3PgEmBG+V8rW4PeI/rfo+SXEl0e+mhyPR/HBaPz+5WAyadexR56Qp2SfxOQVmZB35IhMCQs+BV+Cb8H3kIdfwx/hz0YaBq3nMbkQ4a9/oWrowA=</latexit> Why High-Dimensional? • Data samples with many attributes • high-throughput imaging, hyperspectral imaging, medical records, … • PDE modeling of physical systems • example: Navier Stokes equations ✓ ∂ u ◆ p + µ r 2 u + 1 ρ D u Dt = ρ ∂ t + u · r u = �r ¯ 3 µ r ( r · u ) + ρ g 3 CS530 / Spring 2020 : Introduction to Scientific Visualization. 04/02/2020 Dimensionality Reduction for Visualization

Why High-Dimensional? • Amalgamate individual properties into high-dimensional feature vectors • e.g. , turn images into vectors 4 CS530 / Spring 2020 : Introduction to Scientific Visualization. 04/02/2020 Dimensionality Reduction for Visualization

Challenges • “Curse of dimensionality” • Sampling becomes exponentially costly • Space accumulates in“corners” of hypercube • Data processing becomes extremely expensive / intractable • How to visualize? • Missing intuition for high-dim spatial domain 5 CS530 / Spring 2020 : Introduction to Scientific Visualization. 04/02/2020 Dimensionality Reduction for Visualization

<latexit sha1_base64="2czjdlwhHmVdGfR8iKzpzuVQ97Y=">ACK3icbZDLSgMxFIYzXmu9V26CRZBEcpMFXRZdKHLCvYCnbZk0kwbTDJDckYow7yPG1/FhS684Nb3ML0savVA4OP/z+Hk/EsuAHX/XAWFpeWV1Zza/n1jc2t7cLObt1EiasRiMR6WZADBNcsRpwEKwZa0ZkIFgjuL8a+Y0Hpg2P1B0MY9aWpK94yCkBK3ULl36oCU39mHfSCaosLWdZlvrXREriCxbCEZ6x8An2sK95fwDHuNxRWbdQdEvuPBf8KZQRNOqdgsvfi+iWQKqCDGtDw3hnZKNHAqWJb3E8NiQu9Jn7UsKiKZafjWzN8aJUeDiNtnwI8VmcnUiKNGcrAdkoCAzPvjcT/vFYC4U75SpOgCk6WRQmAkOER8HhHteMghaIFRz+1dMB8TGAjbevA3Bmz/5L9TLJe+0VL49K1Yq0zhyaB8doCPkoXNUQTeoimqIokf0jN7Qu/PkvDqfztekdcGZzuyhX+V8/wCiYafP</latexit> Ratio of hypersphere volume / hypercube volume n 2 π � n � Γ 2 + 1 2 n CS530 / Spring 2020 : Introduction to Scientific Visualization. 04/02/2020 Dimensionality Reduction for Visualization 6

DIMENSION REDUCTION 7 CS530 / Spring 2020 : Introduction to Scientific Visualization. 04/02/2020 Dimensionality Reduction for Visualization

Principal Component Analysis • Assume a point cloud R k ( x i ) i =1 ,..,n ∈ I • Interpret these points as observations of a random variable R k X ∈ I c = 1 • The empirical mean (centroid) is � x i n i • The covariance matrix is given by A = ¯ X¯ X T A jl = 1 � ( x ij − c j )( x il − c l ) n i 8 CS530 / Spring 2020 : Introduction to Scientific Visualization. 04/02/2020 Dimensionality Reduction for Visualization

PCA • The eigenvectors of the covariance matrix form a data-dependent coordinate system • The first m eigenvectors (in decreasing order of associated eigenvalues) span the m principal dimensions of the point cloud. 9 CS530 / Spring 2020 : Introduction to Scientific Visualization. 04/02/2020 Dimensionality Reduction for Visualization

MDS Multidimensional Scaling •Input: dissimilarity matrix R n × n ∆ ∈ I with and ∆ ij = δ i,j ∆ ii = 0 ∆ ij > 0 •Goal: find such that R d x 1 , x 2 , . . . , x n ∈ I ∀ ( i, j ) || x i − x j || ≈ δ i, j 10 CS530 / Spring 2020 : Introduction to Scientific Visualization. 04/02/2020 Dimensionality Reduction for Visualization

<latexit sha1_base64="2ePEJcstSKUfZ7QdDdfORtkckqI=">AB+XicbVDLSsNAFL3xWesr6tLNYBFclaQKuiy6cVnBPqAJYTKZtEMnkzgzKZTQP3HjQhG3/ok7/8Zpm4W2Hhg4nHMu984JM86Udpxva219Y3Nru7JT3d3bPzi0j47Ks0loW2S8lT2QqwoZ4K2NdOc9jJcRJy2g1HdzO/O6ZSsVQ86klG/QPBIsZwdpIgW173IQjHDkDegTcgK75tSdOdAqcUtSgxKtwP7yopTkCRWacKxU3Uy7RdYakY4nVa9XNEMkxEe0L6hAidU+cX8ik6N0qE4lSaJzSaq78nCpwoNUlCk0ywHqplbyb+5/VzHd/4BRNZrqkgi0VxzpFO0awGFDFJieYTQzCRzNyKyBLTLQpq2pKcJe/vEo6jbp7W8XNWat2UdFTiFM7gAF6hCfQgjYQGMzvMKbVgv1rv1sYiuWeXMCfyB9fkDbJKS3Q=</latexit> MDS Method: B = − 1 H = I − 1 • Center(*): where n 11 T 2 H ∆ H • Spectral decomposition: B = U Λ U T • Clamp(*): ( Λ + ) ij = max( Λ ij , 0) 1 • Solution: first d columns of X = U Λ 2 + This is a PCA! ∆ B (*): if measures Euclidean distances, is positive semidefinite ( ) λ i ≥ 0 11 CS530 / Spring 2020 : Introduction to Scientific Visualization. 04/02/2020 Dimensionality Reduction for Visualization

MANIFOLD LEARNING 12 CS530 / Spring 2020 : Introduction to Scientific Visualization. 04/02/2020 Dimensionality Reduction for Visualization

Manifold Learning • Move away from linear assumption • Assume curved low-dimensional geometry • Assume smoothness ➡ MANIFOLD Problem: Given points x 1 , . . . , x n ∈ R D that lie on a d -dimensional manifold M that can be described by a single coordi- nate chart f : M → R d , find y 1 , . . . , y n ∈ • Manifold learning def R d , where y i = f ( x i ). 13 CS530 / Spring 2020 : Introduction to Scientific Visualization. 04/02/2020 Dimensionality Reduction for Visualization

����������������� ������������������������������� Isomap •Form k-NN graph of input points •Form dissimilarity matrix as ∆ square of approximated geodesic distance between points •Compute MDS! 14 CS530 / Spring 2020 : Introduction to Scientific Visualization. 04/02/2020 Dimensionality Reduction for Visualization

Isomap •Computing geodesic distance • standard graph problem in CS ������������������� O ������������������� 15 CS530 / Spring 2020 : Introduction to Scientific Visualization. 04/02/2020 Dimensionality Reduction for Visualization

����������� ������������ Isomap • Successful in computer vision problems 16 CS530 / Spring 2020 : Introduction to Scientific Visualization. 04/02/2020 Dimensionality Reduction for Visualization

���������������������������� Locally Linear Embedding Intuition • smooth manifold is locally close to linear Method • Characterize local linearity through weight matrix W X W ij x j || 2 || x i − such that is minimized j ∈ N ( i ) X X • Find d-dimensional ‘s that minimize W ij y j || 2 || y i − y i j • Solution is obtained as first d eigenvectors of ( I − W ) T ( I − W ) 17 CS530 / Spring 2020 : Introduction to Scientific Visualization. 04/02/2020 Dimensionality Reduction for Visualization

���������������������������� Locally Linear Embedding Intuition • smooth manifold is locally close to linear Method • Characterize local linearity through weight matrix W X W ij x j || 2 || x i − such that is minimized j ∈ N ( i ) X X • Find d-dimensional ‘s that minimize W ij y j || 2 || y i − y i j • Solution is obtained as first d eigenvectors of ( I − W ) T ( I − W ) 17 CS530 / Spring 2020 : Introduction to Scientific Visualization. 04/02/2020 Dimensionality Reduction for Visualization

���������������������������� Locally Linear Embedding Intuition • smooth manifold is locally close to linear Method • Characterize local linearity through weight matrix W X W ij x j || 2 || x i − such that is minimized j ∈ N ( i ) X X • Find d-dimensional ‘s that minimize W ij y j || 2 || y i − y i j • Solution is obtained as first d eigenvectors of ( I − W ) T ( I − W ) 17 CS530 / Spring 2020 : Introduction to Scientific Visualization. 04/02/2020 Dimensionality Reduction for Visualization

LLE 18 CS530 / Spring 2020 : Introduction to Scientific Visualization. 04/02/2020 Dimensionality Reduction for Visualization