Diffusion Processes and Dimensionality Reduction on Manifolds ESI, - - PowerPoint PPT Presentation

Faculty of Science

Diffusion Processes and Dimensionality Reduction on Manifolds

ESI, Vienna, Feb. 2015 Stefan Sommer

Department of Computer Science, University of Copenhagen

February 23, 2015 Slide 1/22

Outline

• Dimensionality Reduction
• Diffusion PCA
• Development and Anisotropic Diffusions
• Examples

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 2/22

Dimensionality Reduction in Non-Linear Manifolds

• dim. reduction and linearizations - mappings from

non-linear manifolds to low dimensional Euclidean space that preserves structure of data

• Non-Euclidean generalizations of PCA:
• Principal Geodesic Analysis (PGA, Fletcher et al., ’04)
• Geodesic PCA (GPCA, Huckeman et al., ’10)
• Horizontal Component Analysis (HCA, Sommer, ’13)
• Principal Nested Spheres ((C)PNS, Jung et al., ’12)
• Barycentric Subspaces (BS, Pennec, ’15)

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 3/22

Dimensionality Reduction in Non-Linear Manifolds

• dim. reduction and linearizations - mappings from

non-linear manifolds to low dimensional Euclidean space that preserves structure of data

• Non-Euclidean generalizations of PCA:
• Principal Geodesic Analysis (PGA, Fletcher et al., ’04)
• Geodesic PCA (GPCA, Huckeman et al., ’10)
• Horizontal Component Analysis (HCA, Sommer, ’13)
• Principal Nested Spheres ((C)PNS, Jung et al., ’12)
• Barycentric Subspaces (BS, Pennec, ’15)

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 3/22

PGA: analysis relative to the data mean

Dimensionality Reduction in Non-Linear Manifolds

• dim. reduction and linearizations - mappings from

non-linear manifolds to low dimensional Euclidean space that preserves structure of data

• Non-Euclidean generalizations of PCA:
• Principal Geodesic Analysis (PGA, Fletcher et al., ’04)
• Geodesic PCA (GPCA, Huckeman et al., ’10)
• Horizontal Component Analysis (HCA, Sommer, ’13)
• Principal Nested Spheres ((C)PNS, Jung et al., ’12)
• Barycentric Subspaces (BS, Pennec, ’15)

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 3/22

PGA: analysis relative to the data mean data points on non-linear manifold

Dimensionality Reduction in Non-Linear Manifolds

• dim. reduction and linearizations - mappings from

non-linear manifolds to low dimensional Euclidean space that preserves structure of data

• Non-Euclidean generalizations of PCA:
• Principal Geodesic Analysis (PGA, Fletcher et al., ’04)
• Geodesic PCA (GPCA, Huckeman et al., ’10)
• Horizontal Component Analysis (HCA, Sommer, ’13)
• Principal Nested Spheres ((C)PNS, Jung et al., ’12)
• Barycentric Subspaces (BS, Pennec, ’15)

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 3/22

PGA: analysis relative to the data mean intrinsic mean µ

Dimensionality Reduction in Non-Linear Manifolds

• dim. reduction and linearizations - mappings from

non-linear manifolds to low dimensional Euclidean space that preserves structure of data

• Non-Euclidean generalizations of PCA:
• Principal Geodesic Analysis (PGA, Fletcher et al., ’04)
• Geodesic PCA (GPCA, Huckeman et al., ’10)
• Horizontal Component Analysis (HCA, Sommer, ’13)
• Principal Nested Spheres ((C)PNS, Jung et al., ’12)
• Barycentric Subspaces (BS, Pennec, ’15)

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 3/22

PGA: analysis relative to the data mean tangent space TµM

Dimensionality Reduction in Non-Linear Manifolds

• dim. reduction and linearizations - mappings from

non-linear manifolds to low dimensional Euclidean space that preserves structure of data

• Non-Euclidean generalizations of PCA:
• Principal Geodesic Analysis (PGA, Fletcher et al., ’04)
• Geodesic PCA (GPCA, Huckeman et al., ’10)
• Horizontal Component Analysis (HCA, Sommer, ’13)
• Principal Nested Spheres ((C)PNS, Jung et al., ’12)
• Barycentric Subspaces (BS, Pennec, ’15)

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 3/22

PGA: analysis relative to the data mean projection of data point to TµM

Dimensionality Reduction in Non-Linear Manifolds

• dim. reduction and linearizations - mappings from

non-linear manifolds to low dimensional Euclidean space that preserves structure of data

• Non-Euclidean generalizations of PCA:
• Principal Geodesic Analysis (PGA, Fletcher et al., ’04)
• Geodesic PCA (GPCA, Huckeman et al., ’10)
• Horizontal Component Analysis (HCA, Sommer, ’13)
• Principal Nested Spheres ((C)PNS, Jung et al., ’12)
• Barycentric Subspaces (BS, Pennec, ’15)

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 3/22

PGA: analysis relative to the data mean Euclidean PCA in tangent space

Dimensionality Reduction in Non-Linear Manifolds

• dim. reduction and linearizations - mappings from

non-linear manifolds to low dimensional Euclidean space that preserves structure of data

• Non-Euclidean generalizations of PCA:
• Principal Geodesic Analysis (PGA, Fletcher et al., ’04)
• Geodesic PCA (GPCA, Huckeman et al., ’10)
• Horizontal Component Analysis (HCA, Sommer, ’13)
• Principal Nested Spheres ((C)PNS, Jung et al., ’12)
• Barycentric Subspaces (BS, Pennec, ’15)

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 3/22

Dimensionality Reduction in Non-Linear Manifolds

• dim. reduction and linearizations - mappings from

non-linear manifolds to low dimensional Euclidean space that preserves structure of data

• Non-Euclidean generalizations of PCA:
• Principal Geodesic Analysis (PGA, Fletcher et al., ’04)
• Geodesic PCA (GPCA, Huckeman et al., ’10)
• Horizontal Component Analysis (HCA, Sommer, ’13)
• Principal Nested Spheres ((C)PNS, Jung et al., ’12)
• Barycentric Subspaces (BS, Pennec, ’15)

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 3/22

PGA: GPCA: HCA:

PGA, GPCA, HCA, PNS, . . .

• search for explicitly constructed parametric

subspaces: geodesic sprays, geodesics, iterated development, . . .

• in general manifolds, these subspaces are not totally

geodesic

• projections to subspaces are problematic: geodesics

may be dense on tori

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 4/22

Generalizing Linear Statistics

Euclidean Riemannian norm x − y distances d(x,y) vectors v0 for geodesics linear subspaces geodesic sprays . . . . . . why are geodesics fundamental when estimating covariance?

• Euclidean space analogies can lead to non-local

constructions

• to goal of this talk is to get closer to constructions

defined “infinitesimally”

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 5/22

Euclidean PCA

Usual formulation:

• eigendecomposition (u1,λ1),...,(ud,λd) of sample
• covar. matrix C
• principal components: xn = UT(yn −µ)

Probabilistic interpretation (Tipping, Bishop, ’99):

• latent variable model

y = Wx +µ+ε, x ∼ N (0,I),

ε ∼ N (0,σ2I)

• marginal distribution y ∼ N (µ,Cσ), Cσ = WW T +σ2I
• MLE of W: WML = U(Λ−σ2I)1/2 + rotation

Λ = diag(λ1,...,λd)

• principal components:

E[xn|yn] = (W T W +σ2I)−1W T

ML(yn −µ)

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 6/22

Diffusion PCA

• probabilistic PCA does not explicitly use subspaces
• on Riemannian manifolds, the

Eells-Elworthy-Malliavin construction gives a map

• Diff : FM → Dens(M)
• Γ ⊂ Dens(M): the image
• Diff(FM), the set of

(normalized) densities resulting from diffusions in FM

• µ ∈ Γ

≈

anisotropic normal distribution

• with µ =
• Diff(x,Xα) = pµµ0, define the log-likelihood

lnL(x,Xα) = lnL(µ) =

N

∑

i=1

lnpµ(yi)

• Diffusion PCA: maxim. lnL(x,Xα) for (x,Xα) ∈ FM
• MLE of data yi under the assumption y ∼ µ ∈ Γ

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 7/22

Diffusion PCA

• all geometric complexities are hidden in the diffusion

processes

• no parametric subspaces, no projections to dense

geodesics

• principal components: mean sample paths reaching yi

ˆ

xi(t) = E[x(t)|x(1) = yi]

• path dependency can be integrated out

˜

xi = 1 d dt ˆ xi(t)dt = ˆ xi(1)

˜

xi provides a linear view of the data

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 8/22

Diffusion PCA

• all geometric complexities are hidden in the diffusion

processes

• no parametric subspaces, no projections to dense

geodesics

• principal components: mean sample paths reaching yi

ˆ

xi(t) = E[x(t)|x(1) = yi]

• path dependency can be integrated out

˜

xi = 1 d dt ˆ xi(t)dt = ˆ xi(1)

˜

xi provides a linear view of the data

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 8/22

Statistical Manifold: Geometry of Γ

• Densities

Dens(M) = {µ ∈ Ωn(M) :

• M µ = 1,µ > 0}
• Fisher-Rao metric: GFR

µ (α,β) =

• M

α µ β µµ

• Γ finite dim. subset of Dens(M)
• properties of
• Diff : FM → Dens(M)
• naturally defined on bundle of symmetric positive T 0

2

tensors

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 9/22

SDEs On Manifolds

• stationary driftless diffusion SDE in Rn:

dXt = σdWt , σ ∈ Mn×d

• diffusion field σ, infinitesimal generator σσT
• curvature: stationary field/generator cannot be

defined due to holonomy

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 10/22

SDEs On Manifolds

• stationary driftless diffusion SDE in Rn:

dXt = σdWt , σ ∈ Mn×d

• diffusion field σ, infinitesimal generator σσT
• curvature: stationary field/generator cannot be

defined due to holonomy

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 10/22

The Frame Bundle

• the manifold and frames (bases) for the tangent

spaces TpM

• F(M) consists of pairs u = (x,Xα), x ∈ M, Xα frame

for TxM

• curves in the horizontal part of F(M) correspond to

curves in M and parallel transport of frames

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 11/22

The Frame Bundle

• the manifold and frames (bases) for the tangent

spaces TpM

• F(M) consists of pairs u = (x,Xα), x ∈ M, Xα frame

for TxM

• curves in the horizontal part of F(M) correspond to

curves in M and parallel transport of frames

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 11/22

The Frame Bundle

• the manifold and frames (bases) for the tangent

spaces TpM

• F(M) consists of pairs u = (x,Xα), x ∈ M, Xα frame

for TxM

• curves in the horizontal part of F(M) correspond to

curves in M and parallel transport of frames

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 11/22

SDEs On Manifolds: Eells-Elworthy-Malliavin construction

• Hi, i = 1...,n horizontal vector fields on F(M):

Hi(u) = π−1

∗ (ui)

• SDE in Rn:

dXt = σ(Xt)dWt , σ(Xt) ∈ Mn×n

• SDE in F(M):

dUt = Hi(Ut)◦ dX i

t = Hi(Ut)◦σ(Xt)i jdW j t

• SDE on M:

πF(M)(Ut)

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 12/22

SDEs On Manifolds: Eells-Elworthy-Malliavin construction

• Hi, i = 1...,n horizontal vector fields on F(M):

Hi(u) = π−1

∗ (ui)

• SDE in Rn:

dXt = σ(Xt)dWt , σ(Xt) ∈ Mn×n

• SDE in F(M):

dUt = Hi(Ut)◦ dX i

t = Hi(Ut)◦σ(Xt)i jdW j t

• SDE on M:

πF(M)(Ut)

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 12/22

Anisotropic Diffusions

• Diff(x,Xα) = πF(M)(U1)
• stochastic development / “rolling without slipping”
• in Rn, sample path increments ∆xti = xti+1 − xti are

normally distributed N (0,(∆t)−1Σ) with log-probability ln˜ pΣ(xt) ∝ − 1

∆t

N−1

∑

i=1

∆xT

ti Σ−1∆xti + c

• Formally, we can set

ln˜ pΣ(xt) ∝ − 1

0 ˙

xt2

Σdt + c

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 13/22

“Most Probable Paths”

• Let φ be path development and Ut a heat diffusion. As

N → ∞, the finite path measure → pullback φ∗v of Wiener measure v on W([0,1],M) (cont. paths from x) (Andersson, Driver, ’99)

• geodesics and be formally viewed as “most probable

paths” (MPPs) for the pull-back path density, i.e. MPPs for the Euclidean Rn diffusion

• Anisotropic case: (xt,Xα,t) path in FM, Xα,t

represents Σ1/2 and defines invertible map

Rn → TxtM. Inner product on TxtM v,wXα,t =

• X−1

α,t v,X−1 α,t w

• Rn

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 14/22

Sub-Riemannian Geometry

• optimal control problem with nonholonomic

constraints xt = arg min

ct,c0=x,c1=y

1

0 ˙

ct2

Xα,tdt

• let

˜

v, ˜ wHFM =

• X−1

α,t π∗(˜

v),X−1

α,t π∗(˜

w)

• Rn
• n H(xt,Xα,t)FM. This defines a sub-Riemannian

metric G on TFM and equivalent problem

(xt,Xα,t) =

arg min

(ct,Cα,t),c0=x,c1=y

1

0 (˙

ct, ˙ Cα,t)2

HFMdt

(1) with constraints (˙ ct, ˙ Cα,t) ∈ H(ct,Cα,t)FM

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 15/22

Evolution Equations

• geodesics satisfy the Hamilton-Jacobi equations

˙

yk

t = Gkj(yt)ξt,j

, ˙ ξt,k = −1

2

∂Gpq ∂yk ξt,pξt,q

• in coordinates (xi) for M, X i

α for Xα, and W encoding

the inner product W kl = δαβX k

αX l β:

˙

xi = W ijξj − W ihΓ

jβ h ξjβ

, ˙

X i

α = −Γiα

h W hjξj +Γiα k W khΓ jβ h ξjβ

˙ ξi = W hlΓkδ

l,iξhξkδ − 1

2

• Γ

hγ k,iW khΓkδ h +Γ hγ k W khΓkδ h,i

• ξhγξkδ

˙ ξiα = Γ

hγ k,iαW khΓkδ h ξhγξkδ −

• W hl

,iα Γkδ

l + W hlΓkδ l,iα

• ξhξkδ

− 1

2

• W hk

,iα ξhξk +Γ

hγ k W kh

,iα Γkδ

h ξhγξkδ

• Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds

Slide 16/22

S2

(a) cov. diag(1,1) (b) cov. diag(2,.5) (c) cov. diag(4,.25)

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 17/22

S2

(d) cov. diag(1,1) (e) cov. diag(2,.5) (f) cov. diag(4,.25)

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 17/22

Landmark LDDMM

• Christoffel symbols (Michelli et al. ’08)

Γk

ij = 1

2gir

• gklgrs

,l − gslgrk ,l − grlgks ,l

• gsj
• mix of transported frame and cometric: F dM bundle
• f rank d linear maps Rd → TxM, ξ,˜

ξ ∈ T ∗F dM,

cometric

• ξ,˜

ξ

• = δαβ(ξ|π−1

∗ Xα)(˜

ξ|π−1

∗ Xβ)+λ

• ξ,˜

ξ

• gR
• the whole frame need not be transported

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 18/22

Landmark LDDMM Examples

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 19/22

Summary

• Diffusion PCA: dim. reduction or linearization by MLE
• f densities generated by diffusion processes
• infinitesimal definition, no subspaces, no projections
• diffusion map
• Diff : FM → Dens(M) from

Eells-Elworthy-Malliavin construction of Brownian motion

• “MPPs” for anisotropic diffusions generalize

geodesics

1 Sommer: Diffusion Processes and PCA on Manifolds, Oberwolfach extended abstract, 2014. 2 Sommer: Anisotropic Distributions on Manifolds: Template Estimation and Most Probable Paths, in preparation.

Stefan Sommer (sommer@diku.dk) (Department of Computer Science, University of Copenhagen) — Diffusion Processes and Dimensionality Reduction on Manifolds Slide 20/22