Lecture 12: SVD, Procrustes Analysis COMPSCI/MATH 290-04 Chris - - PowerPoint PPT Presentation

▶
lecture 12 svd procrustes analysis
SMART_READER_LITE
LIVE PREVIEW

Lecture 12: SVD, Procrustes Analysis COMPSCI/MATH 290-04 Chris - - PowerPoint PPT Presentation

Feb 22, 2024 93 likes •604 views

Lecture 12: SVD, Procrustes Analysis COMPSCI/MATH 290-04 Chris Tralie, Duke University 2/23/2016 COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis Announcements Group Assignment 1 Full Submission Due next Tuesday 11:55 PM

slide-1
SLIDE 1

Lecture 12: SVD, Procrustes Analysis

COMPSCI/MATH 290-04

Chris Tralie, Duke University

2/23/2016

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-2
SLIDE 2

Announcements

⊲ Group Assignment 1 Full Submission Due next Tuesday 11:55 PM ⊲ Hackathon Saturday 2/27 4:00 PM - 10:00 PM Gross Hall 330 ⊲ Rank Top 3 Final Project Choices By Next Wednesday 3/2

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-3
SLIDE 3

Table of Contents

◮ Ray Tracing Special Case ⊲ PCA Review ⊲ Singular Value Decomposition ⊲ Procrustes Distance ⊲ Final Projects

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-4
SLIDE 4

Ray Tracing Special Case

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-5
SLIDE 5

Ray Tracing Special Case

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-6
SLIDE 6

Ray Tracing Special Case

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-7
SLIDE 7

Ray Tracing Special Case

r s1 s s2 Face 1 Face 2

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-8
SLIDE 8

Table of Contents

⊲ Ray Tracing Special Case ◮ PCA Review ⊲ Singular Value Decomposition ⊲ Procrustes Distance ⊲ Final Projects

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-9
SLIDE 9

PCA Review

Organize point cloud into N × d matrix, each point along a column X =    | | . . . |

  • v1
  • v2

. . .

  • vN

| | . . . |    Choose a unit column vector direction u ∈ Rd×1 Then d = uTX gives projections onto u

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-10
SLIDE 10

PCA Review

Organize point cloud into N × d matrix, each point along a column X =    | | . . . |

  • v1
  • v2

. . .

  • vN

| | . . . |    Choose a unit column vector direction u ∈ Rd×1 Then d = uTX gives projections onto u ⊲ More consistent with what we’ve done; points in columns

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-11
SLIDE 11

PCA New Convention

d = uTX

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-12
SLIDE 12

PCA New Convention

d = uTX ⊲ How to express the sum of the squares of the dot products?

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-13
SLIDE 13

PCA New Convention

d = uTX ⊲ How to express the sum of the squares of the dot products? ddT

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-14
SLIDE 14

PCA New Convention

d = uTX ⊲ How to express the sum of the squares of the dot products? ddT ddT = (uTX)(uTX)T = uTXX Tu Want to find u that maximizes the above quadratic form

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-15
SLIDE 15

PCA New Convention

Use eigenvectors of A = XX T to find principal directions maximizing uTAu λ1 = 422

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-16
SLIDE 16

PCA Review

Use eigenvectors of A = XX T to find principal directions maximizing uTAu λ2 = 21.6

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-17
SLIDE 17

Table of Contents

⊲ Ray Tracing Special Case ⊲ PCA Review ◮ Singular Value Decomposition ⊲ Procrustes Distance ⊲ Final Projects

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-18
SLIDE 18

Orthogonal Matrices / Rotation Review

cos(θ) − sin(θ) sin(θ) cos(θ) x y

  • X

Y

cos(θ) sin(θ) θ cos(θ) θ

  • sin(θ)

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-19
SLIDE 19

Orthogonal Matrices / Rotation Review

Inverse rotation: dot product interpretation cos(θ) sin(θ) − sin(θ) cos(θ) x y

  • X

Y

cos(θ) sin(θ) θ cos(θ) θ

  • sin(θ)

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-20
SLIDE 20

Orthogonal Matrices / Rotation Review

In general R =     | | . . . |

  • u1
  • u2

. . .

  • uN

| | . . . |    

  • ui ·

uj = 1, i = j

  • ui ·

uj = 0, i = j In 3D,

  • u1 ×

u2 = u3 for a pure rotation

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-21
SLIDE 21

Orthogonal Matrices / Rotation Review

R =     | | . . . |

  • u1
  • u2

. . .

  • uN

| | . . . |     RT =      −

  • u1

− −

  • u2

− . . . . . . . . . −

  • uN

−     

  • ui ·

uj = 1, i = j

  • ui ·

uj = 0, i = j RTR = RRT = I

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-22
SLIDE 22

Singular Value Decomposition

Given an m × n matrix A, the SVD of A is A = USV T ⊲ U is an M × M rotation matrix ⊲ S is an M × N matrix, where Sij = 0 i = j ⊲ V is an N × N rotation matrix A =     | | . . . |

  • u1
  • u2

. . .

  • uM

| | . . . |            s1 . . . s2 . . . s3 . . . . . . . . . . . . . . . . . . . . . sM . . . 0             −

  • v1

− −

  • v2

− . . . . . . . . . −

  • vN

−

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-23
SLIDE 23

Singular Value Decomposition

A = USV T A =     | | . . . |

  • u1
  • u2

. . .

  • uM

| | . . . |            s1 . . . s2 . . . s3 . . . . . . . . . . . . . . . . . . . . . sM . . . 0             −

  • v1

− −

  • v2

− . . . . . . . . . −

  • vN

−

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-24
SLIDE 24

Singular Value Decomposition

A = USV T A =     | | . . . |

  • u1
  • u2

. . .

  • uM

| | . . . |            s1 . . . s2 . . . s3 . . . . . . . . . . . . . . . . . . . . . sM . . . 0             −

  • v1

− −

  • v2

− . . . . . . . . . −

  • vN

− ⊲ s1 > s2 > s3 > ... > sM

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-25
SLIDE 25

Singular Value Decomposition

A = USV T A =     | | . . . |

  • u1
  • u2

. . .

  • uM

| | . . . |            s1 . . . s2 . . . s3 . . . . . . . . . . . . . . . . . . . . . sM . . . 0             −

  • v1

− −

  • v2

− . . . . . . . . . −

  • vN

− ⊲ s1 > s2 > s3 > ... > sM ⊲ U holds the eigenvectors of AAT

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-26
SLIDE 26

Singular Value Decomposition

A = USV T A =     | | . . . |

  • u1
  • u2

. . .

  • uM

| | . . . |            s1 . . . s2 . . . s3 . . . . . . . . . . . . . . . . . . . . . sM . . . 0             −

  • v1

− −

  • v2

− . . . . . . . . . −

  • vN

− ⊲ s1 > s2 > s3 > ... > sM ⊲ U holds the eigenvectors of AAT ⊲ V holds the eigenvectors of ATA

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-27
SLIDE 27

Singular Value Decomposition

A = USV T A =     | | . . . |

  • u1
  • u2

. . .

  • uM

| | . . . |            s1 . . . s2 . . . s3 . . . . . . . . . . . . . . . . . . . . . sM . . . 0             −

  • v1

− −

  • v2

− . . . . . . . . . −

  • vN

− ⊲ s1 > s2 > s3 > ... > sM ⊲ U holds the eigenvectors of AAT ⊲ V holds the eigenvectors of ATA ⊲ Each s is the square root of corresponding eigenvalue of AAT and ATA (they’re the same!)

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-28
SLIDE 28

Singular Value Decomposition: Example

Ax V x

T

SV x

T

USV x

T COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-29
SLIDE 29

Singular Value Decomposition: Example

Ax V x

T

SV x

T

USV x

T COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-30
SLIDE 30

Singular Value Decomposition: Example

Ax V x

T

SV x

T

USV x

T COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-31
SLIDE 31

Singular Value Decomposition → PCA

A = USV T ⊲ s1 > s2 > s3 > ... > sM ⊲ U holds the eigenvectors of AAT ⊲ V holds the eigenvectors of ATA ⊲ Each s is the square root of corresponding eigenvalue of AAT and ATA Let X be a 3 × N matrix of points along columns. Can we use SVD(X) to do PCA?

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-32
SLIDE 32

Singular Value Decomposition → PCA

X = USV T ⊲ Columns of U give principal components ⊲ Squares of corresponding S gives sum of squared magnitudes along directions of U ⊲ Coordinates along U directions?

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-33
SLIDE 33

Table of Contents

⊲ Ray Tracing Special Case ⊲ PCA Review ⊲ Singular Value Decomposition ◮ Procrustes Distance ⊲ Final Projects

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-34
SLIDE 34

Procrustes Distance

http://www.procrustes.nl/gif/illustr.gif

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-35
SLIDE 35

Procrustes Alignment

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-36
SLIDE 36

Procrustes Distance

Given two point clouds { xi}N

i=1 and {

yi}N

i=1

where xi and yi are in correspondence Seek to minimize

N

  • i=1

||R( xi + t) − yi||2

2

  • ver all orthogonal matrices R and translation vectors t. ||.||2

2 is

squared distance

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-37
SLIDE 37

Translation

N

  • i=1

||R( xi + t) − yi||2

2

Translation is easy! Align centroids

  • t = 1

N N

  • i=1
  • yi −

N

  • i=1
  • xi
  • COMPSCI/MATH 290-04

Lecture 12: SVD, Procrustes Analysis

slide-38
SLIDE 38

Rotating to Align Points: Dot Product Perspective

  • x1 ·

y1 = || x1|||| y1|| cos(θ1)

x1 y1

θ

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-39
SLIDE 39

Rotating to Align Points: Dot Product Perspective

  • x1 ·

y1 + x2 · y2 = || x1|||| y1|| cos(θ1) + || x2|||| y2|| cos(θ2)

x1 x2 y1 y2

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-40
SLIDE 40

Rotating to Align Points: Dot Product Perspective

  • x1 ·

y1 + x2 · y2 = || x1|||| y1|| cos(θ1) + || x2|||| y2|| cos(θ2)

x1 x2 y1 y2

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-41
SLIDE 41

Rotating to Align Points: Dot Product Perspective

  • x1 ·

y1 + x2 · y2 = || x1|||| y1|| cos(θ1) + || x2|||| y2|| cos(θ2)

x1 x2 y1 y2

⊲ Why should points further away from origin get more weight?

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-42
SLIDE 42

Rotating to Align Points: Dot Product Perspective

In general, how to maximize?

N

  • i=1

Rx xi · Ry yi

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-43
SLIDE 43

Rotating to Align Points: Dot Product Perspective

In general, how to maximize?

N

  • i=1

Rx xi · Ry yi VIDEO EXAMPLE

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-44
SLIDE 44

Maximizing Dot Product: First Component

Choose first row of Rx, which is a projection of blue points. Call this row a1 (write as a row vector) a1X

a1

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-45
SLIDE 45

Maximizing Dot Product: First Component

Choose first row of Ry, which is a projection of red points. Call this row b1 (write as row vector) b1Y

a1 b1

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-46
SLIDE 46

Maximizing Dot Product: First Component

a1X, b1Y How to write N

i=1(

a1 · xi)( b1 · yi) in matrix form?

a1 b1

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-47
SLIDE 47

Maximizing Dot Product: First Component

a1X, b1Y How to write N

i=1(

a1 · xi)( b1 · yi) in matrix form? (a1X)(b1Y)T = a1XY TbT

1 a1 b1

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-48
SLIDE 48

Maximizing Dot Product: First Component

How to find u1 and v1 that maximize this product? (a1X)(b1Y)T = a1XY TbT

1

Take SVD: XY T = USV T and substitute in a1USV TbT

1

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis

slide-49
SLIDE 49

Continued next time...

COMPSCI/MATH 290-04 Lecture 12: SVD, Procrustes Analysis