Video Manifolds Selen Atasoy MICCAI 2011 Tutorial Image Spaces - - PowerPoint PPT Presentation

▶
video manifolds
SMART_READER_LITE
LIVE PREVIEW

Video Manifolds Selen Atasoy MICCAI 2011 Tutorial Image Spaces - - PowerPoint PPT Presentation

Mar 09, 2024 449 likes •773 views

Image Similarities for Learning Video Manifolds Selen Atasoy MICCAI 2011 Tutorial Image Spaces Image Manifolds Tenenbaum2000 Roweis2000 Tenenbaum2000 [ Tenenbaum2000: J. B. Tenenbaum, V. Silva, J. C. Langford : A global geometric framework for

slide-1
SLIDE 1

Image Similarities for Learning Video Manifolds

Selen Atasoy MICCAI 2011 Tutorial

slide-2
SLIDE 2

Image Spaces

slide-3
SLIDE 3

Roweis2000 Tenenbaum2000 Tenenbaum2000

Image Manifolds

[Tenenbaum2000: J. B. Tenenbaum, V. Silva, J. C. Langford: A global geometric framework for nonlinear dimensionality reduction. Science, 290(5500), 2000.] [Roweis2000: S. T. Roweis, L. K. Saul: Nonlinear dimensionality reduction by locally linear

  • embedding. Science, 290(5500), 2000]
slide-4
SLIDE 4

Atasoy2010 Pless2003

Video Manifolds

[Pless2003: R. Pless: Using Isomap to Explore Video Sequences: ICCV, 2003.] [Atasoy2010: S. Atasoy, D. Mateus, J. Lallemand, A. Meining, G.Z. Yang, N. Navab: Endoscopic Video Manifolds, MICCAI, 2010.] [Atasoy2011: S. Atasoy, D. Mateus, A. Meining, G.Z. Yang, N. Navab: Targeted Optical Biopsies for Surveillance Endoscopies, MICCAI, 2011.]

slide-5
SLIDE 5

Theoretical Background

slide-6
SLIDE 6

Manifold Learning

  • High dimensional data points

lying on or near a manifold

  • Low dimensional representation
  • Find a mapping

that best preserves ... ???

slide-7
SLIDE 7

Manifold Learning

  • 1. Define a matrix based on the

relations between data points

  • 2. Compute the eigenvectors &

eigenvalues

  • 3. Embed each sample

A General Recipe

slide-8
SLIDE 8

Manifold Learning A General Recipe

Method Operator/Matrix Preserved Objective Function PCA Covariance matrix Variance of the dataset / Euclidean distances between data points Laplacian Eigenmaps Graph Laplacian Distances within the local neighbourhood

  • f each data point

ISOMAP Geodesic distance matrix Geodesic distances between data points LLE Reconstruction weights Reconstruction weights within the local neighbourhood

  • f each data point
slide-9
SLIDE 9

Manifold Learning

  • Rayleigh-Ritz Theorem:
  • Recall:

– Scalar product: – Scalar product in H: – Norm: – Norm in H:

Why does it work?

eigenvalues eigenvectors

slide-10
SLIDE 10

Discrete Domain

  • vectors
  • Continuous Domain
  • functions
  • Manifold Learning

Why does it work?

Schwarz’s Kernel Theorem: Each linear operator is given as an integration against a unique kernel. That kernel is the impulse response of the linear system to an impulse (a delta function).

slide-11
SLIDE 11

Discrete Domain

  • vectors
  • Continuous Domain
  • functions
  • Manifold Learning

Why does it work?

slide-12
SLIDE 12

Discrete Domain

  • vectors
  • Continuous Domain
  • functions
  • Manifold Learning

Why does it work?

The matrix H defines:

  • which operator is applied
  • which (Hilbert) space we are working in
  • which quantity will be conserved
slide-13
SLIDE 13

Laplacian Eigenmaps

slide-14
SLIDE 14

Manifold Learning

  • Solve
  • Find the eigenvectors of the graph Laplacian
  • Equivalent to solving the Helmholtz Equation

Laplacian Eigenmaps

[Belkin2003: M. Belkin, P. Niyogi: Laplacian eigenmaps for dimensionality reduction and data

  • representation. Neural computation, 15(6), 1373-1396. MIT Press, 2003]
slide-15
SLIDE 15

[Levy2010] [Levy2010]

Manifold Learning Laplacian Eigenmaps - Interpretation

[Chladni1787: E. Chladni: Discoveries in the Theory of Sound, 1787.] [Levy2010: B. Levy: Spectral Geometry Processing: ACM SIGGRAPH Course Notes, 2010.]

[Chladni1787]

slide-16
SLIDE 16

Non-linear Manifold Learning Laplacian Eigenmaps - Interpretation

  • Manifold learning as bending, stretching without cutting or creating wholes
  • Vibrational modes are preserved while bending the manifold
slide-17
SLIDE 17

Endoscopic Video Manifolds (EVMs)

slide-18
SLIDE 18

Endoscopic Video Manifolds

  • Clustering Uninformative Frames

Challenges

slide-19
SLIDE 19

Endoscopic Video Manifolds Clustering Uninformative Frames

slide-20
SLIDE 20

Endoscopic Video Manifolds Clustering Uninformative Frames

Informative frame & power spectrum Uninformative frame & power spectrum

[Atasoy2010: S. Atasoy, D. Mateus, J. Lallemand, A. Meining, G.Z. Yang, N. Navab: Endoscopic Video Manifolds, MICCAI, 2010.]

Uninformative frame Informative frame

slide-21
SLIDE 21

Endoscopic Video Manifolds Clustering Uninformative Frames

[Atasoy2010: S. Atasoy, D. Mateus, J. Lallemand, A. Meining, G.Z. Yang, N. Navab: Endoscopic Video Manifolds, MICCAI, 2010.]

slide-22
SLIDE 22

Endoscopic Video Manifolds

  • Significant change in

endoscope viewpoint

  • Small overlap between

frames showing the same scene

  • Scenes do not necessarily

contain distinctive features

Challenges

slide-23
SLIDE 23

Endoscopic Video Manifolds

Cluster 1

338 frames

Cluster 2

150 frames

Cluster 3

137 frames

Cluster 4

102 frames

Cluster 5

98 frames

Cluster 6

78 frames

Cluster 7

71 frames

Cluster 8

71 frames

Cluster 9

44 frames

Cluster 10

38 frames

  • 0.06
  • 0.04
  • 0.02

0.02 0.04 0.06 0.08

  • 0.1
  • 0.05

0.05 0.1

  • 0.12
  • 0.1
  • 0.08
  • 0.06
  • 0.04
  • 0.02

0.02 0.04

Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Cluster 6 Cluster 7 Cluster 8 Cluster 9 Cluster 10

Clustering Endoscopic Scenes – Euclidean Distance

[Belkin2003: M. Belkin, P. Niyogi: Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15(6), 1373-1396. MIT Press, 2003] [Atasoy2010: S. Atasoy, D. Mateus, J. Lallemand, A. Meining, G.Z. Yang, N. Navab: Endoscopic Video Manifolds, MICCAI, 2010.]

slide-24
SLIDE 24

Endoscopic Video Manifolds Clustering Endoscopic Scenes – Euclidean Distances

  • 0.06
  • 0.04
  • 0.02

0.02 0.04 0.06 0.08

  • 0.1
  • 0.05

0.05 0.1

  • 0.12
  • 0.1
  • 0.08
  • 0.06
  • 0.04
  • 0.02

0.02 0.04

Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Cluster 6 Cluster 7 Cluster 8 Cluster 9 Cluster 10

Cluster 3

slide-25
SLIDE 25

Endoscopic Video Manifolds Clustering Endoscopic Scenes - NCC

Cluster 1

389 frames

Cluster 2

137 frames

Cluster 3

103 frames

Cluster 4

98 frames

Cluster 5

85 frames

Cluster 6

82 frames

Cluster 7

81 frames

Cluster 8

64 frames

Cluster 9

44 frames

Cluster 10

44 frames

  • 0.08
  • 0.06
  • 0.04
  • 0.02

0.02 0.04

  • 0.1
  • 0.05

0.05 0.1

  • 0.12
  • 0.1
  • 0.08
  • 0.06
  • 0.04
  • 0.02

0.02 0.04

Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Cluster 6 Cluster 7 Cluster 8 Cluster 9 Cluster 10

[Atasoy2010: S. Atasoy, D. Mateus, J. Lallemand, A. Meining, G.Z. Yang, N. Navab: Endoscopic Video Manifolds, MICCAI, 2010.]

slide-26
SLIDE 26

Endoscopic Video Manifolds Clustering Endoscopic Scenes - NCC

Euclidean Distance Normalized Cross Correlation

[Atasoy2010: S. Atasoy, D. Mateus, J. Lallemand, A. Meining, G.Z. Yang, N. Navab: Endoscopic Video Manifolds, MICCAI, 2010.]

slide-27
SLIDE 27

Endoscopic Video Manifolds Clustering Endoscopic Scenes - NCC

slide-28
SLIDE 28

Endoscopic Video Manifolds

  • Change the adjacency matrix to include temporal

constraints

Clustering Endoscopic Scenes with Temporal Constraints

[Atasoy2011: S. Atasoy, D. Mateus, A. Meining, G.Z. Yang, N. Navab: Targeted Optical Biopsies for Surveillance Endoscopies, MICCAI, 2011]

slide-29
SLIDE 29

Endoscopic Video Manifolds

Clustering Endoscopic Scenes with Temporal Constraints

  • 0.04
  • 0.02

0.02 0.04 0.06 0.08 0.1

  • 0.05

0.05 0.1 0.15

  • 0.1
  • 0.08
  • 0.06
  • 0.04
  • 0.02

0.02 0.04

Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Cluster 6 Cluster 7 Cluster 8 Cluster 9 Cluster 10

Cluster 1

344 frames

Cluster 2

143 frames

Cluster 3

126 frames

Cluster 4

120 frames

Cluster 5

112 frames

Cluster 6

88 frames

Cluster 7

55 frames

Cluster 8

53 frames

Cluster 9

43 frames

Cluster 10

43 frames

[Atasoy2011: S. Atasoy, D. Mateus, A. Meining, G.Z. Yang, N. Navab: Targeted Optical Biopsies for Surveillance Endoscopies, MICCAI, 2011]

slide-30
SLIDE 30
slide-31
SLIDE 31

Acknowledgements

  • Prof. Nassir

Navab

  • Prof. Guang-Zhong

Yang

  • Prof. Alexander

Meining

  • Dr. Diana

Mateus

Thank you for your attention!!!