Deformable Models A powerful, model- A powerful, model -based - - PDF document

▶
deformable models
SMART_READER_LITE
LIVE PREVIEW

Deformable Models A powerful, model- A powerful, model -based - - PDF document

Feb 14, 2024 321 likes •971 views

1. Deformable and Functional Models In Medical Image Analysis 2. A Tensor Algebraic Framework for Image Synthesis, Analysis & Recognition Demetri Terzopoulos Terzopoulos Demetri Computer Science Department Computer Science Department

slide-1
SLIDE 1
  • 1. Deformable and Functional Models

In Medical Image Analysis

Demetri Demetri Terzopoulos Terzopoulos

Computer Science Department Computer Science Department University of California, Los Angeles University of California, Los Angeles

  • 2. A Tensor Algebraic Framework for

Image Synthesis, Analysis & Recognition

Deformable Models

A powerful, model-based medical image analysis approach

  • Proposed in computer vision and graphics
  • Actively explored in medical image analysis
  • Combine bottom-up and top-down analysis
  • Accommodate shape & motion constraints/variability
  • Incorporate a priori anatomical knowledge
  • Support intuitive interaction mechanisms

A powerful, model A powerful, model-

  • based medical image

based medical image analysis approach analysis approach

  • Proposed in computer vision and graphics

Proposed in computer vision and graphics

  • Actively explored in medical image analysis

Actively explored in medical image analysis

  • Combine bottom

Combine bottom-

  • up and top

up and top-

  • down analysis

down analysis

  • Accommodate shape & motion constraints/variability

Accommodate shape & motion constraints/variability

  • Incorporate a priori anatomical knowledge

Incorporate a priori anatomical knowledge

  • Support intuitive interaction mechanisms

Support intuitive interaction mechanisms

slide-2
SLIDE 2

Computing Visible Surfaces from Scattered Visual Data

[Terzopoulos, 1984]

Thin-plate spline under tension Thin Thin-

  • plate

plate spline spline under tension under tension

y y

{ }

k k k k

c z y x , ), , (

{ }

k k k k

c z y x , ), , (

k

zk z

) , (

k k y

x ) , (

k k y

x

) , ( y x f ) , ( y x f z z x x

data point:

2

] ) , ( [ 2 1 ∑ − =

k k k k k d

y x f z c E

2

] ) , ( [ 2 1 ∑ − =

k k k k k d

y x f z c E

( ) ( ) [ ]

∫∫

+ + − + + = y d x d f f f f f f E

yy xy xx y x 2 2 2 2 2

2 ) 1 ( 2 1 ) ( τ τ ρ (

) ( ) [ ]

∫∫

+ + − + + = y d x d f f f f f f E

yy xy xx y x 2 2 2 2 2

2 ) 1 ( 2 1 ) ( τ τ ρ

k

ck c

Discontinuity-Preserving Surface Reconstruction

Make Make “ “rigidity rigidity” ” & & “ “tension tension” ” functions of (x,y) functions of (x,y)

  • Tangent discontinuities:

Tangent discontinuities:

  • Jump discontinuities:

Jump discontinuities:

1 ) , ( = y x τ 1 ) , ( = y x τ ) , ( = y x ρ ) , ( = y x ρ

slide-3
SLIDE 3
  • Curve representation:
  • Curve deformation energy:
  • Equations of motion:
  • Curve representation:

Curve representation:

  • Curve deformation energy:

Curve deformation energy:

  • Equations of motion:

Equations of motion:

Snakes: Active Contours

[Kass, Witkin, Terzopoulos, 1987]

∫

∂ ∂ + ∂ ∂ =

1 2 2 2 2 2 1

2 1 ) ( u d u w u w E c c c

∫

∂ ∂ + ∂ ∂ =

1 2 2 2 2 2 1

2 1 ) ( u d u w u w E c c c

] 1 , [ ; ) , ( ) , ( ) , ( ∈ ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ = u t u y t u x t u c ] 1 , [ ; ) , ( ) , ( ) , ( ∈ ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ = u t u y t u x t u c

f c c c c = ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ∂ ∂ ∂ ∂ + ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ ∂ ∂ ∂ ∂ − +

2 2 2 2 2 1

u w u u w u & & & γ µ f c c c c = ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ∂ ∂ ∂ ∂ + ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ ∂ ∂ ∂ ∂ − +

2 2 2 2 2 1

u w u u w u & & & γ µ f c c c

c

= + + ) ( E δ γ µ & & & f c c c

c

= + + ) ( E δ γ µ & & &

Image Analysis Using Snakes

External forces come from an image

  • Image potential:

External forces come from an image External forces come from an image

  • Image potential:

Image potential:

) (c f P −∇ = ) (c f P −∇ =

) , ( y x P ) , ( y x P

slide-4
SLIDE 4

Motion Tracking in Video

Time-varying image potential Time Time-

  • varying image potential

varying image potential

) , , ( t y x P ) , , ( t y x P

Snake-Based Tracking

(Blake & Isard, Oxford University) (Blake & (Blake & Isard Isard, Oxford University) , Oxford University)

slide-5
SLIDE 5

Discretization

  • Continuous equations of motion
  • Discrete equations of motion
  • Continuous equations of motion

Continuous equations of motion

  • Discrete equations of motion

Discrete equations of motion

f Kc c D c M = + + & & & f Kc c D c M = + + & & &

Mass matrix Damping matrix External forces Stiffness matrix

f c c c c = ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ∂ ∂ ∂ ∂ + ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ ∂ ∂ ∂ ∂ − +

2 2 2 2 2 1

u w u u w u & & & γ µ f c c c c = ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ∂ ∂ ∂ ∂ + ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ ∂ ∂ ∂ ∂ − +

2 2 2 2 2 1

u w u u w u & & & γ µ

Snake Stiffness Matrix

Finite differences: Finite differences: Finite differences:

4 1 ; 2 4 1 ; 2 ; 2 2 ; 1 4 1 ; 2 ; 2 1 ; 2 2 ; 1 1 ; 1 1 2 3 1 1 2 2 3 4 2 3 3 3 4 5 3 3 3 2 1 2 2 2 1 1 1 1 1 1 2

2 2 4 h w c h w w h w b h w w w h w w a a b c c b b a b c c c b a b c c b a b c c b a b c c c b a b b c c b a

i i i i i i i i i i i i N N N N N N N N N N N N N N N N N N + + + − − − − − − − − − − − − − − − − − − − −

= + − − = + + + + = ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ = O O O O O K

4 1 ; 2 4 1 ; 2 ; 2 2 ; 1 4 1 ; 2 ; 2 1 ; 2 2 ; 1 1 ; 1 1 2 3 1 1 2 2 3 4 2 3 3 3 4 5 3 3 3 2 1 2 2 2 1 1 1 1 1 1 2

2 2 4 h w c h w w h w b h w w w h w w a a b c c b b a b c c c b a b c c b a b c c b a b c c c b a b b c c b a

i i i i i i i i i i i i N N N N N N N N N N N N N N N N N N + + + − − − − − − − − − − − − − − − − − − − −

= + − − = + + + + = ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ = O O O O O K

2 1 1 2 2 1

2 1 , , ); ( h u h u N i ih

i i i i i i − + +

+ − ≈ ∂ ∂ − ≈ ∂ ∂ − = = c c c c c c c c c L

2 1 1 2 2 1

2 1 , , ); ( h u h u N i ih

i i i i i i − + +

+ − ≈ ∂ ∂ − ≈ ∂ ∂ − = = c c c c c c c c c L ) ( ) (

2 ; 2 1 ; 1

ih w w ih w w

i i

= = ) ( ) (

2 ; 2 1 ; 1

ih w w ih w w

i i

= =

slide-6
SLIDE 6

Stable, Implicit Euler Time-Integration Method

Solve linear system at each time step

  • Efficient skyline storage of
  • LU factorization of
  • Forward / Back substitution solves for

Solve linear system at each time step Solve linear system at each time step

  • Efficient skyline storage of

Efficient skyline storage of

  • LU factorization of

LU factorization of

  • Forward / Back substitution solves for

Forward / Back substitution solves for

) ( ) ( ) ( ) ( ) ( ) ( ) ( t t t t t t t t t t

dt c c c g c c A + = + =

+

  • +
  • +
  • δ

δ δ

{

A( )

t

A( )

t

) ( t t δ +

  • c
  • Surface representation:
  • Surface deformation energy:
  • Equations of motion:
  • Surface representation:

Surface representation:

  • Surface deformation energy:

Surface deformation energy:

  • Equations of motion:

Equations of motion:

Deformable Surfaces

[Terzopoulos, 1986; Terzopoulos, Witkin, Kass, 1987]

∫ ∫

∂ ∂ + ∂ ∂ ∂ + ∂ ∂ + ∂ ∂ + ∂ ∂ =

1 1 2 2 2 02 2 2 11 2 2 2 20 2 01 2 10

2 2 1 ) ( dv u d v w v u w u w v w u w E s s s s s s

∫ ∫

∂ ∂ + ∂ ∂ ∂ + ∂ ∂ + ∂ ∂ + ∂ ∂ =

1 1 2 2 2 02 2 2 11 2 2 2 20 2 01 2 10

2 2 1 ) ( dv u d v w v u w u w v w u w E s s s s s s ] 1 , [ , ; ) , , ( ) , , ( ) , , ( ) , , ( ∈ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎣ ⎡ = v u t v u z t v u y t v u x t v u s ] 1 , [ , ; ) , , ( ) , , ( ) , , ( ) , , ( ∈ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎣ ⎡ = v u t v u z t v u y t v u x t v u s

f s s s

s

= + + ) ( E δ γ µ & & & f s s s

s

= + + ) ( E δ γ µ & & &

f s s s s s s s = ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ∂ ∂ ∂ ∂ + ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ∂ ∂ ∂ ∂ ∂ ∂ + ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ∂ ∂ ∂ ∂ + ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ ∂ ∂ ∂ ∂ − ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ ∂ ∂ ∂ ∂ − +

2 2 02 2 2 2 11 2 2 2 20 2 2 01 10

2 v w v v u w v u u w u v w v u w u & & & γ µ f s s s s s s s = ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ∂ ∂ ∂ ∂ + ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ∂ ∂ ∂ ∂ ∂ ∂ + ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ∂ ∂ ∂ ∂ + ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ ∂ ∂ ∂ ∂ − ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ ∂ ∂ ∂ ∂ − +

2 2 02 2 2 2 11 2 2 2 20 2 2 01 10

2 v w v v u w v u u w u v w v u w u & & & γ µ

slide-7
SLIDE 7

Deformable Model Reconstruction Reconstruction

3D to 2D projection 3D to 2D projection 3D to 2D projection

Viewpoint Image 3D model

) ( s f Π −∇ = P ) ( s f Π −∇ = P

slide-8
SLIDE 8

Graphics / Vision

Converse problems Converse problems Converse problems

CV “Cooking with Kurt” (1987) Image Reconstructed 3D Scene

Vegetable Model Animation

slide-9
SLIDE 9

Medical Image Analysis Tasks

  • Segmentation
  • Shape modeling
  • Matching
  • Motion recovery and analysis
  • Functional modeling
  • Segmentation

Segmentation

  • Shape modeling

Shape modeling

  • Matching

Matching

  • Motion recovery and analysis

Motion recovery and analysis

  • Functional modeling

Functional modeling

Interactive Medical Image Segmentation using Snakes

[Carlbom, Terzopoulos, Harris, 1994]

EM neuronal tissue sections EM neuronal tissue sections EM neuronal tissue sections

slide-10
SLIDE 10

Reconstruction of Neuronal Dendrite

Cell interiors stacked in 3D Cell interiors stacked in 3D Cell interiors stacked in 3D

Visualization of Dendrite

Ray-traced interpolated volume Ray Ray-

  • traced interpolated volume

traced interpolated volume

slide-11
SLIDE 11

Family of Snakes

Many snakes variants Many snakes variants Many snakes variants

Livewire

Hermite Snakes Fourier Descriptor Snakes Wavelet Snakes B-snakes Active Shape Models Kalman Snakes Condesention Snakes Topologically Adaptive Snakes Finite Element Snakes Discrete Snakes DP Optimal Boundary Tracking

Snakes (Kass, Witkin, Terzopoulos, 1987) Dynamic Programming Snakes

  • “Livewire” or “Intelligent Scissors”:

An interactive boundary tracing tool

  • “

“Livewire Livewire” ” or

  • r “

“Intelligent Scissors Intelligent Scissors” ”: : An interactive boundary tracing tool An interactive boundary tracing tool

Level Set Snakes Level-Set Snakes

Livewire Demo

slide-12
SLIDE 12

Limitations of Livewire

  • No control of trace between

seed points; only backtracking

  • Many seed points needed for

complex boundaries

  • Nearby strong edges can

capture trace (on-the-fly training)

  • Fundamentally image-based

– cannot bridge gaps – smoothness not guaranteed

  • No control of trace between

No control of trace between seed points; only backtracking seed points; only backtracking

  • Many seed points needed for

Many seed points needed for complex boundaries complex boundaries

  • Nearby strong edges can

Nearby strong edges can capture trace capture trace (on

(on-

  • the

the-

  • fly training)

fly training)

  • Fundamentally image

Fundamentally image-

  • based

based

– – cannot bridge gaps cannot bridge gaps – – smoothness not guaranteed smoothness not guaranteed

Combining Snakes and Livewire

“United Snakes”

  • Livewire serves for quick initialization of snakes

– typically requires fewer seed points

  • Livewire-initialized snakes quickly lock on boundaries
  • Snakes enable adjustment of traces between seeds

– snake provides subpixel accuracy

  • Snake energy imposes smoothness and bridges gaps
  • Livewire seed points capture user’s knowledge

– can serve as hard or soft constraints on snake

“ “United Snakes United Snakes” ”

  • Livewire serves for quick initialization of snakes

Livewire serves for quick initialization of snakes – – typically requires fewer seed points typically requires fewer seed points

  • Livewire

Livewire-

  • initialized snakes quickly lock on boundaries

initialized snakes quickly lock on boundaries

  • Snakes enable adjustment of traces between seeds

Snakes enable adjustment of traces between seeds – – snake provides snake provides subpixel subpixel accuracy accuracy

  • Snake energy imposes smoothness and bridges gaps

Snake energy imposes smoothness and bridges gaps

  • Livewire seed points capture user

Livewire seed points capture user’ ’s knowledge s knowledge – – can serve as hard or soft constraints on snake can serve as hard or soft constraints on snake

slide-13
SLIDE 13

Combining Snakes and Livewire

“United Snakes” accrue benefits of both “ “United Snakes United Snakes” ” accrue benefits of both accrue benefits of both

Dynamic Chest Image Analysis

slide-14
SLIDE 14

Vessel Segmentation Vessel Segmentation

slide-15
SLIDE 15

Topologically Adaptive Snakes

(McInerney & Terzopoulos, 1996)

Segmenting Retinal Angiogram

  • T-snake flows and bifurcates

Segmenting Retinal Angiogram Segmenting Retinal Angiogram

  • T

T-

  • snake flows and bifurcates

snake flows and bifurcates

Initial Model Flow Segmented Angiogram

Retinal Angiogram Segmentation

slide-16
SLIDE 16

Affine Cell Image Decomposition

ACID makes snakes topologically flexible

  • ACID grid continually reparameterizes snake

ACID makes snakes topologically flexible ACID makes snakes topologically flexible

  • ACID grid continually

ACID grid continually reparameterizes reparameterizes snake snake

T-Snake Segmentation of Brain Image

slide-17
SLIDE 17

Shrink-Wrap Segmentation Vertebra Reconstruction

slide-18
SLIDE 18

Complex Structure Extraction

Cerebral Vasculature Lung Brain

T-Surface Segmentation of Cortex

[McInerney & Terzopoulos, 1997]

slide-19
SLIDE 19

Tongue Tracking in Ultrasound

[Kambhamettu et al, 1999]

Reconstructed LV

LV Reconstruction

Deformable Balloon in Processed DSR Data

slide-20
SLIDE 20

Cardiac LV Motion Tracking Systolic/Diastolic LV

Computing ejection fraction Computing ejection fraction Computing ejection fraction

slide-21
SLIDE 21

Functional Model of the Heart

[Peskin & McQueen]

Artificial Humans Scanned Data Synthetic Faces

Cyberware Data Synthesized Expressions Range Image Texture Image

slide-22
SLIDE 22

Processed range image RGB texture image

Fitting the Generic Mesh

Feature-based image matching algorithm

localizes facial features in:

Feature Feature-

  • based image matching algorithm

based image matching algorithm

localizes facial localizes facial features in: features in:

Sampling Facial Shape

Fitted mesh nodes sample range data Fitted mesh nodes sample range data Fitted mesh nodes sample range data

slide-23
SLIDE 23

Textured 3D Geometric Model

Texture map coordinates

  • Positions of fitted

mesh nodes in RGB texture image

Texture map Texture map coordinates coordinates

  • Positions of fitted

Positions of fitted mesh nodes in RGB mesh nodes in RGB texture image texture image

Auxiliary Geometric Models

Eyelid Texture Interpolation

slide-24
SLIDE 24

Complete Geometric Model

Neutral expression is estimated Neutral expression Neutral expression is estimated is estimated

Epidermis Dermis Muscle Layer

Facial Anatomy

Skin Model Muscle Model

slide-25
SLIDE 25

35 Muscles

  • Levator Oculii
  • Corrugators
  • Naso-Labial
  • Zygomatics
  • Obicularis Oris

plus

  • Articulate Jaw
  • Eyes/Eyelids

35 Muscles 35 Muscles

  • Levator

Levator Oculii Oculii

  • Corrugators

Corrugators

  • Naso

Naso-

  • Labial

Labial

  • Zygomatics

Zygomatics

  • Obicularis

Obicularis Oris Oris

plus plus

  • Articulate Jaw

Articulate Jaw

  • Eyes/Eyelids

Eyes/Eyelids

Facial Muscle Model Structure Synthetic Face Animation

slide-26
SLIDE 26

Real-Time Facial Simulation Incision on Facial Mesh

slide-27
SLIDE 27

Retriangulation Around Incision Maxillo Surgery

slide-28
SLIDE 28

Craniofacial Surgery

[Gladalin, 2002]

Anatomical Structure of the Neck

slide-29
SLIDE 29

Biomechanical Modeling

What would Leonardo da Vinci Think of This?

slide-30
SLIDE 30

Demo: Gaze Behavior Demo: Autonomous Multi-Head Interaction

slide-31
SLIDE 31

Geometry

Artificial Life Modeling

From physics to intelligence From physics to intelligence From physics to intelligence

Physics Physics Perception Perception Behavior Behavior Cognition Cognition

Artificial Fishes

slide-32
SLIDE 32

Deformable Organisms

[Hamarneh, McInerney, Terzopoulos, 2001]

Corpus Callosum Organism Corpus Corpus Callosum Callosum Organism Organism

Memory and prior knowledge Plan or schedule Sensors Skeleton Muscles and limbs Interactions with

  • ther creatures

Underlying medial based Shape representation Muscle actuation causes shape deformation Brain Perception Perceptual attention mechanism Memory and prior knowledge Plan or schedule Sensors Skeleton Muscles and limbs Interactions with

  • ther creatures

Underlying medial based Shape representation Muscle actuation causes shape deformation Brain Perception Perceptual attention mechanism fornix 2 3 N-2 1 genu splenium rostrum body N N-1 upper/right lower/left fornix 2 3 N-2 1 genu splenium rostrum body N N-1 upper/right lower/left

Deformable Organisms

slide-33
SLIDE 33

Deformable Organisms Conclusion

Deformable models

  • Powerful technique for extracting geometric models
  • f anatomical structures
  • Functional models
  • Development continues

Deformable models Deformable models

  • Powerful technique for extracting geometric models

Powerful technique for extracting geometric models

  • f anatomical structures
  • f anatomical structures
  • Functional models

Functional models

  • Development continues

Development continues

“ “Deformable Models in Medical Image Analysis: Deformable Models in Medical Image Analysis: A Survey A Survey” ”, , Medical Image Analysis Medical Image Analysis, , 1 1(2), 1997 (2), 1997 See deformable.com See deformable.com

slide-34
SLIDE 34

A Tensor Algebraic Framework for Image Synthesis, Analysis & Recognition

  • M. Alex O. Vasilescu

MIT Media Laboratory

Demetri Terzopoulos

University of California, Los Angeles

  • M. Alex O.
  • M. Alex O. Vasilescu

Vasilescu

MIT Media Laboratory MIT Media Laboratory

Demetri Demetri Terzopoulos Terzopoulos

University of California, Los Angeles University of California, Los Angeles

Why is Face Recognition Difficult?

Viewpoint changes Viewpoint changes Viewpoint changes

slide-35
SLIDE 35

Illumination Changes Illumination Changes Illumination Changes

Why is Face Recognition Difficult? Appearance-Based Recognition

Recognition of 3D objects (faces) directly from their appearance in ordinary images

  • PCA / Eigenimages:

–[Sirovich & Kirby 1987]

"Low Dimensional Procedure for the Characterization of Human Faces"

–[Turk & Pentland 1991]

"Face Recognition Using Eigenfaces"

–[Murase & Nayar 1995]

"Visual learning and recognition of 3D objects from appearance"

Recognition of 3D objects (faces) directly from Recognition of 3D objects (faces) directly from their appearance in ordinary images their appearance in ordinary images

  • PCA /

PCA / Eigenimages Eigenimages: : – – [ [Sirovich Sirovich & Kirby 1987] & Kirby 1987]

"Low Dimensional Procedure for the Characterization of Human "Low Dimensional Procedure for the Characterization of Human Faces" Faces"

– – [Turk & [Turk & Pentland Pentland 1991] 1991]

"Face Recognition Using "Face Recognition Using Eigenfaces Eigenfaces" "

– – [ [Murase Murase & & Nayar Nayar 1995] 1995]

"Visual learning and recognition of 3D objects from appearance" "Visual learning and recognition of 3D objects from appearance"

slide-36
SLIDE 36

Linear Algebra

The algebra of vectors and matrices

  • Traditionally of great value in image science

– Fourier transform – Karhunen-Loeve transform

  • Linear methods (PCA, FLD, ICA) model:

– Linear operators over a vector space – Single-factor variation in image formation – The linear combination of multiple sources

The algebra of vectors and matrices The algebra of vectors and matrices

  • Traditionally of great value in image science

Traditionally of great value in image science – – Fourier transform Fourier transform – – Karhunen Karhunen-

  • Loeve

Loeve transform transform

  • Linear methods (PCA, FLD, ICA) model:

Linear methods (PCA, FLD, ICA) model: – – Linear operators over a vector space Linear operators over a vector space – – Single Single-

  • factor variation in image formation

factor variation in image formation – – The linear combination of multiple sources The linear combination of multiple sources

Multilinear Algebra

The algebra of higher-order (>2) tensors

  • Natural images result from the interaction of multiple factors

related to – scene geometry – Illumination – Imaging

  • Multilinear algebra can explicitly represent multifactor variation

– Multilinear operators over a set of vector spaces

  • Multilinear algebra subsumes linear algebra as a special case
  • A unifying mathematical framework

The algebra of higher The algebra of higher-

  • order (>2) tensors
  • rder (>2) tensors
  • Natural images result from the interaction of multiple factors

Natural images result from the interaction of multiple factors related to related to – – scene geometry scene geometry – – Illumination Illumination – – Imaging Imaging

  • Multilinear

Multilinear algebra can explicitly represent multifactor variation algebra can explicitly represent multifactor variation – – Multilinear Multilinear operators over a

  • perators over a set

set of vector spaces

  • f vector spaces
  • Multilinear

Multilinear algebra subsumes linear algebra as a special case algebra subsumes linear algebra as a special case

  • A unifying mathematical framework

A unifying mathematical framework

slide-37
SLIDE 37

1 ×

ℜ ∈

kl

i

An image is a point in dimensional space An image is a point in dimensional space An image is a point in dimensional space

Images

1

× ℜ kl

l k

I

×

ℜ ∈

pixel 1 pixel kl pixel 2 255 255 255

... . . . .. . .. . . . . .. . ... . . . . .. .

Principal components (eigenvectors) of image ensemble Principal components (eigenvectors) of image Principal components (eigenvectors) of image ensemble ensemble

Eigenimages

pixel 1 pixel kl pixel 2 255 255 255

... . . . .. . .. . . . . .. . ... . . . . .. . .

  • Typically computed

using the SVD Algorithm

  • Typically computed

Typically computed using the SVD using the SVD Algorithm Algorithm

slide-38
SLIDE 38

Linear Representation

pixel 1 pixel kl pixel 2 255 255 255

. .

3

c +

1

c

9

c +

28

c +

3

c

2

c

Running Sum:

1 term 3 terms 9 terms 28 terms

1

c

Eigenfaces

  • Facial images
  • Eigenfaces basis vectors capture the variability in facial appearance
  • Eigenfaces have been successful in simple facial recognition problem

– i.e., frontal images with fixed illumination

  • Facial images

Facial images

  • Eigenfaces

Eigenfaces basis vectors capture the variability in facial appearance basis vectors capture the variability in facial appearance

  • Eigenfaces

Eigenfaces have been successful in simple facial recognition problem have been successful in simple facial recognition problem – – i.e., frontal images with fixed illumination i.e., frontal images with fixed illumination

slide-39
SLIDE 39

The Problem with Linear (PCA) Appearance-Based Recognition Methods

Eigenimages work best for recognition when only a single factor – e.g., object identity – is allowed to vary

  • However, natural images are the consequence of multiple factors (or modes)

related to scene structure, illumination and imaging

Eigenimages Eigenimages work best for recognition when only a single work best for recognition when only a single factor factor – – e.g., object identity e.g., object identity – – is allowed to vary is allowed to vary

  • However, natural images are the consequence of

However, natural images are the consequence of multiple factors multiple factors (or modes) (or modes) related to scene structure, illumination and imaging related to scene structure, illumination and imaging

Our Approach

[ Vasilescu & Terzopoulos, ECCV 02, ICPR 02, CVPR 03, CVPR 05 ]

A nonlinear appearance-based technique

  • Our appearance-based model explicitly accounts for each of the

multiple factors inherent in image formation

  • Multilinear algebra, the algebra of higher order tensors
  • Applied to facial images, we call our tensor technique

“TensorFaces”

A A nonlinear nonlinear appearance appearance-

  • based technique

based technique

  • Our appearance

Our appearance-

  • based model

based model explicitly accounts explicitly accounts for each of the for each of the multiple factors inherent in image formation multiple factors inherent in image formation

  • Multilinear

Multilinear algebra, the algebra of higher order tensors algebra, the algebra of higher order tensors

  • Applied to facial images, we call our tensor technique

Applied to facial images, we call our tensor technique “ “TensorFaces TensorFaces” ”

slide-40
SLIDE 40

Linear vs Multilinear Manifolds Preliminary Recognition Results

[Vasilescu & Terzopoulos, ICPR’02]

88% 88%

27% 27%

Training: Training: 23 people, 5 viewpoints (0,+17, 23 people, 5 viewpoints (0,+17,

  • 17,+34,

17,+34,-

  • 34), 3 illuminations

34), 3 illuminations Testing: Testing: 23 people, 5 viewpoints (0,+17, 23 people, 5 viewpoints (0,+17,

  • 17,+34,

17,+34,-

  • 34), 4

34), 4th

th illumination

illumination

80% 80%

61% 61%

Training: Training: 23 people, 3 viewpoints (0,+34, 23 people, 3 viewpoints (0,+34,-

  • 34),

34), 4 illuminations 4 illuminations Testing: Testing: 23 people, 2 viewpoints (+17, 23 people, 2 viewpoints (+17,-

  • 17),

17), 4 illuminations ( 4 illuminations (center,left,right,left+right center,left,right,left+right) )

TensorFaces TensorFaces

PCA PCA PIE Recognition Experiment PIE Recognition Experiment

slide-41
SLIDE 41

views illuminations

expressions

people

PIE Database (Weizmann) Data Organization

Linear/PCA: Data Matrix D

  • Rpixels x images
  • a matrix of image vectors

Multilinear: Data Tensor D

  • Rpeople x views x illums x express x pixels
  • N-dimensional matrix
  • 28 people, 45 images/person
  • 5 views, 3 illuminations,

3 expressions per person

Linear/PCA: Linear/PCA: Data Matrix Data Matrix D

  • R

Rpixels

pixels x images x images

  • a matrix of image vectors

a matrix of image vectors

Multilinear Multilinear: : Data Tensor Data Tensor D

D

  • R

Rpeople

people x views x x views x illums illums x express x pixels x express x pixels

  • N

N-

  • dimensional matrix

dimensional matrix

  • 28 people, 45 images/person

28 people, 45 images/person

  • 5 views, 3 illuminations,

5 views, 3 illuminations, 3 expressions per person 3 expressions per person

ex il vp p , , ,

i

People Illuminations E x p r e s s i

  • n

s Views Pixels Images

slide-42
SLIDE 42

Tensor Decomposition Complete Dataset

Learning Stage

Background on Tensor Decomposition

  • Factor Analysis:

– Psychometrics, Econometrics, Chemometrics,…

  • SVD:

– [Eckart and Young, 1936] (Psychometrika)

“The approximation of one matrix by another of lower rank“

  • 3-Way Factor Analysis:

– [Tucker,1966] (Psychometrika)

“Some mathematical notes on three mode factor analysis“

  • N-Way Factor Analysis:

– [Harshman, 1970] – Parafac – [Carrol and Chang, 1970] – Candecomp – [Kruskal, 1977] – [Kroonenberg and De Leeuw, 1980] – [Kapteyn, Neudecker, and Wansbeek, 1986] – [Franc, 1992] – [de Lathauwer, 1997]

  • Factor Analysis:

Factor Analysis: – – Psychometrics, Econometrics, Psychometrics, Econometrics, Chemometrics Chemometrics, ,… …

  • SVD:

SVD: – – [ [Eckart Eckart and Young, 1936] and Young, 1936] ( (Psychometrika Psychometrika) )

“ “The approximation of one matrix by another of lower rank The approximation of one matrix by another of lower rank“ “

  • 3

3-

  • Way Factor Analysis:

Way Factor Analysis: – – [Tucker [Tucker, ,1966] 1966] ( (Psychometrika Psychometrika) )

“ “Some mathematical notes on three mode factor analysis Some mathematical notes on three mode factor analysis“ “

  • N

N-

  • Way Factor Analysis:

Way Factor Analysis: – – [ [Harshman Harshman, 1970] , 1970] – – Parafac Parafac – – [ [Carrol Carrol and Chang, 1970] and Chang, 1970] – – Candecomp Candecomp – – [ [Kruskal Kruskal, 1977] , 1977] – – [ [Kroonenberg Kroonenberg and De and De Leeuw Leeuw, 1980] , 1980] – – [ [Kapteyn Kapteyn, , Neudecker Neudecker, and , and Wansbeek Wansbeek, 1986] , 1986] – – [Franc, 1992] [Franc, 1992] – – [de [de Lathauwer Lathauwer, 1997] , 1997]

slide-43
SLIDE 43

Matrix Decomposition - SVD

  • A matrix has a column and row space
  • SVD orthogonalizes these spaces and decomposes
  • Rewrite in terms of mode-n products
  • A matrix has a column and row space

A matrix has a column and row space

  • SVD

SVD orthogonalizes

  • rthogonalizes these spaces and decomposes

these spaces and decomposes

  • Rewrite in terms of

Rewrite in terms of mode mode-

  • n products

n products

2 1 I

x I

ℜ ∈ D

T 2 1SU

U D=

2 1

2 1

U U S D

× ×

=

Pixels Images

D

( contains the eigenfaces )

1

U

D

Tensor Decomposition D D is a N-dimensional “matrix”, with N spaces

  • N-mode SVD is the natural generalization of SVD
  • N-mode SVD orthogonalizes these spaces and decomposes D as

the mode-n product of N-orthogonal spaces

  • Core tensor Z governs interaction between mode matrices
  • Mode-n matrix

spans the column space of

D D D D is a

is a N

N-

  • d

dimensional

imensional “ “matrix matrix” ”, with , with N s

N spaces

paces

  • N

N-

  • mode SVD is the natural generalization of SVD

mode SVD is the natural generalization of SVD

  • N

N-

  • mode SVD

mode SVD orthogonalizes

  • rthogonalizes these spaces and decomposes

these spaces and decomposes D

D as

as the mode the mode-

  • n product of N

n product of N-

  • orthogonal spaces
  • rthogonal spaces
  • Core tensor

Core tensor Z

Z governs interaction between mode matrices

governs interaction between mode matrices

  • Mode

Mode-

  • n matrix

n matrix spans the column space of spans the column space of

N

N n

U U U U

n 2 1

2 1

× × × × = L L

Z D

) (n

D

n

U

slide-44
SLIDE 44

D

3 2 1

x x x

3 2 1

U U U

Z

=

Tensor Decomposition

( )

( )

( )

Z D

vec vec 1 2 3

U U U ⊗ ⊗ =

3 2 3 3 2 1 2 1

3 2 1 1 3 1 2 1 1 1 r r r r r r r r r , , ,

u u u

  • ∑

∑ ∑ =

= = = R R R

σ

D

Z

N-Mode SVD Algorithm

Two steps:

  • 1. For n = 1,…,N, compute matrix by computing

the SVD of the flattened matrix and setting to be the left matrix of the SVD

  • 2. Solve for the core tensor as follows

Two steps: Two steps:

1.

  • 1. For n = 1,

For n = 1,… …,N, compute matrix by computing ,N, compute matrix by computing the SVD of the flattened matrix and the SVD of the flattened matrix and setting to be the left matrix of the SVD setting to be the left matrix of the SVD 2.

  • 2. Solve for the core tensor as follows

Solve for the core tensor as follows

) ( n

D

n

U

n

U

T N N T n n T T

U U U U × × × ×

=

L L

2 2 1 1

D Z

slide-45
SLIDE 45

Facial Data Tensor Decomposition

People Illuminations E x p r e s s i

  • n

s Views

D

pixels express illums. views people

5 4 3 2 1

U U U U U × × × × × =

.

Z D

D(illums.)

Computing Uillums

Views

D

Images Illuminations

  • D(illums) - flatten D along the illumination dimension
  • Uillums – orthogonalizes the column space of D(illums)
slide-46
SLIDE 46

Computing Uviews

Illuminations Views

D(views)

D

Images

  • D(views) – flatten D along the viewpoint dimension
  • Uviews – orthogonalize the column space of D(views)

Computing Upixels

Pixels Images

D(pixels)

  • D(pixels) – flatten D along the pixel dimension
  • Upixels – orthogonal column space of D(pixels)

– eigenimages

slide-47
SLIDE 47

Multilinear (Tensor) Algebra

Nth-order tensor

N

I I I L × ×

ℜ ∈

2 1

A

n n I

J ×

ℜ ∈ M

matrix (2nd-order tensor) mode-n product:

M

n

× = A B

( ) ( )

n n

A M B =

where

I1 J1

x1

  • The mode-n product is a generalization of the product of two matrices
  • It is the product of a tensor with a matrix
  • Mode-n product of and

N n

I I I x...x x...x

1

ℜ ∈ A

I1 I2 I3 I2 J2 J2 I2 J2 I2

=

x2

N n n n

I I J I I x..x x x. x...x

1 1 1 + −

ℜ ∈ B

n n I

J x

ℜ = M

M

n

× = A B

Mode-n Product

A

B

M

x3

J3 I3 I3 J3

n n N n n n

i j i i i i i n i N i n i n j n i i n

m a

... ... ... ...

A

1 1 1

1 1 1

+ −

∑

= ⎟ ⎠ ⎞ ⎜ ⎝ ⎛

+ −

× M

slide-48
SLIDE 48

Views Illums.

P e

  • p

l e

TensorFaces:

B = Z x5 Upixels

TensorFaces: explicitly represent covariance across factors

( )

4 4 4 4 4 4 4 4 3 4 4 4 4 4 4 4 4 2 1 3 2 1 3 2 1 matrix t coefficien matrix basis matrix data

express illums. views people (pixels) pixels (pixels) T .

U U U U Z U D ⊗ ⊗ ⊗ =

TensorFaces Subsume Eigenfaces

Multilinear Analysis / TensorFaces: Multilinear Multilinear Analysis / TensorFaces: Analysis / TensorFaces:

pixels x express x illums. x views x people x

5 1

U U U U U

.

Z D

4 3 2

=

( )

T express

U

(pixels)

Z

views

U ⊗

illums.

U ⊗

people

U ⊗ = 3 2 1 matrix data

(pixels)

D 3 2 1 matrix basis

pixels

U

Linear Analysis / Linear Analysis / Eigenfaces Eigenfaces: :

slide-49
SLIDE 49

Dimensionality Reduction ˆ

Iterative dimensionality reduction approach:

  • Optimize mode per mode in an iterative way
  • Alternating Least Squares (ALS) algorithm improves data fit

Iterative dimensionality reduction approach: Iterative dimensionality reduction approach:

  • Optimize mode per mode in an iterative way

Optimize mode per mode in an iterative way

  • Alternating Least Squares (ALS) algorithm improves data fit

Alternating Least Squares (ALS) algorithm improves data fit ∑ + ∑ + ∑ ≤

= = =

−

N N N N

I R i i I R i i I R i i

σ σ σ

2 2 2 2

2 2 2 2 1 1 1 1

L

D ˆ D

TensorFaces Mean Sq. Err. = 409.15 3 illum + 11 people param. 33 basis vectors PCA Mean Sq. Err. = 85.75 33 parameters 33 basis vectors

Strategic Data Compression = Perceptual Quality

Original 176 basis vectors TensorFaces 6 illum + 11 people param. 66 basis vectors

TensorFaces data reduction in illumination space TensorFaces data reduction in illumination space primarily degrades illumination effects (cast primarily degrades illumination effects (cast shadows, highlights) shadows, highlights)

  • PCA has

PCA has lower mean square error lower mean square error but but higher perceptual error higher perceptual error

slide-50
SLIDE 50

Query Image Tensor Decomposition Recognized Person Complete Dataset Face Recognition

Illumination Parameters Person Parameters Viewpoint Parameters TensorFaces

i i

Uc d =

Linear Representation:

3

c +

1

c

9

c +

28

c +

=

i i T

c d U =

Projection Operator

U

Unknown coefficient vector

slide-51
SLIDE 51

Multilinear Representation: Multilinear Representation:

v2 p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 p1 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4

p1 p2 p3 p4

[ [

i1 i2 i3 i4 v1 v2 v3

[ ] ] [

p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 v1 v1 v2 v3 v1 v2 v3 v1 v2 v3 v1 v1 v1 v1 v1 v1 v1 v1 v1 v1 v1 v1

T T T

v i p d r r r r

3 2 1

× × × =T

v3

[

=

Unknown coefficient vectors [Vasilescu & Terzopoulos CVPR’05]

×

  • 1

(pixels)

T

T T T

v i p d r r r r

3 2 1

× × × =T

v2 p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 p1 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4

p1 p2 p3 p4

[ [

i1 i2 i3 i4 v1 v2 v3

[ ] ] [

p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 v1 v1 v2 v3 v1 v2 v3 v1 v2 v3 v1 v1 v1 v1 v1 v1 v1 v1 v1 v1 v1 v1 v3

[

=

Projection Tensor

T

pixels

  • 1

(pixels) ×

T

v i p r

  • r
  • r

T

Response Tensor

slide-52
SLIDE 52

×

  • 1

(pixels)

T

p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 v2 p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4

p1 p2 p3 p4

[ [

i1 i2 i3 i4 v1 v2 v3

[ ] ] [

p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 v1 v1 v2 v3 v1 v2 v3 v1 v2 v3 v1 v1 v1 v1 v1 v1 v1 v1 v1 v1 v1 v1 v3

=

Response Tensor – Rank (1,…,1)

×

  • 1

(pixels)

T

p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 v2 p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4

p1 p2 p3 p4

[ [

i1 i2 i3 i4 v1 v2 v3

[ ] ] [

p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 v1 v1 v2 v3 v1 v2 v3 v1 v2 v3 v1 v1 v1 v1 v1 v1 v1 v1 v1 v1 v1 v1 v3

=

Rensponse Tensor – Rank (1,…,1)

slide-53
SLIDE 53

×

  • 1

(pixels)

T

p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 v2 p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4

p1 p2 p3 p4

[ [

i1 i2 i3 i4 v1 v2 v3

[ ] ] [

p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 p1 p2 p3 p4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4 v1 v1 v2 v3 v1 v2 v3 v1 v2 v3 v1 v1 v1 v1 v1 v1 v1 v1 v1 v1 v1 v1 v3

=

Response Tensor – Rank (1,…,1)

  • 1. Compute the Projection Tensor:
  • 2. Compute the Response Tensor:
  • 3. Extract the coefficient vectors by factorizing the

Response Tensor using the N-mode SVD algorithm

Multilinear Projection

+

=

(mode) (mode) T

P

l v p p v l d

  • =

= =

⊗ ⊗ × ×

) (

mode mode T T T T

J P R

slide-54
SLIDE 54

Higher Higher-

  • Order

Order Statistics Statistics 2 2nd

nd -

  • Order

Order Statistics Statistics (covariance) (covariance) Our Nonlinear Our Nonlinear ( (Multilinear Multilinear) Models ) Models Linear Linear Models Models

Perspective on Multilinear Models Multilinear PCA

TensorFaces

PCA

Eigenfaces

ICA Multilinear ICA

Independent TensorFaces

[Vasilescu & Terzopoulos, Learning 2004]

. . . . .. .. . . . . . . . . . . . . . . . . .. .. .. . .. . . .

pixel 1 pixel kl 255 255

. . . . .. .. . . . . . . . . . . . . . . . . .. .. .. . .. . . .

PCA ICA

T

USV D =

coefficient matrix basis matrix

U D =

T

SV C K =

independent components T T −

W W

coefficient matrix

pixel 1 pixel kl 255 255

slide-55
SLIDE 55

1. For n=1,…,N, compute matrix by computing the SVD of the flattened matrix and setting to be the left matrix of the SVD. Compute using ICA. Our new mode matrix is 2. Solve for the core tensor as follows

T n n T n n

V Z W K

) ( −

=

N-Mode ICA

[Vasilescu & Terzopoulos, CVPR 2005]

N-Mode ICA

[Vasilescu & Terzopoulos, CVPR 2005]

) (n

D

n

U

n

U

1 1 1 2 2 1 1 1 − − − −

× × × × × ×

=

N N n n

K K K K L L

D S

T n n n n

V Z U D

) ( ) (

=

( )

T n (n) T n T n n

V Z W W U

−

=

T n

W

T N N T n n T T − − − −

× × × × × ×

=

W W W W

2 2 1 1

L L

Z S

n

K

PCA: TensorFaces:

  • Multilinear orthog. decomp.
  • Encodes 2nd order statistics
slide-56
SLIDE 56

ICA: Independent TensorFaces:

Multilinear ICA

  • Multilinear decomposition
  • Encodes higher order statistics

Freiburg U. 3D-Morphable Data

slide-57
SLIDE 57
  • 35
  • 30
  • 25
  • 20 -15 -10
  • 5

5 10 15 20 25 30 35

  • 35
  • 20
  • 15
  • 10
  • 5

5 10 15 20 25 30 35

  • 30 -25

Results

Data Set - 16,875

images

  • 75 people
  • 15 viewpoints
  • 15 illuminations

Data Set Data Set -

  • 16,875

16,875

images images

  • 75 people

75 people

  • 15 viewpoints

15 viewpoints

  • 15 illuminations

15 illuminations

Training Images Training Images -

  • 2,700

2,700

  • 75 people

75 people

  • 6 viewpoints

6 viewpoints

  • 6 illuminations

6 illuminations

Test Images: Test Images:

  • 75 people

75 people

  • 9 viewpoints

9 viewpoints

  • 9

9 illums illums

97% 97% 93% 93% 89% 89% 83% 83% Independent Independent TensorFaces TensorFaces TensorFaces TensorFaces ICA ICA PCA PCA Multilinear Multilinear Models Models Linear Models Linear Models

slide-58
SLIDE 58

Illumination Parameters

Query Image Tensor Decomposition Recognized Person Complete Dataset

Partial Image Set for Subject Complete Image Set for Subject

Person Parameters Viewpoint Parameters TensorFaces

Face Recognition Image Synthesis Data Decomposition

? ?

?

Other Applications

  • Human Motion Signatures

–3-Mode Decomposition, Recognition, & Synthesis

[Vasilescu ICPR 02, CVPR 01, SIGGRAPH 01]

  • Multilinear Image-Based Rendering

[Vasilescu & Terzopoulos, SIGGRAPH 04]

  • Human Motion Signatures

Human Motion Signatures – – 3 3-

  • Mode Decomposition, Recognition, & Synthesis

Mode Decomposition, Recognition, & Synthesis

[ [Vasilescu ICPR 02, CVPR 01, SIGGRAPH 01 Vasilescu ICPR 02, CVPR 01, SIGGRAPH 01] ]

  • Multilinear

Multilinear Image Image-

  • Based Rendering

Based Rendering

[ [Vasilescu Vasilescu & & Terzopoulos Terzopoulos, SIGGRAPH 04] , SIGGRAPH 04]

slide-59
SLIDE 59

Multilinear Image-Based Rendering

IBR: Rendering based on sparse samples of

  • bject appearance (images)

[Gortler et al. 1996, Levoy & Hanrahan 1996, …]

  • Surface appearance is determined by the complex

interaction of multiple factors: – Scene geometry – Illumination – Imaging

IBR: Rendering based on sparse samples of IBR: Rendering based on sparse samples of

  • bject appearance (images)
  • bject appearance (images)

[ [Gortler Gortler et al. 1996, et al. 1996, Levoy Levoy & & Hanrahan Hanrahan 1996, 1996, … …] ]

  • Surface appearance is determined by the complex

Surface appearance is determined by the complex interaction of multiple factors: interaction of multiple factors: – – Scene geometry Scene geometry – – Illumination Illumination – – Imaging Imaging

Bidirectional Texture Function Bidirectional Texture Function

BTF: Captures the appearance of extended textured surfaces with

–Spatially varying reflectance –Surface mesostructure (3D texture) –Subsurface scattering –Etc.

  • Generalization of BRDF, which accounts
  • nly for surface microstructure at a point

BTF: BTF: Captures the appearance of extended Captures the appearance of extended textured surfaces with textured surfaces with

– – Spatially varying reflectance Spatially varying reflectance – – Surface Surface mesostructure mesostructure (3D texture) (3D texture) – – Subsurface scattering Subsurface scattering – – Etc. Etc.

  • Generalization of

Generalization of BRDF BRDF, which accounts , which accounts

  • nly for surface microstructure at a point
  • nly for surface microstructure at a point
slide-60
SLIDE 60

BTF BTF BTF

Reflectance as a function of position on surface, Reflectance as a function of position on surface, view direction, and illumination direction view direction, and illumination direction

  • The BTF captures shading and

The BTF captures shading and mesostructural mesostructural self self-

  • shadowing,

shadowing, self self-

  • occlusion,
  • cclusion, interreflection

interreflection, subsurface scattering , subsurface scattering

) , , , , , (

i i v v BTF

y x f φ θ φ θ

position

  • n surface

(texel) view direction illumination direction photometric angles

Plaster Plaster Pebbles Pebbles Concrete Concrete

BTF Texture Mapping

[Dana et al. 1999]

BTF Texture Mapping

[Dana et al. 1999]

Standard Texture Mapping BTF Texture Mapping

slide-61
SLIDE 61

System Diagram

Image Acquisition, Pre-processing & Organization Tensor Decomposition & Dimensionality Reduction Rendering Algorithm Uviews Uillums

T

D

D

T

?

Viewpoint Illumination Geometry

TensorTextures: Multilinear Image-Based Rendering

slide-62
SLIDE 62

Rendered Texture for a Planar Surface

Conclusion

Multilinear algebraic framework for computer vision and computer graphics

  • Tensor approach to the analysis and synthesis of image ensembles

– TensorFaces and TensorTextures – Multilinear PCA and ICA

  • Potentially of interest in all multifactor problems in vision and graphics to which

PCA has been applied; e.g: – Deformable models – Active appearance models [Cootes and Taylor] – Morphable face models [Blanz and Vetter] – Precomputed dynamics [James and Fatahalian]

  • Applications in many other fields of science

Multilinear Multilinear algebraic framework for computer vision and computer algebraic framework for computer vision and computer graphics graphics

  • Tensor approach to the analysis and synthesis of image ensembles

Tensor approach to the analysis and synthesis of image ensembles – – TensorFaces TensorFaces and and TensorTextures TensorTextures – – Multilinear Multilinear PCA and ICA PCA and ICA

  • Potentially of interest in

Potentially of interest in all all multifactor problems in vision and graphics to which multifactor problems in vision and graphics to which PCA has been applied; PCA has been applied; e.g e.g: : – – Deformable models Deformable models – – Active appearance models Active appearance models [ [Cootes Cootes and Taylor] and Taylor] – – Morphable Morphable face models face models [ [Blanz Blanz and Vetter] and Vetter] – – Precomputed Precomputed dynamics dynamics [James and [James and Fatahalian Fatahalian] ]

  • Applications in many other fields of science

Applications in many other fields of science

slide-63
SLIDE 63

TensorTextures - IBR Human Motion Signatures …

Tensor Algebra Foundation Multilinear PCA/ICA Tensor Algebra Foundation Multilinear PCA/ICA

Bioinformatics Machine Learning Econometrics

slide-64
SLIDE 64

Additional Information

www.media.mit.edu/~maov terzopoulos.com Acknowledgement:

Funded by the Technical Support Working Group (TSWG)

  • f the US Department of Defense

www.media.mit.edu/~maov www.media.mit.edu/~maov terzopoulos.com terzopoulos.com Acknowledgement: Acknowledgement:

F Fu un nd de ed d b by y t th he e Technical Support Working Group (TSWG) Technical Support Working Group (TSWG)

  • f the US Department of Defense
  • f the US Department of Defense