computer aided medical procedures & augmented reality | - - PowerPoint PPT Presentation

4 July 2007

1GALEN group | laboratoire de mathématiques appliquées aux systèmes | ecole centrale paris 2computer aided medical procedures & augmented reality | technische universität münchen 3computer science department | university of crete

computer aided medical procedures & augmented reality | campar.cs.tum.edu

Inter and Intra-Modal Deformable Registration:

Continuous Deformations Meet Efficient Optimal Linear Programming

Ben Glocker1,2, Nikos Komodakis1,3, Nikos Paragios1, Georgios Tziritas3, Nassir Navab2

camp+ar | department of computer science | technische universität münchen | 11 July 2007 2 computer aided medical procedures & augmented reality | campar.cs.tum.edu

Outline

• Introduction & Motivation
• Image Registration based on Discrete Labeling
• Optimization using Linear Programming
• Results & Conclusions

In this presentation everything is intensity-based

camp+ar | department of computer science | technische universität münchen | 11 July 2007 3 computer aided medical procedures & augmented reality | campar.cs.tum.edu

Introduction

• Source and target image
• Image relation

g(x) = h ◦ f(T (x)) g : Ω → R Ω ⊂ Rd with d ∈ {2, 3} T : Ω → Ω non-linear transformation f : Ω → R h : R → R non-linear relation on intensities

camp+ar | department of computer science | technische universität münchen | 11 July 2007 4 computer aided medical procedures & augmented reality | campar.cs.tum.edu

Registration as an Optimization Problem

• Energy formulation
• The aim of registration is to recover the transformation

which involves

• the definition of a transformation type
• the definition of a distance/similarity measure
• the definition of an optimization procedure

E(T ) = Z

Ω

ρh(g(x), f(T (x))dx → min! ρh : R × R → R distance measure

camp+ar | department of computer science | technische universität münchen | 11 July 2007 5 computer aided medical procedures & augmented reality | campar.cs.tum.edu

Review of Registration Methods

• Types of transformations
• Rigid, affine, projective
• Basis functions, Spline-based
• Finite Element Models, …
• Distance/Similarity measures
• SAD, SSD, NCC, NMI, CR, …
• Optimization methods
• Variational
• Gradient-based
• Direct search (Simplex, Powell-Brent, Best Neighbor)

camp+ar | department of computer science | technische universität münchen | 11 July 2007 6 computer aided medical procedures & augmented reality | campar.cs.tum.edu

Motivation

• What is expected from an optimal registration method?
• Independent from the choice of the transformation type
• Independent from the choice of the distance/similarity measure
• Guarantee of a globally optimal solution
• Reasonable computational complexity

camp+ar | department of computer science | technische universität münchen | 11 July 2007 7 computer aided medical procedures & augmented reality | campar.cs.tum.edu

Our Contributions

• Novel deformable registration framework based on discrete

labeling and linear programming

• Our framework bridges the gap between continuous

deformations and discrete optimization

• Gradient-free and flexible in the choice of the distance measure
• Guaranteed optimality properties on the solution
• Computational efficient and tractable

camp+ar | department of computer science | technische universität münchen | 11 July 2007 8 computer aided medical procedures & augmented reality | campar.cs.tum.edu

Image Registration based on Discrete Labeling

camp+ar | department of computer science | technische universität münchen | 11 July 2007 9 computer aided medical procedures & augmented reality | campar.cs.tum.edu

Local Registration

• Deformation grid providing a continuous

and dense deformation field

• In our implementation we use

Free Form Deformation D(x) =

3

X

l=0 3

X

m=0

Bl(u)Bm(v) di+l,j+m T (x) = x + D(x) with D(x) = X

p∈G

η(|x − p|) dp

[Rueckert99, Schnabel01, Rohlfing03, …]

camp+ar | department of computer science | technische universität münchen | 11 July 2007 10 computer aided medical procedures & augmented reality | campar.cs.tum.edu

Energy Formulation

• Reformulation of the optimization problem
• Smoothness term
• Registration task

Edata(T ) = X

p∈G

Z

Ω

η−1(|x − p|) · ρh(g(x), f(T (x)))dx Esmooth(T ) = X

p∈G

φ(|∇G dp|) Etotal = Edata + Esmooth → min!

camp+ar | department of computer science | technische universität münchen | 11 July 2007 11 computer aided medical procedures & augmented reality | campar.cs.tum.edu

p q r s t u v w x p q r s t u v w x

• General energy formulation
• Markov Random Field (MRF) formulation for discrete labelings

Discrete Optimization Problem

Etotal = Edata + Esmooth → min!

Data term = singleton potentials Smoothness term = pairwise potentials

Vpq(up, uq) Etotal(u) = X

p∈G

Vp(up) + X

p,q∈E

Vpq(up, uq) Vp(up)

camp+ar | department of computer science | technische universität münchen | 11 July 2007 12 computer aided medical procedures & augmented reality | campar.cs.tum.edu

Discretization of Parameter Space

• Set of labels and a discretized deformation space

L = {u1, ..., ui} Θ = {d1, ..., di}

+X

• X

+Y

• Y

Max Steps +X

• X

+Y

• Y

Sparse sampling Dense sampling

camp+ar | department of computer science | technische universität münchen | 11 July 2007 13 computer aided medical procedures & augmented reality | campar.cs.tum.edu

• MRF singleton potentials:
• Problem: singleton potentials are not independent!

Data Term

|L| × |G| cost matrix Edata(u) = X

p∈G

Z

Ω

η−1(|x − p|)ρh(g(x), f(T (x)))dx ≈ X

p∈G

Vp(up)

camp+ar | department of computer science | technische universität münchen | 11 July 2007 14 computer aided medical procedures & augmented reality | campar.cs.tum.edu

x=0 y=0

• Approximation of label costs simultaneously for all nodes

Fast Approximation of Singleton Potentials

Single potential look-up table Nodes Labels

Current label

camp+ar | department of computer science | technische universität münchen | 11 July 2007 15 computer aided medical procedures & augmented reality | campar.cs.tum.edu

x=10 y=0

• Approximation of label costs simultaneously for all nodes

Fast Approximation of Singleton Potentials

Single potential look-up table Nodes Labels

Current label

camp+ar | department of computer science | technische universität münchen | 11 July 2007 16 computer aided medical procedures & augmented reality | campar.cs.tum.edu

x=10 y=-10

• Approximation of label costs simultaneously for all nodes

Fast Approximation of Singleton Potentials

Single potential look-up table Nodes Labels

Current label

camp+ar | department of computer science | technische universität münchen | 11 July 2007 17 computer aided medical procedures & augmented reality | campar.cs.tum.edu

x=0 y=-10

• Approximation of label costs simultaneously for all nodes

Fast Approximation of Singleton Potentials

Single potential look-up table Nodes Labels

Current label

camp+ar | department of computer science | technische universität münchen | 11 July 2007 18 computer aided medical procedures & augmented reality | campar.cs.tum.edu

x=-10 y=-10

• Approximation of label costs simultaneously for all nodes

Fast Approximation of Singleton Potentials

Single potential look-up table Nodes Labels

Current label

camp+ar | department of computer science | technische universität münchen | 11 July 2007 19 computer aided medical procedures & augmented reality | campar.cs.tum.edu

x=-10 y=0

• Approximation of label costs simultaneously for all nodes

Fast Approximation of Singleton Potentials

Single potential look-up table Nodes Labels

Current label

camp+ar | department of computer science | technische universität münchen | 11 July 2007 20 computer aided medical procedures & augmented reality | campar.cs.tum.edu

x=-10 y=10

• Approximation of label costs simultaneously for all nodes

Fast Approximation of Singleton Potentials

Single potential look-up table Nodes Labels

Current label

camp+ar | department of computer science | technische universität münchen | 11 July 2007 21 computer aided medical procedures & augmented reality | campar.cs.tum.edu

x=0 y=10

• Approximation of label costs simultaneously for all nodes

Fast Approximation of Singleton Potentials

Single potential look-up table Nodes Labels

Current label

camp+ar | department of computer science | technische universität münchen | 11 July 2007 22 computer aided medical procedures & augmented reality | campar.cs.tum.edu

x=10 y=10

• Approximation of label costs simultaneously for all nodes

Fast Approximation of Singleton Potentials

Single potential look-up table Nodes Labels

Current label

camp+ar | department of computer science | technische universität münchen | 11 July 2007 23 computer aided medical procedures & augmented reality | campar.cs.tum.edu

• MRF pairwise potentials:

e.g. truncated absolute difference (piecewise smooth) Note: smoothness function can vary locally

Smoothness Term

Vpq(up, uq) = λpq min (|dup − duq|, T) Esmooth(u) = X

p,q∈E

Vpq(up, uq) |L| × |L| cost matrix

camp+ar | department of computer science | technische universität münchen | 11 July 2007 24 computer aided medical procedures & augmented reality | campar.cs.tum.edu

MRF Formulation of Image Registration

Etotal(u) = X

p∈G

Vp(up) + X

p,q∈E

Vpq(up, uq)

Opens the door to MRF optimization techniques Opens the door to MRF optimization techniques

camp+ar | department of computer science | technische universität münchen | 11 July 2007 25 computer aided medical procedures & augmented reality | campar.cs.tum.edu

Optimization using Linear Programming

camp+ar | department of computer science | technische universität münchen | 11 July 2007 26 computer aided medical procedures & augmented reality | campar.cs.tum.edu

• Say we seek an optimal solution x* to the following integer program (this is our

primal problem):

• To find an approximate solution, we first relax the integrality constraints to get

a primal & a dual linear program:

Primal-Dual Schema

(NP-hard problem) primal LP: dual LP:

camp+ar | department of computer science | technische universität münchen | 11 July 2007 27 computer aided medical procedures & augmented reality | campar.cs.tum.edu

• Goal: find integral-primal solution x, feasible dual solution y

such that their primal-dual costs are “close enough”, e.g.,

Primal-Dual Schema

T

b y

T

c x

primal cost of solution x primal cost of solution x dual cost of solution y dual cost of solution y

* T

c x

cost of optimal integral solution x* cost of optimal integral solution x*

*

f ≤

T T

c x b y

* *

f ≤

T T

c x c x

Then x is an f*-approximation to optimal solution x* Then x is an f*-approximation to optimal solution x*

camp+ar | department of computer science | technische universität münchen | 11 July 2007 28 computer aided medical procedures & augmented reality | campar.cs.tum.edu

• The primal-dual schema works iteratively

Primal-Dual Schema

1 T

b y

1 T

c x

sequence of dual costs sequence of dual costs sequence of primal costs sequence of primal costs

2 T

b y

…

k T

b y

* T

c x

unknown optimum unknown optimum

2 T

c x

…

k T

c x

k * k

f ≤

T T

c x b y

camp+ar | department of computer science | technische universität münchen | 11 July 2007 29 computer aided medical procedures & augmented reality | campar.cs.tum.edu

Primal-Dual Schema for MRFs

(only one label assigned per vertex) enforce consistency between variables xp,a, xq,b and variable xpq,ab

Binary variables

xp,a =1 label a is assigned to node p xpq,ab =1 labels a, b are assigned to nodes p, q xp,a =1 label a is assigned to node p xpq,ab =1 labels a, b are assigned to nodes p, q

camp+ar | department of computer science | technische universität münchen | 11 July 2007 30 computer aided medical procedures & augmented reality | campar.cs.tum.edu

Primal-Dual Schema for MRFs

• During the PD schema for MRFs, it turns out that
• Resulting flows tell us how to update both:

the dual variables, as well as the primal variables

each update of primal and dual variables each update of primal and dual variables solving max-flow in appropriately constructed graph solving max-flow in appropriately constructed graph

for each iteration of primal- dual schema

Fast-PD

camp+ar | department of computer science | technische universität münchen | 11 July 2007 31 computer aided medical procedures & augmented reality | campar.cs.tum.edu

• MRF optimization method based on duality theory of Linear

Programming (the Primal-Dual schema)

• Can handle a very wide class of MRFs
• Can guarantee approximately optimal solutions

(worst-case theoretical guarantees)

• Can provide tight certificates of optimality per-instance

(per-instance guarantees)

• Provides significant speed-up for static and dynamic MRFs

Fast-PD

Komodakis, N., Tziritas, G., Paragios, N. Fast, Approximately Optimal Solutions for Single and Dynamic MRFs. Computer Vision and Pattern Recognition 2007

camp+ar | department of computer science | technische universität münchen | 11 July 2007 32 computer aided medical procedures & augmented reality | campar.cs.tum.edu

MRF Hardness

MRF pairwise potential MRF hardness linear exact global

• ptimum

arbitrary local optimum metric global optimum approximation

Fast-PD

camp+ar | department of computer science | technische universität münchen | 11 July 2007 33 computer aided medical procedures & augmented reality | campar.cs.tum.edu

for i=1:no_iterations 1: setup_label_sets() 2: precompute_single_potential_matrix() 3: precompute_pairwise_potential_matrix() 4: compute_discrete_labeling() 5: update_deformation() end

Registration Algorithm

Gaussian image pyramids & multi-level deformation grids are used for a hierarchical registration approach

camp+ar | department of computer science | technische universität münchen | 11 July 2007 34 computer aided medical procedures & augmented reality | campar.cs.tum.edu

Results & Conclusions

camp+ar | department of computer science | technische universität münchen | 11 July 2007 35 computer aided medical procedures & augmented reality | campar.cs.tum.edu

SSD error

12278 3402 3180 1233

Running time

Before registration > 2 hours < 2 min < 2 min

Schnabel01 Our Method Our Method

Comparison

camp+ar | department of computer science | technische universität münchen | 11 July 2007 36 computer aided medical procedures & augmented reality | campar.cs.tum.edu

Visual Results

camp+ar | department of computer science | technische universität münchen | 11 July 2007 37 computer aided medical procedures & augmented reality | campar.cs.tum.edu

Validation of Distance Measures

camp+ar | department of computer science | technische universität münchen | 11 July 2007 38 computer aided medical procedures & augmented reality | campar.cs.tum.edu

Validation of Distance Measures

camp+ar | department of computer science | technische universität münchen | 11 July 2007 39 computer aided medical procedures & augmented reality | campar.cs.tum.edu

Results (MICCAI 2007)

Method DSC Sensitivity Specificity Interaction Cartilage Grau et al. 0.90 (0.01) 90.03 % 99.87 % 5-10 min Tibia, Femur, Patella Dam et al. 0.92 (n/a) 93.00 % 99.99 % Max 10 min Tibia, Femur Cheong et al. 0.64 (0.15) 74.00 % n/a Medial Tibia Cheomg et al. 0.72 (0.09) 79.00 % n/a Lateral Tibia Folkesson et al. 0.80 (0.03) 90.01 % 99.80 % Tibia, Femur Our Approach 0.83 (0.06) 93.77 % 99.94 % Patella

ρAtlas(Iμ, Iσ2, Inew) =

1 |Ω|

P

x∈Ω (Iμ(x)−Inew(x))2 2Iσ2(x)

Mean Intensity Image Variance Intensity Image Segmentation

camp+ar | department of computer science | technische universität münchen | 11 July 2007 40 computer aided medical procedures & augmented reality | campar.cs.tum.edu

Conclusions

• What is provided by our registration framework?
• Independent from the choice of the transformation type?

Grid-based but independent from weighting function.

• Independent from the choice of the distance/similarity measure?
• Yes. We are gradient-free.
• Guarantee of a globally optimal solution?

Quasi-yes (from an optimization point of view).

• Reasonable computational complexity?

Yes (256x256 in 2 seconds, 256x192x64 in 1 minute).

camp+ar | department of computer science | technische universität münchen | 11 July 2007 41 computer aided medical procedures & augmented reality | campar.cs.tum.edu

Latest Work & Future Directions

• On-the-fly estimation of locally varying

deformation spaces (done!)

• Incorporate global registration & different

deformation models

• Introducing domain knowledge such as

priors on the deformation space

• Belief propagation networks
• GPU implementation