Registration in Computer-Assisted Laparoscopic Surgery Lena - - PowerPoint PPT Presentation

Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery

Lena Maier-Hein, PhD Division of Medical and Biological Informatics (MBI) German Cancer Research Center (DKFZ)

Registration in Computer-Assisted Laparoscopic Surgery

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Laparoscopic surgery

• Access to the abdomen through small incisions
• Instruments/endoscope are inserted through trocars
• Many advantages, but:
• 2D screen vs. 3D world
• Small field-of-view
• Reduced mobility
• No tactile feedback

requires a lot of skill and experience

(Source: D. Stoyanov, PhD thesis 2005)

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Computer-assisted laparoscopic surgery

• Goal: Visualize hidden structures via Augmented Reality (AR)
• Requires: Registration of pre-operative data with intra-operative

data

(Source: M. Sugimoto, J Hepatobiliary PancreatSci 2010) (Source: M. Baumhauer, IntJ CARS 2008)

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Computer-assisted surgery

Pre-operative Intra-operative

Planning data Modality for surgical planning (i.e. CT device) Tracking system (i.e. optical) Intra-operative imaging modalities (i.e. endoscope) Intra-operative patient data Tracking data registration registration

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Outline

• Part I: Basics
• Part II: Global Surface Registration
• Part III: Fine Surface Registration
• Part IV: Registration in practice - state-of-the-art approaches

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Outline

• Part I: Basics
• Part II: Global Surface Registration
• Part III: Fine Surface Registration
• Part IV: Registration in practice - state-of-the-art approaches

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Basics – Input shapes

• Geometry representation
• Points
• Oriented points (i.e., points with normals)
• Surfaces
• Skeletons
• …
• Surface representation
• implicit (e.g. level set)
• Explicit (e.g. triangle meshes)

Triangle mesh and oriented points (blue) (Source: [Johnson1997])

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

• Shape descriptor: Scalar value or vector that describes a given

shape (e.g. moments)

• Local surface descriptor: Captures some property of the shape

around the neighbourhood of a point (e.g. curvature)

• Desirable properties:
• Robust
• Rotation invariant
• Translation invariant
• Local
• Efficient
• ….
• Requires: Local coordinate system

Basics – Surface descriptor

(Source: [Gelfand2005])

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Many types of surface descriptors available

• Curvature
• Spin images
• Integral variants
• Shape context
• Multi-scale features
• Curvature maps
• Spherical harmonics and wavelets
• Salient geometric features
• Part-aware metric
• Heat Kernel Signature
• Shot
• …

(cf. [vanKaick2011] for references)

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

• Curvature κ in a point on a curve

r

Descriptors – Curvature (on a curve)

(Source: Wikipedia)

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Normal plane: Plane containing the normal vector Normal curvature: Curvature of the curve obtained by intersecting the surface with a normal plane Principal curvature: Maximum (κ1) and minimum (κ2) values of the normal curvature at a point.

Descriptors – Principal curvature (on a surface)

(Source: Wikipedia)

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Mean curvature: Gaussian curvature:

(Source: Wikipedia)

Descriptors – Mean/Gaussian curvature

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Many types of surface descriptors available

• Curvature (Example of low-dimensional descriptor)
• Spin images (Example of high-dimensional descriptor)
• Integral variants
• Shape context
• Multi-scale features
• Curvature maps
• Spherical harmonics and wavelets
• Salient geometric features
• Part-aware metric
• Heat Kernel Signature
• Shot
• …

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Descriptors – Spin Images

(Source: [Johnson1997])

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Descriptors – Spin Images: Coordinate System

• Cylinder coordinates
• α: Distance to axis defined by normal vector n
• β: Distance to tangent plane

(Source: [Johnson1997])

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada (Source: [Johnson1997])

Descriptors – Spin Images: Illustration

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada (Source: [Johnson1997])

Descriptors – Spin images: Examples

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

• Define size and bin size of spin image
• For each vertex v:
• For each vertex in the support volume of v:
• Compute coordinates (α,β)
• Increment corresponding bins using bilinear interpolation

(Source: [Johnson1997])

Descriptors – Spin images: Algorithm

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

• Different image sizes (and thus support volumes) yield different spin

images

(Source: [Johnson1997])

Descriptors – Spin images: Image Size

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

• Bin size: Width of bins
• Should be set depending on mesh resolution
• Larger bin sizes imply:
• decreasing influence of individual vertices
• a greater memory requirement

(Source: [Johnson1997])

Descriptors – Spin images: Bin size

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

• Support angle θ:
• nA:normal vector of point of interest
• Only consider those points in the support volumes with normal vector nB that fulfill

the condition:

• The angle enclosed by nB and nA is smaller than or equal to θ
• Useful for range image registration due to occlusion

(Source: [Johnson1997])

Descriptors – Spin images: Support Angle

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Many types of surface descriptors available

• Curvature
• Spin images
• Integral variants
• Shape context
• Multi-scale features
• Curvature maps
• Spherical harmonics and wavelets
• Salient geometric features
• Part-aware metric
• Heat Kernel Signature
• Shot
• …

(Source: [Gelfand2005])

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Outline

• Part I: Basics
• Part II: Global Surface Registration
• Part III: Fine Surface Registration
• Part IV: Registration in practice - state-of-the-art approaches

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Surface Matching: Challenges

(Image: Stefanie Speidel et al., Karlsruhe Institute ofTechnology (KIT))

Partial surface registration Non-rigid surface registration

(Image: Stefanie Speidel et al., KIT)

Noise Real-time registration …

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Outline

• Part I: Basics
• Part II: Global Surface Registration
• Global analysis based registration
• Local analysis based registration
• Part III: Fine Surface Registration
• Part IV: Registration in practice - state-of-the-art approaches

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Surface matching – global analysis

• Compute shape properties and use them for an initial alignment
• Example: Principal Component Analysis (PCA) based alignment
• Compute and align centroids and principal axes of both shapes
• Not suitable for partial surface matching
• Global surface properties are often used for shape retrieval
• Example using Manifold Harmonics Transform [Reuter2006]

(Source: [Reuter2006])

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Outline

• Part I: Basics
• Part II: Global Surface Registration
• Global analysis based registration
• Local analysis based registration
• Part III: Fine Surface Registration
• Part IV: Registration in practice - state-of-the-art approaches

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Descriptor Computation Correspondence Search Transformation Computation Feature Extraction

Registration based on local analysis: Workflow

Remark: Steps can be combined (e.g. Correspondence Search and Transformation Computation)

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Feature Extraction

• Extract features from both shapes
• Points (e.g. based on high curvature or multi-resolution approach)
• Regions (e.g. based on mesh segmentation)
• Contours
• Example: Use points with a high average squared geodesic

distance to other points [Gelfand2005]

(Source: Zhang2008])

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Descriptor Computation Correspondence Search Transformation Computation Feature Extraction

Registration based on local analysis: Workflow

c.f. Basics above

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Descriptor Computation Correspondence Search Transformation Computation Feature Extraction

Registration based on local analysis: Workflow

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Correspondence search

• Requires
• Objective function: quantifies quality of chosen correspondences
• Optimization approach: Aims for minimizing/maximizing the objective

function

• The objective function is composed of a similarity term and (in

the NAP case) a distortion term:

“For two shapes P and Q and a correspondence relation R, the objective takes the form : Obj(P;Q;R) = Sim(P;Q;R)+a Distor(P;Q;R) with a similarity term that is linear on the number of feature points and a distortion term that is usually quadratic on the number of feature points, since it commonly involves comparing properties of pairs of points.” [vanKaick2011]

• In some cases (in particular for α = 0), the problem can be stated

as a Linear Assignment Problem (LAP) [dosSantos2010]

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Descriptor Computation Correspondence Search Transformation Computation Feature Extraction

Registration based on local analysis: Workflow

• Methods for solving LAP
• Tree-based methods
• Voting-based methods
• Graph-based methods
• Embedding-based methods
• …

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Correspondence Search: LAP

• Given:
• A similarity matrix quantifying the similarity of each potential

correspondence

• Goal:
• Find a set of correspondences that minimizes the sum of costs for the

global assignment

• Method
• Hungarian method [Kuhn1955] for one-to-one correspondences
• Variants available for different numbers of features in shapes

Similarity matrix

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Correspondence Search – Distortion term

• If α > 0, a distortion term has to be defined in addition to the

similarity term:

“The distortion term quantifies how much the shapes would be deformed if their corresponding elements were brought into alignment. A common candidate for a distortion measure is the disparity in the distances between pairs of matched

• points. The disparity is an approximate way of measuring the distortion

introduced by the correspondence without having to first align the shapes. It can be expressed as where (p1;q1) Є R and (p2;q2) Є R.” [vanKaick2011]

The disparity term can e.g. represent the difference in the Euclidean or geodesic distances between pairs of points.

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Descriptor Computation Correspondence Search Transformation Computation Feature Extraction

Registration based on local analysis: Workflow

• Methods for solving LAP
• Tree-based methods
• Voting-based methods
• Graph-based methods
• Embedding-based methods
• …

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Correspondences – Tree-based (1/4)

• During correspondence search, each tree node represents a

partial solution

• Full solution represented by path from root to leaf

(Source: [vanKaick2011]/[Zhang2008]) Sample and verify (Source: [vanKaick2011]/[Aiger2008])

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Correspondences – Tree-based (2/4)

• Generally, three steps required [vanKaick2011]:
• Branching, Bounding, Pruning
• Branching: Expanding a node that represents a new partial

solution

• E.g. by adding a new pairwise assignment to a given solution
• Bounding: Estimating how far the partial solution is from the
• ptimum solution
• E.g. by aligning the shapes based on the current correspondences
• Pruning: Eliminating nodes that will not lead to the optimum

solution

• E.g. by testing the compatibility between pairwise assignments

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Correspondences – Tree-based (3/4)

• State-of-the-art example for non-rigid registration [Zhang 2008]:
• Deformation-driven search prioritized by a self-distortion energy measured
• n meshes deformed according to a given correspondence
• Pruning techniques based e.g. on pairwise geodesic distances and feature-

to-feature similarity defined by curvature maps

(Source: [Zhang2008])

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Correspondences – Tree-based (4/4)

• More Examples for search strategies in trees:
• Branch-and-bound [Gelfand2005]
• Priority-driven search [Funkhouser2006]
• Belief propagation [Anguelov2004]
• Integer quadratic programming [Berg2005]

(Source: [Gelfand2005])

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Correspondences – Voting (1/2)

• Originally designed for rigid registration, voting methods

make use of the fact that the rigid transform is low- dimensional and exhaustively search for the small number of parameters [vanKaick2011]

• Sample Algorithm for rigid transform:

1.

Quantize the transformation space into a six dimensional table

2.

For each triplet of points in the model shape and each triplet in the data shape

1.

Compute the transformation between the triplets

2.

Record vote in the corresponding cell of the table.

3.

The entry with the most votes gives the optimal aligning transform.

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Correspondences – Voting (2/2)

• Variant: Vote for correspondences (feature pairs) as opposed

to voting for a transformation

• The assignments that are certain emerge as the ones with the

highest number of votes

• Example: Electors Voting [Au2010]

(Source: [Au2010])

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Descriptor computation Correspondence Search Transformation Computation Feature Extraction

Registration based on local analysis: Workflow

• Methods for solving LAP
• Tree-based methods
• Voting-based methods
• Graph-based methods
• Embedding-based methods
• …

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

• Generate a graph representation of the surfaces to be matched and

apply graph matching methods to establish correspondences (c.f. e.g. [dosSantos2010, Zeng2010]

• Example using mesh segmentation [dosSantos2010] :

Correspondences – Graph-based (1/2)

(Source: [dosSantos2010])

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Descriptor Computation Correspondence Search Transformation Computation Feature Extraction

Workflow for registration based on local analysis

• Methods for solving LAP
• Tree-based methods
• Voting-based methods
• Graph-based methods
• Embedding-based methods
• …

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Correspondences – Embedding

• Idea: Embed the shapes in a different space and look for

correspondences in that space (using one of the previous techniques)

• Advantage: Non-rigid registration problem may become a rigid

alignment problem in the embedding space

(Source: [vanKaick2011])

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Correspondences – Embedding: Example 1

• Motivation: Spectral embeddings normalize shapes with respect

to rigid body transformations and uniform scaling

Eigenvector scaling Non-rigid ICP via thin-plate splines Geodesic-based spectral embedding Best matching

(Source: slides corresponding to [Jain2006])

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Correspondences – Embedding: Example 2

• Utilizing Möbius transformation for shape correspondence

[Lipman2009]

(Source: [vanKaick2011]/[Lipman2009]) (Source: [Lipman2009])

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Descriptor computation Correspondence search Transformation computation Feature Extraction

Workflow for registration based on local analysis

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

• Transformation may be the result of the correspondence search (c.f.

voting techniques)

• Otherwise:
• Point-based registration (e.g. using the algorithm by Horn [Horn1987]) may be used

for rigid alignment

• Splines may be used for non-rigid alignment

(Source: http://elonen.iki.fi/code/tpsdemo/gfx/tpsdemo.png)

Transformation – Surface Alignment (1/2)

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

• Caution: In the case of non-rigid registration, surface matching is not

sufficient: Volume registration required (e.g., using biomechanical model [Cash2007])

• Typically applied after fine registration

(Image: Stefanie Speidel, Karlsruhe Institute ofTechnology (KIT))

Transformation – Surface Alignment (2/2)

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Outline

• Part I: Basics
• Part II: Global Surface Registration
• Part III: Fine Surface Registration
• Part IV: Registration in practice - state-of-the-art approaches

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Iterative Closest Point (ICP) algorithm

• Iterative procedure for registering two sets of points X and Y

[Besl1992]

• Algorithm: Iteratively

1.

Find nearest neighbour in Y to each point in X

2.

Compute „best“ rigid transformation and apply it to X

• Relies on an initial alignment of the point sets
• Converges to nearest local minimum with respect to a mean-

squared distance metric

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Iterative Closest Point (ICP) algorithm

• Iterative procedure for registering two sets of points X and Y

(here: dark and light blue) [Besl1992]

1.

Find nearest neighbour in Y to each point in X

2.

Compute „best“ rigid transformation and apply it to X

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Iterative Closest Point (ICP) algorithm

• Iterative procedure for registering two sets of points X and Y

(here: dark and light blue) [Besl1992]

1.

Find nearest neighbour in Y to each point in X

2.

Compute „best“ rigid transformation and apply it to X

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Iterative Closest Point (ICP) algorithm

• Iterative procedure for registering two sets of points X and Y

(here: dark and light blue) [Besl1992]

1.

Find nearest neighbour in Y to each point in X

2.

Compute „best“ rigid transformation and apply it to X

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Iterative Closest Point (ICP) algorithm

• Iterative procedure for registering two sets of points X and Y

(here: dark and light blue) [Besl1992]

1.

Find nearest neighbour in Y to each point in X

2.

Compute „best“ rigid transformation and apply it to X

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Iterative Closest Point (ICP) algorithm

• Iterative procedure for registering two sets of points X and Y

(here: dark and light blue) [Besl1992]

1.

Find nearest neighbour in Y to each point in X

2.

Compute „best“ rigid transformation and apply it to X

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

ICP Variants

• Many variants of the ICP available (cf. e.g.[Rusinkiewicz2001])
• Example: Generalization for registration of Time-of-Flight (ToF)

data [Maier-Hein2010]: Anisotropic ICP (A-ICP)

• Possibility to assign covariance matrix to each point in both sets
• Incorporates covariance matrices in both steps of the algorithm
• Non-rigid variants available (e.g. [Chui2003] [Amberg2007])

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Fine surface registration - Alternative

• Idea: use sparse set of safe correspondences to generate

bigger set of dense correspondences

• Example: seed-growing method [Sharma2011];

(Source: [Sharma2011])

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Discussion on Surface Matching

• Robust global partial surface matching in real-time not yet

solved

• Research in global surface registration focusses on algorithmic

developments rather than on run-time

• Key challenge: robust correspondence establishment
• Fast algorithms for fine surface registration available (e.g. GPU

implementations of the ICP or ICP variants [Tamaki2010])

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Outline

• Part I: Basics
• Part II: Global Surface Registration
• Part III: Fine Surface Registration
• Part III: Registration in practice - state-of-the-art approaches
• Surface-based
• Marker-based
• Calibration-based

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

• Example [Cash2007]
• Surface registration based on laser range scanner used to generate

boundary conditions for Finite-Element Method (FEM)

• Extension to endoscopic interventions (rigid registration) in [Rauth2007]

(Source: [Cash2007])

Clinical Registration – Surface-based

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Outline

• Part I: Basics
• Part II: Global Surface Registration
• Part III: Fine Surface Registration
• Part III: Registration in practice - state-of-the-art approaches
• Surface-based
• Marker-based
• Calibration-based

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

• Use tracked markers to register pre-operative images with the

patient

• Example: [Baumhauer2007]
• Application: Radical prostatectomy and partial nephrectomy
• Goal: Higher precision by visualizing critical structures in endoscopic

images [Baumhauer2007]

preoperative intraoperative

Insertion of Navigation Aids Data Registration Navigation Aid Tracking Pose Estimation & Visualization Preoperative Planning

< 5 min. < 10 min. Real-time ~ 30 min.

Clinical registration – Marker-based

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Clinical registration – Marker-based: Example(1/5)

Definition of target and critical structures:

• Organs
• Vessels
• Tumor
• Images from different

modalities, e.g. MRI, CT

Insertion of Navigation Aids Data Registration Navigation Aid Tracking Pose Estimation & Visualization Preoperative Planning

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Clinical registration – Marker-based: Example(2/5)

Fiducials (Navigation aids):

• Color coded
• Optimal placement: maximize inter-

needle distance

Insertion of Navigation Aids Data Registration Navigation Aid Tracking Pose Estimation & Visualization Preoperative Planning

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Clinical registration – Marker-based: Example(3/5)

Registration of planning data:

• Segmentation with statistical shape

models

• Surface-based non-rigid registration
• 3D Needle segmentation based on hough

transform

Insertion of Navigation Aids Data Registration Navigation Aid Tracking Pose Estimation & Visualization Preoperative Planning

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Clinical registration – Marker-based: Example(4/5)

Tracking Approach:

• Region of interest by color

analysis

• Hough transformation

Insertion of Navigation Aids Data Registration Navigation Aid Tracking Pose Estimation & Visualization Preoperative Planning

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Inside-Out Tracking:

• Find the pose of an object relative to

the camera

• Use point correspondences from:
• Static 3D image (C-arm or

ultrasound)

• 2D endoscopic images
• Extended Kalman filter

Insertion of Navigation Aids Data Registration Navigation Aid Tracking Pose Estimation & Visualization Preoperative Planning

Clinical registration – Marker-based: Example(5/5)

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Outline

• Part I: Basics
• Part II: Global Surface Registration
• Part III: Fine Surface Registration
• Part IV: Registration in practice - state-of-the-art approaches
• Surface-based
• Marker-based
• Calibration-based

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Clinical registration – Calibration-based

• Calibrate all imaging modalities (e.g. Ultrasound device, C-

arm,…) and track them in a common coordinate system (e.g. coordinate system of tracking system)

• Example: Calibration of an optically tracked C-arm device

according to [Feuerstein2008]

(Source: [Feuerstein2008])

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Summary

• Natural choice for registration method after 3D surface

reconstruction: surface based registration

• Non-rigid partial surface registration in real-time is extremely

challenging

• Requires
• Global registration (rough alignment)
• Fine registration (for computing final transformation)
• Key in global registration: Establishment of correspondences
• Most common methods for correspondence search based on:
• Tree search
• Voting
• Graph matching
• Embedding
• State-of-the-art registration methods are not yet based on

real-time surface matching

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Acknowledgements

Contributors: Thiago R. dos Santos, German Cancer Research Center Stefanie Speidel, University of Karlsruhe Many thanks!

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Literature

[Amberg2007] B. Amberg, S. Romdhani, T. Vetter: “Optimal step nonrigid ICP algorithms for surface registration,” in In CVPR07, 2007. [Au2010] Au O. K.-C., Cohen-Or D., Tai C.-L., Fu H., Zheng Y.: Electors voting for fast automatic shape

• correspondence. Computer Graphics Forum (Proc. EUROGRAPHICS) 29, 2 (2010).

[Audette2000] M. A. Audette, F. P. Ferrie, and T. M. Peters. An algorithmic overview of surface registration techniques for medical imaging. MIA, 4:201-217, 2000. [Anguelov2004] Anguelov D., Srinivasan P., Pang H.-C., Koller D., Thrun S., Davis J.: The correlated correspondence algorithm for unsupervised registration of nonrigid surfaces. In NIPS (2004). [Bao2007] P. Bao, T. K. Sinha, C-C. R Chen, J. R. Warmath, R. L. Galloway, and A. J. Herline. A prototype ultrasound-guided laparoscopic radiofrequency ablation system. Surg. Endosc., 21(1):7479, 2007. [Baumhauer2007] Baumhauer M, Simpfendörfer T, Schwarz R, Seitel M, Müller-Stich B P, Gutt C N, Rassweiler J, Meinzer HP, Wolf I. Soft Tissue Navigation For Laparoscopic Prostatectomy: Evaluation

• f Camera Pose Estimation for Enhanced Visualization. In Kevin R. Cleary, Michael I. Miga (Eds).
• Proc. SPIE Medical Imaging 2007: Visualization and Image-Guided Procedures, Vol. 6509, ArtNr.:

650911, 2007 [Baumhauer2008] M. Baumhauer, M. Feuerstein, H.-P. Meinzer, and J. Rassweiler. Navigation in endoscopic soft tissue surgery: perspectives and limitations. J. Endourol., 22(4):751766, 2008. [Belongie2000] Belongie S., Malik J., Puzicha J.: Shape context: A new descriptor for shape matching and

• bject recognition. In NIPS (2000), pp. 831–837.

[Berg2005] Berg A. C., Berg T. L., Malik J.: Shape matching and object recognition using low distortion

• correspondences. In Proc. IEEE Conf. on CVPR (2005), pp. 26–33.

[Besl1992] P. J. Besl and N. D. McKay. A method for registration of 3-d shapes. IEEE TPAMI, 14:239 256, 1992.

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Literature

[Cash2007] D. Cash, M. Miga, S. Glasgow, B. Dawant, L. Clements, Z. Cao, R. Galloway, and W. Chapman. Concepts and preliminary data toward the realization of image-guided liver surgery. J. Gastrointest. Surg., 11(7):844859, 2007. [Chui2003] H. Chui and A. Rangarajan, “A new point matching algorithm for non-rigid registration,”

• Comput. Vis. Image Underst. 89, pp. 114–141, February 2003.

[Danilchenko2011] A. Danilchenko and J. M. Fitzpatrick. General approach to rst-order error prediction in rigid point registration. IEEE TMI, 30(3):679693, 2011. [dosSantos2010] T. R. dos Santos, A. Seitel, H.-P. Meinzer, and L. Maier-Hein, “Correspondences search for surface-based intra-operative registration,” in MICCAI 2010, LNCS 6362, pp. 660–667, Springer, 2010. [Feuerstein2008] M. Feuerstein, T. Mussack, S. M. Heining, and N. Navab. Intraoperative laparoscope augmentation for port placement and resection planning in minimally invasive liver resection IEEE TMI, 27(3):355369, 2008 [Funkhouser2006] Funkhouser T., Shilane P.: Partial matching of 3D shapes with priority-driven search. In

• Proc. Symp. on Geom. Processing (SGP) (2006), pp. 131–142.

[Gelfand2005] Gelfand N., Mitra N. J., Guibas L. J., Pottmann H.: Robust global registration. In Proc.

• Symp. on Geom. Processing (SGP) (2005), pp. 197–206.

[Hildebrand2007] P. Hildebrand, M. Kleemann, U. J. Roblick, L. Mirow, C. Bürk, and H.-P. Bruch. Technical aspects and feasibility of laparoscopic ultrasound navigation in radiofrequency ablation of unresectable hepatic malignancies. J. Laparoendosc. Adv. Surg. Tech. A., 17(1):5357, 2007. [Johnson1997] A. E. Johnson „Spin-images: A representation for 3-d surface matching”, PhD thesis 1997 [Johnson1999] A. E. Johnson et al. „Using spin images for efficient object recognition in cluttered 3D scenes”, IEEE Transactions on Pattern Analysis and Machine Intelligence 21(5): 433-339, 1999

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Literature

[Kuhn1955] Kuhn, H. W. (1955), "The Hungarian method for the assignment problem", Naval Research Logistics Quarterly, 2:83–97. [Lipman2009] Lipman Y., Funkhouser T.: Möbius voting for surface correspondence. ACM Trans. on Graphics (Proc. SIGGRAPH) 28, 3 (2009).[ Maier-Hein2010] Maier-Hein, L., Schmidt, M., Franz, A., dos Santos, T., Seitel, A., J•ahne, B., Fitzpatrick, J., and Meinzer, H., Accounting for anisotropic noise in ne registration of time-of-ight range data with high-resolution surface data," in [MICCAI 2010], LNCS 6361, 251{258 (2010). [Rauth2007] T. P. Rauth, P. Q. Bao, R. L. Galloway,J. Bieszczad, E. M. Friets, D. A. Knaus, D. B. Kynor, and

• A. J. Herline. Laparoscopic surface scanning and subsurface targeting: Implications for image-guided

laparoscopic liver surgery. Surgery, 142(2):207 214, 2007. [Reuter2006] Martin Reuter, Franz-Erich Wolter, and Niklas Peinecke, Laplace–Beltrami spectra as ‘Shape-DNA’ of surfaces and solids Computer-Aided Design Volume 38, Issue 4, April 2006, Pages 342-366 Symposium on Solid and Physical Modeling [Rusinkiewicz2001] Rusinkiewicz S., Levoy M.: Efficient variants of the ICP algorithm. In Proc. 3rd Int.

• Conf. on 3D Digital Imaging and Modeling (2001), pp. 145–152.

[Sindram2010] D. Sindram, I. H. McKillop, J. B. Martinie, and D. A. Iannitti. Novel 3-d laparoscopic magnetic ultrasound image guidance for lesion targeting. HPB, 12(10):709716, 2010. [Sun2009] Sun J., Ovsjanikov M., Guibas L.: A concise and provably informative multi-scale signature based on heat diffusion. Computer Graphics Forum (Proc. SGP) 28, 5 (2009). [Tamaki2010] T. Tamaki, M. Abe, B. Raytchev, and K. Kaneda, “Softassign and EM-ICP on GPU,” International Conference on Natural Computation 0, pp. 179–183, 2010. [vanKaick2011] van Kaick, O., Zhang, H., Hamarneh, G. and Cohen-Or, D. (2011), A Survey on Shape

• Correspondence. Computer Graphics Forum. doi: 10.1111/j.1467-8659.2011.01884.x

Lena Maier-Hein – Registration Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Literature

[Zhang2008] Zhang H., Sheffer A., Cohen-Or D., Zhou Q., Van Kaick O., Tagliasacchi A.: Deformation- driven shape correspondence. Computer Graphics Forum (Proc. SGP) 27, 5 (2008), 1431–1439. [Zeng2010] Yun Zeng; Chaohui Wang; Yang Wang; Xianfeng Gu; Samaras, D.; Paragios, N.; Dense non-rigid surface registration using high-order graph matching, Computer Vision and Pattern Recognition, IEEE Computer Society Conference on, Vol. 0 (2010), pp. 382-389 [Zheng2010] Zheng Q., Sharf A., Tagliascchi A., Chen B., Zhang H., Sheffer A., Cohen-Or D.: Consensus skeleton for non-rigid space-time registration. Computer Graphics Forum (Proc. EUROGRAPHICS) 29, 2 (2010), 635–644.