Temporal Feature Tracking for Robotic Assisted Endoscopic Surgery - - PowerPoint PPT Presentation

▶
temporal feature tracking for robotic assisted endoscopic
SMART_READER_LITE
LIVE PREVIEW

Temporal Feature Tracking for Robotic Assisted Endoscopic Surgery - - PowerPoint PPT Presentation

May 21, 2023 128 likes •660 views

Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery Temporal Feature Tracking for Robotic Assisted Endoscopic Surgery Haytham Elhawary, Ph.D. Philips Research North America, Briarcliff, New York, USA Objectives of this talk

slide-1
SLIDE 1

Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery

Haytham Elhawary, Ph.D. Philips Research North America, Briarcliff, New York, USA

Temporal Feature Tracking for Robotic Assisted Endoscopic Surgery

slide-2
SLIDE 2

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Objectives of this talk

  • Understand the need to develop robust tracking algorithms

using uncalibrated monocular endoscopes

  • How to choose good features to track in your image
  • Explore the basics of optical flow algorithms for tracking
  • Methods to evaluate your feature tracking algorithm
  • Example applications
  • Conclusions
slide-3
SLIDE 3

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Why develop robust feature tracking algorithms?

You can do lots of cool stuff with it!

  • Motion analysis
  • Mosaicing or image stitching
  • Track moving targets
  • Motion compensation
  • 3D reconstruction
  • …
slide-4
SLIDE 4

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

The need for feature tracking

Lab set-up of the system

  • IBM/JHU LARS robot
  • 7 DOFs

(X-Y-Z stage, 3 rot, 1 insertion)

  • Remote center of motion (RCM)
  • Richard Wolf Inc endoscope
  • Monocular, direct
  • Totally uncalibrated
  • 352 x 240
  • Chamberlain Group

beating heart phantom

  • Realistic outer surface
slide-5
SLIDE 5

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

The need for feature tracking

Clinical requirements of the system

  • Use of a standard 2D monocular endoscope
  • Most used endoscope in manual laparoscopic procedures
  • No calibration of the endoscope
  • Tedious technical process not apt for OR
  • Allow intra-operative replacement of the endoscope
  • Change from angled and straight lens endoscopes
  • No marker placement on the heart surface
  • Adds complexity to the procedure
slide-6
SLIDE 6

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Feature Detection

“I like when a girl knows what she looks like and dresses to accentuate those features.“ Zac Efron

slide-7
SLIDE 7

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Choosing good features to track

Feature Detection

  • There are features in an image that are more prone to be

tracked successfully

  • Features have mathematical characteristics that fit the tracking

method (optical flow) better

  • Several methods: Moravec corner detection, Shi and Tomasi

941, and SURF based feature detector from Bay et al 20062 etc

1 J. Shi and C. Tomasi (June 1994). "Good Features to Track,“ 9th IEEE Conference on Computer Vision and Pattern Recognition.

  • 2H. Bay, et al., "SURF: Speeded Up Robust Features," in Computer Vision – ECCV 2006, vol. 3951, 2006, pp. 404-417.
slide-8
SLIDE 8

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Choosing good features to track

Feature Detection

Learning OpenCV, Gary Bradski & Adrian Kaehler, O’Reilly

slide-9
SLIDE 9

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Choosing good features to track

Feature Detection

  • Good features to track method (Shi and Tomasi 94)
  • Given a point in an image the Hessian matrix at

that point is:

  • The autocorrelation matrix of second order derivative images around a

small window at each point is calculated, and the eigenvalues computed.

  • Ratio between min and max Eigen values are calculated, and if above a

threshold, the point is considered good to track.

  • Implemented by GoodFeaturesToTrack function in OpenCV.

T t t t

y x x ] , [ =

→

) , ( y x I

            ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ =

→ → → → →

) ( ) ( ) ( ) ( ) (

2 2 2 2 2 2 t t t t t

x I y x I y x x I y x x I x x H

slide-10
SLIDE 10

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Choosing good features to track

GFT Feature Detection

  • Varied the ratio between max and min eigen values above which

a point is considered good to track

  • Defined high, medium and low quality points depending on ratio
slide-11
SLIDE 11

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Choosing good features to track

GFT Feature Detection

slide-12
SLIDE 12

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Choosing good features to track

Feature Detection

  • SURF feature detector method (Bay et al, 06, 08)
  • Speeded Up Robust Features (SURF) is a feature descriptor with 64 or 128

dimensions

  • SURF describes distribution of the intensity content within an interest

point neighbourhood, based on Haar wavelets (inspired by SIFT)

  • Detects blob like structures at locations where the determinant of the

Hessian is maximum – Hessian used is covariance of second order Gaussian image

  • Implemented in OpenCV
slide-13
SLIDE 13

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Choosing good features to track

SURF Feature Detection

  • Varied the threshold of the determinant above which a point is

considered good to track

slide-14
SLIDE 14

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Choosing good features to track

SURF Feature Detection

slide-15
SLIDE 15

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Choosing good features to track

SURF and GFT Feature Detection – not the same points!

GFT detector SURF detector

slide-16
SLIDE 16

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Feature Tracking (optical flow)

“Are you stalking me? Because that would be super.“ Ryan Reynolds

slide-17
SLIDE 17

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Optical flow algorithm

  • Optical flow allows you to detect the apparent motion of certain

patterns in an image (displacement vectors)

  • Iterative differential method for tracking point correspondences

between 2 frames

  • Dense and sparse optical flow methods available, although

dense methods are computationally expensive

  • Lucas-Kanade algorithm1 (81) is one of the most popular sparse

tracking methods

  • Dense methods include Horn-Schunck2 (81)

1 B. D. Lucas and T. Kanade (1981), An iterative image registration technique with an application to stereo vision. Proceedings of Imaging

Understanding Workshop, pages 121--130

2 B.K.P. Horn and B.G. Schunck, "Determining optical flow." Artificial Intelligence, vol 17, pp 185-203, 1981

slide-18
SLIDE 18

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Optical flow algorithm

  • Based on 3 assumptions:
  • Brightness constancy:
  • Small motions: motion is slow compared to frame rate (allows

approximating the derivative of intensity over time)

  • Spatial coherence: neighbouring points in a scene belong to the same

surface and have similar motion

Lucas-Kanade method

) , , ( ) , , ( t t v y u x I t y x I ∆ + + + = = + +

t y x

I v I u I

(1)

slide-19
SLIDE 19

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Optical flow algorithm

  • Using a window of pixels w=[wx,wy], (assumption 3) that move in

the same manner, equation (1) can be applied for each pixel in the window to obtain u and v → Minimization problem solved with least squares

  • Implemented in OpenCV
  • Pyramidal version of Lucas-Kanade1 solves the problems of

large motions falling outside of the local window

Lucas-Kanade method

1 J.-Y. Bouguet, "Pyramidal Implementation of the Lucas Kanade Feature Tracker Description of the algorithm," Intel Corporation,

Microprocessor Research Labs, unpublished.

) ( ) ( ) (

t t t t w x x w x x w y y w y y t t t

v x I x I v e

x t x t y t y t

  • +

− =

∆ + + = − = + = − =

∑ ∑

slide-20
SLIDE 20

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Optical flow algorithm

  • Feature Tracking on a beating heart
slide-21
SLIDE 21

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Optical flow algorithm

  • Once tracking is complete, try checking for errors in algorithm.
  • Check for convergence of the LK method
  • Impose a maximum expected inter-frame motion threshold
  • Keep previous frame in memory in case the current frame gives a tracking

error

  • Vary window size to find optimal for your type of images and motion
  • Use periodic nature of motion (if applicable)
  • Etc...

Lucas-Kanade method – tricks!

slide-22
SLIDE 22

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Evaluating Feature Tracking

“Probably about 10-15% of the time it's accurate.“ Justin Timberlake

slide-23
SLIDE 23

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Methods to evaluate feature tracking

  • It is difficult to evaluate the accuracy of a feature tracking

method due to lack of gold standard

  • Segmenting known location of feature in each frame too

tedious and not accurate!

  • Other surrogate metrics are required to evaluate accuracy
  • Other metrics that can be used to evaluate feature tracking:
  • Robustness (how often it fails)
  • Periodic repeatability (for periodic motions)
  • Frequency spectrum
  • Others...
slide-24
SLIDE 24

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Methods to evaluate feature tracking

Artificial Images with known transformation

  • Apply a known transformation to a single image frame, and

compare tracked feature with known location of feature

  • Obtain statistics: rms error, standard deviation, conf interval
  • Just a sanity check for the algorithm
slide-25
SLIDE 25

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Methods to evaluate feature tracking

Artificial Images with known transformation

  • Parameters: 31x31 window, 5

pyramidal levels, interframe thresh 10 px, “high quality” 30 points

  • Motion per frame: 2.5 pixel and 0.25

deg

  • Number of rejections with LK-SURF

(1) was lower than with LK-GFT (2)

  • Both performed at real-time speeds

>30Hz

slide-26
SLIDE 26

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Methods to evaluate feature tracking

Robotic Guided Endoscope

slide-27
SLIDE 27

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Methods to evaluate feature tracking

Experiments with beating heart phantom

  • Same algorithm parameters as in previous experiments
  • Tracking of 30 points selected manually and 30 points selected

with SURF feature detection

  • Recovery of points if rejection occurs using previous frame

SURF detected points Manually selected points

slide-28
SLIDE 28

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Methods to evaluate feature tracking

Performance Metrics with no ground-truth

  • Visual inspection of the trajectory of the point (distance

between point in current frame and point in start frame)

  • Difference in period average – indicates drift of the point over

time

  • Frequency analysis to discern presence of heart-beat frequency
  • Power ratio – checks presence of low frequency components in

the spectrum

slide-29
SLIDE 29

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Methods to evaluate feature tracking

Performance Metrics with no ground-truth

  • Visual Inspection of trajectory and period average
slide-30
SLIDE 30

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Methods to evaluate feature tracking

Performance Metrics with no ground-truth

  • Fourier frequency analysis

Presence of heart beat peak

slide-31
SLIDE 31

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Methods to evaluate feature tracking

Performance Metrics with no ground-truth

  • Fourier frequency analysis can be misleading!

Presence of heart beat peak

slide-32
SLIDE 32

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Methods to evaluate feature tracking

Performance Metrics with no ground-truth

  • Power Ratio

∫ ∫

p p

f f

f P f P

8 .

) ( ) (

  • Indicates the presence of low frequency

components in the spectrum after removing DC component

  • Value between 0 and 1
  • Can better detect drift and other tracking

deviations such as jumps

slide-33
SLIDE 33

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Methods to evaluate feature tracking

Performance Metrics with no ground-truth

  • Power Ratio
slide-34
SLIDE 34

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Methods to evaluate feature tracking

Performance Metrics with no ground-truth

  • Power Ratio
slide-35
SLIDE 35

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Methods to evaluate feature tracking

Performance Metrics with no ground-truth

  • Power Ratio
slide-36
SLIDE 36

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Methods to evaluate feature tracking

Performance Metrics with no ground-truth

  • Tracking results for beating heart phantom
slide-37
SLIDE 37

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Example Applications

“In theory, there is no difference between theory and practice. In practice, there is.“ Yogi Berra

slide-38
SLIDE 38

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Example Applications

Robotic Guided Endoscope

slide-39
SLIDE 39

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Example Applications

Robotic Guided Endoscope

  • Endoscope tip located at 50mm from heart phantom
  • RCM located at 50mm from endoscope tip
  • Robot moves while tracking takes place, with endoscope tip

moving at 5mm/s

  • Robot moves to a location and then moves back to start point
slide-40
SLIDE 40

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Example Applications

Robotic Guided Endoscope

  • Tracking algorithms with LK-SURF and LK-manual
  • Variation in inter-frame threshold (0, 10, 20, 30 and 40 pixels)
  • Variation in window sizes (31, 41 and 51)

SURF detected points Manually selected points

slide-41
SLIDE 41

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Example Applications

Robotic Guided Endoscope

  • Performance metrics:
  • Previous metrics: power ratio, frequency analysis, mean period average
  • Step error: average PI and PII subtracted
slide-42
SLIDE 42

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Example Applications

Robotic Guided Endoscope

  • Evaluation Results

SURFLK51_th20 SURFLK31_th20

slide-43
SLIDE 43

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Example Applications

Robotic Guided Endoscope

  • Evaluation Results

LK31_th20 LK31_th10

slide-44
SLIDE 44

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Example Applications

Robotic Guided Endoscope

  • Evaluation Results

LK31_th20 LK31_th10

slide-45
SLIDE 45

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Example Applications

Robotic Guided Endoscope

  • Results are best with SURF detector for all metrics
slide-46
SLIDE 46

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Example Applications

Visual Servoing in Robotic Guided Endoscopy

slide-47
SLIDE 47

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Example Applications

Visual Servoing in Robotic Guided Endoscopy

  • Visual Servoing allows selection of a point in the image, and the

motion of the robot to locate the point at the centre of image

slide-48
SLIDE 48

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Example Applications

Visual Servoing in Robotic Guided Endoscopy

  • Latency 0.438s
  • Reach all points in

the image in <3s

  • Drift (points with

residual error above 0.5s) < 6 points

slide-49
SLIDE 49

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Example Applications

Image Stitching

slide-50
SLIDE 50

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Conclusions

  • Temporal feature tracking forms the basis of many applications
  • Selection of good features to track (SURF) is very important for

accuracy

  • Optical flow algorithms (Lucas Kanade) are surprisingly robust in

spite of their constraints

  • Tricks to improve accuracy: inter-frame motion threshold and

window size

  • Lack of gold standard makes evaluation difficult, so surrogate

metrics must be used

  • Combination of metrics involving error in position and frequency

analysis

slide-51
SLIDE 51

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Conclusions

Related Publications

  • Haytham Elhawary, Aleksandra Popovic, “Robust Feature Tracking on

the Beating Heart for a Robotic Guided Endoscope”, IJMRCAS, 2011, in press

  • Aleksandra Popovic, Paul Thienphrapa, “An Approach to Robotic

Guidance of an Uncalibrated Endoscope in Beating Heart Surgery”, Proceedings of the 2010 3rd IEEE RAS & EMBS, International Conference on Biomedical Robotics and Biomechatronics, Tokyo, Japan, September 26-29, 2010

  • Jessie Young, Haytham Elhawary, Aleksandra Popovic, “Integration of

a Robotic Arm with the Surgical Assistant Workstation Software Framework “, MICCAI 2011 workshop Systems and Architectures for Computer Assisted Interventions

slide-52
SLIDE 52

Haytham Elhawary – Temporal Feature Tracking using an Endoscope Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

22ndSeptember ,Toronto, Canada

Contact Details

Haytham Elhawary, PhD Member Research Staff Clinical Informatics, Interventional and Translational Solutions Philips Research North America 345 Scarborough Road, Briarcliff Manor, NY 10510 Tel: (914) 945-6175 -- Fax: (914) 945-6330 Email: Haytham.Elhawary@Philips.com