Medical Image Processing and Analysis MSc in Biomedical Engineering - - PowerPoint PPT Presentation

João Manuel R. S. Tavares tavares@fe.up.pt www.fe.up.pt/~tavares

December 10, 2015

Medical Image Processing and Analysis

MSc in Biomedical Engineering

Outline

• Introduction
• Segmentation
• Motion Tracking
• Analysis of Objects: Matching, Morphing and

Registration

• 3D Reconstruction
• Conclusions
• Research Team
• Publications & Events

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 2

Introduction

Presentation

• Associate Professor at FEUP (DEMec)
• Senior Research and Projects Coordinator of the

Institute of Science and Innovation in Mechanical and Industrial Engineering (INEGI)

• Habilitation in Mechanical Engineering from the

University of Porto

• PhD and MSc degrees in Electrical and Computer

Engineering from FEUP

• BSc degree in Mechanical Engineering from FEUP
• Research Areas: Image Processing and Analysis,

Medical Imaging, Biomechanics, Human Posture and Control, Product Development

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 4

Introduction

• The researchersof Image Processing and

Analysis aim the development of algorithms to perform fully or semi-automatically tasks performed by the (quite complex) human vision system

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 5

Original images Computational 3D voxelized and poligonized models built

Azevedo et al. (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(3):359-369

Introduction

• Image processing and analysis are topics of the

most importance for our Society

• Algorithms of image processing and analysis are

frequently used, for example, in:

– Natural Sciences – Sports – Biology – Industry – Engineering – Medicine

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 6

Introduction

• Examples of common tasks involving algorithms
• f image processing and analysis:

– Noise removal – Geometric correction – Segmentation, recognition (2D-4D) – Motion and deformation tracking and analysis, including matching, registration and morphing – 3D reconstruction – Assisted medical diagnosis and intervention

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 7

• Image: a matrix with n rows and m columns (and l

in 3D), being each basic element known as pixel (or voxel in 3D)

Introduction

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 8

• Image Processing: by applying mathematical
• perations/rules using the values of the image

pixels (or voxels in 3D) in the Cartesian or in other domain

Introduction

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 9

∗ −1 −2 −1 +1 +2 +1 ⎡ ⎣ ⎢ ⎢ ⎢ ⎤ ⎦ ⎥ ⎥ ⎥ = Gy ∗ −1 +1 −2 2 −1 +1 ⎡ ⎣ ⎢ ⎢ ⎢ ⎤ ⎦ ⎥ ⎥ ⎥ = Gx Gx

2 + Gy 2 = G

Sobel operator

∗ ( denotes convolution)

• Image Acquisition: a sensor captures the

energy reflected or emitted by the imaged object

Introduction

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 10 http://what-when-how.com/introduction-to-video-and-image-processing/image-acquisition-introduction-to- video-and-image-processing-part-1

• Difficulties: noise, artifacts, occlusion, poor

illumination, reflections, complex objects and backgrounds

Introduction

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 11

https://rahaddadi.files.wordpress.com/2011/05/face_black_and_white_optical_illusion_cool-s453x562-92306-5803.jpg http://s1.cdn.autoevolution.com/images/news/the-longest-traffic-jam-in-history-12-days-62-mile-long-47237_1.jpg

Introduction

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 12

Image(s) enhancement / correction Image(s) segmentation / features extraction tracking matching morphing Image(s) motion analysis registration image (pre)processing image analysis / computational vision 3D vision computer vision

Usual Pipeline

Introduction

• (Pre)processing of noisy images using an

intelligent worm

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 13

Original images (a), noisy corrupted images (b) and smoothed images using different smoothing methods (c-h)

Araujo et al. (2014) Expert Systems with Applications 41(13):5892-5906

Segmentation

Segmentation

• It is intended to identify in a fully or semi-

automatically manner objects (2D/3D) presented in static images or in image sequences

• The most usual methodologies are based on

thresholding, region growing, template matching, statistical, geometric or physical modeling, or artificial classifiers

• It is one of the most usual operations involved in

the computational analysis of objects in images

• Frequent problems: noise, artifacts, low resolution,

reduced contrast, shapes not previously known,

• cclusion, multiple objects, etc.

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 15

Segmentation

• Image segmentation by threshold (binarization)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 16 Ma et al. (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(2):235-246

Segmentation

• Example: segmentation of contours in dynamic

pedobarography (Otsu’s method, morphologic dilation,

xor operation)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 17

Original images Segmented images

Bastos & Tavares (2004) Lecture Notes in Computer Science 3179:39-50

camera mirror contact layer + glass

reflected light glass pressure

• paque layer

lamp lamp transparent layer

Segmentation

• Image segmentation by region growing

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 18 Ma et al. (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(2):235-246

Segmentation

• Example: segmentation of ear structures (region

growing)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 19

Region Growing, x=215; y=254

Segmentation obtained (bony labyrinth)

Barroso et al. (2011) CNME 2011 Ferreira et al. (2014) Computer Methods in Biomechanics and Biomedical Engineering 17(8):888-904

X: 254 Y: 214 Index: 116.7 RGB: 0.459, 0.459, 0.459

Original Image

Segmentation

• Segmentation based on neuronal networks

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 20 Albuquerque et al. (2008) Nondestructive Testing and Evaluation 23(4):273-283 Albuquerque et al. (2009) NDT & E International 42(7):644-651

Original metallographic images After segmentation (material microstructures )

Segmentation

• Segmentation of objects based on deformable

templates

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 21 Carvalho & Tavares (2006) CompIMAGE 2006, 129-134 Carvalho & Tavares (2007) VipIMAGE 2007, 209-215

Example of a deformable template (for the eye)

Segmentation

• Example: segmentation of eye

features (deformable geometric template)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 22

Original image and associated force (or energy) fields Segmentation of the iris using a deformable template (a circle) Segmentation of an eye using an deformable template

Carvalho & Tavares (2006) CompIMAGE 2006, 129-134 Carvalho & Tavares (2007) VipIMAGE 2007, 209-215

• Statistical modeling of objects (point distribution

models)

Segmentation

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 23 Vasconcelos & Tavares (2008) Computer Modeling in Engineering & Sciences 36(3):213-241

Segmentation

• Segmentation of based on active shape models

(point distribution models, optimization)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 24 Vasconcelos & Tavares (2008) Computer Modeling in Engineering & Sciences 36(3):213-241

Segmentation

• Example: analysis of the vocal tract during

speech production from MR images (active shape model)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 25

Original image Final segmentation

Vasconcelos et al. (2011) Journal of Voice 25(6):732-742

• Segmentation of objects based on active

appearance models (statistical models, optimization)

Segmentation

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 26 Vasconcelos & Tavares (2008) Computer Modeling in Engineering & Sciences 36(3):213-241

Segmentation

• Example: analysis of the vocal tract shape

during speech production from MR images

(active appearance model)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 27

Initial segmentation Final segmentation

Vasconcelos et al. (2011) Journal of Engineering in Medicine 225(1):68-76 Vasconcelos et al. (2012) Journal of Engineering in Medicine 226(3):185-196

• Segmentation of objects based on active

contours (i.e. snakes – parametric models)

Segmentation

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 28 Tavares et al. (2009) International Journal for Computational Vision and Biomechanics 2(2):209-220

Esnake = Eint

s=0 1

∫

(v(s))+ Eext(v(s))ds Eint = α(s) dv(s) ds

2

+ β(s) d 2v(s) ds2

2

Segmentation

• Example: segmentation of medical images

(active contours - snakes)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 29

Initial contour Final contour

Tavares et al. (2009) International Journal for Computational Vision and Biomechanics 2(2):209-220 Gonçalves et al. (2008) Computer Modeling in Engineering & Sciences 32(1):45-55

Segmentation

• Segmentation of objects based on the level-set

method (geometrical models)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 30 Ma et al. (2010) Medical Engineering & Physics 32(7):766-774 Ma et al. (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(2):235-246

∂φ ∂t + F Δφ = 0

Typical form of the motion equation:

Segmentation

• Example: segmentation of the carotid

bifurcation in a Doppler image (active contour / level-

set model)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 31

Segmentation using the contour active method (Yessi’s model) Segmentation using the level-set method (Chan-Vese’s model)

Silva et al. (2011) VipIMAGE 2011, 117-122 Santos et al. (2013) Expert Systems with Applications 40(16):6570-6579

• Example: segmentation of skin pigmented

lesions in dermoscopic images (color spaces, level-

set model)

Segmentation

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 32

Segmentation examples under different imaging conditions and different types of skin pigmented lesions Illustration of the segmentation process

Ma & Tavares (2015) IEEE J. of Biomedical and Health Informatics, DOI: 10.1109/JBHI.2015.2390032 (in press) Filho et al. (2015) Journal of Medical Systems 39(11):177

Segmentation

• Segmentation of objects based on the level set

method with a prior knowledge

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 33 Ma et al. (2010) Medical Engineering & Physics 32(7):766-774 Ma et al. (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(2):235-246

Segmentation

• Example: segmentation of the pelvic floor in MR

images (level-set model, a prior knowledge, shape

influence field)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 34

Pelvic floor segmented

Ma et al. (2010) Medical Engineering & Physics 32(7):766-774

Segmentation

• Example: segmentation simultaneously of three
• rgans of the female pelvic cavity in MRI

images (level-set method, a prior knowledge)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 35 Ma et al. (2013) Computers in Biology and Medicine 43(4):248-258 Ma et al. (2012) The Int. Journal for Numerical Methods in Biomedical Engineering 28(6-7):714-726

Segmentation of the bladder, vagina and rectum in pelvic cavity images (3 examples)

• Example: segmentation of the bladder walls in

MR images (level-set method, a prior knowledge)

Segmentation

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 36 Ma et al. (2011) Annals of Biomedical Engineering 39(8):2287-2297

Segmentation of the interior and external walls of the bladder (3 examples)

Motion Tracking

Motion Tracking

• It is intended to track the motion (and/or the

deformation) of objects in image sequences

• In this area, the methodologies based on optical

flow, block matching and stochastic methods are widespread

• Usually, it concerns the estimation of the

motion involved, the management of the features being tracked, the analysis of the motion tracked and its quantification

• Usual problems: non-rigid motions, geometric

distortion, non-constant illumination, occlusion, noise, multiple objects, etc.

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 38

Motion Tracking

• Computational framework

to track features in image sequences (Kalman Filter or

Unscented Kalman Filter,

• ptimization, Mahalanobis

distance, management model)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 39 Pinho et al. (2007) Int. Journal of Simulation Modelling 6(2):84-92 Pinho & Tavares (2009) VipIMAGE 2009, 299-304 Pinho & Tavares (2009) Computer Modeling in Engineering & Sciences 46(1):51-75

Motion Tracking

• Example: tracking marks in gait analysis (Kalman

filter, Mahalanobis distance, optimization, management model)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 40

Prediction Uncertainty Area Measurement Correspondence Result (5 frames)

Pinho et al. (2005) ICCB 2005, 915-926 Pinho & Tavares (2009) Computer Modeling in Engineering & Sciences 46(1):51-75

Motion Tracking

• Example: tracking marks in order to detect gait

events (Kalman filter, Mahalanobis

distance, optimization)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 41 Sousa et al. (2007) ISHF2007, 331-340 Sousa et al. (2007) ICCB2007, 291-296

Motion Tracking

• Example: tracking mice in long image

sequences (Kalman filter, Mahalanobis distance,

• ptimization, management model)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 42

(547 frames)

Pinho et al. (2005) LSCCS, Vol. 4A:463-466 Pinho et al. (2007) International Journal of Simulation Modelling 6(2):84-92

Analysis of Objects: Matching

Matching

• The goal is to establish correspondence

between image features

• It is regularly used in the computational analysis of
• bjects in images, for example, to register (i.e.

align) objects, recognize objects, attain 3D information, analyze tracked motion, etc.

• Generally, it is achieved by considering

invariant characteristics, as curvature, or displacements in a global/eigen spaces (like in modal space)

• Common problems: occlusion, non-rigid

deformations, high shape variations, etc.

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 44

Matching

• Based on physical or geometrical modeling and

modal matching

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 45

Modeling (physical or geometrical) Eigenvalues / eigenvectors computation Matching matrix assembly Contour 1 Contour 2 Matches achievement (optimization) Modeling (physical or geometrical) Eigenvalues / eigenvectors computation

Bastos & Tavares (2006) Inverse Problems in Science and Engineering 14(5):529-541 Tavares & Bastos (2010) Progress in Computer Vision and Image Analysis 339-368

Matching

• Example: matching contours in dynamic

pedobarography (FEM modeling, modal matching,

• ptimization)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 46

Original images Matched contours

camera mirror contact layer + glass

reflected light glass pressure

• paque layer

lamp lamp transparent layer

Bastos & Tavares (2004) LNCS 3179:39-50 Tavares & Bastos (2010) Progress in Computer Vision and Image Analysis, 339-368

Matching

• Example: matching contours and surfaces in

dynamic pedobarography (FEM modeling, modal

analysis, optimization)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 47

Image of dynamic pedobarography

Tavares & Bastos (2005) Electronic Letters on Computer Vision and Image Analysis 5(3):1-20

Matching found between two contours Matching found between two intensity (pressure) surfaces (2 views) Matching found between iso-contours (2 views)

Analysis of Objects: Morphing

Morphing (i.e. simulation)

• It is an especially used in Computer Graphics, but

also very useful in the analysis of objects in images, for example, to estimate the deformation involved between objects or between configurations of an

• bject, to simulate the transitions between shapes

acquired with a high temporal gap, etc.

• Normally, it is attained by considering simple

geometric transformations

• However, when it must be considered the real

behavior of the objects, physical based methodologies and modeling as, for example, FEM, should be considered

– Common difficulties are related to the estimation of the involved forces and with the properties of the adopted (virtual) material – The adequate matching of the objects is crucial

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 49

Morphing

• Physical morphing/simulation of contours (FEM

modeling, modal analysis, optimization, Lagrange’s eq.)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 50

Morphing

• Example: morphing contours (FEM modeling, modal

analysis, optimization, Lagrange’s equation)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 51

Matching found Deformations estimated

Gonçalves et al. (2008) Computer Modeling in Engineering & Sciences 32(1):45-55

Original images

Analysis of Objects: Registration

Registration

• It is commonly required in order to compare
• bjects acquired at different time instants or

according to distinct conditions

• It is essential, for example, in Medicine to follow

up the evaluation of diseases

• Usually, it is achieved by considering characteristic

features, as points of high curvature, and their matching, or by minimization of a similarity measure, followed by the estimation of the involved transformation

• Common problems: key and invariant features not

easily detected, occlusion, non-rigid deformations, severe shape variations, etc.

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 53

Registration

• Registration based on contours matching,
• ptimization and dynamic programming

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 54

The cost matrix is built based on geometric or physical principles The matching is found based on the minimization

• f the sum of the costs

associated to the possible correspondences To search for the best matching is used an

• ptimization assignment

algorithm

Bastos & Tavares (2006) Inverse Problems in Science and Engineering 14(5):529-541 Oliveira & Tavares (2009) Computer Modeling in Engineering & Sciences 43(1):91-110 Oliveira, Tavares, Pataky (2009) Journal of Biomechanics 42(15):2620-2623

Registration

• Example: registration of pedobarography

images (geometrical modeling, matching, optimization,

dynamic programming)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 55

Original images and contours Contours and images before and after the registration

Oliveira et al. (2009) Journal of Biomechanics 42(15):2620-2623

Registration: 2D, monomodal, intrasubject Processing time: 0.5 s (AMD Turion64, 2.0 GHz, 1.0 GB of RAM) Images dimension: 217x140 pixels

Fixed image and contour (MRI) Moving image and contour (MRI) Overlapped images before the registration Overlapped images after the registration Difference between the images after the registration Correspondences found between the Corpus Callosum contours Oliveira & Tavares (2014) Computer Methods in Biomechanics and Biomedical Engineering 17(2):73-93

Registration

• Example: registration of brain MR images

(geometrical modeling, matching, optimization, dynamic programming)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 56

Registration

• Registration of images based on Fourier transf.

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 57

Original images Images before and after the registration

Oliveira, Pataky, Tavares (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(6):731-740

Registration: 2D, monomodal, intrasubject Processing time: 2.1 s (AMD Turion64, 2.0 GHz, 1.0 GB of RAM) Images dimension: 221x257 pixels

Registration

• Example: registration of brain MR images

(Fourier transform)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 58

Registration

• Registration based on the iterative search for the

parameters of the transformation that optimizes a similarity measure between the input images

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 59 Moving image Fixed image Pre-registration transformation (optional) Interpolator Similarity measure Optimizer Geometric transformation

The optimization algorithm stops when a similarity criterion is achieved

Oliveira & Tavares (2014) Computer Methods in Biomechanics and Biomedical Engineering 17(2):73-93

Registration

• Example: registration of pedobarography

images (Hybrid method: Fourier transform based

registration + optimization of a similarity measure)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 60

Original images Images before and after the registration

Oliveira & Tavares (2012) Medical & Biological Engineering & Computing 49,(3):313-323

Registration: 2D, multimodal, intrasubject (without pre-registration) Similarity measure: MI Processing time: 4.6 s (AMD Turion64, 2.0 GHz, 1.0 GB of RAM) Images dimension: 246x234 pixels

Oliveira & Tavares (2014) Computer Methods in Biomechanics and Biomedical Engineering 17(2):73-93

Registration

• Example: registration/fusion of head CT/MR

images (optimization of a similarity measure)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 61

Registration

• Example: applications in plantar pressure

image studies

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 62 Oliveira, Sousa, Santos, Tavares (2012) Computer Methods in Biomechanics and Biomedical Engineering 15(11):1181-1188

A computational platform was developed to assist biomechanical studies that can be used for:

• Foot segmentation
• Foot classification: left/right,

high arched, flat, normal, …

• Foot axis computation
• Footprint indices

computation

• Posterior statistical analysis

“Checkerboard” of the slices before the registration (CT/MRI-PD, brain)

F F F F F F F F M M M M M M M M

(The “checkerboard” slice is built by interchanging square patches of both slices and preserving their original spatial position in the fixed (F) and moving (M) slices)

Registration

• Example: registration/fusion using iterative optimization

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 63

Registration: 3D, multimodal, intrasubject; Similarity measure: MI Checkerboard of the slices after the registration (CT/MRI-PD, brain)

Registration

• Example: registration/fusion using iterative opt. (cont.)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 64

Checkerboard of the slices (CT, thorax, Δt: 8.5 months) before the registration

Oliveira & Tavares (2014) Computer Methods in Biomechanics and Biomedical Engineering 17(2):73-93

Registration

• Example: registration using iterative optimization

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 65

Registration: 3D, monomodal, intrasubject; Similatity measure: MI Checkerboard of the slices (CT, thorax, Δt: 8.5 months) after the registration

Registration

• Example: registration using iterative optimization (cont.)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 66

Normal Alzheimer Idiopathic Parkinsonism Essential tremor

Registration

• Example: application on brain DaTSCAN SPECT

images

Brain DaTSCAN SPECT images are used to assist the diagnosis of the Parkinson’s disease and to distinguish it from other degenerative

• diseases. The solution developed is able to:

– Segment the relevant areas and perform dimensional analysis – Quantify the binding potential of the basal ganglia – Computation of statistical data relatively to a reference population – Image classification for diagnosis purposes

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 67

Mean slice from the population used as reference Corresponding slice of a patient Difference of intensities Z-scores mapping over the slice (red – high Z-

scores) (The blue rectangles represent the 3D ROIs used to compute the binding potentials) Oliveira et al. (2014) The Quarterly Journal of Nuclear Medicine and Molecular Imaging 58(1):74-84

Registration

• Example: application on brain DaTSCAN SPECT

images

3D volume images are automatically registered and statistical analysis relatively to a reference population can be attained

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 68

Basal ganglia from a mean image of a normal population Basal ganglia from a patient with idiopathic Parkinson’s disease Basal ganglia from a patient with vascular Parkinson’s disease Oliveira et al. (2014) The Quarterly Journal of Nuclear Medicine and Molecular Imaging 58(1):74-84

Registration

• Example: application on brain DaTSCAN SPECT

images

Basal ganglia 3D shape reconstruction and quantification

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 69

Three slices (coronal, sagittal and axial) after registration and identification of the potential lesion 3D visualization after CT/SPECT fusion (the lesion identified in the SPECT slices is indicated)

Registration

• Example: SPECT/CT fusion

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 70

Fully automated segmentation and classification of the images based on image registration and an artificial classifier

Template image (top), segmented image (bottom-left) and artery mapping (bottom-right)

Registration

• Example: application on gated myocardial

perfusion SPECT images

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 71

Oliveira, Faria, Tavares (2014) Proceedings of the Institution of Mechanical Engineers, Part H: Journal of Engineering in Medicine 228(8):810-818 3D surface of the incus and malleus surface built TC slices with the incus and malleus ossicles (inside the red ellipse) to be segmented

Registration

• Example: application in the fully automated

segmentation of the incus and malleus ear

• ssicles in conventional CT images

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 72

Registration

• Registration of image sequences: spatial &

temporal registration

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 73 Moving sequence Fixed sequence Apply the spatio & temporal transformation Compute the similarity measure Optimizer Build the spatio & temporal transformation Oliveira et al. (2011) Medical & Biological Engineering & Computing 49(7):843-850 Oliveira & Tavares (2013) Medical & Biological Engineering & Computing 51(3):267-276 Build the temporal representative images Search for the transformation that register the temporal representative images Estimate the linear temporal registration Pre-registration Registration optimization

Registration

• Example: registration of image sequences in

dynamic pedobarography (spatial & temporal

registration)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 74

74

Device: Light reflection (25 fps, resolution 30 pixels/cm2) Image similarity measure: MSD Sequences dimension: 160x288x22, 160x288x25 Processing time: 1 min (using an AMD Turion64, 2.0 GHz, 1.0 GB of RAM) Template sequence Source sequence Overlapped sequences Before the registration After the registration

Registration

• Example: registration of image sequences in

dynamic pedobarography (spatial & temporal registration)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 75

Device: EMED (25 fps, resolution: 2 pixels/cm2, images dimension: 32x55x13; 32x55x18) Registration: rigid (spatial), polynomial (temporal); similarity measure: MSD Processing time: 4 s - AMD Turion64, 2.0 GHz, 1.0 GB of RAM Fixed sequence Moving sequence Overlapped sequences Before the registration After the registration

3D Reconstruction

3D Reconstruction

• It is intended to accomplish the 3D reconstruction of
• bjects or scenes from images
• In this area, the following methodologies are common:

inner shapes – 2D segmentation (contours, for example) and data interpolation, or 3D segmentation; external shapes – active techniques (with energy projection or relative motion), passive techniques (without energy projection nor relative motion) and of space carving; deformation of 3D deformable models prebuilt

• Usually, it involves tasks of camera calibration, data

segmentation, matching, triangulation, registration and fusion

• Common problems: geometric distortion, insufficient or

unstable illumination, occlusion, noise, multiple objects, complex shapes and topologies, etc.

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 77

3D Reconstruction

• 3D reconstruction of organs from medical images

based on 2D segmentation, loft (marching cubes) and smooth

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 78

Reconstructed organs from pelvic cavity

Pimenta et al. (2006) CompIMAGE 2006, 343-348 Alexandre et al. (2007) VipIMAGE 2007, 359-362

3D Reconstruction

• Example: 3D reconstruction from multiple

views (registration/fusion)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 79

Axial and sagittal T2-weighted MR images 3D Reconstruction of the bladder by fusion data from the axial and sagittal images (2 views)

Ma et al. (2013) Medical Engineering & Physics 35(12):1819-1824

3D Reconstruction

• 3D reconstruction of scenes using techniques of

active vision (dense

stereo vision)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 80

Disparity map

• btained

Original image pair

Azevedo et al. (2006) VISAPP 2006, 383-388

3D Reconstruction

• 3D reconstruction of objects by space carving

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 81 Azevedo et al. (2008) Advances in Computational Vision and Medical Image Processing: Methods and Applications, 117-136

Pattern and object turntable image sequence Pattern image sequence Background/object segmentation Camera calibration Volumetric 3D reconstruction 3D model polygonization

3D Reconstruction

• Example: 3D reconstruction of objects by space

carving

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 82 Azevedo et al. (2008) Advances in Computational Vision and Medical Image Processing: Methods and Applications, 117-136 Azevedo et al. (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(3):359-369

Original images Computational 3D models built voxelized and poligonized

3D Reconstruction

• Example: 3D reconstruction of objects by space

carving

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 83

Original images Computational 3D models built voxelized and poligonized

Azevedo et al. (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(3):359-369

3D Reconstruction

• Example: 3D reconstruction of the spine from

two orthogonal X-ray images by adjusting a deformable model (atlas)

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 84 Moura et al. (2010) Computer Modeling in Engineering & Sciences 60(2):115-138 Moura et al. (2011) Medical Engineering & Physics 33(8):924-933

Interface developed Adjusted model (2 views) and reconstruction obtained

Conclusions

Conclusions

• The area of image processing and analysis is very

complex and demand, but of raised importance in many domains

• Numerous hard challenges exist, as for example, adverse

conditions in the image acquisition process, occlusion,

• bjects with complicate shapes, with topological variations
• r undergoing complex motion or deformation
• Considerable work has already been developed, but

important and complex goals still to be reached

• Methods and methodologies of other research areas,

as of Mathematics, Computational Mechanics, Medicine and Biology, can contribute significantly for their reaching

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 86

The work presented has been done with the support of Fundação Gulbenkian, CNPq, CAPES, and FCT, mainly trough the funding of the research projects:

– PTDC/BBB-BMD/3088/2012 – PTDC/SAU-BEB/102547/2008 – PTDC/SAU-BEB/104992/2008 – PTDC/EEA-CRO/103320/2008 – UTAustin/CA/0047/2008 – UTAustin/MAT/0009/2008 – PDTC/EME-PME/81229/2006 – PDTC/SAU-BEB/71459/2006 – POSC/EEA-SRI/55386/2004

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 87

Acknowledgments

Research Team

(Image Processing and Analysis)

Research Team (Image Processing and Analysis)

• Post-Doc students (5):

– Finished: Alexandre Carvalho, Simone Prado, Mercedes Filho – In course: Zhen Ma; Pedro Predosa

• PhD students (15):

– Finished: Zhen Ma, Francisco Oliveira, Teresa Azevedo, Daniel Moura, Sandra Rua, Maria Vasconcelos – In course: João Nunes, Alex Araujo, Carlos Gulo, Roberta Oliveira, Danilo Jodas, Pedro Morais, Andre Pilastri, Nuno Sousa, Domingos Vieira

• MSc students (30):

– Finished: Raquel Alves, Carolina Tabuas, Jorge Pereira, Luis Ribeiro, Luis Ferro, Rita Teixeira, Liliana Azevedo, Diana Cidre, Célia Cruz, Priscila Alves, Pedro Gomes, Nuno Sousa, Diogo Faria, Elisa Barroso, Ana Jesus, Frederico Jacobs, Gabriela Queirós, Daniela Sousa, Francisco Oliveira, Teresa Azevedo, Maria Vasconcelos, Raquel Pinho, Luísa Bastos, Cândida Coelho, Jorge Gonçalves, Frederico Junqueira, JessicaDelmoral, Ricardo Le – In course:

• BSc students (2)

– Finished: Ricardo Ferreira, Soraia Pimenta

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 89

Publications & Events

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 90

Taylor & Francis journal “Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization”

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 91

www.tandfonline.com/tciv

Springer book series “Lecture Notes in Computational Vision and Biomechanics (LNCV&B)” Editors: João Manuel R. S. Tavares, Renato Natal Jorge

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 92

www.springer.com/series/8910

Indexed in

Events

@2015 João Manuel R.S. Tavares Medical Image Processing and Analysis 93

(authenticus.up.pt)

Thank You!

Medical Image Processing and Analysis

MSc in Biomedical Engineering

João Manuel R. S. Tavares tavares@fe.up.pt www.fe.up.pt/~tavares

December 10, 2015