João Manuel R. S. Tavares tavares@fe.up.pt www.fe.up.pt/~tavares
March 2, 2016
Towards Automated Bioimage Analysis: Algorithms & Applications
Towards Automated Bioimage Analysis: Algorithms & Applications - - PowerPoint PPT Presentation
Towards Automated Bioimage Analysis: Algorithms & Applications Joo Manuel R. S. Tavares tavares@fe.up.pt www.fe.up.pt/~tavares March 2, 2016 Outline Introduction Segmentation Analysis of Objects Tracking Matching
João Manuel R. S. Tavares tavares@fe.up.pt www.fe.up.pt/~tavares
March 2, 2016
Towards Automated Bioimage Analysis: Algorithms & Applications
Outline
– Tracking – Matching – Registration
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 2
Presentation
Institute of Science and Innovation in Mechanical and Industrial Engineering (INEGI)
Engineering from FEUP
Biomedical Imaging, Biomechanics, Human Posture and Control, Product Development
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 4
Introduction
Analysis aim the development of algorithms to perform fully or semi-automatically tasks performed by the (quite complex) human vision system
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 5
Original images Computational 3D voxelized and poligonized models built
Azevedo et al. (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(3):359-369
Introduction
most importance for our Society
frequently used, for example, in:
– Natural Sciences – Sports – Biology – Industry – Engineering – Medicine
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 6
Introduction
– Noise removal – Geometric correction – Segmentation, recognition (2D-4D) – Motion and/or deformation tracking and analysis, including matching and registration – 3D reconstruction – Assisted medical diagnosis and intervention
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 7
in 3D), being each basic element known as pixel (or voxel in 3D)
Introduction
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 8
pixels (or voxels in 3D) in the Cartesian or in another domain
Introduction
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 9
∗ −1 −2 −1 +1 +2 +1 ⎡ ⎣ ⎢ ⎢ ⎢ ⎤ ⎦ ⎥ ⎥ ⎥ = Gy ∗ −1 +1 −2 2 −1 +1 ⎡ ⎣ ⎢ ⎢ ⎢ ⎤ ⎦ ⎥ ⎥ ⎥ = Gx Gx
2 + Gy 2 = GSobel operator
∗
( denotes convolution)
energy reflected or emitted by the imaged object
Introduction
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 10 http://what-when-how.com/introduction-to-video-and-image-processing/image-acquisition-introduction-to- video-and-image-processing-part-1
illumination, reflections, complex objects and backgrounds
Introduction
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 11
https://rahaddadi.files.wordpress.com/2011/05/face_black_and_white_optical_illusion_cool-s453x562-92306-5803.jpg http://s1.cdn.autoevolution.com/images/news/the-longest-traffic-jam-in-history-12-days-62-mile-long-47237_1.jpg
Introduction
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 12
Image(s) enhancement / correction Image(s) segmentation / features extraction tracking matching Image(s) motion analysis registration image (pre)processing image analysis / computational vision 3D vision computer vision
Usual Pipeline
Segmentation
automatically manner objects (2D/3D) presented in static images or in image sequences
thresholding, region growing, template matching, statistical, geometric or physical modeling, or artificial classifiers
the computational analysis of objects in images
reduced contrast, shapes not previously known,
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 14
Segmentation
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 15 Ma et al. (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(2):235-246
Segmentation
pedobarography (Otsu’s method, morphologic dilation,
xor operation)
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 16
Original images Segmented images
Bastos & Tavares (2004) Lecture Notes in Computer Science 3179:39-50
camera mirror contact layer + glassreflected light glass pressure
lamp lamp transparent layer
Segmentation
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 17 Ma et al. (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(2):235-246
Segmentation
growing)
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 18
Region Growing, x=215; y=254Segmentation obtained (bony labyrinth)
Barroso et al. (2011) CNME 2011 Ferreira et al. (2014) Computer Methods in Biomechanics and Biomedical Engineering 17(8):888-904
X: 254 Y: 214 Index: 116.7 RGB: 0.459, 0.459, 0.459Original Image
Segmentation
templates
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 19 Carvalho & Tavares (2006) CompIMAGE 2006, 129-134 Carvalho & Tavares (2007) VipIMAGE 2007, 209-215
Example of a deformable template (for the eye)
Segmentation
features (deformable geometric template)
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 20
Original image and associated force (or energy) fields Segmentation of the iris using a deformable template (a circle) Segmentation of an eye using an deformable template
Carvalho & Tavares (2006) CompIMAGE 2006, 129-134 Carvalho & Tavares (2007) VipIMAGE 2007, 209-215
Segmentation
(point distribution models, optimization)
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 21 Vasconcelos & Tavares (2008) Computer Modeling in Engineering & Sciences 36(3):213-241
Segmentation
speech production from MR images (active shape model)
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 22
Original image Final segmentation
Vasconcelos et al. (2011) Journal of Voice 25(6):732-742
contours (i.e. snakes – parametric models)
Segmentation
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 23 Tavares et al. (2009) International Journal for Computational Vision and Biomechanics 2(2):209-220
Esnake = Eint
s=0 1
∫
(v(s))+ Eext(v(s))ds Eint = α(s) dv(s) ds
2
+ β(s) d 2v(s) ds2
2
Segmentation
(active contours - snakes)
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 24
Initial contour Final contour
Tavares et al. (2009) International Journal for Computational Vision and Biomechanics 2(2):209-220 Gonçalves et al. (2008) Computer Modeling in Engineering & Sciences 32(1):45-55
Segmentation
method (geometrical models)
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 25 Ma et al. (2010) Medical Engineering & Physics 32(7):766-774 Ma et al. (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(2):235-246
∂φ ∂t + F Δφ = 0
Typical form of the motion equation:
lesions in dermoscopic images (color spaces, level-
set model)
Segmentation
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 26
Segmentation examples under different imaging conditions and different types of skin pigmented lesions Illustration of the segmentation process
Ma & Tavares (2015) IEEE J. of Biomedical and Health Informatics, DOI: 10.1109/JBHI.2015.2390032 (in press) Filho et al. (2015) Journal of Medical Systems 39(11):177
Segmentation
method with a prior knowledge
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 27 Ma et al. (2010) Medical Engineering & Physics 32(7):766-774 Ma et al. (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(2):235-246
Segmentation
images (level-set method, a prior knowledge)
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 28 Ma et al. (2013) Computers in Biology and Medicine 43(4):248-258 Ma et al. (2012) The Int. Journal for Numerical Methods in Biomedical Engineering 28(6-7):714-726
Segmentation of the bladder, vagina and rectum in pelvic cavity images (3 examples)
Tracking
deformation of objects in image sequences
flow, block matching and stochastic methods are widespread
motion/deformation involved, the management
motion tracked and its quantification
distortion, non-constant illumination, occlusion, noise, multiple objects, etc.
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 30
Tracking
to track features in image sequences (Kalman Filter or
Unscented Kalman Filter,
distance, management model)
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 31 Pinho et al. (2007) Int. Journal of Simulation Modelling 6(2):84-92 Pinho & Tavares (2009) VipIMAGE 2009, 299-304 Pinho & Tavares (2009) Computer Modeling in Engineering & Sciences 46(1):51-75
Tracking
filter, Mahalanobis distance, optimization, management model)
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 32
Prediction Uncertainty Area Measurement Correspondence Result (5 frames)
Pinho et al. (2005) ICCB 2005, 915-926 Pinho & Tavares (2009) Computer Modeling in Engineering & Sciences 46(1):51-75
Tracking
sequences (Kalman filter, Mahalanobis distance,
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 33
(547 frames)
Pinho et al. (2005) LSCCS, Vol. 4A:463-466 Pinho et al. (2007) International Journal of Simulation Modelling 6(2):84-92
Matching
between image features
align) objects, recognize objects, attain 3D information, analyze tracked motion, etc.
invariant characteristics, as curvature, or displacements in global/eigen spaces (like in modal space)
deformations, high shape variations, etc.
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 35
Matching
modal matching
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 36
Modeling (physical or geometrical) Eigenvalues / eigenvectors computation Matching matrix assembly Contour 1 Contour 2 Matches achievement (optimization) Modeling (physical or geometrical) Eigenvalues / eigenvectors computation
Bastos & Tavares (2006) Inverse Problems in Science and Engineering 14(5):529-541 Tavares & Bastos (2010) Progress in Computer Vision and Image Analysis 339-368
Matching
dynamic pedobarography (FEM modeling, modal
analysis, optimization)
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 37
Image of dynamic pedobarography
Tavares & Bastos (2005) Electronic Letters on Computer Vision and Image Analysis 5(3):1-20
Matching found between two contours Matching found between two intensity (pressure) surfaces (2 views) Matching found between iso-contours (2 views)
Registration
according to distinct conditions
up the evaluation of diseases
features, as points of high curvature, and their matching, or by minimization of a similarity measure, followed by the estimation of the involved transformation
easily detected, occlusion, non-rigid deformations, severe shape variations, etc.
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 39
Registration
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 40
The cost matrix is built based on geometric or physical principles The matching is found based on the minimization
associated to the possible correspondences To search for the best matching is used an
algorithm
Bastos & Tavares (2006) Inverse Problems in Science and Engineering 14(5):529-541 Oliveira & Tavares (2009) Computer Modeling in Engineering & Sciences 43(1):91-110 Oliveira, Tavares, Pataky (2009) Journal of Biomechanics 42(15):2620-2623
Registration: 2D, monomodal, intrasubject Processing time: 0.5 s (AMD Turion64, 2.0 GHz, 1.0 GB of RAM) Images dimension: 217x140 pixels
Fixed image and contour (MRI) Moving image and contour (MRI) Overlapped images before the registration Overlapped images after the registration Difference between the images after the registration Correspondences found between the Corpus Callosum contours Oliveira & Tavares (2014) Computer Methods in Biomechanics and Biomedical Engineering 17(2):73-93
Registration
(geometrical modeling, matching, optimization, dynamic programming)
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 41
Registration
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 42
Original images Images before and after the registration
Oliveira, Pataky, Tavares (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(6):731-740
Registration
parameters of the transformation that optimizes a similarity measure between the input images
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 43 Moving image Fixed image Pre-registration transformation (optional) Interpolator Similarity measure Optimizer Geometric transformation
The optimization algorithm stops when a similarity criterion is achieved
Oliveira & Tavares (2014) Computer Methods in Biomechanics and Biomedical Engineering 17(2):73-93
Registration: 2D, multimodal, intrasubject (without pre-registration) Similarity measure: MI Processing time: 4.6 s (AMD Turion64, 2.0 GHz, 1.0 GB of RAM) Images dimension: 246x234 pixels
Oliveira & Tavares (2014) Computer Methods in Biomechanics and Biomedical Engineering 17(2):73-93
Registration
images (optimization of a similarity measure)
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 44
“Checkerboard” of the slices before the registration (CT/MRI-PD, brain)
F F F F F F F F M M M M M M M M
(The “checkerboard” slice is built by interchanging square patches of both slices and preserving their original spatial position in the fixed (F) and moving (M) slices)
Registration
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 45
Registration: 3D, multimodal, intrasubject; Similarity measure: MI Checkerboard of the slices after the registration (CT/MRI-PD, brain)
Registration
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 46
Normal Alzheimer Idiopathic Parkinsonism Essential tremor
Registration
images
Brain DaTSCAN SPECT images are used to assist the diagnosis of the Parkinson’s disease and to distinguish it from other degenerative
– Segment the relevant areas and perform dimensional analysis – Quantify the binding potential of the basal ganglia – Computation of statistical data relatively to a reference population – Image classification for diagnosis purposes
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 47
Mean slice from the population used as reference Corresponding slice of a patient Difference of intensities Z-scores mapping over the slice (red – high Z-
scores) (The blue rectangles represent the 3D ROIs used to compute the binding potentials) Oliveira et al. (2014) The Quarterly Journal of Nuclear Medicine and Molecular Imaging 58(1):74-84
Registration
images
3D volume images are automatically registered and statistical analysis relatively to a reference population can be attained
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 48
Registration
temporal registration
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 49 Moving sequence Fixed sequence Apply the spatio & temporal transformation Compute the similarity measure Optimizer Build the spatio & temporal transformation Oliveira et al. (2011) Medical & Biological Engineering & Computing 49(7):843-850 Oliveira & Tavares (2013) Medical & Biological Engineering & Computing 51(3):267-276 Build the temporal representative images Search for the transformation that register the temporal representative images Estimate the linear temporal registration Pre-registration Registration optimization
Registration
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 50
dynamic pedobarography (spatial & temporal
registration)
Device: Light reflection (25 fps, resolution 30 pixels/cm2) Image similarity measure: MSD Sequences dimension: 160x288x22, 160x288x25 Processing time: 1 min (using an AMD Turion64, 2.0 GHz, 1.0 GB of RAM) Template sequence Source sequence Overlapped sequences Before the registration After the registration
3D Reconstruction
inner shapes – 2D segmentation (contours, for example) and data interpolation, or 3D segmentation; external shapes – active techniques (with energy projection or relative motion), passive techniques (without energy projection nor relative motion) and of space carving; deformation of 3D deformable models prebuilt
segmentation, matching, triangulation, registration and fusion
unstable illumination, occlusion, noise, multiple objects, complex shapes and topologies, etc.
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 52
3D Reconstruction
based on 2D segmentation, loft (marching cubes) and smooth
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 53
Reconstructed organs from pelvic cavity
Pimenta et al. (2006) CompIMAGE 2006, 343-348 Alexandre et al. (2007) VipIMAGE 2007, 359-362
3D Reconstruction
views (registration/fusion)
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 54
Axial and sagittal T2-weighted MR images 3D Reconstruction of the bladder by fusion data from the axial and sagittal images (2 views)
Ma et al. (2013) Medical Engineering & Physics 35(12):1819-1824
Conclusions
complex and demand, but of raised importance in many domains
conditions in the image acquisition process, occlusion,
important and complex goals still to be reached
as of Mathematics, Computational Mechanics, Medicine and Biology, can contribute significantly for their reaching
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 56
The work presented has been done with the support of Fundação Gulbenkian, CNPq, CAPES, and FCT, mainly trough the funding of the research projects:
– PTDC/BBB-BMD/3088/2012 – PTDC/SAU-BEB/102547/2008 – PTDC/SAU-BEB/104992/2008 – PTDC/EEA-CRO/103320/2008 – UTAustin/CA/0047/2008 – UTAustin/MAT/0009/2008 – PDTC/EME-PME/81229/2006 – PDTC/SAU-BEB/71459/2006 – POSC/EEA-SRI/55386/2004
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 57
Acknowledgments
(Image Processing and Analysis)
Research Team (Image Processing and Analysis)
– Finished: Alexandre Carvalho, Simone Prado, Mercedes Filho – In course: Zhen Ma; Pedro Predosa
– Finished: Zhen Ma, Francisco Oliveira, Teresa Azevedo, Daniel Moura, Sandra Rua, Maria Vasconcelos – In course: João Nunes, Alex Araujo, Carlos Gulo, Roberta Oliveira, Danilo Jodas, Pedro Morais, Andre Pilastri, Nuno Sousa, Domingos Vieira
– Finished: Raquel Alves, Carolina Tabuas, Jorge Pereira, Luis Ribeiro, Luis Ferro, Rita Teixeira, Liliana Azevedo, Diana Cidre, Célia Cruz, Priscila Alves, Pedro Gomes, Nuno Sousa, Diogo Faria, Elisa Barroso, Ana Jesus, Frederico Jacobs, Gabriela Queirós, Daniela Sousa, Francisco Oliveira, Teresa Azevedo, Maria Vasconcelos, Raquel Pinho, Luísa Bastos, Cândida Coelho, Jorge Gonçalves, Frederico Junqueira, Jessica Delmoral, Ricardo Le – In course:
– Finished: Ricardo Ferreira, Soraia Pimenta
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 59
Taylor & Francis journal “Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization”
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 61
www.tandfonline.com/tciv
Springer book series “Lecture Notes in Computational Vision and Biomechanics (LNCV&B)” Editors: João Manuel R. S. Tavares, Renato Natal Jorge
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 62
www.springer.com/series/8910
Indexed in
Events
@2016 João Manuel R.S. Tavares Towards Automated Bioimage Analysis: Algorithms & Applications 63
Towards Automated Bioimage Analysis: Algorithms & Applications
João Manuel R. S. Tavares tavares@fe.up.pt www.fe.up.pt/~tavares
March 2, 2016