Robust Automatic Co-segmentation of Multiple Medical Images T. Liu 1 - PowerPoint PPT Presentation

Robust Automatic Co-segmentation of Multiple Medical Images T. Liu 1 , D. Floros 2 , N, Pitsianis 21 , X. Sun 1 , L. Ren 3 , F. Yin 3 59 th AAPM Annual Meeting, Denver, CO July 30, 2017 1 Department of Computer Science, Duke University, USA 2 Department of Electrical and Computer Engineering, Aristotle University of Thessaloniki, Greece 3 Department of Radiation Oncology, Duke University School of Medicine, USA 1 / 9

Introduction: from segmentation to co-segmentation + Texture-based segmentation 1 : – Intra-region texture homogeneity – Inter-region texture heterogeneity Thoracic CT slice i CT and atlas data from the AAPM LCTSC dataset 1 [Belongie et al. ICCV, 1998] [Cimpoi et al. ICCV, 2015] Segmentation of slice i [Shi et al. IEEE Trans. PAML, 2000] 2 [Rother et al. CVPR, 2006] (colors == region labels) [Rubio et al. CVPR, 2012] 3 [Wang et al. IEEE Trans. PAML, 2010], [Iglesias et al. Med Image Anal, 2015] 2 / 9

Introduction: from segmentation to co-segmentation + Texture-based segmentation 1 : – Intra-region texture homogeneity – Inter-region texture heterogeneity + Texture-based co-segmentation 2 : – Simultaneous segmentation – Inter-image region correspondence Thoracic CT slice i Thoracic CT slice j CT and atlas data from the AAPM LCTSC dataset 1 [Belongie et al. ICCV, 1998] [Cimpoi et al. ICCV, 2015] Co-segmentation of slice i Co-segmentation of slice j [Shi et al. IEEE Trans. PAML, 2000] 2 [Rother et al. CVPR, 2006] [Rubio et al. CVPR, 2012] 3 [Wang et al. IEEE Trans. PAML, 2010], [Iglesias et al. Med Image Anal, 2015] 2 / 9

Introduction: from segmentation to co-segmentation + Texture-based segmentation 1 : – Intra-region texture homogeneity – Inter-region texture heterogeneity + Texture-based co-segmentation 2 : – Simultaneous segmentation – Inter-image region correspondence Thoracic CT slice i Thoracic CT slice j * Atlas guided segmentation 3 : special case of co-segmentation CT and atlas data from the AAPM LCTSC dataset 1 [Belongie et al. ICCV, 1998] [Cimpoi et al. ICCV, 2015] Provided labels of slice i Generated labels of slice j [Shi et al. IEEE Trans. PAML, 2000] 2 [Rother et al. CVPR, 2006] [Rubio et al. CVPR, 2012] 3 [Wang et al. IEEE Trans. PAML, 2010], [Iglesias et al. Med Image Anal, 2015] 2 / 9

Purpose: Automatic co-segmentation INPUT: – CT slice i equipped with provided atlas (labels) – CT slice j without labels OUTPUT: – Automatically generated atlas (labels) of CT slice j + Provide texture homogeneity information for further image processing tasks (e.g. adaptive denoising 1 , registration) + Identify (label) organs of interest in query image guided by atlas 1 Dimitris Floros’ talk on Wednesday, August 02: WE-G-201-6 3 / 9

Method: co-segmentation by utilizing texture similarity and atlas Patch: texture element (local signal structure and noise statistics) pixelb pixela P b P a Patch similarity weight: − � P a − P b � 2 � � w ( P a , P b ) = exp 2 σ 2 f 4 / 9

Method: co-segmentation by utilizing texture similarity and atlas Patch: texture element (local signal structure and noise statistics) Joint weight matrix: intra-image inter-image A ij A ii pixelb pixela P b P a A T A jj ij Patch similarity weight: − � P a − P b � 2 � � w ( P a , P b ) = exp 2 σ 2 f 4 / 9

Result part 1: co-segmentation by texture similarity alone Provide texture homogeneity information for further image processing tasks (e.g. adaptive denoising, registration) Thoracic CT slice i Thoracic CT slice j Co-segmentation via texture similarity and graph spectral embedding and clustering 1 1 [Shi et al. IEEE Trans. PAML, 2000] 5 / 9

Results part 2: co-segmentation for label (atlas) transferring Observation: Similar textures between heart and aorta Co-segmentation with texture only Solution: incorporate label and spatial relationship into joint similarity matrix Co-segmentation with atlas labels guidance 6 / 9

Method: co-segmentation for label (atlas) transferring p – a subset of heart patches q – a subset of aorta patches A pp A pq p ⊂ I i q ⊂ I j p ⊂ I i A ij A ii App Apq A qp A qq similarity submatrix: texture-only A T A jj − � P a − P b � 2 � � ij w ( P a , P b ) = exp 2 σ 2 q ⊂ I j Aqp Aqq f 7 / 9

Method: co-segmentation for label (atlas) transferring p – a subset of heart patches q – a subset of aorta patches ˆ ˆ A pp A pq p ⊂ I i q ⊂ I j ˆ ˆ A qp A qq p ⊂ I i A ij A ii App Apq similarity submatrix: with feature, atlas and spatial relationship � − � P a − P b � 2 − � x a − x b � 2 − � 1 − δ ( l a − l b ) � 2 � A T A jj w ( P a , P b ) = exp 2 2 2 σ 2 σ 2 σ 2 ij s f l q ⊂ I j Aqp Aqq x a – spatial coordinates of P a l a – atlas label of P a (if available) 7 / 9

Results part 2: co-segmentation for label (atlas) transferring Thoracic CT slice i Co-segmentation of slice i Provided labels of slice i Thoracic CT slice j Co-segmentation of slice j 8 / 9

Results part 2: co-segmentation for label (atlas) transferring Thoracic CT slice i Co-segmentation of slice i Provided labels of slice i Thoracic CT slice j Co-segmentation of slice j Transferred labels of slice j 8 / 9

Summary – Enable the texture-based co-segmentation for registration and denoising – Transfer labels from reference image (with atlas labels) to query images In Progress: – Transferring labels (atlas info) between different scans/patient – Transferring labels between different patients 9 / 9

Thank you! Tiancheng Liu – tcliu@cs.duke.edu 9 / 9

References i R. Castillo, E. Castillo, R. Guerra, V. E. Johnson, T. McPhail, A. K. Garg, and T. Guerrero. A framework for evaluation of deformable image registration spatial accuracy using large landmark point sets. Physics in medicine and biology , 54(7):1849, 2009. X. Chen, J. K. Udupa, U. Bagci, Y. Zhuge, and J. Yao. Medical image segmentation by combining graph cuts and oriented active appearance models. IEEE Transactions on Image Processing , 21(4):2035–2046, 2012. R. C. Craddock, G. James, P. E. Holtzheimer, X. P. Hu, and H. S. Mayberg. A whole brain fmri atlas generated via spatially constrained spectral clustering. Human Brain Mapping , 33(8):1914–1928, 2012.

References ii A. L. Dulmage and N. S. Mendelsohn. Coverings of bipartite graphs. Canadian Journal of Mathematics , 10(4):516–534, 1958. L. Grady and G. Funka-Lea. Multi-label image segmentation for medical applications based on graph-theoretic electrical potentials. In Computer Vision and Mathematical Methods in Medical and Biomedical Image Analysis , pages 230–245. Springer, 2004. V. Grau, A. Mewes, M. Alcaniz, R. Kikinis, and S. K. Warfield. Improved watershed transform for medical image segmentation using prior information. IEEE transactions on medical imaging , 23(4):447–458, 2004.

References iii B. N. Li, C. K. Chui, S. Chang, and S. H. Ong. Integrating spatial fuzzy clustering with level set methods for automated medical image segmentation. Computers in biology and medicine , 41(1):1–10, 2011. A. Y. Ng, M. I. Jordan, Y. Weiss, et al. On spectral clustering: Analysis and an algorithm. In NIPS , volume 14, pages 849–856, 2001. H. Ng, S. Ong, K. Foong, P. Goh, and W. Nowinski. Medical image segmentation using k-means clustering and improved watershed algorithm. In Image Analysis and Interpretation, 2006 IEEE Southwest Symposium on , pages 61–65. IEEE, 2006.

References iv D. L. Pham, C. Xu, and J. L. Prince. Current methods in medical image segmentation. Annual review of biomedical engineering , 2(1):315–337, 2000. U. Von Luxburg. A tutorial on spectral clustering. Statistics and computing , 17(4):395–416, 2007.