Towards Intelligent Interactive Segmentation of Medical Images - - PowerPoint PPT Presentation

Towards Intelligent Interactive Segmentation of Medical Images

Guotai Wang

University of Electronic Science and Technology of China 2019-9-4

Content

1，Minimally interactive segmentation of the placenta from fetal MRI 2, Interactive segmentation using deep learning and geodesic distance transform 3, Image-specific fine-tuning for interactive segmentation

CNN Train Test

Content

1，Minimally interactive segmentation of the placenta from fetal MRI 2, Interactive segmentation using deep learning and geodesic distance transform 3, Image-specific fine-tuning for interactive segmentation

CNN Train Test

Clinical background of the placenta

Minimally Interactive Placenta Segmentation: Background

Normal Low implantation

Partial placenta previa Complete placenta previa

Placenta has a large variation of shape and position Twin-twin transfusion syndrome Intrauterine growth restriction

Imaging of the placenta

Minimally Interactive Placenta Segmentation: Background

Fetal MRI Fetal Ultrasound

• Low contrast
• Limited filed of view
• Noises
• Good soft tissue contrast
• Large filed of view
• Higher SNR

• Images are acquired as a stack of 2D Slices

Challenges of placenta segmentation from fetal MRI

Minimally Interactive Placenta Segmentation: Background

Stack 1 Stack 2

Axial view Sagittal view

Acquired in axial view Acquired in sagittal view

• Low 3D resolution
• Inter-slice motion
• Inhomogeneous appearance
• Large shape/position variation

The challenges make it hard to

• btain accurate segmentation

results of the placenta automatically

• Minimally interactive segmentation

– Interactions only required for a single slice – Automatic propagation to other slices

Slic-Seg: Slice-by-slice propagation

Interactive Segmentation using Online Random Forest

input scribbles probability segmentation result user interactions DyBa ORF learning CRF

Segmentation of the start slice Automatic propagation

• Segmentation Results

Slic-Seg: Slice-by-slice propagation

Interactive Segmentation using Online Random Forest

• G. Wang et al, Slic-Seg: A Minimally Interactive Segmentation of the Placenta from Sparse and Motion-Corrupted Fetal MRI in Multiple Views, Medical Image Analysis, 2016
• G. Wang et al, Dynamically Balanced Online Random Forests for Interactive Scribble-Based Segmentation, MICCAI 2016
• G. Wang et al, Slic-Seg: Slice-by-slice Segmentation Propagation of the Placenta in Fetal MRI using One-plane Scribbles and Online Learning, MICCAI, 2015

• Making use of the complementary resolution

Co-segmentation of images acquired in different views

Interactive Segmentation using Online Random Forest

• G. Wang et al, Slic-Seg: A Minimally Interactive Segmentation of the Placenta from Sparse and Motion-Corrupted Fetal MRI in Multiple Views, Medical Image Analysis, 2016
• G. Wang et al, Dynamically Balanced Online Random Forests for Interactive Scribble-Based Segmentation, MICCAI 2016
• G. Wang et al, Slic-Seg: Slice-by-slice Segmentation Propagation of the Placenta in Fetal MRI using One-plane Scribbles and Online Learning, MICCAI, 2015

Axial view image Sagittal view image 4D Graph Cut

Co-segmentation of images acquired in different views

Interactive Segmentation using Online Random Forest

• G. Wang et al, Slic-Seg: A Minimally Interactive Segmentation of the Placenta from Sparse and Motion-Corrupted Fetal MRI in Multiple Views, Medical Image Analysis, 2016
• G. Wang et al, Dynamically Balanced Online Random Forests for Interactive Scribble-Based Segmentation, MICCAI 2016
• G. Wang et al, Slic-Seg: Slice-by-slice Segmentation Propagation of the Placenta in Fetal MRI using One-plane Scribbles and Online Learning, MICCAI, 2015

Initial Co- segmentation Axial View of I1 Sagittal View of I1 Axial View of I2 Sagittal View of I2

Content

1，Minimally interactive segmentation of the placenta from fetal MRI 2, Interactive segmentation using deep learning and geodesic distance transform 3, Image-specific fine-tuning for interactive segmentation

CNN Train Test

• Interactive segmentation

– Widely used in practice – Higher robustness for challenging cases

• Existing interactive tools

– Graph Cuts, Random Walker, ITK-SNAP, … – Often require a lot of user interactions – Not intelligent and fast enough

• Existing deep learning methods

– Mainly used for automatic segmentation – Require a large number of annotated training images – Still need to be refined in complex cases

Why combine them

Interactive segmentation using deep learning

Mis-segmentations obtained by CNNs Graph Cuts (Y. Boykov, 2001)

• Two-stage framework

– P-Net: propose an initial segmentation – R-Net: refine the initial segmentation

• User interactions

– Given on the output of P-Net – Used as input of R-Net

Interactive segmentation using CNNs

DeepIGeoS: Deep Interactive Geodesic Framework for Segmentation

yes Final segmentation

R-Net with CRF-Net(fu)

Refined segmentation

P-Net with CRF-Net(f)

Input image Initial segmentation User-interactions

Agreed by the user ?

no

• G. Wang et al. DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation, TPAMI, 2019

• Geodesic distance transforms

– For each class respectively – Obtain additional two distance maps – Encode contextual information

How to encode user interactions

DeepIGeoS: Deep Interactive Geodesic Framework for Segmentation

• G. Wang et al. DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation, TPAMI, 2019

User-interactions on initial segmentation

(a) (b) (c) (d) (e)

Input of R-Net

A seed point Geodesic distance Euclidean distance

• Interactions are based on mis-segmentations

– Compare an initial segmentation with the ground truth – Randomly sample points from mis-segmented regions

Simulated user interactions during training

DeepIGeoS: Deep Interactive Geodesic Framework for Segmentation

• G. Wang et al. DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation, TPAMI, 2019

Interactions for background Interactions for foreground

• P-Net and R-Net share the same structure

– Except the number of input channels

Network structure

DeepIGeoS: Deep Interactive Geodesic Framework for Segmentation

• G. Wang et al. DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation, TPAMI, 2019

2D placenta segmentation from fetal MRI

DeepIGeoS: Deep Interactive Geodesic Framework for Segmentation

• G. Wang et al. DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation, TPAMI, 2019

DeepIGeoS Random Walker (L. Grady, 2006)

DeepIGeoS is 4-5 times faster than traditional interactive segmentation tools

2D placenta segmentation from fetal MRI

DeepIGeoS: Deep Interactive Geodesic Framework for Segmentation

• G. Wang et al. DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation, TPAMI, 2019

DeepIGeoS Random Walker (L. Grady, 2006)

DeepIGeoS is 4-5 times faster than traditional interactive segmentation tools

2D placenta segmentation from fetal MRI

DeepIGeoS: Deep Interactive Geodesic Framework for Segmentation

• G. Wang et al. DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation, TPAMI, 2019

DeepIGeoS Random Walker (L. Grady, 2006)

DeepIGeoS is 4-5 times faster than traditional interactive segmentation tools

• Data from BraTS challenge 2015

– Whole tumor segmentation from FLAIR – Training: 234 images – Testing: 40 images

3D brain tumor segmentation from MRI

DeepIGeoS: Deep Interactive Geodesic Framework for Segmentation

• G. Wang et al. DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation, TPAMI, 2019

• Data from BraTS challenge 2015

– Whole tumor segmentation from FLAIR – Training: 234 images – Testing: 40 images

3D brain tumor segmentation from MRI

DeepIGeoS: Deep Interactive Geodesic Framework for Segmentation

• G. Wang et al. DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation, TPAMI, 2019

• Data from BraTS challenge 2015

– Whole tumor segmentation from FLAIR – Training: 234 images – Testing: 40 images

3D brain tumor segmentation from MRI

DeepIGeoS: Deep Interactive Geodesic Framework for Segmentation

• G. Wang et al. DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation, TPAMI, 2019

Content

1，Minimally interactive segmentation of the placenta from fetal MRI 2, Interactive segmentation using deep learning and geodesic distance transform 3, Image-specific fine-tuning for interactive segmentation

CNN Train Test

• Fetal MRI segmentation

– Multiple-organs – Annotation for all organs ?

How to segment previously unseen objects?

• G. Wang et al. Interactive medical image segmentation using deep learning with image-specific fine-tuning, TMI, 2018

BIFSeg: Segmentation by Bounding Box + Image-Specific Fine-Tuning

Placenta Fetal brain Fetal lung Maternal Kidney Annotated for training Testing Unseen during training

• Proposed framework

– 1, Use CNN to get an initial segmentation inside a bounding box – 2, Fine-tune the CNN with/without scribbles (supervision) – 3, Deal with previously unseen objects

Dealing with unseen objects with image-specific fine-tuning

• G. Wang et al. Interactive medical image segmentation using deep learning with image-specific fine-tuning, TMI, 2018

BIFSeg: Segmentation by Bounding Box + Image-Specific Fine-Tuning

Training stage

Trained CNN model

• …

…

Training images Cropped training images

q0 q q

Pre-trained CNN model

• Initial result

Updated CNN model

• Refined result

… …

Testing stage

Image with user-provided bounding box

q1 q0

Scribbles Image-specific fine- tuning with weighted loss function Weight map

• Fine-tuning

– Feature extractor keep fixed – Classifier is fine-tuned towards a specific image – Update the model and label

Fine-tuning and image-specific model

• G. Wang et al. Interactive medical image segmentation using deep learning with image-specific fine-tuning, TMI, 2018

BIFSeg: Segmentation by Bounding Box + Image-Specific Fine-Tuning

3x3, 64 3x3, 64 3x3, 64 3x3, 64 3x3, 64 3x3, 64 3x3, 64 3x3, 64 3x3, 64 3x3, 64 3x3, 64 3x3, 64 3x3, 64

1 4 1 2 2 4 8 4 8 8 16 16 16 Block 1 Block 2 Block 3 Block 4 Block 5

1x1, 128 1x1, 2

Block 6 1 1

Input Image Output

classifier Feature extractor

• Joint optimization

– CNN parameters – Segmentation

• When is fixed
• When is fixed

A uniform framework for supervised and unsupervised fine-tuning

• G. Wang et al. Interactive medical image segmentation using deep learning with image-specific fine-tuning, TMI, 2018

BIFSeg: Segmentation by Bounding Box + Image-Specific Fine-Tuning

Iterative update (Graph Cut problem) (Back propagation) = ≠

• Update the model based on current segmentation

– Need to reduce the impact of mis-labeled pixels

• Pixels with different confidence

– Network-based confidence – Interaction-based confidence

Weighted loss function for fine-tuning

• G. Wang et al. Interactive medical image segmentation using deep learning with image-specific fine-tuning, TMI, 2018

BIFSeg: Segmentation by Bounding Box + Image-Specific Fine-Tuning

Input Foreground probability Binary output Scribbles Weight map Initial segmentation Foreground scribble Background scribble User-provided bounding box

• Unsupervised Fine-tuning

– No additional user interactions provided

2D segmentation of multiple organs from fetal MRI

• G. Wang et al. Interactive medical image segmentation using deep learning with image-specific fine-tuning, TMI, 2018

BIFSeg: Segmentation by Bounding Box + Image-Specific Fine-Tuning

Train Test Placenta √ √ Fetal brain √ √ Fetal lungs √ Maternal kidneys √ Patient number 10 6

Input P-Net BIFSeg (-w) BIFSeg Foreground probability

Previously seen Previously unseen

• Supervised Fine-tuning

– Guided by scribbles – Only few interactions – Real-time update

2D segmentation of multiple organs from fetal MRI

• G. Wang et al. Interactive medical image segmentation using deep learning with image-specific fine-tuning, TMI, 2018

BIFSeg: Segmentation by Bounding Box + Image-Specific Fine-Tuning

Input Placenta Fetal brain Fetal lungs Maternal kidneys Segmentation result Ground truth Foreground scribble Background scribble User-provided bounding box P-Net Scribbles P-Net + CRF BIFSeg (-w) BIFSeg Previously seen Previously unseen

Segmentation of previously unseen kidney

• Supervised Fine-tuning

– Guided by scribbles – Only few interactions – Real-time update

2D segmentation of multiple organs from fetal MRI

• G. Wang et al. Interactive medical image segmentation using deep learning with image-specific fine-tuning, TMI, 2018

BIFSeg: Segmentation by Bounding Box + Image-Specific Fine-Tuning

Input Placenta Fetal brain Fetal lungs Maternal kidneys Segmentation result Ground truth Foreground scribble Background scribble User-provided bounding box P-Net Scribbles P-Net + CRF BIFSeg (-w) BIFSeg Previously seen Previously unseen

Segmentation of previously unseen kidney

• Supervised Fine-tuning

– Guided by scribbles – Only few interactions – Real-time update

2D segmentation of multiple organs from fetal MRI

• G. Wang et al. Interactive medical image segmentation using deep learning with image-specific fine-tuning, TMI, 2018

BIFSeg: Segmentation by Bounding Box + Image-Specific Fine-Tuning

Input Placenta Fetal brain Fetal lungs Maternal kidneys Segmentation result Ground truth Foreground scribble Background scribble User-provided bounding box P-Net Scribbles P-Net + CRF BIFSeg (-w) BIFSeg Previously seen Previously unseen

Segmentation of previously unseen kidney

• Quantitative evaluation

2D segmentation of multiple organs from fetal MRI

• G. Wang et al. Interactive medical image segmentation using deep learning with image-specific fine-tuning, TMI, 2018

BIFSeg: Segmentation by Bounding Box + Image-Specific Fine-Tuning

Comparison with traditional methods Dice score of different objects

• Data

– T1ce: tumor core (train and test) – Flair: whole tumor (test only)

• Unsupervised fine-tuning

– No interactions provided

3D segmentation of brain tumor from MRI

• G. Wang et al. Interactive medical image segmentation using deep learning with image-specific fine-tuning, TMI, 2018

BIFSeg: Segmentation by Bounding Box + Image-Specific Fine-Tuning

Axial Sagittal Coronal Axial Sagittal Coronal

User-provided bounding box Segmentation result Ground truth (a) Tumor core in T1c (previously seen) (b) Whole tumor in FLAIR (previously unseen) Input PC-Net PC-Net + CRF BIFSeg(-w) BIFSeg Foreground probability PC-Net BIFSeg(-w) BIFSeg Segmentation

• Supervised fine-tuning

– Guided by scribbles for refinement

3D segmentation of brain tumor from MRI

• G. Wang et al. Interactive medical image segmentation using deep learning with image-specific fine-tuning, TMI, 2018

BIFSeg: Segmentation by Bounding Box + Image-Specific Fine-Tuning

Input PC-Net PC-Net + CRF BIFSeg (-w) BIFSeg Scribbles Axial Sagittal Coronal Axial Sagittal Coronal (a) Tumor core in T1c (previously seen) Segmentation result Ground truth Foreground scribble Background scribble User-provided bounding box (b) Whole tumor in FLAIR (previously unseen)

Comparison with traditional methods

• Using deep learning for interactive segmentation is promising

– Outperforms traditional interactive segmentation methods – High accuracy and fast – Only few interactions needed

Conclusion

• Future works

– Segment previously unseen modality – Intelligent guidance for user interactions