Learning Normalized Inputs for Iterative Estimation in Medical - - PDF document

4/6/18 1

Learning Normalized Inputs for Iterative Estimation in Medical Image Segmentation

Michal Drozdzal, Gabriel Chartrand, Eugene Vorontsov, Mahsa Shakeri, Lisa Di Jorio, An Tang, Adriana Romero, Yoshua Bengio, Chris Pal, Samuel Kadoury

Medical imaging modalities - basics

Endoscopy Electron Microscopy Computed Tomography Magnetic Resonance Imaging 2D/3D Temporal dimension Signal scale 2D/3D 3D 3D 2D No No No Yes Grayscale RGB Hounsfield scale

• We will treat all

data as 2D data.

4/6/18 2

Image segmentation

Electron microscopy CT Endoscopy

Medical imaging segmentation pipeline

Pre-processing Model Post-processing

4/6/18 3

Medical imaging segmentation pipeline

Pre-processing Model Post-processing

Modality & model specific 3D for 2D model FP reduction Morphological operations 2D/3D CRF What is used?

Medical imaging segmentation pipeline

Pre-processing Model Post-processing

Fully Convolutional Network FCN8 UNET

Jon Long et. al. CVPR 2015 Olaf Ronneberger et. al. MICCAI 2015

Modality & model specific 3D for 2D model FP reduction Morphological operations 2D/3D CRF What is used?

4/6/18 4

Fully Convolutional Networks

Down sampling Up sampling Skip

• Pooling
• Strided convolutions
• Repeat + convolution
• Transposed convolutions
• Concatenate
• Sum

This can be ResNet, DenseNet, … This can be ResNet, DenseNet, …

Medical imaging segmentation pipeline

Pre-processing Model Post-processing

Modality specific (handcrafted) Fully Convolutional Network FCN8 UNET

Jon Long et. al. CVPR 2015 Olaf Ronneberger et. al. MICCAI 2015

Range normalization Value clipping Standardization N4 (MRI) Modality & model specific 3D for 2D model FP reduction Morphological operations 2D/3D CRF What is used?

4/6/18 5

Examples of segmentation pipelines

Range normalization Value clipping Standardization N4 (MRI) Histogram equalization Gaussian smoothing 2D Unet 2D FCN8 2D FC-ResNet 3D Unet 3D FCN8 3D FC-ResNet 2D CRF 3D CRF Morphological operations

Tools of medical imaging segmentation practitioner:

Pre-processing Model Post-processing

Lung segmentation in CT: Lung segmentation in MRI: Liver segmentation in CT: Standardization + 2DUnet N4 + 2DUNet Value clipping + 3DUnet + morphological operations

Examples of (hypothetical) segmentation pipelines: Let’s design a model that can be trained with any imaging modality and does not require any pre-processing.

4/6/18 6

Let’s use ResNets

F(x) x F(x) + x + F(x) x F(x) + convolution(x) + convolution

*N *N x has high impact on feature maps due to skip connections ResNet uses initial convolution that can adapt input Recent findings suggest that F() is a transformation close to identity [1 ] We found that FC-ResNets are more susceptible to data pre-processing than FCNs

[1] Veit at al. Residual Networks are Exponential Ensembles of Relatively Shallow Networks

Observations about ResNets:

Fully Convolutional Network Fully Convolutional Residual Network

EM CT MRI CT MRI EM

What do we propose?

4/6/18 7

Model

Fully Convolutional Network Fully Convolutional Residual Network

Input [1xaxb] Segmentation map [1xaxb] Feature map [1xaxb] +/

• 1

M parameters +/

• 11

M parameters

Data

Electron Microscopy Computed Tomography Magnetic Resonance Cell segmentation Lesion segmentation Prostate segmentation N= 30 training images 30 testing images N= 105 training volumes 30 testing volumes N= 50 training volumes 30 testing volumes

4/6/18 8

Experimental setup

Data preparation:

> No normalization! > Data augmentation

• Flips
• Rotation
• Shearing
• Elastic transformations
• Cropping

Optimization:

> RMSprop > Weight decay

EM data results (as of mid 2017)

96.8 96.9 97 97 .1 97 .2 97 .2 97 .3 97 .7 97 .8 98 98.1 VRAND Pyramid-LSTM FC-ResNet

• ptree-idsa

SCI motif IDSIA Unet CUMedVision FusionNet IAL Ours

Comparison to published methods Qualitative results (test set)

4/6/18 9

CT data results (as of mid 2017)

53.5 57 61.7 71.1 DICE FCN8 Unet FC-ResNet Ours

Comparison to standard FCNs

Image True segmentation Result

Qualitative results (test set)

MRI data results (as of mid 2017)

79.92 83.02 74.17 82.39 86.65 2D FCN 3D FCN Situs Ours SRIBHME CAMP-TUM2 CUMED

Comparison to published methods Qualitative results (test set)

4/6/18 10

Fully Convolutional Network Fully Convolutional Residual Network

EM CT MRI CT MRI EM EM CT MRI EM CT MRI Input intensity histogram: Normalized intensity histogram:

Pre-processor effect

Can we quantify the pre-processing effect?

Pre-processor quantification

d( , ) > d( , ) d( , ) = d( , )

Mean Jensen–Shannon distance on validation set 2.99 3.57 5.71 2.98 3.35 3.89 2.48 2.87 3.38 EM CT MRI input data standardization pre-processor d(0,0) d(1,0) d(2,0) d(2,0) d(1,0) d(1,1) d(2,1) d(2,2) d(2,1)

4/6/18 11

Wrap up

A low capacity FCN can serve as a learnable pre-processor. Combining learnable pre-processor with FC-ResNet yields very good results on a variety of image modalities. Single pipeline for all type of medical data! No need to handcraft data pre-processing. Want to work at the confluence of academia and industry? MILA has open positions for:

• Professors
• Software engineers
• Director of software
• R&D & technology transfer
• Linux sysadmins

is hiring!

https:/ /tinyurl.com/mila-jobs

4/6/18 12

Learning Normalized Inputs for Iterative Estimation in Medical Image Segmentation

Michal Drozdzal, Gabriel Chartrand, Eugene Vorontsov, Mahsa Shakeri, Lisa Di Jorio, An Tang, Adriana Romero, Yoshua Bengio, Chris Pal, Samuel Kadoury