Towards Practical Problems in Deep Learning for Radiology Image - - PowerPoint PPT Presentation

Towards Practical Problems in Deep Learning for Radiology Image Analysis

Quanzheng Li, Xiang Li, James H.Thrall Center for Clinical Data Science Department of Radiology Massachusetts General Hospital, Harvard Medical School

Background Challenges Solution Performance Our Vision

• Purpose of AI in medical imaging: creating value

in the delivery of medical care and delivery of radiology services:

– increasing diagnostic certainty – decreasing time on task for radiologists – faster availability of results – reducing costs of care.

• Interrogating image data for extracting maximum

value, with/without pre-defined model structure.

• Accuracy, Efficiency, Robustness.

Background Challenges Solution Performance Our Vision

• Automatically detect (i.e. screening) the presence
• f free air lesion regions in the lung CT images.
• Manual

inspection

• f

the incoming medical images can be time-consuming and lack of the efficiency in handling life-threatening cases (such as pneumothorax).

• Certain image abnormities can be subtle human

inspector, leads to potential mistakes in handling the patients.

• No

systematic way

• f

using learning-based methods for fully automatic screening.

Background Challenges Solution Performance Our Vision

• Region with free

air in the lung.

• Can be presented

anywhere.

• Low HUT area in

the image.

• Usually

associated with

• ther conditions.

Background Challenges Solution Performance Our Vision

Data Preprocessing and Quality Control Data Storage, Transfer, and Sharing Preparation for Analysis Analysis Postprocessing

• PACS
• HDFS
• Cloud storage
• Meta-information

processing

• Format conversion
• Image enhancement

/de-noising

• Patch extraction
• Data augmentation
• Network definition
• Model training and

validation

• Model testing
• Model application

for practice

• Heatmap

generation

• Diagnosis
• Visualization

? Positive

Background Challenges Solution Performance Our Vision

1 https://arxiv.org/abs/1512.03385

Data Preprocessing and Quality Control Data Storage, Transfer, and Sharing Preparation for Analysis Analysis Postprocessing

? Heterogeneity ? Missing / erroneous data items ? Online vs. Offline

• PACS
• HDFS
• Cloud storage
• Meta-information

processing

• Fomat conversion
• Image enhancement

/de-noising

• Patch extraction
• Data augmentation
• Network definition
• Model training and

validation

• Model testing
• Model application

for practice

• Heatmap

generation

• Diagnosis
• Visualization

• High

image/feature heterogeneity + lack

• f

training samples: more likely to over-fitting.

• Data items can be missing or wrong (e.g. in

DICOM headers during the scan).

• Most

sophisticated preprocessing (e.g. image restoration, image segmentation) techniques have to be done

• ff-line

with group-wise information provided and/or ground-truth.

Background Challenges Solution Performance Our Vision

1 https://arxiv.org/abs/1512.03385

Background Challenges Solution Performance Our Vision

1 https://arxiv.org/abs/1512.03385

Data Preprocessing and Quality Control Data Storage, Transfer, and Sharing Preparation for Analysis Analysis Postprocessing

? Low speed and intensive I/O for patch extraction ? Lack of training data samples ? Arbitrary parameter / model structure

• PACS
• HDFS
• Cloud storage
• Meta-information

processing

• Fomat conversion
• Image enhancement

/de-noising

• Patch extraction
• Data augmentation
• Network definition
• Model training and

validation

• Model testing
• Model application

for practice

• Heatmap

generation

• Diagnosis
• Visualization

• Deep learning models typically run on small

image patches for increased sample size and better feature representation.

Background Challenges Solution Performance Our Vision

1 https://arxiv.org/abs/1512.03385

Pneumothorax Patches Normal Control Patches

Background Challenges Solution Performance Our Vision

1 https://arxiv.org/abs/1512.03385

Data Preprocessing and Quality Control Data Storage, Transfer, and Sharing Preparation for Analysis Analysis Postprocessing

? Computational time: large data size + complex models ? Needs for real-time results.

• PACS
• HDFS
• Cloud storage
• Meta-information

processing

• Fomat conversion
• Image enhancement

/de-noising

• Patch extraction
• Data augmentation
• Network definition
• Model training and

validation

• Model testing
• Model application

for practice

• Heatmap

generation

• Diagnosis
• Visualization

• More

complex models and deeper networks: Increased computational load for the system.

• Example: >1000 layered Deep Residual Learning

network1 has been evaluated on the ImageNet 2012 dataset consists of 1000 classes, trained on 1.28 million training images.

Background Challenges Solution Performance Our Vision

1 https://arxiv.org/abs/1512.03385

• Large data size of most medical image types,

high performance computing becomes a crucial component for a practical and running solution.

• Example: A typical CT image has more than 30

million voxels (512×512×120). The pneumothorax project dataset constitutes imaging data from >600 subjects.

Background Challenges Solution Performance Our Vision

Background Challenges Solution Performance Our Vision

• Dataset consists of 648 subjects with/without

pneumothorax, 66 of them are annotated.

• Network trained on 31 subjects, totally 21,540

36×36 patches.

• Training a 16-layer, VGG-like 2D CNN for lesion

detection

• n

two classes: pneumothorax vs. normal.

• The whole pipeline takes DICOM images as input,

generates a lesion heatmap, provides diagnosis score for the probability of pneumothorax.

Background Challenges Solution Performance Our Vision

Data Preprocessing and Quality Control Data Storage, Transfer, and Sharing Preparation for Analysis Analysis Postprocessing

? Security ? Privacy ? Transfer speed ? Heterogeneity ? Missing / erroneous data items ? Online vs. Offline ? Low speed and intensive I/O for patch extraction ? Lack of training data samples ? Arbitrary parameter / model structure ? Computational time: large data size + complex models ? Integration into the workflow ? Real-time feedback

• PACS
• HDFS
• Cloud storage
• Meta-information

processing

• Fomat conversion
• Image enhancement

/de-noising

• Patch extraction
• Data augmentation
• Network definition
• Model training and

validation

• Model testing
• Model application

for practice

• Heatmap

generation

• Diagnosis
• Visualization

√ Co-development with high radiologist involvement √ Self-paced learning √ Asynchronous I/O √ HPC platform supported by DGX1 √ CUDA implementation

Background Challenges Solution Performance Our Vision

• Intensive involvement of radiologist During the

training phase: addressing the data heterogeneity and under-coverage of training samples:

• Four types of mis-classification cases identified,

3 false-positive, 1 false-negative:

– Extra small pneumothorax lesions (mainly caused by the image view). – Empyema. – Imaging artifacts (e.g. dark strips). – Irregular trachea/branches shapes.

Background Challenges Solution Performance Our Vision

• Self-paced learning scheme to further increase

the sample size.

spCNN: Close Looped, Multiple Rounds of Training

Distribution of prediction probability Virtual samples at round i-1 Bootstrapping Module Dataset for analyze Bootstrapping CNN k,i New CNN, at round i Original samples

apply virtual sample selection

Classification results and diagnosis

• btain

Bootstrapping CNN 1,i New, unlabeled data

apply apply

• btain

Classification Module

retraining retraining

Original samples

retraining retraining

…

Background Challenges Solution Performance Our Vision

• Tested 35 subjects, patch-wise accuracy: 93.9%.
• Subject-wise accuracy is calculated by counting

the number of detected patches followed by thresholding.

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 True positive rate False positive rate

Subject-wise ROC Curve

Background Challenges Solution Performance Our Vision

• Although the detection is done on each slice (i.e.

2D network), the detected lesion boundary is stable across slices.

Background Challenges Solution Performance Our Vision

• Detection (i.e. generating heatmap) of a single

subject takes less than 3 minutes.

• Enabled by the computing power of DGX1: 50

times faster than single K40, 10 times faster than single P100.

• Most

time consuming step is

• n

the patch extraction, further I/O synchronous will help.

• The detection speed is on the same scale of a

typical CT scan (minutes), thus enables real-time screening of the patients.

Background Challenges Solution Performance Our Vision

• Self-paced learning method helps in increasing

the training sample size by 23,100 patches (>100%) from 200 subjects (independent with the current annotation set).

• Performed in a single round.
• All the patches are consistently classified by the

10 bootstrapping networks, the accuracy is controlled by the Family-wise Error Rate (FWER)

• f p=0.05.
• Training of the 10 bootstrapping networks is

done within one hour.

Background Challenges Solution Performance Our Vision

• The

latest NVIVDIA DGX-1 provides us an unprecedented computational power to support

• ur approach of data augmentation and fast user

interaction.

• Such

massive data-initiated, computational- intensive solution will be the dominant trend for the medical researches and clinical practices.

• A new way for researchers and radiologists to

apply, interpret and convert their domain knowledges (“fourth paradigm”).

Background Challenges Solution Performance Our Vision

• Acknowledgement:

– Radiology at MGH: Subba Digumarthy, Mannudeep Kalra – Gordon Center for Medical Imaging at MGH: Ning Guo, Kyungsang Kim, Dufan Wu, Aoxiao Zhong – Beijing International Center for Mathematical Research at PKU:

• Prof. Bin Dong

– MGH Center for Clinical Data Science: