Interpretable Multimodal Deep Learning for Objective Diagnosis, - - PowerPoint PPT Presentation

March 20, 2019 1

Interpretable Multimodal Deep Learning for Objective Diagnosis, Prognosis and Biomarker Discovery

Faisal Mahmood, PhD Postdoctoral Fellow Department of Biomedical Engineering Johns Hopkins University faisalm@jhu.edu | http://faisal.ai

faisalm@jhu.edu | faisal.ai

Deep Learning for Medical Imaging – Major Challenges

• 1. Limited Annotated Data
• Under representation of rare conditions.
• Limited experts available for annotation.
• Privacy Issues
• Cost

Faisal Mahmood, Nicholas J. Durr et al."Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training." IEEE Transactions on Medical Imaging (2018).

mmmmmmmmmm Pathology 65%

70%

Johns Hopkins Electronic Medical Records

Radiology 15%

... Unlabeled Unstructured Transfer Learning Pre- Trained Does not capture clinical diversity.

• 2. Domain Adaptation
• Diversity in data, different sensors, cites and patients.
• Patient specific texture and color information.

Deep Learning for Medical Imaging – Major Challenges Train Test

How can we train AI systems robust to variability in the data?

• 2. Domain Adaptation
• Diversity in data, different sensors, cites and patients.
• Patient specific texture and color information.

Deep Learning for Medical Imaging – Major Challenges Train Test

(F. Mahmood et al., 2018)

• 3. Structured Prediction
• Global vs Local features.

Deep Learning for Medical Imaging – Major Challenges

Modify

AI System

Input Output Ground Truth

Compare

• F. Mahmood, 2018 – (EN.580.142.13)

• 3. Structured Prediction
• Global vs Local features.

Deep Learning for Medical Imaging – Major Challenges Ai System

Input Output Ground Truth

Compare

Per-pixel classification or regression is unstructured. Each pixel is considered conditionally independent.

How can we develop conditionally dependent deep learning models?

• 4. Incorporating Multimodal Information
• Subjective diagnosis is multimodal.

Deep Learning for Medical Imaging – Major Challenges

Imaging Data

Histopathology Immunohistochemistry Radiology

Endoscopy

…

Multi-omics Data

Genomics Proteomics

Transcriptomic s miRNAomics

Metabolomics

…

Patient Specific Data

Patient History

Familial History Clinical Phenotyping Pharmacologic Data

… …

How Can we Fuse Unregistered, Uncorrelated and Noisy Data? How can we separate patient specific and general information?

Deep Learning for Medical Imaging – Major Challenges

• 1. Limited Annotated Data
• Under representation of rare conditions.
• Limited experts available for annotation.
• Privacy Issues, Cost
• 2. Domain Adaptation
• Diversity in data, different sensors, cites and patients.
• Patient specific texture and color information.
• 3. Structured Prediction
• Global vs Local features.
• 4. Incorporating Multimodal Information
• Subjective diagnosis is multimodal.

Computational Pathology Computational Endoscopy

Computational Pathology Computational Endoscopy

Application: Depth Estimation for Endoscopy Purpose: Predict Topography from Monocular Images Colonoscopy Gives 2D Images Topography Matters

Faisal Mahmood, Nicholas J. Durr et al." Deep learning and conditional random fields-based depth estimation and topographical reconstruction from conventional endoscopy" Medical Image Analysis (2018).

Endoscopic Depth and Topography

Endoscopic Depth and Topography

Application: Depth Estimation for Endoscopy Purpose: Predict Topography from Monocular Images Colonoscopy Gives 2D Images Topography Matters 60% of colorectal cancer cases detected after optical colonoscopy are associated with missed lesions.

How do gastroenterologists predict the presence of a polyp? Predict the size of the perforations. Predict surface topography.

Faisal Mahmood, Nicholas J. Durr et al." Deep learning and conditional random fields-based depth estimation and topographical reconstruction from conventional endoscopy" Medical Image Analysis (2018).

Depth Estimation from Monocular Endoscopy Images

No Ground Truth Depth Data:

• Limited real estate on an endoscope.
• Regulatory approvals required to add depth sensor.

8.8mm

Solution: Generate Synthetic Endoscopy Data

Faisal Mahmood, Nicholas J. Durr et al." Deep learning and conditional random fields-based depth estimation and topographical reconstruction from conventional endoscopy" Medical Image Analysis (2018).

Generating Synthetic Endoscopy Data with GT Depth

(F. Mahmood et al., 2018)

Generating Synthetic Endoscopy Data with GT Depth

(F. Mahmood et al., 2018)

Deep Learning for Medical Imaging – Major Challenges

• 1. Limited Annotated Data
• Under representation of rare conditions.
• Limited experts available for annotation.
• Privacy Issues
• Cost
• 2. Domain Adaptation
• Diversity in data, different sensors, cites and patients.
• Patient specific texture and color information.
• 3. Structured Prediction
• Global vs Local features.
• 4. Incorporating Multimodal Information
• Subjective diagnosis is multimodal.

Training with Endoscopy Synthetic Data

Problem: Standard Deep Learning Networks are not sufficiently context aware.

Solution: Add non-local information using a joint CNN-Graphical Model Setup.

(F. Mahmood et al., 2018)

Solution: Joint CNN-CRF Model

(F. Mahmood et al., 2018)

Typical Deep Learning Flow

Solution: Joint CNN-CRF Model

Adds Non-local Information

(F. Mahmood et al., 2018)

Deep Learning for Medical Imaging – Major Challenges

• 1. Limited Annotated Data
• Under representation of rare conditions.
• Limited experts available for annotation.
• Privacy Issues
• Cost
• 2. Domain Adaptation
• Diversity in data, different sensors, cites and patients.
• Patient specific texture and color information.
• 3. Structured Prediction
• Global vs Local features.
• 4. Incorporating Multimodal Information
• Subjective diagnosis is multimodal.

Adapting Synthetic Networks to Real Data

Train Test Problem: Network trained on synthetic data does not work with real data. Solution: Adversarial Reverse Domain Adaptation.

(Mahmood et al., 2018)

Adversarial Reverse Domain Adaptation

Typical Flow: Transfer Synthetic Data to Real-like Domain. Proposed Flow: Transfer Real Data to Synthetic-like Domain. Remove Patient Specific Features while preserving features necessary for depth estimation.

• F. Mahmood, Nicholas J. Durr et al."Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training."

IEEE Transactions on Medical Imaging (2018).

Adversarial Reverse Domain Adaptation

Faisal Mahmood, Nicholas J. Durr et al."Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training." IEEE Transactions on Medical Imaging (2018).

Patient Specific Details Removed

Shape, Shading, Intensity Preserved

Endoscopy Images

Synthetic-like

Representation (Mahmood et al., 2018)

Endoscopy Depth Estimation

Colonoscopy Video Depth Estimate Colonoscopy Video Depth Estimate

(Mahmood et al., 2018)

Validation – Endoscopy Depth Estimation

(Mahmood et al., 2018)

Estimated Depth to Topography

Faisal Mahmood, Nicholas J. Durr et al."Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training." IEEE Transactions on Medical Imaging (2018).

Polyp Charcterization

Hyperplastic Serrated Adenoma

Can we predict the type of polyp without a biopsy only from RGB Image using limited data?

Polyp Charcterization

Hyperplastic Serrated Adenoma

• 76 Videos
• All videos labeled by 4 Senior Gastroenterologists & 3 Fellows
• Average GI Accuracy: Senior: 63.4% Fellow: 53.7%

Deep Learning for Medical Imaging – Major Challenges

• 1. Limited Annotated Data
• Under representation of rare conditions.
• Limited experts available for annotation.
• Privacy Issues
• Cost
• 2. Domain Adaptation
• Diversity in data, different sensors, cites and patients.
• Patient specific texture and color information.
• 3. Structured Prediction
• Global vs Local features.
• 4. Incorporating Multimodal Information
• Subjective diagnosis is multimodal.

Multimodal Data Fusion

Faisal Mahmood, Nicholas J. Durr et al."Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training." IEEE Transactions on Medical Imaging (2018).

Fusion Network

Depth is an additional predicted modality. Can we fuse depth and RGB to get better polyp classification results? Does depth fusion help train a network that requires less data?

RGB-D Classification via Depth Fusion

(Mahmood et al., 2018)

Data Fusion in Feature Space is better than Concatenation.

Error

Multimodal Densenet

(F. Mahmood et al., 2018)

RGB-D Classification RGB vs RGB-D

Error

Average Accuracy = 93% using RGB-D

20 40 60 80 100 120 140 160 180 200

epochs

0.1 0.2 0.3 0.4 0.5 0.6 0.7

test error DenseNet DenseNet Fit MultiDense MultiDense Fit

RGB vs RGB-D Classification

(F. Mahmood et al., 2018)

RGB+Depth Classificatio n RGB Classificatio n

Gradient Class Activation Maps

RGB Classification RGB-D Classification Adenoma

(F. Mahmood et al., 2018)

RGB Classification RGB-D Classification Input

Gradient Class Activation Maps

(F. Mahmood et al., 2018)

Using just 76 polyp videos with fused depth it is possible to build a classifier with AUC > 0.9

Computational Pathology Computational Endoscopy

Automated Breast Cancer Grading

Nuclear Atypia Tubule/Gland Formation

1 2 3 1 2 3 1 2 3

Mitotic Activity

Typical AI for Pathology Flow

Whole Slide Image

≈ 1 Billion Pixels!

Region of Interest Ground Truth from Different Classes

This Needs a Lot of Labeled Data!

Ductal Carcinoma

Invasive Ductal Carcinoma Other Diagnostic Classes

Convolutional Neural Network

• F. Mahmood, 2018 – (EN.580.142.13)

Grade Ground Truth Grade Interobserver & Intraobserver Variability?

Automated Breast Cancer Grading

• A. Nuclear Atypia
• B. Tubule/Gland Formation

1 2 3 1 2 3 1 2 3

• C. Mitotic Activity

Nuclei Segmentation Mitosis Detection Tubule Detection

A+B+C+

Additional Features

Grade

Can help discover additional morphological features.

Automated Breast Cancer Grading

• A. Nuclear Atypia

1 2 3

Nuclei Segmentation Can we build a single AI model that can segment nuclei from any H&E image regardless or organ?

Labeled Nuclei Segmentation Data

32 1000x1000 Slide Patches from 8 Different Organs

Small subjectively labeled datasets are not enough for capturing the diversity needed for a singular multi-organ nuclei segmentation network.

Sparse Stain Normalization

Breast Kidney Lung Stomach

Synthetic Data Generation for Nuclei Segmentation

Normalized sides from six organs Segmentation Masks Pathology Patches (B) Nuclei Segmentation Patches (A)

The variability in data can be captured using synthetic data. Unpaired Synthetic Data Generation

Unpaired Mapping between random polygons and synthetic H&E patches.

Unpaired Mapping between random polygons and synthetic H&E patches.

Synthetic Pathology Images with Nuclei Segmentation Masks

Deep Learning for Medical Imaging – Major Challenges

• 1. Limited Annotated Data
• Under representation of rare conditions.
• Limited experts available for annotation.
• Privacy Issues
• Cost
• 2. Domain Adaptation
• Diversity in data, different sensors, cites and patients.
• Patient specific texture and color information.
• 3. Structured Prediction
• Global vs Local features.
• 4. Incorporating Multimodal Information
• Subjective diagnosis is multimodal.

Faisal Mahmood, Nicholas J. Durr et al."Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training." IEEE Transactions on Medical Imaging (2018).

Faisal Mahmood, Daniel Borders, et al. "Deep Adversarial Training for Multi-Organ Nuclei Segmentation in Histopathology Images." arXiv preprint arXiv:1810.00236 (2018).

Context-Aware Nuclei Segmentation with No CRF Post-processing Step

Overlapping Nuclei

Aggregated Jaccard Index

(F. Mahmood et al., 2018)

Extension to Mitotic Event Detection

(F. Mahmood et al., 2018)

Synthetic Data Generation Context-Aware Mitosis Detection

94.6% Detection Accuracy

Extension to Epithelium Segmentation

(F. Mahmood et al., 2018)

93.8% Segmentation Accuracy

Extension to Tubule Segmentation

96.9% Segmentation Accuracy

(F. Mahmood et al., 2018)

Tissue Level Semantic Segmentation

TMA Prediction Ground Truth

91.4% Segmentation Accuracy

Predicted Feature Fusion

Grade Prediction

0.2 0.4 0.6 0.8 1

False Positive Rate

0.2 0.4 0.6 0.8 1

True Positive Rate H&E Only - ROC

H&E Only (AUC =0.8425)

Data: 2,210 H&E Slide ROIs (TCGA). Labeling: Labeled by 2 pathologists.

AUC=0.8425

Predicted Feature Fusion

Grade Prediction

AUC Increases by 7.98% - Same Data.

0.2 0.4 0.6 0.8 1

False Positive Rate

0.2 0.4 0.6 0.8 1

True Positive Rate H&E + NS ROC

H&E + NS (AUC =0.9097)

AUC=0.9097

Predicted Feature Fusion

Grade Prediction

AUC Increases by 13.74% - Same Data.

0.2 0.4 0.6 0.8 1

False Positive Rate

0.2 0.4 0.6 0.8 1

True Positive Rate H&E + Nuclei Seg +TF + ME ROC

H&E+NS+TF+ME (AUC =0.9583)

AUC=0.9583

Error

Multimodal Densenet: General Framework for Multimodal Data Fusion

(F. Mahmood et al., 2018)

Multimodal Densenet: General Framework for Multimodal Data Fusion

(F. Mahmood et al., 2018)

Multimodal Fusion Library

Deep Learning for Medical Imaging – Major Challenges

• 1. Limited Annotated Data
• Under representation of rare conditions.
• Limited experts available for annotation.
• Privacy Issues
• Cost
• 2. Domain Adaptation
• Diversity in data, different sensors, cites and patients.
• Patient specific texture and color information.
• 3. Structured Prediction
• Global vs Local features.
• 4. Incorporating Multimodal Information
• Subjective diagnosis is multimodal.

Future Work - Clinical

Aim 1: Segmentation, Classification and Grading

Nuclei Segmentation

Synthetic Data Generation Context-aware Adversarial Methods

Tissue Level Segmentation Whole Slide Classification Methods

Fusing spatial information Classes Malignant Benign DCIS UDH Dense Classifier

…

Aim 2: Biomarker Identification

Identify Features Used to Make Classification Decisions Correlate Identified Features with Patient Diagnosis and Prognosis

Feature-wise Kaplan– Meier Prognostic Model

Features used for classification Aim 4: Multimodal Fusion Aim 3: New Grading System / Response to Specific Therapeutic Agents

Fusing Morphological Features for Classification

Develop New Grading Scheme

Subjective + Cloud Based Analysis of Nottingham vs Proposed Grading Objective Analysis of Nottingham vs Proposed Grading DL Models on Cloud HPC Comparative Analysis

Computational Biophotonics Lab Nicholas J. Durr Richard Chen Greg N. McKay Jordan Sweer Taylor Bobrow Mason Chen Bailey Surtees Eric Chiang JHU School of Medicine Alexander S. Baras Susan Harvey Saowanee Ngamruengphong Department of Computer Science Alan Yuille Greg Haiger Siemens Sandra Sudarsky Daphne Yu

Acknowledgements

Olympus Lee Zhen Google Jesus Trujillo Gomez

faisalm@jhu.edu Code / Data Available at: http://faisal.ai

Thank You.