Computational Pathology In the Midst of a Revolution: How - PowerPoint PPT Presentation

Computational Pathology In the Midst of a Revolution: How Computational Pathology is Transforming Clinical Practice and Biomedical Research Thomas J. Fuchs Associate Member, Memorial Sloan Kettering Cancer Center Associate Professor, Weill Cornell Graduate School of Medical Sciences Director, Computational Pathology and Medical Machine Learning Lab Department of Medical Physics Department of Pathology fuchst@mskcc.org thomasfuchslab.org Disclaimer: co-founder of Paige.AI

Fuchs Lab @ MSKCC + Weill Cornell Thomas Peter Fem Andrew Hassan Gabe Arjun Amanda

Background Graz, Austria Arnold Schwarzenegger 38 th Governor of California

Bladder Computational Pathology Kidney

150 000 pixel

1,000,000 new glass slides per year @ MSKCC

1,000,000 new glass slides per year @ MSKCC

1,000,000 new glass slides per year @ MSKCC

AI AI Comp mp. Pathology Digital Pathology (Scanning, QC, P-PACS, Image Processing, …) Pathology In Info forma matics (EHR, LIS, Barcoding, fRFID, ...) Wet Laboratory (Physical Slide Production, Cutting, Staining , …) Simplified Pathology Department Stack

Definition Computational Pathology investigates a complete probabilistic treatment of scientific and clinical workflows in general pathology, i.e. it combines experimental design, statistical pattern recognition and survival analysis within an unified framework to answer scientific and clinical questions in pathology. [Fuchs 2011]

• Archimedes lever, 1824 • Mechanic Magazine

Ubiquity of Machine Learning Self-driving Cars Knowledge Systems Computational Pathology Surveillance Cyber Security Space Exploration 25

Computer Vision Tasks in Pathology Nuclei Detection and Classification Sub-cellular level Segmentation Structure Estimation Morphology

Dataset Sizes: Computer Vision vs. Computational Pathology 1 Whole Slide = 100,000 x 60,000 = 6 billion pixels CIFAR-10 (32*32)*60K = 61.44 million pixels All 60,000 CIFAR images fit into this box

Dataset Sizes: Computer Vision vs. Computational Pathology n=1 n=474 All of ImageNet 474 Whole Slides 482 x 415 * 14,197,122 100,000 x 60,000 *474 = 2.8 trillion pixels = 2.8 trillion pixels

Ground Truth for Statistical Learning Labeled samples are needed for training and validation. What is the „Ground Truth“?

Expert & Crowd Sourcing Past Present Future [BD2K Proposal 2014]

Expert Staining Estimation

Intra Pathologist Evaluation 50 nuclei were repeated flipped and rotated to test the intra pathologist variability. Original Flipped & Rotated 53/250 mismatches Baseline: Intra-Pathologists classification uncertainty of ~ 20%

Why is Comp. Path. Challenging? Computational hard Pathology chess genomics expert with decades of training humans repetitive amenable to or boring crowdsourcing easy a child’s play filing manufacturing vision robotics for humans easy machines hard structured machine readable data unstructured/visual data

Computational Pathology

Nucleus Based Analysis DAGM 2008

Applications of the Framework Pancreatic Islet Segmentation for T2 Diabetes Spatial Processes for HippocampalSclerosis Original Image Detected Objects Process Intensity Detection in IHC Stained Cell Cultures Counting of Mouse Liver Hepatocytes

Cell Nuclei Detection

Survival Analysis p = 0.043 p = 0.026 low risk low risk high risk high risk

FGI Grant Quantifying and Correlating Tissue Pathology with the MK-IMPACT Genotype Classic UC E-cadherin IHC Plasmacytoid Ca Plasmacytoid Ca Classic UC One tumor with two morphologies with different mutational profile (but also share 2 mutations indicating same origin).

A Joint Effort for Personalized Medicine Pathology Radiology Genomics Computational Pathology Combining quantitative analyses from pathology, radiology and genomics facilitates true personalized medicine. Radiomics @ MSKCC cBioPortal @ MSKCC

Computational Pathology Datasets Google [Liu et al. 2017] 509 Slides Camelyon Challenge 400 Slides First Computational Pathology Paper GLASS challenge [Fuchs et al. 2008] Equivalent to 500 200 slides 1 Slide (Tissue Microarray) 400 200 20 1 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018

State-of-the-art in March 2017 269 slides for training 240 slides for testing binary classification

State-of-the-art vs. Reality in clinical practice State-of-the art datasetsin pathology: Clinical reality : • • tiny (~400 slides) messy • • very well curated diverse • surprising Like training your autonomous car only on How can we ever hope to an empty parking lot. train clinical-grade models? It has never seen rain, snow or a dirt road.

Number of Digitized Whole Slides Clinical Slide Scanning @ Memorial Sloan Kettering 100000 150000 200000 250000 50000 0 1/1/15 2/1/15 3/1/15 4/1/15 5/1/15 6/1/15 2015 7/1/15 8/1/15 9/1/15 10/1/15 11/1/15 12/1/15 1/1/16 2/1/16 3/1/16 4/1/16 5/1/16 6/1/16 2016 7/1/16 8/1/16 9/1/16 10/1/16 11/1/16 12/1/16 1/1/17 2/1/17 2017 3/1/17 4/1/17

1200000 Clinical Slide Scanning @ Memorial Sloan Kettering 1000000 ~ 1 petabyte of compressed image data 800000 Number of Digitized Whole Slides 600000 Projection with current ramp-up to 40,000 slides / / mo month 400000 200000 0 1/1/15 3/1/15 5/1/15 7/1/15 9/1/15 11/1/15 1/1/16 3/1/16 5/1/16 7/1/16 9/1/16 11/1/16 1/1/17 3/1/17 5/1/17 7/1/17 9/1/17 11/1/17 1/1/18 3/1/18 5/1/18 7/1/18 9/1/18 11/1/18 2015 2016 2017 2018

a machine lear arning solution QC: a sharp the Blur detect ctor [Campanella et al. 2017] blurred thumbnail blur mask sharp blurred blurred

Aperio Hamamatsu Philips cBio Consultation ... Scanner Scanner Scanner Portal Portal Aperio Hamamatsu Philips cBio Portal Consultation ... Viewer Viewer Viewer Viewer Viewer ImageScope Nanozoomer IntelliSite Cancer Digital Slide Archive PathXL ....

Aperio Hamamatsu Philips cBio Consultation ... Scanner Scanner Scanner Portal Portal slides.mskcc.org

High Performance Computing for Pathology Awarded “Center of Excellence for GPU Computing” from for our work in Pathology and cBio. GPU MSKCC’s HPC Cluster 120 GPUs in total Pascal TitanX and 1080 (Ti) GPUs dedicated to Computational Pathology

Memorial Sloan Kettering Cancer Center Deep Learning for Decision Support in Skin Cancer

Basal Cell Carcinoma Prediction Segmentation and Diagnosis Prediction

Whole-slide tumor prediction CNN

Convergence Curves: BCC Classification

97% Accuracy in Predicting BCC Classification Error Logistic Random AlexNet AlexNet ResNet 18 ResNet 18 ResNet 34 Regression Forest pretrained pretrained pretrained

Generative Adversarial Networks for Large-Scale Semantic Image Retrieval

Dreaming of Cancer: A Nightmare of Cancer Samples drawn from our Generative Adversarial Network (GAN) Prostate Cancer Model Natural Images (CIFAR-10)

Computational Pathology Datasets Google [Liu et al. 2017] 509 Slides Camelyon Challenge 400 Slides First Computational Pathology Paper GLASS challenge [Fuchs et al. 2008] 500 200 slides 1 Slide (Tissue Microarray) 400 200 20 1 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018

athology rtificial ntelligence uidance ngine

Computational Pathology Datasets Paige.AI Prostate Biopsy Complete Diagnosis 15,000 Slides Camelyon Challenge 400 Slides First Computational Pathology Paper GLASS challenge [Fuchs et al. 2008] Google 500 200 slides 1 Slide (Tissue Microarray) 400 [Liu et al. 2017] 200 509 Slides 20 1 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018

Changing Clinical Practice MSK-P15K Dataset We generated an unrivaled prostate biopsy dataset of 15,000 whole slides of needle biopsy cores with clinical annotation. Deep Learn rning We are training convolutional neural networks and generative models at scale on our HPC cluster Whole slides of Prostate Needle Biopsies Medical Expert rtise MSK is the nations leading center for prostate cancer consultation with world-renown domain experts, who annotate the data and interactively train our AI. Goal: The firs rst ever r clinical-gra rade Computational Pathology model We developed an unified slide viewer for sample annotation slides.mskcc.org

Fuchs Lab @ MSKCC / Weill Cornell Andrew Schaumberg Thomas Fuchs Peter Schüffler Arjun Raj Rajanna Gabriele Campanella Amanda Beras Hassan Muhammad

MSKCC Collaborators David Klimstra Meera Hameed Victor Reuter Malcolm Pike Klaus Busam Joe Sirintrapun Jinru Shia Hikmat Al-Ahmadie Edi Brogi Oscar Lin Joseph O. Deasy Jung Hun Oh Harini Adity Apte John L. Humm Veeraraghavan