Co Comp mput utat ation ional al Pa Path tholo ology gy at at Sca Scale le Changing Clinical Practice One Petabyte at a Time Thomas s J. Fuchs Associate Member, Memorial Sloan Kettering Cancer Center Associate Professor, Weill Cornell Graduate School of Medical Sciences Disclaimer: Director, Computational Pathology and Medical Machine Learning Lab Director, Warren Alpert Center for Digital and Computational Pathology Founder and CSO of PAIGE.ai ai Department of Pathology fuchst@mskcc.org thomasfuchslab.org @Thomas asFu Fuch chsA sAI
Fuchs Lab @ MSKCC + Weill Cornell
The Warren Alpert Center for Digital and Computational Pathology at MSK est 2017
1,000,000 new glass slides per year @ MSKCC
1,000,000 new glass slides per year @ MSKCC
1,000,000 new glass slides per year @ MSKCC
15
150 000 pixel
The whole edifice of medicine rests on the pathologist’s diagnosis Treatment Screening & Detection & Research CT Testing MRI Sequencing Lab Pharma Tissue Diagnosis Pathology Sono Studies Surgical Pathology Derm Insurance Hematopathology Dermatopathology … … Molecular Pathology CTC Follow-up Clinical Workflow 21
Definition Computational Pathology investigates a complete probabilistic treatment of scientific and clinical workflows in general pathology, i.e. it combines experimental design, statistical pattern recognition and survival analysis within an unified framework to answer scientific and clinical questions in pathology. [Fuchs 2011]
Simplified Pathology Department Stack AI AI Comp. Comp Patholog thology Digital Patholog Digital thology (Scanning, QC, P-PACS, Image Processing, …) Patholog thology Inf nfor orma matics tics (EHR, LIS, Barcoding, fRFID, ...) Wet et La Labo borator tory (Physical Slide Production, Cutting, Staining , …)
24
Computer Vision Tasks in Pathology Nuclei Detection and Classification Sub-cellular level Segmentation Structure Estimation Morphology
Dataset Sizes: Computer Vision vs. Computational Pathology 1 Whole Slide = 100,000 x 60,000 CIFAR-10 = 6 billion pixels (32*32)*60K = 61.44 million pixels All 60,000 CIFAR images fit into this box
Dataset Sizes: Computer Vision vs. Computational Pathology n=1 n=474 All of ImageNet 474 Whole Slides 482 x 415 * 14,197,122 100,000 x 60,000 *474 = 2.8 trillion pixels = 2.8 trillion pixels
Ground Truth for Statistical Learning Labeled samples are needed for training and validation. What is the „Ground Truth“?
Expert & Crowd Sourcing Past Present Future [BD2K Proposal 2014]
Expert Staining Estimation
Intra Pathologist Evaluation 50 nuclei were repeated flipped and rotated to test the intra pathologist variability. Original Flipped & Rotated 53/250 mismatches Baseline: Intra-Pathologists classification uncertainty of ~ 20%
Why is Computational Pathology so challenging?
Computational Pathology
Nucleus Based Analysis DAGM 2008
Applications of the Framework Pancreatic Islet Segmentation for T2 Diabetes Spatial Processes for Hippocampal Sclerosis Original Image Detected Objects Process Intensity Detection in IHC Stained Cell Cultures Counting of Mouse Liver Hepatocytes
Cell Nuclei Detection
Survival Analysis p = 0.043 p = 0.026 low risk low risk high risk high risk
Functional Genomics Quantifying and Correlating Tissue Pathology with the MK-IMPACT Genotype Classic UC E-cadherin IHC Plasmacytoid Ca Classic UC Plasmacytoid Ca One tumor with two morphologies with different mutational profile (but also share 2 mutations indicating same origin).
A Joint Effort for Personalized Medicine Pathology Radiology Genomics Computational Pathology Combining quantitative analyses from pathology, radiology and genomics facilitates true personalized medicine. Radiomics @ MSKCC cBioPortal @ MSKCC
Computational Pathology Datasets Camelyon Challenge 400 Slides First Computational Pathology Paper GLASS challenge [Fuchs et al. 2008] 200 slides 1 Slide (Tissue Microarray) 40 20 0 2 1 0 0 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018
State-of-the-art in March 2017 269 slides for training 129 slides for testing binary classification
Computational Pathology Datasets Google [Liu et al. 2017] 509 Slides Camelyon Challenge 400 Slides First Computational Pathology Paper GLASS challenge [Fuchs et al. 2008] Equivalent to 50 200 slides 1 Slide (Tissue Microarray) 40 0 20 0 2 1 0 0 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018
State-of-the-art vs. Reality in clinical practice State-of-the art datasets in pathology: • tiny (~400 slides) • very well curated Like training your autonomous car only on an empty parking lot. It has never seen rain, snow or a dirt road.
State-of-the-art vs. Reality in clinical practice State-of-the art datasets in pathology: Clinical reality : • • tiny (~400 slides) messy • • very well curated diverse • surprising Like training your autonomous car only on How can we ever hope to an empty parking lot. train clinical-grade models? It has never seen rain, snow or a dirt road.
Clinical Slide Scanning @ Memorial Sloan Kettering 250000 Number of Digitized Whole Slides 200000 150000 100000 50000 0 2015 2016 2017
1200000 Clinical Slide Scanning @ Memorial Sloan Kettering 1000000 ~ 1 pe petabyte of compressed image data Number of Digitized Whole Slides 800000 600000 Projection with current ramp-up to 40,00 ,000 sli lides / / mo month 400000 200000 0 1/1/15 1/1/16 1/1/17 1/1/18 2015 2016 2017 2018
Computational Pathology Datasets Google [Liu et al. 2017] 509 Slides Camelyon Challenge 400 Slides First Computational Pathology Paper GLASS challenge [Fuchs et al. 2008] 500 200 slides 1 Slide (Tissue Microarray) 400 200 2 1 0 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018
Computational Pathology Datasets Paige.AI Prostate Biopsy Complete Diagnosis 15,000 Slides Camelyon Challenge 400 Slides First Computational Pathology Paper GLASS challenge [Fuchs et al. 2008] Google 50 200 slides 1 Slide (Tissue Microarray) 40 [Liu et al. 2017] 0 20 0 509 Slides 2 1 0 0 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018
a mach chin ine lea earnin ing g solu solutio ion QC: QC: a sharp the Blu lur detector th [Campanella et al. 2017] blurred thumbnail blur mask sharp blurred blurred
Aperio Hamamatsu Philips cBio Consultation ... Scanner Scanner Scanner Portal Portal Aperio Hamamatsu Philips cBio Portal Consultation ... Viewer Viewer Viewer Viewer Viewer ImageScope Nanozoomer IntelliSite Cancer Digital Slide Archive PathXL ....
Aperio Hamamatsu Philips cBio Consultation ... Scanner Scanner Scanner Portal Portal slides.mskcc.org
High Perf orman c e Co mp u tin g f or Path ology Awarded “Center of Excellence for GPU Computing” from for our work in Pathology and csBio. MSKCC’s HPC Cluster 320 GPUs in total Pascal TitanX and 1080 (Ti) GPUs dedicated to Computational Pathology
Deep Learning Cluster for Computational Pathology @MSKCC
Deep Learning at Scale DGX-1 Cluster for Computational Pathology 6 DGX-1 V100 58
Memorial Sloan Kettering Cancer Center Deep Learning for Decision Support in Skin Cancer
Basal Cell Carcinoma Prediction Segmentation and Diagnosis Prediction
97% Accuracy in Predicting BCC
Frozen H&E IHC Fluorescent Single Cell FNA Bone & Soft Tissue Breast Dermatopathology Gastrointestinal Genitourinary Gynecologic Head and Neck Neuropathology Thoracic Hematopathology Cytology
Frozen H&E IHC Fluorescent Single Cell FNA Bone & Soft Tissue Breast Dermatopathology Gastrointestinal Genitourinary Gynecologic Head and Neck Neuropathology Thoracic Hematopathology Cytology
Frozen H&E IHC Fluorescent Single Cell FNA Bone & Soft Tissue Breast Dermatopathology Gastrointestinal Genitourinary Gynecologic Head and Neck Neuropathology Thoracic Hematopathology Cytology
Frozen H&E IHC Fluorescent Single Cell FNA Bone & Soft Tissue Breast Dermatopathology Gastrointestinal Genitourinary Gynecologic Head and Neck Neuropathology Thoracic Hematopathology Cytology
Frozen H&E IHC Fluorescent Single Cell FNA Bone & Soft Tissue Breast Dermatopathology Gastrointestinal Genitourinary Gynecologic Head and Neck Neuropathology Thoracic Hematopathology Cytology
Frozen H&E IHC Fluorescent Single Cell FNA Bone & Soft Tissue Breast Dermatopathology Gastrointestinal Genitourinary Gynecologic Head and Neck Neuropathology Thoracic Hematopathology Cytology
Recommend
More recommend
