Computational Pathology In the Midst of a Revolution: How - - PowerPoint PPT Presentation

▶
computational pathology
SMART_READER_LITE
LIVE PREVIEW

Computational Pathology In the Midst of a Revolution: How - - PowerPoint PPT Presentation

May 24, 2023 18 likes •695 views

Computational Pathology In the Midst of a Revolution: How Computational Pathology is Transforming Clinical Practice and Biomedical Research Thomas J. Fuchs Associate Member, Memorial Sloan Kettering Cancer Center Associate Professor, Weill

slide-1
SLIDE 1

Computational Pathology

In the Midst of a Revolution: How Computational Pathology is Transforming Clinical Practice and Biomedical Research

Thomas J. Fuchs

Associate Member, Memorial Sloan Kettering Cancer Center Associate Professor, Weill Cornell Graduate School of Medical Sciences Director, Computational Pathology and Medical Machine Learning Lab Department of Medical Physics Department of Pathology fuchst@mskcc.org thomasfuchslab.org Disclaimer: co-founder of Paige.AI

slide-2
SLIDE 2

Fuchs Lab @ MSKCC + Weill Cornell

Thomas Peter Fem Andrew Hassan Gabe Arjun Amanda

slide-3
SLIDE 3

Graz, Austria

Arnold Schwarzenegger 38th Governor of California

Background

slide-4
SLIDE 4

Bladder Kidney Computational Pathology

slide-5
SLIDE 5
slide-6
SLIDE 6
slide-7
SLIDE 7
slide-8
SLIDE 8
slide-9
SLIDE 9
slide-10
SLIDE 10
slide-11
SLIDE 11
slide-12
SLIDE 12
slide-13
SLIDE 13

150 000 pixel

slide-14
SLIDE 14
slide-15
SLIDE 15
slide-16
SLIDE 16
slide-17
SLIDE 17
slide-18
SLIDE 18

1,000,000 new glass slides per year @ MSKCC

slide-19
SLIDE 19

1,000,000 new glass slides per year @ MSKCC

slide-20
SLIDE 20

1,000,000 new glass slides per year @ MSKCC

slide-21
SLIDE 21
slide-22
SLIDE 22

Comp mp. Pathology Digital Pathology (Scanning, QC, P-PACS, Image Processing, …) Pathology In Info forma matics (EHR, LIS, Barcoding, fRFID, ...)

Wet Laboratory

(Physical Slide Production, Cutting, Staining, …)

Simplified Pathology Department Stack

AI AI

slide-23
SLIDE 23

Definition

Computational Pathology investigates a complete probabilistic treatment of scientific and clinical workflows in general pathology, i.e. it combines experimental design, statistical pattern recognition and survival analysis within an unified framework to answer scientific and clinical questions in pathology.

[Fuchs 2011]

slide-24
SLIDE 24
  • Archimedes lever, 1824
  • Mechanic Magazine
slide-25
SLIDE 25

Ubiquity of Machine Learning

25 Self-driving Cars Knowledge Systems Computational Pathology Surveillance Cyber Security Space Exploration

slide-26
SLIDE 26

Computer Vision Tasks in Pathology

Nuclei Detection and Classification

Sub-cellular level

Segmentation Structure Estimation Morphology

slide-27
SLIDE 27

CIFAR-10

(32*32)*60K= 61.44 million pixels

1 Whole Slide = 100,000 x 60,000 = 6 billion pixels

All 60,000 CIFAR images fit into this box

Dataset Sizes: Computer Vision vs. Computational Pathology

slide-28
SLIDE 28

All of ImageNet

482 x 415 * 14,197,122 = 2.8 trillion pixels

n=1 n=474

474 Whole Slides

100,000 x 60,000 *474 = 2.8 trillion pixels

Dataset Sizes: Computer Vision vs. Computational Pathology

slide-29
SLIDE 29

Ground Truth for Statistical Learning

What is the „Ground Truth“?

Labeled samples are needed for training and validation.

slide-30
SLIDE 30

Expert & Crowd Sourcing Past Present Future

[BD2K Proposal 2014]

slide-31
SLIDE 31

Expert Staining Estimation

slide-32
SLIDE 32

Intra Pathologist Evaluation

Original Flipped & Rotated

50 nuclei were repeated flipped and rotated to test the intra pathologist variability.

53/250 mismatches Baseline: Intra-Pathologists classification uncertainty of ~20%

slide-33
SLIDE 33

Why is Comp. Path. Challenging?

easy hard machines easy hard humans

chess manufacturing filing repetitive

  • r boring

robotics expert with decades of training a child’s play for humans vision

Computational Pathology

amenable to crowdsourcing

genomics

structured machine readable data unstructured/visual data

slide-34
SLIDE 34

Computational Pathology

slide-35
SLIDE 35

Nucleus Based Analysis

DAGM 2008

slide-36
SLIDE 36

Original Image Detected Objects Process Intensity

Applications of the Framework

Detection in IHC Stained Cell Cultures Counting of Mouse Liver Hepatocytes Spatial Processes for HippocampalSclerosis Pancreatic Islet Segmentation for T2 Diabetes

slide-37
SLIDE 37

Cell Nuclei Detection

slide-38
SLIDE 38

Survival Analysis

p = 0.043 p = 0.026

low risk high risk low risk high risk

slide-39
SLIDE 39

FGI Grant Quantifying and Correlating Tissue Pathology with the MK-IMPACT Genotype

E-cadherin IHC Plasmacytoid Ca Classic UC Plasmacytoid Ca Classic UC

One tumor with two morphologies with different mutational profile (but also share 2 mutations indicating same origin).

slide-40
SLIDE 40

Pathology Genomics Radiology

Combining quantitative analyses from pathology, radiology and genomics facilitates true personalized medicine.

cBioPortal @ MSKCC Radiomics @ MSKCC

Computational Pathology

A Joint Effort for Personalized Medicine

slide-41
SLIDE 41

2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018

First Computational Pathology Paper [Fuchs et al. 2008] 1 Slide (Tissue Microarray) 1 Google [Liu et al. 2017] 509 Slides Camelyon Challenge 400 Slides GLASS challenge 200 slides 20 200 400 500

Computational Pathology Datasets

Equivalent to

slide-42
SLIDE 42

State-of-the-art in March 2017

269 slides for training 240 slides for testing binary classification

slide-43
SLIDE 43

State-of-the-art vs. Reality in clinical practice

State-of-the art datasetsin pathology:

  • tiny (~400 slides)
  • very well curated

Like training your autonomous car only on an empty parking lot. It has never seen rain, snow or a dirt road. Clinical reality:

  • messy
  • diverse
  • surprising

How can we ever hope to train clinical-grade models?

slide-44
SLIDE 44

50000 100000 150000 200000 250000

1/1/15 2/1/15 3/1/15 4/1/15 5/1/15 6/1/15 7/1/15 8/1/15 9/1/15 10/1/15 11/1/15 12/1/15 1/1/16 2/1/16 3/1/16 4/1/16 5/1/16 6/1/16 7/1/16 8/1/16 9/1/16 10/1/16 11/1/16 12/1/16 1/1/17 2/1/17 3/1/17 4/1/17

Clinical Slide Scanning @ Memorial Sloan Kettering

Number of Digitized Whole Slides

2015 2016 2017

slide-45
SLIDE 45

Number of Digitized Whole Slides

2015 2016 2017

200000 400000 600000 800000 1000000 1200000

1/1/15 3/1/15 5/1/15 7/1/15 9/1/15 11/1/15 1/1/16 3/1/16 5/1/16 7/1/16 9/1/16 11/1/16 1/1/17 3/1/17 5/1/17 7/1/17 9/1/17 11/1/17 1/1/18 3/1/18 5/1/18 7/1/18 9/1/18 11/1/18

2018

Clinical Slide Scanning @ Memorial Sloan Kettering

Projection with current ramp-up to 40,000 slides / / mo month ~ 1 petabyte of compressed image data

slide-46
SLIDE 46

QC: a

a machine lear

arning solution

the Blur detect

ctor

sharp blurred blurred sharp blurred thumbnail blur mask [Campanella et al. 2017]

slide-47
SLIDE 47

Aperio Scanner cBio Portal Consultation Portal ... Aperio Viewer Hamamatsu Viewer Philips Viewer cBio Portal Viewer Consultation Viewer ... Philips Scanner Hamamatsu Scanner

ImageScope Nanozoomer IntelliSite Cancer Digital Slide Archive PathXL ....

slide-48
SLIDE 48

Aperio Scanner cBio Portal Consultation Portal ... Philips Scanner Hamamatsu Scanner slides.mskcc.org

slide-49
SLIDE 49
slide-50
SLIDE 50
slide-51
SLIDE 51

High Performance Computing for Pathology

Awarded “Center of Excellence for GPU Computing” from for our work in Pathology and cBio. 120 GPUs in total Pascal TitanX and 1080 (Ti) GPUs dedicated to Computational Pathology MSKCC’s HPC Cluster GPU

slide-52
SLIDE 52

Memorial Sloan Kettering Cancer Center

Deep Learning

for Decision Support

Skin Cancer

in

slide-53
SLIDE 53

Basal Cell Carcinoma Prediction Segmentation and Diagnosis Prediction

slide-54
SLIDE 54

Whole-slide tumor prediction

CNN

slide-55
SLIDE 55

Convergence Curves: BCC Classification

slide-56
SLIDE 56

97% Accuracy in Predicting BCC

Classification Error

Logistic Regression Random Forest AlexNet AlexNet pretrained ResNet 18 ResNet 18 pretrained ResNet 34 pretrained

slide-57
SLIDE 57

Generative Adversarial Networks for Large-Scale Semantic Image Retrieval

slide-58
SLIDE 58
slide-59
SLIDE 59

Dreaming of Cancer: A Nightmare of Cancer

Samples drawn from our Generative Adversarial Network (GAN)

Prostate Cancer Model Natural Images (CIFAR-10)

slide-60
SLIDE 60

2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018

First Computational Pathology Paper [Fuchs et al. 2008] 1 Slide (Tissue Microarray) 1 Google [Liu et al. 2017] 509 Slides Camelyon Challenge 400 Slides GLASS challenge 200 slides 20 200 400 500

Computational Pathology Datasets

slide-61
SLIDE 61
slide-62
SLIDE 62

athology rtificial ntelligence uidance ngine

slide-63
SLIDE 63

2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018

First Computational Pathology Paper [Fuchs et al. 2008] 1 Slide (Tissue Microarray) 1 Google [Liu et al. 2017] 509 Slides Camelyon Challenge 400 Slides GLASS challenge 200 slides 20 200 400 500

Computational Pathology Datasets

Paige.AI Prostate Biopsy Complete Diagnosis 15,000 Slides

slide-64
SLIDE 64

Changing Clinical Practice

MSK-P15K Dataset We generated an unrivaled prostate biopsy dataset of 15,000 whole slides of needle biopsy cores with clinical annotation. Deep Learn rning We are training convolutional neural networks and generative models at scale on

  • ur HPC cluster

Medical Expert rtise MSK is the nations leading center for prostate cancer consultation with world-renown domain experts, who annotate the data and interactively train our AI. Goal: The firs rst ever r clinical-gra rade Computational Pathology model

Whole slides of Prostate Needle Biopsies We developed an unified slide viewer for sample annotation slides.mskcc.org

slide-65
SLIDE 65

Fuchs Lab @ MSKCC / Weill Cornell

Andrew Schaumberg ThomasFuchs Peter Schüffler Arjun Raj Rajanna Gabriele Campanella Amanda Beras Hassan Muhammad

slide-66
SLIDE 66

MSKCC Collaborators

David Klimstra Meera Hameed Victor Reuter Malcolm Pike Joe Sirintrapun Hikmat Al-Ahmadie Edi Brogi Jinru Shia Klaus Busam Oscar Lin Jung Hun Oh Harini Veeraraghavan Adity Apte John L. Humm Joseph O. Deasy

slide-67
SLIDE 67

Thank you for your attention!

Questions welcomed!

Thomas J. Fuchs fuchst@mskcc.org thomasfuchslab.org Open ML and CS positions in Manhattan jobs@paige.ai