Co Comp mput utat ation ional al Pa Path tholo ology gy at - - PowerPoint PPT Presentation

▶
co comp mput utat ation ional al pa path tholo ology gy
SMART_READER_LITE
LIVE PREVIEW

Co Comp mput utat ation ional al Pa Path tholo ology gy at - - PowerPoint PPT Presentation

Jun 09, 2023 43 likes •937 views

Co Comp mput utat ation ional al Pa Path tholo ology gy at at Sca Scale le Changing Clinical Practice One Petabyte at a Time Thomas s J. Fuchs Associate Member, Memorial Sloan Kettering Cancer Center Associate Professor, Weill

slide-1
SLIDE 1

Co Comp mput utat ation ional al Pa Path tholo

  • logy

gy at at Sca Scale le Changing Clinical Practice One Petabyte at a Time

Thomas s J. Fuchs

Associate Member, Memorial Sloan Kettering Cancer Center Associate Professor, Weill Cornell Graduate School of Medical Sciences Director, Computational Pathology and Medical Machine Learning Lab Director, Warren Alpert Center for Digital and Computational Pathology Department of Pathology fuchst@mskcc.org thomasfuchslab.org

@Thomas asFu Fuch chsA sAI

Disclaimer: Founder and CSO of PAIGE.ai ai

slide-2
SLIDE 2

Fuchs Lab @ MSKCC + Weill Cornell

slide-3
SLIDE 3

The Warren Alpert Center for Digital and Computational Pathology at MSK

est 2017

slide-4
SLIDE 4
slide-5
SLIDE 5
slide-6
SLIDE 6
slide-7
SLIDE 7
slide-8
SLIDE 8
slide-9
SLIDE 9
slide-10
SLIDE 10
slide-11
SLIDE 11
slide-12
SLIDE 12
slide-13
SLIDE 13

1,000,000 new glass slides per year @ MSKCC

slide-14
SLIDE 14

1,000,000 new glass slides per year @ MSKCC

slide-15
SLIDE 15

1,000,000 new glass slides per year @ MSKCC

slide-16
SLIDE 16

15

slide-17
SLIDE 17
slide-18
SLIDE 18

150 000 pixel

slide-19
SLIDE 19
slide-20
SLIDE 20
slide-21
SLIDE 21
slide-22
SLIDE 22

21

CT MRI Lab Sono Derm … CTC Tissue

Pathology

Surgical Pathology Hematopathology Dermatopathology Molecular Pathology

Diagnosis Testing Sequencing Pharma Studies Insurance … Follow-up Screening & Detection Treatment & Research Clinical Workflow

The whole edifice of medicine rests on the pathologist’s diagnosis

slide-23
SLIDE 23

Definition

Computational Pathology investigates a complete probabilistic treatment of scientific and clinical workflows in general pathology, i.e. it combines experimental design, statistical pattern recognition and survival analysis within an unified framework to answer scientific and clinical questions in pathology.

[Fuchs 2011]

slide-24
SLIDE 24

Comp Comp. Patholog thology Digital Digital Patholog thology (Scanning, QC, P-PACS, Image Processing, …) Patholog thology Inf nfor

  • rma

matics tics (EHR, LIS, Barcoding, fRFID, ...)

Wet et La Labo borator tory

(Physical Slide Production, Cutting, Staining, …)

Simplified Pathology Department Stack

AI AI

slide-25
SLIDE 25

24

slide-26
SLIDE 26

Computer Vision Tasks in Pathology

Nuclei Detection and Classification

Sub-cellular level

Segmentation Structure Estimation Morphology

slide-27
SLIDE 27

CIFAR-10

(32*32)*60K= 61.44 million pixels

1 Whole Slide = 100,000 x 60,000 = 6 billion pixels

All 60,000 CIFAR images fit into this box

Dataset Sizes: Computer Vision vs. Computational Pathology

slide-28
SLIDE 28

All of ImageNet

482 x 415 * 14,197,122 = 2.8 trillion pixels

n=1 n=474

474 Whole Slides

100,000 x 60,000 *474 = 2.8 trillion pixels

Dataset Sizes: Computer Vision vs. Computational Pathology

slide-29
SLIDE 29

Ground Truth for Statistical Learning

What is the „Ground Truth“?

Labeled samples are needed for training and validation.

slide-30
SLIDE 30

Expert & Crowd Sourcing Past Present Future

[BD2K Proposal 2014]

slide-31
SLIDE 31

Expert Staining Estimation

slide-32
SLIDE 32

Intra Pathologist Evaluation

Original Flipped & Rotated

50 nuclei were repeated flipped and rotated to test the intra pathologist variability.

53/250 mismatches Baseline: Intra-Pathologists classification uncertainty of ~20%

slide-33
SLIDE 33

Why is Computational Pathology so challenging?

slide-34
SLIDE 34

Computational Pathology

slide-35
SLIDE 35

Nucleus Based Analysis

DAGM 2008

slide-36
SLIDE 36

Original Image Detected Objects Process Intensity

Applications of the Framework

Detection in IHC Stained Cell Cultures Counting of Mouse Liver Hepatocytes Spatial Processes for Hippocampal Sclerosis Pancreatic Islet Segmentation for T2 Diabetes

slide-37
SLIDE 37

Cell Nuclei Detection

slide-38
SLIDE 38

Survival Analysis

p = 0.043 p = 0.026

low risk high risk low risk high risk

slide-39
SLIDE 39

Functional Genomics Quantifying and Correlating Tissue Pathology with the MK-IMPACT Genotype

E-cadherin IHC Plasmacytoid Ca Classic UC Plasmacytoid Ca Classic UC

One tumor with two morphologies with different mutational profile (but also share 2 mutations indicating same origin).

slide-40
SLIDE 40

Pathology Genomics Radiology

Combining quantitative analyses from pathology, radiology and genomics facilitates true personalized medicine.

cBioPortal @ MSKCC Radiomics @ MSKCC

Computational Pathology

A Joint Effort for Personalized Medicine

slide-41
SLIDE 41

2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018

First Computational Pathology Paper [Fuchs et al. 2008] 1 Slide (Tissue Microarray) 1 Camelyon Challenge 400 Slides GLASS challenge 200 slides 2 20 40

Computational Pathology Datasets

slide-42
SLIDE 42

State-of-the-art in March 2017

269 slides for training 129 slides for testing binary classification

slide-43
SLIDE 43

2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018

First Computational Pathology Paper [Fuchs et al. 2008] 1 Slide (Tissue Microarray) 1 Google [Liu et al. 2017] 509 Slides Camelyon Challenge 400 Slides GLASS challenge 200 slides 2 20 40 50

Computational Pathology Datasets

Equivalent to

slide-44
SLIDE 44

State-of-the-art vs. Reality in clinical practice

State-of-the art datasets in pathology:

  • tiny (~400 slides)
  • very well curated

Like training your autonomous car only on an empty parking lot. It has never seen rain, snow or a dirt road.

slide-45
SLIDE 45

State-of-the-art vs. Reality in clinical practice

State-of-the art datasets in pathology:

  • tiny (~400 slides)
  • very well curated

Like training your autonomous car only on an empty parking lot. It has never seen rain, snow or a dirt road. Clinical reality:

  • messy
  • diverse
  • surprising

How can we ever hope to train clinical-grade models?

slide-46
SLIDE 46

50000 100000 150000 200000 250000

Clinical Slide Scanning @ Memorial Sloan Kettering

Number of Digitized Whole Slides

2015 2016 2017

slide-47
SLIDE 47

Number of Digitized Whole Slides

2015 2016 2017

200000 400000 600000 800000 1000000 1200000

1/1/15 1/1/16 1/1/17 1/1/18

2018

Clinical Slide Scanning @ Memorial Sloan Kettering

Projection with current ramp-up to 40,00 ,000 sli lides / / mo month ~ 1 pe petabyte of compressed image data

slide-48
SLIDE 48

2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018

First Computational Pathology Paper [Fuchs et al. 2008] 1 Slide (Tissue Microarray) 1 Google [Liu et al. 2017] 509 Slides Camelyon Challenge 400 Slides GLASS challenge 200 slides 2 200 400 500

Computational Pathology Datasets

slide-49
SLIDE 49
slide-50
SLIDE 50
slide-51
SLIDE 51

2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018

First Computational Pathology Paper [Fuchs et al. 2008] 1 Slide (Tissue Microarray) 1 Google [Liu et al. 2017] 509 Slides Camelyon Challenge 400 Slides GLASS challenge 200 slides 2 20 40 50

Computational Pathology Datasets

Paige.AI Prostate Biopsy Complete Diagnosis 15,000 Slides

slide-52
SLIDE 52

QC: QC: a

a mach

chin ine lea earnin ing g solu solutio ion

th the Blu

lur detector

sharp blurred blurred sharp blurred thumbnail blur mask [Campanella et al. 2017]

slide-53
SLIDE 53

Aperio Scanner cBio Portal Consultation Portal ... Aperio Viewer Hamamatsu Viewer Philips Viewer cBio Portal Viewer Consultation Viewer ... Philips Scanner Hamamatsu Scanner

ImageScope Nanozoomer IntelliSite Cancer Digital Slide Archive PathXL ....

slide-54
SLIDE 54

Aperio Scanner cBio Portal Consultation Portal ... Philips Scanner Hamamatsu Scanner slides.mskcc.org

slide-55
SLIDE 55
slide-56
SLIDE 56
slide-57
SLIDE 57

High Perf orman c e Co mp u tin g f or Path ology

Awarded “Center of Excellence for GPU Computing” from for our work in Pathology and csBio. 320 GPUs in total Pascal TitanX and 1080 (Ti) GPUs dedicated to Computational Pathology MSKCC’s HPC Cluster

slide-58
SLIDE 58

Deep Learning Cluster for Computational Pathology @MSKCC

slide-59
SLIDE 59

58

6 DGX-1 V100

Deep Learning at Scale DGX-1 Cluster for Computational Pathology

slide-60
SLIDE 60

Memorial Sloan Kettering Cancer Center

Deep Learning

for Decision Support

Skin Cancer

in

slide-61
SLIDE 61

Basal Cell Carcinoma Prediction Segmentation and Diagnosis Prediction

slide-62
SLIDE 62

97% Accuracy in Predicting BCC

slide-63
SLIDE 63

Frozen H&E IHC Fluorescent Single Cell FNA Bone & Soft Tissue Breast Dermatopathology Gastrointestinal Genitourinary Gynecologic Head and Neck Neuropathology Thoracic Hematopathology Cytology

slide-64
SLIDE 64

Frozen H&E IHC Fluorescent Single Cell FNA Bone & Soft Tissue Breast Dermatopathology Gastrointestinal Genitourinary Gynecologic Head and Neck Neuropathology Thoracic Hematopathology Cytology

slide-65
SLIDE 65

Frozen H&E IHC Fluorescent Single Cell FNA Bone & Soft Tissue Breast Dermatopathology Gastrointestinal Genitourinary Gynecologic Head and Neck Neuropathology Thoracic Hematopathology Cytology

slide-66
SLIDE 66

Frozen H&E IHC Fluorescent Single Cell FNA Bone & Soft Tissue Breast Dermatopathology Gastrointestinal Genitourinary Gynecologic Head and Neck Neuropathology Thoracic Hematopathology Cytology

slide-67
SLIDE 67

Frozen H&E IHC Fluorescent Single Cell FNA Bone & Soft Tissue Breast Dermatopathology Gastrointestinal Genitourinary Gynecologic Head and Neck Neuropathology Thoracic Hematopathology Cytology

slide-68
SLIDE 68

Frozen H&E IHC Fluorescent Single Cell FNA Bone & Soft Tissue Breast Dermatopathology Gastrointestinal Genitourinary Gynecologic Head and Neck Neuropathology Thoracic Hematopathology Cytology

slide-69
SLIDE 69

Frozen H&E IHC Fluorescent Single Cell FNA Bone & Soft Tissue Breast Dermatopathology Gastrointestinal Genitourinary Gynecologic Head and Neck Neuropathology Thoracic Hematopathology Cytology

slide-70
SLIDE 70

Frozen H&E IHC Fluorescent Single Cell FNA Bone & Soft Tissue Breast Dermatopathology Gastrointestinal Genitourinary Gynecologic Head and Neck Neuropathology Thoracic Hematopathology Cytology

slide-71
SLIDE 71

Frozen H&E IHC Fluorescent Single Cell FNA Bone & Soft Tissue Breast Dermatopathology Gastrointestinal Genitourinary Gynecologic Head and Neck Neuropathology Thoracic Hematopathology Cytology

slide-72
SLIDE 72

Frozen H&E IHC Fluorescent Single Cell FNA Bone & Soft Tissue Breast Dermatopathology Gastrointestinal Genitourinary Gynecologic Head and Neck Neuropathology Thoracic Hematopathology Cytology

slide-73
SLIDE 73

N u m b e r o f S l i d e s

PAIGE.AI 20,000 MSKCC 12,000 Google 500

slide-74
SLIDE 74

73

Strongly Supervised Learning Pixel-level Annotation “Classical” supervised model. Weakly Supervised Learning Image-level Annotation Multiple Instance learning Dictionary Learning, etc. Binary label for the whole slide from the pathology report.

slide-75
SLIDE 75

74

slide-76
SLIDE 76

0,1 labeled bags … instances bag … ranked instances model inference cross-entropy loss model learning

Multiple Instance Learning

for Prostate Needle Biopsies

slide-77
SLIDE 77
slide-78
SLIDE 78
slide-79
SLIDE 79

Multiple Instance Learning

for Prostate Needle Biopsies

88.0% 87.6% 83.7%

0.00 1.00 0.50 0.75 0.25

slide-80
SLIDE 80

79

slide-81
SLIDE 81

Changing Clinical Practice

MSK-P15 P15K K Data Dataset et We generated an unrivaled prostate biopsy dataset of 15,000 whole slides of needle biopsy cores with clinical annotation. Deep Deep Le Learnin rning We are training convolutional neural networks and generative models at scale on

  • ur HPC cluster

Medi edical l Expert xpertis ise MSK is the nations leading center for prostate cancer consultation with world- renown domain experts, who annotate the data and interactively train our AI. Goal: Th The e fi first rst ev ever er cli clini nical-gra rade Compu Computa tatio tional nal Pa Patho tholo logy y model el

Whole slides of Prostate Needle Biopsies We developed an unified slide viewer for sample annotation slides.mskcc.org

slide-82
SLIDE 82

Fuchs Lab @ MSKCC / Weill Cornell

Andrew Schaumberg Thomas Fuchs Peter Schüffler Arjun Raj Rajanna Gabriele Campanella Amanda Beras Hassan Muhammad

slide-83
SLIDE 83

MSKCC Collaborators

David Klimstra Meera Hameed Victor Reuter Malcolm Pike Joe Sirintrapun Hikmat Al-Ahmadie Edi Brogi Jinru Shia Klaus Busam Oscar Lin Jung Hun Oh Harini Veeraraghavan Adity Apte John L. Humm Joseph O. Deasy

slide-84
SLIDE 84

Thank you for your attention!

Questions welcomed!

Thomas J. Fuchs fuchst@mskcc.org thomasfuchslab.org

Open ML and CS positions in Manhattan paige.ai

@ThomasFuchsAI

slide-85
SLIDE 85

84

slide-86
SLIDE 86

85

slide-87
SLIDE 87

Generative Adversarial Networks for Large-Scale Semantic Image Retrieval

slide-88
SLIDE 88
slide-89
SLIDE 89

Dreaming of Cancer: A Nightmare of Cancer

Samples drawn from our Generative Adversarial Network (GAN)

Prostate Cancer Model Natural Images (CIFAR-10)