Radiogenomics: Medical images predicting genomics and impact of - - PowerPoint PPT Presentation

Bradley J Erickson, MD PhD, FSIIM, CIIP Professor and Associate Chair, Radiology Director, Radiology Informatics Lab Mayo Clinic, USA

Radiogenomics: Medical images predicting genomics and impact of multi-dimensions

Disclosures

• Relationships with commercial interests:

– Board / Stock Owner:

• FlowSIGMA
• OneMedNet
• VoiceIT

– Research Support: NVidia Corporation, NIH

Radiologists Replaced in Five Years!!! Geoffrey Hinton:

3

“I think that if you work as a radiologist you are like Wile E. Coyote in the cartoon. You’re already over the edge of the cliff, but you haven’t yet looked down. There’s no ground underneath. It’s just completely

• bvious that in five years deep learning is going to do better than

radiologists, it might be ten years.” Quoted in The New Yorker, April 3, 2017

• 1. Middle Management
• 2. Commodity Salespeople
• 3. Report writers, journalists,

Authors & Announcers

• 4. Accountants & Bookkeepers
• 5. Medical Doctors

(There are 3.5 million professional truck drivers vs 1.1 million doctors)

Meanwhile… Precision Medicine

• Is the identification of unique properties (genes) of a patient or disease that one

can use to select a targeted therapy

• By targeting the unique gene or its pathway, one may be able to deliver a high

dose with minimal side effects

• Investment in precision medicine is $42 billion since 2014. Investment in AI in

that period is about $1.5 billion

Treatment of Disease Now Depends on Genomics

• Brain tumor Tx used to be based on microscope analysis

– Astrocytomas, Oligodendrogliomas, Glioblastomas. – Grade 1-4

• Now, 3 markers:

– MGMT Methylation is a modifier that is also important

Glioma

IDH-WT IDH-mut Bad Prognosis 1p19q Co-Del 1p19q Intact Intermediate Prognosis Good Prognosis

What is the Vector for Precision Imaging?

• Genomics and precision medicine is a requirement
• Big Data/Imaging and Learning from it, is critical

Some people skate to the puck is. I skate to where the puck is going to be. (Wayne Gretzky)

Radiomics & Radiogenomics

• Many diseases are largely determined by genome
• Some diseases are determined by multiple genes
• Some diseases are gene + environment
• Radiomics: Identify properties in images that reflect disease or prognosis
• Radiogenomics: Identify genomics of tissues using radiological images

Why DL Will Advance Radiomics

• Deep Learning Finds Features and Connections vs Just Connections

“Computers Programming Computers”

Hand-Crafted Feature Extraction Learning Feature Extractor Classifier Classifier

Radiogenomics: 1p19q

• 159 astrocytomas with known 1p19q status split for train/test (114) and

validation (45) Sens Sens Spec Spec Accuracy Accuracy CNN-MultiRes no augmentation 0.84 0.73 0.79 CNN-MultiRes augmented 0.93 0.82 0.88

Akkus, J Digit Im, 2017

Deep Learning: IDH1 Mutation

• 155 patients (110 train/test, 45 validation) :

Recall Recall Pr Precision ecision F1 scor F1 score ResNet18 0.61 (+/- 0.12) 0.78 (+/- 0.13) 0.68 (+/- 011) ResNet34 0.91 (+/- 0.09) 0.92 (+/- 0.08) 0.91 (+/- 0.08) ResNet50 0.95 (+/- 0.04) 0.95 (+/- 0.04) 0.95 (+/- 0.04)

Deep Learning: MGMT Methylation

• 155 patients (110 train/test, 45 validation) :

Korfiatis, J Digit Im, 2017

Recall Recall Pr Precision ecision F1 scor F1 score ResNet18 0.78 (+/- 0.19) 0.80 (+/- 0.07) 0.75 (+/- 0.16) ResNet34 0.91 (+/- 0.04) 0.80 (+/- 0.18) 0.82 (+/- 0.15) ResNet50 0.97 (+/- 0.02) 0.97 (+/- 0.02) 0.97 (+/- 0.02)

Machine Learning: Prediction of Renal Failure

Az for predicting who will be in ESRD 8 years after baseline testing: HtTKV & baseline eGFR: 0.67 [0.61-0.72] Textures: 0.85 [0.75-0.94] Textures + TKV: 0.89 [0.79-0.95]

*Kline, Accepted in JASN

Deep Learning: Body Composition

• Amount and distribution of fat, muscle, bone predicts survival in some types of

cancer, and may predict overall long term survival and health

• Automated large scale measurement of CT and correlation with health record

can help validate

• Dice Score 0.98 on 2D. Now extending to 3D…

Source CT Image Human Drawn Truth U-Net Segmentation

Extending to Higher Dimensions

Patches 2D 3D nD

Extending to Higher Dimensions

• Most work today is 2D
• Extension to 3D is fairly straightforward but

– Most medical images not isotropic – Extension to 3D also demands more memory

• Extension to other dimensions should also be valuable

– Other image contrast types (radiologists routinely do this in our heads) – Time (change detection is critical for therapy assessment)

Recent Work on 3D Spatial Dimensions

• V-Net: Volumetric extension of U-net

Recent Work on 3D Spatial Dimensions

• V-Net

– Uses Dice to address class imbalance – Achieved good (not yet great) performance with rather limited training data

• Much non-medical work, particularly 3D face recognition

Other Dimensions

• Medical imagery often has multiple contrast types for a given anatomy

– Before IV contrast / after IV contrast – MRI: T1, T2, DWI, PWI, FLAIR, SWI

• Limited experience with using use despite photographs having 3 color channels

that could map to 3 image types.

• Used VGG Net with MRI image types into the 3 color channels. Applied to a data

set of 204 MRIs labeled as ‘significant’ or ‘not significant’

• Had 4 MRI image types: DWI, ADC, Ktrans, T2

– Since only 3 inputs, they tried combinations of 3 at a time – Using both DWI and ADC seems odd, as ADC is a quantitative version of DWI – T1 ignored – Got good results vs GOFAI – But human was “100%” by definition

Other Dimensions

• Medical imagery often has multiple contrast types for a given anatomy

– Before IV contrast / after IV contrast – MRI: T1, T2, DWI, PWI, FLAIR, SWI

• Limited experience with using use despite photographs having 3 color channels

that could map to 3 image types. Why? – Image Registration (patient motion) – Limited compute? – Patches versus ‘Full View’

Other Dimensions: Time

• Monitoring disease is critical to understanding it

– “The prior exam is the radiologist’s best friend” – Recognizing subtle change is easier if you are looking for a difference

Baseline Follow-up

Progression Regression

Patriarche & Erickson, JDI, 2007

Other Dimensions: Time

• While there are some types of Medical Images that are cinegraphic, and thus

could benefit from ‘movie’ analysis methods, when time is weeks or months, patient/organ deformation is a major challenge

Impediments: Technological

• Image registration

– Current AI algorithms expect that a given X,Y,Z are same tissue – Some deformity occurs over time – Can deep learning be taught to ‘see through’ mis-registration?

• Computation Capacity

– Just as going 2D to 3D requires much more compute power (cores + RAM) extending to nD will increase demand that much more

Into the Future with N-Dimensions

• U-Nets will increasingly be replaced with V-Nets (and T-Nets?)
• Medical-Optimized nets will increasingly be used (versus VGG/Alex Net)

– Nature of problem is often different (‘Is there a cat’ vs ‘Is there a tumor; what type; what is stage; or something else’) – Nature of image is different (lower resolution in 2D) – Third spatial dimension is non-isotropic – Time dimension is different – Other parameters are different than color (collected separately) so need to ‘understand’ margins that are not quite right

Impediments: Political

• Traditional scientists: “No Hypothesis = No Science”
• FDA Clearance
• Finances & Reimbursement Model