Ra Radi diology ology Ron onal ald d M. M. Sum ummers, - - PowerPoint PPT Presentation

▶
ra radi diology ology
SMART_READER_LITE
LIVE PREVIEW

Ra Radi diology ology Ron onal ald d M. M. Sum ummers, - - PowerPoint PPT Presentation

Feb 06, 2023 195 likes •731 views

Th The I e Imp mpac act t of of De Deep ep Le Lear arning ning on on Ra Radi diology ology Ron onal ald d M. M. Sum ummers, mers, M. M.D., ., Ph. h.D. D. Se Seni nior or In Investiga estigato tor Imaging Biomarkers

slide-1
SLIDE 1

Th The I e Imp mpac act t of

  • f De

Deep ep Le Lear arning ning on

  • n

Ra Radi diology

  • logy

Ron

  • nal

ald d M.

  • M. Sum

ummers, mers, M. M.D., ., Ph. h.D. D. Se Seni nior

  • r In

Investiga estigato tor

Imaging Biomarkers and CAD Laboratory Radiology and Imaging Sciences NIH Clinical Center Bethesda, MD www.cc.nih.gov/drd/summers.html

slide-2
SLIDE 2

Disclosure

  • Patent royalties from iCAD

Disclaimer

  • Opinions discussed are my alone and

do not necessarily represent those of NIH or DHHS.

slide-3
SLIDE 3
slide-4
SLIDE 4

Overview

  • Background
  • Radiology imaging applications
  • Data mining radiology reports and images
  • Challenges and pitfalls
slide-5
SLIDE 5

We’ve Entered the Deep Learning Era

  • Hand-crafted features less important
  • Large annotated datasets more important
  • Impact: More and varied researchers can

contribute, accelerating the pace of progress

slide-6
SLIDE 6

Deep Learning

  • Convolutional neural networks (ConvNets)
  • An improvement to neural networks
  • More layers permit higher levels of abstraction
  • Similarities to low level vision processing in

animals

  • Marked improvements in solving hard problems

like object recognition in pictures

slide-7
SLIDE 7

H Roth et al., SPIE MI 2015

slide-8
SLIDE 8

Deep Learning Improves CAD

Roth et al. IEEE TMI 2015

slide-9
SLIDE 9

Deep Learning Improves CAD

Roth et al. IEEE TMI 2015

slide-10
SLIDE 10

Lymphadenopathy CAD

Hua, Liu, Summers et al. ARRS 2012

slide-11
SLIDE 11
  • 90 CTs with

388 mediastinal LNs

  • 86 CTs with

595 abdominal LNs

  • Sensitivities

70%/83% at 3 FP/vol. and 84%/90% at 6 FP/vol., respectively

H Roth et al., MICCAI 2014

slide-12
SLIDE 12
  • Deeper CNN model performed best
  • GoogLeNet for mediastinal LNs
  • Sensitivity 85% at 3 FP/vol.

HC Shin et al., IEEE TMI 2016

slide-13
SLIDE 13

Lymph Node CT Dataset

  • doi.org/10.7937/K9/TCIA.2015.AQIIDCNM
  • TCIA CT Lymph Node
  • 176 scans, 58 GB
  • Also: annotations, candidates, masks
slide-14
SLIDE 14

Detection of Conglomerate Lymph Node Clusters

A Gupta et al.

slide-15
SLIDE 15

Pancreas CAD

Dice 87.5%

A Farag et al. MICCAI Abd WS 2014; RSNA 2014

slide-16
SLIDE 16

Pancreas CAD using CNN

H Roth et al., SPIE MI 2015

slide-17
SLIDE 17

Pancreas CT Dataset

  • doi.org/10.7937/K9/TCIA.2016.tNB1kqBU
  • TCIA CT Pancreas
  • 82 scans, 10 GB
slide-18
SLIDE 18

Gao et al. IEEE ISBI 2016

Segmentation Label Propagation

slide-19
SLIDE 19

Gao et al. IEEE ISBI 2016

Segmentation Label Propagation

slide-20
SLIDE 20

Colitis CAD

Wei et al. SPIE, ISBI 2013

slide-21
SLIDE 21

Colitis CAD

J Liu et al. SPIE Med Imaging 2016

slide-22
SLIDE 22

Colitis CAD

J Liu et al. SPIE Med Imaging and ISBI 2016

  • 26 CT scans of patients with colitis
  • 260 images
  • 85% sensitivity at 1 FP/image
slide-23
SLIDE 23

Spine Metastasis CAD

J Burns, J Yao et al. RSNA 2011; Radiology 2013

slide-24
SLIDE 24

Deep Learning Improves CAD

Roth et al. IEEE TMI 2015

slide-25
SLIDE 25

Vertebral Fracture CAD

Yao et al. CMIG 2014

slide-26
SLIDE 26

Vertebral Fracture CAD

Burns et al. Radiology 2016

  • 92% sensitivity for fracture localization
  • 1.6 FPs per patient
  • Most common FP: nutrient foramina

(39% of all FPs)

slide-27
SLIDE 27

Posterior Elements Fracture CAD

Roth et al. SPIE Med Imaging 2016

  • 18 trauma

patients

  • 55 fractures
  • Test set AUC

0.857

  • 71% / 81%

sensitivities at 5 / 10 FP/ patient

slide-28
SLIDE 28

H Roth et al., IEEE ISBI 2015

Anatomy Classification Using Deep Convolutional Nets

  • 1,675

patients

  • 4,298 images
  • Test set AUC

0.998

  • 5.9%

classification error

slide-29
SLIDE 29

ImageNet

  • 14,197,122 images, 21841 synonym sets

indexed

  • 1,034,908 bounding box annotations
  • Annual challenge inspires fierce

competition

  • ImageNet Large Scale Visual Recognition

Challenge (ILSVRC)

Image credit: http://www.image-net.org

slide-30
SLIDE 30

HC Shin et al. CVPR 2015

Data Mining Reports & Images

slide-31
SLIDE 31

Data Mining Reports & Images

  • Trained on 216,000 key images (CT, MR, …)
  • 169,000 CT images
  • 60,000 patient scans
  • Recall-at-K, K=1 (R@1 score)) was 0.56
slide-32
SLIDE 32

HC Shin et al. CVPR 2015 & JMLR 2016

Data Mining Reports & Images

slide-33
SLIDE 33

Topic: Metastases

HC Shin et al. CVPR 2015 & JMLR 2016

slide-34
SLIDE 34

Data Mining Reports & Images

HC Shin et al. CVPR 2015 & JMLR 2016

slide-35
SLIDE 35

HC Shin et al. CVPR 2016

Data Mining Reports & Images

slide-36
SLIDE 36

HC Shin et al. CVPR 2016

Data Mining Reports & Images

slide-37
SLIDE 37

HC Shin et al. CVPR 2016

Data Mining Reports & Images

slide-38
SLIDE 38

Challenges and Pitfalls

  • Network architecture
  • Convolution
  • DropOut
  • Memory (e.g., LSTM)
  • Max pooling
  • Softmax
  • Number of layers
  • Combining classifiers
slide-39
SLIDE 39

Challenges and Pitfalls

  • Data
  • Data augmentation
  • Dataset size
  • Annotation quality
  • Disease (focused vs. comprehensive)
  • Availability
slide-40
SLIDE 40

Data Augmentation

  • During training, input images are sampled

at different scales and random non-rigid deformations

  • Degree of deformation is chosen such that

the resulting warped images resemble plausible physical variations of the medical images

  • Can help avoid overfitting
slide-41
SLIDE 41

ConvNet training with scales and non-rigid deformations

TPS deformation fields Data augmentation at each superpixel bounding box:

  • Ns scales (zoom-out)
  • Nd deformations

~800k training images from 60 patients

Roth et al. RSNA 2015

slide-42
SLIDE 42

Approach

  • If we can create databases of the entire

radiology image & report collection of one or more hospitals, we will have large datasets amenable for deep learning any radiology CAD task.

slide-43
SLIDE 43

Challenges and Pitfalls

  • Need labels for the images
  • Radiology reports
  • Crowdsourcing
  • Weakly-supervised learning
  • Transfer learning
slide-44
SLIDE 44

Approach

  • ImageNet approach using crowdsourcing

annotations is not feasible due to lack of radiology expertise.

  • The radiologist reports are the annotations.
  • Since every radiology study has a report,

every study has already been annotated by an expert.

slide-45
SLIDE 45

Challenges and Pitfalls

  • Computation
  • GPU acceleration allows efficient training
  • Few implementations currently permit use of

GPU clusters (MxNet)

  • Learning curve varies widely for publicly

available software platforms

slide-46
SLIDE 46

Publicly Available Code

  • Caffe (AlexNet, VGGNet, GoogLeNet)
  • Theano
  • Torch
  • TensorFlow
  • CNTK (ResNet)
  • MxNet
slide-47
SLIDE 47
slide-48
SLIDE 48

Conclusions

  • Deep learning leading to large

improvements in CAD and segmentation

  • Pace of deep learning technology

exceptionally fast

  • Big data permit new advances
  • Interest in deep learning and big data in

radiology image processing is soaring

slide-49
SLIDE 49

Acknowledgments

  • Jack Yao
  • Jiamin Liu
  • Le Lu
  • Nathan Lay
  • Evrim Turkbey
  • Amal Farag
  • Holger Roth
  • Hoo-Chang Shin
  • Xiaosong Wang
  • Andrew Sohn
  • Nicholas Petrick
  • Berkman Sahiner
  • Joseph Burns
  • Perry Pickhardt
  • Mingchen Gao
  • Daniel Mollura
  • Nvidia for GPU

card donations

slide-50
SLIDE 50

Acknowledgements

 NCI  NHLBI  NIDDK  CC  FDA  Mayo Clinic  DOD  U. Wisconsin  Stanford U.  NIH Fellowship Programs:

 Fogarty  ISTP  IRTA  BESIP  CRTP

slide-51
SLIDE 51

To Learn More …

www.cc.nih.gov/drd/summers.html www.cc.nih.gov/drd/info/cips.html

Lung Nodule Detection Tissue Classification Atlas Spine Labeling Virtual Bronchoscopy Angiography (COW) Tumor Analysis