Identifying Carotid Plaque Composition in MRI with Convolutional - - PowerPoint PPT Presentation

▶
identifying carotid plaque composition in mri with
SMART_READER_LITE
LIVE PREVIEW

Identifying Carotid Plaque Composition in MRI with Convolutional - - PowerPoint PPT Presentation

Feb 27, 2024 214 likes •692 views

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks Author: Yuxi Dong Director: Wei Xu 1 Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks Background: Atherosclerosis Caused by

slide-1
SLIDE 1

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

Author: Yuxi Dong Director: Wei Xu 1

slide-2
SLIDE 2
  • Caused by accumulation of

substances in arteries

  • Cause stroke, the second place in

global death ranks from 1990 to 2010

Background: Atherosclerosis

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

2

Background Dataset Method Results Conclusion

slide-3
SLIDE 3
  • What we see: Plaque

n Reduced or blocked blood flow

  • When a plaque breaks up

n Rupture from vessel n Flow with blood to other parts of body n May block the vessel somewhere

  • Composition of the plaque =>

different risk level

Background: Dangerous of carotid plaques

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

Atherosclerosis[1]

[1] What Is Atherosclerosis?, http://www.nhlbi.nih.gov/health/health-topics/topics/atherosclerosis/

Vessel Plaque Blood flow 3

Background Dataset Method Results Conclusion

slide-4
SLIDE 4
  • We focus on carotid vessels (arteries on the neck)
  • Traditional method: MRI + Trained radiologist

n Time consuming n Requires expertise n Inter-reviewer variability

  • We want to identify the composition of carotid plaques in MRI automatically

Our goal: Identify composition of plaques

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

4

Background Dataset Method Results Conclusion

slide-5
SLIDE 5
  • Background of MRI and plaques
  • Dataset and preprocessing
  • Our model
  • Evaluation

Outline

5

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

Background Dataset Method Results Conclusion

slide-6
SLIDE 6

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

MRI produces multi-contrast images

  • 4 contrast weightings: T1W, T2W, TOF, MP-RAGE
  • Each from a different physical scanning method

T1W T2W TOF MP-RAGE Cross section 6

Background Dataset Method Results Conclusion

slide-7
SLIDE 7

The vessel: when it is normal

T1W T2W TOF MP-RAGE Manual 7

Background Dataset Method Results Conclusion

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

slide-8
SLIDE 8

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

When there is a plaque

  • Calcification: calcium builds up in blood vessels

T1W T2W TOF MP-RAGE Manual 8

Background Dataset Method Results Conclusion

slide-9
SLIDE 9

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

When there is a plaque

  • Calcification: calcium builds up in blood vessels
  • Lipid-rich/necrotic core (LR/NC): extracellular mass in the intima

T1W T2W TOF MP-RAGE Manual 9

Background Dataset Method Results Conclusion

slide-10
SLIDE 10

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

When there is a plaque

  • Calcification: calcium builds up in blood vessels
  • Lipid-rich/necrotic core (LR/NC): extracellular mass in the intima
  • Hemorrhage: liquid plaque component

T1W T2W TOF MP-RAGE Manual 10

Background Dataset Method Results Conclusion

slide-11
SLIDE 11

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

When there is a plaque

  • Calcification: calcium builds up in blood vessels
  • Lipid-rich/necrotic core (LR/NC): extracellular mass in the intima
  • Hemorrhage: liquid plaque component
  • Loose matrix: tissues that are loosely woven

T1W T2W TOF MP-RAGE Manual 11

Background Dataset Method Results Conclusion

slide-12
SLIDE 12
  • MEPPS

n Morphology-enhanced probability map n Intensity + morphology information

  • Van et al.

n Bayes classifier n Intensity + zero-, first and second derivatives

  • Using deep learning, we can improve the performance

up to 2x compared to MEPPS

  • Do not need ad hoc features

Previous work requires hand-crafted features, yet not achieving usable accuracy

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

12

Background Dataset Method Results Conclusion

slide-13
SLIDE 13
  • Background
  • Dataset and preprocessing
  • Our model
  • Evaluation

Outline

13

Background Dataset Method Results Conclusion

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

slide-14
SLIDE 14
  • Collected 13 medical centers and hospitals all over China
  • Over 1000 patients, we used ~580, age between 18 and 80
  • All patients have stroke or transient ischemic attack within two weeks after
  • nsets of symptoms
  • Professionally labeled to identify all plaques.

Dataset: Chinese Atherosclerosis Risk Evaluation study (CARE II)

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

14

Background Dataset Method Results Conclusion

slide-15
SLIDE 15
  • Each case has 16 slices with 4 contrast weightings
  • Different slice thickness => requires an alignment

Dataset labeling: Alignment of different contrasts

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

T1 T2 TOF MP-RAGE 2mm 1mm

bifurcation

15

Background Dataset Method Results Conclusion

slide-16
SLIDE 16
  • Each case has 16 slices with 4 contrast weightings
  • Different slice thickness => requires an alignment

Dataset labeling: Alignment of different contrasts

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

bifurcation T1W T2W TOF MP-RAGE Manual 16

Background Dataset Method Results Conclusion

slide-17
SLIDE 17
  • Each case has 16 slices with 4 contrast weightings
  • Different slice thickness => requires an alignment

Dataset labeling: Alignment of different contrasts

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

T1 T2 TOF MP-RAGE 2mm 1mm

bifurcation

17

Background Dataset Method Results Conclusion

slide-18
SLIDE 18

Dataset Labeling: segment all the component => pixel level labeling

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

18

Background Dataset Method Results Conclusion

slide-19
SLIDE 19
  • Reviewers provide a 5-level quality score
  • We ignore the lowest quality ones

Dataset labeling: Image quality filtering

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

1 2 3 4 5 high low 19

Background Dataset Method Results Conclusion

slide-20
SLIDE 20
  • We choose 1098 vessels (16 slices each), from ~580 people
  • 20% test set
  • 80% training + validation

Our training / testing set selection

20

Background Dataset Method Results Conclusion

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

slide-21
SLIDE 21
  • Background
  • Dataset and preprocessing
  • Our model
  • Evaluation

Outline

21

Background Dataset Method Results Conclusion

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

slide-22
SLIDE 22
  • We use convolutional neural networks (CNN) to learn the input

n Base models: VGG-16[1], GoogLeNet[2], ResNet-101[3]

  • Key questions:

n Still not enough training data.

n

Natural image datasets, e.g. ImageNet, 1.26 million

n Does ImageNet pre-trained models help? n How to adapt the multi-contrast images to a pre-trained model? n Plaques is very small in the image, pretrained CNN does not offer not enough resolution

Our Approach

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

[1] Simonyan K, Zisserman A. Very Deep Convolutional Networks for Large-Scale Image Recognition[J]. Computer Science, 2014. [2] Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions[J]. 2015:1-9. [3] He K, Zhang X, Ren S, et al. Deep Residual Learning for Image Recognition[J]. 2015:770-778.

22

Background Dataset Method Results Conclusion

slide-23
SLIDE 23
  • We fine-tune base models pre-trained with ImageNet
  • Allow inputting the 4 contrast weightings with reasonable overhead
  • Maintaining high resolution by reducing the down-sampling factor from 32x

to 8x

Our Key Ideas

23

Background Dataset Method Results Conclusion

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

slide-24
SLIDE 24
  • Low-level features of pretrained model contains texture information

n Similar for natural images vs. medical ??

  • We can re-use them through fine-tuning

Key idea 1: Fine-tuning a Pre-trained Model

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

24

Background Dataset Method Results Conclusion

Figure from: Zeiler M D, Fergus R. Visualizing and understanding convolutional networks.

slide-25
SLIDE 25
  • Input: RGB Image (3 input channels) -> Multi-contrast MR Images (4 input channels)

Key idea 2: Adapting multi-contrast images into the pre- trained network (VGG, GoogLeNet, ResNet…)

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

data conv1 pool1 RGB Image data1 conv1_1 conv1_SUM T1W/3 data2 conv1_2 T2W/3 data3 conv1_3 TOF/3 data4 conv1_4 MP-Rage/3 pool1 … … (a). The original structure (b). Modified Structure 25

Background Dataset Method Results Conclusion

slide-26
SLIDE 26
  • Pretrained model has 32x reduction on input images
  • Our input size is 320x320
  • Plaque composition may be less than 32x32, => less than 1 pixel
  • Thus: 8x reduction
  • Modify two strides of 2 to 1, and add dilation kernels[1]

Key Idea 3: Maintaining high resolution

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

[1] Chen L C, Papandreou G, Kokkinos I, et al. Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs[J]. Computer Science, 2014(4):357-361.

Less than 32x32 26

Background Dataset Method Results Conclusion

slide-27
SLIDE 27
  • Imbalance Data

n Many slices only have normal tissues n Features of normal tissues can also be learned from other slices, while learning abnormal classes

simultaneously

n Thus we throw normal slices away

  • Data augmentation

n Since there isn’t any known impact of the position (left or right) of the carotid on plaques n We flip the image horizontally with ½ probability n With an input of 320*320, we randomly rescale the image to 1x~1.25x, and randomly crop 320*320

  • n the rescaled image and put into the net

Implementation Details

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

27

Background Dataset Method Results Conclusion

slide-28
SLIDE 28
  • Background
  • Dataset and preprocessing
  • Our model
  • Evaluation

Outline

28

Background Dataset Method Results Conclusion

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

slide-29
SLIDE 29

Results

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

Our Result 29

Background Dataset Method Results Conclusion

slide-30
SLIDE 30

Results

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

Our Result T1W T2W TOF MP-RAGE Manual ResNet-101 30

Background Dataset Method Results Conclusion

slide-31
SLIDE 31
  • Pixel-wise accuracy

n Recall, precision and f-score

  • Pixel-wise accuracy is strict

n Recall: 0.967; Precision: 0.789; F-score: 0.869

Results: Metrics

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

T1W T2W TOF MP-RAGE Manual ResNet-101 31

Background Dataset Method Results Conclusion

slide-32
SLIDE 32
  • Metric

n Precision n Recall n F-measure

Results

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

MEPPS GoogLeNet VGG-16 ResNet-101 Calcification 0.698/0.457 0.673/0.446 0.663/0.481 0.704/0.492 Lipid Core 0.373/0.273 0.533/0.419 0.536/0.372 0.576/0.474 Hemorrhage 0.526/0.299 0.710/0.499 0.717/0.487 0.729/0.622 Loose Matrix 0.103/0.253 0.422/0.091 0.522/0.138 0.488/0.246

Precision/Recall

MEPPS GoogLeNet VGG-16 ResNet-101 Calcification 0.552 0.536 0.557 0.580 Lipid Core 0.315 0.469 0.439 0.520 Hemorrhage 0.382 0.586 0.580 0.671 Loose Matrix 0.146 0.150 0.218 0.327

F-measure

32

Background Dataset Method Results Conclusion

slide-33
SLIDE 33
  • Metric

n Precision n Recall n F-measure

Results: Comparing to MEPPS

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

MEPPS GoogLeNet VGG-16 ResNet-101 Calcification 0.698/0.457 0.673/0.446 0.663/0.481 0.704/0.492 Lipid Core 0.373/0.273 0.533/0.419 0.536/0.372 0.576/0.474 Hemorrhage 0.526/0.299 0.710/0.499 0.717/0.487 0.729/0.622 Loose Matrix 0.103/0.253 0.422/0.091 0.522/0.138 0.488/0.246

Precision/Recall

MEPPS GoogLeNet VGG-16 ResNet-101 Calcification 0.552 0.536 0.557 0.580 Lipid Core 0.315 0.469 0.439 0.520 Hemorrhage 0.382 0.586 0.580 0.671 Loose Matrix 0.146 0.150 0.218 0.327

F-measure

33

Background Dataset Method Results Conclusion

slide-34
SLIDE 34
  • Metric

n Precision n Recall n F-measure

Results: Different CNNs

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

MEPPS GoogLeNet VGG-16 ResNet-101 Calcification 0.698/0.457 0.673/0.446 0.663/0.481 0.704/0.492 Lipid Core 0.373/0.273 0.533/0.419 0.536/0.372 0.576/0.474 Hemorrhage 0.526/0.299 0.710/0.499 0.717/0.487 0.729/0.622 Loose Matrix 0.103/0.253 0.422/0.091 0.522/0.138 0.488/0.246

Precision/Recall

MEPPS GoogLeNet VGG-16 ResNet-101 Calcification 0.552 0.536 0.557 0.580 Lipid Core 0.315 0.469 0.439 0.520 Hemorrhage 0.382 0.586 0.580 0.671 Loose Matrix 0.146 0.150 0.218 0.327

F-measure

34

Background Dataset Method Results Conclusion

slide-35
SLIDE 35
  • Metric

n Precision n Recall n F-measure

Results: Different accuracy on different compositions

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

MEPPS GoogLeNet VGG-16 ResNet-101 Calcification 0.698/0.457 0.673/0.446 0.663/0.481 0.704/0.492 Lipid Core 0.373/0.273 0.533/0.419 0.536/0.372 0.576/0.474 Hemorrhage 0.526/0.299 0.710/0.499 0.717/0.487 0.729/0.622 Loose Matrix 0.103/0.253 0.422/0.091 0.522/0.138 0.488/0.246

Precision/Recall

MEPPS GoogLeNet VGG-16 ResNet-101 Calcification 0.552 0.536 0.557 0.580 Lipid Core 0.315 0.469 0.439 0.520 Hemorrhage 0.382 0.586 0.580 0.671 Loose Matrix 0.146 0.150 0.218 0.327

F-measure

35

Background Dataset Method Results Conclusion

slide-36
SLIDE 36

Results: False Positive

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

Our Result T1W T2W TOF MP-RAGE Manual ResNet-101 Loose matrix Calcification 36

Background Dataset Method Results Conclusion

slide-37
SLIDE 37

Contrast Weighting Calcification Lipid/Necrotic Core Hemorrhage Loose Matrix T1W 0.538 0.496 0.443 0.020 T2W 0.494 0.515 0.323 0.387 TOF 0.468 0.465 0.487 0.080 MP-RAGE 0.337 0.437 0.681 0.015 ALL 0.580 0.520 0.671 0.327

Contributions of Each Contrast Weighting

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

  • Use each contrast weighting to train separate models
  • F-measure of each tissue class

37

Background Dataset Method Results Conclusion

slide-38
SLIDE 38

Model Ensemble

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

38

Background Dataset Method Results Conclusion

  • Average: average over the softmax layer of four models
  • Learning: learn the weights of feature maps of upscore layer

Model Fibrous Tissue Calcification Lipid/Necrotic Core Hemorrhage Loose Matrix Average 0.963 0.518 0.522 0.608 0.009 Learning 0.963 0.585 0.557 0.691 0.335 ResNet-101 0.962 0.580 0.520 0.671 0.327 softmax 8xupscore … Learning Average

slide-39
SLIDE 39

Model Ensemble: Learning

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

39

Background Dataset Method Results Conclusion

data1 conv1_1 conv1_SUM T1W/3 data2 conv1_2 T2W/3 data3 conv1_3 TOF/3 data4 conv1_4 MP-Rage/3 pool1 … (a). Aggregate early data1 conv1_1 T1W/3 data2 conv1_2 T2W/3 data3 conv1_3 TOF/3 data4 conv1_4 MP-Rage/3 (b). Aggregate later … … … …

8xupscore_1 8xupscore_2 8xupscore_3 8xupscore_4

8xupscore prediction

slide-40
SLIDE 40

Model Ensemble: weights of each feature map

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

40

Background Dataset Method Results Conclusion

  • 4 models trained with 4 channels separately
  • 5 score maps for each contrast weighting in each model
  • 20 feature maps

Fibrous Tissue Calcification Lipid/Necrotic Core Hemorrhage Loose Matrix T1W T2W TOF MP-Rage

slide-41
SLIDE 41

Model Ensemble: weights of each feature map

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

41

Background Dataset Method Results Conclusion

Fibrous Tissue Calcification Lipid/Necrotic Core Hemorrhage Loose Matrix T1W T2W TOF MP-Rage Contrast Weighting Calcification Lipid/Necrotic Core Hemorrhage Loose Matrix T1W 0.538 0.496 0.443 0.020 T2W 0.494 0.515 0.323 0.387 TOF 0.468 0.465 0.487 0.080 MP-RAGE 0.337 0.437 0.681 0.015 ALL 0.580 0.520 0.671 0.327

slide-42
SLIDE 42

Model Ensemble: weights of each feature map

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

42

Background Dataset Method Results Conclusion

Fibrous Tissue Calcification Lipid/Necrotic Core Hemorrhage Loose Matrix T1W T2W TOF MP-Rage Contrast Weighting Calcification Lipid/Necrotic Core Hemorrhage Loose Matrix T1W 0.538 0.496 0.443 0.020 T2W 0.494 0.515 0.323 0.387 TOF 0.468 0.465 0.487 0.080 MP-RAGE 0.337 0.437 0.681 0.015 ALL 0.580 0.520 0.671 0.327

slide-43
SLIDE 43

Confusion Matrix

Calcification 202 14 139 3085 0.592 0.995 Lipid Core 306 69 379 2686 0.447 0.975 Hemorrhage 81 98 19 3242 0.810 0.971 Loose Matrix 72 634 85 2649 0.459 0.807 Calcification 270 42 71 3057 0.792 0.986 Lipid Core 567 169 118 2586 0.827 0.939 Hemorrhage 84 34 16 3306 0.840 0.990 Loose Matrix 83 91 74 3192 0.529 0.972

MEPPS ResNet

TP FP FN TN sensitivity specificity

43

Background Dataset Method Results Conclusion

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

slide-44
SLIDE 44
  • Takes ~11s on a Titan X GPU
  • For whole-slice (16 slices) prediction

Results: Running time

MEPPS GoogLeNet VGG-16 ResNet-101 Time (sec) 10.0 9.1 8.9 11.4

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

44

Background Dataset Method Results Conclusion

slide-45
SLIDE 45
  • We apply CNNs to automatically recognize carotid plaque components
  • Modify the network to receive multi-contrast input
  • Lower the down sampling ratio to maintain high resolution
  • CNNs achieve better accuracy than traditional Bayesian methods while

running in acceptable time

Conclusion

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

45

Background Dataset Method Results Conclusion

slide-46
SLIDE 46
  • CNNs can replace many traditional methods in medical image processing
  • Key challenge: la

labele led data

  • esp. high quality label for CNN training ≠ medical report
  • Future direction: reducing the labeling requirement, transfer? Active learning?

Final Remarks

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

46

Background Dataset Method Results Conclusion

T1W T2W TOF MP-RAGE Manual ResNet-101

slide-47
SLIDE 47

Q&A

Identifying Carotid Plaque Composition in MRI with Convolutional Neural Networks

Background Motivation Dataset Method Results Conclusion

47