Neuroimage Analysis for Automated Brain Disease Diagnosis Mingxia - - PowerPoint PPT Presentation

▶
neuroimage analysis for automated brain disease diagnosis
SMART_READER_LITE
LIVE PREVIEW

Neuroimage Analysis for Automated Brain Disease Diagnosis Mingxia - - PowerPoint PPT Presentation

Oct 14, 2023 125 likes •697 views

Neuroimage Analysis for Automated Brain Disease Diagnosis Mingxia Liu University of North Carolina at Chapel Hill mxliu@med.unc.edu http://mingxia.web.unc.edu/ 07-17-2019 Background Healthy Brain vs. Alzheimers Sulcus Sulcus Gyrus

slide-1
SLIDE 1

Neuroimage Analysis for Automated Brain Disease Diagnosis

University of North Carolina at Chapel Hill mxliu@med.unc.edu http://mingxia.web.unc.edu/

07-17-2019

Mingxia Liu

slide-2
SLIDE 2

Healthy Brain Alzheimer’s

Ventricle Sulcus Gyrus Sulcus Gyrus

Healthy Brain Alzheimer’s

Healthy Brain vs. Alzheimer’s

Background

slide-3
SLIDE 3

Background

  • Alzheimer’s Disease (AD)

– A progressive disease

Normal Control Mild Cognitive Impairment (MCI) Alzheimer’s Disease

Brain Health Time

Stable MCI (sMCI) Progressive MCI (pMCI)

  • Calling Need

– Developing computer-aided methods for MCI/AD diagnosis

slide-4
SLIDE 4
  • Structural Magnetic Resonance Imaging (MRI)
  • FDG-Positron Emission Tomography (PET)
  • Cerebrospinal Fluid (CSF) ‐‐‐ Aβ42, t‐tau and p‐tau

Biomarkers for early diagnosis of AD and MCI

MRI PET CSF CSF Brain Dura

Multi-modal data

Background

slide-5
SLIDE 5
  • M. Liu, etc., Landmark-based Deep Multi-Instance Learning for Brain Disease Diagnosis, Medical Image Analysis, 2018.
  • M. Liu, etc. Joint Classification and Regression via Deep Multi-task Multi-channel Learning for Alzheimer’s Disease Diagnosis. IEEE
  • Trans. on Biomedical Engineering, 2018.

fMRI PET sMRI

Neuroimaging Data

Brain Disease Diagnosis – Typical Pipeline

Image Preprocessing Feature Extraction/Selection Classifier Learning Machine Learning & Deep Learning

slide-6
SLIDE 6

Challenges in Computer-aided Disease Diagnosis

  • Effective feature representation of neuroimages
  • Missing multi-modal data
  • Heterogeneous data at different imaging sites
slide-7
SLIDE 7

Outline

  • Missing Data
  • Multi-modal Data Fusion
  • Domain Adaptation

Multi-modal Neuroimage

CSF PET sMRI

Single-modal Neuroimage

  • Structural MRI (sMRI)

sMRI

slide-8
SLIDE 8

Part I. Single-modal Neuroimage Analysis

  • Structural MRI based Brain Disease Diagnosis
slide-9
SLIDE 9

Anatomical Landmarks for Structural MRI

  • Landmark-based Deep Representation of sMRI
  • C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
  • M. Liu, J. Zhang, C. Lian, and D. Shen. IEEE Trans. on Cybernetics, 2019.
  • M. Liu, J. Zhang, E. Adeli, and D. Shen. IEEE Trans. on Biomedical Engineering, 2019.
  • M. Liu, J. Zhang, E. Adeli, and D. Shen. Medical Image Analysis, 2018.
  • M. Liu, J. Zhang, D. Nie, P.T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.
  • J. Zhang, M. Liu, and D. Shen. IEEE Trans. on Image Processing, 2017.
slide-10
SLIDE 10

Training Data

...

Patch Extraction

ata Training Data

...

Landmark Discovery

ata

Test MRI

ata

Landmark Detection

Training Data

...

Training MRIs

Pre-processed MR Images …

… CNN

Deep Multi-channel Convolutional Neural Network (CNN) Disease Classification Clinical Score Prediction * Featured Article of IEEE Journal of Biomedical and Health Informatics, 2018

Anatomical Landmark-based Deep Network

slide-11
SLIDE 11

0.01 p-values

1,740 landmarks via group comparison between AD and NC Top 50 landmarks

Anatomical Landmark-based Deep Network

  • M. Liu, J. Zhang, E. Adeli, and D. Shen. Medical Image Analysis, 2018.
  • M. Liu, J. Zhang, D. Nie, P.T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.
slide-12
SLIDE 12

Concatenated

sMRI

FC8-32 FC8-32 FC8-32

…

FC9-32L Conv1- 32@3×3×3 Conv2- 32@3×3×3 Max-pooling Conv3- 64@2×2×2 Conv4- 64@2×2×2 Max-pooling Conv5- 128@2×2×2 Conv6- 128@2×2×2 Max-pooling Conv1- 32@3×3×3 Conv2- 32@3×3×3 Max-pooling Conv3- 64@2×2×2 Conv4- 64@2×2×2 Max-pooling Conv5- 128@2×2×2 Conv6- 128@2×2×2 Max-pooling FC7-128

…

FC7-128 Conv1- 32@3×3×3 Conv2- 32@3×3×3 Max-pooling Conv3- 64@2×2×2 Conv4- 64@2×2×2 Max-pooling Conv5- 128@2×2×2 Conv6- 128@2×2×2 Max-pooling

Patch 1 Patch 2

…

Patch L

Landmark-based Deep Network

FC7-128 FC10-8L FC11-2

Class Label

Soft-max

Global Image-level Representation Local Patch-level Representation

Global Representation?

  • M. Liu, J. Zhang, E. Adeli, and D. Shen. Medical Image Analysis, 2018.
  • M. Liu, J. Zhang, D. Nie, P.T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.
slide-13
SLIDE 13

0.431

Overall Accuracy

0.404 0.46 7 0.486 0.487

0.518

0.325

Correlation Coefficient

0.289 0.468 0.492 0.538

0.567

Classification Results for AD vs. sMCI vs. pMCI vs. NC Regression Results for MMSE

Results of Classification and Regression

  • M. Liu, J. Zhang, E. Adeli, and D. Shen. Medical Image Analysis, 2018.
  • M. Liu, J. Zhang, D. Nie, P.T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.
slide-14
SLIDE 14

End-to-end Disease Diagnosis with sMRI

  • Hierarchical Fully Convolutional network
  • C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
  • M. Liu, J. Zhang, C. Lian, and D. Shen. IEEE Trans. on Cybernetics, 2019.
  • M. Liu, J. Zhang, E. Adeli, and D. Shen. IEEE Trans. on Biomedical Engineering, 2019.
  • M. Liu, J. Zhang, E. Adeli, and D. Shen. Medical Image Analysis, 2018.
  • M. Liu, J. Zhang, D. Nie, P.T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.
  • J. Zhang, M. Liu, and D. Shen. IEEE Trans. on Image Processing, 2017.
slide-15
SLIDE 15

15

Hierarchical Network for ROI Identification

  • Hierarchical Fully Convolutional Network (H-FCN)

– Automatically and identify disease-related ROIs in the whole sMR image – Jointly learn multi-scale features and construct a classification model

  • C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
  • C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.
slide-16
SLIDE 16

Hierarchical Network for ROI Identification

Input: sMRI

  • C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
  • C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.
slide-17
SLIDE 17

Input: sMRI

1) Location proposals

Hierarchical Network for ROI Identification

  • C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
  • C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.
slide-18
SLIDE 18

1) Location proposals

Input: sMRI

2) Patch-level sub- networks (PSN) (shared weights)

Hierarchical Network for ROI Identification

PSN PSN

Class_P 64 64 64 128 128 32

  • C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
  • C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.
slide-19
SLIDE 19

PSN

1) Location proposals

… …

PSN PSN PSN PSN PSN PSN PSN Input: sMRI

2) Patch-level sub- networks (PSN) (shared weights)

Hierarchical Network for ROI Identification

PSN

Class_P 64 64 64 128 128 32

  • C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
  • C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.
slide-20
SLIDE 20

…

64 Conv_R Class_R

…

PSN PSN PSN PSN PSN PSN PSN PSN

2) Patch-level sub- networks (PSN) (shared weights) 3) Region-level sub-networks 1) Location proposals

Input: sMRI

Hierarchical Network for ROI Identification

PSN

Class_P 64 64 64 128 128 32

  • C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
  • C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.
slide-21
SLIDE 21

…

64 Conv_R Class_R

…

64 Conv_R Class_R

…

PSN PSN PSN PSN PSN PSN PSN PSN

2) Patch-level sub- networks (PSN) (shared weights) 3) Region-level sub-networks 1) Location proposals

Input: sMRI

Hierarchical Network for ROI Identification

PSN

Class_P 64 64 64 128 128 32

  • C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
  • C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.
slide-22
SLIDE 22

…

64 Conv_S Class_S 64 Conv_R Class_R 64 Conv_R Class_R

… …

Output: Class label

PSN PSN PSN PSN PSN PSN PSN PSN

3) Region-level sub-networks 4) Subject-level sub-network 1) Location proposals

Input: sMRI

2) Patch-level sub- networks (PSN) (shared weights)

Classification (1 × 1 × 1 Conv) 4 × 4 × 4 Conv 1 × 1 × 1 Conv Channel concatenation Region-level Conv 2 × 2 × 2 max pooling 3 × 3 × 3 Conv Spatial concatenation Subject-level Conv Potentially pruned sub-networks Skipped connection Conv: Convolution PSN

Class_P 64 64 64 128 128 32

Hierarchical Network for ROI Identification

  • C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
  • C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.

Network Pruning: To remove less informative regions

slide-23
SLIDE 23

Identified Voxel-level Discriminative Locations

1 2 3 1 2 3 1 2 3 1 2 3 4 3 4 3 4 1 2 1 2 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 1 2 1 2 1 2 3 1 2 3 1 2 3 1 2 3

(a) AD Subject #1 (b) AD Subject #2 (c) AD Subject #3 (d) AD Subject #4 (e) AD Subject #5 (f) AD Subject #6

  • C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
  • C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.
slide-24
SLIDE 24

Identified Patch-level Discriminative Locations

Sagittal View Axial View Coronal View 3D View

  • C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
  • C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.
slide-25
SLIDE 25

Identified Region-level Discriminative Locations

Sagittal View Axial View Coronal View 3D View

  • C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
  • C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.
slide-26
SLIDE 26

0.5 0.6 0.7 0.8 0.9 1

Patch Region Subject ACC

0.5 0.6 0.7 0.8 0.9 1

Patch Region Subject AUC

H-FCN before network pruning H-FCN after network pruning Results of AD vs. NC classification obtained by patch-, region-, and subject-level sub- networks in H-FCN without /with network pruning

Classification Results

  • C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
  • C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.

1) Global subject-level representation is more useful 2) Network pruning promotes the classification performance

slide-27
SLIDE 27

Outline

  • Missing Data
  • Multi-modal Data Fusion
  • Domain Adaptation

Multi-modal Neuroimage

CSF PET sMRI

Single-modal Neuroimage

  • Structural MRI (sMRI)

sMRI

slide-28
SLIDE 28

Part II. Multi-modal Neuroimage Analysis

  • Multi-modality Fusion for Disease Diagnosis
  • Imaging Synthesis for Missing Modalities
  • Domain Adaptation for Multi-site Data
slide-29
SLIDE 29
  • M. Liu, J. Zhang, P.-T. Yap, and D. Shen. MICCAI, 2016.
  • M. Liu, J. Zhang, P.-T. Yap, and D. Shen. Medical Image Analysis, 2017.
  • M. Liu, Y. Gao, P.-T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.

Multi-modality Fusion for Disease Diagnosis

  • Hypergraph Learning with Missing Data
slide-30
SLIDE 30

30

Multi-View Data Grouping

  • Given 3 modalities (i.e., MRI, PET and CSF), 6 views are constructed

according to the availability of different modalities

Multi-Modality Data CSF PET MRI

PET CSF MRI Missing Data

* MICCAI Young Scientist Award Nomination, 2016 * MICCAI Travel Award, 2016

slide-31
SLIDE 31

31

Multi-View Data Grouping

  • Given 3 modalities (i.e., MRI, PET and CSF), 6 views are constructed

according to the availability of different modalities

Multi-Modality Data CSF PET MRI Multi-View Data Grouping

View 1 View 2 View 3 View 4 View 5 View 6 PET CSF MRI Missing Data

* MICCAI Young Scientist Award Nomination, 2016 * MICCAI Travel Award, 2016

slide-32
SLIDE 32
  • Construct multiple hypergraphs, with each hypergraph corresponding to

a specific view

Multi-Modality Data CSF PET MRI Multi-View Data Grouping

View 1 View 2 View 3 View 4 View 5 View 6 Sparse Representation Sparse Representation

Sparse Representation based Hypergraph Construction

Sparse Representation Sparse Representation Sparse Representation Sparse Representation PET CSF MRI Missing Data

  • M. Liu, J. Zhang, P.-T. Yap, and D. Shen. MICCAI, 2016.
  • M. Liu, J. Zhang, P.-T. Yap, and D. Shen. Medical Image Analysis, 2017.
  • M. Liu, Y. Gao, P.-T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.

Hypergraph Construction

slide-33
SLIDE 33
  • A view-aligned hypergraph classification model to capture the coherence

among different views

View-Aligned Hypergraph Classification

View-Aligned Hypergraph Classification

Multi-Modality Data CSF PET MRI Multi-View Data Grouping

View 1 View 2 View 3 View 4 View 5 View 6 Sparse Representation Sparse Representation

Sparse Representation based Hypergraph Construction

Sparse Representation Sparse Representation Sparse Representation Sparse Representation PET CSF MRI Missing Data

  • M. Liu, J. Zhang, P.-T. Yap, and D. Shen. MICCAI, 2016.
  • M. Liu, J. Zhang, P.-T. Yap, and D. Shen. Medical Image Analysis, 2017.
  • M. Liu, Y. Gao, P.-T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.
slide-34
SLIDE 34
  • Multi-view label fusion strategy (weighted voting)

Multi-View Label Fusion

Multi-Modality Data CSF PET MRI Multi-View Data Grouping

View 1 View 2 View 3 View 4 View 5 View 6 Sparse Representation Sparse Representation

Sparse Representation based Hypergraph Construction

Sparse Representation Sparse Representation Sparse Representation Sparse Representation

View-Aligned Hypergraph Classification

PET CSF MRI Missing Data

  • M. Liu, J. Zhang, P.-T. Yap, and D. Shen. MICCAI, 2016.
  • M. Liu, J. Zhang, P.-T. Yap, and D. Shen. Medical Image Analysis, 2017.
  • M. Liu, Y. Gao, P.-T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.

Multi-view Label Fusion

slide-35
SLIDE 35

Label Space

𝐲𝑜1

MRI

𝐲𝑜1

PET

𝐲𝑜1

CSF

𝑔

𝑜1 MRI

𝑔

𝑜1 PET

𝑔

𝑜1 CSF

𝑔

𝑜2 MRI

𝑔

𝑜2 PET

𝐲𝑜2

MRI

𝐲𝑜2

PET

View-Aligned Constraint

Coherence among views View-Aligned Regularizer

  • M. Liu, J. Zhang, P.-T. Yap, and D. Shen. MICCAI, 2016.
  • M. Liu, J. Zhang, P.-T. Yap, and D. Shen. Medical Image Analysis, 2017.
  • M. Liu, Y. Gao, P.-T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.

Missing CSF MRI PET Subject #2 CSF MRI PET Subject #1

slide-36
SLIDE 36

36

𝑛𝑗𝑜

𝐆, 𝛃, {𝐗𝑛}𝑛=1

𝑁

𝑛=1

𝑁

𝛁𝑛 (𝐠𝑛 − 𝐳) 2

2

+𝜇 𝑛=1

𝑁

𝐗𝑛

𝐺 2

+𝜈 𝑜=1

𝑂

𝑛=1

𝑁

𝑞=1

𝑁

𝛻𝑜,𝑜

𝑛 𝛻𝑜,𝑜 𝑞

𝑔

𝑜 𝑛 − 𝑔 𝑜 𝑞 2

+ 𝑛=1

𝑁

𝛽𝑛 2 𝐠𝑛 T 𝐌𝑛𝐠𝑛 𝑡. 𝑢. 𝑛=1

𝑁

𝛽𝑛 = 1, ∀𝛽𝑛 ≥ 0; 𝑗=1

𝑂𝑓

𝑛

𝑋

𝑗,𝑗 𝑛 = 1, ∀𝑋 𝑗,𝑗 𝑛 ≥ 0.

Step 3: View-Aligned Hypergraph Classification

Formulation of view-aligned hypergraph classification (VAHC)

View-aligned Regularizer Hypergraph Laplacian matrix , and .

  • M. Liu, J. Zhang, P.-T. Yap, and D. Shen. MICCAI, 2016.
  • M. Liu, J. Zhang, P.-T. Yap, and D. Shen. Medical Image Analysis, 2017.
  • M. Liu, Y. Gao, P.-T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.
slide-37
SLIDE 37

Learned Weights for Views

Learned weights for views in four classification tasks

AD vs. NC MCI vs. NC pMCI vs. sMCI pMCI vs. NC 0.0 0.1 0.2 0.3 0.4 0.5 0.6

Weights

MRI PET CSF PET+MRI MRI+CSF PET+MRI+CSF

  • M. Liu, J. Zhang, P.-T. Yap, and D. Shen. MICCAI, 2016.
  • M. Liu, J. Zhang, P.-T. Yap, and D. Shen. Medical Image Analysis, 2017.
  • M. Liu, Y. Gao, P.-T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.

Using all 3 modalities (MRI+PET+CSF) achieves the best results

slide-38
SLIDE 38

Experimental Results

pMCI vs. sMCI classification

ACC SEN SPE BAC PPV NPV AUC 50 60 70 80 90 100

Results (%)

(c) pMCI vs. sMCI classification

Zero KNN EM SVD VAHL

  • M. Liu, J. Zhang, P.-T. Yap, and D. Shen. MICCAI, 2016.
  • M. Liu, J. Zhang, P.-T. Yap, and D. Shen. Medical Image Analysis, 2017.
  • M. Liu, Y. Gao, P.-T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.
slide-39
SLIDE 39

Imaging Synthesis for Missing Modalities

  • Deep Learning based Automatic PET Synthesis from sMRI
  • Y. Pan, M. Liu*, C. Lian, Y. Xia, and D. Shen. MICCAI, 2019.
  • Y. Pan, M. Liu, C. Lian, T. Zhou, Y. Xia, and D. Shen. MICCAI, 2018.
  • Y. Pan, M. Liu*, C. Lian, L. Yue, S. Xiao, Y. Xia and D. Shen. ISMRM, 2019
slide-40
SLIDE 40

Pre-processed MR and PET Images Stage 2: Brain Disease Classification Stage 1: PET Image Synthesis based on sMRI Synthesizing Missing PET based on MRI for Brain Disease Diagnosis

Hybrid Cycle-GAN for Missing PET Synthesis

  • Y. Pan, M. Liu, C. Lian, T. Zhou, Y. Xia, and D. Shen. MICCAI, 2018
  • Y. Pan, M. Liu*, C. Lian, L. Yue, S. Xiao, Y. Xia and D. Shen. ISMRM, 2019
  • Among 800+ subjects in ADNI, all subjects have sMRI, while
  • nly half of them have FDG-PET scans.

MRI PET

slide-41
SLIDE 41

Hybrid cycle-consistent generative adversarial network (HGAN) to impute missing PET scans based on sMRI

Synthetic PET Synthetic MRI

32 32

Residual Net Block (RNB)

Real PET

1 128 64 32 16

𝑬𝑸

1 128

64

32 16

𝑬𝑵

1 16 32 32 16 8

𝑯𝑵

⋯

RNB RNB

8 16 32

𝑯𝑸

⋯

RNB RNB

32 16 1 3×3×3 Convolution 7×7×7 Convolution 3×3×3 Deconvolution 4×4×4 Convolution Addition

Real MRI

6 6 1/0 1/0

𝐘𝑁 𝐘𝑄 𝐻1 𝐘𝑁 𝐻2 𝐘𝑄

Hybrid Cycle-GAN for Missing PET Synthesis

  • Y. Pan, M. Liu, C. Lian, T. Zhou, Y. Xia, and D. Shen. MICCAI, 2018
  • Y. Pan, M. Liu*, C. Lian, L. Yue, S. Xiao, Y. Xia and D. Shen. ISMRM, 2019
slide-42
SLIDE 42

Synthetic PET Images

PET (RID: 5016)

  • Y. Pan, M. Liu, C. Lian, T. Zhou, Y. Xia, and D. Shen. MICCAI, 2018
  • Y. Pan, M. Liu*, C. Lian, L. Yue, S. Xiao, Y. Xia and D. Shen. ISMRM, 2019

(a) GAN (b) CGAN (c) VGAN (d) HGAN (Ours) (e) Ground Truth

PET (RID: 4352)

slide-43
SLIDE 43

Synthetic MRI Scans

MRI (RID: 5016)

  • Y. Pan, M. Liu, C. Lian, T. Zhou, Y. Xia, and D. Shen. MICCAI, 2018
  • Y. Pan, M. Liu*, C. Lian, L. Yue, S. Xiao, Y. Xia and D. Shen. ISMRM, 2019

(a) GAN (b) CGAN (c) VGAN (d) HGAN (Ours) (e) Ground Truth

MRI (RID: 4352)

slide-44
SLIDE 44

PET (Real + Synthetic) MRI (Real)

24 24 24 24 24 24 24 24 24 24 24 24 … 32 32 32 64 64 64

Sub-network 1

DCM DCM DCM 16 16 16

Sub-network 2

32 32 32 64 64 64 DCM DCM DCM 16 16 16 Concatenation 32 8K …

Clinical scores at four time-points

8 32 8 128 32 BL M06 M24 M12

Fully-connected Down-sampling Copy 3×3×3 Convolution 2×2×2 Max-pooling Channel concatenation

Hybrid Cycle-GAN for Missing PET Synthesis

Landmark-based Deep Network for Brain Disease Classification using MRI and PET (Real+Synthetic)

  • Y. Pan, M. Liu, C. Lian, T. Zhou, Y. Xia, and D. Shen. MICCAI, 2018
  • Y. Pan, M. Liu*, C. Lian, L. Yue, S. Xiao, Y. Xia and D. Shen. ISMRM, 2019

PET (Real + Synthetic)

slide-45
SLIDE 45

MCI conversion prediction with complete MRI and complete (after imputation) PET

Hybrid Cycle-GAN for Missing PET Synthesis

  • Y. Pan, M. Liu, C. Lian, T. Zhou, Y. Xia, and D. Shen. MICCAI, 2018
  • Y. Pan, M. Liu*, C. Lian, L. Yue, S. Xiao, Y. Xia and D. Shen. ISMRM, 2019
slide-46
SLIDE 46

How to generating classification-oriented PET/MRI scans for diagnosis?

Feature-consistent GAN for Joint PET Synthesis and Classification

1 128 64 32 16

𝐸

3×3×3 Conv 7×7×7 Conv 3×3×3 Deconv 4×4×4 Conv Addition Difference

32 32 RNB 1 16 32 32 16 8

𝐻

RNB RNB ⋯ 6

𝐻𝑁 𝐻𝑄 𝐸𝑄 𝐸𝑁

𝔐𝑕 𝔐𝑕

𝐺

𝑄

𝐺

𝑁

𝔐c 𝔐c FG AN

Synthetic MRI Real PET Synthetic PET Real MRI Synthetic PET Real PET

1 2 K

1 2 K

⋮

1/0 𝑚2 64 64 32 16 64 64 64 32 16

𝔐c

64

Feature-consistent Component 𝐺

𝑄

Cosine Kernel

  • Y. Pan, M. Liu*, C. Lian, Y. Xia, and D. Shen. MICCAI, 2019

Feature-consistent Component 𝐺

𝑄

slide-47
SLIDE 47

30 50 70 90

AUC ACC SEN SEP ROI LMF LDMIL DSNN (Ours)

MCI conversion prediction with complete MRI and complete (after imputation) PET

Feature-consistent GAN for Joint PET Synthesis and Classification

  • Y. Pan, M. Liu*, C. Lian, Y. Xia, and D. Shen. MICCAI, 2019

Generating task-oriented PET scans helps boost performance

slide-48
SLIDE 48

Domain Adaptation for Multi-site Data

  • Low-rank Representation for Multi-site Data Adaptation
  • M. Wang, D. Zhang, J. Huang, D. Shen, and M. Liu*, and D. Shen. MICCAI, 2018
slide-49
SLIDE 49

Low-rank Representation for Domain Adaptation

ABIDE: 17 imaging sites with resting-state fMRI data

Scanners Scanning Parameters Population Noise Level Image Contrast

  • M. Wang, D. Zhang, J. Huang, D. Shen, and M. Liu*, and D. Shen. MICCAI, 2018

*Best Poster Award, MICS, 2019

slide-50
SLIDE 50

50

…

P2 P1 PS P New Representation

Latent Representation Space

Linear Representation

Site T Site S Site 1 Site 2

Low-rank Representation for Domain Adaptation

  • M. Wang, D. Zhang, J. Huang, D. Shen, and M. Liu*, and D. Shen. MICCAI, 2018

Mapping to a common latent space Representing source data using target data

Source Domains Target Domain

*Best Poster Award, MICS, 2019

slide-51
SLIDE 51

51

Low-rank Representation for Domain Adaptation

Formulation of multi-center low-rank representation (maLRR) min𝐊,𝐐,𝐐𝑗,𝐚𝑗,𝐅𝑇𝑗,𝐅𝑄𝑗,𝐆𝑗 𝐊 ∗ + 𝑗=1

𝐿

𝐆𝑗

∗ + 𝛽 𝐅𝑇𝑗 1 + 𝛾 𝐅𝑄𝑗 1

  • s. t. 𝐐𝑗𝐘𝑇𝑗 = 𝐐𝐘𝑈𝐚𝑗 + 𝐅𝑇𝑗,

𝐐𝑗= 𝐐 + 𝐅𝑄𝑗, 𝑗 = 1, … , 𝐿 𝐐 = 𝐊, 𝐚𝑗 = 𝐆𝑗, 𝐐𝐐𝑈 = 𝐉

1) Mapping to a latent space 2) Representing source data using target data

  • M. Wang, D. Zhang, J. Huang, D. Shen, and M. Liu*, and D. Shen. MICCAI, 2018

*Best Poster Award, MICS, 2019

slide-52
SLIDE 52

52

Low-rank Representation for Domain Adaptation

Performance of different methods in the task of Autism vs. NC classification, with NYU as the target domain and the other four sites as the source domains

ABIDE: 5 imaging sites with resting-state fMRI data

  • M. Wang, D. Zhang, J. Huang, D. Shen, and M. Liu*, and D. Shen. MICCAI, 2018

*Best Poster Award, MICS, 2019

slide-53
SLIDE 53

Outline

  • Missing Data
  • Multi-modal Data Fusion
  • Domain Adaptation

Multi-modal Neuroimage

CSF PET sMRI

Single-modal Neuroimage

  • Structural MRI (sMRI)

sMRI

slide-54
SLIDE 54

Acknowledge

  • Dr. Dinggang Shen
  • Dr. Daoqiang Zhang
  • Dr. Pew-Thian Yap
  • Dr. Jun Zhang
  • Dr. Chunfeng Lian
  • Dr. Ling Yue
  • Dr. Jing Zhang
  • Dr. Aimei Dong
  • Dr. Bo Wang

Collaborators

  • Dr. Ehsan Adeli
  • Dr. Yue Gao
  • Dr. Biao Jie
  • Dr. Tao Zhou
  • Dr. Yong Xia

Visiting Scholars and Students

  • Mr. Mingliang Wang
  • Mr. Yongsheng Pan
  • Mr. Dongren Yao
  • Mr. Jiashuang Huang
slide-55
SLIDE 55

Thanks for Your Attention!

mxliu@med.unc.edu http://mingxia.web.unc.edu/

Mingxia Liu