Breaking Speed Limits with Simultaneous Ultra-Fast MRI Reconstruction and Tissue Segmentation Paper #239 Francesco Calivà, Andrew P. Leynes, Rutwik Shah, Upasana U. Bharadwaj, Sharmila Majumdar, Peder E. Z. Larson, Valentina Pedoia
Disclosure I have no financial interests or relationships to disclose with regard to the subject matter of this presentation. Funding source This project was supported by R00AR070902 from the National Institute of Arthritis and Musculoskeletal and Skin Diseases, National Institutes of Health, (NIH-NIAMS).
Deep Learning in Magnetic Resonance Imaging (MRI) Long scanning time is the main limitation of MRI • We devised a DL framework for MRI reconstruction and segmentation from highly undersampled MRIs • We bridged image reconstruction and analysis by proposing a task-based reconstruction approach •
Hypotheses: Reconstruction in the image-domain provides higher • data interpretability over k-space IFT CNN Reconstruction and segmentation are similar and • Image-Domain Learning related tasks Simultaneous segmentation and image reconstruction
Proposed approach: Task-based image reconstruction: TB-recon Shared Encoding Path f=16 f=32 344 344 f=64 160 344 172 80 160 172 344 86 40 86 Reconstruction Decoding path f=32 f=64 160 80 344344 172 172 344 f=16 Shared Features 160 344 Embedding f = 6 4 f=32 f=64 Segmentation Decoding path 86 40 80 160 86 172172 344 344 f=16 Residual skip Strided 3D Conv (2x2x2) Intra-task skip Out Ch = 2x In Ch Inter-task skip Strided 3D Conv (2x2x2) Concatenate Out Ch = In Ch/2 344 3D Conv (5x5x5) 3D Conv (5x5x5) Out Ch = 16x In Ch Out Ch = In Ch/2 160 344 3D Conv (5x5x5) 3D Conv (1x1x1) Out Ch = In Ch Out Ch = Num Classes
Shared Encoding Path Shared Encoding Path Input: f=16 f=16 f=32 f=32 zero-filled MRI 344 344 344 344 f=64 f=64 160 160 344 344 172 172 80 80 160 160 172 172 344 344 86 86 40 40 86 86 Reconstruction Decoding path Reconstruction Decoding path f=32 f=32 f=64 f=64 160 160 80 80 344344 344344 172 172 172 172 344 344 f=16 f=16 Shared Features Shared Features 160 160 344 344 Embedding Embedding f f = = 6 6 4 4 f=32 f=32 f=64 f=64 Segmentation Decoding path Segmentation Decoding path 86 86 40 40 80 80 160 160 86 86 172172 172172 344 344 344 344 f=16 f=16 Residual skip Residual skip Strided 3D Conv (2x2x2) Strided 3D Conv (2x2x2) Intra-task skip Intra-task skip Out Ch = 2x In Ch Out Ch = 2x In Ch Inter-task skip Inter-task skip Strided 3D Conv (2x2x2) Strided 3D Conv (2x2x2) Concatenate Concatenate Out Ch = In Ch/2 Out Ch = In Ch/2 344 344 3D Conv (5x5x5) 3D Conv (5x5x5) 3D Conv (5x5x5) 3D Conv (5x5x5) Out Ch = 16x In Ch Out Ch = 16x In Ch Out Ch = In Ch/2 Out Ch = In Ch/2 160 160 344 344 3D Conv (5x5x5) 3D Conv (5x5x5) 3D Conv (1x1x1) 3D Conv (1x1x1) Out Ch = In Ch Out Ch = In Ch Out Ch = Num Classes Out Ch = Num Classes
Shared Encoding Path Shared Encoding Path Input: shared encoder f=16 6 1 = f f=32 2 zero-filled MRI 3 = 344 344 f 344 344 f=64 4 160 160 6 = 344 344 f 172 172 80 80 160 160 172 172 344 344 86 86 40 40 86 86 Reconstruction Decoding path Reconstruction Decoding path f=32 2 f=64 4 6 3 = = f f 160 160 80 80 344344 344344 172 172 172 172 344 344 f=16 6 1 = f Shared Features Shared Features 160 160 344 344 Embedding Embedding f f=64 = 6 4 f=32 2 f=64 4 Segmentation Decoding path Segmentation Decoding path 3 6 = = 86 86 f f 40 40 80 80 160 160 86 86 172172 172172 344 344 344 344 6 f=16 1 = f Residual skip Residual skip Strided 3D Conv (2x2x2) Strided 3D Conv (2x2x2) Intra-task skip Intra-task skip Out Ch = 2x In Ch Out Ch = 2x In Ch Inter-task skip Inter-task skip Strided 3D Conv (2x2x2) Strided 3D Conv (2x2x2) Concatenate Concatenate Out Ch = In Ch/2 Out Ch = In Ch/2 344 344 3D Conv (5x5x5) 3D Conv (5x5x5) 3D Conv (5x5x5) 3D Conv (5x5x5) Out Ch = 16x In Ch Out Ch = 16x In Ch Out Ch = In Ch/2 Out Ch = In Ch/2 160 160 344 344 3D Conv (5x5x5) 3D Conv (5x5x5) 3D Conv (1x1x1) 3D Conv (1x1x1) Out Ch = In Ch Out Ch = In Ch Out Ch = Num Classes Out Ch = Num Classes
Shared Encoding Path Shared Encoding Path Input: shared encoder f=16 6 1 = f f=32 2 zero-filled MRI 3 = 344 344 f 344 344 f=64 4 160 160 6 = 344 344 f 172 172 80 80 160 160 172 172 344 344 86 86 40 40 86 86 Reconstruction Decoding path Reconstruction Decoding path f=32 2 f=64 4 6 3 = = f f 160 160 80 80 344344 344344 172 172 172 172 344 344 f=16 6 1 = f Shared Features Shared Features Common 160 160 344 344 Embedding Embedding Feature f f=64 = 6 Embedding 4 f=32 2 f=64 4 Segmentation Decoding path Segmentation Decoding path 3 6 = = 86 86 f f 40 40 80 80 160 160 86 86 172172 172172 344 344 344 344 f=16 6 1 = f Residual skip Residual skip Strided 3D Conv (2x2x2) Strided 3D Conv (2x2x2) Intra-task skip Intra-task skip Out Ch = 2x In Ch Out Ch = 2x In Ch Inter-task skip Inter-task skip Strided 3D Conv (2x2x2) Strided 3D Conv (2x2x2) Concatenate Concatenate Out Ch = In Ch/2 Out Ch = In Ch/2 344 344 3D Conv (5x5x5) 3D Conv (5x5x5) 3D Conv (5x5x5) 3D Conv (5x5x5) Out Ch = 16x In Ch Out Ch = 16x In Ch Out Ch = In Ch/2 Out Ch = In Ch/2 160 160 344 344 3D Conv (5x5x5) 3D Conv (5x5x5) 3D Conv (1x1x1) 3D Conv (1x1x1) Out Ch = In Ch Out Ch = In Ch Out Ch = Num Classes Out Ch = Num Classes
Shared Encoding Path Shared Encoding Path Shared Encoding Path Input: shared encoder f=16 6 f=16 1 = f f=32 2 zero-filled MRI f=32 3 = 344 344 f 344 344 344 344 f=64 4 160 160 f=64 6 160 = 344 344 f 344 172 172 80 80 172 80 160 160 160 172 172 172 344 344 344 86 86 40 40 86 40 86 86 86 Reconstruction Decoding path Reconstruction Decoding path Reconstruction Decoding path f=32 2 f=64 4 f=64 f=32 6 3 = = f f 160 160 160 80 80 80 344344 344344 172 172 344344 172 172 172 172 344 344 344 f=16 6 f=16 1 = f Shared Features Shared Features Shared Features Common 160 160 160 344 344 344 Embedding Embedding Embedding Feature 2 decoders f f=64 = f = 6 Embedding 6 4 4 f=32 2 f=64 4 f=32 f=64 Segmentation Decoding path Segmentation Decoding path 3 6 Segmentation Decoding path = = 86 86 f f 86 40 40 80 80 40 160 160 80 160 86 86 172172 172172 344 344 86 172172 344 344 344 344 f=16 6 f=16 1 = f Residual skip Residual skip Strided 3D Conv (2x2x2) Strided 3D Conv (2x2x2) Residual skip Strided 3D Conv (2x2x2) Intra-task skip Intra-task skip Out Ch = 2x In Ch Out Ch = 2x In Ch Intra-task skip Out Ch = 2x In Ch Inter-task skip Inter-task skip Strided 3D Conv (2x2x2) Strided 3D Conv (2x2x2) Inter-task skip Strided 3D Conv (2x2x2) Concatenate Concatenate Out Ch = In Ch/2 Out Ch = In Ch/2 Concatenate Out Ch = In Ch/2 344 344 344 3D Conv (5x5x5) 3D Conv (5x5x5) 3D Conv (5x5x5) 3D Conv (5x5x5) 3D Conv (5x5x5) 3D Conv (5x5x5) Out Ch = 16x In Ch Out Ch = 16x In Ch Out Ch = In Ch/2 Out Ch = In Ch/2 160 160 Out Ch = 16x In Ch Out Ch = In Ch/2 160 344 344 344 3D Conv (5x5x5) 3D Conv (5x5x5) 3D Conv (1x1x1) 3D Conv (1x1x1) 3D Conv (5x5x5) 3D Conv (1x1x1) Out Ch = In Ch Out Ch = In Ch Out Ch = Num Classes Out Ch = Num Classes Out Ch = In Ch Out Ch = Num Classes
Shared Encoding Path Shared Encoding Path Shared Encoding Path Input: shared encoder f=16 6 f=16 1 = f f=32 f=32 2 zero-filled MRI 3 = 344 344 344 f 344 344 344 f=64 f=64 4 160 160 160 6 = 344 344 344 f 172 172 172 80 80 80 160 160 160 172 172 172 344 344 344 86 86 86 40 40 40 86 86 86 Reconstruction Decoding path Reconstruction Decoding path Reconstruction Decoding path f=32 2 f=64 f=64 4 f=32 6 3 = = f f 160 160 160 80 80 80 344344 344344 344344 172 172 172 172 172 172 344 344 344 f=16 f=16 6 1 = f Shared Features Shared Features Shared Features Common 160 160 160 reconstruction 344 344 344 Embedding Embedding Embedding Feature decoder f f=64 f = = 6 6 Embedding 4 4 f=32 f=32 2 f=64 f=64 4 Segmentation Decoding path Segmentation Decoding path Segmentation Decoding path 3 6 = = 86 86 86 f f 40 40 40 80 80 80 160 160 160 86 86 86 172172 172172 172172 344 344 344 344 344 344 f=16 f=16 6 1 = f Residual skip Residual skip Residual skip Strided 3D Conv (2x2x2) Strided 3D Conv (2x2x2) Strided 3D Conv (2x2x2) Intra-task skip Intra-task skip Intra-task skip Out Ch = 2x In Ch Out Ch = 2x In Ch Out Ch = 2x In Ch Inter-task skip Inter-task skip Inter-task skip Strided 3D Conv (2x2x2) Strided 3D Conv (2x2x2) Strided 3D Conv (2x2x2) Concatenate Concatenate Concatenate Out Ch = In Ch/2 Out Ch = In Ch/2 Out Ch = In Ch/2 344 344 344 3D Conv (5x5x5) 3D Conv (5x5x5) 3D Conv (5x5x5) 3D Conv (5x5x5) 3D Conv (5x5x5) 3D Conv (5x5x5) Out Ch = 16x In Ch Out Ch = 16x In Ch Out Ch = 16x In Ch Out Ch = In Ch/2 Out Ch = In Ch/2 Out Ch = In Ch/2 160 160 160 344 344 344 3D Conv (5x5x5) 3D Conv (5x5x5) 3D Conv (5x5x5) 3D Conv (1x1x1) 3D Conv (1x1x1) 3D Conv (1x1x1) Out Ch = In Ch Out Ch = In Ch Out Ch = In Ch Out Ch = Num Classes Out Ch = Num Classes Out Ch = Num Classes
Recommend
More recommend
