VS-Net: Variable Splitting Network for Accelerated Parallel MRI - - PowerPoint PPT Presentation

VS-Net: Variable Splitting Network for Accelerated Parallel MRI Reconstruction

Jinming Duan†1,2, Jo Schlemper†2,3, Chen Qin2, Cheng Ouyang2, Wenjia Bai2, Carlo Biffi2, Ghalib Bello2, Ben Statton2, Declan P O’Regan2, Daniel Rueckert2

1School of Computer Science, University of Birmingham, UK 2Imperial College London, UK 3Hyperfine Research, CT, USA †contributed equally. 1/ 14

Motivation:

◮ Slow MR imaging process leads to low patient throughput, patient’s discomfort,

image artifacts, etc.

2/ 14

Motivation:

◮ Slow MR imaging process leads to low patient throughput, patient’s discomfort,

image artifacts, etc.

◮ Compressed sensing and parallel MRI are often used to speed up

2/ 14

Motivation:

◮ Slow MR imaging process leads to low patient throughput, patient’s discomfort,

image artifacts, etc.

◮ Compressed sensing and parallel MRI are often used to speed up ◮ Parallel MRI is common in clinical practice currently

2/ 14

Motivation:

◮ Slow MR imaging process leads to low patient throughput, patient’s discomfort,

image artifacts, etc.

◮ Compressed sensing and parallel MRI are often used to speed up ◮ Parallel MRI is common in clinical practice currently ◮ Deep learning based approaches [Schlemper, et al, Hammernik, et al, Yan, et al]

have shown great promise

2/ 14

Motivation:

◮ Slow MR imaging process leads to low patient throughput, patient’s discomfort,

image artifacts, etc.

◮ Compressed sensing and parallel MRI are often used to speed up ◮ Parallel MRI is common in clinical practice currently ◮ Deep learning based approaches [Schlemper, et al, Hammernik, et al, Yan, et al]

have shown great promise

◮ Combing parallel MRI, compressed sensing and deep learning is the aim

2/ 14

Motivation:

◮ Slow MR imaging process leads to low patient throughput, patient’s discomfort,

image artifacts, etc.

◮ Compressed sensing and parallel MRI are often used to speed up ◮ Parallel MRI is common in clinical practice currently ◮ Deep learning based approaches [Schlemper, et al, Hammernik, et al, Yan, et al]

have shown great promise

◮ Combing parallel MRI, compressed sensing and deep learning is the aim

2/ 14

General CS parallel MRI model:

Technically, one can reconstruct a clean image m ∈ CN by minimizing min

m

λ 2

nc

• i=1

DFSim − yi2

2 + R (m),

(1)

3/ 14

General CS parallel MRI model:

Technically, one can reconstruct a clean image m ∈ CN by minimizing min

m

λ 2

nc

• i=1

DFSim − yi2

2 + R (m),

(1) where

◮ λ is a smooth parameter

3/ 14

General CS parallel MRI model:

Technically, one can reconstruct a clean image m ∈ CN by minimizing min

m

λ 2

nc

• i=1

DFSim − yi2

2 + R (m),

(1) where

◮ λ is a smooth parameter ◮ yi ∈ CM (M < N) is undersampled k-space of the ith coil (nc coils)

3/ 14

General CS parallel MRI model:

Technically, one can reconstruct a clean image m ∈ CN by minimizing min

m

λ 2

nc

• i=1

DFSim − yi2

2 + R (m),

(1) where

◮ λ is a smooth parameter ◮ yi ∈ CM (M < N) is undersampled k-space of the ith coil (nc coils) ◮ D ∈ RM×N is sampling matrix that zeros out entries not acquired

3/ 14

General CS parallel MRI model:

Technically, one can reconstruct a clean image m ∈ CN by minimizing min

m

λ 2

nc

• i=1

DFSim − yi2

2 + R (m),

(1) where

◮ λ is a smooth parameter ◮ yi ∈ CM (M < N) is undersampled k-space of the ith coil (nc coils) ◮ D ∈ RM×N is sampling matrix that zeros out entries not acquired ◮ F ∈ CN×N is the Fourier transform matrix

3/ 14

General CS parallel MRI model:

Technically, one can reconstruct a clean image m ∈ CN by minimizing min

m

λ 2

nc

• i=1

DFSim − yi2

2 + R (m),

(1) where

◮ λ is a smooth parameter ◮ yi ∈ CM (M < N) is undersampled k-space of the ith coil (nc coils) ◮ D ∈ RM×N is sampling matrix that zeros out entries not acquired ◮ F ∈ CN×N is the Fourier transform matrix ◮ Si ∈ CN×N is the ith coil sensitivity, precomputed from fully sampled k-space

center using E-SPIRiT [Uecker, et al]

3/ 14

General CS parallel MRI model:

Technically, one can reconstruct a clean image m ∈ CN by minimizing min

m

λ 2

nc

• i=1

DFSim − yi2

2 + R (m),

(1) where

◮ λ is a smooth parameter ◮ yi ∈ CM (M < N) is undersampled k-space of the ith coil (nc coils) ◮ D ∈ RM×N is sampling matrix that zeros out entries not acquired ◮ F ∈ CN×N is the Fourier transform matrix ◮ Si ∈ CN×N is the ith coil sensitivity, precomputed from fully sampled k-space

center using E-SPIRiT [Uecker, et al]

◮ R (m) is a regularization term, e.g. TV, TGV, wavelet L1, etc.

3/ 14

Variable splitting optimization:

We introduce splitting variables u ∈ CN and {xi ∈ CN}nc

i=1, converting (1) into

min

m,u,xi

λ 2

nc

• i=1

DFxi − yi2

2 + R (u) s.t. m = u, Sim = xi, ∀i ∈ {1, 2, ..., nc} .

4/ 14

Variable splitting optimization:

We introduce splitting variables u ∈ CN and {xi ∈ CN}nc

i=1, converting (1) into

min

m,u,xi

λ 2

nc

• i=1

DFxi − yi2

2 + R (u) s.t. m = u, Sim = xi, ∀i ∈ {1, 2, ..., nc} .

We add constraints back via penalty function method and minimize min

m,u,xi

λ 2

nc

• i=1

DFxi − yi2

2 + R (u) + α

2

nc

• i=1

xi − Sim2

2 + β

2 u − m2

2 ,

(2)

4/ 14

Variable splitting optimization:

We introduce splitting variables u ∈ CN and {xi ∈ CN}nc

i=1, converting (1) into

min

m,u,xi

λ 2

nc

• i=1

DFxi − yi2

2 + R (u) s.t. m = u, Sim = xi, ∀i ∈ {1, 2, ..., nc} .

We add constraints back via penalty function method and minimize min

m,u,xi

λ 2

nc

• i=1

DFxi − yi2

2 + R (u) + α

2

nc

• i=1

xi − Sim2

2 + β

2 u − m2

2 ,

(2) To minimize (2), we alternatively optimize m, u and xi w.r.t. the three subproblems:          uk+1 = arg min

u β 2 u − mk2 2 + R (u)

xk+1

i

= arg min

xi

λ nc

i=1 DFxi − yi2 2 + α 2

nc

i=1 xi − Simk2 2

mk+1 = arg min

m α 2

nc

i=1 xk+1 i

− Sim2

2 + β 2 uk+1 − m2 2

, (3)

4/ 14

Variable splitting optimization:

We introduce splitting variables u ∈ CN and {xi ∈ CN}nc

i=1, converting (1) into

min

m,u,xi

λ 2

nc

• i=1

DFxi − yi2

2 + R (u) s.t. m = u, Sim = xi, ∀i ∈ {1, 2, ..., nc} .

We add constraints back via penalty function method and minimize min

m,u,xi

λ 2

nc

• i=1

DFxi − yi2

2 + R (u) + α

2

nc

• i=1

xi − Sim2

2 + β

2 u − m2

2 ,

(2) To minimize (2), we alternatively optimize m, u and xi w.r.t. the three subproblems:          uk+1 = arg min

u β 2 u − mk2 2 + R (u)

xk+1

i

= arg min

xi

λ nc

i=1 DFxi − yi2 2 + α 2

nc

i=1 xi − Simk2 2

mk+1 = arg min

m α 2

nc

i=1 xk+1 i

− Sim2

2 + β 2 uk+1 − m2 2

, (3) Here k ∈ {1, ..., nit} denotes the kth iteration. α and β are introduced penalty weights.

4/ 14

Variable splitting optimization:

An optimal solution (m∗) may be found by iterating over uk+1, xk+1

i

and mk+1 using    uk+1 = denoiser(mk) xk+1

i

= F−1((λDTD + αI)−1(αFSimk + λDTyi)) ∀i ∈ {1, 2, ..., nc} mk+1 = (βI + α nc

i=1 SH i Si)−1(βuk+1 + α nc i=1 SH i xk+1 i

) . (4)

5/ 14

Variable splitting optimization:

An optimal solution (m∗) may be found by iterating over uk+1, xk+1

i

and mk+1 using    uk+1 = denoiser(mk) xk+1

i

= F−1((λDTD + αI)−1(αFSimk + λDTyi)) ∀i ∈ {1, 2, ..., nc} mk+1 = (βI + α nc

i=1 SH i Si)−1(βuk+1 + α nc i=1 SH i xk+1 i

) . (4)

◮ 1st Eq. is an image denoising solver (ie. denoiser)

5/ 14

Variable splitting optimization:

An optimal solution (m∗) may be found by iterating over uk+1, xk+1

i

and mk+1 using    uk+1 = denoiser(mk) xk+1

i

= F−1((λDTD + αI)−1(αFSimk + λDTyi)) ∀i ∈ {1, 2, ..., nc} mk+1 = (βI + α nc

i=1 SH i Si)−1(βuk+1 + α nc i=1 SH i xk+1 i

) . (4)

◮ 1st Eq. is an image denoising solver (ie. denoiser) ◮ 2nd Eq. is a closed-form, point-wise and coil-wise data consistency term

5/ 14

Variable splitting optimization:

An optimal solution (m∗) may be found by iterating over uk+1, xk+1

i

and mk+1 using    uk+1 = denoiser(mk) xk+1

i

= F−1((λDTD + αI)−1(αFSimk + λDTyi)) ∀i ∈ {1, 2, ..., nc} mk+1 = (βI + α nc

i=1 SH i Si)−1(βuk+1 + α nc i=1 SH i xk+1 i

) . (4)

◮ 1st Eq. is an image denoising solver (ie. denoiser) ◮ 2nd Eq. is a closed-form, point-wise and coil-wise data consistency term ◮ 3rd Eq. is a closed-form, point-wise weighted average

5/ 14

Variable splitting optimization:

An optimal solution (m∗) may be found by iterating over uk+1, xk+1

i

and mk+1 using    uk+1 = denoiser(mk) xk+1

i

= F−1((λDTD + αI)−1(αFSimk + λDTyi)) ∀i ∈ {1, 2, ..., nc} mk+1 = (βI + α nc

i=1 SH i Si)−1(βuk+1 + α nc i=1 SH i xk+1 i

) . (4)

◮ 1st Eq. is an image denoising solver (ie. denoiser) ◮ 2nd Eq. is a closed-form, point-wise and coil-wise data consistency term ◮ 3rd Eq. is a closed-form, point-wise weighted average

5/ 14

VS-Net architecture:

Figure: Architecture of the proposed variable splitting network (VS-Net). DCL and WAL represents Data Consistency Layer and Weighted Average Layer, respectively.

6/ 14

VS-Net architecture:

Denoiser ReLU 3x3 Conv ReLU 3x3 Conv ReLU 3x3 Conv ReLU 3x3 Conv 3x3 Conv

nf=64 nf=64 nf=64 nf=64 nf=2

Denoiser Layer (via CNN)

DCL WAL mk , Si , k0 , mask

• Eq. 4 middle

…

mk uk+1 𝑦"

#$"

• Eq. 4 middle
• Eq. 4 middle

Input

i=1 i=2 i=nc

𝑦%

#$"

𝑦&'

#$"

… .. .. ..

Output Input Output Data Consistency Layer

𝑦"

#$", 𝑇"

𝑦%

#$", 𝑇%

𝑦&'

#$", 𝑇&'

…

uk+1

• Eq. 4 bottom

mk+1

Input Output Weight Average Layer

Figure: Detailed structure of each layer in VS-net. DCL and WAL represents Data Consistency Layer and Weighted Average Layer, respectively.

7/ 14

VS-Net loss and parameterizations:

We use the mean squared error (MSE) for VS-net loss L(Θ) = min

Θ

1 2ni

ni

• i=1

mnit

i (Θ) − gi2 2,

(5)

8/ 14

VS-Net loss and parameterizations:

We use the mean squared error (MSE) for VS-net loss L(Θ) = min

Θ

1 2ni

ni

• i=1

mnit

i (Θ) − gi2 2,

(5)

◮ Two parameterizations are studied for Θ, i.e., Θ1 =

• {Wl}nit

l=1, λ, α, β

• and

Θ2 =

• {Wl, λl, αl, βl}nit

l=1

• 8/ 14

VS-Net loss and parameterizations:

We use the mean squared error (MSE) for VS-net loss L(Θ) = min

Θ

1 2ni

ni

• i=1

mnit

i (Θ) − gi2 2,

(5)

◮ Two parameterizations are studied for Θ, i.e., Θ1 =

• {Wl}nit

l=1, λ, α, β

• and

Θ2 =

• {Wl, λl, αl, βl}nit

l=1

• ◮ {Wl}nit

l=1 means W are learnable parameters in each CNN. For Θ1 the weights λ,

α and β are shared, while for Θ2 they are not

8/ 14

Experiments results:

Figure: Quantitative measures versus number of epochs at testing. Top: network performance verse cascade numbers. Bottom: network performance verse parameterizations (Θ1 and Θ2).

9/ 14

Figure: Visual comparison using different methods for Cartesian undersampling with AF 4 (top) and 6 (bottom). From left to right: zero-filling results, ℓ1-SPIRiT, VN, VS-Net, and ground truth.

10/ 14

Table: Quantitative results obtained by different methods on the test set including ∼2000 image slices across 5 acquisition protocols. Each metric was calculated on ∼400 image slices, and mean ± standard deviation are reported.

4-fold AF 6-fold AF Protocol Method PSNR SSIM PSNR SSIM Coronal fat-sat. PD Zero-filling 32.34±2.83 0.80±0.11 30.47±2.71 0.74±0.14 ℓ1-SPIRiT 34.57±3.32 0.81±0.11 31.51±2.21 0.78±0.08 VN 35.83±4.43 0.84±0.13 32.90±4.66 0.78±0.15 VS-Net 36.00±3.83 0.84±0.13 33.24±3.44 0.78±0.15 Coronal PD Zero-filling 31.35±3.84 0.87±0.11 29.39±3.81 0.84±0.13 ℓ1-SPIRiT 39.38±2.16 0.93±0.03 34.06±2.41 0.88±0.04 VN 40.14±4.97 0.94±0.12 36.01±4.63 0.90±0.13 VS-Net 41.27±5.25 0.95±0.12 36.77±4.84 0.92±0.14 Axial fat-sat. T2 Zero-filling 36.47±2.34 0.94±0.02 34.90±2.39 0.92±0.02 ℓ1-SPIRiT 39.38±2.70 0.94±0.03 35.44±2.87 0.91±0.03 VN 42.10±1.97 0.97±0.01 37.94±2.29 0.94±0.02 VS-Net 42.34±2.06 0.96±0.01 39.40±2.10 0.94±0.02 Sagittal fat-sat. T2 Zero-filling 37.35±2.69 0.93±0.07 35.25±2.68 0.90±0.09 ℓ1-SPIRiT 41.27±2.95 0.94±0.06 36.00±2.67 0.92±0.05 VN 42.84±3.47 0.95±0.07 38.92±3.23 0.93±0.09 VS-Net 43.10±3.44 0.95±0.07 39.07±3.33 0.92±0.09 Sagittal PD Zero-filling 37.12±2.58 0.96±0.04 35.96±2.57 0.94±0.05 ℓ1-SPIRiT 44.52±1.94 0.97±0.02 39.14±2.12 0.96±0.02 VN 46.34±2.75 0.98±0.05 39.71±2.58 0.96±0.05 VS-Net 47.22±2.89 0.98±0.04 40.11±2.46 0.96±0.05

11/ 14

Conclusion:

◮ We designed a variable splitting (VS) for the general CS parallel MRI model

12/ 14

Conclusion:

◮ We designed a variable splitting (VS) for the general CS parallel MRI model ◮ We proposed a VS-Net by unrolling VS process

12/ 14

Conclusion:

◮ We designed a variable splitting (VS) for the general CS parallel MRI model ◮ We proposed a VS-Net by unrolling VS process ◮ VS-Net imposes data consistency explicitly and coil-wisely

12/ 14

Conclusion:

◮ We designed a variable splitting (VS) for the general CS parallel MRI model ◮ We proposed a VS-Net by unrolling VS process ◮ VS-Net imposes data consistency explicitly and coil-wisely ◮ VS-Net has closed-form, point-wise solutions, thus easy to code and fast to train

12/ 14

Conclusion:

◮ We designed a variable splitting (VS) for the general CS parallel MRI model ◮ We proposed a VS-Net by unrolling VS process ◮ VS-Net imposes data consistency explicitly and coil-wisely ◮ VS-Net has closed-form, point-wise solutions, thus easy to code and fast to train ◮ We made all the model weights learnable (ie, λ and penalty weights α and β )

12/ 14

Conclusion:

◮ We designed a variable splitting (VS) for the general CS parallel MRI model ◮ We proposed a VS-Net by unrolling VS process ◮ VS-Net imposes data consistency explicitly and coil-wisely ◮ VS-Net has closed-form, point-wise solutions, thus easy to code and fast to train ◮ We made all the model weights learnable (ie, λ and penalty weights α and β ) ◮ Our experiments showed improved results using VS-Net

12/ 14

References:

[1] Schlemper, J., et al.: A deep cascade of convolutional neural networks for dynamic MR image reconstruction. IEEE Trans. Med. Imag. 37(2) (2018) 491–503 [2] Yan, Y., et al.: Deep ADMM-Net for compressive sensing MRI. In: NIPS. (2016) [3] Hammernik, K., et al.: Learning a variational network for reconstruction of accelerated MRI data. Magn. Reson. Med. 79(6) (2018) 3055–3071 [4] Aggarwal, H.K., et al.: MoDL: Model-based deep learning architecture for inverse

• problems. IEEE Trans. Med. Imag. 38(2) (2019) 394–405

[5] Akcakaya, M., et al.: Scan-specific robust artificial-neural-networks for k-space interpolation (RAKI) reconstruction: Database-free deep learning for fast imaging.

• Magn. Reson. Med. 81(1) (2018) 439–453

[6] Uecker, M., et al.: Espirit an eigenvalue approach to autocalibrating parallel MRI: where sense meets grappa. Magn. Reson. Med. 71(3) (2014) 990–1001 [7] Uecker, M., et al.: Software toolbox and programming library for compressed sensing and parallel imaging, Citeseer [8] Murphy, M., et al.: Fast l1-spirit compressed sensing parallel imaging MRI: Scalable parallel implementation and clinically feasible runtime. IEEE Trans. Med. Imag. 31(6) (2012) 1250–1262

13/ 14

Thank you!

Code is available at https://github.com/j-duan/VS-Net

14/ 14