Microscopic Super-Resolution in the lateral plane by sparse - - PowerPoint PPT Presentation

Microscopic Super-Resolution in the lateral plane by sparse regularization in the covariance domain

Vasiliki Stergiopoulou Laure Blanc-Féraud (CNRS), Luca Calatroni (CNRS), Sebastien Schaub (IMEV) and Henrique de Morais Goulart (ENSEEIHT) Thematic Day: "Non-convex sparse optimization" Friday, October 9

Table of Contents Introduction Exploiting temporal fluctuations COL0RME Results Conclusions and Future work

2 / 23

Diffraction Limited Resolution of Conventional Fluorescence Microscopy ◮ Spatial resolution is limited by light diffraction phenomena ◮ Smallest resolvable distance (Rayleigh Criterion): d = 0.61λ

NA (≈ 200nm)

λ: emission wavelength, NA: Numerical Aperture.

3 / 23

Diffraction Limited Resolution of Conventional Fluorescence Microscopy ◮ Spatial resolution is limited by light diffraction phenomena ◮ Smallest resolvable distance (Rayleigh Criterion): d = 0.61λ

NA (≈ 200nm)

λ: emission wavelength, NA: Numerical Aperture.

3 / 23

State-of-the art methods for SR STED (Hell and Wichmann, 1994) STimulation-Emission-Depletion ◮ Depletes some of the excited fluorophores and limits the area of illumination SMLM (Betzig, Zhuang, Hess, 2006) Single Molecule Localization Microscopy ◮ Activation of few molecules, imaging and localization

http://zeiss-campus. magnet.fsu.edu/ 4 / 23

State-of-the art methods for SR STED (Hell and Wichmann, 1994) STimulation-Emission-Depletion ◮ Depletes some of the excited fluorophores and limits the area of illumination ◮ Requires special equipment, potentially harmful excitation levels SMLM (Betzig, Zhuang, Hess, 2006) Single Molecule Localization Microscopy ◮ Activation of few molecules, imaging and localization ◮ Time consuming acquisition, poor temporal resolution, potentially harmful excitation levels

http://zeiss-campus. magnet.fsu.edu/ 4 / 23

State-of-the art methods for SR STED (Hell and Wichmann, 1994) STimulation-Emission-Depletion ◮ Depletes some of the excited fluorophores and limits the area of illumination ◮ Requires special equipment, potentially harmful excitation levels SMLM (Betzig, Zhuang, Hess, 2006) Single Molecule Localization Microscopy ◮ Activation of few molecules, imaging and localization ◮ Time consuming acquisition, poor temporal resolution, potentially harmful excitation levels

http://zeiss-campus. magnet.fsu.edu/

We aim to design a Super-Resolution model with the following features: ◮ improved temporal resolution ◮ harmless excitation levels ◮ use of standard equipment / conventional fluorophores

4 / 23

Table of Contents Introduction Exploiting temporal fluctuations COL0RME Results Conclusions and Future work

5 / 23

Temporal fluctuations

Temporal profile of a fluorescent molecule: Temporal profile of a pixel (real data):

Idea : Acquire short videos with high-density of molecules per frame and use a reconstruction algorithm that exploits spatial and temporal independence

• f the emitters

6 / 23

Related approaches SOFI (Dertinger et al., 2009) Super resolution Optical Fluctuation Imaging ◮ Shrinkage of PSF via computation of higher-order statistics SRRF (Gustafsson et al., 2016) Super-Resolution Radial Fluctuations ◮ Non-linear transformation of each frame based

• n radial symmetry (the degree of local

gradient convergence) SPARCOM (Solomon et al., 2019) SPARsity based super-resolution COrrelation Mi- croscopy ◮ Exploits sparsity in the correlation domain

7 / 23

Table of Contents Introduction Exploiting temporal fluctuations COL0RME Results Conclusions and Future work

8 / 23

Method

COℓ0RME

COvariance-based ℓ0 super-Resolution Microscopy with intensity Estimation

1st Step: Support Estimation

(Covariance domain)

= ⇒ Support Ω 9 / 23

Method

COℓ0RME

COvariance-based ℓ0 super-Resolution Microscopy with intensity Estimation

1st Step: Support Estimation

(Covariance domain)

= ⇒ Support Ω

2nd Step: Intensity Estimation

9 / 23

Image model Yt = Mq(H (Xt)) + Nt + B t = 1, ..., T: all the frames Yt Xt Yt ∈ RN×N raw data Xt ∈ RL×L high-resolution image L = qN

Mq: down-sizing operator H: convolution operator Nt: additive white Gaussian noise B: stationary background

q=4

10 / 23

1st Step: Support Estimation ◮ Yt = Mq(H (Xt)) + Nt + B

vectorized form

− − − − − − − − − − → yt = Ψxt + nt + b

11 / 23

1st Step: Support Estimation ◮ Yt = Mq(H (Xt)) + Nt + B

vectorized form

− − − − − − − − − − → yt = Ψxt + nt + b

y1

=

Ψ

. . .

x1

. . .

y1y2

. . .

yT

=

Ψ

. . .

x1

. . .

x2

. . .

x3

. . . . . .

xT

. . .

Ry =

1 T −1 T

• t=1

(yt − µy)(yt − µy)T

Ry

=

Ψ

. . .

Rx

. . . . . . . . . . . . .

ΨT

. . . Rx = diag(rx) rx ∈ RL2

11 / 23

1st Step: Support Estimation ◮ Yt = Mq(H (Xt)) + Nt + B

vectorized form

− − − − − − − − − − → yt = Ψxt + nt + b ◮ Covariance matrices: Ry = ΨRxΨT

◮ Rx is diagonal, rx = diag(Rx)

11 / 23

1st Step: Support Estimation ◮ Yt = Mq(H (Xt)) + Nt + B

vectorized form

− − − − − − − − − − → yt = Ψxt + nt + b ◮ Covariance matrices: Ry = ΨRxΨT + Rn

◮ Rx is diagonal, rx = diag(Rx) ◮ Rn = sI, s ∈ R+

11 / 23

1st Step: Support Estimation ◮ Yt = Mq(H (Xt)) + Nt + B

vectorized form

− − − − − − − − − − → yt = Ψxt + nt + b ◮ Covariance matrices: Ry = ΨRxΨT + Rn

◮ Rx is diagonal, rx = diag(Rx) ◮ Rn = sI, s ∈ R+

11 / 23

1st Step: Support Estimation ◮ Yt = Mq(H (Xt)) + Nt + B

vectorized form

− − − − − − − − − − → yt = Ψxt + nt + b ◮ Covariance matrices: Ry = ΨRxΨT + Rn

◮ Rx is diagonal, rx = diag(Rx) ◮ Rn = sI, s ∈ R+

◮ Ry = ΨRxΨT + sI

vectorized form

− − − − − − − − − − → ry = (Ψ ⊙ Ψ)rx + sIv Problem l2-l0 : arg min

rx≥0,s≥0

1 2ry − (Ψ ⊙ Ψ)rx − sIv2

F + λrx0

rx = 0− → Support Ω

11 / 23

1st Step: Support Estimation l2 - l0 by continuous relaxation1with noise variance estimation G(rx, s) = 1

2ry − (Ψ ⊙ Ψ)rx − sIv2 2 + λrx0 →

˜ G(rx, s) = 1

2ry − (Ψ ⊙ Ψ)rx − sIv2 2 + ΦCEL0(rx)

ΦCEL0(rx) =

L2

• i=1

φCEL0(ai, λ; (rx)i) =

L2

• i=1

λ− ai2

2

• |(rx)i| −

√ 2λ ai

• 1{|(rx)i|≤

√ 2λ ai }

and ai = (Ψ ⊙ Ψ)i

1Soubies et al., A Continuous Exact l0 penalty (CEL0) for least squares regularized

problem, SIAM J. Imaging Sciences, 2015

2Ochs et al., On Iteratively Reweighted Algorithms for Nonsmooth Nonconvex

Optimization in Computer Vision, SIAM J. Imaging Sciences, 2014

12 / 23

1st Step: Support Estimation l2 - l0 by continuous relaxation1with noise variance estimation G(rx, s) = 1

2ry − (Ψ ⊙ Ψ)rx − sIv2 2 + λrx0 →

˜ G(rx, s) = 1

2ry − (Ψ ⊙ Ψ)rx − sIv2 2 + ΦCEL0(rx)

ΦCEL0(rx) =

L2

• i=1

φCEL0(ai, λ; (rx)i) =

L2

• i=1

λ− ai2

2

• |(rx)i| −

√ 2λ ai

• 1{|(rx)i|≤

√ 2λ ai }

and ai = (Ψ ⊙ Ψ)i

Alternating minimization: Iterative Reweighted l1 algorithm2& explicit expression:

Require: rx0 ∈ RL2 , s0 ∈ R+ repeat calculate the weights: ω

rn x i

rxn+1 = arg min

rx∈RL2 1 2 ry − (Ψ ⊙ Ψ)rx − snIv2 2 + λ L2

• i=1

ωi

rn x |(rx)i| + X≥0(rx)

sn+1 = arg min

s∈R 1 2 ry − (Ψ ⊙ Ψ)rn+1 x

− sIv2

2 + X≥0(s)

until convergence print rx, s

1Soubies et al., A Continuous Exact l0 penalty (CEL0) for least squares regularized

problem, SIAM J. Imaging Sciences, 2015

2Ochs et al., On Iteratively Reweighted Algorithms for Nonsmooth Nonconvex

Optimization in Computer Vision, SIAM J. Imaging Sciences, 2014

12 / 23

2nd Step: Intensity estimation

✘✘✘ ✘

x∈ RL2 x∈ R|Ω| The model: yt = Ψxt + nt + b

arg min

x≥0,b≥0 1 2 Y1T − ΨΩx − Tb2 2 + β i∈Ω

• j∈N(i)∩Ω

(xi − xj)2 + α∇b2

2

13 / 23

Table of Contents Introduction Exploiting temporal fluctuations COL0RME Results Conclusions and Future work

14 / 23

Simulated acquisition Ground truth Size of the image: 160 x 160 Fluorescent Molecules (FM): 5471 Average FM/pixel/frame: 10.7 Observation (Low Background)

(SNR = 21.26dB)

Size of the image: 40 x 40 Pixel size: 100 µm Video rate: 100 fps Acquisition time: 10 s Spatial pattern: Microtubules dataset from the SMLM challenge 2016 3 Temporal profiles: SOFI simulation tool (Girsault et al.. 2016)

3http://bigwww.epfl.ch/smlm/datasets/index.html 15 / 23

Simulated results Ground Truth COL0RME

T = 100

COL0RME

T = 700

SRRF

T = 100

SRRF

T = 700

16 / 23

Simulated results Ground Truth COL0RME

T = 100

COL0RME

T = 700

SPARCOM

T = 100

SPARCOM

T = 700

16 / 23

Simulated acquisition Ground truth Size of the image: 160 x 160 Fluorescent Molecules (FM): 5471 Average FM/pixel/frame: 10.7 Observation (High Background)

(SNR = 19.64dB)

Size of the image: 40 x 40 Pixel size: 100 µm Video rate: 100 fps Acquisition time: 10 s Spatial pattern: Microtubules dataset from the SMLM challenge 2016 4 Temporal profiles: SOFI simulation tool (Girsault et al.. 2016)

4http://bigwww.epfl.ch/smlm/datasets/index.html 17 / 23

Simulated results Ground Truth COL0RME

T = 100

COL0RME

T=700

SRRF

T = 100

SRRF

T = 700

18 / 23

Simulated results Ground Truth COL0RME

T = 100

COL0RME

T=700

SPARCOM

T = 100

SPARCOM

T = 700

18 / 23

Simulated results Ground Truth COL0RME

T = 100

COL0RME

T=700

SPARCOM

T = 100

SPARCOM

T = 700

18 / 23

Experimental data: High Density SMLM ◮ Tubulin 2D High Density dataset from the SMLM challenge 2013 5 ◮ T = 500 ◮ Video rate = 25 fps ◮ Acquisition time = 20 s Observation (4x zoom) COL0RME SRRF

5http://bigwww.epfl.ch/smlm/datasets/index.html 19 / 23

Experimental data: High Density SMLM ◮ Tubulin 2D High Density dataset from the SMLM challenge 2013 5 ◮ T = 500 ◮ Video rate = 25 fps ◮ Acquisition time = 20 s

Observation (4x zoom) Support estimation Background estimation Intensity estimation

5http://bigwww.epfl.ch/smlm/datasets/index.html 19 / 23

Real data Observation (4x zoom) COL0RME SRRF

20 / 23

Table of Contents Introduction Exploiting temporal fluctuations COL0RME Results Conclusions and Future work

21 / 23

Conclusions and Future work Main points: ◮ Sparse ℓ0 optimisation methods for SR in covariance domain ◮ Precise localization - Elimination of artifacts under difficult imaging conditions ◮ Noise variance, background and intensity estimation (= SPARCOM) Future work ◮ Go to 3D super-resolution ◮ SR using learning approaches

22 / 23

Thank you Thank you for your attention. Do you have any questions?

23 / 23

Supplementary Slides

23 / 23

Temporal Fluctuations of our data SMLM HD data Our Real Data

23 / 23

LSPARCOM Simulated Data (Low Background - T = 500) Real Data

23 / 23

LSPARCOM Simulated Data (Low Background - T = 500) Real Data

23 / 23