Computational aspects of SIM Rainer Heintzmann , - Leibniz - - PowerPoint PPT Presentation

▶
computational aspects of sim
SMART_READER_LITE
LIVE PREVIEW

Computational aspects of SIM Rainer Heintzmann , - Leibniz - - PowerPoint PPT Presentation

Dec 02, 2023 238 likes •976 views

Computational aspects of SIM Rainer Heintzmann , - Leibniz Institute of Photonic Technology (IPHT), - Friedrich Schiller University of Jena Trieste, 23/02/2017 1 Paradigm: Optimize for direct visibility + Image Object Optics E.g.:

slide-1
SLIDE 1

Computational aspects of SIM

Rainer Heintzmann,

  • Leibniz Institute of Photonic Technology (IPHT),
  • Friedrich Schiller University of Jena

1

Trieste, 23/02/2017

slide-2
SLIDE 2

Paradigm: Optimize for direct visibility

E.g.: Widefield, Confocal, STED

Does not necessarily optimize information content!

+

  • Optics

Object Image

slide-3
SLIDE 3

Paradigm: Optimize for information content

Data +

  • Optics

Object +

  • Data

Image

Computation

slide-4
SLIDE 4

Examples in Medical Imaging

MRI PET SPECT fMRI

CT

http://www.vetmed.lsu.edu/vth&c/Orthopedics/Images/ Computed%20Tomography%20(CT)%20Scanner.RV.jpg http://www.cis.rit.edu/htbooks/mri/images/head.gif http://www.cerebromente.org.br/n01/pet/petdep.gif http://www.fmri.wfubmc.edu/other%20pics/ lab_brain_logo.JPG http://www.physics.ubc.ca/research/images/spect.gif

slide-5
SLIDE 5

5

Structured Illumination (SIM)

slide-6
SLIDE 6

Moiré Demonstration

slide-7
SLIDE 7

Moiré effect

7 high frequency detail

Sample

Aurélie Jost Aurélie Jost

... is lost

slide-8
SLIDE 8

Moiré effect

8 high frequency grid high frequency detail low frequency Moiré

Sample Illumination

Aurélie Jost Aurélie Jost

HF information is present

slide-9
SLIDE 9

The Moiré effect

Moiré fringes

Image: Wikipedia (author:Ildar Sagdejev)

slide-10
SLIDE 10

Multiplication in real space ↕ Convolution in Fourier space

Iem (x) = Obj(x) · Iex (x) Iem (k) = Obj(k) Iex (k)

~ ~ ~

Image formation in FLUORESCENCE

slide-11
SLIDE 11

detectable region

Object Fluorescence Distribution

+K

  • K

magnitude spatial freq.

Iem (x) = Obj(x) · Iex (x) Iem (k) = Obj(k) Iex (k)

~ ~ ~

Iem (k)

~

Image (k)

~

Obj +1 (k)

~

slide-12
SLIDE 12

detectable region

Piecing Parts Together

+K

  • K

magnitude spatial freq.

Iem (k)

~

Image (k)

~

  • Correct for OTF
  • Extract components
  • Shift into place
  • Weighted average

Obj +1 (k)

~

  • Apodize
slide-13
SLIDE 13

Sample Illumination

Structured Illumination Micropscopy

Sample with structured illumination Multiplication of sample and illumination

slide-14
SLIDE 14

Sample Illumination

Structured Illumination Micropscopy

slide-15
SLIDE 15

Sample

Structured Illumination Micropscopy

slide-16
SLIDE 16

Structured Illumination Micropscopy

Sample Sample & llumination

slide-17
SLIDE 17

Sample Sample & llumination Imaging leads to loss of high frequencies (OTF)

slide-18
SLIDE 18

Separating the components… Sample

slide-19
SLIDE 19

Separating the components… Shifting the components… Sample

slide-20
SLIDE 20

Separating the components… Shifting the components… Recombining the components… Sample

slide-21
SLIDE 21

Separating the components… Shifting the components… Recombining the components… using the correct weights. Sample Reconstructed sample

Image processing !

slide-22
SLIDE 22

Laser CCD

x z

Tube lens Filter Dichromatic reflector Tube lens Objective Sample

Diffraction grating, SLM, etc…

Multiple images for order separation

slide-23
SLIDE 23

3D Structured Illumination

M.G.L Gustafsson et al., Three-dimensional Resolution Doubling in Widefield Fluorescence Microscopy by Structured Illumination, Biophys.

  • J. (BioFAST), 2008

Microtubule cytoskeleton in HeLa cells

slide-24
SLIDE 24

Microscopy image Resolution map

shifted information shifted information shifted information shifted information

Reconstruct high resolution image like a puzzle

slide-25
SLIDE 25

Separated puzzle pieces

larger resolution available

slide-26
SLIDE 26

Separated puzzle pieces

slide-27
SLIDE 27

extra information

Joined puzzle

slide-28
SLIDE 28

1 m

slide-29
SLIDE 29

1 m

slide-30
SLIDE 30

Proof of Principle 1999

Heintzmann & Cremer 1999 Proc. SPIE 3568, 185-196

slide-31
SLIDE 31

Structured Illumination

2011 3D live video

slide-32
SLIDE 32

3d live cell SIM

cytosol (red), actin (green) Images by Reto Fiolka, Janelia Farm Research Campus, HHMI, Ashburn, VA, USA

slide-33
SLIDE 33

Rainer Heintzmann, 2012 33

The nitty gritty details

Unknowns: grating constant (precise value) grating orientation local phase global phase

  • rder contrast

illumination intensity sample position (drift)

slide-34
SLIDE 34

Rainer Heintzmann, 2012 34

The nitty gritty details

Unknowns: grating constant (precise value) grating orientation local phase global phase

  • rder contrast

illumination intensity sample position (drift)

slide-35
SLIDE 35

Rainer Heintzmann, 2012 35

The nitty gritty details

correct grating constant wrong grating constant intensity beating, splitting of structures Cave! Hard to distinguish from real data dark bright

slide-36
SLIDE 36

Rainer Heintzmann, 2012 36

The nitty gritty details

Same information: Use overlap and cross correlation

SNR-weighted cross correlation for best results (assume contant variance in Fourer space) typically iterative (3 iterations)

slide-37
SLIDE 37

Rainer Heintzmann, 2012 37

The nitty gritty details

Unknowns: grating constant (precise value) grating orientation local phase global phase

  • rder contrast

illumination intensity sample position (drift)

slide-38
SLIDE 38

Rainer Heintzmann, 2012 38

The nitty gritty details

separated components matrix wrong phases correct phases Global phase: Correlation needs to be real valued

slide-39
SLIDE 39

Rainer Heintzmann, 2012 39

The nitty gritty details

Collaboration: Dithmar, Ach, Best, Cremer (Heidelberg University) Algorithm: Kai Wicker

slide-40
SLIDE 40

Rainer Heintzmann, 2012 40

The nitty gritty details Global phase errors:

destructive "interference" in Fourier space

slide-41
SLIDE 41

Rainer Heintzmann, 2012 41

The nitty gritty details

Determine

  • rder strength from
  • verlap:

Order contrast errors: Part of the matrix M

  • rder 1 pixel value
  • rder 0 pixel value
slide-42
SLIDE 42

Rainer Heintzmann, 2012 42

The nitty gritty details

Speed up: Avoid recalculation of cross correlations (Kai Wicker) pre calculate image correlations: and use unmixing matrix

correlations do not need to be recomputed

slide-43
SLIDE 43

Rainer Heintzmann, 2012 43

fast processing

Doing it faster? Phase of a single image by peaks in weighted autocorrelation

slide-44
SLIDE 44

Rainer Heintzmann, 2012 44

The nitty gritty details

K.Wicker, Opt. Expr. 2013

slide-45
SLIDE 45

Rainer Heintzmann, 2012 45

Single image autocorr. optimization

Collaboration: Dithmar, Ach, Best, Cremer (Heidelberg University) K.Wicker, Opt. Expr. 2013

slide-46
SLIDE 46

Rainer Heintzmann, 2012 46

The Wiener Filter Problem

Wiener Filtering assumes constant noise in image and a known spectrum But

  • noise variance is proportional to signal
  • spectrum is unknown
slide-47
SLIDE 47

Rainer Heintzmann, 2012 47

The Apodization function (goal function)

ideal: no (or small) negative values, small sidelobes, small width

Using the “Lucosz-bound” Stalllinga et al. 2013

slide-48
SLIDE 48

Rainer Heintzmann, 2012 48

fast SIM

slide-49
SLIDE 49

Rainer Heintzmann, 2012 49

fast SIM setup

Foerster, R., et al., Optics Express 22 22, 20663-20677 (2014) Foerster, R., et al., Optics Express 22 22, 20663-20677 (2014)

slide-50
SLIDE 50

Rainer Heintzmann, 2012 50

162 raw frames/s, Orca FLASH 4V2

Hui-Wen Lu-Walter Hui-Wen Lu-Walter

High-Speed SIM: Freely diffusing 100nm beads High-Speed SIM: Freely diffusing 100nm beads

Foerster, R., et al., Optics Express 22 22, 20663-20677 (2014) Foerster, R., et al., Optics Express 22 22, 20663-20677 (2014)

slide-51
SLIDE 51

Rainer Heintzmann, 2012 51

Problem: Rolling shutter readout

http://www.matrix-vision.com/glossar.html

Typical: Two rolling shutters per camera

slide-52
SLIDE 52

Rainer Heintzmann, 2012 52

sCMOS Cameras: rolling shutter

http://www.matrix-vision.com/glossar.html

slide-53
SLIDE 53

Rainer Heintzmann, 2012 53

sCMOS Cameras

sCMOS rolling shutters

http://en.wikipedia.org/wiki/File:CMOS_rolling_shutter_distortion.jpg

slide-54
SLIDE 54

Rainer Heintzmann, 2012 54

Solution: Synchronised partial frames Solution: Synchronised partial frames

Song et al., Measurement Science Technology 27,066401 (2016)

slide-55
SLIDE 55

Rainer Heintzmann, 2012 55

Solution: Synchronised partial frames Solution: Synchronised partial frames

FWHM= 108nm Rate: 714 fps (raw) 79 fps (SIM)

Song et al., Measurement Science Technology 27,066401 (2016)

slide-56
SLIDE 56

Rainer Heintzmann, 2012 56

Overview

  • Introduction: Resolution, Fourier and Abbe
  • Superresolution
  • Structured Illumination
  • Circumventing the limit: Nonlinearity
slide-57
SLIDE 57

Rainer Heintzmann, 2012 57

Non-linearity

  • R. Heintzmann, T.M. Jovin, and C. Cremer., J. Opt. Soc. Am. A,19 (8), 1599-1609

2002

magnitude spatial frequency Obj +2 (k) ~ 20

  • 30 -20

Border of detection OTF

magnitude spatial frequency Obj +1 (k) ~

  • K0

K0

  • 2K0
  • K0

K0 Border of detection OTF Linear Excitation (low intensity) Non-Linear Excitation (high intensity)

  • 3K0

Past

Iem (x) = Obj(x) · Iex (x) Iem (k) = Obj(k) Iex (k)

~ ~ ~

Iem (x) = Obj(x) · f(Iex (x)) Iem (k) = Obj(k) f(Iex (k))

~ ~ ~

slide-58
SLIDE 58

Rainer Heintzmann, 2012 58

Photoswitchable Proteins

IrisFP (Tetrameric)

(Ulrich Nienhaus, Susan Böhme, Elisabeth Ehler)

Widefield Linear SI Nonlinear SI

Data: Enno Oldewurtel

slide-59
SLIDE 59

Rainer Heintzmann, 2012 59

History

2011 TIRF NL-SIM TIRF NL-SIM

  • n biological objects

using saturated switching (Dronpa) Nuclear Pores (Nup98):

slide-60
SLIDE 60

Rainer Heintzmann, 2012 60

Science 28 August 2015: vol. 349 no. 6251 DOI: 10.1126/science.aab3500 Extended-resolution structured illumination imaging of endocytic and cytoskeletal dynamics Li et al. (Betzig lab)

mApple-F-tractin (purple) and the focal adhesion protein mEmerald-paxillin (green) in a U2OS cell (movie S2).

evolution of cortical f-actin in a COS-7 cell at 23°C transfected with Skylan-NS-Lifeact,

slide-61
SLIDE 61

Overview

  • High-res modes: SIM
  • Blind: PSF, illumination estimation
slide-62
SLIDE 62
  • always slightly underdetermined

(like blind source separation)

  • sum of all illumination is assumed constant

(also for 200 speckle patterns)

  • tiny Fourier-space support

Blind deconvolution (illumination)

slide-63
SLIDE 63

Overview

Object distorted Pattern SIM reconstruction blind deconvolution

slide-64
SLIDE 64

5 µm

Blind-SIM: experimental TIRF-SIM data

Widefield TIRF image

Image courtesy Philipp von Olshausen / Alexander Rohrbach, Freiburg Image courtesy Philipp von Olshausen / Alexander Rohrbach, Freiburg

WF image

Aurélie Jost Aurélie Jost

WF deconvolution classical SIM „classical“ SIM blind-SIM

slide-65
SLIDE 65

Blind-SIM on thick samples

Principle of the thick slice deconvolution:

  • 2-beam illumination
  • Single-slice acquisition at z = z0
  • 3D blind-SIM deconvolution using 3D PSF and extended

stack X Y Z=Z0 Single acquired slice

Additional planes

BlindSIM: Aurélie Jost BlindSIM: Aurélie Jost

No contribution to the cost functional

slide-66
SLIDE 66

Blind-SIM on thick samples

BlindSIM: Aurélie Jost BlindSIM: Aurélie Jost

2D blind SIM thick slice blind SIM

slide-67
SLIDE 67

Rainer Heintzmann, 2012 67

Blind-SIM on thick samples

67 Experimental thick samples: WF image WF 2D deconvolution

BlindSIM

Aurélie Jost

BlindSIM

Aurélie Jost

Image courtesy Elena Tolstik Data acquired on the Elyra (3-beam) Image courtesy Elena Tolstik Data acquired on the Elyra (3-beam)

Elyra result 5 µm WF 3D deconvolution

slide-68
SLIDE 68

standard SIM

Blind-SIM on thick samples

3D WF deconv 2D blind-SIM Thick slice blind-SIM Experimental thick samples: Yeast, csiLSFM set-up (SIM-SPIM)

Image Data: Bo-Jui Chang Ernst Stelzer, Frankfurt

Data information Sample: yeast mitochondiral GFP label Excitation: 488 nm Emission: 509 nm Pixel size: 57,6 nm NA: 1,0 (water-imm.) n: 1,33 Grating: 307,2 nm Reconstruction parameters Reconstructed Slices: 8 Scale z PSF: 200 nm Good‘s roughness penalty λ = 0,02 Number of iterations: 30

2 µm

slide-69
SLIDE 69

Blind-SIM on thick samples

Image Data: Bo-Jui Chang Ernst Stelzer, Frankfurt Yeast mitochondria 2 µm Thick slice reconstruction: slice by slice, Aurélie Jost

slide-70
SLIDE 70

Rainer Heintzmann, 2012 70

Summary

Linear fluorescence microscopy methods (structured illumination) can

  • Enhance resolution (2x limit frequency)
  • Increase HF detection

Non-linear methods are unlimited in resolution (NL-SIM, STED)

slide-71
SLIDE 71

Collaborations

  • Research: Ondrej Mandula, Susan Cox, Rolf Beutel,
  • Y. Matsumura
  • Ideas: Anne Sentenac
  • Images: Mats Gustafsson, Alexander Rohrbach,

Ernst Stelzer, Bo-Jui Chang

  • Samples: Christopher Williams, James Moneypenny,

Gareth Jones, Jürgen Rybak, Rolf Beutel, Y. Matsumura

  • Probes: Ullrich Nienhaus, Susan Böhme
  • Airy Scan Slides: Allex Sossic, Uros Krzic, Chris Power

+ DFG, JSMC, KCL, Zeiss

slide-72
SLIDE 72

+ Collaborators, DFG, JSMC, KCL

Acknowledgement

Kai Wicker SIM Ondrej Mandula SIM Walter Müller Raman Ulrich Leischner Light-Sheet Aurélie Jost Deconvolution

slide-73
SLIDE 73

Summary

Many modes of microscopy exist Linear methods yield a factor of 2 Light-sheet microscopy makes cool images Computer-based imaging has great potential