Image Restoration by Deconvolution: Concepts and Applications Chong - - PowerPoint PPT Presentation
Leica - CNIC 1st Practical School in Super-Resolution Microscopy, Centro Nacional de Investigaciones Cardiovasculares Carlos III (CNIC) Image Restoration by Deconvolution: Concepts and Applications Chong Zhang SIMBioSys, Depertment of
Chong Zhang
SIMBioSys, Depertment of Information and Communication Technologies Universitat Pompeu Fabra
15th March, 2016
Leica - CNIC 1st Practical School in Super-Resolution Microscopy, Centro Nacional de Investigaciones Cardiovasculares Carlos III (CNIC)
Deconvolution is a computational technique allowing to partly compensate for the image distortion caused by a microscope. The betterment can be signifjcant both in terms of attenuation of the out of focus light and increase of the spatial resolution. It was fjrst devised at the MIT for seismology (Robinson, Wiener, early 50’), then applied to astronomy and fjnally found its way to 3D optical fmuorescence microscopy (Agard 1984). It should not be seen as a “black box” to enhance image quality since it can introduce artifacts or further degrade low quality images. It is compatible with quantitative measurements (should even improve). It works best for thin (<50 um), optically transparent, fjxed, bright samples. Challenging for live microscopy: short exposure (limit motion blur), objective adapted to medium (limit spherical aberrations).
Courtesy of S. Tosi
Courtesy of P. Bankhead
Courtesy of P. Bankhead
Courtesy of P. Bankhead
Courtesy of S. Tosi
“stamping” the kernel on each pixel of the image the kernel is scaled (multiplied) by the intensity of the central pixel Accumulated (summed) in the output image. Filter kernel
Original image Filtered image Original image Filtered image
Courtesy of S. Tosi
Convolution consists of replacing each point in the original object with its blurred image in all dimensions and summing together overlapping contributions from adjacent points to generate the resulting image
A 3D kernel is a stack holding the fjlter coefficients
Courtesy of S. Tosi
Convolution consists of replacing each point in the original object with its blurred image in all dimensions and summing together overlapping contributions from adjacent points to generate the resulting image
If we take the FT of the equation, the is replaced by multiplication, thus image restoration might be achievable by: dividing the FT of the image by the FT of the kernel and then taking the inverse Fourier transform.
Spatial Domain Fourier Domain
Courtesy of S. Tosi
Widefjeld PSF
Cannell 2006 Bankhead 2014
A record of how much the image created by a microscope spreads/blurs an object of a single point (thus determines the way in which images of objects blur into each other in the fjnal image).
Courtesy of S. Tosi Bankhead 2014
Measured PSF (experimental PSF) Theoretical PSF
Parton 2006
Measured PSF: an image resulting from a single small spherical fmuorescent bead (smaller than the optical resolution, thus forms effectively a point source of light) Theoretical PSF: generated by a computer program, after input of values describing the optical system – the magnifjcation, NA of the objective, the illuminating and emitted wavelengths, and the refractive index of the objective lens immersion medium. It gives an indication of the best possible resolution for a given objective but these limits are not achievable. Real PSFs are typically >20% bigger than calculated versions.
Gated ¡ STED ¡ <50 ¡ 560 ¡ 70 ¡ /GSD 70 ¡ 3D ¡ STED ¡ <130 ¡ <130 ¡
Courtesy of N. Garin
λ : fmuorophore emission wavelength NA : objective numerical aperture n : refractive index of the objective lens immersion medium NA can never exceed n, which itself has fjxed values (e.g. 1.0 for air, 1.33 for water,
NA = n sinθ
Notes:
into account the effects of: brightness, pixel size, noise
immersion refractive index is high Rayleigh criterion
Widefjeld PSF: 100nm beads, excitation 520nm, emission 617nm
Parton 2006 Bankhead 2014
Spatial resolution: radius of the smallest point source in the image, i.e. fjrst minimum of Airy disk
Bankhead 2014
Airy disk ¡
Asymmetry radial (x-y): commonly misalignment of optical components about the z-axis, either as tilt or decentration along the optical axis (z-axis): commonly due to spherical aberration, which may result from refractive index mismatches between the objective, immersion medium, and sample or tube length/coverslip thickness errors.
Notes:
avoid spherical aberration
larger PSF than a highly irregular one.
Pawley 2006
The deconvolution fjlter F should “undo” the effect of the microscope PSF H by processing the sampled image R, ideally D = S.
Object (sample) Microscope PSF Noise Digital fjlter Deconvolved image
Courtesy of S. Tosi
= R = D
The deconvolution fjlter F should “undo” the effect of the microscope PSF H by processing the sampled image R, ideally D = S.
Object (sample) Microscope PSF Noise Digital fjlter Deconvolved image
Courtesy of S. Tosi
= R = D
Linear inverse Constrained iterative Regularized inverse Deblurring subtractive
Nearest neighbors No neighbors Richardson-Lucy Adaptive blind Fast Software available 2D only No need PSF Not for quantitative intensity measures Tikhonov-Miller fjlter Wiener fjlter Jansson van Cittert Fast Software available 3D Needs PSF Do not count for noise Not for super resolution Trade-off between sharpness & noise Slow Good 3D results 3D Needs PSF Also for super resolution 3D Estimates PSF May not be always quantitative Use noise models
Lifting the Fog: Image Restoration by Deconvolution, Parton 2006
Courtesy of S. Tosi
A very simple model for the PSF H (Gaussian std = 1 pixel) 1 ¡ 4,1·√10-‑8 ¡ 1 ¡ 2,4·√107 ¡ H power spectrum (log display)
H-1 power spectrum (log display)
But in practice… Noise enhancement ruins our efforts! For example, assuming H known, F linear (convolution) and no noise (N = 0) leads to:
H +N H-1
Original image S H is a Gaussian with std = 2 pixels Noise std = 10-4 Noise std = 10-12 S after convolution by H No noise
Courtesy of S. Tosi
Courtesy of S. Tosi
A very simple model for the PSF H (Gaussian std = 1 pixel) 1 ¡ 4,1·√10-‑8 ¡ H power spectrum (log display)
(H-1)reg (1% clipping) power spectrum (log display)
1 ¡ 2,4·√107 ¡ 1 ¡ 100 ¡
Courtesy of S. Tosi
H +N (H-1)trunc
Original image S S after convolution by H Noise std = 10-4
Courtesy of S. Tosi
Minimizing the expectation of ||E|| over all possible noise realizations assuming a white Gaussian noise: Coming back to: Bands free of noise: |N(u,v)| = 0 F(u,v) = H(u,v)-1 (inverse fjlter) Strong noise bands: |N(u,v)| ∞ F(u,v) 0 (cut-off) Intermediate bands: best trade-off
Wiener fjlter
As ¡noise ¡at ¡certain ¡frequencies ¡increases, ¡the ¡signal-‑to-‑noise ¡ra-o ¡drops, ¡so ¡F ¡also ¡
signal-to-noise ratio.
Courtesy of S. Tosi
Regularized inverse fjlter result Noise std = 10-4 Wiener fjlter result Noise std = 10-4
Courtesy of S. Tosi
The best deconvolution algorithms for 3D microscopy are typically non-linear. Principle of Maximum A Priori algorithms (MAP): The second equality comes from Bayes theorem. In the optimization S is usually constrained to be positive and somehow spatially smooth (TV regularization term) Pr(S). The statistical distribution of the noise has to be known to derive the maximum likelihood term Pr(R|S) the algorithm is tuned to a particular noise (e.g. Poisson or Gaussian noise). There is usually no known analytical solution to the problem, the algorithms proceeds by iterations (candidate Si at iteration i) to refjne the estimate of the data at each iteration. The Richardson-Lucy algorithm is among the most well known MAP deconvolution algorithm. Some algorithms also simultaneously estimate the PSF from the sampled image (blind deconvolution).
Quantitative intensity measurements, e.g. intensity ratio: controls, also report on un-deconvolved data for comparison Quantitative positional or structural analysis, e.g. centroid, tracking, volume analysis, (object based) colocalisation, etc: relatively less critical the choice For all analysis:
Deconvolution process comparable between datasets Compare with control/un-deconvolved data Understand algorithm used and choose most suitable Report possible artifacts and confjrm it, if possible
Diffraction PSF 3D PSF generator Parallel iterative deconvolution (http://imagej.net/Parallel_Iterative_Deconvolution): 4 deconvolution algorithms Parallel spectral deconvolution (http://imagej.net/Parallel_Spectral_Deconvolution): not iterative, no constraint e.g. non- negativity Iterative Deconvolve 3D (http://imagej.net/Iterative_Deconvolve_3D): non-negative, iterative, similar to WPL algorithm. The execution is way slower on modern (multicore) computers but the memory requirement is less stringent DeconvolutionLab (http://bigwww.epfm.ch/algorithms/deconvolutionlab/): different algorithms including a custom version
Squassh (http://imagej.net/Squassh): joint deconvolution-segmentation procedure
Commercial software
Fiji plugins
Original AutoquantX (30IT, bead PSF) Huygens (50IT, bead distilled PSF) PID (WPL, Wiener Gamma 0.1, 50IT, bead PSF) Original PID (WPL, 50IT, true PSF) AutoquantX (30IT, true PSF) Huygens (50IT, distilled true PSF) Original AutoquantX (30IT, bead PSF) Huygens (50IT, bead distilled PSF) PID (WPL, 50IT, bead PSF)
+ ¡Microscope ¡specific ¡ PSF ¡ ¡ + ¡depth-‑varying ¡PSF ¡ + ¡supports ¡spinning ¡ disk ¡M. ¡ + ¡Visually ¡appealing ¡ results ¡
source ¡ + ¡Free ¡& ¡Open ¡source ¡ & ¡full ¡control ¡ ¡ + ¡Reasonably ¡fast ¡ + ¡Support ¡for ¡ spatially-‑variant ¡ PSF ¡(un-‑tested) ¡
crispy ¡ + ¡Fast ¡convergence ¡ ¡ + ¡Robust ¡algorithms ¡ + ¡Very ¡simple ¡to ¡use ¡ + ¡Visually ¡appealing ¡ results ¡ + ¡2D ¡mode ¡for ¡thin ¡ samples ¡
source ¡
Courtesy of S. Tosi
Reviews (overviews): 1. Waters, Accuracy and precision in quantitative fmuorescence microscopy, JCB 2009 2. Parton et al., Lifting the fog: Image restoration by deconvolution, Cell biology 2006 3. Pawley, Chapter 25: “Image enhancement by deconvolution” , Handbook of biological confocal microscopy, 2006 4. McNally et al., Three-Dimensional Imaging by Deconvolution Microscopy, Methods 1999 Technical articles: 1. Zanella et al., Towards real-time image deconvolution: application to confocal and STED microscopy, Scientifjc Reports 2013 2. Bertero et al., Image deconvolution, Proc. NATO A.S.I. 2004 3. Thiébaut, Introduction to image reconstruction and inverse problems, Proc. NATO A.S.I. 2002 Websites: 1. Olympus microscopy center (overview): http://www.olympusmicro.com/primer/digitalimaging/deconvolution/deconvolutionhome.html 2. Textbook: http://blogs.qub.ac.uk/ccbg/fmuorescence-image-analysis-intro 3. http://fjji.sc/Deconvolution_tips
Slides courtesy:
Sébastien Tosi IRB Barcelona
Pete Bankhead Queen’s University Belfast