Image Analysis and Classification in Nanoscale Scanning Tunneling - - PowerPoint PPT Presentation

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Image Analysis and Classification in Nanoscale Scanning Tunneling Microscopy

Nen Huynh, Tawny Lim, Jonathan Siegel Konstantin Dragomiretskiy, Travis Meyer, Joseph Woodworth

Mentors: Dr. Christoph Brune

• Prof. Andrea Bertozzi

California NanoSystems Institute:

• Dr. Shelley Claridge

Miles Silverman

• Prof. Paul Weiss

UCLA REU Final Presentation, August 8, 2012

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Outline

1 Introduction 2 Fourier Transform 3 Preprocessing 4 Segmentation 5 Nonlocal Methods 6 Conclusion

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

What is STM?

Scanning Tunneling Microscopy (STM) is a type of Scanning Probe Microscopy. STM generates images based on changes in distance between tip and sample, measured by the current of electrons running from the tip through the sample.

Figure: STM Tip

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Scanning Tunneling Microscope Setup

Figure: STM Microscope

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

What is our Goal?

Use image analysis to extract patterns in data for peptide identification.

Figure: Image and Underlying Beta Sheet

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

What types of images are we working with?

Figure: Topography and Polarizability Images Subtracting average of image from original can reduce illumination and enhance contrast between light and dark.

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

What types of images are we working with?

Figure: Topography and Polarizability Images Subtracting average of image from original can reduce illumination and enhance contrast between light and dark.

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

What types of images are we working with?

Figure: Topography and Polarizability Images Subtracting average of image from original can reduce illumination and enhance contrast between light and dark.

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

What types of images are we working with?

Figure: Topography and Polarizability Images Subtracting average of image from original can reduce illumination and enhance contrast between light and dark.

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Decomposition

First, we must decompose our images into structure and texture. Structure: Piecewise constant approximation to image Texture: Regular periodic patterns TV-min:

minu

• Ω(u−f )2dx+λ
• Ω |∇u|dx

G-Norm

minu,v

• Ω(f −(u+v))2dx+λ1
• Ω |∇u|+λ2
• Ω vG dx

Figure: Original Image = Structure + Texture + Noise

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Prior Fourier Transform in Action

Prokhorov et al. use the Fourier transform to obtain the orientation and separation between molecular structures on AFM images.

[Prokhorov, et al. Langmuir, 2011], [Stabel, et al. J. Phys. Chem., 1995]

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Discrete Fourier Transform

We advance on the work of Prokhorov et al. by performing the Fourier transform on the texture patch. Xk =

N−1

• n=0

xn · e−2πi k

N n

(a) Selected

Patch

(b) Patch

Close-up

(c) DFT: Top

View

(d) DFT: Side View

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Discrete Fourier Transform

We can now extract the angles and periods of the patch.

(a) Image

Patch

(b) DFT: Top

View

Angles Periods 61.6992 5.0119 65.5560 6.1243 60.2551 9.1786 68.1986 13.7415 62.2415 3.4465 64.7989 3.9386

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Segmentation

Images are divided into regions for further analysis. Segmentation-related techniques: Structure Tensor Principle Component Analysis on Histograms Chan-Vese Model Spectral Clustering

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Structure Tensor

The structure tensor gives us orientation information by using the gradient. Vector-valued structure tensor: Jρ = Kρ ∗ n

i=1 ∇fi∇f T i

Coherency: cw = ( λ1−λ2

λ1+λ2 )2

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Structure Tensor

The structure tensor gives us orientation information by using the gradient. Vector-valued structure tensor: Jρ = Kρ ∗ n

i=1 ∇fi∇f T i

Coherency: cw = ( λ1−λ2

λ1+λ2 )2

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Structure Tensor

The structure tensor gives us orientation information by using the gradient. Vector-valued structure tensor: Jρ = Kρ ∗ n

i=1 ∇fi∇f T i

Coherency: cw = ( λ1−λ2

λ1+λ2 )2

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Structure Tensor

The structure tensor shows there are four regions of different

• rientations in the texture image.

(a) Texture (b) Structure Tensor on Texture

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Principle Component Analysis (PCA) on Histogram Transforms

Using PCA on the histogram of structure tensor images reduces the significance of the low coherency (gray) areas.

Figure: Taking the patch histograms.

For each patch, we find the histogram and represent the histogram values as a vector hi.

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

PCA on Histograms

Next, apply PCA on the set of histogram vectors to find the direction v1 with the most variance.

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

PCA on Histograms

Vector v1 allows transformation of structure tensor image. Projecting patch histogram onto v1 generates scalar value.

Figure: Projecting a patch’s histogram onto vector v1

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

PCA on Histograms

The PCA Histogram transform creates a new image better suited for intensity-based segmentation.

(a) Image (b) PCA on Structure

Tensor Histogram

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Segmentation: Chan-Vese Model

Segmentation using the Chan-Vese model: The algorithm divides image into regions by fitting a constant to each of the regions.

(a) Image (b) Level Set

Function, φ

E(φ,c1,c2) = λ1

• {φ<0}(I(x)−c1)2 dx

+ λ2

• {φ>0}(I(x)−c2)2 dx+µLength({φ=0})

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Chan-Vese (CV) Model

The optimal segmentation of an image comes from disjoint sets {φ > 0} and {φ < 0} where φ is a level set function minimizing: E(φ, c1, c2) = λ1

• {φ<0}

(I(x) − c1)2 dx + λ2

• {φ>0}

(I(x) − c2)2 dx + µLength({φ = 0}) and c1, c2 are the fitting constants.

(a) (b) (c) Figure: (a-b) Level Set φ. Black line is the zero level set (contour). (c) Image and segmentation contour in red.

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Chan-Vese on PCA

(a) Image (b) PCA on Structure

Tensor Histogram

(c) Segmentation of

PCA Image

Figure: Example of Chan-Vese on PCA Histogram Transform of image.

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Segmentation: Chan-Vese Model

Example for consideration:

(a) (b) (c) (d) Figure: (a) Image (b) PCA Hist on texture of image (c) Level Set Function (d) Multiple region segmentation

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Graph-based Methods

Segmentation of Graph

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Graph-based Methods

Segmentation of Graph Orientation Test Image Spectral Clustering Result

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Non-local Laplacian

Continuous Laplacian △u :=

N

• i=1

∂2u ∂x2

i

5-point finite difference approximation to the Laplacian △u(x, y) = u(x − h, y) + u(x, y − h) − 4u(x, y) + u(x + h, y) + u(x, y + h) h2 +O(h2) Graph Laplacian for a generalized similarity function w((x1, y1), (x2, y2)) △wu((xi, yi)) :=

• (xj ,yj )∈Ω

(u((xj, yj)) − u((xi, yi))) · w((xi, yi), (xj, yj))

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Metrics

Some metrics we used: L2 distance Wasserstein distance

(a) Image (b) L2 distance (c) Wasserstein

distance

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Mean-Diffused Orientation

Coherency measures orientation homogeneity in a pixel’s neighborhood. Mean-Diffused Orientation is our method for homogenizing the

• utput of the structure tensor. We choose a patch Pq around each

pixel q and then average the angles given by the structure tensor in each patch as follows a∗

q = arg(

• p∈Pq

coh(p)>t

e2iap) Here a∗

q is the new angle associated to q and t is a threshhold

value for the coherency.

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Mean-Diffused Orientation

(a) (b) Figure: Mean-Diffused Orientation works better on images with high coherency.

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

PCA vs. Mean-Diffused Orientation

(a) PCA Histogram

Transform

(b) Mean-Diffused

Orientation

Figure: We will examine PCA and MDO on these two images.

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

PCA vs. Mean-Diffused Orientation

(a) PCA Histogram

Transform

(b) Mean-Diffused

Orientation

Figure: For this sample image, Mean-Diffused Orientation does not work as well because of the low coherency.

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

PCA vs. Mean-Diffused Orientation

(a) PCA Histogram

Transform

(b) Mean-Diffused

Orientation

Figure: For this sample image, PCA does not work as well because of the holes in the structure.

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Spectral Clustering

Spectral Clustering segments an image by taking a similarity graph and creating a graph Laplacian from it. After creating a matrix from the eigenvectors of the graph Laplacian, it clusters the matrix using K-means.

(a) (b) (c) (d) Figure: (a) Image (b) Coherency-based orientation inpainting (Gray removal) (c) Binary segmentation (d) Multiple region segmentation

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Classification

After segmentation, the resultant regions can be quantified for further analysis.

Figure: Analysis of angle orientations using rose plots

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Classification

Some Defining Features: Pattern Frequency Amplitude Fourier Transform Orientation

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Conclusion

Decompose the STM images into structure and texture. Segment structure images using Chan-Vese model. Preprocess texture image using structure tensor and PCA Histogram Transform to create a more suitable image for Chan-Vese segmentation. Propose Spectral Clustering method using Mean-Diffused Orientation. Extract orientation and period.

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Conclusion

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

Acknowledgements

Thanks for your attention! CNSI’s:

• Dr. Shelley Claridge, Miles Silverman, Dr. Paul Weiss

UCLA Math: Joseph Woodworth, Konstantin Dragomiretskiy Travis Meyer, Dr. Christoph Brune Ekaterina Merkurjev, Dr. J´ erˆ

• me Gilles, Prof. Andrea Bertozzi.

Introduction Fourier Transform Preprocessing Segmentation Nonlocal Methods Conclusion

References

Brox, T., J.Weickert, B. Burgeth, and P. Mrazek. “Nonlinear Structure Tensors.” Image and Vision Computing 24.1 (2006): 41-55. Print. Chan, Tony F., and Luminita A. Vese. “Active Contours Without Edges.” IEEE Transactions on Image Processing 10.2 (2001): 266-77. Print. Prokhorov, V. V., D. V. Klinov, A. A. Chinarev, A. B. Tuzikov, I. V. Gorokhova, and N. V. Bovin. ”High-Resolution Atomic Force Microscopy Study of Hexaglycylamide Epitaxial Structures on Graphite.” Langmuir 27 (2011): 5879-890. Print. Vese, Luminita A., and Stanley J. Osher. “Modeling Textures with Total Variation Minimization and Oscillating Patterns in Image Processing.” Journal of Scientific Computing 19.1-3 (2003): 553-72. Print.