S5326 Recovering Structural Information about Nanoparticle Systems - - PowerPoint PPT Presentation

S5326 Recovering Structural Information about Nanoparticle Systems

Abhinav Sarje

Computational Research Division Lawrence Berkeley National Laboratory

03.19.15 GPU Technology Conference 2015 San Jose, CA

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Nanoparticle Systems

• Materials (natural or artificial) made up of nanoparticles.
• Sizes ranging from 1 nanometer to 1000s nanometers.
• Wide variety of applications in optical, electronic and biomedical fields. E.g.:
• Inorganic nanomaterials in optoelectronics.
• Organic material based nano-devices such as Organic Photovoltaics (OPVs), OLEDs.
• Chemical catalysts, drug design and discovery, biological process dynamics.

Importance of structural information:

• Nanomaterials exhibit shape and size-dependent properties, unlike bulk materials which have

constant physical properties regardless of size.

• Nanoparticle characterization is necessary to establish understanding and control of material

synthesis and applications.

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Nanoparticle Systems

• Materials (natural or artificial) made up of nanoparticles.
• Sizes ranging from 1 nanometer to 1000s nanometers.
• Wide variety of applications in optical, electronic and biomedical fields. E.g.:
• Inorganic nanomaterials in optoelectronics.
• Organic material based nano-devices such as Organic Photovoltaics (OPVs), OLEDs.
• Chemical catalysts, drug design and discovery, biological process dynamics.

Importance of structural information:

• Nanomaterials exhibit shape and size-dependent properties, unlike bulk materials which have

constant physical properties regardless of size.

• Nanoparticle characterization is necessary to establish understanding and control of material

synthesis and applications.

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Measuring Structural Information at Nano-scale

• Electron microscopy (TEM, SEM),
• atomic force microscopy (AFM),
• X-ray photoelectron spectroscopy (XPS),
• X-ray diffraction (XRD),
• X-ray scattering,
• and more.

X-ray scattering:

• Determine the size distribution profile of nanoparticles in suspension or polymers in solution.
• Probe the behavior of complex fluids such as polymer solutions.
• Probe structures of non-crystalline thin-film materials.

Examples:

• Small-Angle X-ray Scattering (SAXS)
• Grazing-Incidence SAXS (GISAXS)

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Measuring Structural Information at Nano-scale

• Electron microscopy (TEM, SEM),
• atomic force microscopy (AFM),
• X-ray photoelectron spectroscopy (XPS),
• X-ray diffraction (XRD),
• X-ray scattering,
• and more.

X-ray scattering:

• Determine the size distribution profile of nanoparticles in suspension or polymers in solution.
• Probe the behavior of complex fluids such as polymer solutions.
• Probe structures of non-crystalline thin-film materials.

Examples:

• Small-Angle X-ray Scattering (SAXS)
• Grazing-Incidence SAXS (GISAXS)

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

X-ray Scattering at Synchrotrons

asarje@lbl.gov Lawrence Berkeley National Laboratory graphic: courtesy of A. Meyer, www.gisaxs.de

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

X-Ray Scattering: Examples

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

X-Ray Scattering: Complex Examples

Gratings Organic Photovoltaics Real Sample Model Scattering Pattern

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Computational Problems in Structure Recovery: Inverse Modeling

asarje@lbl.gov Lawrence Berkeley National Laboratory

Start Initial guess

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Computational Problems in Structure Recovery: Inverse Modeling

asarje@lbl.gov Lawrence Berkeley National Laboratory

Start Initial guess forward simulation

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Computational Problems in Structure Recovery: Inverse Modeling

asarje@lbl.gov Lawrence Berkeley National Laboratory

Start Initial guess forward simulation compute error w.r.t. experimental data

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Computational Problems in Structure Recovery: Inverse Modeling

asarje@lbl.gov Lawrence Berkeley National Laboratory

Start Initial guess forward simulation compute error w.r.t. experimental data tune model parameters

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Computational Problems in Structure Recovery: Inverse Modeling

asarje@lbl.gov Lawrence Berkeley National Laboratory

Start Initial guess forward simulation compute error w.r.t. experimental data tune model parameters

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Computational Problems in Structure Recovery: Inverse Modeling

asarje@lbl.gov Lawrence Berkeley National Laboratory

Start Initial guess forward simulation compute error w.r.t. experimental data tune model parameters End

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Need for High-Performance Computing

Data generation and analysis gap:

• High measurement rates of current state-of-the-art light beam detectors.
• Wait for days for analyzing data with previous softwares.
• Extremely inefficient utilization of facilities due to mismatch.
• Example: 100 MB raw data per second. Up to 12 TB per week.

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Need for High-Performance Computing

High computational and accuracy requirements:

• Errors are proportional to the resolutions of various computational discretization.
• Higher resolutions require higher computational power.
• Example:
• O(107) to O(1015) kernel computations for one simulation.
• O(102) experiments per material sample.
• O(10) to O(103) forward simulations for inverse modeling per scattering pattern.

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Need for High-Performance Computing

Science Gap:

• Beam-line scientists lack access to high-performance algorithms and codes.
• In-house developed codes limited in compute capabilities and performance.
• Also, they are extremely slow – wait for days and weeks to obtain basic results.

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Forward Simulations: Computing Scattered Light Intensities

Given:

1 a sample structure model, and 2 experimental configuration,

simulate scattering patterns. Based on Distorted Wave Born Approximation (DWBA) theory.

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Inverse Modeling

Forward simulation kernel: computing the scattered light intensities. E.g.

• FFT computations (SAXS)
• Complex form factor and structure factor computations (GISAXS)

Various inverse modeling algorithms:

• Reverse Monte-Carlo simulations for SAXS.
• Sophisticated optimization algorithms for GISAXS.
• Gradient based: LMVM (Limited-Memory Variable-Metric.)
• Derivative-free trust region-based: POUNDerS.
• Stochastic: Particle Swarm Optmization.

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Inverse Modeling

Forward simulation kernel: computing the scattered light intensities. E.g.

• FFT computations (SAXS)
• Complex form factor and structure factor computations (GISAXS)

Various inverse modeling algorithms:

• Reverse Monte-Carlo simulations for SAXS.
• Sophisticated optimization algorithms for GISAXS.
• Gradient based: LMVM (Limited-Memory Variable-Metric.)
• Derivative-free trust region-based: POUNDerS.
• Stochastic: Particle Swarm Optmization.

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Reverse Monte Carlo Simulations

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Reverse Monte Carlo Simulations: Validation

Actual Models

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Reverse Monte Carlo Simulations: Validation

Actual Models Initial Models

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Reverse Monte Carlo Simulations: Validation

Actual Models Initial Models Recovered Models

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Reverse Monte Carlo Simulations: Strong and Weak Scaling

Titan

103 104 105 106 107 108 1 2 4 8 16 32 64 128 256 512

Time [ms] Number of Nodes (GPUs)

t=1024, s=2048 t=1024, s=512 t=128, s=512 104 105 106 1 2 4 8 16 32 64 128 256 512 1024

Time [ms] Number of Nodes (GPUs)

t=2p, s=2048 t=p, s=2048 t=2p, s=512 t=p, s=512

Hopper

104 105 106 107 108 109 1010 24 48 96 192 384 768 1536 3072 6144 12288

Time [ms] Number of Cores

t=1024, s=2048 t=1024, s=512 t=128, s=2048 t=128, s=512 104 105 106 107 24 48 96 192 384 768 1536 3072 6144 12288

Time [ms] Number of Cores

t=2p, s=2048 t=p, s=2048 t=p, s=512 t=p/2, s=512

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Limited-Memory Variable-Metric and POUNDerS

• Methods from the optimization package TAO.
• LMVM is a gradient-based method.
• POUNDerS is a derivative-free trust-region-based method.

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Limited-Memory Variable-Metric and POUNDerS: Two Parameter Case

A Single Cylindrical Nanoparticle

16 18 20 22 24 x1 16 18 20 22 24 x2 20 40 60 80 100 120 140 160 180 16 18 20 22 24 x1 16 18 20 22 24 x2 15 30 45 60 75 90 105 16 18 20 22 24 x1 16 18 20 22 24 x2 3 6 9 12 15 18 21 24 27

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Limited-Memory Variable-Metric and POUNDerS: Two Parameter Case

A Single Cylindrical Nanoparticle X-Ray Scattering Pattern

16 18 20 22 24 x1 16 18 20 22 24 x2 20 40 60 80 100 120 140 160 180 16 18 20 22 24 x1 16 18 20 22 24 x2 15 30 45 60 75 90 105 16 18 20 22 24 x1 16 18 20 22 24 x2 3 6 9 12 15 18 21 24 27

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Limited-Memory Variable-Metric and POUNDerS: Two Parameter Case

A Single Cylindrical Nanoparticle X-Ray Scattering Pattern

16 18 20 22 24 x1 16 18 20 22 24 x2 20 40 60 80 100 120 140 160 180

Objective Function Map

16 18 20 22 24 x1 16 18 20 22 24 x2 15 30 45 60 75 90 105 16 18 20 22 24 x1 16 18 20 22 24 x2 3 6 9 12 15 18 21 24 27

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Limited-Memory Variable-Metric and POUNDerS: Two Parameter Case

A Single Cylindrical Nanoparticle X-Ray Scattering Pattern

16 18 20 22 24 x1 16 18 20 22 24 x2 20 40 60 80 100 120 140 160 180

Objective Function Map

16 18 20 22 24 x1 16 18 20 22 24 x2 15 30 45 60 75 90 105

LMVM Convergence Map

16 18 20 22 24 x1 16 18 20 22 24 x2 3 6 9 12 15 18 21 24 27

POUNDerS Convergence Map

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Limited-Memory Variable-Metric and POUNDerS: Six Parameter Case

Pyramidal Nanoparticles forming a Lattice

36 37 38 39 40 41 42 43 44 x1 46 48 50 52 54 x3 6.0 7.5 9.0 10.5 12.0 13.5 15.0 16.5 18.0 36 37 38 39 40 41 42 43 44 x1 46 48 50 52 54 x3 12 14 16 18 20 22 24 26 28

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Limited-Memory Variable-Metric and POUNDerS: Six Parameter Case

Pyramidal Nanoparticles forming a Lattice X-Ray Scattering Pattern

36 37 38 39 40 41 42 43 44 x1 46 48 50 52 54 x3 6.0 7.5 9.0 10.5 12.0 13.5 15.0 16.5 18.0 36 37 38 39 40 41 42 43 44 x1 46 48 50 52 54 x3 12 14 16 18 20 22 24 26 28

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Limited-Memory Variable-Metric and POUNDerS: Six Parameter Case

Pyramidal Nanoparticles forming a Lattice X-Ray Scattering Pattern

36 37 38 39 40 41 42 43 44 x1 46 48 50 52 54 x3 6.0 7.5 9.0 10.5 12.0 13.5 15.0 16.5 18.0

Objective Function Map

36 37 38 39 40 41 42 43 44 x1 46 48 50 52 54 x3 12 14 16 18 20 22 24 26 28

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Limited-Memory Variable-Metric and POUNDerS: Six Parameter Case

Pyramidal Nanoparticles forming a Lattice X-Ray Scattering Pattern

36 37 38 39 40 41 42 43 44 x1 46 48 50 52 54 x3 6.0 7.5 9.0 10.5 12.0 13.5 15.0 16.5 18.0

Objective Function Map LMVM does not converge

36 37 38 39 40 41 42 43 44 x1 46 48 50 52 54 x3 12 14 16 18 20 22 24 26 28

POUNDerS Convergence Map

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Particle Swarm Optimization

• Stochastic method.
• Multiple agents, “particle swarm”, search for optimal points in the parameter space.
• Agent velocities influenced by history of traveled paths.

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Particle Swarm Optimization

• Stochastic method.
• Multiple agents, “particle swarm”, search for optimal points in the parameter space.
• Agent velocities influenced by history of traveled paths.

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Particle Swarm Optimization: Fitting X-Ray Scattering Data

2 4 6 8 10 12 14 16 18 20 Number of agents 1 4 9 16 25 36 49 64 81 100 Parameter space volume 80 160 240 320 400 480 560 640 Function evaluations

Fitting 2 Parameters

2 4 6 8 10 12 14 16 18 20 Number of agents 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Parameter space volume 100 200 300 400 500 600 700 800 900 Function evaluations

Fitting 6 Parameters

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Particle Swarm Optimization: Performance

4 16 36 64 100 144 Search Space Volume ([x1 ]x[x2 ]) 5 10 15 20 25 30 Iterations for Convergence

15 agents 20 agents

Convergence w.r.t. Search Space Volume

4 16 64 256 Number of GPUs 102 103 104 Time [s]

sample 1 sample 2

Strong Scaling on Titan

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Particle Swarm Optimization: Agents vs. Generations

10 20 30 40 50 Generation 10-12 10-11 10-10 10-9 10-8 10-7 10-6 10-5 10-4 10-3 10-2 10-1 100 Relative Error

5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100

10 20 30 40 Generation 20 40 60 80 100

• Num. Agents

10-11 10-10 10-9 10-8 10-7 10-6 10-5 10-4 10-3 10-2 10-1 Relative Error

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

An Ongoing Work

We saw that:

• Derivative-based methods converge only for simple cases.
• Trust-region-based methods are very sensitive to initial guess.
• PSO is robust, nearly always converging, but expensive.
• Need better optimization algorithms.

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

An Ongoing Work

We saw that:

• Derivative-based methods converge only for simple cases.
• Trust-region-based methods are very sensitive to initial guess.
• PSO is robust, nearly always converging, but expensive.
• Need better optimization algorithms.

Near future:

• GPUs have brought data analysis time from days and weeks to just minutes and seconds.
• Opening gates to much more sophisticated analyses.
• We are applying Deep Learning for feature and structural classification to generate initial

models to fit.

• Our codes are already being used at various synchrotrons world-wide.

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Our Current Team

• Alexander Hexemer, Advanced Light Source, Berkeley Lab.
• Dinesh Kumar, Advanced Light Source, Berkeley Lab.
• Xiaoye S. Li, Computational Research Division, Berkeley Lab.
• Abhinav Sarje, Computational Research Division, Berkeley Lab.
• Singanallur Venkatakrishnan, Advanced Light Source, Berkeley Lab.

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Our Current Team

• Alexander Hexemer, Advanced Light Source, Berkeley Lab.
• Dinesh Kumar, Advanced Light Source, Berkeley Lab.
• Xiaoye S. Li, Computational Research Division, Berkeley Lab.
• Abhinav Sarje, Computational Research Division, Berkeley Lab.
• Singanallur Venkatakrishnan, Advanced Light Source, Berkeley Lab.

And we are open for collaborations ...

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Acknowledgments

• Thanks to NVIDIA for donating several GPU cards to make this possible.
• Supported by the Office of Science of the U.S. Department of Energy under Contract No.

DE-AC02-05CH11231.

• Also supported by DoE Early Career Research grant awarded to Alexander Hexemer.
• Used resources of the National Energy Research Scientific Computing Center, which is

supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231.

• Used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National

Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725.

asarje@lbl.gov Lawrence Berkeley National Laboratory

Introduction Motivation X-Ray Scattering Inverse Modeling Reverse Monte-Carlo Simulations LMVM & POUNDerS PSO Conclusions

Thank you!

Contact: asarje@lbl.gov Code: http://portal.nersc.gov/project/als/hipgisaxs

asarje@lbl.gov Lawrence Berkeley National Laboratory