[PPT] - Projector Augmented Wave based Kohn Sham Density Functional Theory PowerPoint Presentation

SLIDE 1

Projector Augmented Wave‐based Kohn‐Sham Density Functional Theory in OpenAtom with N2 log N scaling

Qi Li2, Eric Bohm2, Raghavendra Kanakagiri3 & Glenn J. Martyna1

1. Pimpernel Science, Software and Information Technology, USA
2. Computer Science, University of Illinois, Champaign‐Urbana, USA
3. Computer Science, IIT Tirupati, Tirupati, India

Funding: NSF SI2

SLIDE 2

Goal: The study of complex heterogeneous systems to discern emergent and new physics and create impact

Light in

Hardware Software Methods Scientific Insight From Modeling

Physics‐based solutions for complex systems

Approach:

SLIDE 3

OpenAtom Concept: Statistical Sampling

f

Complex Environments is Key to Understanding many Physical Systems.

OpenAtom: Pimpernel (Martyna), UIUC (Kale) and Yale (Ismail‐Beigi) collaborate to build the Electronic Ground and Excited State parallel software and methods including classical and quantum nuclear motion capabilities to realize this vision.

Biological function : enabled by fluctuations

in both the environment and the biomolecules.

Pollutant detection:

requires sampling complex aqueous systems and then exporting the results to a GW/GW‐BSE app for computation of spectra.

Understanding chemical reactions in dense arrays:

requires non‐trivial sampling of the full system due to complex many‐body reaction paths.

SLIDE 4

Key Project Accomplishments thus Far:

54 Si at 12 Ry 16 Si at 12 Ry6

Reduced order GW software soon to be released!

Electronic Excited States (charm++ parallelization)

1. High Parallel Scaling for O(N4) GW
2. O(N3) GW method based on a shredded propagator, complex time formalism

Electronic Ground State (charm++ parallelization):

1. High Parallel Scaling allows study of hydrogen storage in MOF’s via Path Integral CPAIMD.
2. Exact Exchange N2 N1/3 log N (for metals & insulators): 10x speed 32 waters! (SIAM in prep).
3. Projector Augmented Wave method in N2 log N (new results!).

High parallel scaling enabled by O(N2 log N) methods & charm++

SLIDE 5

Kohn‐Sham Density Functional Theory (KS‐DFT):

A workhorse of computational science.

KS‐DFT: Ground state electronic energy expressed exactly as the minimum of a

functional of the zero temperature, 1‐body density written in terms of 𝜍 𝒔, 𝒔′ 𝜔𝒔𝜔

∗𝒔′

,

𝑜 𝒔 𝜍 𝒔, 𝒔 , 𝑂 # electrons/2

an orthonormal set of KS states, 𝜔 | 𝜔 2𝜀.

Walter Kohn, Nobel Chemistry 1998

KS Density Functional: Sum of the kinetic energy of non‐interacting electrons,

Hartree energy, electron‐ion/external energy and an unknown correction term, exchange correlation energy functional, 𝐹 𝑜 𝒔 ℏ 2𝑛 𝑒𝒔 𝛼𝜍 𝒔, 𝒔 |𝒔𝒔 𝑓 2 𝑒𝒔𝑒𝒔 𝑜 𝒔 𝑜 𝒔 𝒔 𝒔 𝑓 𝑒𝒔 𝑜 𝒔 𝑊

𝒔; 𝑂 𝐹 𝑜 𝒔 ,

𝑂 # ions, 𝑂~𝑂.

Generalized Gradient Approximation (GGA): Tractable approx. to 𝐹

𝐹 𝑜 𝒔 𝑒𝒔 𝜁 𝑜 𝒔 , 𝛼𝑜 𝒔

SLIDE 6

KS‐DFT in OpenAtom

OpenAtom: Plane‐wave (PW) based KS‐DFT within the GGA – expand

KS states in the delocalized PW basis.

𝜔 𝒔 in D(h)

PW‐KS‐DFT in OpenAtom ‐ Disadvantages:
Large basis set required ‐ millions and millions (c.f. Carl Sagan).
Large memory required – need large machines.
Heavy atoms (impossibly) computationally intensive.
PW‐KS‐DFT in OpenAtom ‐ Advantages:
N2 log N or better scaling of interactions & derivatives ‐

Euler Exponential Spline (EES) Interpolation.

Only orthogonalization is ~N3 .
High parallelism under charm++.
k‐points, path integrals, LSDA &

tempering implemented.

SLIDE 7

Projector Augmented Wave Method (PAW)

P. E. Blöchl, Phys. Rev. B 50, 17953 (1994)

Goal: Implement N2 log N EES‐based PAW with high parallel efficiency in OpenAtom.

Projector‐Augmented Wave (PAW) : accurate treatment of

heavy atoms in KS‐DFT with low computational cost.

PAW‐KS‐DFT Advantages
KS states split into localized and delocalized/smooth parts –

small basis possible even for heavy atoms.

NMR and some other linear response methods require the core –

PAW makes it easy.

Small memory requirement.
PAW‐KS‐DFT Disadvantages
Implemented with inefficient N3 methods for interactions.
Parallel performance of standard implementations poor.
Accuracy control poor.

SLIDE 8

PAW Basics: KS states

𝜔 𝒔 𝜔

𝒔 𝜔 𝒔 ,

KS states: delocalized/smooth part, (S), + localized/core part, (core).

Core localized within a sphere of radius 𝑆 around each ion:

𝜔

𝒔 0, |𝒔 𝑺| 𝑆

Core: localized, written in terms of fixed core projectors, Δ𝑞, 𝑞

∗:

2𝑆

𝜔

𝒔 Δ𝑞 𝒔 𝑺 𝑎 , Δ𝑞 𝒔 𝑺 0, 𝒔 𝑺 𝑆

𝑎

p |𝜔 𝑒𝑠𝑞 𝒔 𝑺 𝜔 𝒔 , 𝑞 𝒔 𝑺 0, 𝒔 𝑺 𝑆

∗ 1 ion type, 1 channel for simplicity

Smooth: fills all spaces & varies, expanded in plane‐waves:

𝜔

𝒕 1

𝑊

𝜔
𝒉 exp

𝑗𝒉 𝒕

𝒉 / 𝒉

𝒔 𝒊𝒕, V = det h, 𝒉 =2π𝒊𝒉 , 𝒉 ∈ integer

𝜔

𝒉

𝜔 𝜔

𝒕

SLIDE 9

PAW Basics: Example KS state

𝑀 𝑀 𝑀

Localized ion core states, 𝜔

𝒔 embedded

𝜔,

𝒔

in the smooth part of the state, 𝜔

𝒔 , that fills D(h).

𝜔,

𝒔

𝑀 0 0 𝒊 0 𝑀 0 0 0 𝑀

SLIDE 10

PAW Basics: KS‐DFT within LDA under periodic boundary conditions at Γ

𝐹 ℏ 2𝑛

𝑒𝒔

𝒊

𝜔 𝛼|𝜔

𝐹

𝑒𝒔

𝒊

𝜁 𝑜 𝒔 𝐹 𝑓 2 𝑒𝒔

𝒊

𝑒𝒔

𝒊

′ 𝑜 𝒔 𝑜 𝒔 𝒔 𝒔 𝒏𝒊

𝒏

𝐹 𝑒𝒔

𝒊

𝑓𝑅𝑜 𝒔 |𝒔 𝒔 𝒏𝒊|

𝒏

𝐹 𝑜 𝒔

𝐹 𝐹 𝐹 𝐹 The whole enchilada: 𝐹 𝐹

𝐹

𝐹

Non‐interacting electron kinetic energy: Smooth and core terms

𝐹

ℏ

2𝑛

𝑒𝒔

𝒊

𝜔𝑱

𝛼|𝜔𝑱 ,

𝐹

ℏ

2𝑛 𝑎

𝑎 ,∆

, 𝐹

ℏ

2𝑛 𝑎

, ∆𝑞 𝛼|∆𝑞

𝐹 𝐹

𝐹

𝑒𝒔

𝒊

𝜁 𝑜 𝒔 𝑒𝒔

𝜁 𝑜 𝒔 𝜁 𝑜

𝒔

𝑜 𝒔 |𝜔𝑱

𝑻 𝒔 |,

𝑜 𝒔 𝑜 𝒔 𝑺 𝑜 𝒔 𝑺 + 𝑜 𝒔 𝑺 , 𝑜

𝒔 𝑜 𝒔 𝑺

∀ |𝒔 𝑺| < 𝑆 ∀ |𝒔 𝑺| < 𝑆 ∀ 𝒔 in 𝐸𝒊

core Exchange Correlation energy: Smooth and core terms

SLIDE 11

PAW Basics: KS‐DFT long/short‐range decomposition

Choose 𝛽, such that the g‐space cutoff = 𝐻 pw 𝑒𝑓𝑜𝑡𝑗𝑢𝑧 cutoff. Ensure r‐space cutoff, 𝑆 3.5 / 𝛽) > 𝑆, confines the m‐sum to the 1st image. Decompose short‐range into smooth, core1 and core2 type terms, (not shown).

𝐹

𝑓

𝑊

4𝜌

𝒉

exp 𝒉

4𝛽 𝑜 𝒉 𝑇̅ 𝒉 𝜌𝑓𝑜 0𝑇̅ 0 𝑊𝛽 𝑇̅ 𝒉 𝑅𝑲

exp 𝑗𝒉 · 𝑺

Using Poisson summation and Ewald’s decomposition of 1/r:

𝐹 𝐹

𝐹

𝐹 𝐹

𝐹

𝐹

𝑓

2 𝑒𝒔

𝒊

𝑒𝒔′

𝒊

𝑜 𝒔 𝑜 𝒔 erfc 𝛽 𝒔 𝒔 𝒔 𝒔 𝐹

𝑓

2𝑊

4𝜌

|𝒉|

exp 𝒉

4𝛽 |𝑜 𝒉| 𝜌𝑓 |𝑜 0| 2𝑊𝛽 𝐹

𝑓

𝑒𝒔

𝒊

𝑜 𝒔 erfc 𝛽 𝒔 𝑺 𝒔 𝑺

𝑲

𝐹 𝑓 2 𝑒𝒔

𝒊

𝑒𝒔′

𝒊

𝑜 𝒔 𝑜 𝒔

𝒔 𝒔 𝒏𝒊

𝒏

, 𝐹 𝑒𝒔

𝒊

𝑜 𝒔 𝑓𝑅𝑲 |𝒔 𝑺 𝒏𝒊|

𝒏

Due to the mixed localized and delocalized basis, there is no natural truncation scale

for the long‐range interactions of EHand Eextin g‐space or r‐space alone.

SLIDE 12

Accuracy of long/short decomposition

PW cutoff: ℏ𝟑𝐻

/2me Ryd

𝛿=𝛽𝑆 erfc(γ 5.1 3.0 2.21e‐05 9.4 3.5 7.43e‐07 16 4.0 1.54e‐08 𝐻

4 𝛿

𝑆

, 𝛿 𝛽𝑆

To approximately match long/short range accuracy:

𝑆 4 bohr

High accuracy can be obtained with both 𝑆 and 𝐻 small !

PW cutoff: ℏ𝟑𝐻

/2me Ryd

𝛿=𝛽𝑆 erfc(γ 20.3 3.0 2.21e‐05 37.5 3.5 7.43e‐07 64 4.0 1.54e‐08 𝑆 2 bohr

SLIDE 13

PAW Basics: Multi‐Resolution, Grids, EES and N2 log N scaling

How do we reduce scaling by one order in N and maintain accuracy?

Given a discrete, 𝒉 =2π𝒊𝒉

, finite g‐space, |g|< 𝐻 , the Fourier coefficients, 𝑔̅𝒉 of f(r), can be converted to 𝑔 𝒉 from f m(r) exactly using an equally spaced s‐space grid, r=hs, of side NFFT,β > 2𝑛𝑕 ,, β ∆𝑡β 1/NFFT,β. ∀ 𝑛 ∈ 𝑎 0

Using FFTs, the 𝑔 𝒉 , can be computed exactly in N log N as:

𝑔 𝒕

𝐺𝐺𝑈, 𝑔̅ 𝒉 , 𝐻 , 𝑔𝒉 =
𝐺𝐺𝑈, 𝑔 𝒕 , 𝑛𝐻 , V det h
1. Discrete real‐space: Fourier Coefficients and FFTs

Using 3 FFT grids, (1) Psi EES, (2) Density, (3) Density EES, and 1 discrete spherical polar grid around each ion, |r|<Rpc , all PAW energy terms & their derivatives can be accurately computed in N2 log N .

2. Euler Exponential Spline Interpolation and FFTs
To compute the Z‐matrices, structure factors, 𝑇̅ 𝒉 , and core functions, fast, it is useful develop a

differentiable controlled approximation to exp 𝑗𝒉 · 𝒔 on a discrete g‐space for all r=hs in D(h) via interpolation from an equally spaced s‐space grid, enabling the use of FFTs.

The Euler exponential spline (EES) delivers where Mp are the cardinal B‐splines and p the spline order,

𝑓

𝐸 𝑕

, 𝑂 𝑁 𝑣 𝑡̂ 𝑓

̂

𝜀̂,

̂

𝒫 2𝑕

𝑂
,

NFFT > 2𝑕 2.8𝑕

𝑁 has compact supp.

𝑣 s 𝑂 𝑚 int 𝑣

SLIDE 14

PAW Basics: g‐space to s‐space and back

Density EES: 𝑂

~𝑂

Density EES: 𝑂

,~𝑂

𝑜 𝒉

𝑜,𝒕

𝐺𝐺𝑈 ,, 𝑜 , 𝒉, 𝐻

𝒔 𝒊𝒕

𝒉 =2π𝒊𝒉

𝑜

, 𝒉 𝑜 𝒉𝐸 𝒉)

The 𝐸 𝒉 ∏ 𝐸

𝑕

, 𝑂,

, enables B‐spline interpolation

Density: 𝑂

~𝑂

𝜔

𝒉

Density: 𝑂

~𝑂

|𝜔

𝒕 |

𝒔 𝒊𝒕

𝐺𝐺𝑈 𝝎,, 𝜔

𝒉 , 𝐻

2

𝒉 =2π𝒊𝒉

𝜔

𝒕

𝑊

Density: 𝑂

~𝑂

𝑜 𝒉

Density: 𝑂

~𝑂

𝑜𝒕

𝐺𝐺𝑈 , 𝑜 𝒕, 𝐻

𝒔 𝒊𝒕

𝒉 =2π𝒊𝒉

𝑊

𝑂

𝒔 𝒊𝒕

Psi EES: 𝑂

~𝑂

𝜔

𝒉

Psi: 𝑂

,~𝑂

𝜔

, 𝒕

𝒉 =2π𝒊𝒉

𝑡 ≡ 𝑡̂/ 𝑂,

,

0 𝑡̂< 𝑂,

,

𝜔

, 𝒉 𝜔
𝒉 𝐸𝒉

𝐺𝐺𝑈 𝝎,, 𝜔

, 𝒉 , 𝐻

2

𝒔 𝒊𝒕

SLIDE 15

PAW Basics: r‐space interpolation

EES provides an accurate, differentiable interpolation between the different resolutions and length scales of PAW

FFT grid points, {𝑂

,/, 𝑂
,/},

s ∈ near ion 𝐾 𝑂

,~ 1

for EES interpolation. 1 N‐ion cores in D(h): 𝑂

~ 1

𝑆~ 1 All grid spacings are independent of system size. 2 N

B‐Spline Interpolation Fine spherical polar grid (𝑂

)

…

not to scale

2𝑆 Psi EES: 𝑂

,~𝑂

h defines D(h) = cuboid V =det h ~ N

…

1 2 N

𝜔

, 𝒕

N‐Partition Density EES: 𝑂

,~𝑂

h defines D(h) = cuboid V =det h ~ N

…

1 2 N

𝑜,𝒕 N‐Partition

SLIDE 16

Creating the r‐space representation of the e‐density

In the following, the multi‐length scale PAW method is used to construct the electron density in N2 log N as a demonstration: 𝑜

𝒔 𝑜 𝒔 𝑜 𝒔 𝑜 𝒔 𝐾 1. . 𝑂

𝑜 𝒔 : outside of cores (1) Create the smooth KS states in real space, 𝜔

𝒕 : N2 log N.

(2) Create the smooth density in real space, 𝑜 𝒕 : N2. (3) *Create the smooth density in the ion cores, 𝑜

𝒔 : N log N.

(4) Create the smooth Z‐matrix,

𝑎

: N2 log N.

(5) *Create the core‐2 densities, 𝑜

𝒔 : N2.

(6) *Create the core‐1 densities,

𝑜

𝒔 : N2 log N.

Formulae for all other components of PAW‐DFT have been derived including ionic and pw expansion coefficient derivatives.

* New terms.

SLIDE 17

3. Creating the smooth density,
around each ion J, on the fine grid, f
EES weighted

smooth density in g‐space 𝑜 , 𝒉 𝐸𝒉 𝑜 𝒉 , 𝒉 𝐻

𝐺𝐺𝑈 ,, 𝑜 , 𝒉, 𝐻

EES weighted smooth density

n discrete s‐space

𝑜 , 𝒕 , s ∈ 𝑂

, ,

r 𝒊𝒕

…

B‐Spline interpolation

2 1 3

N

𝑜

𝒔 ,

f ∈ 𝑂

EES interpolated

smooth density around each J N = number of ions, J=1..N, 𝑂 𝑂 𝑂

= number points on spherical‐polar grid around each ion.

𝑂

and 𝑂

,s ∈ near 𝐾 independent system size .

…

2 1 3

N

EES weighted smooth density around each J 𝑜

, 𝒕 ,

𝑂

,: s ∈ near 𝐾

SLIDE 18

Creating the

matrix for all ions of type jtyp and channel c

A set of points, s ∈ near ion J* , of type jtyp interpolated to obtain 𝑎

for all I,J ∈ jtyp (𝑂 ,𝝎~1,

𝑎

𝜔

,, 𝒕 𝑁, 𝒕

,𝝎

s ∈ s independent of I,c as are B splines, 𝑁

, 𝒕

…

2 1 3 NKS

𝜔

,,

𝒉

𝐸𝒉 𝜔

𝒉 𝑞
𝑻

𝒉 𝐻/2 𝐸𝒉 𝑞

𝒉

weighted KS states in g‐space

…

2 1 3 NKS

𝐺𝐺𝑈 𝝎,, 𝜔

,,

𝒉 , 𝐻

2 𝜔

,, 𝒕 ,

s ∈ 𝑂

𝝎,

EES x smooth projector weighted KS states in s‐space

…

2 1 3

𝑂

𝑂

–Partition,

𝑂

B‐Spline Interps

per KS state

𝑎

𝐾 ∈ jtyp

𝑂

,𝝎

s ∈ near ion 1 𝑂

,𝝎

s ∈ near ion 2 𝑂

,𝝎

s ∈ near ion 3

f type jtyp

𝑂

,𝝎

s ∈ near ion 𝑂

f type jtyp

𝑂

‐Partition,

𝑂

B‐Spline Interps.

𝑎

𝐾 ∈ jtyp

𝑎

𝐾 ∈ jtyp

𝑎

𝐾 ∈ jtyp

𝑎

𝑲 ∈ jtyp

Smooth Z‐matrix for ions of type jtyp and channel, c *Note, index J need not be contiguous in list of all atoms

SLIDE 19

5. Creating the core density component,
around each ion J, on the fine grid,
…

𝑎

𝐾 1 … 𝑂

𝑎

𝐾 1 … 𝑂

𝑎

𝐾 1 … 𝑂

𝑎

𝐾 1 … 𝑂

Reduction of 𝑎

𝑎

,

…

𝑎

,

𝑎

,

𝑎

,

Each KS state contributes to N unique reductions 𝑎

, |𝑎 |

𝑎
𝑎
𝑎
Reduction of 𝑎
∆𝑞𝒔

∆𝑞𝒔 𝑜

𝒔

𝑜

𝒔

In this example we have 1 projector 𝑜

𝒔 𝑎 , ∆𝑞 𝒔 ∀ 𝑔 ∈ 𝑂

𝑜

𝒔

𝑜

𝒔

SLIDE 20

6. Creating the core density component,
around each ion J, on the fine grid,
Each KS state contributes to N unique reductions

𝜔

,, 𝒕 ∑ 𝑎𝜔 , 𝒕 ∀

s ∈ near 𝐾: 𝑂

,

𝑎 = weight for points s ∈ near 𝐾 from KS state, I.

…

2 1 3 NKS

𝜔

, 𝒉

𝐸𝒉𝜔

𝒉 ,

𝒉 𝐻/2 EES weighted KS states in g‐space

…

2 1 3 NKS

𝐺𝐺𝑈 𝝎,, 𝜔

, 𝒉 , 𝐻

2 𝜔

, 𝒕 ,

s ∈ 𝑂

𝝎,

EES weighted KS states in s‐space

…

N reductions of s around J=1 N reductions of s around J=N Z11 Z21 Z31

2 1 3

N

𝜔

,, 𝒕 ,

𝑂

,: s ∈ near 𝐾

EES weighted KS states around J 𝑎

…

B‐Spline interpolation

2 1 3

N

𝑜

𝒔 ,

f ∈ 𝑂

EES interpolated

PAW 1 density around each J

∆𝑞𝒔

SLIDE 21

PAW Charm++ Implementation Progress:

Chare arrays defined and communication patterns established

in PowerPoint form.

Full PAW‐KS‐DFT flow chart for energies. Forces in progress.
Model Charm++ software outside of OpenAtom to test fine

grid spacing, Coulomb cusp smoothing, convergence with real‐ space cutoff, ... Complete.

N‐partition and N‐consolidation operations added to
charmFFT. Periodic boundary conditions need to be added.
Ready to begin integration into OpenAtom. Maybe with new

funding.

SLIDE 22

Grand Challenge Application: Perovskite solar cells

MAI‐term. PbI2‐defect. PbI2‐term.

Pros:

High eff., low cost, tunable band gap (ABX3)

Cons:

Instability: water, air, light, interface … & toxic compounds.

CH3NH3PbX3 PAW in OpenAtom

Understand: mechanism of

instability/degradation.

Search: non‐toxic B2+ (Fe, Co, Ni,…)

for new high perf. materials.

Design: new interface/encapsulation

for novel devices with long lifetime.

System size: 512 atoms (4x4x2

MAPbI3 +128 water), 1264 states

CH3NH3 Perovskite solar cells reach Si eff.

SLIDE 23

Conclusions

PAW‐KS‐DFT is an important method in computational science

that allows computations beyond PW‐KS‐DFT – heavy atoms.

Using EES Interpolation, we have derived a multi‐length scale

PAW technique that scales as N2 log N (all energy terms and all derivatives) – an important advance and 100 pages of latex.

Charm++ parallel framework developed; communication scaling