Scalable GW software for excited electrons using OpenAtom Kavitha - - PowerPoint PPT Presentation

Scalable GW software for excited electrons using OpenAtom

Kayahan Saritas, Minjung Kim and Sohrab Ismail-Beigi Yale University Kavitha Chandrasekar, Eric Mikida, Eric Bohm and Laxmikant Kale University of Illinois at Urbana-Champaign Glenn Martyna Pimpernel Science, Software and Information Technology

Electronic structure calculations

𝑗ℏ 𝜖 𝜖𝑢 | ⟩ Ψ(𝑢) = + 𝐼| ⟩ Ψ(𝑢) § Time independent Schrodinger equation for a many-body system § Density functional theory (DFT) simplifies this to one-body problem Many Ri & rj

Solve for wavefunctions 𝜔!(𝑠) and energies 𝜗!

Comparison of the methods

1 100 10,000 10 1,000

FCI O(N!) Tight binding O(N3) HF, DFT O(N3) QMC GW CCSD(T) O(N7) Chemical Chemical Accuracy Accuracy Transition Transition States? States? Relative Relative Energies Energies Number of atoms Computational Cost Exact Schrödinger Equation O(N3-4)

DFT problem with excitations

ground state

𝜗! 𝜗!"# . . . . . . 𝐹$%& = % 𝜖𝐹 𝜖𝑂 !"' − % 𝜖𝐹 𝜖𝑂 !(' = 𝜗!"# − 𝜗!

DFT:

𝜗) Conduction band (empty) Valence band (filled) Band gap Janak’s theorem

DFT problem with excitations

ground state

𝜗! 𝜗!"# . . . . . . 𝐹$%& = % 𝜖𝐹 𝜖𝑂 !"' − % 𝜖𝐹 𝜖𝑂 !(' = 𝜗!"# − 𝜗!

DFT:

𝜗) Conduction band (empty) Valence band (filled) Band gap

Material DFT GW Expt. Diamond 3.9 5.6* 5.48 Si 0.5 1.3* 1.17 LiCl 6.0 9.1* 9.4 SrTiO3 2.0 3.4-3.8 3.25 Band gaps (eV)

Why band gap/excitations in a material is important? § Metallic, semiconducting or insulating? § Light-matter interactions in general § A lot of engineering implications: PV, lasers, luminescence …

GW method

Challenges § Memory intensive § Much larger number of conduction bands: Huge number of FFTs § Large and dense matrix multiplications § Unfavorable scaling 𝑃(𝑂4) Goal § Efficient and highly scalable GW software § 𝑃(𝑂3) scaling method

𝑄 𝑠, 𝑠) = −2 4

* +,--./

4

.1234 𝜔*(𝑠)𝜔0(𝑠)𝜔*(𝑠))𝜔0(𝑠))

𝐹* − 𝐹0

§ Lots of FFTs to get 𝜔!(𝑠) functions § However, 𝜗"# can converge using a small r-grid

What is expensive in GW?

~ 𝑂* + 𝑂+ 𝑂, ln 𝑂, ~2𝑂,

• ln 𝑂,

~ 𝑂*𝑂+ 𝑂,

• = 𝑃(𝑂.)

* Kim et al., (2020), Phys. Rev. B., 101, pp. 035139

𝑄<,<! = −2 4

* >"##

4

>$%"## 𝜔<,* ∗ 𝜔<,0𝜔<!,0 ∗

𝜔<!,* 𝐹0 − 𝐹*

1 𝐹$ − 𝐹% = *

& '

𝑓" (!"(" )𝑒𝜐 = *

& '

𝑓"(!)𝑓(")𝑒𝜐 = *

& '

𝑔(𝜐)𝑓") 𝑒𝜐

(1) Laplace transform: (2) Gauss-Laguerre quadrature: *

& '

𝑔(𝜐)𝑓") 𝑒𝜐 ≈ 0

* +#

𝜕* 𝑔 𝜐*

Nr2NunoccNocc~ N4 Nr2Nq(Nunocc+Nocc)~ N3

O(N3) algorithm (CTSP) for P

𝑂! 𝑂"

𝑌<,<! = 4

@ >&

4

A >'

𝐵<,<)𝐶<,<) 𝑥 + 𝑏@ − 𝑐

A

CTSP: Complex time shredded propagator

𝑄<,<! = −2 4

* >"##

4

>$%"##

𝜔<,*

∗ 𝜔<,0𝜔<!,0 ∗

𝜔<!,* 4

B >(

𝜕B 𝑔 𝜐B = 4

B >(

𝜕B [4

* >"##

𝜔<,*𝜔<!,*

∗

𝑓C)D*][ 4

>$%"##

𝜔<,0𝜔<!,0

∗

𝑓EC#D*] Nq(Nunocc+Nocc) Nr2~ N3

O(N3) algorithm (CTSP) for P

(3) Energy windows:

𝑄

$,$& = ( ' (/0

(

) (10

𝑄

$,$& ') !!,# "#$($#$, &; ( = 0) ,#$

(&'()

,#$

(*+)

• ,,$

,',- ,',# ,',$ ,*,- ,*,# ,*,$ ,*,. ,*,/ ,& a) b)

𝐹!

Most expensive

• Real-space P
• O(N3) method

Also expensive - O(N4)

Steps for typical GW calculations

Σ± (𝜕)<,<!

GHI = 4 J,I

𝐶<,<!

J 𝜔<I𝜔<!I ∗

𝜕 − 𝐹I ± 𝜕J

𝐶$,$!

& : residues

𝜕&: energies of the poles of 𝑋(𝑠)$,$'

§ 𝜕 − 𝜗I ± 𝜕J=0 is possible: Gauss-Laguerre quadrature not applicable

O(N3) method for self-energy

§ New quadrature is needed and was developed: Hermite-Gauss-Laguerre quadrature 1 𝜕 − 𝐹I ± 𝜕J = 𝐽𝑛 L

K L

𝑒𝜐𝑓EDED+/N𝑓@(OEC%±O,)D 𝑌<,<! = 4

@ >&

4

A >'

𝐵<,<)𝐶<,<) 𝑥 + 𝑏@ − 𝑐

A

§ Si crystal (16 atoms) § Number of bands: 399 § 𝑂Q*=1, 𝑂Q0=4 § MgO crystal (16 atoms) § Number of bands: 433 § 𝑂Q*=1, 𝑂Q0=4

Results: Energy gap

* Kim et al., (2020), Phys. Rev. B., 101, pp. 035139

Performance against other codes

§ Si crystal (16 atoms) § Number of bands: 399 § 𝑂JQ=15, 𝑂IQ=30

* Kim et al., (2019), Comput. Phys. Commun., 244, pp. 427-441

http://charm.cs.illinois.edu/OpenAtom/

OpenAtom GW Parallel Scaling

OpenAtom Team

GW-BSE Parallelization Phase Serial Parallel 1 Compute P in Rspace (N4 and N3 methods) Complete Complete 2 FFT P to GSpace Complete Complete 3 Invert epsilon Complete Complete 4 Plasmon pole Complete Future Work 5 COHSEX Self-energy Complete Complete 6 Dynamic Self-energy Complete Future Work

14

GW Phase-I P Matrix Computation (N4 and N3 method)

15

P Matrix R R M unoccupied R Ψ Vectors L occupied

…

1D Chare Array 2D Chare Array 2D Tiles

Parallel Decomposition: Input state vectors

16

Duplicate occupied and unoccupied states on each node

ψ ψ

ψ ψ ψ

Computation of Pmatrix using N3 method

• Outer loops are windows of occupied and unoccupied states
• Most expensive computation - 𝜍 and 𝜍)matrices

for l = 1:Nvw for m = 1:Ncw for j = 1:Nquadlm calculate 𝜍01')! calculate 𝜍&01')! P[r,r’] += 𝜍01')![r,r’] x 𝜍&01')![r,r’]

Computation 𝜍 matrix (Using occupied states)

• State vectors are represented with ψ

○ Number of occupied states = L, each state has N elements ○ All occupied states can be represented as a matrix ψV[1:L][1:N])

𝜍2345) -> Add elements of outer product of ψV[1:L] for l=1:L for r=1:N for r’=1:N 𝜍2345) [r,r’] += ψV [l] T[r] x ψV[l][r’] 𝜍2345) -> Same as ZGEMM of all ψV and all ψVT ZGEMM (ψVT[1:N][1:L] , ψV[1:L][1:N]) (i.e matrix multiply ) for r=1:N for r’=1:N for l=1:L 𝜍2345) [r,r’] += ψVT[r][l] x ψV[l][r’]

Computation 𝜍’ matrix (Using unoccupied states)

• Number of unoccupied states = M, each state has N elements
• All unoccupied states can be represented as a matrix ψC[1:M][1:N])

𝜍2345) -> Add elements of outer product of ψC[1:M] for m=1:M for r=1:N for r’=1:N 𝜍′2345) [r,r’] += ψC [m] T[r] x ψC[m][r’] 𝜍2345) -> Same as ZGEMM of all ψC and all ψCT ZGEMM (ψCT[1:N][1:M] , ψC[1:M][1:N]) (i.e matrix multiply ) for r=1:N for r’=1:N for m=1:M 𝜍′2345) [r,r’] += ψCT[r][m] x ψC[m][r’]

Computation of P-matrix (tiled) (N3)

P Matrix Unoccupied states ψC(1:M) Occupied states ψV(1:L) (ZGEMM) 𝜍 matrix (ZGEMM) 𝜍’ matrix (Element-wise multiply)

• f 𝜍 & 𝜍’ matrix

L N N L N N M M N N N N N N

Performance of N3 method

• N3 method is an order faster than

N4 method for Si108 atoms dataset

○ 20k X 20k output matrix size

• Scales well on Intel KNL and

SkyLake nodes

• Future scaling results for larger

datasets

10 100 1000 10000 8 16 32 64 Execution Time Node count (48 cores per node) N4 method N3 method 100 1000 10000 8 16 32 64 Execution Time Node count (128 cores per node) N4 method N3 method

Intel KNL nodes (Stampede2) Intel Skylake nodes (Stampede2)

Questions?