GENE Training D. R. Hatch Oct. 2018 ICTP, Trieste, IT Ge=ng GENE - PowerPoint PPT Presentation

GENE Training D. R. Hatch Oct. 2018 ICTP, Trieste, IT

Ge=ng GENE • Go to genecode.org • Register • Follow instrucDons in registraDon email • DocumentaDon is in ‘doc’ directory in GENE folder 2

Many GyrokineDc Codes with Different Algorithms, Geometry, ComputaDonal Domains • Two main algorithms • ParDcle in Cell (PIC) [ORB5, GTC, GEM, XGC, etc.] • The code tracks parDcle trajectories in response to self-consistent fields • Sum over parDcles at new posiDon, determine moments, calculate new fields • ConDnuum [GENE, GS2, GYRO, Gkeyll, GKV, etc.] • DiscreDze enDre distribuDon funcDon on a grid and solve PDE • Full f: doesn’t separate background from fluctuaDng distribuDon funcDon— very challenging and computaDonally expensive • δ f: separate background from fluctuaDons, evolve only fluctuaDons—more efficient and usually sufficient for tokamak core 3

The GyrokineDc GENE Code (genecode.org; Jenko et al PoP 2000) • ConDnuum approach to gyrokineDcs (evolve GENE on top-level HPC resources distribuDon funcDon on grid). • Publicly available, world-wide user base from ~30 scienDfic insDtuDons (US), ~100 INCITE Award (2016) worldwide • Modes of operaDon: • delta-f & full-f (gradient-driven, flux- driven) • flux-tube & full-flux-surface & global • Unique combinaDon of various FDM, and spectral methods • Extensive physics • Part of fusion whole device modeling ECP project Strong scaling of GENE on Titan (2k-16k nodes) 4

How the GENE Code Works • GENE can operate in mulDple modes Flux tube takes points • Local flux tube: take gradients, parameters at radial points • JusDfied by large scale separaDon between background profiles and fluctuaDon scales • Global: simulate a radial domain accounDng for full variaDon of profiles, and magneDc equilibrium • Necessary when there is not large scale separaDon Global covers a conDnuous radial domain 5

How the GENE Code Works: Geometry • Tokamak geometry can be complex • Define computaDonal domain to follow magneDc field lines to exploit anisotropy • Extended domain along field line • Smaller domain perpendicular to field • Incorporate complexity of the simulaDon domain into metric coefficients grid • Allows for simple, structured grid in spite of complex geometry 6

Five Phase Space Dimensions • x: radial coordinate • Domain typically ~100 gyroradii • GENE uses Fourier (local) or finite difference (global) in x • Requires ~100 to ~1000 grid points • y: binormal or field line label • perpendicular to magneDc field (but not always perpendicular to x!) • Domain typically ~100 gyroradii • GENE uses Fourier in y • Requires ~16-~100 in y 7

Five Phase Space Dimensions • z: coordinate labeling distance along the field line • This is coordinate is constructed so that field lines are straight in the poloidal-toroidal plane • Domain extends from –pi to pi (inside of the torus to outside) • GENE uses finite differences in z • Requires approximately 16-100 grid points (several hundred in extreme cases) 8

Five Phase Space Dimensions • Two velocity space coordinates • Velocity parallel to magneDc field: v || • Domain plus/minus v th • GENE uses finite differences • Requires ~16-100 grid points • MagneDc moment captures perpendicular velocity: • Domain plus/minus v th • GENE uses finite differences • Requires ~8-100 grid points 9

How the GENE Code Works: Time Advance • GENE solves the gyrokineDc equaDon Runge-Kuwa for Dme advance • ConvenDonal algorithms that have proven very robust and efficient 10

How the GENE Code Works: Time Advance • GENE solves the gyrokineDc equaDon Fourier for perpendicular (x,y) derivaDves • ConvenDonal algorithms that have proven very robust and efficient 11

How the GENE Code Works: Time Advance • GENE solves the gyrokineDc equaDon Finite Differences for parallel derivaDves • ConvenDonal algorithms that have proven very robust and efficient 12

How the GENE Code Works: Time Advance • GENE solves the gyrokineDc equaDon Finite Differences for velocity derivaDves • ConvenDonal algorithms that have proven very robust and efficient 13

How the GENE Code Works: ParallelizaDon • Total grid points: • 3x10 6 (smallest possible nonlinear simulaDon) • A few 100 cpu hours • 3x10 10 (large global simulaDon) • Million cpu hours • Typically with mulDple kineDc species (ions, electrons, and oxen an impurity species) • DistribuDon funcDon can be several TB • Requires extreme scalability for both memory and computaDon Dme • DistribuDon 5D grid on MPI (message passing interface) processor grid • Processors talk to each other only when exchanging informaDon needed for derivaDves, integrals, etc. • GENE can parallelize in 5+ dimensions 14

GENE Extreme Scale Computing Strong scaling of GENE on Titan (2k-16k nodes) 15

Exascale Computing: ~50x Increase over Present-Day ¤ DOE iniDaDve (execuDve order in 2015): develop exascale compuDng capability ¤ Fundamental change in how high performance compuDng works ¤ GENE was awarded exascale compuDng project (ECP) grant to prepare for exascale computers (one of ~15 naDonwide) 16

GENE: Types of Simulations • Linear simulaDons: • Set nonlinear term to 0 • è Fourier modes are decoupled • è This becomes an eigenvalue problem • The complex frequency defines the growth rate and frequency of the instability • These are very informaDve and not too expensive computaDonally • Can even run on desktop computer 17

GENE: Types of Simulations • Nonlinear simulaDons: • Saturates via mode-mode coupling • è Requires many Fourier modes in x and y • IniDal linear growth phase followed by a staDsDcally saturated state • Produces fluctuaDon amplitudes, staDsDcal properDes of turbulence, transport levels, etc. • Needs many processors 18