GENE Training D. R. Hatch Oct. 2018 ICTP, Trieste, IT Ge=ng GENE - - PowerPoint PPT Presentation

GENE Training

• D. R. Hatch
• Oct. 2018

ICTP, Trieste, IT

Ge=ng GENE

• Go to genecode.org
• Register
• Follow instrucDons in registraDon email
• DocumentaDon is in ‘doc’ directory in GENE folder

2

Many GyrokineDc Codes with Different Algorithms, Geometry, ComputaDonal Domains

• Two main algorithms
• ParDcle in Cell (PIC) [ORB5, GTC, GEM, XGC, etc.]
• The code tracks parDcle trajectories in response to self-consistent

fields

• Sum over parDcles at new posiDon, determine moments, calculate

new fields

• ConDnuum [GENE, GS2, GYRO, Gkeyll, GKV, etc.]
• DiscreDze enDre distribuDon funcDon on a grid and solve PDE
• Full f: doesn’t separate background from fluctuaDng distribuDon funcDon—

very challenging and computaDonally expensive

• δf: separate background from fluctuaDons, evolve only fluctuaDons—more

efficient and usually sufficient for tokamak core

3

The GyrokineDc GENE Code

4

• ConDnuum approach to gyrokineDcs (evolve

distribuDon funcDon on grid).

• Publicly available, world-wide user base

from ~30 scienDfic insDtuDons (US), ~100 worldwide

• Modes of operaDon:
• delta-f & full-f (gradient-driven, flux-

driven)

• flux-tube & full-flux-surface & global
• Unique combinaDon of various FDM,

and spectral methods

• Extensive physics
• Part of fusion whole device modeling ECP

project

GENE on top-level HPC resources INCITE Award (2016)

Strong scaling of GENE on Titan (2k-16k nodes)

(genecode.org; Jenko et al PoP 2000)

How the GENE Code Works

• GENE can operate in mulDple modes
• Local flux tube: take gradients,

parameters at radial points

• JusDfied by large scale

separaDon between background profiles and fluctuaDon scales

• Global: simulate a radial domain

accounDng for full variaDon of profiles, and magneDc equilibrium

• Necessary when there is not

large scale separaDon

Flux tube takes points Global covers a conDnuous radial domain

5

How the GENE Code Works: Geometry

• Tokamak geometry can be complex
• Define computaDonal domain to follow magneDc field lines to

exploit anisotropy

• Extended domain along field line
• Smaller domain perpendicular to field
• Incorporate complexity of the simulaDon domain into metric

coefficients grid

• Allows for simple, structured grid in spite of complex

geometry

6

Five Phase Space Dimensions

• x: radial coordinate
• Domain typically ~100 gyroradii
• GENE uses Fourier (local) or finite difference (global) in x
• Requires ~100 to ~1000 grid points
• y: binormal or field line label
• perpendicular to magneDc field (but not always perpendicular to

x!)

• Domain typically ~100 gyroradii
• GENE uses Fourier in y
• Requires ~16-~100 in y

7

• z: coordinate labeling distance along the field line
• This is coordinate is constructed so that field lines are straight

in the poloidal-toroidal plane

• Domain extends from –pi to pi (inside of the torus to outside)
• GENE uses finite differences in z
• Requires approximately 16-100 grid points (several hundred in

extreme cases)

Five Phase Space Dimensions

8

• Two velocity space coordinates
• Velocity parallel to magneDc field: v||
• Domain plus/minus vth
• GENE uses finite differences
• Requires ~16-100 grid points
• MagneDc moment captures perpendicular velocity:
• Domain plus/minus vth
• GENE uses finite differences
• Requires ~8-100 grid points

Five Phase Space Dimensions

9

How the GENE Code Works: Time Advance

• GENE solves the gyrokineDc equaDon

Runge-Kuwa for Dme advance

• ConvenDonal algorithms that have proven very robust and efficient

10

How the GENE Code Works: Time Advance

• GENE solves the gyrokineDc equaDon

Fourier for perpendicular (x,y) derivaDves

• ConvenDonal algorithms that have proven very robust and efficient

11

How the GENE Code Works: Time Advance

• GENE solves the gyrokineDc equaDon

Finite Differences for parallel derivaDves

• ConvenDonal algorithms that have proven very robust and efficient

12

How the GENE Code Works: Time Advance

• GENE solves the gyrokineDc equaDon

Finite Differences for velocity derivaDves

• ConvenDonal algorithms that have proven very robust and efficient

13

How the GENE Code Works: ParallelizaDon

• Total grid points:
• 3x106 (smallest possible nonlinear simulaDon)
• A few 100 cpu hours
• 3x1010 (large global simulaDon)
• Million cpu hours
• Typically with mulDple kineDc species (ions, electrons,

and oxen an impurity species)

• DistribuDon funcDon can be several TB
• Requires extreme scalability for both memory and

computaDon Dme

• DistribuDon 5D grid on MPI (message passing interface)

processor grid

• Processors talk to each other only when exchanging

informaDon needed for derivaDves, integrals, etc.

• GENE can parallelize in 5+ dimensions

14

GENE Extreme Scale Computing

Strong scaling of GENE on Titan (2k-16k nodes)

15

Exascale Computing: ~50x Increase over Present-Day

¤ DOE iniDaDve (execuDve order in 2015): develop exascale compuDng capability ¤ Fundamental change in how high performance compuDng works ¤ GENE was awarded exascale compuDng project (ECP) grant to prepare for exascale computers (one of ~15 naDonwide)

16

GENE: Types of Simulations

• Linear simulaDons:
• Set nonlinear term to 0
• èFourier modes are decoupled
• èThis becomes an eigenvalue problem
• The complex frequency defines the growth rate and frequency of

the instability

• These are very informaDve and not too expensive computaDonally
• Can even run on desktop computer

17

GENE: Types of Simulations

• Nonlinear simulaDons:
• Saturates via mode-mode coupling
• èRequires many Fourier modes in x and y
• IniDal linear growth phase followed by a staDsDcally saturated state
• Produces fluctuaDon amplitudes, staDsDcal properDes of

turbulence, transport levels, etc.

• Needs many processors

18