Applica'on Accelera'on on Current and Future Cray Pla4orms Alice - - PowerPoint PPT Presentation
Applica'on Accelera'on on Current and Future Cray Pla4orms Alice Koniges, NERSC, Berkeley Lab David Eder, Lawrence Livermore Na'onal Laboratory (speakers) Robert Preissl, Jihan Kim (NERSC LBL), Aaron Fisher, Nathan Masters, Velimir Mlaker
CUG 2010
Alice Koniges, NERSC, Berkeley Lab David Eder, Lawrence Livermore Na'onal Laboratory (speakers) Robert Preissl, Jihan Kim (NERSC LBL), Aaron Fisher, Nathan Masters, Velimir Mlaker (LLNL), Stephan Ethier, Weixing Wang (PPPL), Mar'n Head‐Gordon (UC Berkeley), Nathan Wichmann (CRAY Inc.) CRAY User Group Mee'ng May 2010
CUG 2010
CUG 2010
CUG 2010
CUG 2010
P0 P1 P2
CUG 2010
OpenMP tasks enables us to overlap MPI communica+on with independent Computa+on and therefore the overall run+me can be reduced by the costs
CUG 2010
Work on par+cle array (packing for sending, reordering, adding acer sending) can be
INDEPENDENT INDEPENDENT INDEPENDENT
GTS shi( roun-ne
CUG 2010
Overlapping MPI_Allreduce with par-cle work
Overlap: Master thread encounters (!$omp master) tasking statements and creates work for the thread team for deferred execu+on. MPI Allreduce call is immediately executed. MPI implementa+on has to support at least MPI_THREAD_FUNNELED Subdividing tasks into smaller chunks to allow beler load balancing and scalability among threads.
CUG 2010
Overlapping par-cle reordering Overlapping remaining MPI_Sendrecv
Par+cle reordering of remaining par+cles (above) and adding sent par+cles into array (right) & sending or receiving of shiced par+cles can be independently executed.
CUG 2010
!" #!" $!!" $#!" %!!" %#!" &!!"
'()*+,"
2)..)3456.+" '+3/7+18"
9):+";<+1=" !"#$%&"&'()*$%>"?"%"!(+,&-%./")*
9@<A)34" B,)4)3@."
!" #!" $!!" $#!" %!!" %#!"
&'()*+" ,--+*./0*" 1(--(2345-*" &*2.6*07"
8(9*":;*0<" !"#$%&"&'()*$%=">"$?"!(+,&-%./")*
8@;A(23" B+(3(2@-"
Performance breakdown of GTS shicer rou+ne using 4 OpenMP threads per MPI process with varying domain decomposi+on and par+cles per cell on Franklin Cray XT4. MPI communica+on in the shic phase uses a toroidal MPI communicator (constantly 128) However, performance differences in the 256 MPI run compared to 2048 MPI run! Speed‐Up is expected to be higher on larger GTS runs with hundreds of thousands CPUs since MPI communica+on is more expensive
CUG 2010
!" #!" $!" %!" &!" '!!" '#!" '$!" '%!" '&!" #!!" '!#$" ('#" #(%" )*+,"-.,/0" 123"245/,..,.6"7#6$6&8"59,:12";<4,=>."9,4"123"945/,.."
)=.?*:@"" A4*@*:=B"
!" #!" $!!" $#!" %!!" %#!" &!!" $!%'" #$%" %#("
)*+,"-.,/0"
123"245/,..,.6"7%6'689"5:,;12"<=4,>?.":,4"123":45/,.."
)>.@*;A"" B4*A*;>C"
Communica+on
messages on Hopper Cray XT5
threads/sockets per node is reasonable
CUG 2010
CUG 2010
an input to classical molecular dynamics/Monte Carlo simula+ons)
theory
– Treat correla+on energy with 2nd order Moller‐Plesset theory – U+lize auxiliary basis sets to approximate atomic orbital densi+es – Strengths: no self‐interac+on problem (DFT), 80‐90% of correla+on energy – Weakness: fich‐order computa+onal dependency on system size (expensive)
quin+c computa+on
CUG 2010
into seven major steps
becomes the dominant wall +me (e.g. glycine‐16, 83% of total wall +me is spent in step 4)
quin+c computa+on rou+nes (BLAS3 matrix mul+plica+ons) and quar+c I/O read
Greta Cluster (M. Head‐Gordon) : AMD Quad‐core Opterons
Wall +me in seconds
CUG 2010
computa+on (CPU: cache, loop control)
Architecture): parallel architecture developed by NVIDIA
(~ 75 GFLOPS TESLA, ~225 GFLOPS FERMI for double precision)
CUG 2010
Ttot ≈ Tload + Tmm1 + Tmm2 + Tmm3 + Trest Ttot ≈ max (Tload , Tmm1) + Tmm3 + max(Tmm2 , Tcopy) + Trest
CUG 2010
– Sun SunFire x4600 Servers – AMD Quad Core processors (“Shanghai”), 4 NVidia FX‐5800 Quadro GPUs (4GB memory) – CUDA 2.3 gcc 4.4.2 ACML 4.3.0
– Cray XT4 system – 2.3 GHz AMD Opteron Quad Core
RI‐MP2 wall +me (seconds) 4945 6542 1405 600 to 800 (?) Franklin TnT(CPU) TnT(GPU) TnT(GPU, beler I/O)
4.7x improvement, more so for beler I/O systems
CUG 2010
CUG 2010
CUG 2010
Diffusion equa+on Heat Conduc+on Radia+on Diffusion
CUG 2010
Level representa+on Composite representa+on Special transi+on elements and basis func+ons
CUG 2010
CUG 2010
CUG 2010
CUG 2010
CUG 2010
CUG 2010
CUG 2010
CUG 2010
5 10 15 20 25 30 35 40 16 32 64
Run'me Memory
Number of cores Memory and Run'me
CUG 2010
CUG 2010
int i, j; int a[100][100]; for (i=0; i<100; i++){ for (j=0; j<100; j++){ a[i][j]=a[i][j]+1; } } #include ”omp.h” int i, j; int a[100UL][100UL]; #pragma omp parallel for private (i,j) for (i=0; i<=100-1; i++){ #pragma omp parallel for private (j) for (j=0; j<=100-1; j++){ a[i][j]=a[i][j]+1; } }
incr.C rose_incr.C autoPar
... #CXX = g++ CXX = autoPar ...
.mk translate compile link .exe
CUG 2010
CUG 2010