Mapping MPI+X Applications to Multi-GPU Architectures A - - PowerPoint PPT Presentation

LLNL-PRES-746812

This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC

Mapping MPI+X Applications to Multi-GPU Architectures

A Performance-Portable Approach

Edgar A. León Computer Scientist San Jose, CA March 28, 2018 GPU Technology Conference

LLNL-PRES-746812

2

Application developers face greater complexity

§ Hardware architectures

— SMT — GPUs — FPGAs — NVRAM — NUMA, multi-rail

§ Programming abstractions

— MPI — OpenMP, POSIX threads — CUDA, OpenMP 4.5, OpenACC — Kokkos, RAJA

§ Applications

— Need the hardware topology to

run efficiently

— Need to run on more than one

architecture

— Need multiple programming

abstractions Hybrid applications

LLNL-PRES-746812

3

§ Compute power is not the bottleneck § Data movement dominates energy consumption § HPC applications dominated by the memory system

— Latency and bandwidth — Capacity tradeoffs (multi-level memories)

§ Leverage local resources

— Avoid remote accesses

How do we map hybrid applications to increasingly complex hardware?

More than compute resources, it is about the memory system!

LLNL-PRES-746812

4

The Sierra system that will replace Sequoia features a GPU-accelerated architecture

Mellanox Interconnect Single Plane EDR InfiniBand 2 to 1 Tapered Fat Tree IBM POWER9

• Gen2 NVLink

NVIDIA Volta

• 7 TFlop/s
• HBM2
• Gen2 NVLink

Components Compute Node

2 IBM POWER9 CPUs 4 NVIDIA Volta GPUs NVMe-compatible PCIe 1.6 TB SSD 256 GiB DDR4 16 GiB Globally addressable HBM2 associated with each GPU Coherent Shared Memory

Compute Rack

Standard 19” Warm water cooling

Compute System

4320 nodes 1.29 PB Memory 240 Compute Racks 125 PFLOPS ~12 MW

GPFS File System

154 PB usable storage 1.54 TB/s R/W bandwidth

LLNL-PRES-746812

5

2017 CORAL EA

Figure generated with hwloc

NVMe SSD 1 IB NIC per socket 2 GPUs per socket 10 cores per socket SMT-8 Private L1, L2, L3 2 Ethernet NICs

IBM Power8+ SL822LC NVIDIA Pascal Tesla P100

LLNL-PRES-746812

6

Existing approaches and their limitations

§ MPI/RM approaches

— By thread — By core — By socket — Latency (IBM Spectrum MPI) — Bandwidth (IBM Spectrum MPI)

§ OpenMP approaches

— Policies

Spread, Close, Master

— Predefined places

Threads, Cores, Sockets

§ Limitations

— Memory system is not primary

concern

— No coherent mapping across

programming abstractions

— No heterogeneous devices

support

LLNL-PRES-746812

7

A portable algorithm for multi-GPU architectures: mpibind

§ 2 IBM Power8+

processors

§ Per socket

— 10 SMT-8 cores — 1 InfiniBand NIC — 2 Pascal GPUs

§ NVMe SSD § Private L1, L2, L3

per core

Node Socket Memory G G Socket Memory G G

Machine L3 (8192KB) L2 (512KB) L1d (64KB) Core P#0 PU P#0 PU P#1 PU P#2 PU P#3 PU P#4 PU P#5 PU P#6 PU P#7

LLNL-PRES-746812

8

mpibind’s primary consideration: The memory hierarchy

L1 L2 L3 L1 L2 L3 L1 L2 L3 L1 L2 L3 L1 L2 L3 L1 L2 L3 L1 L2 L3 L1 L2 L3 L1 L2 L3 L1 L2 L3 NUMA L1 L2 L3 L1 L2 L3 NUMA

Start

...

NIC-0 GPU-0 GPU-1 NVM-0 NIC-1 GPU-2 GPU-3 p0-7 c0 p8-15 c1 p16-23 c2 p24-31 c3 p32-39 c4 p40-47 c5 p48-55 c6 p56-63 c7 p64-71 c8 p72-79 c9 p80-87 c10 p152-159 c19

l0 l1 l2 l3 l4 l5

1 2 3

Workers = 8 Vertices(l2) = 20 8 workers / 2 NUMA 4 workers / 2 GPUs

LLNL-PRES-746812

9

§ Get hardware topology § Devise memory tree G

— Assign devices to memory vertices

§ Calculate # workers w (all processes and threads) § Traverse tree to determine level k with at least w vertices § Traverse subtrees selecting compute resources for each vertex

m’: vertices(k) → PU

§ Map workers to vertices respecting NUMA boundaries

m: workers → vertices(k) → PU

The algorithm

L1 L1 L2 L2 NUMA NUMA Start ... ... GPU0 NIC0 GPU1 NIC1 Core0 p4 p5 p6 p7 Core1 l0 l1 l2 l3 l4

LLNL-PRES-746812

10

Example mapping: One task per GPU

Node Socket Memory G G Socket Memory G G

def lat bw mb

0-79 0-7 0-7 0,8,16,24,32

1

80-159 8-15 8-15 40,48,56,64,72

2

0-79 16-23 80-87 80,88,96,104,112

3

80-159 24-31 88-95 120,128,136,144,152

2 1 3 1 2 3 3 1 3 2 1 3 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 3 1 2 2

LLNL-PRES-746812

11

Evaluation: Synchronous collectives, GPU compute and bandwidth, app benchmark

Machine CORAL EA system Affinity Spectrum-MPI default Spectrum-MPI latency Spectrum-MPI bandwidth mpibind Benchmarks MPI Barrier MPI Allreduce Bytes&Flops Compute Bytes&Flops Bandwidth SW4lite Number

• f Nodes

1, 2, 4, 8, 16 Processes (tasks) per node 4, 8, 20

LLNL-PRES-746812

12

Enabled uniform access to GPU resources

§ Execute multiple instances

concurrently

— 4 and 8 PPN

§ Measure GPU FLOPS § Processes time-share GPUs

by default

§ Performance without

mpibind severely limited because of GPU mapping

bandwidth default latency mpibind bandwidth default latency mpibind Mean GFlops per Process 1000 2000 3000 4000 5000 GPU 0 4 4 4 4 5 5 4 8 PPN Target PPN: 4 Target PPN: 8

Compute micro-benchmark*

*kokkos/benchmarks/bytes_and_flops

Better

LLNL-PRES-746812

13

Enabled access to the memory of all GPUs without user intervention

§ Execute multiple instances

concurrently

— 4 or 8 PPN

§ Measure GPU global

memory bandwidth

§ Processes time-share GPUs

by default

§ Without mpibind some

processes fail running out

• f memory

bandwidth default latency mpibind bandwidth default latency mpibind Mean GBps per Process 100 200 300 400 500 GPU 0 4 4 4 4 5 4 4 8 PPN Target PPN: 4 Target PPN: 8

Memory bandwidth micro-benchmark*

*kokkos/benchmarks/bytes_and_flops

Better

LLNL-PRES-746812

14

Impact on SW4lite–Earthquake ground motion simulation

§ Simplified version of SW4

— Layer over half space (LOH.2) — 17 million grid points (h100)

§ Multiple runs, calculate mean

— 6 times for GPU — 10 times for CPU

§ Performance speedup

— CPU: mpibind over default: 3.7x — GPU: mpibind over bandwidth: 2.2x — GPU over CPU: 9.7x

bandwidth default latency mpibind bandwidth default latency mpibind Forcing Supergrid Scheme BC_phys BC_comm Mean Execution Time (secs) 200 400 600 800 CPU: MPI + OpenMP GPU: MPI + RAJA

Better

TPP → CPP x PPN → CPN bandwidth 8 1 4 4 under default 80 10 4 40

• ver

latency 8 1 4 4 under mpibind 5 5 4 20

TPP: Threads per process, CPP: Cores per process, PPN: Processes per node, CPN: Cores per node

LLNL-PRES-746812

15

mpibind: A memory-driven mapping algorithm for multi-GPU architectures

§ Focuses on hierarchical

nature of memory system

§ Provides portability and user

transparency

— Same algorithm on GPU-based,

KNL-based, and commodity- based systems

§ Encompasses

— Hybrid programming

abstractions

— Heterogeneous devices

§ Outperforms existing

approaches without user intervention

— Reduces runtime variability — Competitive performance on

collective operations

— Enables uniform access to all

GPU resources

LLNL-PRES-746812

16

Bibliography and related GTC talks

§ mpibind: A memory-centric affinity

algorithm for hybrid applications. MEMSYS 2017.

§ System noise revisited: Enabling

application scalability and reproducibility with SMT. IPDPS 2016.

§ 3D ground motion simulation in

• basins. Final report to Pacific

Earthquake Engineering Research Center, 2005.

§ SW4lite, Kokkos, and RAJA

github.com/geodynamics/sw4lite github.com/kokkos/kokkos github.com/LLNL/RAJA

§ S8270 – Acceleration of an

LLNL production Fortran application on the Sierra supercomputer

§ S8470 – Using RAJA for

accelerating LLNL production applications on the Sierra supercomputer

§ S8489 – Scaling molecular

dynamics across 25,000 GPUs

• n Sierra & Summit