Optimizing MPI Communication on Multi-GPU Systems using CUDA - - PowerPoint PPT Presentation

▶
optimizing mpi communication on multi gpu systems using
SMART_READER_LITE
LIVE PREVIEW

Optimizing MPI Communication on Multi-GPU Systems using CUDA - - PowerPoint PPT Presentation

Apr 22, 2023 543 likes •779 views

Optimizing MPI Communication on Multi-GPU Systems using CUDA Inter-Process Communication Sreeram Potluri* Hao Wang* Devendar Bureddy* Ashish Kumar Singh* Carlos Rosales + Dhabaleswar K. Panda* *Network-Based Computing Laboratory

slide-1
SLIDE 1

Optimizing MPI Communication

  • n Multi-GPU Systems using

CUDA Inter-Process Communication

Sreeram Potluri* Hao Wang* Devendar Bureddy*

Ashish Kumar Singh* Carlos Rosales+ Dhabaleswar K. Panda*

*Network-Based Computing Laboratory Department of Computer Science and Engineering The Ohio State University

+Texas Advanced Computing Center 1

slide-2
SLIDE 2

Outline

  • Motivation
  • Problem Statement
  • Using CUDA IPC
  • CUDA IPC based Designs in MVAPICH2

– Two Sided Communication – One-sided Communication

  • Experimental Evaluation
  • Conclusion and Future Work

2

slide-3
SLIDE 3
  • GPUs are becoming a common component of modern clusters – higher

compute density and performance/watt

  • 3 of the top 5 systems in the latest Top 500 list use GPUs
  • Increasing number of HPC workloads are being ported to GPUs - many
  • f these use MPI
  • MPI libraries are being extended to support communication from GPU

device memory

GPUs for HPC

3

slide-4
SLIDE 4

4

At Sender:

  • cudaMemcpy (sbuf, sdev)

MPI_Send (sbuf, . . . )

  • At Receiver:
  • MPI_Recv (rbuf, . . . )

cudaMemcpy (rdev, rbuf)

MVAPICH/MVAPICH2 for GPU Clusters

Earlier

At Sender:

  • MPI_Send (sdev, . . . )
  • At Receiver:
  • MPI_Recv (rdev, . . . )

Now PCIe GPU CPU NIC Switch inside MVAPICH2

  • Efficient overlap copies over the PCIe with RDMA transfers over the network
  • Allows us to select efficient algorithms for MPI collectives and MPI datatype

processing

  • Available with MVAPICH2 v1.8 (http://mvapich.cse.ohio-state.edu)

4

slide-5
SLIDE 5

Motivation

5

CPU

GPU 1 GPU 0

Memory I/O Hub Process 0 Process 1

  • Multi-GPU node architectures are

becoming common

  • Until CUDA 3.2

– Communication between processes staged through the host – Shared Memory (pipelined) – Network Loopback [asynchronous)

  • CUDA 4.0

– Inter-Process Communication (IPC) – Host bypass – Handled by a DMA Engine – Low latency and Asynchronous – Requires creation, exchange and mapping of memory handles - overhead

HCA

slide-6
SLIDE 6

6

Comparison of Costs

Copy Latency (usec)

50 100 150 200

CUDA IPC Copy Copy Via Host CUDA IPC Copy + Handle Creation & Mapping Overhead 49 usec 3 usec 228 usec

  • Comparison of bare copy costs between two processes on one node,

each using a different GPU (outside MPI)

  • 8 Bytes
slide-7
SLIDE 7

Outline

  • Motivation
  • Problem Statement
  • Basics of CUDA IPC
  • CUDA IPC based Designs in MVAPICH2

– Two Sided Communication – One-sided Communication

  • Experimental Evaluation
  • Conclusion and Future Work

7

slide-8
SLIDE 8

Problem Statement

8

  • Can we take advantage of CUDA IPC to improve performance of MPI

communication between GPUs on a node?

  • How do we address the memory handle creation and mapping
  • verheads?
  • What kind of performance do the different MPI communication

semantics deliver with CUDA IPC?

– Two-sided Semantics – One-sided Semantics

  • How do CUDA IPC based designs impact the performance of end-

applications?

slide-9
SLIDE 9

Outline

  • Motivation
  • Problem Statement
  • Basics of CUDA IPC
  • CUDA IPC based Designs in MVAPICH2

– Two Sided Communication – One-sided Communication

  • Experimental Evaluation
  • Conclusion and Future Work

9

slide-10
SLIDE 10

Basics of CUDA IPC

Process 0 Process 1

cudaIpcGetMemhandle (&handle, base_ptr) cudaIpcOpenMemhandle (&base_ptr, handle) IPC handles cudaMemcpy (rbuf_ptr, base_ptr + displ) cudaEventRecord (&ipc_event, event_handle) cudaIpcGetEventHandle (&event_handle, event) cudaStreamWaitEvent (0, event)

  • ther CUDA calls that can

modify the sbuf

IPC memory handle should be closed at Process 1 before the buffer is freed at Process 0

cudaIpcOpenEventhandle (&ipc_event, event_handle) cuMemGetAddressRange (&base_ptr, sbuf_ptr) Done sbuf_ptr rbuf_ptr

10

slide-11
SLIDE 11

Outline

  • Motivation
  • Problem Statement
  • Basics of CUDA IPC
  • CUDA IPC based Designs in MVAPICH2

– Two Sided Communication – One-sided Communication

  • Experimental Evaluation
  • Conclusion and Future Work

11

slide-12
SLIDE 12

Design of Two-sided Communication

  • MPI communication costs

– synchronization – data movement

  • Small message communication

– minimize synchronization overheads – pair-wise eager buffers for host-host communication – associated pair-wise IPC buffers on GPU – synchronization using CUDA Events

  • Large message communication

– minimize number for copies - rendezvous protocol – minimize memory mapping overheads using a mapping cache

12

slide-13
SLIDE 13

Design of One-sided Communication

13

  • Separates communication from synchronization
  • Window
  • Communication calls - put, get, accumulate
  • Synchronization calls

– active - fence, post-wait/start-complete – passive – lock-unlock – period between two synchronization calls is a communication epoch

  • IPC memory handles created and mapped during window creation
  • Put/Get implemented as cudaMemcpyAsync
  • Synchronization using CUDA Events
slide-14
SLIDE 14

Outline

  • Motivation
  • Problem Statement
  • Basics of CUDA IPC
  • CUDA IPC based Designs in MVAPICH2

– Two Sided Communication – One-sided Communication

  • Experimental Evaluation
  • Conclusion and Future Work

14

slide-15
SLIDE 15
  • Intel Westmere node

– 2 NVIDIA Tesla C2075 GPUs – Red Hat Linux 5.8 and CUDA Toolkit 4.1

  • MVAPICH/MVAPICH2 - High Performance MPI Library for IB,

10GigE/iWARP and RoCE

– Available since 2002 – Used by more than 1.930 organizations (HPC centers, Industries and Universities) in 68 countries – More than 111,000 downloads from OSU site directly – Empowering many TOP500 clusters

  • 5th ranked 73,278-core cluster (Tsubame 2.0) at Tokyo Institute of Technology
  • 7th ranked 111,104-core cluster (Pleiades) at NASA
  • 25th ranked 62,976-core cluster (Ranger) at TACC

– http://mvapich.cse.ohio-state.edu

Experimental Setup

15

slide-16
SLIDE 16

1000 2000 3000 4000 5000 6000 1 16 256 4K 64K 1M Bandwidth (MBps) Message Size (Bytes) 500 1000 1500 2000 4K 16K 64K 256K 1M 4M Latency (usec) Message Size (Bytes) 10 20 30 40 50 1 4 16 64 256 1K Latency (usec) Message Size (Bytes)

16

Two-sided Communication Performance

SHARED-MEM CUDA IPC

70% 46% 78% considerable

improvement in MPI performance due to host bypass

slide-17
SLIDE 17

1000 2000 3000 4000 5000 6000 1 16 256 4K 64K 1M Bandwidth (MBps) Message Size (Bytes) 10 20 30 40 50 2 8 32 128 512 Latency (usec) Message Size (Bytes)

17

One-sided Communication Performance

(get + active synchronization vs. send/recv)

SHARED-MEM-1SC CUDA-IPC-1SC CUDA-IPC-2SC

30% 27%

Better performance compared to two-sided semantics.

500 1000 1500 2000 4K 16K 64K 256K 1M 4M Latency (usec) Message Size (Bytes)

slide-18
SLIDE 18

18

One-sided Communication Performance

(get + passive synchronization)

SHARED-MEM CUDA IPC

100 200 300 400 500 600 100 200 300 Latency (usec) Target Busy Loop (usec)

true asynchronous progress

  • Lock + 8 Gets + Unlock with the target in a busy loop (128KB

messages)

slide-19
SLIDE 19

Lattice Boltzmann Method

20 40 60 80 100 120 140 256x256x64 256x512x64 512x512x64 LB Step Latency (msec) Dataset per GPU 2SIDED-SHARED-MEM 2SIDED-IPC 1SIDED-IPC

  • Computation fluid dynamics code with support for multi-phase flows

with large density ratios

  • Modified to use MPI communication from GPU device memory - one-

sided and two-sided semantics

  • Up to 16% improvement in per step

16%

19

slide-20
SLIDE 20

Outline

  • Motivation
  • Problem Statement
  • Basics of CUDA IPC
  • CUDA IPC based Designs in MVAPICH2

– Two Sided Communication – One-sided Communication

  • Experimental Evaluation
  • Conclusion and Future Work

20

slide-21
SLIDE 21

21

Conclusion and Future Work

  • Take advantage of CUDA IPC to improve MPI communication between GPUs
  • n a node
  • 70% improvement in latency and 78% improvement in bandwidth for two-

sided communication

  • One-sided communication gives better performance and allows for truly

asynchronous communication

  • 16% improvement in execution time of Lattice Boltzmann Method code
  • Studying the impact on other applications while exploiting computation-

communication overlap

  • Exploring efficient designs for inter-node one-sided communication on GPU

clusters

slide-22
SLIDE 22

Thank You!

{potluri, wangh, bureddy, singhas, panda} @cse.ohio-state.edu carlos@tacc.utexas.edu

Network-Based Computing Laboratory http://nowlab.cse.ohio-state.edu/ MVAPICH Web Page http://mvapich.cse.ohio-state.edu/

22