Performance of HPC Middleware over Infiniband WAN Designing - - PowerPoint PPT Presentation

Performance of HPC Middleware

• ver Infiniband WAN

Presented by: Ashish Kumar Singh

Designing Efficient FTP Mechanisms for High Performance Data –Transfer

• ver Infiniband

High Performance Data Transfer in Grid Environment Using GridFTP over Infiniband

Performance of HPC Middleware

• ver Infiniband WAN
• S. Narravula, H. Subramoni, P. Lai, R. Noronha and D.K.

Panda

Motivation

• Multi-Cluster needs of organizations
• Advent of long haul Infiniband (IB WAN)

– Infiniband range extenders like Intel Connects and Obsidian Longbows

• IB applications and libraries like, MPI, NFS over

RDMA, etc. developed for Intra-cluster environments

Contributions

• Analyzes the general communication

performance of HPC middleware

• Proposes basic design optimizations for

enhancing communication performance over WAN

• Demonstrates the potential benefits obtained

by enhancing internal protocols of middleware

IB Range Extension

• Obsidian Longbows provide range extension

for Infiniband fabrics over 10 Gigabits/s WAN

Verbs-level Performance (UD)

• UD does not involve any acknowledgements from the remote side
• UD is scalable with higher delays
• Higher level protocols need to take care of reliability and flow control mechanisms

Verbs-level Performance (RC)

• RC guarantees in-order delivery by ACKs and NACKs, which limits the number of messages

that can be in flight to a maximum supported window size

• Fewer large messages can fill the pipeline and so large messages are less effected

IPoIB Performance (UD)

• TCP needs larger window sizes to achieve good bandwidth
• More streams – more UD packets with independent flow control, so more outstanding

packets that can be pushed out from source at any given time frame

IPoIB Performance (RC)

• Advantage of RC transport mode over IPoIB is that RC can handle larger packet sizes.

Larger packet sizes can achieve better bandwidth and per byte TCP processing decreases

MPI-level Performance (Delay)

• Trends similar to basic verbs-level evaluation

MPI-level Performance (Tuning)

• Protocol choice changes for medium sized messages in high delay scenario
• Rendezvous protocol involves an additional message exchange

MPI-level Performance (Streams)

• For small messages, messaging rate increases proportionally with number of

communicating streams

• For higher delay networks, additional parallel streams are better for overall network

bandwidth utilization a) 100 us delay b) 1 ms delay c) 10 ms delay

MPI-level Performance (Collective)

• Simple optimized broadcast that performs the bcast operation hierarchially over the two

connected clusters, minimizing the traffic on the WAN

• For small messages, as the WAN link is able to handle all the traffic, the congestion is very

minor a) 10 us delay b) 100 us delay c) 1000 us delay

Conclusions

• Applications usually absorb smaller network

delay fairly well

• Many protocols get severely impacted in high

delay scenarios

• Protocols can be optimized for high delay

scenarios to improve the performance

• With long-haul IB WAN technology cluster-of-

clusters architecture for HPC systems is feasible

Designing Efficient FTP Mechanisms for High Performance Data – Transfer over Infiniband

• P. Lai, H. Subramoni, S. Narravula, A. Mamidala and D.K.

Panda

Motivation

• FTP - most popular method to transfer bulk

data

• Typically used in applications like data staging,

content replication and remote site backup

• Advent of long haul Infiniband (IB WAN) made

cluster-of-cluster architecture possible

• IPoIB and SDP lose significant native

performance

Possible Approaches

• Existing sockets based FTP through intermediate drivers (#1, #2 and #3). IPoIB

and SDP are the popular schemes for this choice.

• #4, new FTP mechanism using the Native IB features.

Performance of Communication Protocols

• Native IB verbs achieve much higher bandwidth as compared to other protocols.
• Performance for FTP, e.g., GridFTP, using IPoIB and SDP is even more worse.

Contributions

• Design an Advanced Data Transfer Service

(ADTS) that leverages zero-copy capabilities

• Leverage ADTS to design a high performance

zero-copy FTP library

• Provide a robust and inter-operable

mechanism to support zero-copy capable clients and the traditional TCP/UDP clients

• Performance study

FTP-ADTS Architecture

• Clients may be capable of performing zero-copy data transfer or only support the

TCP/UDP based communication.

• Once the transport protocol is negotiated , Data Connection Management

component initiates a connection.

Design of Zero-Copy Channel

• Memory Semantics using RDMA vs. Channel

semantics using Send-Recv

• Drawbacks of Memory Semantics:

– Pre-allocation, registration and communication of target RDMA buffers – Explicit flow control – Notification of completion – Latency benefits for small messages is marred by high network delay

Design of Zero-Copy Channel

• Advantages of Send-Recv Semantics:

– Identical zero-copy benefits – Simpler flow control, with use of SRQ – Sender is not throttled down due to lack of buffers

• n remote node

– Both RC and UD transports available

Design Enhancements

• Buffer/File Management component keeps a

small set of pre-allocated and registered buffers

• Memory Registration Cache and Persistent

Sessions

• Pipelined Data Transfers
• Prefork Server to handle bursts of requests

Performance

• Site Replication over IB WAN using FTP.
• FTP-ADTS speeds up data transfer by up to 65%.
• Much lesser CPU utilization.

Conclusions

• Existing TCP or UDP or SCTP based FTP

implementations are not suitable for WAN capable interconnects like IB WAN

• FTP-ADTS efficiently transfers data by

leveraging zero-copy operations of modern interconnects

High Performance Data Transfer in Grid Environment Using GridFTP

• ver Infiniband
• H. Subramoni, P. Lai, R. Kettimuthu and D.K. Panda

Overview

• GridFTP is a high-performance, secure, reliable

extension of the standard FTP optimized for WAN

• Globus XIO framework, used to design

GridFTP, offers easy-to-use interface

• The framework hides the complications of

communication semantics of underlying devices (network or disk)

Contribution

• Combining the ease of use of Globus XIO

framework and the high performance achieved through IB

• Enhancing the disk I/O performance of the

existing ADTS library

– By decoupling the network processing from disk I/O

• perations
• Evaluation of the design

– micro-benchmark level – applications like Community Climate System Model and ultra scale visualization

Design Issues

• Most HPC applications require movement of

huge amount of data

– Needs slower hard disks and RAIDs for storage – With low bandwidth provided by TCP/UDP based FTP, this was not an issue – Will be an issue for Globus ADTS XIO

• Solution

– decoupling of network from disk I/O

Design Changes in ADTS

• Introduction of :
• multiple threads (read, write and network thread)
• set of buffers to stage the data
• Read thread prefetches a set of locations from the disk and keeps it ready for the

network thread to send over the physical link

• How to avoid frequent context switches?
• Low and High Water Marks, High water mark is set to max size of circular buf
• Read only available buffers less than low-water mark

Application Level Improvements