Performance of RDMA-Capable Storage Performance of RDMA-Capable - PowerPoint PPT Presentation

Performance of RDMA-Capable Storage Performance of RDMA-Capable Storage Protocols on Wide-Area Network Protocols on Wide-Area Network Weikuan Yu Weikuan Yu Nageswara S.V. Rao Nageswara S.V. Rao Pete Wyckoff* Pete Wyckoff* Jeffrey S. Vetter Jeffrey S. Vetter Ohio Supercomputer Center* Supercomputer Center* Ohio Managed by UT-Battelle for the Department of Energy

InfiniBand Clusters around the World InfiniBand Clusters around the World SGI (US) CEA (France) Ranger (US) Tsubame (Japan) Dawning (China) EKA (India) 2 Managed by UT-Battelle for the Department of Energy PDSW'08, Austin, TX

The Problem of Computing Islands The Problem of Computing Islands • Islands of InfiniBand (IB) clusters Islands of InfiniBand (IB) clusters • – More IB clusters are deployed More IB clusters are deployed – – Some already connected, e.g. through Some already connected, e.g. through TeraGrid TeraGrid – • But only via TCP/IP protocols But only via TCP/IP protocols • • Data transfer across these islands Data transfer across these islands • – Need ever-greater data movement capabilities. – Need ever-greater data movement capabilities. – GridFTP, BBCP or other special storage configuration GridFTP, BBCP or other special storage configuration – – TCP performance on Long Distance can be low – TCP performance on Long Distance can be low • With 10GigE on UltraScience Net (no tuning) With 10GigE on UltraScience Net (no tuning) • – 9.2 Gbps at 0.2 mile 9.2 Gbps at 0.2 mile – – 8.2 Gbps at 1400 miles 8.2 Gbps at 1400 miles – – 2.3-2.5 Gbps at 6600+ miles 2.3-2.5 Gbps at 6600+ miles – 3 Managed by UT-Battelle for the Department of Energy PDSW'08, Austin, TX

RDMA (IB) in Clusters and Local Area Networks RDMA (IB) in Clusters and Local Area Networks Sub-microsecond latency Sub-microsecond latency • • Superb bandwidth (32Gbps with IB QDR) Superb bandwidth (32Gbps with IB QDR) • • Heavily used for clustering Heavily used for clustering • • Getting popular in storage environment Getting popular in storage environment • • – NFS over RDMA ( NFS over RDMA (NFSoRDMA NFSoRDMA) ) – – SCSI RDMA Protocol (SRP) – SCSI RDMA Protocol (SRP) – iSCSI over RDMA ( iSCSI over RDMA (iSER iSER) ) – Applications Applications MPI NFS/iSERI/SRP MPI NFS/iSERI/SRP Verbs Verbs InfiniBand HCA InfiniBand HCA 1 µ sec 4 Managed by UT-Battelle for the Department of Energy PDSW'08, Austin, TX

Sample Performance of RDMA-based Storage Sample Performance of RDMA-based Storage RDMA
enables
good
iSCSI
bandwidth
within
LAN RDMA
enables
good
iSCSI
bandwidth
within
LAN • • Nearly
doubled
the
performance
for
iSCSI Nearly
doubled
the
performance
for
iSCSI • • 5 Managed by UT-Battelle for the Department of Energy PDSW'08, Austin, TX

Feasibility of RDMA (IB) on WAN Feasibility of RDMA (IB) on WAN Long-range Extensions for InfiniBand available Long-range Extensions for InfiniBand available • • – Network Equipment Technologies (NET): NX5010 Network Equipment Technologies (NET): NX5010 – – Obsidian Research: Longbow Obsidian Research: Longbow – Long latency (10 4 Long latency (10 4 ~10 ~10 5 5 µ sec) µ sec) • • High bandwidth yet feasible High bandwidth yet feasible • • – – Good Good distance scalability and tolerance to interfering traffic distance scalability and tolerance to interfering traffic – Good network throughput and MPI-level Performance – Good network throughput and MPI-level Performance Can RDMA provide a good transport protocol for storage on WAN? Can RDMA provide a good transport protocol for storage on WAN? • • Applications Applications MPI NFS/iSERI/SRP MPI NFS/iSERI/SRP Verbs Verbs InfiniBand HCA InfiniBand HCA 10 4 ~10 5 µ sec 6 Managed by UT-Battelle for the Department of Energy PDSW'08, Austin, TX

Experimental Environment Experimental Environment Hardware Hardware • • – Long-range IB extension devices from NET (Network Equipment – Long-range IB extension devices from NET (Network Equipment Technologies, Inc) Technologies, Inc) – Mellanox PCI-Express 4x DDR HCAs HCAs (InfiniHost-III and Connect-X) (InfiniHost-III and Connect-X) – Mellanox PCI-Express 4x DDR Software Packages Software Packages • • – OFED-1.3 from openfabrics openfabrics.org .org – OFED-1.3 from – Linux-2.6.25 with Linux-2.6.25 with NFSoRDMA NFSoRDMA and and iSER iSER support support – Performance of RDMA-based Storage Protocols on WAN Performance of RDMA-based Storage Protocols on WAN • • – NFS over RDMA – NFS over RDMA – – iSCSI over RDMA iSCSI over RDMA 7 Managed by UT-Battelle for the Department of Energy PDSW'08, Austin, TX

UltraScience Net at ORNL UltraScience Net at ORNL Experimental WAN Network Experimental WAN Network • • – Oak Ridge, Atlanta, Chicago, Seattle, and Sunnyvale Atlanta, Chicago, Seattle, and Sunnyvale – Oak Ridge, – OC192 backbone connections – OC192 backbone connections – 4300 miles one way, 8600 miles loop-back – 4300 miles one way, 8600 miles loop-back 8 Managed by UT-Battelle for the Department of Energy PDSW'08, Austin, TX

RDMA-based Transport RDMA-based Transport Request
and
request
becomes
pure
control
messages, Request
and
request
becomes
pure
control
messages, • • and
have
to
travel
long
distance
on
WAN and
have
to
travel
long
distance
on
WAN Use
of
RDMA
read
(round‐trip
operations)
for
clients
to
write
data Use
of
RDMA
read
(round‐trip
operations)
for
clients
to
write
data • • Possible
additional
control
messages
for
NFSoRDMA
for
long
arguments Possible
additional
control
messages
for
NFSoRDMA
for
long
arguments • • Further
fragmentation
due
to
the
use
of
page‐based
operations Further
fragmentation
due
to
the
use
of
page‐based
operations • • 9 Managed by UT-Battelle for the Department of Energy PDSW'08, Austin, TX

RDMA on WAN RDMA on WAN RDMA
has
good
network‐level
performance
within
short
distance
WAN RDMA
has
good
network‐level
performance
within
short
distance
WAN • • High
bandwidth
at
long
distance
is
only
possible
for
large
messages High
bandwidth
at
long
distance
is
only
possible
for
large
messages • • Low
RDMA‐read
performance
for
page‐based
messages
(4KB),
even
at Low
RDMA‐read
performance
for
page‐based
messages
(4KB),
even
at • • 0.2
mile
when
using
InfiniHost‐III
 0.2
mile
when
using
InfiniHost‐III 
HCAs HCAs 10 Managed by UT-Battelle for the Department of Energy PDSW'08, Austin, TX

NFS over RDMA NFS over RDMA NFS
over
RDMA
achieves
good
 NFS
over
RDMA
achieves
good 
bandwidth
within
short
distance bandwidth
within
short
distance • • But
significant
optimizations
are
needed
for
long
distance But
significant
optimizations
are
needed
for
long
distance • • 11 Managed by UT-Battelle for the Department of Energy PDSW'08, Austin, TX

NFS - Large block size NFS - Large block size NFS
over
IPoIB‐CM
benefits
from
large
block
size NFS
over
IPoIB‐CM
benefits
from
large
block
size • • NFS
over
RDMA
needs
to
support
large
block
size
for
better
fit NFS
over
RDMA
needs
to
support
large
block
size
for
better
fit • • on
long‐distance
WAN on
long‐distance
WAN 12 Managed by UT-Battelle for the Department of Energy PDSW'08, Austin, TX

NFS over RDMA - using Connect-X NFS over RDMA - using Connect-X Better
RDMA
read
in
connect‐X
improves
 Better
RDMA
read
in
connect‐X
improves 
the
performance the
performance • • of
file
write
for
NFS
over
RDMA of
file
write
for
NFS
over
RDMA Performance
at
long
distance
is
yet
to
determine Performance
at
long
distance
is
yet
to
determine • • 13 Managed by UT-Battelle for the Department of Energy PDSW'08, Austin, TX

iSCSI over RDMA (iSER iSER) ) iSCSI over RDMA ( RDMA
enables
high‐performance
iSCSI
within
short
distance RDMA
enables
high‐performance
iSCSI
within
short
distance • • RDMA
has
good
promise
over
long
distance
as
shown
with
large RDMA
has
good
promise
over
long
distance
as
shown
with
large • • messages messages 14 Managed by UT-Battelle for the Department of Energy PDSW'08, Austin, TX