NFS over RDMA Brent Callaghan, Theresa Lingutla-Raj, Alex Chiu, - - PowerPoint PPT Presentation

NFS over RDMA 1 of 17 SIGCOMM 2003, NICELI Workshop

NFS over RDMA

Brent Callaghan, Theresa Lingutla-Raj, Alex Chiu, Peter Staubach, Omer Asad Sun Microsystems, Inc.

NFS

NFS over RDMA 2 of 17 SIGCOMM 2003, NICELI Workshop

Why RDMA as a Transport?

• Nice to have at 1 Gb/sec but

must have for 10 Gb/sec

• Offload protocol processing from general

purpose CPU to dedicated protocol hardware

• Offload host memory/IO bus with direct

data placement (DDP)

NFS over RDMA 3 of 17 SIGCOMM 2003, NICELI Workshop

NFS is an RDMA Sweet Spot

• Clients and servers are close

– Most commonly on a LAN – Often in the same server room or rack – Bandwidth high - latency low

• NFS moves big chunks of data

– 8 KB for NFS version 2 – No limit for NFS version 3

• Most clients read & write 32 KB chunks
• Solaris servers accept up to 1 MB reads/writes

NFS over RDMA 4 of 17 SIGCOMM 2003, NICELI Workshop

NFS

RDMA as a new RPC Transport

TCP UDP

RDMA

IP RPC/XDR

CHANGES

NFS NFS

NLM ACL

NFS over RDMA 5 of 17 SIGCOMM 2003, NICELI Workshop

Small RPC Messages

SEND SEND RPC Call RPC Reply

Small pre-posted receive buffer Small pre-posted receive buffer

• Most NFS messages are quite small

– Less than 1 KB

• No RDMA needed - just use SENDs

NFS over RDMA 6 of 17 SIGCOMM 2003, NICELI Workshop

Moving NFS data with RDMA

Data RPC Header

An NFS read reply or write request is a large chunk

• f data with a variable length RPC & NFS header.

That large chunk of data could be moved more efficiently if we could move it instead with DDP.

Stag Address Length

DDP Header

NFS over RDMA 7 of 17 SIGCOMM 2003, NICELI Workshop

XDR De-Chunking the Message

Chunk Chunk list entry XDR encoded RPC Message

XDR Offset Chunk Address

Non Chunks RDMA Send RDMA Read or Write TCP Conn

• Encoded message for TCP transport
• Encoded message for RDMA transport

NFS over RDMA 8 of 17 SIGCOMM 2003, NICELI Workshop

RDMA Transport Header

XID Version Message Type Chunk List

RPC Message sans chunks

XDR Stream Offset Chunk Length Source STag Source Address Next Chunk

NFS over RDMA 9 of 17 SIGCOMM 2003, NICELI Workshop

Read-Read Protocol

SEND SEND SEND READ READ RPC Call Arg chunks Result Chunks RPC Reply RPC Done

Message + Chunk list Message + Chunk list Free chunks Client Server

NFS over RDMA 10 of 17 SIGCOMM 2003, NICELI Workshop

NFS/TCP Throughput

Peak throughput 60 MB/sec @ 256 KB reads & 4 reads-ahead

NFS over RDMA 11 of 17 SIGCOMM 2003, NICELI Workshop

NFS/RDMA Throughput

Peak throughput 102 MB/sec @ 256 KB reads & 8 reads-ahead

NFS over RDMA 12 of 17 SIGCOMM 2003, NICELI Workshop

CPU Utilization

(with no async read-ahead)

NFS over RDMA 13 of 17 SIGCOMM 2003, NICELI Workshop

Further Work

• NFS/RDMA protocol Internet Drafts

submitted to IETF

• Extends basic “read-read” protocol to use RDMA

write with ULP hooks: “read-write”

• Includes receive buffer request/grant credit control
• Support for alignment padding in RDMA SENDs
• Receive buffer size negotiation protocol
• Support in NFS version 4.1

NFS over RDMA 14 of 17 SIGCOMM 2003, NICELI Workshop

Extended RDMA Transport Header

XID Version Message Type Chunk List XID Version Message Type Credits Read List Write List Reply Read List Write List Reply Threshold Alignment

Old Header Extended Header

Padding Control

Receive Buffer Credit Control Long replies Direct write from server

NFS over RDMA 15 of 17 SIGCOMM 2003, NICELI Workshop

Read-Write Protocol

SEND SEND READ RPC Call Arg chunks Result Chunks RPC Reply

Message + Write list Message + Write list

WRITE

Client Server

NFS over RDMA 16 of 17 SIGCOMM 2003, NICELI Workshop

Project Status

• Solaris prototype

– kVIPL with Emulex GN9000/VI, 1Gb link – Like a normal NFS mount – Demonstrated good performance

• Infiniband

– Implementing extended “read-write” protocol – Mellanox Tavor, 10 Gb (4x) link – Evaluating performance

NFS over RDMA 17 of 17 SIGCOMM 2003, NICELI Workshop