RPC-over-RDMA Version 2 Protocol

1.1. Design Goals

The general mission of RPC-over-RDMA transports is to leverage network hardware capabilities to reduce host CPU needs related to the transport of RPC messages. In particular, this includes mitigating host interrupt rates and limiting the necessity to copy RPC payload bytes on receivers.¶

These hardware capabilities benefit both RPC clients and servers. On balance, however, the RPC-over-RDMA protocol design approach has been to bolster clients more than servers, as the client is typically where applications are most hungry for CPU resources.¶

Additionally, RPC-over-RDMA transports are designed to support RPC applications transparently. However, such transports can also provide mechanisms that enable further optimization of data transfer when RPC applications are structured to exploit direct data placement. In this context, the Network File System (NFS) family of protocols (as described in [RFC1094], [RFC1813], [RFC7530], [RFC5661], [RFC7862], and subsequent NFSv4 minor versions) are all potential beneficiaries of RPC-over-RDMA.¶

A complete problem statement appears in [RFC5532].¶

1.2. Motivation for a New Version

Storage administrators have broadly deployed the RPC-over-RDMA version 1 protocol specified in [RFC8166]. However, there are known shortcomings to this protocol:¶

• The protocol's default size of Receive buffers forces the use of RDMA Read and Write transfers for small payloads, and limits the size of reverse-direction messages.¶
• It is difficult to make optimizations or protocol fixes that require changes to on-the-wire behavior.¶
• For some RPC procedures, the maximum reply size is difficult or impossible for an RPC client to estimate in advance.¶

To address these issues in a way that preserves interoperation with existing RPC-over-RDMA version 1 deployments, the current document presents an updated version of the RPC-over-RDMA transport protocol.¶

This version of RPC-over-RDMA is extensible, enabling the introduction of OPTIONAL extensions without impacting existing implementations. See Appendix C.1 for further discussion. It introduces a mechanism to exchange implementation properties to automatically provide further optimization of data transfer.¶

This version also contains incremental changes that relieve performance constraints and enable recovery from unusual corner cases. These changes are outlined in Appendix C and include a larger default inline threshold, the ability to convey a single RPC message using multiple RDMA Send operations, support for authentication of connection peers, richer error reporting, improved credit-based flow control, and support for Remote Invalidation.¶

3.1. Remote Procedure Calls

This section highlights critical elements of the RPC protocol [RFC5531] and the External Data Representation (XDR) [RFC4506] it uses. RPC-over-RDMA version 2 enables the transmission of RPC messges built using XDR and also uses XDR internally to describe its header format.¶

3.2. Remote Direct Memory Access

When a third party transfers large RPC payloads, RPC Requesters and Responders can become more efficient. An example of such a third party might be an intelligent network interface (data movement offload), which places data in the receiver's memory so that no additional adjustment of data alignment is necessary (direct data placement or "DDP"). RDMA transports enable both of these optimizations.¶

In the current document, the standalone term "RDMA" refers to the physical mechanism an RDMA transport utilizes when moving data.¶

4.1. Message Framing

Each RPC-over-RDMA version 2 message consists of at most two XDR streams:¶

• The "Transport stream" contains a header that describes and controls the transfer of the Payload stream in this RPC-over-RDMA message. Every RDMA Send on an RPC-over-RDMA version 2 connection MUST begin with a Transport stream.¶
• The "Payload stream" contains part or all of a single RPC message. The sender MAY divide an RPC message at any convenient boundary but MUST send RPC message fragments in XDR stream order and MUST NOT interleave Payload streams from multiple RPC messages.¶

The RPC-over-RDMA framing mechanism described in this section replaces all other RPC framing mechanisms. Connection peers use RPC-over-RDMA framing even when the underlying RDMA protocol runs on a transport type with well-defined RPC framing, such as TCP. However, a ULP can negotiate the use of RDMA, dynamically enabling the use of RPC-over-RDMA on a connection established on some other transport type. Because RPC framing delimits an entire RPC request or reply, the resulting shift in framing must occur between distinct RPC messages, and in concert with the underlying transport.¶

4.2. Managing Receiver Resources

If any pre-posted Receive buffer on the connection is not large enough to accept an incoming RDMA Send, the RDMA provider can terminate the connection. Likewise, if a pre-posted Receive buffer is not available to accept an incoming RDMA Send, the RDMA provider can terminate the connection. Therefore, a sender needs to respect the resource limits of its peer receiver to ensure the longevity of each connection. Two operational parameters communicate these limits between connection peers: credit grants, and inline threshold.¶

4.3. Using Direct Data Placement

RPC-over-RDMA version 2 provides a mechanism for moving part of an RPC message via a data transfer distinct from RDMA Send and Receive. For example, a sender can remove one or more XDR data items from the Payload stream. These items are then conveyed via other mechanisms, such as one or more RDMA Read or Write operations.¶

4.4. Encoding Chunks

The RPC-over-RDMA version 2 transport protocol does not place a limit on chunk size. However, each ULP may cap the amount of data that can be transferred by a single RPC transaction. For example, NFS implementations typically have settings that restrict the payload size of NFS READ and WRITE operations. The Responder can use such limits to sanity check chunk sizes before using them in RDMA operations.¶

4.5. Reverse-Direction Operation

The terminology used in this section is introduced in Section 3.1.3.3.¶

5.1. Transport Properties Model

The current document specifies a basic set of receiver and sender properties. Such properties are specified using a code point that identifies the particular transport property and a nominally opaque array containing the XDR encoding of the property.¶

The following XDR types handle transport properties:¶

<CODE BEGINS>

typedef rpcrdma2_propid uint32;

struct rpcrdma2_propval {
        rpcrdma2_propid rdma_which;
        opaque          rdma_data<>;
};

typedef rpcrdma2_propval rpcrdma2_propset<>;

typedef uint32 rpcrdma2_propsubset<>;


<CODE ENDS>

The rpcrdma2_propid type specifies a distinct transport property. The property code points are defined as const values rather than elements in an enum type to enable the extension by concatenating XDR definition files.¶

The rpcrdma2_propval type carries the value of a transport property. The rdma_which field identifies the particular property, and the rdma_data field contains the associated value of that property. A zero-length rdma_data field represents the default value of the property specified by rdma_which.¶

Although the rdma_data field is opaque, receivers interpret its contents using the XDR type associated with the property specified by rdma_which. When the contents of the rdma_data field do not conform to that XDR type, the receiver MUST return the error RDMA2_ERR_BAD_PROPVAL using the header type RDMA2_ERROR, as described in Section 6.3.1.¶

For example, the receiver of a message containing a valid rpcrdma2_propval returns this error if the length of rdma_data is greater than the length of the transferred message. Also, when the receiver recognizes the rpcrdma2_propid contained in rdma_which, it MUST report the error RDMA2_ERR_BAD_PROPVAL if either of the following occurs:¶

• The nominally opaque data within rdma_data is not valid when interpreted using the property-associated typedef.¶
• The length of rdma_data is insufficient to contain the data represented by the property-associated typedef.¶

A receiver does not report an error if it does not recognize the value contained in rdma_which. In that case, the receiver does not process that rpcrdma2_propval. Processing continues with the next rpcrdma2_propval, if any.¶

The rpcrdma2_propset type specifies a set of transport properties. The protocol does not impose a particular ordering of the rpcrdma2_propval items within it.¶

The rpcrdma2_propsubset type identifies a subset of the properties in a rpcrdma2_propset. Each bit in the mask denotes a particular element in a previously specified rpcrdma2_propset. If a particular rpcrdma2_propval is at position N in the array, then bit number N mod 32 in word N div 32 specifies whether the defined subset includes that particular rpcrdma2_propval. Words beyond the last one specified are assumed to contain zero.¶

5.2. Current Transport Properties

Table 1 specifies a basic set of transport properties. The columns contain the following information:¶

• The column labeled "Property" contains a name of the transport property described by the current row.¶
• The column labeled "Code" specifies the code point that identifies this property.¶
• The column labeled "XDR type" gives the XDR type of the data used to communicate the value of this property. This data type overlays the data portion of the nominally opaque rdma_data field.¶
• The column labeled "Default" gives the default value for the property.¶
• The column labeled "Section" indicates the section within the current document that explains the use of this property.¶

<CODE BEGINS>

const uint32 RDMA2_PROPID_SBSIZ = 1;
typedef uint32 rpcrdma2_prop_sbsiz;


<CODE ENDS>

<CODE BEGINS>

const uint32 RDMA2_PROPID_RBSIZ = 2;
typedef uint32 rpcrdma2_prop_rbsiz;


<CODE ENDS>

<CODE BEGINS>

const uint32 RDMA2_PROPID_RSSIZ = 3;
typedef uint32 rpcrdma2_prop_rssiz;


<CODE ENDS>

<CODE BEGINS>

const uint32 RDMA2_PROPID_RCSIZ = 4;
typedef uint32 rpcrdma2_prop_rcsiz;


<CODE ENDS>

<CODE BEGINS>

const uint32 RDMA_RVRSDIR_NONE = 0;
const uint32 RDMA_RVRSDIR_SIMPLE = 1;
const uint32 RDMA_RVRSDIR_CONT = 2;
const uint32 RDMA_RVRSDIR_GENL = 3;

const uint32 RDMA2_PROPID_BRS = 5;
typedef uint32 rpcrdma2_prop_brs;


<CODE ENDS>

<CODE BEGINS>

const uint32 RDMA2_PROPID_HOSTAUTH = 6;
typedef opaque rpcrdma2_prop_hostauth<>;


<CODE ENDS>

6.1. Transport Header Types

Table 2 lists the RPC-over-RDMA version 2 header types. The columns contain the following information:¶

• The column labeled "Operation" names the particular operation.¶
• The column labeled "Code" specifies the value of the header type for this operation.¶
• The column labeled "XDR type" gives the XDR type of the data structure used to organize the information in this new header type. This data immediately follows the universal portion on the transport header present in every RPC-over-RDMA transport header.¶
• The column labeled "Msg" indicates whether this operation is followed (or not) by an RPC message payload.¶
• The column labeled "Section" refers to the section within the current document that explains the use of this header type.¶

RPC-over-RDMA version 2 peers are REQUIRED to support all message header types in Table 2. RPC-over-RDMA version 2 implementations that receive an unrecognized header type MUST respond with an RDMA2_ERROR message with an rdma_err field containing RDMA2_ERR_INVAL_HTYPE and drop the incoming message without processing it further.¶

6.2. Headers and Chunks

Most RPC-over-RDMA version 2 data structures have antecedents in corresponding structures in RPC-over-RDMA version 1. As is typical for new versions of an existing protocol, the XDR data structures have new names, and there are a few small changes in content. In some cases, there have been structural re-organizations to enable protocol extensibility.¶

<CODE BEGINS>

struct rpcrdma_common {
             uint32         rdma_xid;
             uint32         rdma_vers;
             uint32         rdma_credit;
             uint32         rdma_htype;
};


<CODE ENDS>

6.3. Header Types

The header types defined and used in RPC-over-RDMA version 1 are not carried over into RPC-over-RDMA version 2, although there are easy equivalents to the version 1 procedures:¶

• The RDMA2_ERROR header (defined in Section 6.3.1) has an XDR definition that differs from that in RPC-over-RDMA version 1, and its modifications are all compatible extensions.¶
• Senders use RDMA2_CALL_INLINE or RDMA2_REPLY_FINAL (defined in Sections 6.3.7 and 6.3.10) in place of RDMA_MSG. There are minor differences in the on-the-wire format between the version 1 procedure and the version 2 header types.¶
• Senders use RDMA2_CALL_EXTERNAL or RDMA2_REPLY_EXTERNAL (defined in Sections 6.3.5 and 6.3.8) in place of RDMA_NOMSG. There are minor differences in the on-the-wire format between the version 1 procedure and the version 2 header types.¶
• RDMA2_CONNPROP_MIDDLE and RDMA2_CONNPROP_FINAL (defined in Sections 6.3.3 and 6.3.4) are new header types devoted to enabling connection peers to exchange information about their transport properties.¶

<CODE BEGINS>

const rpcrdma2_proc RDMA2_ERROR = 4;

struct rpcrdma2_err_vers {
        uint32 rdma_vers_low;
        uint32 rdma_vers_high;
};

struct rpcrdma2_err_write {
        uint32 rdma_chunk_index;
        uint32 rdma_length_needed;
};

union rpcrdma2_hdr_error switch (rpcrdma2_errcode rdma_err) {
        case RDMA2_ERR_VERS:
          rpcrdma2_err_vers rdma_vrange;
        case RDMA2_ERR_READ_CHUNKS:
          uint32 rdma_max_chunks;
        case RDMA2_ERR_WRITE_CHUNKS:
          uint32 rdma_max_chunks;
        case RDMA2_ERR_SEGMENTS:
          uint32 rdma_max_segments;
        case RDMA2_ERR_WRITE_RESOURCE:
          rpcrdma2_err_write rdma_writeres;
        case RDMA2_ERR_REPLY_RESOURCE:
          uint32 rdma_length_needed;
        default:
          void;
};


<CODE ENDS>

<CODE BEGINS>

const rpcrdma2_proc RDMA2_GRANT = 5;


<CODE ENDS>

<CODE BEGINS>

const rpcrdma2_proc RDMA2_CONNPROP_MIDDLE = 6;

struct rpcrdma2_hdr_connprop {
        rpcrdma2_propset rdma_props;
};


<CODE ENDS>

<CODE BEGINS>

const rpcrdma2_proc RDMA2_CONNPROP_FINAL = 7;

struct rpcrdma2_hdr_connprop {
        rpcrdma2_propset rdma_props;
};


<CODE ENDS>

<CODE BEGINS>

const rpcrdma2_proc RDMA2_CALL_EXTERNAL = 8;

struct rpcrdma2_hdr_call_external {
        uint32                      rdma_inv_handle;

        struct rpcrdma2_read_list   *rdma_call;
        struct rpcrdma2_read_list   *rdma_reads;
        struct rpcrdma2_write_list  *rdma_provisional_writes;
        struct rpcrdma2_write_chunk *rdma_provisional_reply;
};


<CODE ENDS>

<CODE BEGINS>

const rpcrdma2_proc RDMA2_CALL_MIDDLE = 9;

struct rpcrdma2_hdr_call_middle {
        uint32                      rdma_remaining;

        /* The rpc message starts here and continues
         * through the end of the transmission. */
        uint32                      rdma_rpc_first_word;
};


<CODE ENDS>

<CODE BEGINS>

const rpcrdma2_proc RDMA2_CALL_INLINE = 10;

struct rpcrdma2_hdr_call_inline {
        uint32                      rdma_inv_handle;

        struct rpcrdma2_read_list   *rdma_reads;
        struct rpcrdma2_write_list  *rdma_provisional_writes;
        struct rpcrdma2_write_chunk *rdma_provisional_reply;

        /* The rpc message starts here and continues
         * through the end of the transmission. */
        uint32                      rdma_rpc_first_word;
};


<CODE ENDS>

<CODE BEGINS>

const rpcrdma2_proc RDMA2_REPLY_EXTERNAL = 11;

struct rpcrdma2_hdr_reply_external {
        struct rpcrdma2_write_list  *rdma_writes;
        struct rpcrdma2_write_chunk *rdma_reply;
};


<CODE ENDS>

<CODE BEGINS>

const rpcrdma2_proc RDMA2_REPLY_MIDDLE = 12;

struct rpcrdma2_hdr_reply_middle {
        uint32                      rdma_remaining;

        /* The rpc message starts here and continues
         * through the end of the transmission. */
        uint32                      rdma_rpc_first_word;
};


<CODE ENDS>

<CODE BEGINS>

const rpcrdma2_proc RDMA2_REPLY_INLINE = 13;

struct rpcrdma2_hdr_reply_inline {
        struct rpcrdma2_write_list  *rdma_writes;

        /* The rpc message starts here and continues
         * through the end of the transmission. */
        uint32                      rdma_rpc_first_word;
};


<CODE ENDS>

6.4. Transport Header Prefix

The following prefix structure appears at the start of each RPC-over-RDMA version 2 transport header.¶

<CODE BEGINS>

struct rpcrdma2_hdr_prefix {
        struct rpcrdma_common       rdma_start;
};


<CODE ENDS>

6.5. Remote Invalidation

To solicit the use of Remote Invalidation, a Requester sets the value of the rdma_inv_handle field in an RPC Call's transport header to a non-zero value that matches one of the rdma_handle fields in that header. If the Responder may invalidate none of the rdma_handle values in the header conveying the Call, the Requester sets the RPC Call's rdma_inv_handle field to the value zero.¶

If the Responder chooses not to use remote invalidation for this particular RPC Reply, or the RPC Call's rdma_inv_handle field contains the value zero, the Responder simply uses RDMA Send to transmit the matching RPC reply. However, if the Responder chooses to use Remote Invalidation, it uses RDMA Send With Invalidate to transmit the RPC Reply. It MUST use the value in the corresponding Call's rdma_inv_handle field to construct the Send With Invalidate Work Request.¶

A Responder never uses a Send With Invalidate Work Request when sending a control plane header type. This includes the RDMA2_ERROR header type, the RDMA2_GRANT header type, the RDMA2_CONNPROP_MIDDLE header type, and the RDMA2_CONNPROP_FINAL header type.¶

6.6. Payload Formats

RPC-over-RDMA version 2 provides several ways, known as "payload formats", to convey an RPC-over-RDMA message. A sender chooses the payload format for each message based on several factors:¶

• The existence of DDP-eligible data items in the RPC message payload¶
• The size of the RPC message payload¶
• The direction of the RPC message (i.e., Call or Reply)¶
• The available hardware resources¶
• The arrangement of source and sink memory buffers¶

The following subsections describe in detail how Requesters and Responders format RPC-over-RDMA message payloads.¶

     Requester                                 Responder
         |      RDMA Send (RDMA2_CALL_INLINE)      |
    Call |   ---------------------------------->   |
         |                                         |
         |                                         | Processing
         |                                         |
         |      RDMA Send (RDMA2_REPLY_INLINE)     |
         |   <----------------------------------   | Reply

     Requester                                 Responder
         |      RDMA Send (RDMA2_CALL_INLINE)      |
    Call |   ---------------------------------->   |
         |               RDMA Read                 |
         |   <----------------------------------   |
         |         RDMA Response (arg data)        |
         |   ---------------------------------->   |
         |                                         |
         |                                         | Processing
         |                                         |
         |      RDMA Send (RDMA2_REPLY_INLINE)     |
         |   <----------------------------------   | Reply

     Requester                                 Responder
         |      RDMA Send (RDMA2_CALL_INLINE)      |
    Call |   ---------------------------------->   |
         |                                         |
         |                                         | Processing
         |                                         |
         |         RDMA Write (result data)        |
         |   <----------------------------------   |
         |      RDMA Send (RDMA2_REPLY_INLINE)     |
         |   <----------------------------------   | Reply

     Requester                                 Responder
         |      RDMA Send (RDMA2_CALL_MIDDLE)      |
    Call |   ---------------------------------->   |
         |      RDMA Send (RDMA2_CALL_MIDDLE)      |
         |   ---------------------------------->   |
         |      RDMA Send (RDMA2_CALL_INLINE)      |
         |   ---------------------------------->   |
         |                                         |
         |                                         | Processing
         |                                         |
         |      RDMA Send (RDMA2_REPLY_MIDDLE)     |
         |   <----------------------------------   | Reply
         |      RDMA Send (RDMA2_REPLY_MIDDLE)     |
         |   <----------------------------------   |
         |      RDMA Send (RDMA2_REPLY_INLINE)     |
         |   <----------------------------------   |

     Requester                                 Responder
         |      RDMA Send (RDMA2_CALL_MIDDLE)      |
    Call |   ---------------------------------->   |
         |      RDMA Send (RDMA2_CALL_MIDDLE)      |
         |   ---------------------------------->   |
         |      RDMA Send (RDMA2_CALL_INLINE)      |
         |   ---------------------------------->   |
         |              RDMA Read                  |
         |   <----------------------------------   |
         |         RDMA Response (arg data)        |
         |   ---------------------------------->   |
         |                                         |
         |                                         | Processing
         |                                         |
         |      RDMA Send (RDMA2_REPLY_INLINE)     |
         |   <----------------------------------   | Reply

     Requester                                 Responder
         |      RDMA Send (RDMA2_CALL_INLINE)      |
    Call |   ---------------------------------->   |
         |                                         |
         |                                         | Processing
         |                                         |
         |         RDMA Write (result data)        |
         |   <----------------------------------   |
         |      RDMA Send (RDMA2_REPLY_MIDDLE)     |
         |   <----------------------------------   | Reply
         |      RDMA Send (RDMA2_REPLY_MIDDLE)     |
         |   <----------------------------------   |
         |      RDMA Send (RDMA2_REPLY_INLINE)     |
         |   <----------------------------------   |

     Requester                                 Responder
         |     RDMA Send (RDMA2_CALL_EXTERNAL)     |
    Call |   ---------------------------------->   |
         |               RDMA Read                 |
         |   <----------------------------------   |
         |         RDMA Response (RPC call)        |
         |   ---------------------------------->   |
         |                                         |
         |                                         | Processing
         |                                         |
         |      RDMA Send (RDMA2_REPLY_INLINE)     |
         |   <----------------------------------   | Reply

     Requester                                 Responder
         |      RDMA Send (RDMA2_CALL_INLINE)      |
    Call |   ---------------------------------->   |
         |                                         |
         |                                         | Processing
         |                                         |
         |          RDMA Write (RPC reply)         |
         |   <----------------------------------   |
         |     RDMA Send (RDMA2_REPLY_EXTERNAL)    |
         |   <----------------------------------   | Reply

7.1. Basic Transport Stream Parsing Errors

7.2. XDR Errors

A receiver might encounter an XDR parsing error that prevents it from processing an ingress Transport stream. Examples of such errors include:¶

• The value of the rdma_xid field does not match the value of the XID field in the accompanying RPC message.¶
• The receive buffer ends before the end of a data object contained in the Transport stream.¶

Moreover, when a Responder receives a valid RPC-over-RDMA header but the Responder's ULP implementation cannot parse the RPC arguments in the RPC Call, the Responder returns an RPC Reply with status GARBAGE_ARGS, using an RDMA2_REPLY_INLINE message type. This type of parsing failure might be due to mismatches between chunk sizes or offsets and the contents of the Payload stream, for example. In this case, the error is permanent, but the Requester has no way to know how much processing the Responder has completed for this RPC transaction.¶

7.3. Responder RDMA Operational Errors

In RPC-over-RDMA version 2, the Responder initiates RDMA Read and Write operations that target the Requester's memory. Problems might arise as the Responder attempts to use Requester-provided resources for RDMA operations. For example:¶

• Usually, chunks can be validated only by using their contents to perform data transfers. If chunk contents are invalid (e.g., a memory region is no longer registered or a chunk length exceeds the end of the registered memory region), a Remote Access Error occurs.¶
• If a Requester's Receive buffer is too small, the Responder's Send operation completes with a Local Length Error.¶
• If the Requester-provided Reply chunk is too small to accommodate a large RPC Reply message, a Remote Access Error occurs. A Responder might detect this problem before attempting to write past the end of the Reply chunk.¶

RDMA operational errors can be fatal to the connection. To avoid a retransmission loop and repeated connection loss that deadlocks the connection, once the Requester has re-established a connection, the Responder SHOULD send an RDMA2_ERROR response to indicate that no RPC-level reply is possible for that transaction.¶

7.4. Other Operational Errors

While a Requester is constructing an RPC Call message, an unrecoverable problem might occur that prevents the Requester from posting further RDMA Work Requests on behalf of that message. As with other transports, if a Requester is unable to construct and transmit an RPC Call, the associated RPC transaction fails immediately.¶

After a Requester has received a Reply, if it is unable to invalidate a memory region due to an unrecoverable problem, the Requester MUST close the connection to protect that memory from Responder access before the associated RPC transaction is complete.¶

While a Responder is constructing an RPC Reply message or error message, an unrecoverable problem might occur that prevents the Responder from posting further RDMA Work Requests on behalf of that message. If a Responder is unable to construct and transmit an RPC Reply or RPC-over-RDMA error message, the Responder MUST close the connection to signal to the Requester that a reply was lost.¶

7.5. RDMA Transport Errors

The RDMA connection and physical link provide some degree of error detection and retransmission. The Marker PDU Aligned Framing (MPA) protocol (as described in Section 7.1 of [RFC5044]) as well as the InfiniBand link layer [IBA] provide Cyclic Redundancy Check (CRC) protection of RDMA payloads. CRC-class protection is a general attribute of such transports.¶

Additionally, the RPC layer itself can accept errors from the transport and recover via retransmission. RPC recovery can typically handle complete loss and re-establishment of a transport connection.¶

The details of reporting and recovery from RDMA link-layer errors are described in specific link-layer APIs and operational specifications and are outside the scope of this protocol specification. See Section 11 for further discussion of RPC-level integrity schemes.¶

8.1. Code Component License

Code Components extracted from the current document must include the following license text. When combining the extracted XDR code with other XDR code which has an identical license, only a single copy of the license text needs to be retained.¶

<CODE BEGINS>

/// /*
///  * Copyright (c) 2010, 2020 IETF Trust and the persons
///  * identified as authors of the code.  All rights reserved.
///  *
///  * The authors of the code are:
///  * B. Callaghan, T. Talpey, C. Lever, and D. Noveck.
///  *
///  * Redistribution and use in source and binary forms, with
///  * or without modification, are permitted provided that the
///  * following conditions are met:
///  *
///  * - Redistributions of source code must retain the above
///  *   copyright notice, this list of conditions and the
///  *   following disclaimer.
///  *
///  * - Redistributions in binary form must reproduce the above
///  *   copyright notice, this list of conditions and the
///  *   following disclaimer in the documentation and/or other
///  *   materials provided with the distribution.
///  *
///  * - Neither the name of Internet Society, IETF or IETF
///  *   Trust, nor the names of specific contributors, may be
///  *   used to endorse or promote products derived from this
///  *   software without specific prior written permission.
///  *
///  *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS
///  *   AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED
///  *   WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
///  *   IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
///  *   FOR A PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO
///  *   EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
///  *   LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
///  *   EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
///  *   NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
///  *   SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
///  *   INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
///  *   LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
///  *   OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
///  *   IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
///  *   ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
///  */
///


<CODE ENDS>

8.2. Extraction of the XDR Definition

Implementers can apply the following sed script to the current document to produce a machine-readable XDR description of the base RPC-over-RDMA version 2 protocol.¶

<CODE BEGINS>

sed -n -e 's:^ */// ::p' -e 's:^ *///$::p'


<CODE ENDS>

That is, if this document is in a file called "spec.txt", then implementers can do the following to extract an XDR description file and store it in the file rpcrdma-v2.x.¶

<CODE BEGINS>

sed -n -e 's:^ */// ::p' -e 's:^ *///$::p' \
 < spec.txt > rpcrdma-v2.x


<CODE ENDS>

Although this file is a usable description of the base protocol, when extensions are to be supported, it may be desirable to divide the description into multiple files. The following script achieves that purpose:¶

<CODE BEGINS>

#!/usr/local/bin/perl
open(IN,"rpcrdma-v2.x");
open(OUT,">temp.x");
while(<IN>)
{
  if (m/FILE ENDS: (.*)$/)
    {
      close(OUT);
      rename("temp.x", $1);
      open(OUT,">temp.x");
    }
    else
    {
      print OUT $_;
    }
}
close(IN);
close(OUT);


<CODE ENDS>

Running the above script results in two files:¶

• The file common.x, containing the license plus the shared XDR definitions that need to be made available to both the base protocol and any subsequent extensions.¶
• The file baseops.x containing the XDR definitions for the base protocol defined in this document.¶

Extensions to RPC-over-RDMA version 2, published as Standards Track documents, should have similarly structured XDR definitions. Once an implementer has extracted the XDR for all desired extensions and the base XDR definition contained in the current document, she can concatenate them to produce a consolidated XDR definition that reflects the set of extensions selected for her RPC-over-RDMA version 2 implementation.¶

Alternatively, the XDR descriptions can be compiled separately. In that case, the combination of common.x and baseops.x defines the base transport. The combination of common.x and the XDR description of each extension produces a full XDR definition of that extension.¶

8.3. XDR Definition for RPC-over-RDMA Version 2 Core Structures

<CODE BEGINS>

/// /***************************************************************
///  *    Transport Header Prefixes
///  ***************************************************************/
///
/// struct rpcrdma_common {
///         uint32         rdma_xid;
///         uint32         rdma_vers;
///         uint32         rdma_credit;
///         uint32         rdma_htype;
/// };
///
/// struct rpcrdma2_hdr_prefix {
///         struct rpcrdma_common       rdma_start;
/// };
///
/// /***************************************************************
///  *    Chunks and Chunk Lists
///  ***************************************************************/
///
/// struct rpcrdma2_segment {
///         uint32 rdma_handle;
///         uint32 rdma_length;
///         uint64 rdma_offset;
/// };
///
/// struct rpcrdma2_read_segment {
///         uint32                  rdma_position;
///         struct rpcrdma2_segment rdma_target;
/// };
///
/// struct rpcrdma2_read_list {
///         struct rpcrdma2_read_segment rdma_entry;
///         struct rpcrdma2_read_list    *rdma_next;
/// };
///
/// struct rpcrdma2_write_chunk {
///         struct rpcrdma2_segment rdma_target<>;
/// };
///
/// struct rpcrdma2_write_list {
///         struct rpcrdma2_write_chunk rdma_entry;
///         struct rpcrdma2_write_list  *rdma_next;
/// };
///
/// /***************************************************************
///  *    Transport Properties
///  ***************************************************************/
///
/// /*
///  * Types for transport properties model
///  */
/// typedef rpcrdma2_propid uint32;
///
/// struct rpcrdma2_propval {
///         rpcrdma2_propid rdma_which;
///         opaque          rdma_data<>;
/// };
///
/// typedef rpcrdma2_propval rpcrdma2_propset<>;
/// typedef uint32 rpcrdma2_propsubset<>;
///
/// /*
///  * Transport propid values for basic properties
///  */
/// const RDMA2_PROPID_SBSIZ = 1;
/// const RDMA2_PROPID_RBSIZ = 2;
/// const RDMA2_PROPID_RSSIZ = 3;
/// const RDMA2_PROPID_RCSIZ = 4;
/// const RDMA2_PROPID_BRS = 5;
/// const RDMA2_PROPID_HOSTAUTH = 6;
///
/// /*
///  * Types specific to particular properties
///  */
/// typedef uint32 rpcrdma2_prop_sbsiz;
/// typedef uint32 rpcrdma2_prop_rbsiz;
/// typedef uint32 rpcrdma2_prop_rssiz;
/// typedef uint32 rpcrdma2_prop_rcsiz;
/// typedef uint32 rpcrdma2_prop_brs;
/// typedef opaque rpcrdma2_prop_hostauth<>;
///
/// const RDMA2_RVRSDIR_NONE = 0;
/// const RDMA2_RVRSDIR_SIMPLE = 1;
/// const RDMA2_RVRSDIR_CONT = 2;
/// const RDMA2_RVRSDIR_GENL = 3;
///
/// /* FILE ENDS: common.x; */



<CODE ENDS>

8.4. XDR Definition for RPC-over-RDMA Version 2 Base Header Types

<CODE BEGINS>

/// /***************************************************************
///  *    Descriptions of RPC-over-RDMA Header Types
///  ***************************************************************/
///
/// /*
///  * Header Type Codes: Control plane operations.
///  */
/// const RDMA2_ERROR = 4;
/// const RDMA2_GRANT = 5;
/// const RDMA2_CONNPROP_MIDDLE = 6;
/// const RDMA2_CONNPROP_FINAL = 7;
///
/// /*
///  * Header Type Codes: Call messages.
///  */
/// const RDMA2_CALL_EXTERNAL = 8;
/// const RDMA2_CALL_MIDDLE = 9;
/// const RDMA2_CALL_INLINE = 10;
///
/// /*
///  * Header Type Codes: Reply messages.
///  */
/// const RDMA2_REPLY_EXTERNAL = 11;
/// const RDMA2_REPLY_MIDDLE = 12;
/// const RDMA2_REPLY_INLINE = 13;
///
/// /*
///  * Header Type to Report Errors.
///  */
/// const RDMA2_ERR_VERS = 1;
/// const RDMA2_ERR_BAD_XDR = 2;
/// const RDMA2_ERR_BAD_PROPVAL = 3;
/// const RDMA2_ERR_INVAL_HTYPE = 4;
/// const RDMA2_ERR_INVAL_CONT = 5;
/// const RDMA2_ERR_READ_CHUNKS = 6;
/// const RDMA2_ERR_WRITE_CHUNKS = 7;
/// const RDMA2_ERR_SEGMENTS = 8;
/// const RDMA2_ERR_WRITE_RESOURCE = 9;
/// const RDMA2_ERR_REPLY_RESOURCE = 10;
/// const RDMA2_ERR_SYSTEM = 100;
///
/// struct rpcrdma2_err_vers {
///         uint32 rdma_vers_low;
///         uint32 rdma_vers_high;
/// };
///
/// struct rpcrdma2_err_write {
///         uint32 rdma_chunk_index;
///         uint32 rdma_length_needed;
/// };
///
/// union rpcrdma2_hdr_error switch (rpcrdma2_errcode rdma_err) {
///         case RDMA2_ERR_VERS:
///           rpcrdma2_err_vers rdma_vrange;
///         case RDMA2_ERR_READ_CHUNKS:
///           uint32 rdma_max_chunks;
///         case RDMA2_ERR_WRITE_CHUNKS:
///           uint32 rdma_max_chunks;
///         case RDMA2_ERR_SEGMENTS:
///           uint32 rdma_max_segments;
///         case RDMA2_ERR_WRITE_RESOURCE:
///           rpcrdma2_err_write rdma_writeres;
///         case RDMA2_ERR_REPLY_RESOURCE:
///           uint32 rdma_length_needed;
///         default:
///           void;
/// };
///
/// /*
///  * Header Type to Exchange Transport Properties.
///  */
/// struct rpcrdma2_hdr_connprop {
///         rpcrdma2_propset rdma_props;
/// };
///
/// /*
///  * Header Types to Convey RPC Messages.
///  */
/// struct rpcrdma2_hdr_call_external {
///         uint32                      rdma_inv_handle;
///
///         struct rpcrdma2_read_list   *rdma_call;
///         struct rpcrdma2_read_list   *rdma_reads;
///         struct rpcrdma2_write_list  *rdma_provisional_writes;
///         struct rpcrdma2_write_chunk *rdma_provisional_reply;
/// };
///
/// struct rpcrdma2_hdr_call_middle {
///         uint32                      rdma_remaining;
///
///         /* The rpc message starts here and continues
///          * through the end of the transmission. */
///         uint32                      rdma_rpc_first_word;
/// };
///
/// struct rpcrdma2_hdr_call_inline {
///         uint32                      rdma_inv_handle;
///
///         struct rpcrdma2_read_list   *rdma_reads;
///         struct rpcrdma2_write_list  *rdma_provisional_writes;
///         struct rpcrdma2_write_chunk *rdma_provisional_reply;
///
///         /* The rpc message starts here and continues
///          * through the end of the transmission. */
///         uint32                      rdma_rpc_first_word;
/// };
///
/// struct rpcrdma2_hdr_reply_external {
///         struct rpcrdma2_write_list  *rdma_writes;
///         struct rpcrdma2_write_chunk *rdma_reply;
/// };
///
/// struct rpcrdma2_hdr_reply_middle {
///         uint32                      rdma_remaining;
///
///         /* The rpc message starts here and continues
///          * through the end of the transmission. */
///         uint32                      rdma_rpc_first_word;
/// };
///
/// struct rpcrdma2_hdr_reply_inline {
///         struct rpcrdma2_write_list  *rdma_writes;
///
///         /* The rpc message starts here and continues
///          * through the end of the transmission. */
///         uint32                      rdma_rpc_first_word;
/// };
///
/// /* FILE ENDS: baseops.x; */



<CODE ENDS>

8.5. Use of the XDR Description

The files common.x and baseops.x, when combined with the XDR descriptions for extension defined later, produce a human-readable and compilable description of the RPC-over-RDMA version 2 protocol with the included extensions.¶

Although this XDR description can generate encoders and decoders for the Transport and Payload streams, there are elements of the operation of RPC-over-RDMA version 2 that cannot be expressed within the XDR language. Implementations that use the output of an automated XDR processor need to provide additional code to bridge these gaps.¶

• The Transport stream is not a single XDR object. Instead, the header prefix is one XDR data item, and the rest of the header is a separate XDR data item. Table 2 expresses the mapping between the header type in the header prefix and the XDR object representing the header type.¶
• The relationship between the Transport stream and the Payload stream is not specified using XDR. Comments within the XDR text make clear where transported messages, described by their own XDR definitions, need to appear. Such data is opaque to the transport.¶
• Continuation of RPC messages across transport message boundaries requires that message assembly facilities not specifiable within XDR are part of transport implementations.¶
• Transport properties are constant integer values. Table 1 expresses the mapping between each property's code point and the XDR typedef that represents the structure of the property's value. XDR does not possess the facility to express that mapping in an extensible way.¶

The role of XDR in RPC-over-RDMA specifications is more limited than for protocols where the totality of the protocol is expressible within XDR. XDR lacks the facility to represent the embedding of XDR-encoded payload material. Also, the need to cleanly accommodate extensions has meant that those using rpcgen in their applications need to take an active role to provide the facilities that cannot be expressed within XDR.¶

11.1. Memory Protection

A primary consideration is the protection of the integrity and confidentiality of host memory by an RPC-over-RDMA transport. The use of an RPC-over-RDMA transport protocol MUST NOT introduce vulnerabilities to system memory contents nor memory owned by user processes. Any RDMA provider used for RPC transport MUST conform to the requirements of [RFC5042] to satisfy these protections.¶

11.2. RPC Message Security

ONC RPC provides cryptographic security via the RPCSEC_GSS framework [RFC7861]. RPCSEC_GSS implements message authentication (rpc_gss_svc_none), per-message integrity checking (rpc_gss_svc_integrity), and per-message confidentiality (rpc_gss_svc_privacy) in a layer above the RPC-over-RDMA transport. The integrity and privacy services require significant computation and movement of data on each endpoint host. Some performance benefits enabled by RDMA transports can be lost.¶

11.3. Transport Properties

Like other fields that appear in the Transport stream, transport properties are sent in the clear with no integrity protection, making them vulnerable to man-in-the-middle attacks.¶

For example, if a man-in-the-middle were to change the value of the Receive buffer size, it could reduce connection performance or trigger loss of connection. Repeated connection loss can impact performance or even prevent a new connection from being established. The recourse is to deploy on a private network or use transport layer encryption.¶

11.4. Host Authentication

[ cel: This subsection is unfinished. ]¶

Wherein we use the relevant sections of [RFC3552] to analyze the addition of host authentication to this RPC-over-RDMA transport.¶

The authors refer readers to Appendix C of [RFC8446] for information on how to design and test a secure authentication handshake implementation.¶

A.1. DDP-Eligibility

A ULB designates specific XDR data items as eligible for DDP. As a sender constructs an RPC-over-RDMA message, it can remove DDP-eligible data items from the Payload stream so that the RDMA provider can place them directly in the receiver's memory. An XDR data item should be considered for DDP-eligibility if there is a clear benefit to moving the contents of the item directly from the sender's memory to the receiver's memory.¶

Criteria for DDP-eligibility include:¶

• The XDR data item is frequently sent or received, and its size is often much larger than typical inline thresholds.¶
• If the XDR data item is a result, its maximum size must be predictable in advance by the Requester.¶
• Transport-level processing of the XDR data item is not needed. For example, the data item is an opaque byte array, which requires no XDR encoding and decoding of its content.¶
• The content of the XDR data item is sensitive to address alignment. For example, a data copy operation would be required on the receiver to enable the message to be parsed correctly, or to enable the data item to be accessed.¶
• The XDR data item itself does not contain DDP-eligible data items.¶

In addition to defining the set of data items that are DDP-eligible, a ULB may limit the use of chunks to particular upper-layer procedures. If more than one data item in a procedure is DDP-eligible, the ULB may limit the number of chunks that a Requester can provide for a particular upper-layer procedure.¶

Senders never reduce data items that are not DDP-eligible. Such data items can, however, be part of a Special Format payload.¶

The programming interface by which an upper-layer implementation indicates the DDP-eligibility of a data item to the RPC transport is not described by this specification. The only requirements are that the receiver can re-assemble the transmitted RPC-over-RDMA message into a valid XDR stream and that DDP-eligibility rules specified by the ULB are respected.¶

There is no provision to express DDP-eligibility within the XDR language. The only definitive specification of DDP-eligibility is a ULB.¶

In general, a DDP-eligibility violation occurs when:¶

• A Requester reduces a non-DDP-eligible argument data item. The Responder reports the violation as described in Section 6.3.1.¶
• A Responder reduces a non-DDP-eligible result data item. The Requester terminates the pending RPC transaction and reports an appropriate permanent error to the RPC consumer.¶
• A Responder does not reduce a DDP-eligible result data item into an available Write chunk. The Requester terminates the pending RPC transaction and reports an appropriate permanent error to the RPC consumer.¶

A.2. Maximum Reply Size

When expecting small and moderately-sized Replies, a Requester should rely on Message Continuation rather than provision a Reply chunk. For each ULP procedure where there is no clear Reply size maximum and the maximum can be substantial, the ULB should specify a dependable means for determining the maximum Reply size.¶

A.3. Reverse-Direction Operation

The direction of operation does not preclude the need for DDP-eligibility statements.¶

Reverse-direction operation occurs on an already-established connection. Specification of RPC binding parameters is usually not necessary in this case.¶

Other considerations may apply when distinct RPC Programs share an RPC-over-RDMA transport connection concurrently.¶

A.4. Additional Considerations

There may be other details provided in a ULB.¶

• A ULB may recommend inline threshold values or other transport-related parameters for RPC-over-RDMA version 2 connections bearing that ULP.¶
• A ULP may provide a means to communicate transport-related parameters between peers.¶
• Multiple ULPs may share a single RPC-over-RDMA version 2 connection when their ULBs allow the use of RPC-over-RDMA version 2 and the rpcbind port assignments for those protocols permit connection sharing. In this case, the same transport parameters (such as inline threshold) apply to all ULPs using that connection.¶

Each ULB needs to be designed to allow correct interoperation without regard to the transport parameters actually in use. Furthermore, implementations of ULPs must be designed to interoperate correctly regardless of the connection parameters in effect on a connection.¶

A.5. ULP Extensions

An RPC Program and Version tuple may be extensible. For instance, the RPC version number may not reflect a ULP minor versioning scheme, or the ULP may allow the specification of additional features after the publication of the original RPC Program specification. ULBs are provided for interoperable RPC Programs and Versions by extending existing ULBs to reflect the changes made necessary by each addition to the existing XDR.¶

[ cel: The final sentence is unclear, and may be inaccurate. I believe I copied this section directly from RFC 8166. Is there more to be said, now that we have some experience? ]¶

B.1. Documentation Requirements

As described earlier, a Standards Track document introduces a set of new protocol elements. Together these elements are considered an OPTIONAL feature. Each implementation is either aware of all the protocol elements introduced by that feature or is aware of none of them.¶

Documents specifying extensions to RPC-over-RDMA version 2 should contain:¶

• An explanation of the purpose and use of each new protocol element.¶
• An XDR description including all of the new protocol elements, and a script to extract it.¶
• A discussion of interactions with other extensions. This discussion includes requirements for other OPTIONAL features to be present, or that a particular level of support for an OPTIONAL facility is required.¶

Implementers combine the XDR descriptions of the new features they intend to use with the XDR description of the base protocol in the current document. This combination is necessary to create a valid XDR input file because extensions are free to use XDR types defined in the base protocol, and later extensions may use types defined by earlier extensions.¶

The XDR description for the RPC-over-RDMA version 2 base protocol combined with that for any selected extensions should provide a human-readable and compilable definition of the extended protocol.¶

B.2. Adding New Header Types to RPC-over-RDMA Version 2

New transport header types are defined similar to Sections 6.3.5 through 6.3.10. In particular, what is needed is:¶

• A description of the function and use of the new header type.¶
• A complete XDR description of the new header type.¶
• A description of how receivers report errors, including mechanisms for reporting errors outside the available choices already available in the base protocol or other extensions.¶
• An indication of whether a Payload stream must be present, and a description of its contents and how receivers use such Payload streams to reconstruct RPC messages.¶
• As appropriate, a statement of whether a Responder may use Remote Invalidation when sending messages that contain the new header type.¶

There needs to be additional documentation that is made necessary due to the OPTIONAL status of new transport header types:¶

• The document should discuss constraints on support for the new header types. For example, if support for one header type is implied or foreclosed by another one, this needs to be documented.¶
• The document should describe the preferred method by which a sender determines whether its peer supports a particular header type. It is always possible to send a test invocation of a particular header type to see if support is available. However, when more efficient means are available (e.g., the value of a transport property), this should be noted.¶

B.3. Adding New Transport properties to the Protocol

A Standards Track document defining a new transport property should include the following information paralleling that provided in this document for the transport properties defined herein:¶

• The rpcrdma2_propid value identifying the new property.¶
• The XDR typedef specifying the structure of its property value.¶
• A description of the new property.¶
• An explanation of how the receiver can use this information.¶
• The default value if a peer never receives the new property.¶

There is no requirement that propid assignments occur in a continuous range of values. Implementations should not rely on all such values being small integers.¶

Before the defining Standards Track document is published, the nfsv4 Working Group should select a unique propid value, and ensure that:¶

• rpcrdma2_propid values specified in the document do not conflict with those currently assigned or in use by other pending working group documents defining transport properties.¶

• rpcrdma2_propid values specified in the document do not conflict with the range reserved for experimental use, as defined in Section 8.2.¶

  [ cel: There is no longer a section 8.2 or an experimental range of propid values. Should we request the creation of an IANA registry for propid values? ].¶

When a Standards Track document proposes additional transport properties, reviewers should deal with possible security issues exposed by those new transport properties.¶

B.4. Adding New Error Codes to the Protocol

The same Standards Track document that defines a new header type may introduce new error codes used to support it. A Standards Track document may similarly define new error codes that an existing header type can return.¶

For error codes that do not require the return of additional information, a peer can use the existing RDMA_ERR2 header type to report the new error. The sender sets the new error code as the value of rdma_err with the result that the default switch arm of the rpcrdma2_error (i.e., void) is selected.¶

For error codes that do require the return of related information together with the error, a new header type should be defined that returns the error together with the related information. The sender of a new header type needs to be prepared to accept header types necessary to report associated errors.¶

C.1. Changes to the XDR Definition

Several XDR structural changes enable within-version protocol extensibility.¶

[RFC8166] defines the RPC-over-RDMA version 1 transport header as a single XDR object, with an RPC message potentially following it. In RPC-over-RDMA version 2, there are separate XDR definitions of the transport header prefix (see Section 6.4), which specifies the transport header type to be used, and the transport header itself (defined within one of the subsections of Section 6.3). This construction is similar to an RPC message, which consists of an RPC header (defined in [RFC5531]) followed by a message defined by an Upper-Layer Protocol.¶

As a new version of the RPC-over-RDMA transport protocol, RPC-over-RDMA version 2 exists within the versioning rules defined in [RFC8166]. In particular, it maintains the first four words of the protocol header, as specified in Section 4.2 of [RFC8166], even though, as explained in Section 6.2.1 of the current document, the XDR definition of those words is structured differently.¶

Although each of the first four fields retains its semantic function, there are differences in interpretation:¶

• The first word of the header, the rdma_xid field, retains the format and function that it had in RPC-over-RDMA version 1. Because RPC-over-RDMA version 2 messages can convey non-RPC messages, a receiver should not use the contents of this field without consideration of the protocol version and header type.¶
• The second word of the header, the rdma_vers field, retains the format and function that it had in RPC-over-RDMA version 1. To clearly distinguish version 1 and version 2 messages, senders need to fill in the correct version (fixed after version negotiation). Receivers should check that the content of the rdma_vers is correct before using the content of any other header field.¶
• The third word of the header, the rdma_credit field, retains the size and general purpose that it had in RPC-over-RDMA version 1. However, RPC-over-RDMA version 2 divides this field into two 16-bit subfields. See Section 4.2.1 for further details.¶
• The fourth word of the header, previously the union discriminator field rdma_proc, retains its format and general function even though the set of valid values has changed. Within RPC-over-RDMA version 2, this word is the rdma_htype field of the structure rdma_start. The value of this field is now an unsigned 32-bit integer rather than an enum type, to facilitate header type extension.¶

Beyond conforming to the restrictions specified in [RFC8166], RPC-over-RDMA version 2 attempts to limit the scope of the changes made to ensure interoperability. Although it introduces the Call chunk and splits the two version 1 workhorse procedure types RDMA_MSG and RDMA_NOMSG into several variants, RPC-over-RDMA version 2 otherwise expresses chunks in the same format and utilizes them the same way.¶

C.2. Transport Properties

RPC-over-RDMA version 2 provides a mechanism for exchanging an implementation's operational properties. The purpose of this exchange is to help endpoints improve the efficiency of data transfer by exploiting the characteristics of both peers rather than falling back on the lowest common denominator default settings. A full discussion of transport properties appears in Section 5.¶

C.3. Credit Management Changes

RPC-over-RDMA transports employ credit-based flow control to ensure that a Requester does not emit more RDMA Sends than the Responder is prepared to receive.¶

Section 3.3.1 of [RFC8166] explains the operation of RPC-over-RDMA version 1 credit management in detail. In that design, each RDMA Send from a Requester contains an RPC Call with a credit request, and each RDMA Send from a Responder contains an RPC Reply with a credit grant. The credit grant implies that enough Receives have been posted on the Responder to handle the credit grant minus the number of pending RPC transactions (the number of remaining Receive buffers might be zero).¶

Each RPC Reply acts as an implicit ACK for a previous RPC Call from the Requester. Without an RPC Reply message, the Requester has no way to know that the Responder is ready for subsequent RPC Calls.¶

Because version 1 embeds credit management in each message, there is a strict one-to-one ratio between RDMA Send and RPC message. There are interesting use cases that might be enabled if this relationship were more flexible:¶

• RPC-over-RDMA operations that do not carry an RPC message, e.g., control plane operations.¶
• A single RDMA Send that conveys more than one RPC message, e.g., for interrupt mitigation.¶
• An RPC message that requires several sequential RDMA Sends, e.g., to reduce the use of explicit RDMA operations for moderate-sized RPC messages.¶
• An RPC transaction that requires multiple exchanges or an odd number of RPC-over-RDMA operations to complete.¶

RPC-over-RDMA version 2 provides a more sophisticated credit accounting mechanism to address these shortcomings. Section 4.2.1 explains the new mechanism in detail.¶

C.4. Inline Threshold Changes

An "inline threshold" value is the largest message size (in octets) that can be conveyed on an RDMA connection using only RDMA Send and Receive. Each connection has two inline threshold values: one for messages flowing from client-to-server (referred to as the "client-to-server inline threshold") and one for messages flowing from server-to-client (referred to as the "server-to-client inline threshold").¶

A connection's inline thresholds determine, among other things, when RDMA Read or Write operations are required because an RPC message cannot be conveyed via a single RDMA Send and Receive pair. When an RPC message does not contain DDP-eligible data items, a Requester can prepare a Special Format Call or Reply to convey the whole RPC message using RDMA Read or Write operations.¶

RDMA Read and Write operations require that data payloads reside in memory registered with the local RNIC. When an RPC completes, that memory is invalidated to fence it from the Responder. Memory registration and invalidation typically have a latency cost that is insignificant compared to data handling costs.¶

When a data payload is small, however, the cost of registering and invalidating memory where the payload resides becomes a significant part of total RPC latency. Therefore the most efficient operation of an RPC-over-RDMA transport occurs when the peers use explicit RDMA Read and Write operations for large payloads but avoid those operations for small payloads.¶

When the authors of [RFC8166] first conceived RPC-over-RDMA version 1, the average size of RPC messages that did not involve a significant data payload was under 500 bytes. A 1024-byte inline threshold adequately minimized the frequency of inefficient Long messages.¶

With NFS version 4 [RFC7530], the increased size of NFS COMPOUND operations resulted in RPC messages that are, on average, larger than previous versions of NFS. With a 1024-byte inline threshold, frequent operations such as GETATTR and LOOKUP require RDMA Read or Write operations, reducing the efficiency of data transport.¶

To reduce the frequency of Special Format messages, RPC-over-RDMA version 2 increases the default size of inline thresholds. This change also increases the maximum size of reverse-direction RPC messages.¶

C.5. Message Continuation Changes

In addition to a larger default inline threshold, RPC-over-RDMA version 2 introduces Message Continuation. Message Continuation is a mechanism that enables the transmission of a data payload using more than one RDMA Send. The purpose of Message Continuation is to provide relief in several essential cases:¶

• If a Requester finds that it is inefficient to convey a moderately-sized data payload using Read chunks, the Requester can use Message Continuation to send the RPC Call.¶
• If a Requester has provided insufficient Reply chunk space for a Responder to send an RPC Reply, the Responder can use Message Continuation to send the RPC Reply.¶
• If a sender has to convey a sizeable non-RPC data payload (e.g., a large transport property), the sender can use Message Continuation to avoid having to register memory.¶

C.6. Host Authentication Changes

For the general operation of NFS on open networks, we eventually intend to rely on RPC-on-TLS [I-D.ietf-nfsv4-rpc-tls] to provide cryptographic authentication of the two ends of each connection. In turn, this can improve the trustworthiness of AUTH_SYS-style user identities that flow on TCP, which are not cryptographically protected. We do not have a similar solution for RPC-over-RDMA, however.¶

Here, the RDMA transport layer already provides a strong guarantee of message integrity. On some network fabrics, IPsec or TLS can protect the privacy of in-transit data. However, this is not the case for all fabrics (e.g., InfiniBand [IBA]).¶

Thus, RPC-over-RDMA version 2 introduces a mechanism for authenticating connection peers (see Section 5.2.6). And like GSS channel binding, there is also a way to determine when the use of host authentication is unnecessary.¶

C.7. Support for Remote Invalidation

When an RDMA consumer uses FRWR or Memory Windows to register memory, that memory may be invalidated remotely [RFC5040]. These mechanisms are available when a Requester's RNIC supports MEM_MGT_EXTENSIONS.¶

For this discussion, there are two classes of STags. Dynamically-registered STags appear in a single RPC, then are invalidated. Persistently-registered STags survive longer than one RPC. They may persist for the life of an RPC-over-RDMA connection or even longer.¶

An RPC-over-RDMA Requester can provide more than one STag in a transport header. It may provide a combination of dynamically- and persistently-registered STags in one RPC message, or any combination of these in a series of RPCs on the same connection. Only dynamically-registered STags using Memory Windows or FRWR may be invalidated remotely.¶

There is no transport-level mechanism by which a Responder can determine how a Requester-provided STag was registered, nor whether it is eligible to be invalidated remotely. A Requester that mixes persistently- and dynamically-registered STags in one RPC, or mixes them across RPCs on the same connection, must, therefore, indicate which STag the Responder may invalidate remotely via a mechanism provided in the Upper-Layer Protocol. RPC-over-RDMA version 2 provides such a mechanism.¶

A sender uses the RDMA Send With Invalidate operation to invalidate an STag on the remote peer. It is available only when both peers support MEM_MGT_EXTENSIONS (can send and process an IETH).¶

Existing RPC-over-RDMA transport protocol specifications [RFC8166] [RFC8167] do not forbid direct data placement in the reverse direction. Moreover, there is currently no Upper-Layer Protocol that makes data items in reverse-direction operations eligible for direct data placement.¶

When chunks are present in a reverse-direction RPC request, Remote Invalidation enables the Responder to trigger invalidation of a Requester's STags as part of sending an RPC Reply, the same way as is done in the forward direction.¶

However, in the reverse direction, the server acts as the Requester, and the client is the Responder. The server's RNIC, therefore, must support receiving an IETH, and the server must have registered its STags with an appropriate registration mechanism.¶

C.8. Integration of Reverse-Direction Operation

Because [RFC5666] did not include specification of reverse-direction operation, [RFC8166] does not include it either. Reverse-direction operation in RPC-over-RDMA version 1 is specified by a separate standards track document [RFC8167].¶

Reverse-direction operation in RPC-over-RDMA version 1 was constrained by the limited ability to extend that version of the protocol. The most awkward issue is that a receiver needs to peek at ingress RPC message payloads to determine whether it is a Call or Reply message. This is necessary because the meaning of several fields in the RPC-over-RDMA transport header is determined by the direction of the RPC message payload:¶

• The meaning of the value in the rdma_xid field is determined by the direction of the message because the XID spaces in the forward and reverse directions are distinct.¶
• The meaning of the value in the rdma_credits field is determined by the direction of the message because credits are granted separately for forward and reverse direction operation.¶
• The purpose of Write chunks and the meaning of their length fields is determined by the direction of the message because in Call messages, they are provisional, but in Reply messages, they represent returned results.¶

The current document remedies this awkwardness by integrating reverse-direction operation into RPC-over-RDMA version 2 so that it can make use of all facilities that are available in the forward-direction, including body chunks, remote invalidation, and message continuation. To enable this integration, the direction of the RPC message payload is encoded in each RPC-over-RDMA version 2 transport header.¶

C.9. Error Reporting Changes

RPC-over-RDMA version 2 expands the repertoire of errors that connection peers may report to each other. The goals of this expansion are:¶

• To fill in details of peer recovery actions.¶
• To enable retrying certain conditions caused by mis-estimation of the maximum reply size.¶
• To minimize the likelihood of a Requester waiting forever for a Reply when there are communications problems that prevent the Responder from sending it.¶

C.10. Changes in Terminology

The RPC-over-RDMA version 2 specification makes the following changes in terminology. These changes do not result in changes in the behavior or operation of the protocol.¶

• The current document explicitly acknowledges the different semantics and purpose of Write chunks appearing in Call messages and those appearing in Reply messages.¶
• The current document introduces the term "payload format" to describe the selection of a mechanism for reducing and conveying an RPC message payload. It replaces the terms "short message" and "long message" with the terms "simple format" and "special format" because this selection is not based only on the size of the payload.¶
• The current document introduces the terms "data item chunk" and "body chunk" in order to distinguish the purpose and operation of these two categories of chunk.¶
• For improved readability, the current document replaces the terms "RDMA segment" and "plain segment" with the term "segment", and the term "RDMA read segment" with the term "Read segment".¶
• The current document refers specifically to the RDMAP, DDP, and MPA standards track protocols rather than using the nebulous term "iWARP".¶