RPC-over-RDMA Version 2 Protocol

3.1.1. Upper-Layer Protocols

RPCs are an abstraction used to implement the operations of an Upper-Layer Protocol (ULP). For RPC-over-RDMA, "ULP" refers to an RPC Program and Version tuple, which is a versioned set of procedure calls that comprise a single well-defined API. One example of a ULP is the Network File System Version 4.0 [RFC7530]. In the current document, the term "RPC consumer" refers to an implementation of a ULP running on an RPC client.¶

3.1.2. Requesters and Responders

Like a local procedure call, every RPC procedure has a set of "arguments" and a set of "results". A calling context invokes a procedure, passing arguments to it, and the procedure subsequently returns a set of results. Unlike a local procedure call, the called procedure is executed remotely rather than in the local application's execution context.¶

The RPC protocol as described in [RFC5531] is fundamentally a message-passing protocol between one or more clients, where RPC consumers are running, and a server, where a remote execution context is available to process RPC transactions on behalf of these consumers.¶

ONC RPC transactions consist of two types of messages:¶

• A CALL message, or "Call", requests work. An RPC Call message is designated by the value zero (0) in the message's msg_type field. The sender places a unique 32-bit value in the message's XID field to match this RPC Call message to a corresponding RPC Reply message.¶
• A REPLY message, or "Reply", reports the results of work requested by an RPC Call message. An RPC Reply message is designated by the value one (1) in the message's msg_type field. The sender copies the value contained in an RPC Reply message's XID field from the RPC Call message whose results the sender is reporting.¶

Each RPC client endpoint acts as a "Requester", which serializes the procedure's arguments and conveys them to a server endpoint via an RPC Call message. A Call message contains an RPC protocol header, a header describing the requested upper-layer operation, and all arguments.¶

An RPC server endpoint acts as a "Responder", which deserializes the arguments and processes the requested operation. It then serializes the operation's results into an RPC Reply message. An RPC Reply message contains an RPC protocol header, a header describing the upper-layer reply, and all results.¶

The Requester deserializes the results and allows the RPC consumer to proceed. At this point, the RPC transaction designated by the XID in the RPC Call message is complete, and the XID is retired.¶

In summary, Requesters send RPC Call messages to Responders to initiate RPC transactions. Responders send RPC Reply messages to Requesters to complete the processing on an RPC transaction.¶

3.1.3. RPC Transports

The role of an "RPC transport" is to mediate the exchange of RPC messages between Requesters and Responders. An RPC transport bridges the gap between the RPC message abstraction and the native operations of a network transport (e.g., a socket).¶

RPC-over-RDMA is a connection-oriented RPC transport. When a transport type is connection-oriented, clients initiate transport connections, while servers wait passively to accept incoming connection requests.¶

3.1.4. External Data Representation

One cannot assume that all Requesters and Responders represent data objects in the same way internally. RPC uses External Data Representation (XDR) to translate native data types and serialize arguments and results [RFC4506].¶

XDR encodes data independently of the endianness or size of host-native data types, enabling unambiguous decoding of data by a receiver.¶

XDR assumes only that the number of bits in a byte (octet) and their order are the same on both endpoints and the physical network. The smallest indivisible unit of XDR encoding is a group of four octets. XDR can also flatten lists, arrays, and other complex data types into a stream of bytes.¶

We refer to a serialized stream of bytes that is the result of XDR encoding as an "XDR stream". A sender encodes native data into an XDR stream and then transmits that stream to a receiver. The receiver decodes incoming XDR byte streams into its native data representation format.¶

3.2.1. Direct Data Placement

Typically, RPC implementations copy the contents of RPC messages into a buffer before being sent. An efficient RPC implementation sends bulk data without first copying it into a separate send buffer.¶

However, socket-based RPC implementations are often unable to receive data directly into its final place in memory. Receivers often need to copy incoming data to finish an RPC operation, if only to adjust data alignment.¶

Although it may not be efficient, before an RDMA transfer, a sender may copy data into an intermediate buffer. After an RDMA transfer, a receiver may copy that data again to its final destination. In this document, the term "DDP" refers to any optimized data transfer where a receiving host's CPU does not move transferred data to another location after arrival.¶

RPC-over-RDMA version 2 enables the use of RDMA Read and Write operations to achieve both data movement offload and DDP. However, note that not all RDMA-based data transfer qualifies as DDP, and some mechanisms that do not employ explicit RDMA can place data directly.¶

3.2.2. RDMA Transport Operation

RDMA transports require that RDMA consumers provision resources in advance to achieve good performance during receive operations. An RDMA consumer might provide Receive buffers in advance by posting an RDMA Receive Work Request for every expected RDMA Send from a remote peer. These buffers are provided before the remote peer posts RDMA Send Work Requests. Thus this is often referred to as "pre-posting" buffers.¶

An RDMA Receive Work Request remains outstanding until the RDMA provider matches it to an inbound Send operation. The resources associated with that Receive must be retained in host memory, or "pinned", until the Receive completes.¶

Given these tenets of operation, the RPC-over-RDMA version 2 protocol assumes each transport provides the following abstract operations. A more complete discussion of these operations appears in [RFC5040].¶

4.2.1. Flow Control

RPC-over-RDMA version 2 employs end-to-end credit-based flow control on each connection to prevent senders from transmitting more messages than a receiver is prepared to accept [CBFC]. Credit-based flow control is relatively simple, providing robust operation in the face of bursty traffic and automated management of receive buffer allocation while enabling effective pipelining. A simplified sliding window approach is all that is necessary for our purposes because the underlying RDMA transport service already guarantees reliable and in-order message delivery.¶

An RPC-over-RDMA version 2 credit represents the capability to convey exactly one RPC-over-RDMA version 2 message, regardless of its size, via an RDMA Send/Receive pair. This arrangement enables RPC-over-RDMA version 2 transport connections to support multiple unacknowledged messages in each direction.¶

Because an RPC-over-RDMA version 2 connection is full-duplex, each connection peer has its own set of credits. The two receivers manage their credits independently, although they typically communicate these values by piggy-backing them on a payload-bearing message in the opposite direction.¶

Each RPC-over-RDMA version 2 message header contains two fields that handle credit accounting:¶

• The rdma_credit field contains the receiver's credit window size. This field functions in much the same way as the RPC-over-RDMA version 1 used "credit grants". When sending an RPC-over-RDMA message, a peer fills in this field with the number of unacknowledged messages it is prepared to receive on this connection.¶
• A second field, which is yet to be defined, reports the number of messages received so far. When sending an RPC-over-RDMA message, a peer fills in this field with the count of messages it has already received on this connection.¶

A sender also keeps track of the count of messages it has sent on the connection. The sender MUST stop transmitting messages when the number of messages it has already sent is about to exceed the number of messages the receiver has acknowledged plus the receiver's credit window.¶

A receiver MAY adjust the credit window to match the needs or policies in effect on either peer. For instance, a peer may reduce the size of its credit window to accommodate the available resources in a Shared Receive Queue. Certain RDMA implementations may impose additional flow-control restrictions, such as limits on RDMA Read operations in progress at the Responder. Accommodation of such checks is considered the responsibility of each RPC-over-RDMA version 2 implementation.¶

The credit window size MUST be less than the total sequence number range (let's make this a more quantitative statement later). The receiver generally chooses a credit window that is large enough to maximize throughput, given the bandwidth-delay product of the connection, while not overwhelming memory resources on the local system.¶

4.2.2. Inline Threshold

An "inline threshold" value is the largest message size (in octets) that can be conveyed in one direction between peer implementations using RDMA Send and Receive channel operations. An inline threshold value is less than or equal to the largest number of octets the sender can post in a single RDMA Send operation. It is also less than or equal to the largest number of octets the receiver can reliably accept via a single RDMA Receive operation.¶

Each connection has two inline threshold values. There is one for messages flowing from Requester-to-Responder (referred to as the "call inline threshold"), and one for messages flowing from Responder-to-Requester (referred to as the "reply inline threshold").¶

Peers can advertise their inline threshold values via RPC-over-RDMA version 2 Transport Properties (see Section 5). In the absence of an exchange of Transport Properties, connection peers MUST assume both inline thresholds are 4096 octets.¶

4.3.1. Server Supports RPC-over-RDMA Version 2

If the server supports RPC-over-RDMA version 2, it sends RPC-over-RDMA messages back to the client with the value two (2) in the rdma_vers field. Both peers may assume the default inline threshold value for RPC-over-RDMA version 2 connections (4096 octets).¶

4.3.2. Server Does Not Support RPC-over-RDMA Version 2

If the server does not support RPC-over-RDMA version 2, it MUST send an RPC-over-RDMA message to the client with an XID that matches the client's first message, RDMA2_ERROR in the rdma_start.rdma_htype field, and with the error code RDMA2_ERR_VERS. This message also reports the range of RPC-over-RDMA protocol versions that the server supports. To continue operation, the client selects a protocol version in that range for subsequent messages on this connection.¶

If the connection is dropped immediately after an RDMA2_ERROR/RDMA2_ERR_VERS message is received, the client should try to avoid a version negotiation loop when re-establishing another connection. It can assume that the server does not support RPC-over-RDMA version 2. A client can assume the same situation (i.e., no server support for RPC-over-RDMA version 2) if the initial negotiation message is lost or dropped. Once the version negotiation exchange is complete, both peers may use the default inline threshold value for the negotiated transport protocol version.¶

4.3.3. Client Does Not Support RPC-over-RDMA Version 2

The server examines the RPC-over-RDMA protocol version used in the first RPC-over-RDMA message it receives. If it supports this protocol version, it MUST use it in all subsequent messages it sends on that connection. The client MUST NOT change the protocol version for the duration of the connection.¶

4.4.1. Chunks and Segments

A Requester records the location information for each registered memory region associated with an RPC payload in the transport header of an RPC-over-RDMA message. With this information, the Responder uses RDMA Read and Write operations to retrieve arguments contained in the specified region of the Requester's memory or place results in that region.¶

A "segment" is a transport header data object that contains the precise coordinates of a contiguous registered memory region. Each segment contains the following information:¶

Handle:

A steering Tag (STag) or R_key generated by registering this memory with the RDMA provider.¶

Length:

The length of the segment's memory region, in octets. The length of a segment MAY be aligned to a single octet. An "empty segment" is defined as a segment with the value zero (0) in its length field.¶

Offset:

The offset or beginning memory address of the segment's memory region.¶

The meaning of the values contained in these fields is elaborated in [RFC5040].¶

A "chunk" is simply a set of segments that have a related purpose. A Requester MAY divide a chunk into segments using any convenient boundaries. The length of a chunk is defined as the sum of the lengths of the segments that comprise it.¶

4.4.2. Reducing a Payload Stream

We refer to a data item that a sender removes from a Payload stream to transmit separately as a "reduced" data item. After a sender has finished removing XDR data items from a Payload stream, we refer to it as a "reduced" Payload stream. A set of segments that describe memory regions containing a single reduced data item is categorized as a "data item chunk."¶

Not all XDR data items benefit from Direct Data Placement. For example, small data items or data items that require XDR unmarshaling by the receiver do not benefit from DDP. Moreover, it is impractical for receivers to prepare for every possible XDR data item in a protocol to appear in a data item chunk.¶

Specifying which data items are DDP-eligible is done in separate standards track documents known as "Upper Layer Bindings". A ULB identifies which XDR data items a peer MAY transfer using DDP. We refer to such data items as "DDP-eligible." Senders MUST NOT reduce any other XDR data items. Detailed requirements for ULB specifications appear in Appendix A of the current document.¶

4.4.3. Moving Whole RPC Messages using Explicit RDMA

RPC-over-RDMA version 2 also enables the movement of a whole RPC message via data transfer distinct from RDMA Send and Receive. A sender registers the memory containing a Payload stream without regard to data item boundaries or DDP-eligibility. The Payload stream is then conveyed via other mechanisms, such as one or more RDMA Read or Write operations. A set of segments that describe memory regions containing a Payload stream is categorized as a "body chunk".¶

A sender may first reduce that Payload stream if it contains one or more DDP-eligible data items. The sender moves these data items using data items chunks, and the reduced Payload stream using a body chunk.¶

4.5.1. Read Chunks

A "Read chunk" contains data that its receiver pulls from the sender. Each Read chunk is a set of one or more "Read segments" encoded as a list. A Read segment consists of a Position field followed by a segment, as defined in Section 4.4.1.¶

Position:

The byte offset in the unreduced Payload stream where the receiver reinserts the data item conveyed in the chunk. The sender MUST compute the Position value from the beginning of the unreduced Payload stream, which begins at Position zero. All segments in the same Read chunk share the same Position value, even if one or more of the segments have a non-four-byte-aligned length. The value in this field MUST be a multiple of four.¶

When constructing an RPC-over-RDMA message, the sender registers memory regions containing data intended for RDMA Read operations. It advertises the coordinates of these regions in Read chunks added to the transport header of an RPC-over-RDMA message.¶

The receiver of this message then pulls the chunk's data from the sender using RDMA Read operations. When receiving a Read chunk, the receiver inserts the first Read segment in a Read chunk into the Payload stream at the byte offset indicated by its Position field. The receiver concatenates Read segments whose Position field value matches this offset until there are no more Read segments at that Position value.¶

4.5.2. Write Chunks

Each "Write chunk" consists of a counted array of zero or more segments, as defined in Section 4.4.1. The function of a Write chunk depends on the direction of the containing RPC-over-RDMA message. In a Call message, a Write chunk advertises registered memory regions into which the Responder may push data. In a Reply message, a Write chunk reports how much data has been pushed.¶

A Requester provisions Write chunks for an RPC transaction long before the Responder has constructed a corresponding Reply message. A Requester typically does not know the actual length of the result data items or Reply to be returned, since the Reply does not yet exist. Thus, a Requester MUST provision Write chunks large enough to accommodate the maximum possible size of each returned data item.¶

An "empty Write chunk" is a Write chunk with a zero segment count. By definition, the length of an empty Write chunk is zero. An "unused Write chunk" has a non-zero segment count, but all of its segments are empty segments.¶

4.5.3. Reducing Complex XDR Data Types

XDR data items may appear in body chunks without regard to their DDP-eligibility. As body chunks contain a Payload stream, they MUST include all appropriate XDR roundup padding to maintain proper XDR alignment of their contents.¶

However, a data item chunk MUST contain only one XDR data item, and the chunk MUST occupy a four-byte aligned length in the Payload stream so that subsequent data items remain properly aligned once the reduced data item is removed from the Payload stream.¶

4.6.1. Sending a Reverse-Direction RPC Call

An RPC-over-RDMA server endpoint constructs the transport header for a reverse-direction RPC Call as follows:¶

• The server generates a new XID value (see Section 3.1.3.5 for full requirements) and places it in the rdma_xid field of the transport header and the xid field of the RPC Call message. The RPC Call header MUST start with the same XID value that is present in the transport header.¶
• The rdma_vers field of each reverse-direction Call MUST contain the same value as forward-direction Calls on the same connection.¶
• The server fills in the rdma_credit field with the credit values for the connection, as described in Section 4.2.1.¶
• The server determines the Payload format for the RPC message and fills in the rdma_htype field as appropriate (see Sections 6.6 and 4.6.4). Section 4.6.4 also covers the disposition of the chunk lists.¶

4.6.2. Sending a Reverse-Direction RPC Reply

An RPC-over-RDMA client endpoint constructs the transport header for a reverse-direction RPC Reply as follows:¶

• The client copies the XID value from the matching RPC Call and places it in the rdma_xid field of the transport header and the xid field of the RPC Reply message. The RPC Reply header MUST start with the same XID value that is present in the transport header.¶
• The rdma_vers field of each reverse-direction Call MUST contain the same value as forward-direction Replies on the same connection.¶
• The client fills in the rdma_credit field with the credit values for the connection, as described in Section 4.2.1.¶
• The client determines the Payload format for the RPC message and fills in the rdma_htype field as appropriate (see Sections 6.6 and 4.6.4). Section 4.6.4 also covers the disposition of the chunk lists.¶

4.6.3. When Reverse-Direction Operation is Not Supported

An RPC-over-RDMA transport endpoint does not have to support reverse-direction operation. There might be no mechanism in the transport implementation to do so. Or, the transport implementation might support operation in the reverse direction, but the Upper-Layer Protocol might not configure the transport to handle reverse-direction traffic.¶

If an endpoint is unprepared to receive a reverse-direction message, loss of the RDMA connection might result. Thus a denial of service can occur if an RPC server continues to send reverse-direction messages after a client that is not prepared to receive them reconnects to that server.¶

Connection peers indicate their support for reverse-direction operation as part of the exchange of Transport Properties just after a connection is established (see Section 5.2.5).¶

When dealing with the possibility that the remote peer has no transport level support for reverse-direction operation, the Upper-Layer Protocol is responsible for informing peers when reverse-direction operation is supported. Otherwise, even a simple reverse-direction RPC NULL procedure from a peer could result in a lost connection. Therefore, an Upper-Layer Protocol MUST NOT perform reverse-direction RPC operations until the RPC client indicates support for them.¶

4.6.4. Using Chunks During Reverse-Direction Operation

Reverse-direction operations can use chunks for DDP-eligible data items and Special payload formats the same way chunks are used in forward-direction operation. Connection peers indicate their support for using chunks in the reverse direction as part of the exchange of Transport Properties just after a connection is established (see Section 5.2.5).¶

However, an implementation might support only Upper-Layer Protocols that have no DDP-eligible data items. Such Upper-Layer Protocols can use only small messages, or they might have a native mechanism for restricting the size of reverse-direction RPC messages, obviating the need to handle chunks in the reverse direction.¶

When there is no Upper-Layer Protocol need for chunks in the reverse direction, implementers MAY choose not to provide support for chunks in the reverse direction, thus avoiding the complexity of implementing support for RDMA Reads and Writes in the reverse direction. When an RPC-over-RDMA transport implementation does not support chunks in the reverse direction, RPC endpoints use only the Simple Payload format without data item chunks or the Continued Payload format without data item chunks to send RPC messages in the reverse direction.¶

If a reverse-direction Requester provides a non-empty chunk list to a Responder that does not support chunks, the Responder MUST report its lack of support using one of the error values defined in Section 7.3.¶

4.6.5. Reverse-Direction Retransmission

In rare cases, an RPC server cannot complete an RPC transaction and cannot send a Reply. In these cases, the Requester may send the RPC transaction again using the same RPC XID. We refer to this as an "RPC retransmission" or a "replay."¶

In the forward direction, an RPC client is the Requester. The client is always responsible for ensuring a transport connection is in place before sending a dropped Call again.¶

With reverse-direction operation, an RPC server is the Requester. Because an RPC server is not responsible for establishing transport connections with clients, the Requester is unable to retransmit a reverse-direction Call whenever there is no transport connection. In this case, the RPC server must wait for the RPC client to re-establish a transport connection before it can retransmit reverse-direction RPC Calls.¶

If the forward-direction Requester has no work to do, it can be some time before the RPC client re-establishes a transport connection. An RPC server may need to abandon a pending reverse-direction RPC Call to avoid waiting indefinitely for the client to re-establish a transport connection.¶

Therefore forward-direction Requesters SHOULD maintain a transport connection as long as the RPC server might send reverse-direction Calls. For example, while an NFS version 4.1 client has open delegated files or active pNFS layouts, it maintains one or more transport connections to enable the NFS server to perform callback operations.¶

5.2.1. Maximum Send Size

The value of this property specifies the maximum size, in octets, of Send payloads. The endpoint receiving this value can size its Receive buffers based on the value of this property.¶

<CODE BEGINS>

const uint32 RDMA2_PROPID_SBSIZ = 1;
typedef uint32 rpcrdma2_prop_sbsiz;


<CODE ENDS>

5.2.2. Receive Buffer Size

The value of this property specifies the minimum size, in octets, of pre-posted receive buffers.¶

<CODE BEGINS>

const uint32 RDMA2_PROPID_RBSIZ = 2;
typedef uint32 rpcrdma2_prop_rbsiz;


<CODE ENDS>

A sender can subsequently use this value to determine when a message to be sent fits in pre-posted receive buffers that the receiver has set up. In particular:¶

• Requesters may use the value to determine when to use a Call chunk or Message Continuation when sending a Call.¶
• Requesters may use the value to determine when to provide a Reply chunk when sending a Call, based on the maximum possible size of the Reply.¶
• Responders may use the value to determine when to use a Reply chunk provided by the Requester, given the actual size of a Reply.¶

5.2.3. Maximum Segment Size

The value of this property specifies the maximum size, in octets, of a segment this endpoint is prepared to send or receive.¶

<CODE BEGINS>

const uint32 RDMA2_PROPID_RSSIZ = 3;
typedef uint32 rpcrdma2_prop_rssiz;


<CODE ENDS>

5.2.4. Maximum Segment Count

The value of this property specifies the maximum number of segments that can appear in a Requester's transport header.¶

<CODE BEGINS>

const uint32 RDMA2_PROPID_RCSIZ = 4;
typedef uint32 rpcrdma2_prop_rcsiz;


<CODE ENDS>

5.2.5. Reverse-Direction Support

The value of this property specifies a client implementation's readiness to process messages that are part of reverse-direction RPC requests.¶

<CODE BEGINS>

const uint32 RDMA_RVRSDIR_NONE = 0;
const uint32 RDMA_RVRSDIR_SIMPLE = 1;
const uint32 RDMA_RVRSDIR_CONT = 2;
const uint32 RDMA_RVRSDIR_GENL = 3;

const uint32 RDMA2_PROPID_BRS = 5;
typedef uint32 rpcrdma2_prop_brs;


<CODE ENDS>

Multiple levels of support are distinguished:¶

• The value RDMA2_RVRSDIR_NONE indicates that the sender does not support reverse-direction operation.¶
• The value RDMA2_RVRSDIR_SIMPLE indicates that the sender supports using only Simple Format messages without data item chunks for reverse-direction messages.¶
• The value RDMA2_RVRSDIR_CONT indicates that the sender supports using either Simple Format without data item chunks or Continued Format messages without data item chunks for reverse-direction messages.¶
• The value RDMA2_RVRSDIR_GENL indicates that the sender supports reverse-direction messages in the same way as forward-direction messages.¶

When a peer does not provide this property, the default is the peer does not support reverse-direction operation.¶

5.2.6. Host Authentication Message

The value of this transport property enables the exchange of host authentication material. This property can accommodate authentication handshakes that require multiple challenge-response interactions and potentially large amounts of material.¶

<CODE BEGINS>

const uint32 RDMA2_PROPID_HOSTAUTH = 6;
typedef opaque rpcrdma2_prop_hostauth<>;


<CODE ENDS>

When this property is not present, the peer(s) remain unauthenticated. Local security policy on each peer determines whether the connection is permitted to continue.¶

6.2.1. Common Transport Header Prefix

The rpcrdma_common structure defines the initial part of each RPC-over-RDMA transport header for RPC-over-RDMA version 2 and subsequent versions.¶

<CODE BEGINS>

struct rpcrdma_common {
             uint32         rdma_xid;
             uint32         rdma_vers;
             uint32         rdma_credit;
             uint32         rdma_htype;
};


<CODE ENDS>

RPC-over-RDMA version 2's use of these first four words aligns with that of version 1 as required by Section 4.2 of [RFC8166]. However, there are crucial structural differences in the XDR definition of RPC-over-RDMA version 2: in the way that these words are described by the respective XDR descriptions:¶

• The header type is represented as a uint32 rather than as an enum type. An enum would need to be modified to reflect additions to the set of header types made by later extensions.¶
• The header type field is part of an XDR structure devoted to representing the transport header prefix, rather than being part of a discriminated union, that includes the body of each transport header type.¶
• There is now a prefix structure (see Section 6.4) of which the rpcrdma_common structure is the initial segment. This prefix is a newly defined XDR object within the protocol description, which constrains the universal portion of all header types to the four words in rpcrdma_common.¶

These changes are part of a more considerable structural change in the XDR definition of RPC-over-RDMA version 2 that facilitates a cleaner treatment of protocol extension. The XDR appearing in Section 8 reflects these changes, which Appendix C.1 discusses in further detail.¶

6.3.1. RDMA2_ERROR: Report Transport Error

RDMA2_ERROR reports a transport layer error on a previous transmission.¶

<CODE BEGINS>

const rpcrdma2_proc RDMA2_ERROR = 4;

struct rpcrdma2_err_vers {
        uint32 rdma_vers_low;
        uint32 rdma_vers_high;
};

struct rpcrdma2_err_write {
        uint32 rdma_chunk_index;
        uint32 rdma_length_needed;
};

union rpcrdma2_hdr_error switch (rpcrdma2_errcode rdma_err) {
        case RDMA2_ERR_VERS:
          rpcrdma2_err_vers rdma_vrange;
        case RDMA2_ERR_READ_CHUNKS:
          uint32 rdma_max_chunks;
        case RDMA2_ERR_WRITE_CHUNKS:
          uint32 rdma_max_chunks;
        case RDMA2_ERR_SEGMENTS:
          uint32 rdma_max_segments;
        case RDMA2_ERR_WRITE_RESOURCE:
          rpcrdma2_err_write rdma_writeres;
        case RDMA2_ERR_REPLY_RESOURCE:
          uint32 rdma_length_needed;
        default:
          void;
};


<CODE ENDS>

See Section 7 for details on the use of this header type.¶

6.3.2. RDMA2_GRANT: Grant Credits

The RDMA2_GRANT header type enables a connection peer to update credit information without conveying a payload.¶

<CODE BEGINS>

const rpcrdma2_proc RDMA2_GRANT = 5;


<CODE ENDS>

This message carries no payload except for a struct rpcrdma2_hdr_prefix. The rdma_xid field is unused. Senders MUST set the rdma_xid field to zero and receivers MUST ignore the value in this field.¶

6.3.3. RDMA2_CONNPROP_MIDDLE: Exchange Transport Properties

The RDMA2_CONNPROP_MIDDLE header type enables a connection peer to publish the properties of its implementation to its remote peer.¶

<CODE BEGINS>

const rpcrdma2_proc RDMA2_CONNPROP_MIDDLE = 6;

struct rpcrdma2_hdr_connprop {
        rpcrdma2_propset rdma_props;
};


<CODE ENDS>

A peer sends an RDMA2_CONNPROP_MIDDLE header type when it has one or more properties to send that do not fit within the default inline threshold for the RPC-over-RDMA version that is in effect.¶

A peer may encounter properties that it does not recognize or support. In such cases, the receiver ignores unsupported properties without generating an error response.¶

If a peer sends follows an RDMA2_CONNPROP_MIDDLE header type with anything other than another RDMA2_CONNPROP_MIDDLE message or an RDMA2_CONNPROP_FINAL message, the receiver MUST respond with an RDMA2_ERROR header type and set its rdma_err field to RDMA2_ERR_INVAL_CONT and drop the incoming message without processing it further.¶

6.3.4. RDMA2_CONNPROP_FINAL: Exchange Transport Properties

The RDMA2_CONNPROP_FINAL header type enables a connection peer to publish the properties of its implementation to its remote peer.¶

<CODE BEGINS>

const rpcrdma2_proc RDMA2_CONNPROP_FINAL = 7;

struct rpcrdma2_hdr_connprop {
        rpcrdma2_propset rdma_props;
};


<CODE ENDS>

Each peer sends an RDMA2_CONNPROP_FINAL header type as the final CONNPROP-type message after the client has established a connection. The size of this message is limited to the default inline threshold for the RPC-over-RDMA version that is in effect.¶

A peer may encounter properties that it does not recognize or support. In such cases, the receiver ignores unsupported properties without generating an error response.¶

If a peer sends a CONNPROP-type message on a connection after it has sent an RDMA2_CONNPROP_FINAL message, the receiver MUST respond with an RDMA2_ERROR header type and set its rdma_err field to RDMA2_ERR_INVAL_CONT and drop the incoming message without processing it further.¶

6.3.5. RDMA2_CALL_EXTERNAL: Convey External RPC Call Message

RDMA2_CALL_EXTERNAL conveys an RPC Call message payload using explicit RDMA operations. The Responder reads the Payload stream from a memory area specified by the Call chunk. The sender MUST set the rdma_xid field to the same value as the xid of the RPC Reply message payload.¶

<CODE BEGINS>

const rpcrdma2_proc RDMA2_CALL_EXTERNAL = 8;

struct rpcrdma2_hdr_call_external {
        uint32                      rdma_inv_handle;

        struct rpcrdma2_read_list   *rdma_call;
        struct rpcrdma2_read_list   *rdma_reads;
        struct rpcrdma2_write_list  *rdma_provisional_writes;
        struct rpcrdma2_write_chunk *rdma_provisional_reply;
};


<CODE ENDS>

rdma_inv_handle:

The rdma_inv_handle field contains a 32-bit RDMA handle that the Responder may use in a Send With Invalidation operation. See Section 6.5.¶

rdma_call:

The rdma_call field anchors a list of one or more Read segments that contain the RPC Call's Payload stream.¶

rdma_reads:

The rdma_reads field anchors a list of zero or more Read segments that contain data item chunks.¶

rdma_provisional_writes:

The rdma_writes field anchors a list of zero or more provisional Write chunks.¶

rdma_provisional_reply:

The rdma_reply field is a list containing zero or one provisional Reply chunk.¶

6.3.6. RDMA2_CALL_MIDDLE: Convey Continued RPC Call Message

RDMA2_CALL_MIDDLE conveys a beginning or middle portion of an RPC Call message immediately following the transport header in the send buffer. The sender MUST set the rdma_xid field to the same value as the xid of the RPC Reply message payload. The sender sets the rdma_remaining field to the number of bytes in the RPC Call message payload that remain to be sent. The rdma_rpc_first_word field demarks the first word of the Payload stream.¶

<CODE BEGINS>

const rpcrdma2_proc RDMA2_CALL_MIDDLE = 9;

struct rpcrdma2_hdr_call_middle {
        uint32                      rdma_remaining;

        /* The rpc message starts here and continues
         * through the end of the transmission. */
        uint32                      rdma_rpc_first_word;
};


<CODE ENDS>

If a peer sends follows an RDMA2_CALL_MIDDLE header type with anything other than an RDMA2_CALL_MIDDLE message or an RDMA2_CALL_INLINE message, the receiver MUST respond with an RDMA2_ERROR header type and set its rdma_err field to RDMA2_ERR_INVAL_CONT and drop the incoming message without processing it further.¶

6.3.7. RDMA2_CALL_INLINE: Convey Inline RPC Call Message

RDMA2_CALL_INLINE conveys the only or final portion of an RPC Call message. The rdma_rpc_first_word field demarks the first word of this Payload stream.¶

<CODE BEGINS>

const rpcrdma2_proc RDMA2_CALL_INLINE = 10;

struct rpcrdma2_hdr_call_inline {
        uint32                      rdma_inv_handle;

        struct rpcrdma2_read_list   *rdma_reads;
        struct rpcrdma2_write_list  *rdma_provisional_writes;
        struct rpcrdma2_write_chunk *rdma_provisional_reply;

        /* The rpc message starts here and continues
         * through the end of the transmission. */
        uint32                      rdma_rpc_first_word;
};


<CODE ENDS>

rdma_inv_handle:

The rdma_inv_handle field contains a 32-bit RDMA handle that the Responder may use in a Send With Invalidation operation. See Section 6.5.¶

rdma_reads:

The rdma_reads field anchors a list of zero or more Read segments that contain only data item chunks. A Requester MUST NOT insert Position-zero Read chunks in this list.¶

rdma_provisional_writes:

The rdma_writes field anchors a list of zero or more provisional Write chunks.¶

rdma_provisional_reply:

The rdma_reply field is a list containing zero or one provisional Reply chunk.¶

6.3.8. RDMA2_REPLY_EXTERNAL: Convey External RPC Reply Message

RDMA2_REPLY_EXTERNAL conveys an RPC Reply message payload using explicit RDMA operations. In particular, it is referred to as a Special Format Reply when the Responder writes the RPC payload into a memory area specified by a Reply chunk. The sender MUST set the rdma_xid field to the same value as the xid of the RPC Reply message payload.¶

<CODE BEGINS>

const rpcrdma2_proc RDMA2_REPLY_EXTERNAL = 11;

struct rpcrdma2_hdr_reply_external {
        struct rpcrdma2_write_list  *rdma_writes;
        struct rpcrdma2_write_chunk *rdma_reply;
};


<CODE ENDS>

rdma_writes:

The rdma_writes field anchors a list of zero or more Write chunks that are either empty or contain reduced data items.¶

rdma_reply:

The rdma_reply field is a list that MUST contain exactly one Reply chunk.¶

6.3.9. RDMA2_REPLY_MIDDLE: Convey Continued RPC Reply Message

RDMA2_REPLY_MIDDLE conveys a beginning or middle portion of an RPC Reply message immediately following the transport header in the send buffer. The sender MUST set the rdma_xid field to the same value as the xid of the RPC Reply message payload. The sender sets the rdma_remaining field to the number of bytes in the RPC Call message payload that remain to be sent. The rdma_rpc_first_word field demarks the first word of the Payload stream.¶

<CODE BEGINS>

const rpcrdma2_proc RDMA2_REPLY_MIDDLE = 12;

struct rpcrdma2_hdr_reply_middle {
        uint32                      rdma_remaining;

        /* The rpc message starts here and continues
         * through the end of the transmission. */
        uint32                      rdma_rpc_first_word;
};


<CODE ENDS>

If a peer sends follows an RDMA2_REPLY_MIDDLE header type with anything other than an RDMA2_REPLY_MIDDLE message or an RDMA2_REPLY_INLINE message, the receiver MUST respond with an RDMA2_ERROR header type and set its rdma_err field to RDMA2_ERR_INVAL_CONT and drop the incoming message without processing it further.¶

6.3.10. RDMA2_REPLY_INLINE: Convey RPC Reply Message Inline

RDMA2_REPLY_INLINE conveys the only or final portion of an RPC Reply message immediately following the transport header in the send buffer. If the Reply message payload has been reduced, the rdma_chunks object carries the reduced data item chunks.¶

<CODE BEGINS>

const rpcrdma2_proc RDMA2_REPLY_INLINE = 13;

struct rpcrdma2_hdr_reply_inline {
        struct rpcrdma2_write_list  *rdma_writes;

        /* The rpc message starts here and continues
         * through the end of the transmission. */
        uint32                      rdma_rpc_first_word;
};


<CODE ENDS>

rdma_writes:

The rdma_writes field anchors a list of zero or more Write chunks that are either empty or contain reduced data items.¶

6.6.1. Simple Format

All RPC messages conveyed via RPC-over-RDMA version 2 need at least one RDMA Send operation to convey. Thus, the most efficient way to send an RPC message that is smaller than the inline threshold is to append the Payload stream directly to the Transport stream and use an RDMA Send to convey both. When no chunks are present, senders construct Calls and Replies the same way, and no other operations are needed.¶

     Requester                                 Responder
         |      RDMA Send (RDMA2_CALL_INLINE)      |
    Call |   ---------------------------------->   |
         |                                         |
         |                                         | Processing
         |                                         |
         |      RDMA Send (RDMA2_REPLY_INLINE)     |
         |   <----------------------------------   | Reply

     Requester                                 Responder
         |      RDMA Send (RDMA2_CALL_INLINE)      |
    Call |   ---------------------------------->   |
         |               RDMA Read                 |
         |   <----------------------------------   |
         |         RDMA Response (arg data)        |
         |   ---------------------------------->   |
         |                                         |
         |                                         | Processing
         |                                         |
         |      RDMA Send (RDMA2_REPLY_INLINE)     |
         |   <----------------------------------   | Reply

     Requester                                 Responder
         |      RDMA Send (RDMA2_CALL_INLINE)      |
    Call |   ---------------------------------->   |
         |                                         |
         |                                         | Processing
         |                                         |
         |         RDMA Write (result data)        |
         |   <----------------------------------   |
         |      RDMA Send (RDMA2_REPLY_INLINE)     |
         |   <----------------------------------   | Reply

6.6.2. Continued Format

For various reasons, a sender can choose to split a message payload over multiple RPC-over-RDMA messages. The Payload stream of each RPC-over-RDMA message contains a part of the RPC message. The receiver reconstructs the original RPC message by concatenating the Payload stream of each RPC-over-RDMA message in received order. A sender MAY split the Payload stream on any convenient boundary.¶

     Requester                                 Responder
         |      RDMA Send (RDMA2_CALL_MIDDLE)      |
    Call |   ---------------------------------->   |
         |      RDMA Send (RDMA2_CALL_MIDDLE)      |
         |   ---------------------------------->   |
         |      RDMA Send (RDMA2_CALL_INLINE)      |
         |   ---------------------------------->   |
         |                                         |
         |                                         | Processing
         |                                         |
         |      RDMA Send (RDMA2_REPLY_MIDDLE)     |
         |   <----------------------------------   | Reply
         |      RDMA Send (RDMA2_REPLY_MIDDLE)     |
         |   <----------------------------------   |
         |      RDMA Send (RDMA2_REPLY_INLINE)     |
         |   <----------------------------------   |

     Requester                                 Responder
         |      RDMA Send (RDMA2_CALL_MIDDLE)      |
    Call |   ---------------------------------->   |
         |      RDMA Send (RDMA2_CALL_MIDDLE)      |
         |   ---------------------------------->   |
         |      RDMA Send (RDMA2_CALL_INLINE)      |
         |   ---------------------------------->   |
         |              RDMA Read                  |
         |   <----------------------------------   |
         |         RDMA Response (arg data)        |
         |   ---------------------------------->   |
         |                                         |
         |                                         | Processing
         |                                         |
         |      RDMA Send (RDMA2_REPLY_INLINE)     |
         |   <----------------------------------   | Reply

     Requester                                 Responder
         |      RDMA Send (RDMA2_CALL_INLINE)      |
    Call |   ---------------------------------->   |
         |                                         |
         |                                         | Processing
         |                                         |
         |         RDMA Write (result data)        |
         |   <----------------------------------   |
         |      RDMA Send (RDMA2_REPLY_MIDDLE)     |
         |   <----------------------------------   | Reply
         |      RDMA Send (RDMA2_REPLY_MIDDLE)     |
         |   <----------------------------------   |
         |      RDMA Send (RDMA2_REPLY_INLINE)     |
         |   <----------------------------------   |

6.6.3. Special Format

Even after DDP-eligible data items have been removed, a Payload stream can sometimes be too large to send using only RDMA Send operations. In those cases, the sender can use RDMA Read or Write operations to convey the entire RPC message. We refer to this as a "Special Format" message.¶

To transmit a Special Format message, the sender transmits only the Transport stream with an RDMA Send operation. The sender does not include the Payload stream in the send buffer. Instead, the Requester provides a body chunk that the Responder uses to move the Payload stream.¶

Because chunks are always present in Special Format messages, the sender always handles Calls and Replies differently.¶

Special Call

The Requester provides a Read chunk that contains the RPC Call message's Payload stream. Every Read segment in this chunk MUST contain zero (0) in its Position field. This type of Read chunk is a body chunk known as a Call chunk.¶

Special Reply

The Requester provisions a Reply chunk in advance. This body chunk is a Write chunk into which the Responder places the RPC Reply message's Payload stream. The Requester provisions the Reply chunk to accommodate the maximum expected reply size for that upper-layer operation.¶

One purpose of a Special Format message is to handle large RPC messages. However, Requesters MAY use a Special Format message at any time to convey an RPC Call message.¶

When it has alternatives, a Responder chooses which Format to use based on the chunks provided by the Requester. If a Requester provided a Write chunk and the Responder has a DDP-eligible result, it first reduces the reply Payload stream. If a Requester provided a Reply chunk and the reduced Payload stream is larger than the reply inline threshold, the Responder MUST use the Requester-provided Reply chunk for the reply.¶

     Requester                                 Responder
         |     RDMA Send (RDMA2_CALL_EXTERNAL)     |
    Call |   ---------------------------------->   |
         |               RDMA Read                 |
         |   <----------------------------------   |
         |         RDMA Response (RPC call)        |
         |   ---------------------------------->   |
         |                                         |
         |                                         | Processing
         |                                         |
         |      RDMA Send (RDMA2_REPLY_INLINE)     |
         |   <----------------------------------   | Reply

     Requester                                 Responder
         |      RDMA Send (RDMA2_CALL_INLINE)      |
    Call |   ---------------------------------->   |
         |                                         |
         |                                         | Processing
         |                                         |
         |          RDMA Write (RPC reply)         |
         |   <----------------------------------   |
         |     RDMA Send (RDMA2_REPLY_EXTERNAL)    |
         |   <----------------------------------   | Reply

6.6.4. Choosing a Reply Payload Format

A Requester provisions all necessary registered memory resources for both an RPC Call and its matching RPC Reply. A Requester constructs each RPC Call, thus it can compute the exact memory resources needed to send every Call. However, the Requester allocates memory resources to receive the corresponding Reply before the Responder has constructed it. Occasionally, it is challenging for the Requester to know in advance precisely what resources are needed to receive the Reply.¶

In RPC-over-RDMA version 2, a Requester can provide a Reply chunk for any transaction. The Responder can use the provided Reply chunk or it can decide to use another means to convey the RPC Reply. If the combination of the provided Write chunk list and Reply chunk is not adequate to convey a Reply, the Responder SHOULD use Message Continuation to send that Reply. If even that is not possible, the Responder sends an RDMA2_ERROR message to the Requester, as described in Section 6.3.1:¶

• If the Write chunk list cannot accommodate the ULP's DDP-eligible data payload, the Responder sends an RDMA2_ERR_WRITE_RESOURCE error.¶
• If the Reply chunk cannot accommodate the parts of the Reply that are not DDP-eligible, the Responder sends an RDMA2_ERR_REPLY_RESOURCE error.¶

When receiving such errors, the Requester can retry the ULP call using more substantial reply resources. In cases where retrying the ULP request is not possible (e.g., the request is non-idempotent), the Requester terminates the RPC transaction and presents an error to the RPC consumer.¶

7.1.1. RDMA2_ERR_VERS

When a Responder detects an RPC-over-RDMA header version that it does not support (the current document defines version 2), it MUST respond with an RDMA2_ERROR message type and set its rdma_err field to RDMA2_ERR_VERS. The Responder then fills in the rpcrdma2_err_vers structure with the RPC-over-RDMA versions it supports. The Responder MUST silently discard the ingress message without passing it to the RPC layer.¶

When a Requester receives this error message, it uses the information in the rpcrdma2_err_vers structure to select an RPC-over-RDMA version that both peers support for subsequent operations on the connection. A Requester MUST NOT subsequently send a message that uses a version that the Responder has indicated it does not support. RDMA2_ERR_VERS indicates a permanent error. Receipt of this error completes the RPC transaction associated with XID in the rdma_xid field.¶

7.1.2. RDMA2_ERR_VERS_MISMATCH

When a Responder receives a message with a transport protocol version that does not match the protocol version that was used in previous successful exchanges on the same connection, it MUST respond with an RDMA2_ERROR message type and set its rdma_err field to RDMA2_ERR_VERS_MISMATCH. The Responder MUST silently discard the ingress message without passing it to the RPC layer.¶

A Requester MUST NOT subsequently send a message that uses a protocol version that the Responder has indicated it does not recognize on this connection. The Requester can recover by sending the message again using a corrected protocol version, or it can terminate the RPC transaction associated with the XID in the rdma_xid field with an error.¶

7.1.3. RDMA2_ERR_INVAL_HTYPE

If a Responder recognizes the value in an ingress rdma_vers field, but it does not recognize the value in the rdma_htype field or does not support that header type, it MUST set the rdma_err field to RDMA2_ERR_INVAL_HTYPE. The Responder MUST silently discard the incoming message without passing it to the RPC layer.¶

A Requester MUST NOT subsequently send a message on the connection that uses an htype that the Responder has indicated it does not support. RDMA2_ERR_INVAL_HTYPE indicates a permanent error. Receipt of this error completes the RPC transaction associated with XID in the rdma_xid field.¶

7.1.4. RDMA2_ERR_INVAL_CONT

If a Responder detects a problem with an ingress RPC-over-RDMA message that is part of a Message Continuation sequence, the Responder MUST set the rdma_err field to RDMA2_ERR_INVAL_CONT. The Responder MUST silently discard all ingress messages with an rdma_xid field that matches the failing message without reassembling the payload.¶

RDMA2_ERR_INVAL_CONT indicates a permanent error. Receipt of this error completes the RPC transaction associated with XID in the rdma_xid field.¶

7.2.1. RDMA2_ERR_BAD_XDR

If a Responder recognizes the values in the rdma_vers field, but it cannot otherwise parse the ingress Transport stream, it MUST set the rdma_err field to RDMA2_ERR_BAD_XDR. The Responder MUST silently discard the ingress message without passing it to the RPC layer.¶

RDMA2_ERR_BAD_XDR indicates a permanent error. Receipt of this error completes the RPC transaction associated with XID in the rdma_xid field.¶

7.2.2. RDMA2_ERR_BAD_PROPVAL

If a receiver recognizes the value in an ingress rdma_which field, but it cannot parse the accompanying propval, it MUST set the rdma_err field to RDMA2_ERR_BAD_PROPVAL (see Section 5.1). The receiver MUST silently discard the ingress message without applying any of its property settings.¶

7.3.1. RDMA2_ERR_READ_CHUNKS

If a Requester presents more DDP-eligible arguments than a Responder is prepared to Read, the Responder MUST set the rdma_err field to RDMA2_ERR_READ_CHUNKS and set the rdma_max_chunks field to the maximum number of Read chunks the Responder can process. If the Responder implementation cannot handle any Read chunks for a request, it MUST set the rdma_max_chunks to zero in this response. The Responder MUST silently discard the ingress message without processing it further.¶

The Requester can reconstruct the Call using Message Continuation or a Special Format payload and resend it. If the Requester chooses not to resend the Call, it MUST terminate this RPC transaction with an error.¶

7.3.2. RDMA2_ERR_WRITE_CHUNKS

If a Requester has constructed an RPC Call with more DDP-eligible results than the Responder is prepared to Write, the Responder MUST set the rdma_err field to RDMA2_ERR_WRITE_CHUNKS and set the rdma_max_chunks field to the maximum number of Write chunks the Responder can return. The Requester can reconstruct the Call with no Write chunks and a Reply chunk of appropriate size. If the Requester does not resend the Call, it MUST terminate this RPC transaction with an error.¶

If the Responder implementation cannot handle any Write chunks for a request and cannot send the Reply using Message Continuation, it MUST return a response of RDMA2_ERR_REPLY_RESOURCE instead (see below).¶

7.3.3. RDMA2_ERR_SEGMENTS

If a Requester has constructed an RPC Call with a chunk that contains more segments than the Responder supports, the Responder MUST set the rdma_err field to RDMA2_ERR_SEGMENTS and set the rdma_max_segments field to the maximum number of segments the Responder can process. The Requester can reconstruct the Call and resend it. If the Requester does not resend the Call, it MUST terminate this RPC transaction with an error.¶

7.3.4. RDMA2_ERR_WRITE_RESOURCE

If a Requester has provided a Write chunk that is not large enough to contain a DDP-eligible result, the Responder MUST set the rdma_err field to RDMA2_ERR_WRITE_RESOURCE. The Responder MUST set the rdma_chunk_index field to point to the first Write chunk in the transport header that is too short, or to zero to indicate that it was not possible to determine which chunk is too small. Indexing starts at one (1), which represents the first Write chunk. The Responder MUST set the rdma_length_needed to the number of bytes needed in that chunk to convey the result data item.¶

The Requester can reconstruct the Call with more reply resources and resend it. If the Requester does not resend the Call (for instance, if the Responder set the index and length fields to zero), it MUST terminate this RPC transaction with an error.¶

7.3.5. RDMA2_ERR_REPLY_RESOURCE

If a Responder cannot send an RPC Reply using Message Continuation and the Reply does not fit in the Reply chunk, the Responder MUST set the rdma_err field to RDMA2_ERR_REPLY_RESOURCE. The Responder MUST set the rdma_length_needed to the number of Reply chunk bytes needed to convey the reply. The Requester can reconstruct the Call with more reply resources and resend it. If the Requester does not resend the Call (for instance, if the Responder set the length field to zero), it MUST terminate this RPC transaction with an error.¶

7.4.1. RDMA2_ERR_SYSTEM

If some problem occurs on a Responder that does not fit into the above categories, the Responder MAY report it to the Requester by setting the rdma_err field to RDMA2_ERR_SYSTEM. The Responder MUST silently discard the message(s) associated with the failing transaction without further processing.¶

RDMA2_ERR_SYSTEM is a permanent error. This error does not indicate how much of the transaction the Responder has processed, nor does it indicate a particular recovery action for the Requester. A Requester that receives this error MUST terminate the RPC transaction associated with the XID value in the RDMA2_ERROR message's rdma_xid field.¶

11.1.1. Protection Domains

The use of a Protection Domain to limit the exposure of memory regions to a single connection is critical. Any attempt by an endpoint not participating in that connection to reuse memory handles needs to result in immediate failure of that connection. Because ULP security mechanisms rely on this aspect of Reliable Connected behavior, implementations SHOULD cryptographically authenticate connection endpoints.¶

11.1.2. Handle (STag) Predictability

Implementations should use unpredictable memory handles for any operation requiring exposed memory regions. Exposing a continuously registered memory region allows a remote host to read or write to that region even when an RPC involving that memory is not underway. Therefore, implementations should avoid the use of persistently registered memory.¶

11.1.3. Memory Protection

Requesters should register memory regions for remote access only when they are about to be the target of an RPC transaction that involves an RDMA Read or Write.¶

Requesters should invalidate memory regions as soon as related RPC operations are complete. Invalidation and DMA unmapping of memory regions should complete before the receiver checks message integrity, and before the RPC consumer can use or alter the contents of the exposed memory region.¶

An RPC transaction on a Requester can terminate before a Reply arrives, for example, if the RPC consumer is signaled, or a segmentation fault occurs. When an RPC terminates abnormally, memory regions associated with that RPC should be invalidated before the Requester reuses those regions for other purposes.¶

11.1.4. Denial of Service

A detailed discussion of denial-of-service exposures that can result from the use of an RDMA transport appears in Section 6.4 of [RFC5042].¶

A Responder is not obliged to pull unreasonably large Read chunks. A Responder can use an RDMA2_ERROR response to terminate RPCs with unreadable Read chunks. If a Responder transmits more data than a Requester is prepared to receive in a Write or Reply chunk, the RDMA provider typically terminates the connection. For further discussion, see Section 6.3.1. Such repeated connection termination can deny service to other users sharing the connection from the errant Requester.¶

An RPC-over-RDMA transport implementation is not responsible for throttling the RPC request rate, other than to keep the number of concurrent RPC transactions at or under the per connection credit window (see Section 4.2.1). A sender can trigger a self-denial of service by exceeding the credit window repeatedly.¶

When an RPC transaction terminates due to a signal or premature exit of an application process, a Requester should invalidate the RPC's Write and Reply chunks. Invalidation prevents the subsequent arrival of the Responder's Reply from altering the memory regions associated with those chunks after the Requester has released that memory.¶

On the Requester, a malfunctioning application or a malicious user can create a situation where RPCs initiate and abort continuously, resulting in Responder replies that terminate the underlying RPC-over-RDMA connection repeatedly. Such situations can deny service to other users sharing the connection from that Requester.¶

11.2.1. RPC-over-RDMA Protection at Other Layers

For any RPC transport, utilizing RPCSEC_GSS integrity or privacy services has performance implications. Protection below the RPC implementation is often a better choice in performance-sensitive deployments, especially if it, too, can be offloaded. Certain implementations of IPsec can be co-located in RDMA hardware, for example, without change to RDMA consumers and with little loss of data movement efficiency. Such arrangements can also provide a higher degree of privacy by hiding endpoint identity or altering the frequency at which messages are exchanged, at a performance cost.¶

Implementations MAY negotiate the use of protection in another layer through the use of an RPCSEC_GSS security flavor defined in [RFC7861] in conjunction with the Channel Binding mechanism [RFC5056] and IPsec Channel Connection Latching [RFC5660].¶

11.2.2. RPCSEC_GSS on RPC-over-RDMA Transports

Not all RDMA devices and fabrics support the above protection mechanisms. Also, NFS clients, where multiple users can access NFS files, still require per-message authentication. In these cases, RPCSEC_GSS can protect NFS traffic conveyed on RPC-over-RDMA connections.¶

RPCSEC_GSS extends the ONC RPC protocol without changing the format of RPC messages. By observing the conventions described in this section, an RPC-over-RDMA transport can convey RPCSEC_GSS-protected RPC messages interoperably.¶

Senders MUST NOT reduce protocol elements of RPCSEC_GSS that appear in the Payload stream of an RPC-over-RDMA message. Such elements include control messages exchanged as part of establishing or destroying a security context, or data items that are part of RPCSEC_GSS authentication material.¶