| Internet-Draft | Recursively Setting Directoriestle | March 2024 |
| Zhang, et al. | Expires 26 September 2024 | [Page] |
In recent years, the concept of near-data computing has been widely recognized in storage architectures. The core idea is to process data nearby, reduce the overhead of network transmission, and utilize the computing capability of smart devices (such as intelligent NICs, smart SSDs, and DPUs). This reduces CPU and memory usage of clients (computing nodes) and improves data processing efficiency. This design idea is applied in NFSv4.2 or future NFS verions, such as Server-Side Copy, in which client sends the control command and the storage server copies data without passing through the data between the client and storage server. Compared with traditional copy operations, data is read from the source storage server and then written to the target storage server after two network transmissions. Data transmission on the network is reduced, and bandwidth resources are greatly released. In addition, the client changes from an original data copy executor to a data copy controller, and a specific execution action is executed by the storage server. Therefore, a large amount of computing resources and memory resources are saved on the client side.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 26 September 2024.¶
Copyright (c) 2024 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
In actual storage applications, users often recursively set the attributes of directories and subitems(their subfiles and subdirectories). Message interaction between client and server is complex, and the client consumes a lot of resources, which does not match the concept of near-data computing. FIG. 1 shows the existing flowchart of recursively setting the attributes of all files under dir1.¶
Step 1: The client sends the readdir command to obtain the list of all files in dir1.¶
Step 2: The storage server responds to the readdir operation. If the directory contains many subitems, the client needs to run the readdir operation for multiple times.¶
Step 3: The client obtains the file list based on the response information, executes the getattr request for each file, and obtains the file attributes.¶
Step 4: The storage server responds to the getattr request.¶
Step 5: The client sends a setattr request.¶
Step 6: The storage server responds to the setattr request.¶
Repeat step 1 to step 6 to traverse all files. If the dir1 directory contains 100,000 files, repeat step 1 to step 6 for 100,000 times to recursively set the dir1 attribute. It can be learned that the process consumes CPU resources and memory resources of the client, and a large number of messages are exchanged between the client and the storage server. As a result, an end-to-end time for the attribute setting operation is relatively long.¶
preamble to the figure.¶
Client Server
+ +
| |
|------ Readdir ----------------------->|
|<--------------------------------------|
|------ Getattr ----------------------->|
|<--------------------------------------|
|------ Setattr ----------------------->|
|<--------------------------------------|
| .... |
| |
Figure 1: Existing flowchart for recursive set operation
¶
As you can see, this figure doodled and dawdled.¶
After adopting the concept of near data calculation, the above scenario can be optimized.¶
Step 1: The client identifies that the operation object of the attribute setting is a directory and the attribute setting is recursive, and invokes a new operation (for example, RECURSIVE_SET).¶
Step 2: The storage server recursively queries all files in the directory, sets attributes for each file, and sends a RECURSIVE_SET response.¶
Compared with the original process, this process not only saves the CPU and memory usage of the client, but also significantly reduces the number of messages exchanged between the client and server. This greatly improves the efficiency in recursive attribute setting scenarios.¶
Similar to the design of Server-Side Copy, the new operation word RECURSIVE_SET is used to recursively set the attributes of a directory and its subitems. This operation can be in synchronous or asynchronous mode, which is determined by the input parameter:¶
o If no back channel is created when the client and server establish a connection, the client can only use the synchronous mode in the RECURSIVE_SET request. If the client uses the asynchronous mode, the server returns the error code NFS4ERR_CB_PATH_DOWN.¶
o If a back channel is created when the client and server establish a connection, the RECURSIVE_SET request can be in synchronous or asynchronous mode.¶
Note to RFC Editor: this section may be removed on publication as an RFC.¶
A recommended Recursive Set operation in an synchronous mode is shown in Figure 2.¶
Step 1: The client sends a RECURSIVE_SET request. In the request, select the synchronization setting mode and set rsa_sync to true.¶
Step 2: If the storage server completes the setting within the timeout period, the storage server returns the request result to the client. If the setting is not complete within the timeout period, generate rsr_callback_id and rsr_recursiveverf. In addition, the rsr_callback_id, rsr_recursiveverf, and error code are sent to the client.¶
Step 3: The client sends a RECURSIVE_SET_STATUS query request. The request contains the information of rss_stateid. The value of rss_stateid is obtained from the response packet of RECURSIVE_SET which value is the value rsr_callback_id. If the value of rss_stateid is the same as the value of rsr_callback_id cached on the storage server, the storage server returns the setting result. Storage server return NFS4_OK if the server successfully finish the recursive setting or NFS4_Pending if the setting is till on going, or else set failure error code. If the value of rss_stateid in the request is different from the value cached on the server, the storage server returns the error code NFS4ERR_BAD_STATEID.¶
Step 4: After receiving the RECURSIVE_SET_STATUS response, the client delays sending the RECURSIVE_SET_STATUS request if the error code is NFS4_PENDING. The delay setting request must contain rss_stateid. If the error code returned by the server is NFS4_OK, the recursive attribute setting is successful. If the error code is the error code of setattr defined in NFSv4.2, the recursive attribute setting fails. In this case, the client returns a response to the application.¶
preamble to the figure.¶
Client Server
+ +
| |
|------ RECURSIVE_SET(rsa_sync = 1) --------------------->|
| |
|<-----Response(rsr_callback_id=0, rsr_recursiveverf=0)---| within the timeout period
| |
| |
|<----Response(rsr_callback_id=1, rsr_recursiveverf=1)----| beyond the timeout period
| |
| |
| |
|-------RECURSIVE_SET_STATUS(rss_stateid=1)-------------->|
| |
|<------Response------------------------------------------|
| |
| |
Figure 2: A synchronous Recursive Set
¶
An alternative Recursive Set operation in an synchronous mode is also given in Figure 3.¶
As you can see, this figure doodled and dawdled.¶
Step 1: The client sends a RECURSIVE_SET request. In the request, select the synchronization setting mode and set rsa_sync to true.¶
Step 2: If the storage server completes the setting within the timeout period, the storage server returns the request result to the client. If the setting is not complete within the timeout period, generate rsr_callback_id and rsr_recursiveverf, and set the error code to NFS4_PENDING. In addition, the rsr_callback_id, rsr_recursiveverf, and error code are sent to the client.¶
Step 3: The client sends a RECURSIVE_SET_STATUS query request. The request contains the information of rss_stateid. The value of rss_stateid is obtained from the response packet of RECURSIVE_SET which value is the value rsr_callback_id. If the value of rss_stateid is the same as the value of rsr_callback_id cached on the storage server, the storage server returns the setting result. (if Success, set NFS4_Pending, or else set failure error code). If the value of rss_stateid in the request is different from the value cached on the server, the storage server returns the error code NFS4ERR_BAD_STATEID.¶
Step 4: After receiving the RECURSIVE_SET response, the client delays sending the RECURSIVE_SET_STATUS request if the error code is NFS4_PENDING. The delay setting request must contain rss_stateid. If the error code returned by the server is NFS4_OK, the recursive attribute setting is successful. If the error code is the error code of setattr defined in NFSv4.2, the recursive attribute setting fails. In this case, the client returns a response to the application.¶
A recommended Recursive Set operation in an asynchronous mode is shown in Figure 3.¶
Step 1: The client sends a RECURSIVE_SET request. In the request, the asynchronous setting mode is rsa_sync=false.¶
Step 2: The storage server needs to generate rsr_callback_id and rsr_recursiveverf, and set the error code to NFS4_OK. In addition, the rsr_callback_id, rsr_recursiveverf, and error code are sent to the client. The storage server continues the recursive setting operation.¶
Step 3: After receiving the response, the client checks the error code and starts an asynchronous task to receive the callback message from the server.¶
Step 4: The client create an asynchronous listening task and matches rsr_callback_id and rsr_recursiveverf. If the matching succeeds, the task is successfully executed. If rsr_callback_id can be matched but rsr_recursiveverf cannot be matched, client skip the message.¶
Step 5: After recursively setting attributes, the storage server sends a message to the client through the backchannel of the NFS4 and notifies the client of the setting result.¶
Step 6: If the client does not receive the asynchronous message, the started task is forcibly terminated when the session is destroyed. If an error occurs when the storage server recursively sets subitem attributes, the storage server terminates the task and returns the error code to the client. All possible errors are subject to the error codes defined by setattr.¶
preamble to the figure.¶
Client Server
+ +
| |
|------ RECURSIVE_SET(rsa_sync = 0) --------------------->|
| |
|<------Response(rsr_callback_id=1, rsr_recursiveverf=1)--|
| |
| |
|<------CB_RECURSIVE_SET----------------------------------|
| |
| |
| |
Figure 3: An asynchronous Recursive Set
¶
As you can see, this figure doodled and dawdled.¶
4.1 Operation TBD1: RECURSIVE_SET – Recursively sets the attributes of a directory and its subitems¶
ARGUMENT¶
<CODE BEGINS>¶
Struct RECURSIVE_SET4args {¶
bool rsa_sync;¶
};¶
<CODE ENDS>¶
RESULT¶
<CODE BEGINS>¶
struct recursive_set_response4 {¶
stateid4 rsr_callback_id;¶
verifier4 rsr_recursiversr_recursiveverf;¶
};¶
union RECURSIVE_SET4res (nfsstat4 rsr_status) {¶
case NFS4_OK:¶
recursive_set_response4 rsr_resok4;¶
default:¶
void;¶
};¶
<CODE ENDS>¶
DESCRIPTION¶
The RECURSIVE_SET operation is used by the client to recursively set the attributes of a directory and all its subitems. The operation should be placed after setattr in the existing setattr operation. After the storage server receives the setattr combination operation, if the setattr operation does not followed by RECURSIVE_SET, the original process remains unchanged. If the setattr operator is followed by the RECURSIVE_SET operation, the storage server considers the attributes of the directory and its subitems to initiate recursive set mode.¶
If the storage is successfully executed, the values of rsr_callback_id and rsr_recursiversr_recursiveverf are 0.¶
If the storage is not complete, the values of rsr_callback_id and rsr_recursiversr_recursiveverf are generated.¶
If rsa_sync is set to true, there are two processing policies:¶
o According to the NFSv4 protocol, the client must wait for the response from the server. Therefore, the client can wait for the processing result from the server. A problem in this mode is that the current request occupies a slot in a session, resulting in a decrease in the number of available slots. If multiple tasks of the same type are being executed, no slot is available on the client in severe cases.¶
o The storage server determines the implementation duration. If the implementation duration is too long, the storage server may return non-zero values of rsr_callback_id and rsr_recursiversr_recursiveverf.¶
After the client receives the request, the client waits for a period of time and executes RECURSIVE_SET_STATUS to query the execution progress of the current task. If the server does not finish the execution , NFS4ERR_PENDING is returned. After receiving the error code, the client retryes the query after a period of time. If the execution is complete, NFS4_OK is returned.¶
4.2 Operation TBD2: RECURSIVE_SET_STATUS – Query the result of the recursively setting the attributes of directories and their subitems¶
ARGUMENT¶
<CODE BEGINS>¶
struct RECURSIVE_SET_STATUS4args {¶
stateid4 rssa_stateid;¶
};¶
<CODE ENDS>¶
RESULT¶
<CODE BEGINS>¶
#define NFS4ERR_PENDING 10090¶
struct RECURSIVE_SET_STATUS4res {¶
nfsstat4 rssr_status;¶
};¶
<CODE ENDS>¶
DESCRIPTION¶
The RECURSIVE_SET_STATUS operation is used by the client to query the status of a recursively set task (attributes of a directory and its subitems). If the task on the storage server is complete, NFS4_OK is returned. If any error occurs during task execution, a response error code is returned and the error code is not extended or modified in this case so the error code is the same as the error code that may occur during the setattr operation. If the current setting task is not complete, NFS4_PENDING is returned.¶
4.3 Operation TBD3: RECURSIVE_SET_CANCEL – Canceling a Running Task on the Client¶
ARGUMENT¶
<CODE BEGINS>¶
//The following operation is used to cancel the recursive setting task that is being executed.¶
struct RECURSIVE_SET_CANCEL4args {¶
stateid4 rsca_stateid;¶
};¶
<CODE ENDS>¶
RESULT¶
<CODE BEGINS>¶
struct RECURSIVE_SET_CANCEL4res {¶
nfsstat4 rscr_status;¶
};¶
<CODE ENDS>¶
DESCRIPTION¶
RECURSIVE_SET_CANCEL is used to cancel the task that is being executed. The request packet contains rsca_stateid. The value of rsca_stateid is obtained from the response of RECURSIVE_SET. If the storage server fails to cancel the task, NFS4ERR_DELAY is returned. When receiving the message, the client delays the retry. If the current task is complete, NFS4_OK is returned.¶
4.4 Operation TBD4: CB_RECURSIVE_SET_NOTIFY – Notify the recursively setting result to client¶
ARGUMENT¶
<CODE BEGINS>¶
struct CB_RECURSIVE_SET_NOTIFY4args {¶
nfs_fh4 crsna_fh;¶
stateid4 crsna_stateid;¶
verifier4 crsna_recursiveverf;¶
nfsstat4 crsna_status;¶
};¶
<CODE ENDS>¶
RESULT¶
<CODE BEGINS>¶
struct CB_RECURSIVE_SET_NOTIFY4res {¶
nfsstat4 crsnr_status;¶
};¶
<CODE ENDS>¶
DESCRIPTION¶
CB_RECURSIVE_SET_NOTIFY is used to send the server callback to client to notify the client of the result of the task of recursively setting the attributes of directories and their subitems. Client check the crsna_stateid and crsna_recursiveverf and client will finish the wait task if they are matched the value received from previous RECURSIVE_SET response or will skip the notification in case of not match and return NFS4ERR_BAD_STATEID to server.¶
TBD¶
TBD¶