| NFSv4 | D. Noveck, Ed. |
| Internet-Draft | EMC |
| Intended status: Informational | P. Shivam |
| Expires: April 08, 2012 | C. Lever |
| B. Baker | |
| ORACLE | |
| October 06, 2011 |
NFSv4.0 migration: Implementation experience and spec issues to resolve
draft-dnoveck-nfsv4-migration-issues-01
The migration feature of NFSv4 provides for moving responsibility for a single filesystem from one server to another, without disruption to clients. Recent implementation experience has shown problems in the existing specification for this feature. This document discusses the issues which have arisen and explores the options available for curing the issues via clarification and correction of the NFSv4.0 specification.
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on April 08, 2012.
Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
This document is in the informational category, and while the facts it reports may have normative implications, any such normative significance reflects the readers' preferences. For example, we may report that the reboot of a client with migrated state results in state not being promptly cleared and that this will prevent granting of conflicting lock requests at least for the lease time, which is a fact. While it is to be expected that client and server implementers will judge this to be a situation that is best avoided, the judgment of how pressing this issue is a judgment for the reader, and eventually the nfsv4 working group to make.
We do explore possible ways in which such issues can be avoided, with minimal negative effects, in the expectation that the working group will choose to address these issues, but the choice of exactly how to address this is best given effect in a working group document.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119].
In the context of this informational document, these normative keywords will always occur in the context of a quotation, most often direct but sometimes indirect. The context will make it clear whether the quotation is from:
Note that the examples below reflect current experience which arises from clients implementing the recommendation to use different nfs_client_id4 id strings for different server addresses, i.e. using what is later referred to herein as the "non-uniform client-string model"
This is simply because that is the experience implementers have had. The reader should not assume that in all cases, this practice is the source of the difficulty. It may be so in some cases but clearly it is not in all cases.
The following sort of situation has proved troublesome:
Note here that while it seems clear to us in this example that C-XYZ and C-ABC are from the same client, the server has no way to determine the structure of the "opaque" id. In the protocol, it really is opaque. Only the client knows which nfs_client_id4 values designate the same client on a different server.
Further problems arise from scenarios like the following.
Note that if the client used "C" (rather than "C-ABC") as the nfs_client_id4 id string, the exact same situation would arise.
One of the first cases in which this sort of situation has resulted in difficulties is in connection with doing a SETCLIENTID for callback update.
The SETCLIENTID for callback update only includes the nfs_client_id4, assuming there can only be one such with a given nfs_client_id4 value. If there are multiple, confirmed client records with identical nfs_client_id4 values, there is no way to map the callback update request to the correct client record.
One possible accommodation for this particular issue that has been used is to add a RENEW operation along with SETCLIENTID (on a callback update) to disambiguate the client.
When the client updates the callback info to the destination, the client would, by convention, send a compound like this:
{ RENEW clientid4, SETCLIENTID nfs_client_id4,verf,cb }
The presence of the clientid4 in the compound would allow the server to differentiate among the various leases that it knows of, all with the same nfs_client_id4 value.
While this would be a reasonable patch for an isolated protocol weakness, interoperable clients and servers would require that the protocol truly be updated to allow such a situation, specifically that of multiple clientid4's with the same nfs_client_id4 value. The protocol is currently designed and implemented assuming this can't happen. We need to either prevent the situation from happening, or fully adapt to the possibilities which can arise. See Section 4 for a discussion of such issues.
Consider the following situation:
Now, instead of a clientid4 identifying a client-server pair, we have many more entities for the client to deal with. In addition, it isn't clear how new state is to be incorporated in this structure.
The limitations of the migrated state (inability to be freed on reboot) would argue against adding more such state but trying to avoid that would run into its own difficulties. For example, a single lockowner string presented under two different clientids would appear as two different entities.
The current definitive definition of the NFSv4.0 protocol [RFC3530], and the current pending draft of RFC3530bis [cur-v4.0-bis] both agree. The section entitled "Client ID" says:
There are two possible interpretations of the phrase "uniquely defines" in the above:
The first interpretation would make these client-strings like phone numbers (a single person can have several) while the second would make them like social security numbers.
Endless debate about the true meaning of "uniquely defines" in this context is quite possible but not very helpful. The following points should be noted though:
Given the need for the server to be aware of client identity with regard to migrated state, either client-string construction rules will have to change or there will be need to get around current issues, or perhaps a combination of these two will be required. Later sections will examine the options and propose a solution.
One consideration that may indicate that this cannot remain exactly as it is today has to do with the fact that the current explanation for this behavior is not correct. The current definitive definition of the NFSv4.0 protocol [RFC3530], and the current pending draft of RFC3530bis [cur-v4.0-bis] both agree. The section entitled "Client ID" says:
In point of fact, a "SETCLIENTID with the same id string" sent to multiple network addresses will be treated as all from the same client but will not "cause the server to begin the process of removing the client's previous leased state" unless the server believes it is a newer instance of the same client, i.e. if the id is the same and there is a different verifier. If the client does not reboot, the verifier should not change. If it does reboot, the verifier will change, and the server should "begin the process of removing the client's previous leased state.
The situation of multiple SETCLIENTID requests received by a server on multiple network addresses is exactly the same, from the protocol design point of view, as when multiple (i.e. duplicate) SETCLIENTID requests are received by the server on a single network address. The same protocol mechanisms that prevent erroneous state deletion in the latter case prevent it in the former case. There is no reason for special handling of the multiple-network-appearance case, in this regard.
It is often felt that this is a consequence of the client-string construction issues, and it is certainly the case that the two are closely connected in that non-uniform client-strings make it impossible for the server to appropriately combine leases from the same client. See Section 5.2.1 for a discussion of non-uniform client-strings.
However, even where the server could combine leases from the same client, it needs to be clear how and when it will do so, so that the client will be prepared. These issues will have to be addressed at various places in the spec.
This could be enough only if we are prepared to do away with the "should" recommending non-uniform client-strings and replace it with a "should not" or even a "SHOULD NOT". Current client implementation patterns make this an unpalatable option. Alternatively, a way needs to be found for the server to infer from client behavior which leases are held by the same client and use this information to do appropriate lease mergers.
The fact that the reason given in client-string-BP3 is not valid makes the existing "should" insupportable. We can't either
So unless someone has a valid new reason, the "should" will have to go. The question is what to replace it with.
Given the difficulties caused by having different nfs_client_id4 client-string values for the same client, we have two choices:
There are a number of issues where the existing text is unclear and/or wrong and needs to be fixed in some way.
There are a number of cases in which certain sections, not specifically related to migration require additional clarification. This is generally because text that is clear in a context in which leases and clientids are created in one place and live there forever may need further refinement in the more dynamic environment that arises as part of migration.
Some examples:
We propose replacing client-string-BP3 with the following text and adding the following proposed Section 5.2 to provide implementation guidance.
One particular aspect of the construction of the nfs4_client_id4 string has proved recurrently troublesome. The client has a choice of:
Construction of the client-string has been a troublesome issue because of the way in which the NFS protocols have evolved.
NFSv4.0 is unfortunately halfway between these two. The two client-string models have arisen in attempts to deal with the changing requirements of the protocol as implementation has proceeded and features that were not very substantial in [RFC3530], got more substantial.
Both models have to deal with the asymmetry in client and server identity information between client and server. Each seeks to make the client's and the server's views match. In the process, each encounters some combination of inelegant protocol features and/or implementation difficulties. The choice of which to use is up to the client implementer and the sections below try to give some useful guidance.
The non-uniform client-string model is an attempt to handle these matters in NFSv4.0 client implementations in as NFSv3-like a way as possible.
For a client using the non-uniform model, all internal recording of clientid4 values is to include, whether explicitly or implicitly, the server IP address so that one always has an (IP-address, clientid4) pair. Two such pairs from different servers are always distinct even when the clientid4 values are the same, as they may occasionally be. In this model, such equality is always treated as simple happenstance.
Making the client-string different on different servers means that a server has no way of tying together information from the same client and so will treat a single client as multiple clients with multiple leases for each server network address. Since there is no way in the protocol for the client to determine if two network addresses are connected to the same server, the resulting lack of knowledge is symmetrical and can result in simpler client implementations in which there is a single clientid/lease per server network addresses.
Support for migration, particularly with transparent state migration, is more complex in the case of non-uniform client-strings. For example, migration of a lease can result in multiple leases for the same client accessing the same server addresses, vitiating many of the advantages of this approach. Therefore, client implementations using the non-uniform client-string model that support migration with transparent state migration SHOULD use the facilities described in the section entitled "Migration and Lease Merger" in order to maintain the property that for a single client there is simple one-to-one map to get from server network address to the corresponding lease.
These can be fairly described as lacking in elegance and this can be a factor in selecting a client-string model, when support for migration with transparent state migration is required. The inelegance arises from the fact that in the non- uniform model the client is telling two different servers that there are two clients when there is in fact only one. Migration causes a situation in which the client knows there is one client and one server (a single IP address), while the server thinks otherwise, which requires some sort of cleanup to be performed. If the inelegance of that process is troubling to the client implementer, he is free to use the uniform model.
When the client-string is kept uniform, the server has the basis to have a single clientid4/lease for each distinct client. The problem that has to be addressed is the lack of explicit server identity information, which is made available in NFSv4.1.
When the same client-string is given to multiple IP addresses, the client can determine whether two IP addresses correspond to a single server, based on the server's behavior. This is the inverse of the strategy adopted for the non-uniform model in which different server IP addresses are told about different clients, simply to prevent a server from manifesting behavior that is inconsistent with there being a single server for each IP address, in line with the traditions of NFS. So, to compare:
For a client using the uniform model, clientid4 values are treated as important information in determining server trunking patterns. For two different IP addresses to return the same clientid4 value is a necessary, though not a sufficient condition for them to be considered as connected to the same server. As a result, when two different IP addresses return the same clientid4, the client needs to determine, using the procedure given below or otherwise, whether the IP addresses are connected to the same server. For such clients, all internal recording of clientid4 values needs to include, whether explicitly or implicitly, identification of the server from which the clientid4 was received so that one always has a (server clientid4) pair. Two such pairs from different servers are always considered distinct even when the clientid4 values are the same, as they may occasionally be.
In order to make this approach work, the client must have accessible, for each nfs4_client_id4 used (only one in the uniform model) a list of all server IP addresses, together with the associated clientid4 values. As a part of the associated data structures, there should be the ability to mark a server IP structure as having the same server as another and to mark an IP-address as currently unresolved. One way to do this is to a allow each such entry to point to another with the pointer value being one of:
When a SETCLIENTID is done and a clientid4 returned, the data structure is searched for a matching clientid4 and processing depends on what is found. We will refer to the IP address on which this SETCLIENTID is done as X. The SETCLIENTID will use the common nfs_client_id4 and specify X as part of the callback parameters. We call the clientid4 and verifier returned by this operation XC and XV.
Note that at this point no SETCLIENTID_CONFIRM has yet been done. This is because we have either established a new clientid4 on a previously unknown server or changed the callback parameters on a clientid4 associated with some already known server. We don't want to confirm something that we are not sure we want to happen.
So for each lead IP address IPn with a clientid4 matching XC, the following steps are done.
The procedure above has made no explicit mention of the possibility that server reboot can occur at any time. To address this possibility the client should periodically use the clientid4 XC in RENEW operations, directed to both the IP address X and the current lead IP address that is currently being tested for identity.
If we have run out of IPn's without finding a matching server, X is considered as having no existing known IP addresses trunked with it. The IP address is marked as a lead IP address for a new server. A SETCLIENTID_CONFIRM is done using XC and XV.
The following are advantages for the implementation of using the uniform client-string model:
The following implementation considerations might cause issues for client implementations.
How to balance these considerations depends on implementation goals.
The current definitive definition of the NFSv4.0 protocol [RFC3530], and the current pending draft of RFC3530bis [cur-v4.0-bis] both agree. The section entitled "Migration and State" says:
There are a number of problems with this and any resolution of our difficulties must address them somehow.
To avoid client complexity, we need to have no more than one lease between a single client and a single server. This requires merger of leases since there is no real help from synchronizing them at a single instant.
We have to have support for both styles of client-string assignment: uniform and non-uniform.
We have made the following decisions as far as proposed normativity for state merger. They reflect the fact that we want to support migration for both id models and that we can't say MUST since we have older clients and servers to deal with.
If the clients and the servers obey the SHOULD's, having more than a single lease for a given client-server pair will be a transient situation, cleaned up as part of adapting to use of migrated state.
Since clients and servers will be a mixture of old and new and because nothing is a MUST we have to ensure that no combination will show worse behavior than is exhibited by current (i.e. old) clients and servers.
The current definitive definition of the NFSv4.0 protocol [RFC3530], and the current pending draft of RFC3530bis [cur-v4.0-bis] both agree. The section entitled "Migration and State" says:
This poses some difficulties, mostly because the part about "client ID" is not clear:
We have decided to address this issue as follows, with the relevant changes all reflected in Section 5.6.
The current definitive definition of the NFSv4.0 protocol [RFC3530], and the current pending draft of RFC3530bis [cur-v4.0-bis] both agree. The section entitled "Migration and State" says:
The above will need to be fixed to reflect the possibility of merging of leases and the text to do this appears as part of Section 5.6.
The current definitive definition of the NFSv4.0 protocol [RFC3530], and the current pending draft of RFC3530bis [cur-v4.0-bis] both agree. The section entitled "Notification of Migrated Lease" says:
There is a lack of clarity that is prompted by ambiguity about what exactly probing is and what the interlock between client and server must be. This has led to some worry about the scalability of the probing process, and although the time required does scale linearly with the number of fs's that the client may have state for with respect to a given server, the actual process can be done efficiently.
To address these issues we propose replacing the above with the following paragraphs, as is reflected in Section 5.6.
Some changes are necessary to reduce confusion about the process of callback information update and in particular to make it clear that no state is freed as a result:
To address both of the clientid4-related issues mentioned in Section 4.4, we propose replacing the last three paragraphs of the section entitled "Client ID" with the following:
When responsibility for handling a given filesystem is transferred to a new server (migration) or the client chooses to use an alternate server (e.g., in response to server unresponsiveness) in the context of filesystem replication, the appropriate handling of state shared between the client and server (i.e., locks, leases, stateids, and client IDs) is as described below. The handling differs between migration and replication.
If a server replica or a server immigrating a filesystem agrees to, or is expected to, accept opaque values from the client that originated from another server, then it is a wise implementation practice for the servers to encode the "opaque" values in network byte order. When doing so, servers acting as replicas or immigrating filesystems will be able to parse values like stateids, directory cookies, filehandles, etc. even if their native byte order is different from that of other servers cooperating in the replication and migration of the filesystem.
In the case of migration, the servers involved in the migration of a filesystem SHOULD transfer all server state from the original to the new server. This must be done in a way that is transparent to the client. This state transfer will ease the client's transition when a filesystem migration occurs. If the servers are successful in transferring all state, the client will continue to use stateids assigned by the original server. Therefore the new server must recognize these stateids as valid.
If transferring stateids from server to server would result in a conflict for an existing stateid for the destination server with the existing client, transparent state migration MUST NOT happen for that client. Servers participating in using transparent state migration should co-ordinate their stateid assignment policies to make this situation unlikely or impossible. The means by which this might be done, like all of the inter-server interactions for migration, are not specified by the NFS version 4.0 protocol.
Handling of clientid values is similar but not identical. The clientid4 and nfs_client_id4 information (id and verifier) will be transferred with the rest of the state information and the destination server should use that information to determine appropriate clientid4 handling. Although the destination server may make state stored under an existing lease available under the clientid4 used on the source server, the client should not assume that this is always so. In particular,
When leases are not merged, the transfer of state should result in creation of a confirmed client record with empty callback information but matching the {v, x, c} for the transferred client information. This should enable establishment of new callback information using SETCLIENTID and SETCLIENTID_CONFIRM.
A client may determine the disposition of migrated state by using a stateid associated with the migrated state and in an operation on the new server and using the associated clientid4 in a RENEW on the new server.
Since responsibility for an entire filesystem is transferred with a migration event, there is no possibility that conflicts will arise on the new server as a result of the transfer of locks.
The servers may choose not to transfer the state information upon migration. However, this choice is discouraged, except where specific issues such as stateid conflicts make it necessary. In the case of migration without state transfer, when the client presents state information from the original server (e.g. in a RENEW op or a READ op of zero length), the client must be prepared to receive either NFS4ERR_STALE_CLIENTID or NFS4ERR_STALE_STATEID from the new server. The client should then recover its state information as it normally would in response to a server failure. The new server must take care to allow for the recovery of state information as it would in the event of server restart.
When a lease is transferred to a new server (as opposed to being merged with a lease already on the new server), a client SHOULD re-establish new callback information with the new server as soon as possible, according to sequences described in sections "Operation 35: SETCLIENTID - Negotiate Client ID" and "Operation 36: SETCLIENTID_CONFIRM - Confirm Client ID". This ensures that server operations are not blocked by the inability to recall delegations.
The server SHOULD merge state for the same server that has come to be registered under different clientids because of migration, if the client clearly indicates its awareness of the identity of the clientids sharing the server state. This will simplify the client's state management task since there would then be only be a single clientid4 and a single lease for every client-server pair.
If a client references a stateids or a clientid4 that are associated with two different leases in the same COMPOUND, it can infer that the two leases are associated with the same client, making lease merger a possibility. If it were to act on this inference without any additional restrictions, the client might be exposed to confusion as a result of unexpected lease mergers occurring.
To prevent confusion and ensure interoperability without requiring undue client complexity the following rules apply:
When lease merger occurs the server moves state under the source lease together with that under the destination lease so that it is all designated by the target clientid4 and then removes the source clientid4. Owner sequencing information remains as it was under the target clientid4.
Given the above rules, the simplest of effecting such merger, when the server support lease merger as above, is to use a COMPOUND such as the following:
When a client finds out that migration has occurred, it should use the procedure described in Section 5.6.1 to determine the disposition of the client state associated with the migrated files. When it is determined that client state has been migrated but not merged into an existing lease on the client, the client can determine whether there is already an existing lease for the client in connection with the destination server. When there is, the client SHOULD use the COMPOUND above to merge the multiple per-clientid state corpora, so that the client will have only a single lease for each server on which it has state. In the case in which the client uses the non-uniform id model, the result will be a clientid4 for the merged lease whose associated nfs_client_id4 reflects the destination server. This is important so that when the verifier for the nfs_client_id4 changes, due a client reboot, the associated state will be promptly freed.
The client should note that when lease merger occurs it applies to all state under the specified clientid4's and not just to that for the fs motivating the operation. A client may note when a merger has already been done because of a migration event dealt with previously and avoid the operation or it can recover when the second RENEW finds that the clientid4-for-migrated-lease no longer exists, as will be the case when a previous merger has caused it to be deleted.
Since client switch-over in the case of replication is not under server control, the handling of state is different. In this case, leases, stateids and client IDs do not have validity across a transition from one server to another. The client must re-establish its locks on the new server. This can be compared to the re-establishment of locks by means of reclaim-type requests after a server reboot. The difference is that the server has no provision to distinguish requests reclaiming locks from those obtaining new locks or to defer the latter. Thus, a client re-establishing a lock on the new server (by means of a LOCK or OPEN request), may have the requests denied due to a conflicting lock. Since replication is intended for read-only use of filesystems, such denial of locks should not pose large difficulties in practice. When an attempt to re-establish a lock on a new server is denied, the client should treat the situation as if its original lock had been revoked.
In the case of lease renewal, the client may not be submitting requests for a filesystem that has been migrated to another server. This can occur because of the implicit lease renewal mechanism. The client renews a lease containing state of multiple filesystems when submitting a request to any one filesystem at the server.
In order for the client to schedule renewal of leases that may have been relocated to the new server, the client must find out about lease relocation before those leases expire. To accomplish this, all operations which implicitly renew leases for a client (such as OPEN, CLOSE, READ, WRITE, RENEW, LOCK, and others), will return the error NFS4ERR_LEASE_MOVED if responsibility for any of the leases to be renewed has been transferred to a new server. Note that when the transfer of responsibility leaves remaining state for that lease on the source server, the lease is renewed as normal despite returning the error. The transfer of responsibility happens when the server receives a GETATTR(fs_locations) from the client for each filesystem for which a lease has been moved to a new server. Normally it does this after receiving an NFS4ERR_MOVED for an access to the filesystem but the server is not required to verify that this happens in order to terminate the return of NFS4ERR_LEASE_MOVED. By convention, the compounds containing GETATTR(fs_locations) SHOULD include an appended RENEW operation to permit the server to identify the client getting the information.
Note that the NFS4ERR_LEASE_MOVED error is only required when responsibility for at least one stateid has been transferred. In the case of a null lease, where the only associated state is a clientid, no NFS4ERR_LEASE_MOVED error need be generated.
Upon receiving the NFS4ERR_LEASE_MOVED error, a client that supports filesystem migration MUST perform the necessary GETATTR operation for each of the filesystems containing state that have been migrated and so give the server evidence that it is aware of the migration of the filesystem. Once the client has done this for all migrated filesystems on which the client holds state, the server MUST resume normal handling of stateful requests from that client.
One way in which clients can do this efficiently in the presence of large numbers of filesystems is described below. This approach divides the process into two phases, one devoted to finding the migrated filesystems and the second devoted to doing the necessary GETATTRs.
The client can find the migrated filesystems by building and issuing one or more COMPOUND requests, each consisting of a set of PUTFH/GETFH pairs, each pair using an fh in one of the filesystems in question. All such COMPOUND requests can be done in parallel. The successful completion of such a request indicates that none of the fs's interrogated have been migrated while termination with NFS4ERR_MOVED indicates that the filesystem getting the error has migrated while those interrogated before it in the same COMPOUND have not. Those whose interrogation follows the error remain in an uncertain state and can be interrogated by restarting the requests from after the point at which NFS4ERR_MOVED was returned or by issuing a new set of COMPOUND requests for the filesystems which remain in an uncertain state.
Once the migrated filesystems have been found, all that is needed is for client to give evidence to the server that it is aware of the migrated status of filesystems found by this process, by interrogating the fs_locations attribute for an fh each of the migrated filesystems. The client can do this building and issuing one or more COMPOUND requests, each of which consists of a set of PUTFH operations, each followed by a GETATTR of the fs_locations attribute. A RENEW follows to help tie the operations to the lease returning NFS4ERR_LEASE_MOVED. Once the client has done this for all migrated filesystems on which the client holds state, the server will resume normal handling of stateful requests from that client.
In order to support legacy clients that do not handle the NFS4ERR_LEASE_MOVED error correctly, the server SHOULD time out after a wait of at least two lease periods, at which time it will resume normal handling of stateful requests from all clients. If a client attempts to access the migrated files, the server MUST reply NFS4ERR_MOVED.
When the client receives an NFS4ERR_MOVED error, the client can follow the normal process to obtain the new server information (through the fs_locations attribute) and perform renewal of those leases on the new server. If the server has not had state transferred to it transparently, the client will receive either NFS4ERR_STALE_CLIENTID or NFS4ERR_STALE_STATEID from the new server, as described above. The client can then recover state information as it does in the event of server failure.
Aside from recovering from a migration, there are other reasons a client may wish to retrieve fs_locations information from a server. When a server becomes unresponsive, for example, a client may use cached fs_locations data to discover an alternate server hosting the same fs data. A client may periodically request fs_locations data from a server in order to keep its cache of fs_locations data fresh.
Since a GETATTR(fs_locations) operation would be used for refreshing cached fs_locations data, a server could mistake such a request as indicating recognition of an NFS4ERR_LEASE_MOVED condition. Therefore a compound which is not intended to signal that a client has recognized a migrated lease SHOULD be prefixed with a guard operation which fails with NFS4ERR_MOVED if the file handle being queried is no longer present on the server. The guard can be as simple as a GETFH operation.
Though unlikely, it is possible that the target of such a compound could be migrated in the time after the guard operation is executed on the server but before the GETATTR(fs_locations) operation is encountered. When a client issues a GETATTR(fs_locations) operation as part of a compound not intended to signal recognition of a migrated lease, it SHOULD be prepared to process fs_locations data in the reply that shows the current location of the fs is gone.
In order that the client may appropriately manage its leases in the case of migration, the destination server must establish proper values for the lease_time attribute.
When state is transferred transparently, that state should include the correct value of the lease_time attribute. The lease_time attribute on the destination server must never be less than that on the source since this would result in premature expiration of leases granted by the source server. Upon migration in which state is transferred transparently, the client is under no obligation to re-fetch the lease_time attribute and may continue to use the value previously fetched (on the source server).
In the case in which lease merger occurs as part of state transfer, the lease_time attribute of the destination lease remains in effect. The client can simply renew that lease with its existing lease_time attribute. State in the source lease is renewed at the time of transfer so that it cannot expire, as long as the destination lease is appropriately renewed.
If state has not been transferred transparently (i.e., the client sees a real or simulated server reboot), the client should fetch the value of lease_time on the new (i.e., destination) server, and use it for subsequent locking requests. However the server must respect a grace period at least as long as the lease_time on the source server, in order to ensure that clients have ample time to reclaim their locks before potentially conflicting non-reclaimed locks are granted. The means by which the new server obtains the value of lease_time on the old server is left to the server implementations. It is not specified by the NFS version 4.0 protocol.
The purpose of this section is to examine the troubling results reported in Section 3.1. We will look at the scenarios as they would be handled within the proposal.
Because the choice of uniform vs. non-uniform nfs_client_id4 id strings is a "MAY", we want to show that the problems are fixed with either one of the choices, which we abbreviate as follows;
We will also have to take account of the various merger-related "SHOULD" clauses to better understand how they have addressed the issues seen, we abbreviate these (collectively known as "SHOULD-merges") as follows:
Let's look at the troublesome situation cited in Section 3.1.1. We have already seen what happens when MAY-NUF-CID holds but none of the SHOULD-merges are in effect. Now let's look at the situation if MAY-UF-CID holds, whether any of the SHOULD-merge conditions are in effect or not.
Now let's look at the troublesome situation when MAY-NUF-CID and SHOULD-BOTH-DM hold.
The correctness signature for this issue is
so if you have clients and servers that obey the SHOULD clauses, the problem is gone regardless of the choice on the MAY.
Now let's consider the scenario given in Section 3.1.2. We have already seen what happens when MAY-NUF-CID holds but none of the SHOULD-merges are in effect. Now let's look at the situation in MAY-NUF-CID holds and SHOULD-SVR-AM holds as well.
Now let's consider the same scenario in the situation in which MAY-UF-CID holds and SHOULD-SVR-AM holds as well.
Now let's consider the same scenario in a third situation in which MAY-NUF-CID holds and SHOULD-BOTH-DM holds as well.
The correctness signature for this issue is
so if you have clients and servers that obey the SHOULD clauses, the problem is gone regardless of the choice on the MAY.
Consider the following situation:
Now look what will happen under various scenarios:
The correctness signature for this issue is
so if you have clients and servers that obey the SHOULD clauses, the problem is gone regardless of the choice on the MAY.
We have seen that (SHOULD-SVR-AM & SHOULD-BOTH-DM) are sufficient to solve the problems people have experienced.
There are a few further points to be made.
The current definitive definition of the NFSv4.0 protocol [RFC3530], and the current pending draft of RFC3530bis [cur-v4.0-bis] both agree. The section entitled "Security Considerations" encourages that clients protect the integrity of the SECINFO operation, any GETATTR operation for the fs_locations attribute, and the operations SETCLIENTID/SETCLIENTID_CONFIRM. A migration recovery event can use any or all of these operations. We do not recommend any change here.
An unprotected state merge compound would allow a man-in-the-middle attacker to replace the migrated clientid. This attack prevents the destination server from properly identifying two leases to merge, or can even cause the server to merge leases from two unrelated clients.
If the attacker replaces the result of the state merge compound, the client is tricked into believing that either a merger succeeded when it didn't, or didn't succeed when it did. The consequences of this are that client can perform unneeded state recovery.
Thus it is imperative that the client protect the action of merging lease state, introduced in the present document, using an integrity-protecting security flavor. See section 9.1.1 of the current pending draft of RFC3530bis [cur-v4.0-bis] for further details.
This document does not require actions by IANA.
The editor and authors of this document gratefully acknowledge the contributions of Trond Myklebust of NetApp and Robert Thurlow of Oracle. We also thank Tom Haynes of NetApp and Spencer Shepler of Microsoft for their guidance and suggestions.
Special thanks go to members of the Oracle Solaris NFS team, especially Rick Mesta and James Wahlig, for their work implementing an NFSv4.0 migration prototype and identifying many of the issues documented here.
| [RFC2119] | Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. |
| [RFC3530] | Shepler, S., Callaghan, B., Robinson, D., Thurlow, R., Beame, C., Eisler, M. and D. Noveck, "Network File System (NFS) version 4 Protocol", RFC 3530, April 2003. |
| [RFC5661] | Shepler, S., Eisler, M. and D. Noveck, "Network File System (NFS) Version 4 Minor Version 1 Protocol", RFC 5661, January 2010. |
| [cur-v4.0-bis] |
Haynes, T. and D. Noveck, "Network File System (NFS) Version 4 Protocol", 2011. Work in progress. |