Internet-Draft | EVPN-IRB Extended Mobility | October 2023 |
Malhotra, et al. | Expires 13 April 2024 | [Page] |
The procedure to handle host mobility in a layer 2 Network with EVPN control plane is defined as part of [RFC7432]. EVPN has since evolved to find wider applicability across various IRB use cases that include distributing both MAC and IP reachability via a common EVPN control plane. MAC Mobility procedures defined in [RFC7432] are extensible to IRB use cases if a fixed 1:1 mapping between host IP and MAC is assumed across host moves. Generic mobility support for IP and MAC addresses that allows these bindings to change across moves IS REQUIRED to support a broader set of EVPN IRB use cases. EVPN all-active multi-homing further introduces scenarios that require additional consideration from mobility perspective. This document enumerates a set of design considerations applicable to mobility across these EVPN IRB use cases and updates sequence number assignment procedures defined in [RFC7432] to address these IRB use cases.¶
NOTE TO IESG (TO BE DELETED BEFORE PUBLISHING): This draft lists six authors which is above the required limit of five. Given significant and active contributions to the draft from all six authors over the course of six years, we would like to request IESG to allow publication with six authors. Specifically, the three Cisco authors are the original inventors of these procedures and contributed heavily to rev 0 draft, most of which is still intact. AT&T is also a key contributor towards defining the use cases that this document addresses as well as the proposed solution. Authors from Nokia and Juniper have further contributed to revisions and discussions steadily over last six years to enable respective implementations and a wider adoption.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 13 April 2024.¶
Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
EVPN-IRB enables advertising both MAC and IP routes via a single MAC+IP RT-2 advertisement. The MAC address is imported into the local bridge MAC table and enables L2 bridged traffic across the network overlay. The IP address is imported into the local ARP table in an asymmetric IRB design or imported into the IP routing table in a symmetric IRB design, and enables routed traffic across the layer 2 network overlay. Please refer to [RFC9135] for more background on EVPN IRB forwarding modes.¶
To support EVPN mobility procedure, a single sequence number mobility attribute is advertised with the combined MAC+IP route. A single sequence number advertised with the combined MAC+IP route to resolve both MAC and IP reachability implicitly assumes a 1:1 fixed mapping between IP and MAC. While a fixed 1:1 mapping between IP and MAC is a common use case that is addressed via existing MAC mobility procedure defined in [RFC7432], additional IRB scenarios need to be considered, that don't necessarily adhere to this assumption. Such use cases are common in a virtualized host envoronments where hosts attached to an EVPN network are virtual machines (VM) or containarized workloads. Following IRB mobility scenarios are considered:¶
While existing MAC mobility procedure can be used for MAC+IP move in the first scenario, subsequent scenarios result in a new MAC- IP association. As a result, a single sequence number assigned independently per-{MAC, IP} is not sufficient to determine most recent reachability for both MAC and IP, unless the sequence number assignment algorithm allows for changing MAC-IP bindings across moves.¶
This document updates sequence number assignment procedures defined in [RFC7432] to adequately address mobility support across EVPN-IRB overlay use cases that allow MAC-IP bindings to change across VM moves and can support mobility for both MAC and IP components carried in an EVPN RT-2 for these use cases.¶
In addition, for hosts on an ESI multi-homed to multiple PE devices, additional procedures are specified to ensure synchronized sequence number assignments across the multi-homing devices.¶
This document covers mobility for the following cases, independent of the overlay encapsulation (e.g.: MPLS, NVO Tunnel):¶
Following sections of the document are informative:¶
Following sections of the document are normative:¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
In an EVPN IRB scenario, where a single MAC+IP RT-2 advertisement carries both IP and MAC routes, a MAC only RT-2 advertisement is redundant for host MACs that are advertised via MAC+IP RT-2. As a result, a MAC only RT-2 is an optional route that may not be advertised from or received at an EVPN PE. This is an important consideration for mobility scenarios discussed in subsequent sections.¶
MAC only RT-2 may still be advertised for non-IP host MACs that are not advertised via MAC+IP RT-2.¶
This section describes the IRB mobility use cases considered in this document. Procedures to address them are covered later in section 6 and section 7.¶
This is the baseline case, wherein a host move results in both host MAC and IP moving together with no change in MAC-IP binding across a move. Existing MAC mobility defined in [RFC7432] may be leveraged to apply to corresponding MAC+IP route to support this mobility scenario.¶
This is the case, where a host move results in VM IP moving to a new MAC binding.¶
A host reload or an orchestrated host move that results in a host being re-spawned at a new location may result in host getting a new MAC assignment, while maintaining its existing IP address. This results in a host IP move to a new MAC binding:¶
IP-a, MAC-a ---> IP-a, MAC-b¶
This takes into account scenarios, where multiple hosts, each with a unique IP, may share a common MAC binding, and a host move results in a new MAC binding for the host IP.¶
As an example, hosts running on a single physical server, each with a unique IP, may share the same physical server MAC. In yet another scenario, an L2 access network may be behind a firewall, such that all hosts IPs on the access network are learnt with a common firewall MAC. In all such "shared MAC" use cases, multiple local MAC-IP ARP entries may be learnt with the same MAC. A host IP move, in such scenarios (for example, to a new physical server), could result in new MAC association for the host IP.¶
In both of the above scenarios, a combined MAC+IP EVPN RT-2 advertised with a single sequence number attribute implicitly assumes a fixed IP to MAC mapping. A host IP move to a new MAC breaks this assumption and results in a new MAC+IP route. If this new MAC+IP route is independently assigned a new sequence number, the sequence number can no longer be used to determine most recent host IP reachability in a symmetric EVPN-IRB design OR the most recent IP to MAC binding in an asymmetric EVPN-IRB design.¶
As an example, consider a topology shown in Figure 1, with host VMs sharing the physical server MAC. In steady state, IP1-M1 route is learnt at PE1, PE2 and advertised to remote PEs with a sequence number N. Now, VM-IP1 is moved to MAC Server-M2. ARP or ND based local learning at PE3, PE4 would now result in a new IP1-M2 route being learnt. If route IP1-M2 is learnt as a new MAC+IP route and assigned a new sequence number of say 0, mobility procedure for VM-IP1 will not trigger across the overlay network.¶
A sequence number assignment procedure needs to be defined to unambiguously determine the most recent IP reachability, IP to MAC binding, and MAC reachability for such a MAC sharing scenario.¶
This is a scenario where a host move or re-provisioning behind a new gateway location may result in the host getting a new IP address assigned, while keeping the same MAC.¶
The complication with this scenario is that MAC reachability could be carried via a combined MAC+IP route while a MAC only route may not be advertised at all. A single sequence number association with the MAC+IP route again implicitly assumes a fixed mapping between MAC and IP. A MAC move resulting in a new IP association for the host MAC breaks this assumption and results in a new MAC+IP route. If this new MAC+IP route independently assumes a new sequence number, this mobility attribute can no longer be used to determine the most recent host MAC reachability.¶
As an example, consider a host VM IP1-M1 that is learnt locally at PE1, PE2 and advertised to remote hosts with a sequence number N. Consider a scenario where this VM with MAC M1 is re-provisioned at server 2, however, as part of this re-provisioning, assigned a different IP address say IP7. IP7-M1 is learnt as a new route at PE3, PE4 and advertised to remote PEs with a sequence number of 0. As a result, L3 reachability to IP7 would be established across the overlay, however, MAC mobility procedure for M1 will not trigger as a result of this MAC-IP route advertisement. If an optional MAC only route is also advertised, sequence number associated with the MAC only route would trigger MAC mobility as per [RFC7432]. However, in the absence of an additional MAC only route advertisement, a single sequence number advertised with a combined MAC+IP route may not be sufficient to update MAC reachability across the overlay.¶
A MAC-IP sequence number assignment procedure needs to be defined to unambiguously determine the most recent MAC reachability in such a scenario without a MAC only route being advertised.¶
Further, PE1/PE2, on learning new reachability for IP7-M1 via PE3/PE4 MUST probe and delete any local IPs associated with MAC M1, such as IP1-M1 in the above example.¶
Arguably, MAC mobility sequence number defined in [RFC7432], could be interpreted to apply only to the MAC part of MAC-IP route, and would hence cover this scenario. This interpretation could be considered a clarification to [RFC7432] and one of the reasons for the common sequence number assignment procedure across all MAC-IP mobility scenarios detailed in this document.¶
Consider an EVPN-IRB overlay network shown in Figure 2, with hosts multi-homed to two or more PE devices via an all-active multi-homed ES. MAC and ARP entries learnt on a local ES may also be synchronized across the multi-homing PE devices sharing this ES. This MAC and ARP SYNC enables local switching of intra and inter subnet ECMP traffic flows from remote hosts. In other words, local MAC and ARP entries on a given ES may be learnt via local learning and / or via sync from another PE device sharing the same ES.¶
For a host that is multi-homed to multiple PE devices via an all- active ES interface, local learning of host MAC and MAC-IP at each PE device is an independent asynchronous event, that is dependent on traffic flow and or ARP / ND response from the host hashing to a directly connected PE on the MC-LAG interface. As a result, sequence number mobility attribute value assigned to a locally learnt MAC or MAC-IP route at each device may not always be the same, depending on transient states on the device at the time of local learning.¶
As an example, consider a host VM that is deleted from ESI-2 and moved to ESI-1. It is possible for host to be learnt on PE1 following deletion of the remote route from PE3, PE4, while being learnt on PE2 prior to deletion of remote route from PE3, PE4. If so, PE1 would process local host route learning as a new route and assign a sequence number of 0, while PE2 would process local host route learning as a remote to local move and assign a sequence number of N+1, N being the existing sequence number assigned at PE3, PE4.¶
Inconsistent sequence numbers advertised from multi-homing devices introduces:¶
As an example, in this inconsistent state, PE2 would drop a remote route received for the same host with sequence number N (as its local sequence number is N+1), while PE1 would install it as the best route (as its local sequence number is 0).¶
There is need for a mechanism to ensure consistency of sequence numbers advertised from a set of multi-homing devices for EVPN mobility to work reliably.¶
In order to support mobility for multi-homed hosts using the sequence number mobility attribute, local MAC and MAC-IP routes learnt on a multi-homed ES MUST be advertised with the same sequence number by all PE devices that the ES is multi-homed to. There is need for a mechanism to ensure consistency of sequence numbers assigned across these PEs.¶
To summarize, sequence number assignment scheme and implementation must take following considerations into account:¶
This section goes over the main components of the EVPN IRB mobility solution specified in this document. Later sections will specify exact sequence number assignment procedures resulting from concepts described in this section.¶
The main idea presented here is to view a local MAC-IP route as a child of the corresponding local MAC only route that inherits the sequence number attribute from the parent local MAC only route:¶
Mx-IPx -----> Mx (seq# = N)¶
As a result, both parent MAC and child MAC-IP routes share one common sequence number associated with the parent MAC route. Doing so ensures that a single sequence number attribute carried in a combined MAC+IP route represents sequence number for both a MAC only route as well as a MAC+IP route, and hence makes advertisement of the MAC only route truly optional. As a result, optional MAC only route with its own sequence number is not required to establish the most recent reachability for a MAC in the overlay network. Specifically, this enables a MAC to assume a different IP address on a move, and still be able to establish the most recent reachability to the MAC across the overlay network via the mobility attribute associated with the MAC+IP route advertisement. As an example, when Mx moves to a new location, it would result in local Mx being assigned a higher sequence number at its new location as per [RFC7432]. If this move results in Mx assuming a different IP address, IPz, local Mx+IPz route would inherit the new sequence number from Mx.¶
Local MAC and local MAC-IP routes would typically be sourced from data plane learning and ARP learning respectively, and could get learnt in the control plane in any order. Implementation could either replicate the inherited sequence number in each MAC-IP entry OR maintain a single attribute in the parent MAC by creating a forward reference local MAC object for cases where a local MAC-IP is learnt before the local MAC.¶
Further, for the shared MAC scenario, this results in multiple local MAC-IP siblings inheriting a sequence number attribute from the common parent MAC route:¶
In such a case, a host-IP move to a different physical server would result in IP moving to a new MAC binding. A new MAC-IP route resulting from this move must now be advertised with a sequence number that is higher than the previous MAC-IP route for this IP, advertised from the prior location. As an example, consider a route Mx-IPx that is currently advertised with sequence number N from PE1. IPx moving to a new physical server behind PE2 results in IPx being associated with MAC Mz. A new local Mz-IPx route resulting from this move at PE2 must now be advertised with a sequence number higher than N and higher than the previous Mz sequence number M. This is so that PE devices, including PE1, PE2, and other remote PE devices that are part of the overlay can clearly determine and program the most recent MAC binding and reachability for the IP. PE1, on receiving this new Mz-IPx route with sequence number say, N+1, for symmetric IRB case, would update IPx reachability via PE2 in forwarding, for asymmetric IRB case, would update IPx's ARP binding to Mz. In addition, PE1 would clear and withdraw the stale Mx-IPx route with the lower sequence number.¶
This also implies that sequence number associated with local MAC Mz and all local MAC-IP children of Mz at PE2 must now be incremented to N+1 or to M+1 if the previous Mz sequence number M is greater than N, and re-advertised across the overlay. While this re-advertisement of all local MAC-IP children routes affected by the parent MAC route is an overhead, it avoids the need for two separate sequence number attributes to be maintained and advertised for IP and MAC components of MAC+IP RT-2. Implementation would need to be able to lookup MAC-IP routes for a given IP and update sequence number for it's parent MAC and its MAC-IP children.¶
In order to support mobility for multi-homed hosts, local MAC and MAC-IP routes learnt on a shared ES MUST be advertised with the same sequence number by all PE devices that the ES is multi-homed to. This also applies to local MAC only routes. local MAC and MAC-IP may be learnt natively via data plane and ARP/ND respectively as well as via SYNC from another multi-homing PE to achieve local switching. Local and SYNC route learning can happen in any order. Local MAC-IP routes advertised by all multi-homing PE devices sharing the ES must carry the same sequence number, independent of the order in which they are learnt. This implies:¶
If an update to local MAC-IP sequence number is required as a result of the above comparison with SYNC MAC-IP route, it would essentially amount to a sequence number update on the parent local MAC, resulting in inherited sequence number update on the MAC-IP route.¶
Following sections specify sequence number assignment procedure needed on local and SYNC MAC and MAC-IP route learning events in order to accomplish the above.¶
A local Mx-IPx learning via ARP or ND should result in computation OR re-computation of the parent MAC Mx's sequence number, following which the MAC-IP route Mx-IPx would simply inherit parent MAC's sequence number. The parent MAC Mx Sequence number MUST be computed as follows:¶
Once the new sequence number for MAC route Mx is computed as per above, all local MAC-IPs associated with MAC Mx MUST inherit the updated sequence number.¶
The local MAC Mx Sequence number MUST be computed as follows:¶
Note that the local MAC sequence number might already be present if there was a local MAC-IP learnt prior to the local MAC, in which case the above may not result in any change in local MAC's sequence number.¶
On receiving a remote MAC OR MAC-IP route update associated with a MAC Mx with a sequence number that is¶
following handling is required on the receiving PE:¶
On receiving a REMOTE SYNC, the corresponding local MAC Mx (if present) sequence number should be re- computed as follows:¶
If this is a SYNC MAC-IP for a locally attached host, it would also result in a derived SYNC MAC Mx route entry, as MAC only RT-2 advertisement is optional. The corresponding local MAC Mx (if present) sequence number should be re-computed as follows:¶
In general, if all PE nodes in the overlay network follow the above sequence number assignment procedures, and the PE is advertising both MAC+IP and MAC routes, sequence numbers advertised with the MAC and MAC+IP routes with the same MAC would always be the same. However, an inter-op scenario with a different implementation could arise, where a PE implementation non-compliant with this document or with [RFC7432] assigns and advertises independent sequence numbers to MAC and MAC+IP routes. To handle this case, if different sequence numbers are received for remote MAC+IP and corresponding remote MAC routes from a remote PE, sequence number associated with the remote MAC route MUST be computed as:¶
A MAC and / or IP move to the local PE would now result in the MAC (and hence all MAC-IP) sequence numbers being incremented from the above computed remote MAC sequence number.¶
If MAC only routes are not advertised at all, and different sequence numbers are received with multiple MAC+IP routes for a given MAC, the sequence number associated with the derived remote MAC route should still be computed as the highest of all of the received MAC+IP sequence numbers with the same MAC.¶
In a MAC sharing use case described in section 6.2, a race condition is possible with simultaneous host moves between a pair of PEs. As an example, consider PE1 with local host IPs I1 and I2 sharing MAC M1, and PE2 with local host IPs I3 and I4 sharing MAC M2. A simultaneous move of I1 from PE1 to PE2 and of I3 from PE2 to PE1, such that I3 is learnt on PE1 before I1's local entry has been probed out on PE1 and/or I1 is learnt on PE2 before I3's local entry has been probed out on PE2 may trigger a race condition. This race condition together with MAC sequence number assignment rules defined in section 7.1 can cause new mac-ip routes I1-M2 and I3-M1 to bounce a couple of times with an incremented sequence number until stale entries I1-M1 and I3-M2 have been probed out from PE1 and PE2 respectively. An implementation MUST ensure proper probing procedures to remove stale ARP, ND, and local MAC entries, following a move, on learning remote routes as defined in section 7.3 (and as per [RFC9135]) to minimize exposure to this race condition.¶
This sections is optional and details ARP and ND probing procedures that MAY be implemented to achieve faster host re- learning and convergence on mobility events.¶
The above probing logic may be generalized as probing for an IP neighbor anytime a resolving parent MAC route is "inconsistent" with the MAC- IP neighbor route, where being inconsistent is defined as being not present or conflicting in terms of the route source being local OR remote. The MAC-IP to MAC parent relationship described earlier in this document in section 6.1 MAY be used to achieve this logic.¶
An additional use case is possible, such that traffic to an end host in the overlay is always IP routed. In a purely routed overlay such as this:¶
Please refer to [RFC7814] for more details.¶
Host mobility within the stretched subnet would still need to be supported for this use. In the absence of any host MAC routes, sequence number mobility Extended Community specified in [RFC7432], section 7.7 may be associated with a /32 OR /128 host IP prefix advertised via EVPN route type 5. MAC mobility procedures defined in [RFC7432] can now be applied as is to host IP prefixes:¶
Note that there is still only one sequence number associated with a host route at any time. For earlier use cases where a host MAC is advertised along with the host IP, a sequence number is only associated with a MAC. Only if the MAC is not advertised at all, as in this use case, is a sequence number associated with a host IP.¶
Note that this mobility procedure would not apply to "anycast IPv6" hosts advertised via NA messages with 0-bit=0. Please refer to [RFC9161].¶
Duplicate host detection scenarios across EVPN IRB can be classified as follows:¶
Duplicate detection procedures for scenario B and C would not apply to "anycast IPv6" hosts advertised via NA messages with 0-bit=0. Please refer to [RFC9161].¶
For all use cases where duplicate hosts have the same MAC, the MAC is detected as duplicate via the duplicate MAC detection procedure described in [RFC7432]. Corresponding MAC-IP routes with the same MAC do not require duplicate detection and MUST simply inherit the duplicate property from the corresponding MAC route. In other words, if a MAC route is in duplicate state, all corresponding MAC-IP routes MUST also be treated as duplicate. Duplicate detection procedure need only be applied to MAC routes.¶
Due to misconfiguration, a situation may arise where hosts with different MACs are configured with the same IP. This scenario would not be detected by [RFC7432] duplicate MAC detection procedures and would result in incorrect forwarding of routed traffic destined to this IP.¶
Such a situation, on local MAC-IP learning, would be detected as a move scenario via the following local MAC sequence number computation procedure described earlier in section 8.1:¶
Such a move that results in sequence number increment on local MAC because of a remote MAC-IP route associated with a different MAC MUST be counted as an "IP move" against the "IP" independent of the MAC. Duplicate detection procedure described in [RFC7432] can now be applied to an "IP" entity independent of MAC. Once an IP is detected as duplicate, corresponding MAC-IP route should be treated as duplicate. Associated MAC routes and any other MAC-IP routes associated with this MAC should not be affected.¶
The duplicate IP detection procedure for such a scenario are specified in [RFC9161]. What counts as an "IP move" in this scenario is further clarified as follows:¶
A MAC-IP route SHOULD be treated as duplicate if either of the following two conditions are met:¶
For a purely routed overlay scenario described in section 8, where only a host IP is advertised via EVPN RT-5, together with a sequence number mobility attribute, duplicate MAC detection procedures specified in [RFC7432] can be intuitively applied to IP only host routes for the purpose of duplicate IP detection.¶
Once a MAC or IP is marked as duplicate and frozen, corrective action must be taken to un-provision one of the duplicate MAC or IP. Un- provisioning a duplicate MAC or IP in this context refers to a corrective action taken on the host side. Once one of the duplicate MAC or IP is un-provisioned, normal operation would not resume until the duplicate MAC or IP ages out, following this correction, unless additional action is taken to speed up recovery.¶
This section lists possible additional corrective actions that could be taken to achieve faster recovery to normal operation.¶
Unfreezing the duplicate or frozen MAC or IP via a CLI can be used to recover from duplicate and frozen state following corrective un-provisioning of the duplicate MAC or IP.¶
Unfreezing the frozen MAC or IP via a CLI at a PE should result in that MAC or IP being advertised with a sequence number that is higher than the sequence number advertised from the other location of that MAC or IP.¶
Two possible corrective un-provisioning scenarios exist:¶
Unfreezing the duplicate and frozen MAC or IP, following the above corrective un-provisioning scenarios would result in recovery to steady state as follows:¶
Note that the probes referred to in the above scenarios are event driven probes resulting from receiving a route with a higher sequence number. Periodic probes resulting from refresh timers may also occur in addition as completely independent probes.¶
In addition to the above, route clearing CLIs may also be used to clear the local MAC or IP route, to be executed AFTER the duplicate host is un-provisioned:¶
Note that the route unfreeze CLI may still need to be run if the route was un-provisioned and cleared from the non-duplicate / non-frozen location. Given that unfreezing of the route via the un-freeze CLI would any ways result in auto-clearing of the route from the "un- provisioned" location, as explained in the prior section, need for a route clearing CLI for recovery from duplicate / frozen state is truly optional.¶
Security considerations discussed in [RFC7432] and [RFC9135] apply to this document. Methods described in this document further extend the consumption of sequence numbers for IRB deployments. They are hence subject to same considerations if the control plane or data plane was to be compromised. As an example, if host facing data plane is compromised, spoofing attempts could result in a legitimate host being perceived as moved, eventually resulting in the host being marked as duplicate. Considerations for protecting control and data plane described in [RFC7432] are equally applicable to such mobility spoofing use cases.¶
None.¶
Authors would like to thank Vibov Bhan and Patrice Brissete for feedback the process of design and implementation of procedures defined in this document. Authors would like to thank Wen Lin for a detailed review and valuable comments related to MAC sharing race conditions. Authors would also like to thank Saumya Dikshit for a detailed review and valuable comments across the document.¶