BGP Extension for 5G Edge Service Metadata

3.1. Metadata Influenced BGP Path Selection

When an ingress router receives BGP updates for the same IP prefix from multiple egress routers, all these egress routers' loopback addresses are considered as the next hops for the IP prefix. For the selected low latency edge services, the ingress router BGP engine would call an edge service Management function that can select paths based on the edge service Metadata received. Section 5.1 has an exemplary algorithm to compute the weighted path cost based on the edge service Metadata carried by the Sub-TLV(s) specified in this document.¶

Section 5 has the detailed description of the edge service Metadata influenced optimal path selection.¶

3.2. Ingress Router Forwarding Behavior

When the ingress router receives a packet and does a lookup on the route in the FIB, it gets the destination prefix's whole path. It encapsulates the packet destined towards the optimal egress node.¶

For subsequent packets belonging to the same flow, the ingress router needs to forward them to the same egress router unless the selected egress router is no longer reachable. Keeping packets from one flow to the same egress router, a.k.a. Flow Affinity, is supported by many commercial routers. Most registered EC services have relatively short flows.¶

How Flow Affinity is implemented is out of the scope for this document.¶

3.3. Forwarding Behavior when UEs Move

When a UE moves to a new 5G gNB which is anchored to the same UPF, the packets from the UE traverse to the same ingress router. Path selection and forwarding behavior are same as before.¶

If the UE maintains the same IP address when anchored to a new UPF, the directly connected ingress router might use the information passed from a neighboring router to derive the optimal Next Hop for this route. The detailed algorithm is out of the scope of this document.¶

4.1. Metadata Path Attribute

The Metadata Path Attribute is an optional BGP Path attribute to carry metrics and metadata about the edge services attached to the egress router. The Metadata Path Attribute, to be assigned by IANA [RFC2042], consists of a set of Sub-TLVs, and each Sub-TLV contains information for specific metrics of the edge services.¶

      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |   Attr. Flags |MetaDataPathAtt|        Length (2 Octets)      |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                                                               |
     |         Value (multiple Metadata Sub-TLVs)                    |
     |                                                               |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

4.2. The Site Preference Index Sub-TLV

Different services might have different preference index values configured for the same site. For example, Service-A requires high computing power, Service-B requires high bandwidth among its microservices, and Service-C requires high volume storage capacity. For a DC with relatively low storage capacity but high bisectional bandwidth, its preference index value for Service-B is higher and lower for Service-C. Site Preference Index can also be used to achieve stickiness for some services.¶

It is out of the scope of this document how the preference index is determined or configured.¶

The Preference Index Sub-TLV has the following format:¶

   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |Site-Preference-Index Sub-Type |               Length          |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                   Preference Index value                      |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

-

Site-Preference-Index Sub-Type =1 (specified in this document).¶

-

Preference Index value: 1 .. (2^32-1); the higher the value, the more preference the site. Preference Index value == 0 is reserved.¶

4.3. Site Physical Availability Index Metadata

The Site Physical Availability Index indicates the percentage of impact on a group of routes associated with a common physical characteristic, for example, a pod, a row of server racks, a floor, or an entire DC. The purpose is to use one UPDATE message to indicate a group of routes of different NLRIs impacted by a physical event. For example, a power outage to a pod can cause the Site Physical Availability Index to be 0% for all the routes in the pod. Partial fiber cut to a row of shelves can cause the Site Physical Availability Index to 50% for all the routes in those shelves. The value is 0-100, with 100% indicating the site is fully functional, 0% indicating the site is entirely out of service, and 50% indicating the site is 50% degraded.¶

It is recommended to assign each route with one Site-ID. Depending on deployment, one DC can use POD number as Site-ID, another DC can use Row of Shelves as the Site-ID.¶

Cloud Site/Pod failures and degradation include but are not limited to, a site degradation or an entire site going down caused by a variety of reasons, such as fiber cut connecting to the site or among pods, cooling failures, insufficient backup power, cyber threats attacks, too many changes outside of the maintenance window, etc. Fiber-cut is not uncommon within a Cloud site or between sites.¶

When those failure events happen, the edge (egress) router is running fine. Therefore, the ingress routers with paths to the egress router can't use BFD to detect the failures.¶

When there is a failure occurring at an edge site (or a pod), many instances can be impacted. In addition, the routes (i.e., the IP addresses) in the site might not be aggregated nicely. Instead of many BGP UPDATE messages to the ingress routers for all the instances, i.e. routes, impacted, the egress router can send one single BGP UPDATE to indicate the capacity availability of the site. The ingress routers can switch all or a portion of the instances associated with the site depending on how much the site is degraded.¶

The BGP UPDATE for the individual instances (i.e., the routes) can include the Capacity Availability Index solely for ingress routers to associate the routes with the Side-ID. The actual Capacity Availability Index value, i.e., the percentage for all the routes associated with the Side-ID, is generated by the egress routers with the egress routers' loopback address as the NLRI.¶

The Site Physical Availability Index Sub-TLV has fixed length of 4 Octets. Therefore there is no Length field.¶

 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      PhyAvailIdx Sub-Type     |I|         Reserved            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|        Site-ID (2 octets)     | Site Availability Percentage  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

- PhyAvailIdx:

Site-Physical-Availability-Index Sub-Type=2 (Specified in this document).¶

- Site ID:

is an identifier for a group of routes associated with a common physical characteristic, for example, a pod, a row of server racks, a floor, or an entire DC. The purpose is to use one UPDATE message to indicate a group of routes impacted by a physical event. Those routes might be from different address families or NLRIs. There could be multiple sites connected to one egress router (a.k.a. Edge DC GW).¶

Route-Flag I:

When set to 1, the Site Availability Index is for BGP speakers (receivers) to associate the routes with the Site-ID. The Site Availability Percentage value is ignored. When set to 0, the BGP speakers (receivers) should apply the Site Availability Index value to all the routes associated with the Site-ID.¶

Site Availability Percentage:

When the RouteFlag-I is 1, the Site Availability Percentage is ignored by the Ingress routers. When the RouteFlag I is set to 0, the Site Availability Percentage represents the percentage of the site availability for all the routes associated with the Site-ID, e.g., 100%, 50%, or 0%. When a site goes dark, the Index is set to 0. 50 means 50% functioning. When the value is outside the 0-100% range, the value carried in this Sub-TLV is ignored.¶

4.4. Service Delay Prediction Index

It is desirable for an ingress router to select a site with the shortest processing time for an ultra-low latency service. But it is not easy to predict which site has "the fastest processing time" or "the shortest processing delay" for an incoming service request because:¶

-

The given service instance shares the same physical infrastructure with many other applications and service instances. Service requests by other applications, UEs, or applications running behavior can impact the processing time for the given service instance.¶

-

The given service instance can be served by a cluster of servers behind a Load Balancer. To the network, the service is identified by one service ID.¶

-

The service complexity is different. One service may call many microservices, need to access multiple backend databases, and need to go through sophisticated security scrubbing functions, etc. Another service can be processed by a few simple steps. Without the application internal logic, it is not easy to estimate the processing time for future service requests.¶

Even though utilization measurements, like those below, are collected by most data centers, they cannot indicate which site has the shortest processing time. A service request might be processed faster on Site-A even if Site-A is overutilized.¶

o

Server utilization for the server where the instance is instantiated.¶

o

The network utilization for the links to the server where the instance is instantiated.¶

o

The number of databases that the service instance will access.¶

o

The memory utilization of the databases¶

The remaining available resource at a site is a more reasonable indication of process delay for future service requests.¶

o

The remaining available Server resources.¶

o

The remaining available network utilization for the links to the server where the instance is instantiated.¶

o

The number of databases that the service instance will access.¶

o

The remaining storage available for the databases.¶

The Service Delay Prediction Index is a value that predicts processing delays at the site for future service requests. The higher the value, the longer of the delay.¶

 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ServiceDelayPredict Sub-Type  |   Length      |F| Reserved    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|         Service Delay Predication Value                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  | Raw-Load-Measurement Sub-Type |               Length          |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                   Measurement Period                          |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           total number of packets to the Edge Service         |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           total number of packets from the Edge Service       |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           total number of bytes to the Edge Service           |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           total number of bytes from the Edge Service         |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

5.1. Egress Node Behavior

Multiple instances of the same service could be attached to one egress router. When all instances of the same service are grouped behind one application layer load balancer, they appear as one single route to the egress router, i.e., the application loader balancer's prefix. Under this scenario, the compute metrics for all those instances behind one application layer balancer are aggregated under the application load balancer's prefix. In this case, the compute metrics aggregated by the Load Balancer are visible to the egress router as associated with the Load Balancer's prefix. However, how the application layer Load Balancers distribute the traffic among different instances is out of the scope of this document. When multiple instances of the same service have different paths or links reachable from the egress router, multiple groups of metrics from respective paths could be exposed to the egress router. The egress router can have preconfigured policies on aggregating various metrics from different paths and the corresponding policies in selecting a path for forwarding the packets received from ingress routers. The aggregated metrics can be carried in the BGP Update messages instead of detailed measurements to reduce the entries advertised by the control plane and dampen the routes update in the forwarding plane. Upon receiving packets from ingress routers, the egress router can use its policies to choose an optimal path to one service instance. It is out of the scope of this document how the measurements are aggregated on egress routers and how ingress routers are configured with the algorithms to integrate the aggregated metrics with network layer metrics.¶

Many measurements could impact and correspondingly reflect service performance. In order to simplify an optimal selection process, egress routers can have preconfigured policies or algorithms to aggregate multiple metrics into one simple one to ingress routers. Though out of the scope of this document, an egress router can also have an algorithm to convert multiple metrics to network metrics, an IGP cost for each instance, to pass to ingress nodes. This decision-making process integrates network metrics computed by traditional IGP/BGP and the service delay metrics from egress routers to achieve a well-informed and adaptive routing approach. This intelligent orchestration at the edge enhances the service's overall performance and optimizes resource utilization across the distributed infrastructure. When the egress has merged the compute metrics from the local sites behind it, it can include one or more aggregated compute metrics in the Metadata Path Attribute in the BGP UPDATE to the Ingress. Also, an identifier or flag can be carried to indicate that the metrics are merged ones. After receiving the routes for the Service ID with the identifier, the ingress would do the route selection based on pre-configured algorithms (see Section 3 of this document).¶

5.2. Integrating Network Delay with the Service Metrics

As the service metrics and network delays are in different units, here is an exemplary algorithm for an ingress router to compare the cost to reach the service instances at Site-i or Site-j.¶

                ServD-i * CP-j               Pref-j * NetD-i
Cost-i=min(w *(----------------) + (1-w) *(------------------))
                ServD-j * CP-i               Pref-i * NetD-j

CP-i:

Capacity Availability Index at Site-i. A higher value means higher capacity available.¶

NetD-i:

Network latency measurement (RTT) to the Egress Router at the site-i.¶

Pref-i:

Preference Index for Site-i, a higher value means higher preference.¶

ServD-i:

Service Delay Predication Index at Site-i for the service, i.e., the ANYCAST address [RFC4786] for the service.¶

w:

Weight is a value between 0 and 1. If smaller than 0.5, Network latency and the site Preference have more influence; otherwise, Service Delay and capacity availability have more influence.¶

When a set of service Metadata is converted to a simple metric, a decision process is determined by the metric semantics and deployment situations. The goal is to integrate the conventional network decision process with the service Metadata into a unified decision-making process for path selection.¶

5.3. Integrating with BGP decision process

When an ingress router receives BGP updates for the same IP address from multiple egress routers, all those egress routers are considered as the next hops for the IP address. For the selected services configured to be influenced by the edge service Metadata, the ingress router BGP Decision process [IDR-CUSTOM-DECISION] would trigger the edge service Management function to compute the weight to be applied to the route's next hop in the forwarding plane. The decision process is influenced by the edge service Metadata associated with the client routes, such as Capacity Availability Index, Site Preference, and Service Delay Prediction Index, in addition to the traditional BGP multipath computation algorithm, such as the Weight, Local preference, Origin, MED, etc., shown below:¶

                     BGP ANYCAST Update
   +--------+ with Metadata    +---------------+
   | BGP    |----------------->| EdgeServiceMgn|
   |Decision|< - - - - - - - - |               |
   +---^-|--+                  +-------|-------+
       | | BGP ANYCAST                 | Update Anycast
       | | Route                       | Route Nexthops
       | | Multi-path NH install       | with weight
   +---|-V--+                          |
   |   RIB  |                          |
   +----+---+                          |
        |                              |
    +---V------------------------------V-------+
    |               Forwarding Plane           |
    |                                          |
    +------------------------------------------+

When any of those metadata value goes to 0, the effect is the same as the routes becoming ineligible via the egress router who originates the metadata UPDATE. But when any of those metadata just degrade, there is possibility, even though smaller, for the egress router to continue as the optimal next hop.¶

Suppose a destination address for aa08::4450 can be reached by three next hops (R1, R2, R3). Further, suppose the local BGP's Decision Process based on the traditional network layer policies and metrics identifies the R1 as the optimal next hop for this destination (aa08::4450). If the edge service Metadata results in R2 as the optimal next hop for the prefix, the Forwarding Plane will have R2 as the next-hop for the destination address of aa08::4450.¶

The edge service Metadata influencing next hop selection is different from the metric (or weight) to the next hop. The metric to a next hop can impact many (sometimes, tens of thousands) routes that have the node as their next hop. while as the edge service Metadata only impact the optimal next hop selection for a subset of client routes that are identified as the edge services.¶

When the BGP custom decision [idr-custom-decision] is used, the edge service Management function would have algorithm to combine the edge service Metadata attributes with the custom decision to derive the optimal next hop for the Edge service routes.¶

Note: For a BGP UPDATE message that includes the edge service Metadata Path Attribute with the RouteFlag-I=0 and the egress router's loopback prefix as the NLRI, the Site Capacity Availability Index value is applied to all the routes associated with the Site-ID.¶

11.1. Metadata Path Attribute

IANA is requested to assign a new path attribute from the "BGP Path Attributes" registry. The symbolic name of the attribute is "Metadata", and the reference is [This Document].¶

    +=======+======================================+=================+
    | Value |             Description              |    Reference    |
    +=======+======================================+=================+
    |  TBD1 |      Metadata Path Attribute         | [this document] |
    +-------+--------------------------------------+-----------------+

11.2. Metadata Path Attribute Sub-Types

IANA is requested to create a new sub-registry under the Metadata Path Attribute registry as follows:¶

Name:

Sub-TLVs under the "Metadata Path Attribute"¶

Registration Procedure:

Expert Review [RFC8126].¶

Detailed Expert Review procedure will be added per RFC8126.¶

Reference:

[this document]¶

+========+==========================+=================+
|Sub-Type|   Description            | Reference       |
+========+==========================+=================+
|      0 | reserved                 | [this document] |
+--------+--------------------------+-----------------+
|      1 | Site Preference Index    | [this document] |
+--------+--------------------------+-----------------+
|      2 | Site Availability Index  | [this document] |
+--------+--------------------------+-----------------+
|      3 | Service Delay Predication| [this document] |
+--------+--------------------------+-----------------+
|      4 | Raw Load Measurement     | [this document] |
+--------+--------------------------+-----------------+
|  5-254 | unassigned               | [this document] |
+--------+--------------------------+-----------------+
|    255 | reserved                 | [this document] |
+--------+--------------------------+-----------------+

13.1. Normative References

[RFC2119]

Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/info/rfc2119>.

[RFC4271]

Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A Border Gateway Protocol 4 (BGP-4)", RFC 4271, DOI 10.17487/RFC4271, January 2006, <https://www.rfc-editor.org/info/rfc4271>.

[RFC4360]

Sangli, S., Tappan, D., and Y. Rekhter, "BGP Extended Communities Attribute", RFC 4360, DOI 10.17487/RFC4360, February 2006, <https://www.rfc-editor.org/info/rfc4360>.

[RFC4760]

Bates, T., Chandra, R., Katz, D., and Y. Rekhter, "Multiprotocol Extensions for BGP-4", RFC 4760, DOI 10.17487/RFC4760, January 2007, <https://www.rfc-editor.org/info/rfc4760>.

[RFC4786]

Abley, J. and K. Lindqvist, "Operation of Anycast Services", BCP 126, RFC 4786, DOI 10.17487/RFC4786, December 2006, <https://www.rfc-editor.org/info/rfc4786>.

[RFC5291]

Chen, E. and Y. Rekhter, "Outbound Route Filtering Capability for BGP-4", RFC 5291, DOI 10.17487/RFC5291, August 2008, <https://www.rfc-editor.org/info/rfc5291>.

[RFC7606]

Chen, E., Ed., Scudder, J., Ed., Mohapatra, P., and K. Patel, "Revised Error Handling for BGP UPDATE Messages", RFC 7606, DOI 10.17487/RFC7606, August 2015, <https://www.rfc-editor.org/info/rfc7606>.

[RFC8126]

Cotton, M., Leiba, B., and T. Narten, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 8126, DOI 10.17487/RFC8126, June 2017, <https://www.rfc-editor.org/info/rfc8126>.

[RFC8277]

Rosen, E., "Using BGP to Bind MPLS Labels to Address Prefixes", RFC 8277, DOI 10.17487/RFC8277, October 2017, <https://www.rfc-editor.org/info/rfc8277>.

[RFC9012]

Patel, K., Van de Velde, G., Sangli, S., and J. Scudder, "The BGP Tunnel Encapsulation Attribute", RFC 9012, DOI 10.17487/RFC9012, April 2021, <https://www.rfc-editor.org/info/rfc9012>.

13.2. Informative References

[ATTRIBUTE-ESCAPE]

J. Haas, "BGP Attribute Escape", July 2023, <https://datatracker.ietf.org/doc/draft-haas-idr-bgp-attribute-escape/>.

[IANA-BGP-PARAMS]

IANA, "BGP Path Attributes", BGP Path Attributes https://www.iana.org/assignments/bgp-parameters/.

[IDR-CUSTOM-DECISION]

A. Retana, R. White, "BGP Custom Decision Process", August 2017, <https://datatracker.ietf.org/doc/draft-ietf-idr-custom-decision/>.

[RFC2042]

Manning, B., "Registering New BGP Attribute Types", RFC 2042, DOI 10.17487/RFC2042, January 1997, <https://www.rfc-editor.org/info/rfc2042>.

[RFC4684]

Marques, P., Bonica, R., Fang, L., Martini, L., Raszuk, R., Patel, K., and J. Guichard, "Constrained Route Distribution for Border Gateway Protocol/MultiProtocol Label Switching (BGP/MPLS) Internet Protocol (IP) Virtual Private Networks (VPNs)", RFC 4684, DOI 10.17487/RFC4684, November 2006, <https://www.rfc-editor.org/info/rfc4684>.

[RFC8174]

Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/info/rfc8174>.

[RFC8799]

Carpenter, B. and B. Liu, "Limited Domains and Internet Protocols", RFC 8799, DOI 10.17487/RFC8799, July 2020, <https://www.rfc-editor.org/info/rfc8799>.

[TS.23.501-3GPP]

3rd Generation Partnership Project (3GPP), "System Architecture for 5G System; Stage 2, 3GPP TS 23.501 v2.0.1", December 2017.