Coordinated Congestion Management

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶

This Internet-Draft will expire on 22 April 2024.¶

Copyright Notice

Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved.¶

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶

Table of Contents

1. Introduction

ML/AI has been progressing rapidly over the last decade. ML/AI model compute, which is measured in FLOPs, are constantly increasing. It is imperative to employ distributed parallel training to train such large models in AI cluster.¶

The communication in AI cluster is bandwidth sensitive. Analyzing data parallelism and model parallelism which are the 2 acceleration methods in AI training, it shows an on-off type of burst traffic pattern with huge traffic amount in each iteration.¶

Therefore, it is important that AI fabric should provide high effective bandwidth, so to shorten communication time and improve computation efficiency. Effective bandwidth indicates fully utilization of link bandwidth to achieve high throughput. Congestion management is the key technology, including congestion control mechanisms and load balancing mechanisms.¶

This document discusses the uncoordinated mechanisms in current congestion management. That leads to throughput issues which are particularly harmful in AI fabric. A scheme for coordinating different congestion management mechanisms is provided in this document, which can be effectively and widely deployed in AI fabric.¶

2. Terminology

• ML: Machine Learning¶

• AI: Artificial Intelligence¶

• FLOPs: Floating-Point Operations¶

• ECN: Explicit Congestion Notification¶

• AR: Adaptive Routing¶

• DCQCN: Data center QCN [DCQCN]¶

• CNP: Congestion Notification Packet¶

• PLB: Protective Load Balancing [PLB]¶

• CC: Congestion Control¶

• ECMP: Equal-cost multi-path routing¶

• Incast congestion: the congesiton is caused by mutiple sources sending traffic to the same destination simultaneously.¶

• Incast flow: the flow causing incast congestion¶

• Incast traffic: packets in incast flows¶

• CC congestion: the congestion is casued by incast or by high-speed port sending traffic to low-speed port¶

• CC flow: the flow causing CC congestion¶

• CC traffic: packets in CC flows¶

3. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶

4. Existing congestion management

Congestion is usually caused by in-cast traffic and/or imbalanced network load. Incast traffic is the traffic from multiple source hosts, but towards to the same destination host. Commonly used solutions include congestion control algorithms that control sending rates and load balancing algorithms that adjust paths for traffic.¶

• The congestion control algorithm, such as DCQCN [DCQCN], Timely [Timely], identifies network congestion by network status, like queue length of switch port, end-to-end delay RTT, etc., then adjust the sending rate at the sender to alleviate congestion. How to quickly flatten down the rate curve to avoid packet loss and how to recover the rate for less throughput reduction are essential to congestion control mechanism in AI fabric.¶

• Adaptive routing is a way for load balancing. According to network status, network switch dynamically change traffic path of a flow in order to fully utilize network resource. Network status could be indicated by local link status, downstream link status etc. How to locate the proper new traffic path without back-and-forth path switching is critical in AI fabric, because each path switching may increase the systeme complexity, like re-ordering.¶

Currently, congestion control mechanism and adaptive routing work independently, without coordination. That results in negative impact on system performance. For example, when congestion caused by imbalanced load on network occurs on a switch, both DCQCN and adaptive routing are activated. ECN in data packets is marked causing the CNP to be sent back to sender. Thus, sender slows down the sending rate of the congested flow. Meanwhile, the switch changes the path for congested flow, traversing the new incoming packets to a light-loaded path. The result is that the congested flow is forwarded on the light-loaded path at a low rate. Then, DCQCN needs some time to recover the sending rate at the new path. It reduces effective bandwidth and seriously impact computation efficiency in AI training. Another example, if the congestion is caused by in-cast traffic, congestion control should be enough. Additional adaptive routing adjustments not only fail to mitigate congestion, but may also introduce more out-of-order issue.¶

It is shown that current congestion management cannot efficiently handle congestion issue in AI fabric. Uncoordinated behaviors reduce effective network bandwidth which is essential for AI workload.¶

5. Design principle of coordinated congestion management

Coordinated congestion management is designed to coordinate congestion control and adaptive routing. Design principle is shown as below.¶

• Avoid unnecessary sending rate reduction
  AI fabric is bandwidth sensitive. High throughput is extremely important. Multipath is needed to make full use of network bandwidth. Slowing down the sending rate while there are still available paths for traffic will be a waste of network resource, thereby increasing communication time in AI cluster and reducing AI training performance.¶

• Fully use multipath while reducing invalid path switching
  While searching for light-loaded paths for load balancing, new paths should be located quickly and accurately. The new path should not be restricted to local paths but extends the search to available paths upstream. Invalid path switching should be avoided. Invalid path switching includes switching in-cast traffic as no matter how to switch the traffic path, it will final get congested on the last hop.¶

• Reuse current CC algorithm and AR algorithm
  There are already a variety of CC algorithm and AR algorithms. Those can still be used in the congestion management coordination scheme. The scheme enables CC and AR be triggered coordinately, adjusting sending rate or switching path depending on different reasons of congestion.¶

• Applicable to various topologies
  Most AI fabrics use CLOS or FATTREE topologies, but there are also new studies considering the use of direct topologies, such as torus, dragonfly, dragonfly+. Some of existing solutions for CC and AR coordination, e.g PLB [PLB], relies on ECMP which can only be used in topologies with equal cost paths like CLOS. For those topologies without equal cost paths, like dragonfly+, such solutions do not work. The coordination scheme should be applicable to different topologies.¶

6. Coordinated congestion management scheme

The key to the coordinated congestion management is to identify CC traffic and non-CC traffic, thereby they are treated differently in network when congestion occurs. CC flow recognized by network is notified to the source host and the subsequent packets of the CC flow are tagged by the source host. This indicates the network switch to perform CC mechanism on the flow instead of AR. For non-CC traffic, the network switch first performs AR. Only when AR mechansim cannot find light-loaded path for switching, the traffic turns to be CC traffic and CC will be run to alleviate congestion.¶

Coordinated congestion management requires interaction between network switches and source hosts, and adds a new tag to data packets for the coordination. The following sections explain the detail of the scheme.¶

7. An example of end-to-end procedure

Network topology is shown in Figure 1. This is a 4 layer fattree topology. There are n computing racks and m switching racks. Computing racks have source hosts, layer 1 switches and layer 2 switches. Swithcing racks contain layer 3 and layer 4 switches.¶

      Switching Rack 1    Switching Rack m
      +---------------+   +---------------+
      |L4-1-1...L4-1-e|   |L4-m-1...L4-m-e|
      |  | \    / |   |   |  | \    / |   |
      |  |  \  /  |   |   |  |  \  /  |   |
      |  |   \/   |   |   |  |   \/   |   |
      |  |   /\   |   |...|  |   /\   |   |
      |  |  /  \  |   |   |  |  /  \  |   |
      |  | /    \ |   |   |  | /    \ |   |
      |L3-1-1...L3-1-d|   |L3-m-1...L3-m-d|
      +--+-----------\    +-/----------+--+
         |            \    /           |
         |             \  /            |
         |  ......      \/     ......  |
         |              /\             |
         |             /  \            |
         |            /    \           |
      +--+-----------/      \----------+---+
      |L2-1-1...L1-1-c|    |L2-n-1...L2-n-c|
      |  | \    / |   |    |  | \    / |   |
      |  |  \  /  |   |    |  |  \  /  |   |
      |  |   \/   |   |    |  |   \/   |   |
      |  |   /\   |   |... |  |   /\   |   |
      |  |  /  \  |   |    |  |  /  \  |   |
      |  | /    \ |   |    |  | /    \ |   |
      |L1-1-1...L1-1-b|    |L1-n-1...L1-n-b|
      |  +        +   |    |  +        +   |
      | H-1-1... H-1-a|    | H-n-1... H-n-a|
      +---------------+    +---------------+
      Computing Rack 1     Computing Rack n

• Host H-1-1 in computing rack 1sends out a data packet P1 belonging to flow F1 to H-n-1 in computing rack n. The value of CC indicator in the packet tag is not set indicating this packet is in a non-incast flow. The AR indicator in the packet tag does not point to any available AR point.¶

• P1 arrives at switch L1-1-1 in computing rack 1. L1-1-1 has multiple light-loaded paths for AR. Path from L1-1-1 to L2-1-1 is selected for P1. AR indicator in P1 tag is updated to L1-1-1.¶

• P1 arrives at switch L2-1-1. L2-1-1 also has multiple light-loaded paths for AR. Path from L2-1-1 to L3-1-1 is selected for P1. AR indicator in P1 tag is updated to L2-1-1.¶

• P1 arrives at switch L3-1-1. L3-1-1 only has one light-loaded paths. The only path from L3-1-1 to L4-1-1 is selected for P1. AR indicator in P1 tag keeps to be L2-1-1.¶

• P1 arrives at switch L4-1-1. L4-1-1 is congested and no local path available for performing AR. By reading AR indicator in P1, L4-1-1 sends an type 3 notification to L2-1.¶

• After receiving AR notification, L2-1-1 switches path from L2-1-1->L3-1-1 to L2-1-1->L3-m-1 for the new incoming packets of flow F1.¶

• After a while, L1-n-1 is congested due to incast. The flow F1 is identified as incast flow. L1-n-1 sends type 1 notification to H-1-1.¶

• By receiving the type 1notification, H-1-1 sets CC indicator of the subsequent packets of F1 indicating the packets are in a incast flow. Thus those packets will not be performed AR. Sending rate of F1 will also be reduced according to congestion control algorithm.¶

8. Security Considerations

TBD.¶

9. IANA Considerations

TBD.¶

Authors' Addresses