Congestion Signaling (CSIG)

4.1.1. Compact Format

CSIG-tag compact format is as shown, with 2B allocated for the CSIG Tag Protocol ID (TPID) and 2B allocated for the data fields.¶

   0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |             TPID              |  T  |R|    S    |      LM     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   |0-15|  TPID  : IEEE allocated Tag Protocol ID for 4 Byte CSIG tag
   |16-18| T     : Signal Type (0:min(ABW), 1: min(ABW/C), 2:max(PD))
   |19|    R     : Reserved
   |20-24| S     : Signal Value: Bucketed (32 configurable buckets)
   |25-31| LM    : Locator Metadata of bottleneck device / port

4.1.2. Expanded Format

CSIG-tag expanded format is as shown, with 2B allocated for the Tag Protocol ID (TPID) and 6B allocated for the data fields¶

   0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |             TPID              |               LM              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |   T   |                  S                    |       R       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   |0-15|  TPID : IEEE allocated Tag Protocol ID for 8 Byte CSIG tag
   |16-31| LM   : Locator Metadata of bottleneck device / port
   |0-3|   T    : Signal Type (0:min(ABW), 1: min(ABW/C), 2:max(PD))
   |4-23|  S    : Signal Value: Uniformly quantized
   |24-31| R    : Reserved for future use

4.1.3. CSIG-tag Data fields Description

This section describes the format and usage of data fields within the CSIG-tag¶

4.2.1. Reflection in TCP

Reflection in TCP is typically achieved via TCP options. CSIG Reflection can be implemented via a new TCP Option, identified by a unique Kind.¶

   0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |     Kind     |    Length     |       CSIG data fields         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Kind              : Unique codepoint to recognize TCP CSIG option
   Length            : Length in bytes of the CSIG data fields
                       carried in the options payload
   CSIG data fields  : Values reflected from receiver to sender

4.2.2. Reflection in non-TCP Transports

Several transports such as QUIC [RFC9000] and PonyExpress [PONYEXPRESS] are built atop UDP. Reflection in UDP can be achieved by including CSIG data fields in the UDP payload from receiver to sender. For unidirectional UDP traffic, an out-of-band reverse connection from the receiver to the sender may be necessary for CSIG reflection.¶

As an example, PonyExpress [PONYEXPRESS] is a custom transport implemented within a userspace host networking stack. It supports a flexible L4 wire protocol that periodically changes as new features are added (Sec 3.1 in Snap). CSIG reflection can be implemented as additional bytes within this wire format.¶

                   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6
                   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                   |    Flags    | CSIG data fields|
                   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

For simplicity and to avoid the need for negotiation, the CSIG reflection header can be carried on all packets independent of whether CSIG is enabled on them. The Valid bit in the Flags field can be set to 1 for packets that carry valid data fields in the reflection header. In certain deployments, negotiation is unavoidable for a variety of reasons. Section 6.3.3 provides details regarding options for negotiation.¶

4.3.1. Forward Path

The sender end-host first constructs a CSIG-tagged packet for a flow of interest and sends out the packet with the tag data fields initialized. The transport determines these initial values for the packet, including Signal Type to request and default values for the other data fields. Each transit device performs a compare-and-replace on the CSIG-tag to optionally update the Signal Value and Locator Metadata fields on the tag. As the packet traverses through the network, the CSIG-tag data fields accumulate the desired aggregation of the requested signal.¶

4.3.2. Reverse Path

When the CSIG-tagged packet reaches the receiver end-host, the data fields in the CSIG tag are extracted and delivered to the transport layer at the receiver. The transport stores the data fields of the packet to be reflected, or a summary of these fields across packets. It reflects these data fields in the layer-4 CSIG reflection header on packets traversing the reverse path from receiver to sender. The CSIG reflection header is unmodified as the packet travels from receiver to sender. The sender extracts the CSIG data fields from the CSIG reflection header of the incoming packet, and hands it to the transport layer for use in applications at the sender. As a result, the sender transport learns the desired signal for a flow within approximately one round-trip time.¶

4.3.3. Multiple signals

The transport layer has a significant role to play in making CSIG usable. Although the CSIG data fields are carried on packets, the measurements are ultimately relevant at the flow / connection level for specific paths. If the sender transport desires to obtain multiple signals for the same flow, it MAY choose Signal Type on a per-packet basis (e.g., in a round robin fashion across the flow's packets), and internally keep track of all of the requested signals as part of the flow's state variables. This approach allows the sender transport to use all supported CSIG signals for use cases such as congestion control, load balancing and multipathing.¶

4.4.1. Sender host

The sender host is responsible for¶

(i) Constructing CSIG-tagged packets for flows of interest and initializing the CSIG-tag data fields on each packet as specified by the transport, and¶

(ii) Parsing the CSIG reflection header received in incoming packets and extracting CSIG data fields for use in the sender transport / applications.¶

Only the sender is allowed to insert CSIG-tags into packets.¶

4.4.2. Transit device

Transit devices are responsible for¶

(i) Computing and tracking Congestion signals such as ABW and ABW/C of each port and hop delay per packet¶

(ii) Parsing the CSIG-tag based on the TPID code point on incoming packets to identify the Signal type being requested, and¶

(iii) Performing compare-and-replace on the Signal value and locator metadata fields on the CSIG-tag based on the aggregation corresponding to the requested signal type (min / max)¶

Transit devices MUST NOT add CSIG tags to incoming packets that are not already CSIG-tagged. Transit devices MAY delete the CSIG tag before forwarding the packet. This functionality can be exercised when downstream devices are not CSIG-capable. Further discussion on this topic is in Section 6 on Incremental Deployment of CSIG.¶

4.4.3. Receiver host

The receiver host is responsible for¶

(i) Extracting the CSIG-tag on incoming packets and exposing the data fields to the transport layer and/or receiver-driven applications¶

(ii) Inserting and populating the CSIG Reflection header at the transport layer for packets traversing the reverse path to the sender.¶

4.4.4. Host roles for bidirectional flows

Note that for bi-directional flows, the Sender and Receiver are specific to each direction within the flow. For a bi-directional flow between hosts A and B,¶

(i) A plays the Sender host role and B plays the Receiver host role for data packets traveling from A to B, and similarly¶

(ii) B plays the Sender host role and A plays the Receiver host role for data packets traveling from B to A.¶

In this scenario, packets traversing from A to B contain both a CSIG- tag that captures the congestion signals on the forward A-->B path, and a CSIG reflection header that captures the CSIG data fields of the reverse B-->A path. Equivalently, packets traversing from B to A contain both a CSIG-tag that captures the congestion signals on the forward B-->A path, and a CSIG reflection header that captures the CSIG data fields of the reverse A-->B path¶

5.1.1. ABW Computation

ABW can be computed using one of many algorithm variants, each having implications on HW or SW implementation complexity, timescales of computation and accuracy of the signal. In its rudimentary form, the raw ABW for a given egress port p over a time interval delta_t can be computed as follows:¶

// delta_txbit is the number of bits that exited on the wire
utilization_bps[p] = (delta_txbit[p]) / delta_t;
// capacity_bps[p] captures the link speed of port p
abw_bps[p] = capacity_bps[p] - utilization_bps[p];

Implementation of these computations relies on at least one of the following capabilities in the devices:¶

• Timer-based computations: Most networking ASICs maintain hardware counters that track the number of bits that exit on each egress port. To compute available bandwidth, a periodic-timer thread in SW or HW triggers the computation and update of available bandwidth every delta_t time interval , where delta_t is a configurable parameter.¶

• Per-packet computations: In this alternative, available bandwidth is computed and updated on every packet that is processed via the egress pipeline, typically in HW e.g., via Exponential Weighted Moving Average (EWMA) estimation where the weights are configurable. delta_t is not an explicit parameter in this approach, and is implicitly determined by EWMA weights.¶

Variants such as Discounted Rate Estimator (DRE) [CONGA] use a combination of per-packet updates and timer-based approaches.¶

5.2.1. ABW/C Computation

ABW/C can be computed from ABW as follows:¶

// Represents fraction of available bandwidth on port p
// relative to the port's capacity.
abwc_frac[p] = abw_bps[p] / capacity_bps[p];

Algorithms for ABW computation described in Section 5.1.1 also apply to ABW/C computation, except that the resulting value is normalized by the port capacity. Quantization / bucketing is performed after normalization.¶

5.2.2. min(ABW) vs min(ABW/C) bottlenecks

On paths with heterogeneous link speeds, min(ABW) and min(ABW/C) bottlenecks are not necessarily the same ports. Figure 2 shows an example where these two bottlenecks are different. Each type of bottleneck has its own value, as demonstrated in Section 8.¶

5.3.1. Algorithm Requirements

To support min(ABW) or min(ABW/C) in CSIG, the device SHOULD support raw ABW computation with a configurable delta_t, and MAY support additional algorithms such as EWMA or DRE. This requirement enables the consistent interpretation of timescale over which available bandwidth is computed. This consistent interpretation allows end-hosts to tune their control decisions based on this timescale e.g., in relation to the flow's RTT.¶

5.3.2. Timescale and Accuracy Requirements

CSIG does not set strict requirements on the delta_t values to be supported by the implementation, except that it SHOULD be configurable to cover the range of RTTs in the network e.g., {10us, 100us, 1ms, 10ms, 100ms, 1s etc.}. Although one would expect all devices on a packet path to compute ABW at similar timescales to provide a consistent path-wide view, CSIG does NOT set strict requirements on the consistency of delta_t parameters chosen across the devices of a packet path. Choices of signal accuracy and timescales are a function of the use case and are not enforced by CSIG. End hosts MAY use EWMA across packets of a flow to calculate ABW or ABW/C over a longer timescale when CSIG on each packet carries ABW or ABW/C over shorter timescales. This technique is useful when flows traversing a given egress port span a wide range of RTTs while ABW computation over the egress port is fixed to a chosen timescale at each transit device.¶

5.3.3. Bucketing / Quantization Requirements

The computed ABW or ABW/C values MUST be compressed to fit in the available Signal value bits on the CSIG-tag. The device MUST support 32 fully configurable ABW buckets and ABW/C buckets for compact CSIG, and configurable quanta for uniform quantization in expanded CSIG. All devices along the packet path MUST be configured with the same buckets / quanta per signal type in order to correctly compute min(ABW) or min(ABW/C) along the path. Appendix A provides examples of these configurations.¶

Each transit device performs a compare-and-replace, i.e., updates the signal value on the CSIG tag if the incoming ABW or ABW/C signal value on the packet is higher than the device's locally computed ABW or ABW/C value for the packet's egress port, post bucketization / quantization. E.g.,¶

// Update the signal value on packet if current hop is the bottleneck
pkt->csig_tag->abw = min(pkt->csig_tag->abw, egr_port->abw)

5.3.4. QoS requirements

min(ABW) and min(ABW/C) are unambiguous signals with low implementation complexity on network devices. For simplicity, these definitions intentionally do NOT distinguish across QoS classes that may share the egress port. Available bandwidth per QoS class on an egress port is complex to define and meaningfully interpret since it depends on the scheduling policy (Strict Priority / WRR / Deficit WRR), buffer carving configuration and other policies (e.g., AQM) associated with QoS. Section 8 describes the applications of min(ABW) and min(ABW/C) as defined. We leave QoS-based variations of these signals and their potential use cases as future work.¶

5.4.1. Per-hop Delay Computation

Unlike ABW and ABW/C which are per-port signals, PD is a per-packet signal. It consists of PHY, MAC and switch pipeline delay experienced by the packet. Pipeline delay is the most relevant component as it captures congestion related queueing delay. Device implementations MAY track ingress and egress timestamps explicitly for each packet and perform a diff in the final stages of the pipeline. Precise definitions of these stages depend on the architecture of the device. For example, some devices could leverage existing timestamping support from tail timestamping capabilities for this purpose.¶

5.4.2. Requirements

// Update the signal value on packet if current hop is the bottleneck
pkt->csig_tag->pd = max(pkt->csig_tag->pd, device->pkt->pd)

5.5.1. Requirements

A single deployment MAY choose a subset of the attributes in Section 5.5.2 and/or newly defined attributes beyond those listed in Section 5.5.2 to include in LM. However, the total size of the individual attributes MUST be within 7 bits for Compact CSIG and within 16 bits for Expanded CSIG.¶

CSIG does not set strict requirements on the LM internal format i.e., how the individual attributes are organized among the available LM bits. However, this LM internal format MUST be consistent across devices in the deployment domain so that the end hosts can consistently interpret these bits. The LM internal format MAY be specific to each signal type.¶

Devices SHOULD support configuring per-port values for LM to be written on the CSIG-tag. Devices MAY provide more granular configurability of LM based on Signal type as well. CSIG packets egressing on a given port that have their Signal Value updated by the device MUST be updated with the LM corresponding to the port and Signal Type.¶

5.5.2. Attributes

Attributes can be designed to capture the level of resolution desired by use cases for pinpointing the bottleneck. Attributes may be encoded to fit within the limited number of LM bits available in CSIG.¶

We separate the list of attributes into compact attributes and expanded attributes. Compact attributes are motivated by the limited number of LM bits available in Compact CSIG, and therefore capture only the essential information about the bottleneck that is necessary for the use cases i.e., to inform control decisions or telemetry. Expanded attributes provide higher resolution information about the bottleneck, and can aid in directly pinpointing bottleneck devices or ports. Expanded attributes typically require more bits and are hence more suited for Expanded CSIG.¶

Examples of attributes are listed below.¶

6.2.1. Discard

Devices in this category are not capable of recognizing or parsing CSIG tagged packets. If such packets are received, they will simply be dropped.¶

6.2.2. Pass-through

Devices in this category are able to recognize and parse CSIG tagged packets, and transparently forward the packet with the tag intact or with the tag stripped to neighboring devices (in the case of transit devices) or to the end host transport layer (in the case of end hosts). However, they do not support updating the CSIG data fields on the tag.¶

Some devices that do not natively support CSIG may be configured to support pass-through mode for CSIG if they support VLAN tags with configurable TPIDs. This is discussed in more detail in Section 6.4.¶

6.2.3. Complete

Devices in this category support the complete CSIG protocol, including recognition, parsing, forwarding, tag-stripping, signal computation, and signal updates on the tag. However, only a subset of signal types may be supported.¶

6.3.1. Requirements for interoperability

Forwarding: The fundamental requirement is that no CSIG-tagged packet should be dropped in the network due to a lack of CSIG support on a device. This requirement means packets with CSIG-tags MUST never reach devices in the Discard category, or MUST have their CSIG-tag stripped before reaching such devices.¶

Negotiation: End hosts / flows SHOULD ensure that the path (including end hosts and transit devices) is CSIG-capable before enabling CSIG- tagging on packets. Devices in the Discard category should not require any changes in order to achieve negotiation. This requirement is to ensure correctness of data fields in end-to-end CSIG operation, and to interoperate with legacy devices or software stacks.¶

6.3.2. Forwarding

To achieve forwarding interoperability requirements for CSIG, CSIG stripping may be exercised as shown below¶

• When a neighboring device connected to a given egress port is a Discard device and cannot parse CSIG packets, this egress port MUST be configured to strip the tag on outgoing packets to ensure that the packet does not get dropped downstream.¶

• When a device supports Pass-through only or does not support the requested signal type on a CSIG packet, egress ports on this device MAY be configured to strip the tag on outgoing packets to ensure that CSIG does not carry inaccurate information. In some use cases where it is acceptable for CSIG to miss capturing signals on certain hops, pass-through devices MAY transparently forward the packet with the CSIG tag intact.¶

• At the boundary of a CSIG domain, device ports that are connected to devices outside of the CSIG domain MUST strip the tag to ensure that packets exiting the domain do not contain CSIG-tags. Only egress ports connected to devices within the CSIG domain SHOULD retain CSIG-tags on outgoing packets.¶

CSIG packets and non-CSIG packets can be used together in a brownfield setting. This requirement means that end hosts MUST be capable of transmitting and receiving both CSIG packets and non-CSIG packets, including for the same flow. A packet marked with CSIG-tag at the sender host may arrive at the receiver host without the tag. In addition, Compact CSIG and Expanded CSIG packets may be used together on the same network.¶

6.3.3. Negotiation

Support for sending and receiving CSIG-tagged packets may require software and/or hardware changes on transit devices and end hosts. In many deployments, particularly those requiring hardware upgrades to support CSIG (such as Switch or NIC support), version stragglers continue to exist for long time horizons for a variety of reasons, and interoperability with such stragglers is a critical requirement. Without negotiation for CSIG capability, devices that are not CSIG- compliant may drop CSIG packets and thus blackhole traffic. Negotiating for CSIG-capability of a path is critical to ensure that CSIG protocol operates safely end-to-end in a brownfield deployment.¶

A path is considered CSIG-capable if end-hosts have at least Pass-through CSIG support and transit devices have Complete CSIG support (native or software-assisted). Before sending CSIG-tagged packets on a network flow, end-hosts must negotiate for path CSIG-capability. We discuss one approach to negotiation for path CSIG-capability, which involves two parts: negotiation for transit device support and negotiation for end host support.¶

8.1.1. Using maximum per-hop delay in E2E CC

E2E RTT measurements used in Swift include the queueing delays on all hops along the flows' path, including the forward and reverse paths. A consequence of using a lumped delay signal is that a flow reduces its sending rate in response to delays that it may not be able to directly control. Furthermore, in deployments where there can be multiple congested links along the path of a flow, it is desirable to modulate the sending rate of a flow in response to just the maximum of the per-hop delays, max(PD), along a flows' path. Replacing the end-to-end measured delay with bottleneck delay into Swift's equation yields the following:¶

// Reduce the congestion window when bottleneck hop delay
// exceeds a chosen target hop delay
if (max(PD) > target_delay) then
  md = beta * (max(PD) - target_delay) / max(PD)
  cwnd = (1 - md) *cwnd

Poseidon [POSEIDON] is a CC proposed in literature that exemplifies the use of maximum per-hop delay in reducing its congestion window. By incorporating bottleneck information in congestion control response, POSEIDON flows achieve higher flow throughputs in presence of reverse path congestion, and congestion across multiple network hops. Algorithm 1 in [POSEIDON] details the use of maximum per-hop delay in both the increase and the decrease of the congestion window.¶

8.1.2. Using maximum link utilization in E2E CC

E2E CC uses heuristics to determine by how much to increase the congestion window, e.g., in the case of Swift, when the measured round-trip time is lower than the target delay, Swift increments the congestion window by one per round-trip time. BBR [BBR] increases the rate as a function of the flow's measured delivery rate.¶

The problem with these heuristics is that they don't get the rate or window adjustments just right and either under or overshoot. Undershooting the rate would mean that transfers take longer to complete even when the bottleneck link has a low utilization, while overshooting can cause an unnecessary increase in queueing delay and packet losses.¶

In the following example, we integrate the maximum utilization signal into Swift's congestion window update equation to ramp up adaptively faster when the bottleneck link has low utilization. The congestion window evolution is represented below:¶

// Increase congestion window in proportion
// to the utilization headroom
if (rtt < target_rtt) then
  fcwnd <-- fcwnd + additive_increment
            + kLambda . fcwnd . (1 - max(U/C))

As an example, the fixed additive increase in Swift of rate <-- rate + Additive Increment, means that it takes 200 RTTs to take 80 Gbps of bandwidth with an Additive Increment of 400 Mbps. The fast ramp-up with CSIG using the bottleneck link utilization takes <10 RTTs to safely ramp to 80 Gbps.¶

8.1.3. Using minimum available bandwidth in E2E CC

E2E CC uses heuristics to determine the initial transfer rate for newly established connections. Starting too slowly would cause the transfer to take longer than necessary while wasting available bandwidth, whereas starting too quickly would cause queue delays and packet drops. The same dilemma exists for transfers that are starting on a connection that has been idle for multiple round-trip times.¶

In networks where we know ahead of time that the degree of multiplexing is low i.e., just a handful of flows co-existing on the link at any point in time, transfers complete quickly when they "jump-start" to use up all of the bottleneck bandwidth. This is especially helpful when transports employ robust loss recovery mechanisms such that even if the queue overflows, any lost packets can be quickly recovered.¶

As an example, on an empty network of 200Gbps, a single transfer can use up the entire 200Gbps in the second RTT, after the CSIG feedback in the first RTT indicates the availability of 200Gbps at the bottleneck link.¶

CSIG's min(ABW) bottleneck bandwidth allows transfers to start safely at line-rate.¶

8.2.1. Load Balancing and Multipathing

Datacenter topologies employ a diverse set of paths between any source-destination pairs. Transports employ techniques such as Protective Load Balancing [PLB] and Multipathing [RFC8684] to spread traffic across the multitude of paths. Load balancing and multipathing in transports use a combination of end-to-end signals and heuristics to select which paths to use and how much traffic to channel in each of the paths.¶

Using CSIG signals from bottleneck links along the diverse set of paths, load balancing and multipathing schemes can select high quality paths with lower congestion, and spread traffic across them in a congestion-aware manner.¶

Locator metadata can also be used to distinguish between incast congestion and core network congestion, which can then be used to adjust load balancing / multipathing actions. For instance, the stage of the bottleneck and link orientation attributes are enough to determine whether the last hop is the bottleneck or not. When the last hop is the bottleneck, flow-level load balancing / multipathing actions may not be effective and may, in fact, worsen incasts. Such cases may require application-level load balancing or job scheduling techniques to distribute traffic. However, when congestion is instead known to be in the core network, flow-level load balancing / multipathing actions can route around congested areas and improve performance.¶

8.2.2. Traffic Engineering

Traffic Engineering carves out paths with apt bandwidth across aggregate source-destination pairs. Examples within a datacenter include Datacenter Network Interconnection Layer (DCNI) [JUPITEREVOL]. CSIG can be used to provide fine-grained path level information, including short timescale microburst congestion, to TE systems. By using summarized CSIG signals aggregated both spatially and temporally across flows, TE can select paths and balance traffic at the datacenter level to accommodate bursty traffic, e.g., from ML.¶