| Internet-Draft | sceb | November 2020 |
| Morton, et al. | Expires 6 May 2021 | [Page] |
This memo reclassifies ECT(1) to be an early notification of congestion on ECT(0) marked packets, which can be used by AQM algorithms and transports as an earlier signal of congestion than CE. It is a simple, transparent, and backward compatible upgrade to existing IETF-approved AQMs, RFC3168, and nearly all congestion control algorithms.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 6 May 2021.¶
Copyright (c) 2020 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119] and [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
Traditional TCP congestion control exhibits a "sawtooth" pattern which, in the most favourable cases, oscillates around the optimum operating point of maximum throughput and minimum delay, which exists at the point where the congestion window equals path BDP. The term "sawtooth" brings to mind the straight-edged graphs of TCP Reno, but the equally common TCP CUBIC is essentially similar in character, as are other AIMD-derived algorithms.¶
A number of proposals have sought to improve this, but introduce various other tradoffs in return. TCP Vegas is consistently outcompeted by standard TCPs, DCTCP proved to be too aggressive for deployment in the public Internet, and while BBR appears to have avoided both of these problems, its complexity makes it difficult to implement correctly. Each of these proposals is characterised by primarily changing only the endpoints, not the network nodes on the path between them; though DCTCP is intended for use with a specific style of AQM, it can work with standard AQMs as long as there is no competing non-DCTCP traffic.¶
Some other proposals have attempted to convey information about the network path explicitly, by having network nodes inject data about link capacity and/or utilisation into passing traffic. These proposals have generally been unsuccessful due to the complex slow-path processing required in network nodes, and are not widely deployed. The only successful proposal of this type is Explicit Congestion Notification [RFC3168] which allows an AQM to signal congestion by marking packets with (essentially) a one-bit signal in preference to dropping them.¶
ECN defines a two-bit field supporting four codepoints, of which three are in active use and the fourth is a semantic duplicate. It was explicitly suggested during ECN's development that new meaning could be given to this spare codepoint, including as a lesser indication of congestion in [RFC3168] (section 20.2). With an alternative use of this codepoint having fallen out of favour, the time is right to revisit this suggestion and propose a workable method of applying it.¶
In so doing, care must be taken that backwards compatibility is maintained with existing traffic, endpoints and network nodes that are known or suspected to have been deployed. Keeping the changes to on-wire protocols minimal, and the complexity of implementation low, are also highly desirable.¶
This memo reclassifies ECT(1) to be an early notification of congestion on ECT(0) marked packets, which can be used by AQM algorithms and transports as an earlier signal of congestion than CE ("Congestion Experienced").¶
This memo also briefly discusses how transports should respond to ECT(1) marked packets. Detailed specifications of this behaviour are left to transport-specific memos.¶
[RFC3168] defines the lower two bits of the (former) TOS byte in the IPv4/6 header as the ECN field. This may take four values: Not-ECT, ECT(0), ECT(1) or CE.¶
| Binary | Keyword | References |
|---|---|---|
| 00 | Not-ECT (Not ECN-Capable Transport) | [RFC3168] |
| 01 | ECT(1) (ECN-Capable Transport(1)) | [RFC3168] |
| 10 | ECT(0) (ECN-Capable Transport(0)) | [RFC3168] |
| 11 | CE (Congestion Experienced) | [RFC3168] |
Research has shown that the ECT(1) codepoint goes essentially unused, with the "Nonce Sum" extension to ECN having not been implemented in practice and thus subsequently obsoleted by [RFC8311] (section 3). Additionally, known [RFC3168] compliant senders do not emit ECT(1), and compliant middleboxes do not alter the field to ECT(1), while compliant receivers all interpret ECT(1) identically to ECT(0). These are useful properties which represent an opportunity for improvement.¶
Experience gained with 7 years of [RFC8290] deployment in the field suggests that it remains difficult to maintain the desired 100% link utilisation, whilst simultaneously strictly minimising induced delay due to excess queue depth - irrespective of whether ECN is in use. This leads to a reluctance amongst hardware vendors to implement the most effective AQM schemes because their headline benchmarks are throughput-based.¶
The underlying cause is the very sharp "multiplicative decrease" reaction required of transport protocols to congestion signalling (whether that be packet loss or CE marks), which tends to leave the congestion window significantly smaller than the ideal BDP when triggered at only slightly above the ideal value. The availability of this sharp response is required to assure network stability (AIMD principle), but there is presently no standardised and backwards-compatible means of providing a less drastic signal.¶
As consensus has arisen that some form of ECN signaling should be an earlier signal than drop, this memo changes the meaning of ECT(1) to SCE, meaning "Some Congestion Experienced". Since there is no longer ambiguity between two ECT codepoints, ECT(0) is referred to as ECT. The ECN-field codepoint table then becomes:¶
| Binary | Keyword | References |
|---|---|---|
| 00 | Not-ECT (Not ECN-Capable Transport) | [RFC3168] |
| 01 | SCE (Some Congestion Experienced) | [This draft] |
| 10 | ECT (ECN-Capable Transport) | [RFC3168] |
| 11 | CE (Congestion Experienced) | [RFC3168] |
This permits middleboxes implementing AQM to signal incipient congestion, below the threshold required to justify setting CE, by converting some proportion of ECT codepoints to SCE ("SCE marking"). Existing [RFC3168] compliant receivers MUST transparently ignore this new signal with respect to congestion control, and both existing and SCE-aware middleboxes SHOULD convert SCE to CE in the same circumstances as for ECT, thus ensuring backwards compatibility with [RFC3168] ECN endpoints.¶
The permitted ECN codepoint transitions by middleboxes are:¶
| From | To |
|---|---|
| Not-ECT | Not-ECT |
| ECT | ECT or SCE or CE |
| SCE | SCE or CE |
| CE | CE |
Note that dropping a packet is an allowed action for any ECN codepoint. While that is the only way of indicating congestion with Not-ECT, it may also be used to both indicate and reduce congestion in any state.¶
To re-state the allowed transitions another way: for ECN-aware flows, the ECN marking of an individual packet MAY be increased by a middlebox to signal congestion, but MUST NOT be decreased, and packets SHALL NOT be altered to appear to be ECN-aware if they were not originally, nor vice versa. Note however that SCE is numerically less than ECT, but semantically greater, and the latter definition applies for this rule.¶
Receivers and transport protocols conforming to this specification SHALL continue to apply the [RFC3168] interpretation of the CE codepoint, that is, to signal the sender to back off send rate to the same extent as if a packet loss were detected. This maintains compatibility with existing middleboxes, senders and receivers.¶
New SCE-aware receivers and transport protocols SHOULD interpret the SCE codepoint as an indication of mild congestion, and respond accordingly by applying send rates intermediate between those resulting from a continuous sequence of ECT codepoints, and those resulting from a CE codepoint. The ratio of ECT and SCE codepoints received indicates the relative severity of such congestion, with a higher proportion of SCE codepoints indicating more congestion.¶
The intent of SCE marking is a "cruise control" signal which permits middleboxes to request relatively small reductions in send rate, or merely a slowing of send rate growth. Accordingly, SCE marks SHOULD progressively trigger exit from exponential slow-start growth, then reduction to Reno-linear growth (for congestion control algorithms which support higher growth rates in congestion-avoidance phase), then a halt to send rate growth, then a gradual reduction of send rate. For immediate large reductions of send rate, the CE mark MUST retain its original Multiplicative Decrease power as per [RFC8511], and compliant AQMs SHOULD retain the ability to employ it where appropriate.¶
Details of how to implement SCE awareness at the transport layer are left to additional Internet Drafts. To ensure RTT-fair convergence with single-queue SCE AQMs, transports SHOULD stabilise at lower SCE-mark ratios for higher BDPs, and MAY reduce their response to CE marks IFF they are responding to SCE signals received at around the same time (eg. within 1-2 RTTs) in the same flow.¶
To maximise the benefit of SCE, middleboxes SHOULD begin to produce SCE marks at lower congestion levels than they begin to produce CE marks. This will usually ensure that SCE-aware flows avoid receiving CE marks. When a single-queue AQM is upgraded to SCE awareness, this will tend to cause SCE flows to give way to non-SCE flows; to avoid this behaviour, single-queue AQMs MAY be left as [RFC3168] compliant without SCE support.¶
For the avoidance of doubt, a decision to mark CE or to drop a packet always takes precedence over SCE marking.¶
The SCE design sees ECN as a "network feature". The risks with ECN signaling (Section 5.1), the need to handle unresponsive flows (Section 5.2), the utility of fairness (Section 5.3), and the availability of only one ECN codepoint all influenced the SCE signaling design. This section discusses these related concerns, along with what is needed from middleboxes to address them, and how that ultimately led to the selection of ECT(1) as an additional signal of lesser congestion (Section 5.4).¶
The safety and effectiveness of ECN signaling depends upon the unaltered transmission of the ECN bits, both for the indication of ECN support, and for ECN signaling. Unlike a drop, which is reliably and irrevocably signaled, ECN signals may be erased or manipulated. Specifically, any of the following results in the lack of a congestion response, which is likely to lead to the near starvation of competing flows:¶
Although the lack of a congestion response is similar to when transports do not respond appropriately to drop, the difference is that with ECN, the behavior can be brought about in the network, without changes to the endpoint. This may happen by accident, for example due to a broken network configuration or endpoint implementation, or on purpose, e.g. using a simple firewall rule.¶
Unresponsive flow mitigation, discussed in the next section, deals with flows that are not responding to congestion signals, including for the reasons listed above.¶
A single unresponsive flow has the potential to nearly starve all other competing flows in a congested bottleneck, resulting in unacceptable network delays and collapses in throughput. The need to handle unresponsive flows is corroborated in [RFC7567] (section 4), stating:¶
"Research, engineering, and measurement efforts are needed regarding the design of mechanisms to deal with flows that are unresponsive to congestion notification or are responsive, but are more aggressive than present TCP."¶
The source language from [RFC2309] (section 5) is more direct:¶
"It is urgent to begin or continue research, engineering, and measurement efforts contributing to the design of mechanisms to deal with flows that are unresponsive to congestion notification or are responsive but more aggressive than TCP."¶
The [COBALT] AQM algorithm is one example of how unresponsive flows can be dealt with, using the [BLUE] algorithm to detect overload and trigger drops.¶
Regardless of how it's done exactly, unresponsive flow mitigation is most effectively implemented with some level of flow awareness, so that drops may be directed to the offending flow/s. Once flow awareness is available, fairness steering becomes possible, discussed further in the following section.¶
In order for SCE flows to compete fairly with non-SCE flows, at least one of the following is required: some form of fairness steering, or some way of separating SCE and non-SCE flows. Following is a non-exhaustive list of options:¶
When available, fairness is viewed as an advantage, in that it:¶
The abundance of new and proposed congestion controls is making their fair competition across bandwidths, RTTs and network conditions more difficult if not impossible to ensure in the endpoint alone [CC-REVOLUTION] [CC-COMPAT]. Congestion control implementations may dominate one another under different conditions, e.g. [BBR-CUBIC], while the widespread deployment of potentially beneficial congestion controls that seek to minimize delay is discouraged by the fact that they are often out-competed in bottlenecks by standard TCP. Fairness in the network both improves these conditions and assists transports responding to SCE.¶
With only a single ECN codepoint remaining, options are limited for how to signal congestion with high fidelity. Meanwhile, the recent rise in ECN signaling makes backwards compatibility with [RFC3168] a practical requirement.¶
Fortunately, the same network technologies that mitigate the well recognized risks listed in Section 5 above, also make the use of ECT(1) as defined by SCE possible, without a separate traffic identifier. Where those technologies cannot be deployed, Diffserv may be used to identify SCE traffic (see Section 6), a purpose for which it was expressly designed. Where that is impossible, SCE allows a graceful fallback to [RFC3168] ECN. SCE's usage of ECT(1) provides a safe and solid foundation on which future innovations in the network can improve the availability and performance of high-fidelity congestion signaling.¶
SCE is not dependent on Diffserv [RFC2474] for its signaling, but makes use of it in the following ways:¶
All SCE DSCPs indicate SCE support in the originating endpoint. This MAY assist SCE marking middleboxes in their operation, but MUST NOT be depended upon for effective congestion control. See Section 7.3 for an example of such a usage.¶
SCE middleboxes MUST retain any SCE DSCPs that arrive on incoming packets, and MUST NOT set them on packets that do not already have them.¶
The SCE DSCPs MAY be set on TCP ACK and control packets which have the Not-ECT codepoint set in the ECN field, IFF the TCP connection as a whole is SCE capable (or in the process of being negotiated as such). This allows all packets relating to that connection to be treated equally by middleboxes which distinguish them. Should ECN negotiation fail, the DSCP should be changed to some non-SCE value for subsequent traffic on that connection.¶
The SCE-CAPABLE DSCP indicates SCE support, with standard, best-effort service implied. This is the appropriate service for capacity-seeking traffic, for which latency is a secondary consideration.¶
The SCE-LOWDELAY DSCP is used to both indicate SCE support and request low-delay service. This MAY be used by AQMs to select a low delay queue with tighter marking parameters that reduce delay, at the possible expense of throughput.¶
The SCE-LOWCOST DSCP is used to both indicate SCE support and request altruistic low-cost service. This MAY be used by AQMs to deprioritise this traffic in favour of low-delay and best-effort traffic, similar to the LE PHB [RFC8622].¶
Prior to approval for public experiment, the SCE DSCPs are defined in the experimental pool xxxx11, and the following rules MUST be observed to contain SCE traffic within the experimental network:¶
The following values are proposed for guidance only. Because they are in the experimental pool, they may be changed to suit the environment:¶
| Name | Value (Binary) | Value (Decimal) |
|---|---|---|
| SCE-CAPABLE | 000111 | 7 |
| SCE-LOWDELAY | 001011 | 11 |
| SCE-LOWCOST | 000011 | 3 |
In the event that SCE is approved for public experiment, the DSCPs will be allocated in an appropriate standards action pool, using a value that is intended to be treated as best-effort traffic by existing deployed devices.¶
One of the SCE DSCPs SHOULD be set by sending endpoints on all SCE capable traffic. However, they neither need to be checked by middleboxes that do not require them before marking SCE, nor by receiving endpoints before returning SCE feedback. That way, they can serve as hints for middleboxes, but the SCE signaling mechanism is not dependent on end-to-end DSCP traversal.¶
Unless and until a public experiment is approved, the guidance in Section 6.2 MUST be followed.¶
A simple and natural way to implement SCE in a Codel-type AQM is to mark all ECT packets as SCE if they are over half the Codel target sojourn time, and not marked CE by Codel itself. This threshold function does not necessarily produce the best performance, but is very easy to implement and provides useful information to SCE-aware flows, often sufficient to avoid receiving CE marks whilst still efficiently using available capacity.¶
For a more sophisticated approach avoiding even small-scale oscillation, a stochastic ramp function may be implemented with 100% marking at the Codel target, falling to 0% marking at or above zero sojourn time. The lower point of the ramp should be chosen so that SCE is not accidentally signalled due to CPU scheduling latencies or serialisation delays of single packets. Absent rigorous analysis of these factors, setting the lower limit at half the Codel target should be safe in many cases.¶
The default configuration of Codel is 100ms interval, 5ms target. A typical ramp function for these parameters might cease marking below 2.5ms sojourn time, increase marking probability linearly to 100% at 5ms, and mark at 100% for sojourn times above 5ms (in which CE marking is also possible).¶
In single-queue AQMs, the above strategy will result in SCE flows yielding to pressure from non-SCE flows, since CE marks do not occur until SCE marking has reached 100%. A balance between smooth SCE behaviour and fairness versus non-SCE traffic can be found by having the marking ramp cross the Codel target at some lower SCE marking rate, perhaps even 0%. A two-part ramp, reaching 1/sqrt(X) at the Codel target (for some chosen X, a cwnd at which the crossover between smoothness and fairness occurs) and ramping up more steeply thereafter, has been implemented successfully for experimentation.¶
The CNQ algorithm [I-D.morton-tsvwg-cheap-nasty-queueing] offers a relatively simple way to limit this yielding behaviour and ensure that, even in competition with non-SCE flows, SCE flows maintain a reasonable minimum throughput capability. This may be sufficient to avoid the need for the two-part ramp described above.¶
Flow-isolating AQMs, including especially CNQ and DRR++ based algorithms, should avoid signalling SCE to flows classified as "sparse", in order to encourage the fastest possible convergence to the fair share.¶
There are several reasonable methods of producing SCE signals in a RED-type AQM.¶
The simplest would be a threshold function, giving a hard boundary in queue depth between 0% and 100% SCE marking. This could be a sensible option for limited hardware implementations. The threshold should be set below the point at which a growing queue might trigger CE marking or packet drops.¶
Another option would be to implement a second marking probability function, occupying a queue-depth space just below that occupied by the main marking probability function. This should be arranged so that high marking rates (ideally 100%) are achieved at or before the point at which CE marking or packet drops begin.¶
For PIE specifically, a second marking probability function could be added with the same parameters as the main marking probability function, except for a lower QDELAY_REF value. This would result in the SCE marking probability remaining strictly higher than the CE marking probability for ECT flows.¶
In high-capacity or resource constrained SCE marking middleboxes, DSCP may be used to select one of two queues, in lieu of implementing fairness steering. Packets marked with an SCE DSCP are placed in an SCE queue, where an AQM instance may mark congestion with either SCE or CE. Packets not marked with an SCE DSCP are placed in a second [RFC3168] queue, whose AQM instance may only mark congestion with CE. For approximate flow fairness, the queues may be scheduled in proportion to the number of flows they contain.¶
Note that as long as the SCE DSCP remains intact from the sending endpoint to the marking queue, the SCE queue may be used. If it has been erased or altered to a non-SCE DSCP, the packet will be placed in the [RFC3168] queue, and may still benefit from standard ECN.¶
If this middlebox is to be used in public environments, some form of unresponsive flow mitigation is warranted to ensure that flows haven't indicated their support for either SCE or [RFC3168] ECN incorrectly. If flows do not respond to the signals they advertise support for, they will dominate competing traffic in the same queue.¶
The proposed mechanism for TCP to feed back SCE signals to the sender is outlined in [I-D.grimes-tcpm-tcpsce]. Use is made of the redundant NS bit in the TCP header, which was formerly associated with ECT(1) in the Nonce Sum specification.¶
The recommended response to each single segment marked with SCE is to reduce cwnd by an amortised 1/sqrt(cwnd) segments. Other responses, such as the 1/cwnd from DCTCP, are also acceptable but may perform less well.¶
SCE explicitly retains [RFC8511] compliant Multiplicative Decrease responses to CE marks, and conventional Multiplicative Decrease responses to packet loss. SCE senders' behaviour is thus naturally compliant with existing specifications when running over existing networks.¶
Existing endpoints, supporting Not-ECT or [RFC3168] compliant congestion control, are required to treat SCE marks (that is, ECT(1)) as identical to ECT(0), and will thus transparently ignore SCE marks. This is allowed for in SCE's design, and allows SCE middleboxes to be deployed into a heterogeneous network.¶
Hence the incremental deployability of SCE endpoints and middleboxes is good.¶
L4S [I-D.ietf-tsvwg-l4s-arch] also claims the ECT(1) codepoint, with significantly different semantic meaning than SCE, so a discussion around the potential for L4S and SCE compatibility is warranted. In the L4S system, ECT(1) is used to identify L4S flows, to distinguish them from [RFC3168] flows - necessary since in L4S, the semantic meaning of CE marks is also changed.¶
Since L4S connections are explicitly negotiated through support of AccECN, and AccECN doesn't support SCE, there is no ambiguity regarding the mode of the connection as far as endpoints are concerned.¶
SCE middleboxes will treat L4S flows in the same way as [RFC3168] does. However, because SCE middleboxes are likely to upgrade ECT(1) marked packets to CE at a higher threshold than L4S middleboxes would, L4S flows will outcompete non-L4S flows in a single SCE-aware queue. This is the same known safety concern with L4S deployment in regards to existing [RFC3168] queues, resulting from the redefinition of CE in L4S. Fairness steering in SCE middleboxes could mitigate this.¶
L4S middleboxes may interpret ECT packets which have received SCE markings at some other SCE-aware middlebox as though they were L4S traffic. This may result in a higher CE marking rate and/or different queuing behaviour. It may also result in the reordering of packets for both SCE and non-SCE aware flows through L4S middleboxes, as packets marked ECT(1) will on average traverse the bottleneck with lower delay than packets not marked ECT(1). Although this could be mitigated by [I-D.ietf-tcpm-rack], it may lead to reduced throughput and head-of-line blocking for flows that traverse both SCE and L4S bottlenecks.¶
There are at least two secondary concerns brought about by the L4S use of ECT(1) as a traffic identifier:¶
Lastly, an ambiguous definition of ECT(1) complicates network debugging with packet captures, since it would be unclear whether a packet was marked ECT(1) due to congestion at an SCE bottleneck, or because it is an L4S flow. Although examination of other packets in the flow could reduce this ambiguity, the necessity of observing flow state is generally discouraged for debugging purposes.¶
Thus far, the working group is operating under the assumption that coexistence of SCE and L4S is not an option.¶
The SCE proposal is a work in progress, with ongoing or planned work in at least the following areas:¶
There are no IANA considerations.¶
An adversary could inappropriately set SCE marks at middleboxes he controls to slow down SCE-aware flows, eventually reaching a minimum congestion window. However, the same threat already exists with respect to inappropriately setting CE marks on normal ECN flows, and this would have a greater impact per mark. Therefore no new threat is exposed by SCE in practice.¶
An adversary could also simply ignore SCE marks at the receiver, or ignore SCE information fed back from the receiver to the sender, in an attempt to gain some advantage in throughput. Again, the same could be said about ignoring CE marks, so no truly new threat is exposed. Additionally, correctly implemented SCE detection may actually improve long-term goodput compared to ignoring SCE.¶
An adversary could erase congestion information by converting SCE marks to ECT or Not-ECT codepoints, thus hiding it from the receiver. This has equivalent effects to ignoring SCE signals at the receiver. An identical threat already exists for erasing congestion information from CE marked packets, and may be mitigated by AQMs switching to dropping packets from flows observed to be non-responsive to CE.¶
An adversary could drop SCE-marked packets, believing them to be bogons (see also L4S Compatibility, above). Endpoints should be able to recover from this through retransmission and a reduction of cwnd. However, it is possible for this to lead to a significant denial of service. A workaround is to disable ECN for connections over the affected path.¶
Thanks to Dave Taht for his contributions to the SCE effort, and his work on writing the original draft-morton-taht-sce-00 that was submitted for IETF/104 on which this draft is based.¶
Many thanks to John Gilmore, the members of the ecn-sane project and the cake@lists.bufferbloat.net mailing list, and the former IETF AQM working group.¶