| Internet-Draft | ISP Dual Queue Networking Deployment Rec | October 2024 |
| Livingood | Expires 17 April 2025 | [Page] |
The IETF's Transport Area Working Group (TSVWG) has finalized experimental RFCs for Low Latency, Low Loss, Scalable Throughput (L4S) and new Non-Queue-Building (NQB) per hop behavior. These documents describe a new architecture and protocol for deploying low latency networking. Since deployment decisions are left to implementers, this document explores the potential implications of those decisions and makes recommendations that can help drive adoption and acceptance of L4S and NQB.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 17 April 2025.¶
Copyright (c) 2024 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document.¶
The IETF's Transport Area Working Group (TSVWG) has finalized experimental RFCs for Low Latency, Low Loss, Scalable Throughput (L4S) and Non-Queue-Building (NQB) per hop behavior [RFC9330] [RFC9331] [RFC9332] [I-D.ietf-tsvwg-l4sops] [I-D.ietf-tsvwg-nqb] [I-D.ietf-tsvwg-dscp-considerations]. These documents do a good job of describing a new architecture and protocol for deploying low latency networking. But as is normal for many such standards, especially those in experimental status, certain deployment decisions are ultimately left to implementers.¶
This document explores the potential implications of key deployment decisions and makes recommendations for those decisions that may help drive adoption. In particular, there are best practices based on prior experience as a network operator that should be considered and there are network neutrality types of considerations as well. These technologies are benign on their own, but the way they are operationally implemented can determine whether they are ultimately perceived positively and adopted by the broader Internet ecosystem. That is a key issue for low latency networking, because the more applications developers and edge platforms that adopt new packet marking for low latency traffic, then the greater the value to end users, so ensuring it is received well is key to driving strong initial adoption.¶
It is worth stating though that these decisions are not embedded in or inherent to L4S and NQB per se, but are decisions that can change depending upon differing technical, regulatory, business or other requirements. Even two network operators with the same type of access technology and in the same market area may choose to implement in different ways. Nevertheless, this document suggests that certain specific deployment decisions can help maximize the value of low latency networking to both users and network operators.¶
The IETF documents on L4S and NQB also made clear that nearly all modern application types - from video conferencing to web browsing - can benefit from low latency networking. In addition, the design of the protocols also make clear that applications developers are best positioned to understand the needs of their applications and to, by extension, express any such low latency needs via appropriate L4S or NQB packet marking. Furthermore, unlike with bandwidth priority on a highly/fully utilized link, low latency networking can better balance the needs of different types of best effort flows (with some caveats - see Section 3).¶
For additional background on latency and why latency matters so much to the Internet, please read [BITAG].¶
In the course of working to improve the responsiveness of network protocols, the IETF concluded with their L4S and NQB work that there were fundamentally two types of Internet traffic and that these two major traffic types could benefit from having separate network processing queues in order to improve the way the Internet works for all applications, and especially for interactive applications.¶
One of the two major traffic types is Queue Building (QB) - things like file downloads and backups that are designed utilize as much network capacity as possible but with which users are usually not interacting with in real-time. The other was Non-Queue-Building (NQB) - such as DNS lookups, voice interaction with artificial intelligence (AI) assistants, video conferencing, gaming, and so on. NQB flows tend to be ones where the end user is sensitive to any delays.¶
Thus, the IETF created specifications for how two different network processing queues. The performance value of dual queue networking (simply "low latency networking" hereafter) has proven out in Comcast's dual queue networking field trial [Comcast]. That field trial lasted for over a year and was regularly reported on over time at several IETF TSVWG meetings [IETF-TSVWG-117] [IETF-TSVWG-118] [IETF-TSVWG-119] [IETF-TSVWG-120] and demonstrated that L4S and NQB can work deliver excellent responsiveness for a variety of applications - from video conferencing to cloud gaming, DNS and other applications. It seems likely that this new capability will enable entirely new classes of applications to become possible, driving a wave of new Internet innovation, while also improving the applications people use today.¶
| May Benefit from L4S/NQB | Classic Queue Building |
| User interacting with screen | Unattended background process |
| Video conference, audio conference, live event video stream, cloud document editing | Cloud file backup, cloud file restoration, game platform update, operating system update |
The Introduction says that unlike with bandwidth priority on a highly/fully utilized link, low latency networking can better balance the needs of different types of best effort flows. But this bears a bit of further discussion to understand more fully.¶
L4S does *not* provide low latency in the same way as previous technologies like DiffServ Quality of Service (QoS). That prior QoS approach used packet prioritization, where it was possible to assign a higher relative priority to certain application traffic, such as Voice over IP (VoIP) telephony. This approach could provide consistent and relatively low latency by assigning high priority to a partition of the capacity of a link, and then policing the rate of packets using that partition. This traditional approach to QoS is hierarchical in nature.¶
That QoS approach is to some extent predicated on an idea that network capacity is very limited and that links are often highly utilized. But in today's Internet, it is increasingly the case that there is an abundance of capacity to end users (e.g., symmetric 1 Gbps), which makes such traditional QoS approaches ineffective in delivering ever-lower latency. This new low latency networking approach is not based on hierarchical QoS prioritization. Rather, it is built upon conditional priority scheduling between its two queues that operate at best effort QoS priority.¶
Low latency traffic to is not prioritized over other (best effort priority) "classic" Internet traffic. That is the case over the ISP network and the broader internet, though it may not not necessarily be the case for a user's in-home Wi-Fi network due to the particulars of how the IEEE 802.11 wireless protocol [IEEE] functions at the current time - see [RFC8325]). In addition, some user access points may prioritize certain traffic (such as gaming) and some traffic such as NQB may use the AC_VI Wi-Fi link layer queue [I-D.ietf-tsvwg-nqb]. This best effort approach stands in contrast to prior differential quality of service (QoS) approaches or to what has been discussed for 5G network slicing [CDT-NN] [van-Schewick-1A] [van-Schewick-1B] [van-Schewick-2] [van-Schewick-3].¶
Low latency networking flows do not get access to greater throughput than "classic" flows. Thus, a user's total provisioned or permitted throughput on an ISP access network link is shared between both classic and low latency queues.¶
Ultimately, the emergence of low latency networking represents a fundamental new network capability that applications can choose to utilize as their needs dictate. It reflects a new ground truth about two fundamentally different types of application traffic and demonstrates that networks continue to evolve in exciting ways. This new network capability can be implemented in a variety of network technologies. For example in access network technologies this could be implemented in DOCSIS [LLD], 5G [Ericsson], PON [CTI], and many other types of networks. Anywhere that a network bottleneck could occur may benefit from this technology.¶
Like any network or system, a good deployment design and configuration matters and can be the difference between a well-functioning and accepted system and one that experiences problems and faces user and/or developer opposition. In the context of deploying low latency networking in an ISP network, this document describes some recommendations that should help to ensure a deployment is resilient, well-accepted, and creates the environment for generating strong network effects. Following these recommendations should help avoid barriers to adoption and help provide a strong foundation for growth.¶
The best approach is for applications to mark traffic to indicate their preference for the low latency queue, not the network making such a decision on its own. This is for several reasons:¶
Any application provider should be able to mark their traffic for the low latency queue, with no restrictions other than standards compliance or other reasonable and openly documented technical guidelines. This maintains the loose cross-layer coupling that is a key tenet of the Internet's architecture by eliminating the need for application providers and networks to coordinate and creates an environment of so-called "permissionless innovation".¶
To create the best foundation for growth, both customer-owned and ISP-administered Customer Premises Equipment (CPE) should be supported (assuming that is typical and technically feasible in a particular type of access network), when applicable. In mobile networks, where a user device connects directly to the network, this may be easier as it simply depends upon operating system software in that end user device. In fixed networks, there is usually CPE that demarcates between the ISP network and the user's home LAN. The software running on those CPE devices will also need to support low latency networking, as well as software in other ISP network devices where CPE connections are aggregated. The more CPE devices that are enabled, the greater the pool of potential users, and thus the broader adoption would be, positively driving network effects.¶
The specifications in [I-D.ietf-tsvwg-nqb] describe a concept of Traffic Protection, also known as a Queue Protection Function [I-D.briscoe-docsis-q-protection] or simply Queue Protection. The document says that Traffic Protection is optional and may not be needed in certain networks. In the case of an ISP deploying low latency networking with two queues, an ISP should consider deploying such a network function to at least detect mismarking (if not necessarily to correct mismarking). This may be implemented, for example, in end user CPE, last mile network equipment, and/or elsewhere in the ISP network - or closely monitors network statistics and user feedback for any indication of widespread low latency packet mismarking by applications.¶
If possible, for operational simplicity, a network should try to maintain the use of DSCP 45 on an end-to-end basis without remarking in their interior network hops. This may not be possible in all networks, because some may already use DSCP 45 for some private internal reason. In such cases, packets must obviously be remarked to and from DSCP 45 at the customer edge (CPE) and network ingress/egress to other networks. But if DSCP 45 is not used internally, it is far simpler for network operations and troubleshooting to preserve that mark on an end to end basis.¶
As noted above with respect to prioritization of packets in the ISP network, all packets should be handled with the same best effort priority. However, in a user's home Wi-Fi (wireless) local area network (WLAN), this is more complicated, because there is not a precise mapping between IETF packet marking and IEEE 802.11 marking, explored in [RFC8325]. In short, today's 802.11 specifications enable a Wi-Fi network to have multiple queues, using different "User Priority" and "Access Category" values. At the current time, these queues are AC_BK (Background), AC_BE (Best Effort), AC_VI (Video), and AC_VO (Voice).¶
As explored in [I-D.ietf-tsvwg-nqb], packets in the low latency queue may be marked for the best effort (AC_BE) or video (AC_VI) wireless queue. For additional context, please refer to Section 8.1 of [I-D.ietf-tsvwg-nqb]. In some situations, such as a user-owned wireless access point or CPE, it may not be possible for the user to select which wireless queue is used. In cases where the CPE is ISP-administered, selecting a specific wireless queue may be possible - though it is not yet clear what the best practice may be for this selection until ISPs and application developers have more experience with low latency networking.¶
Since the IETF's NQB specification is only recently completed, many applications that have been using other DSCP marks for their latency-sensitive flows have not yet shifted to adopt DSCP-45. One example is the Microsoft Xbox platform [Microsoft], which is using DSCP-46. So in the relatively short-term, ISPs may find it beneficial to their customers to use a more permissive upstream NQB admission policy, allowing DSCP-40, 45, 46, and 56 admission into the low latency queue. It may take a year or more after the NQB DSCP assignment is made by IANA for developers to shift to DSCP-45, given other items in their development backlog and their software release schedule.¶
In some cases, dual queue networking and associated packet marking is supported up to the ISP's demarcation point - such as in a cable modem. It is recommended that packet markings should pass from such a demarcation point to any attached customer-administered CPE, such as a router or wireless access point. That enables a customer-administered router to implement dual queue networking, rather that it only being possible with ISP-administered CPE.¶
Thanks to Bob Briscoe, Mat Ford, Vidhi Goel, Eliot Lear, Sebastian Moeller, Sebnem Ozer, Jim Rampley, Dan Rice, Greg Skinner, Greg White, and Yiannis Yiakoumis for their review and feedback on this document.¶
RFC Editor: Please remove this section before publication.¶
This memo includes no requests to or actions for IANA.¶
The key security consideration pertains to Queue Protection. As the current time, it is recommended that implementers utilize Queue Protection, to ensure that any traffic that is incorrectly marked for low latency can be detected and remarked for the classic queue. The necessity of Queue Protection remains something of a debate, with some firmly believing it is necessary but others believing that it is not needed. The latter view is that application developers have a natural incentive to correctly mark their traffic, because to do otherwise would worsen the quality of experience (QoE) for their users. In that line of thinking, if a developer mismarks, they and/or their users will notice and they will fix that error. However, it is also conceivable that malicious software could be operating on a user's device or home network and that malicious software could try to send some much traffic to the low latency queue that the queue or both queues become unusable. This is quite similar to other "traditional" denial of servce (DoS) attacks, so it does not necessarily seems unique to low latency networking. But due to the possibility of this occuring, and low latency networking being such a new approach, it seems prudent to implement Queue Protection.¶
Network Neutrality (a.k.a. Net Neutrality) can mean a variety of things within a country, as well as between different countries, based on different opinions, market structures, business practices, laws, and regulations. Generally speaking, In the context of the United States' market, it has come to mean that Internet Service Providers (ISPs) should not block, throttle, or deprioritize lawful application traffic, and should not engage in paid prioritization, among other things. Net Neutrality concerns can sometimes affect the deployment of new technologies by ISPs, so they should carefully consider regulatory issues when making deployment decisions.¶
As it is envisioned in the design of the IETF's new low latency networking protocols, the addition of a low latency packet processing queue at a network link is merely a second packet queue and does not mean that this queue is hierarchically prioritized or that it has more capacity. As a result, low latency networking appears to pose no new Net Neutrality issues. However, it is important for ISPs to keep these risks in mind as they make deployment design decisions.¶
One key aspect of low latencty networking is that it operates, from the perspective of an ISP's deployment, is application-agnostic. The ISP creates a second network queue on key network links, but does not decide on their own what applications can use this queue. Rather, they add the queue and packet flows are sent to that queue based on packet marking set by application developers. This approach is far superior to older approaches, which caused significant Net Neutrality risks [Lotus], that used middleboxes to attempt to infer applications based on observing packet flows on ISP network links.¶
RFC Editor: Please remove this section before publication.¶
v00: First draft¶
v01: Incorporate comments from 1st version after IETF-115¶
v02: Incorporate feedback from the TSVWG mailing list¶
v03: Final feedback from TSVWG and prep for sending to ISE¶
v04: Refresh expiration before major revision¶
v05: Changes from Greg Skinner and Eliot Lear¶
v06: More changes from Eliot Lear¶
RFC Editor: Please remove this section before publication.¶
- Open issues are being tracked in a GitHub repository for this document at https://github.com/jlivingood/IETF-L4S-Deployment/issues¶