| TOC |
|
This document describes the congestion management system of Comcast Cable, a large cable broadband Internet Service Provider (ISP) in the U.S. Comcast completed deployment of this congestion management system on December 31, 2008.
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as “work in progress.”
This Internet-Draft will expire on March 5, 2011.
Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
1.
Introduction
2.
Applicability to Other Types of Networks
3.
Key Terminology
4.
Historical Overview
5.
Summary
6.
Relationship Between Managing Congestion and Adding Capacity
7.
Implementation and Configuration
7.1.
Thresholds For Determining When a CMTS Port Is in a Near Congestion State
7.2.
Thresholds For Determining When a User Is in an Extended High Consumption State and for Release from that Classification
7.3.
Effect of BE Quality of Service on Users' Broadband Experience
7.4.
Equipment/Software Used and Location
8.
Conclusion
9.
Exceptional Network Utilization Considerations
10.
Limitations of This Congestion Management System
11.
Low Extra Delay Background Transport and Other Possibilities
12.
Security Considerations
13.
IANA Considerations
14.
Acknowledgements
15.
Change Log
16.
Open Issues
17.
Notes for RFC Editor
18.
Informative References
§
Authors' Addresses
| TOC |
Comcast Cable is a large broadband Internet Service Provider (ISP), based in the U.S., serving the majority of its customers via cable modem technology. During the late part of 2008, and completing on December 31, 2008, Comcast deployed a new congestion management system across its entire network. This new system was developed in response to dissatisfaction in the Internet community as well as complaints to the U.S. Federal Communications Commission (FCC) regarding Comcast's old system, which targeted specific peer-to-peer (P2P) applications. This new congestion management system is protocol-agnostic, meaning that it does not examine or impact specific user applications or network protocols, which is perceived as a more fair system for managing network resources at limited times when congestion may occur.
It is important for readers to note that congestion can occur in any IP network, and, when it does, packets can be delayed or dropped. As Bob Briscoe has pointed out on an IETF mailing list, some amount of packet loss can be normal and/or tolerable, noting "But a single TCP flow with a round trip time (RTT) of 80ms can attain 50 Mbps with a loss fraction of 0.0013% (1 in ~74,000 packets) so there's no need to try to achieve loss figures much lower than this. And indeed, if flows aren't bottlenecked elsewhere, TCP will drive the system until it gets such loss levels. If, instead, a customer is downloading five separate 10 Mbps TCP flows still with an 80ms RTT, TCP will drive losses up to 1 in ~3,000, or 0.03%, and any lower loss rates won't be able to improve performance." As a result, applications and protocols have been designed to deal with the reality that congestion can occur in any IP network, the mechanics of which we explain in detail later in this document.
The purpose of this document is to describe how this example of a large scale congestion management system functions. This is partially in response to questions from other ISPs as well as solution developers, who are interested in learning from and/or deploying similar systems in other networks. In addition, it is hoped that such a document may help inform new work in the IETF, in the hope that better systems and protocols may be possible in the future. Lastly, the authors wish to transparently and openly document this system, so that there could be no doubt about how the system functioned.
| TOC |
Several document reviewers and other IETF participants have pointed out that, though we refer to functional elements that are specific to a DOCSIS-based network implementation, this type of congestion management system could be generally applied to nearly any type of network. Thus, it is important for readers to take note of this and take into consideration that this sort of protocol-agnostic congestion management system could certainly fit in a wide variety of network types and implementations.
| TOC |
This section defines the key terms used in this document. Some terms below refer to elements of the Comcast network. As a result, it may be helpful to refer to Figure 1 when reviewing some of these terms.
| TOC |
A device located at the customer premise used to access the Comcast High Speed Internet (HSI) network. In some cases, the cable modem is owned by the customer, and in other cases it is owned by the cable operator. This device has an interface (i.e., someplace to plug in a cable) for connecting the coaxial cable provided by the cable company to the modem, as well as one or more interfaces for connecting the modem to a customer's PC or home gateway device (e.g., home gateway, router, firewall, access point, etc.). In some cases, the cable modem function, i.e., the ability to access the Internet, is integrated into a home gateway device or Embedded Multimedia Terminal Adapter (eMTA). Once connected, the cable modem links the customer to the HSI network and ultimately the broader Internet.
| TOC |
A piece of hardware located in a cable operator's local network (generally in a "headend", Section 3.10 (Headend)) that acts as the gateway to the Internet for cable modems in a particular geographic area. A simple way to think of the CMTS is as a router with interfaces on one side leading to the Internet and interfaces on the other connecting to Optical Nodes and then customers, in a so-called "last mile" network.
| TOC |
Also referred to simply as a "port". A port is a physical interface on a device used to connect cables in order to connect with other devices for transferring information/data. An example of a physical port is a CMTS port. A CMTS has both upstream and downstream network interfaces to serve the local access network, which are referred to as upstream or downstream ports. A port generally serves a neighborhood of hundreds of homes. Over time, CMTS ports tend to serve fewer and fewer homes, as the network is segmented for capacity growth purposes. Prior to DOCSIS version 3, a single CMTS physical port was used for either transmitting or receiving data downstream or upstream to a given neighborhood. With DOCSIS version 3, and the channel bonding feature, multiple CMTS physical ports can be combined to create a virtual port. A CMTS is also briefly defined in Section 2.6 of [RFC3083] (Woundy, R., “Baseline Privacy Interface Management Information Base for DOCSIS Compliant Cable Modems and Cable Modem Termination Systems,” March 2001.).
| TOC |
A technique for combining multiple downstream and/or upstream channels to increase customers' download and/or upload speeds, respectively. Multiple channels from the Hybrid Fiber Coax (HFC, Section 3.11 (Hybrid Fiber Coax (HFC))) network can be bonded into a single virtual port (called a bonded group), which acts as a large single channel or port to provide increased speeds for customers. Channel bonding is a feature of Data Over Cable Service Interface Specification (DOCSIS) version 3, as described in the [CableLabs DOCSIS 3.0 MAC and Upper Layer Protocols Interface Specification] (CableLabs, “Data-Over-Cable Service Interface Specifications - DOCSIS 3.0 - MAC and Upper Layer Protocols Interface Specification,” October 2009.).
| TOC |
A type of cable used by a cable operator to connect customer premise equipment (CPE) -- such as TVs, cable modems (including eMTAs), and Set Top Boxes -- to the HFC network. This cable may be used both within the home as well as in segments of the last mile network running to a home or customer premise location. There are many grades of coaxial cable that are used for different purposes. Different types of coaxial cable are used for different purposes on the network.
| TOC |
A service/product offered by Comcast for delivering Internet service over a broadband connection.
| TOC |
Any device that resides at the customer's residence, connected to the Comcast network, whether controlled by Comcast or not.
| TOC |
A reference standard developed by CableLabs that specifies how components on cable networks need to be built to enable HSI service over an HFC network, as noted in the [CableLabs DOCSIS 3.0 Cable Modem to Customer Premise Equipment Interface Specification] (CableLabs, “Data-Over-Cable Service Interface Specifications - DOCSIS 3.0 - Cable Modem to Customer Premise Equipment Interface Specification,” March 2008.), [CableLabs DOCSIS 3.0 Physical Layer Specification] (CableLabs, “Data-Over-Cable Service Interface Specifications - DOCSIS 3.0 - Physical Layer Specification,” January 2009.), [CableLabs DOCSIS 3.0 MAC and Upper Layer Protocols Interface Specification] (CableLabs, “Data-Over-Cable Service Interface Specifications - DOCSIS 3.0 - MAC and Upper Layer Protocols Interface Specification,” October 2009.), [CableLabs DOCSIS 3.0 Security Specification] (CableLabs, “Data-Over-Cable Service Interface Specifications - DOCSIS 3.0 - Security Specification,” March 2008.), and [CableLabs DOCSIS 3.0 Operations Support System Interface Specification] (CableLabs, “Data-Over-Cable Service Interface Specifications - DOCSIS 3.0 - Operations Support System Interface Specification,” October 2009.). These standards define the specifications for the cable modem and the CMTS such that any DOCSIS certified cable modem will work on any DOCSIS-certified CMTS, independent of the selected vendor. The interoperability of cable modems and CMTSs allows customers to purchase a DOCSIS-certified modem from a retail outlet and use it on their cable-networked home. All DOCSIS-related standards are available to the public at the CableLabs website, at http://www.cablelabs.com.
| TOC |
Description of the direction in which a signal travels, in this case from the network to a user. Downstream traffic occurs when users are downloading something from the Internet, such as watching a web-based video, reading web pages, or downloading software updates.
| TOC |
A cable facility responsible for receiving TV signals for distribution over the HFC network to the end customers. This facility typically also houses one or more CMTSs. This is sometimes also called a "hub".
| TOC |
A network architecture used primarily by cable companies, comprised of fiber optic and coaxial cables that currently deliver Voice, Video, and Internet services to customers, as defined in Section 1.2 of the [CableLabs DOCSIS 3.0 MAC and Upper Layer Protocols Interface Specification] (CableLabs, “Data-Over-Cable Service Interface Specifications - DOCSIS 3.0 - MAC and Upper Layer Protocols Interface Specification,” October 2009.).
| TOC |
Standardized technology for monitoring and/or recording subscribers' upstream and downstream Internet usage data based on their cable modem. The data is collected from the CMTS and sent to a server for further processing. Additional information is available at http://www.ipdr.org, as well as [IPDR Standard] (Cotton, S., Cockrell, B., Walls, P., and T. Givoly, “Network Data Management &nbhy; Usage (NDM-U) For IP-Based Services Service Specification &nbhy; Cable Labs DOCSIS 2.0 SAMIS,” November 2004.) and [CableLabs DOCSIS IPDR Support] (Yassini, R., “Data-Over-Cable Service Interface Specifications - DOCSIS 2.0 - Operations Support System Interface Specification,” November 2008.).
| TOC |
A component of the HFC network generally located in customers' local neighborhoods which is used to convert the optical signals sent over fiber-optic cables to electrical signals that can be sent over coaxial cable to customers' cable modems, or vice versa. A fiber optic cable connects the Optical Node, through distribution hubs, to the CMTS and coaxial cable connects the Optical Node to customers' cable modems.
| TOC |
The peak speed associated with a tier of service purchased by a customer. For example, a customer with a 105 Mbps downstream and 10 Mbps upstream speed tier would be said to be provisioned with 105 Mbps of downstream bandwidth and 10 Mbps of upstream bandwidth. This is often referred to as 105/10 service in industry parlance.
The Provisioned Bandwidth is the speed that a customer's modem is configured (and the network is engineered) to deliver on a regular basis (which is not the same as a "Committed Information Rate" or a guaranteed rate). Internet speeds are generally a best effort service that are dependent on a number of variables, many of which are outside the control of an Internet Service Provide (ISP). In general, speeds do not typically exceed a customer's provisioned speed. Comcast, however, invented a technology called "PowerBoost" [PowerBoost Specification] (Comcast Cable Communications Management LLC, “Comcast PowerBoost Specification,” June 2010.) that, for example, enables users to experience brief boosts above their provisioned speeds while they transfer large files over the Internet, by utilizing excess capacity which may be available in the network at that time.
| TOC |
A set of techniques to manage network resources to ensure a level of performance to specific data flows, as described in [RFC1633] (Braden, B., Clark, D., and S. Shenker, “Integrated Services in the Internet Architecture: an Overview,” June 1994.) and [RFC2475] (Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., and W. Weiss, “An Architecture for Differentiated Services,” December 1998.). One method for providing QoS to a network is by differentiating the type of traffic by class or flow and assigning priorities to each type. When the network becomes congested, the data packets that are marked as having higher priority will have higher likelihood of being serviced.
| TOC |
Description of the direction in which a signal travels, in this case from the user to the network. Upstream traffic occurs when users are uploading something to the network, such as sending email, sending files to another computer, or uploading photos to a digital photo website.
| TOC |
Comcast began the engineering project to develop a new congestion management system in March 2008, the same month that Comcast hosted the 71st meeting of the IETF in Philadelphia, PA, USA. On May 28, 2008, Comcast participated in an IETF Peer-to-Peer Infrastructure Workshop [RFC5594] (Peterson, J. and A. Cooper, “Report from the IETF Workshop on Peer-to-Peer (P2P) Infrastructure, May 28, 2008,” July 2009.), hosted by the Massachusetts Institute of Technology (MIT) in Cambridge, MA, USA.
In order to participate in this workshop, interested attendees were asked to submit a paper to a technical review team, which Comcast did on May 9, 2008, in the [Comcast P2Pi Position Paper] (Livingood, J. and R. Woundy, “Comcast's IETF P2P Infrastructure Workshop Position Paper,” May 2008.). Comcast subsequently attended and participated in this valuable workshop. During the workshop, Comcast outlined the high-level design for a new congestion management system [Comcast IETF P2P Infrastructure Workshop Presentation] (Livingood, J. and R. Woundy, “Comcast's IETF P2P Infrastructure Workshop Presentation on May 28, 2008,” May 2008.) and solicited comments and other feedback from attendees and other members of the Internet community (presentations were also posted to the IETF's P2Pi mailing list). The congestion management system outlined in that May 2008 workshop was later tested in trial markets and is in essence what was then deployed by Comcast later in 2008.
Following an August 2008 FCC document [FCC Memorandum and Opinion ‑ August 2008] (Martin, K., Copps, M., Adelstein, J., Tate, D., and R. McDowell, “FCC Memorandum and Opinion Regarding Reasonable Network Management,” August 2008.) regarding how Comcast managed congestion on its High-Speed Internet ("HSI") network, Comcast disclosed to the FCC [FCC Network Management Response ‑ September 2008] (Zachem, K., “Letter to the FCC Regarding Comcast's Network Management Practices,” September 2008.) and the public additional technical details of the congestion management system that it intended to and did implement by the end of 2008 [FCC Congestion Management Deployment Completion Letter ‑ January 2009] (Zachem, K., “Letter to the FCC Advising of Successful Deployment of Comcast's New Congestion Management System,” January 2009.), including the thresholds involved in this new system. While the description of how this system is deployed in the Comcast network is necessarily specific to the various technologies and designs specific to that network, a similar system could be deployed on virtually any large scale ISP network or other IP network.
| TOC |
Comcast's HSI network has elements which are shared across many subscribers. This means that Comcast's HSI customers share upstream and downstream bandwidth with their neighbors. Although the available bandwidth is substantial, so, too, is the demand. Thus, when a relatively small number of customers in a neighborhood place disproportionate demands on network resources, this can cause congestion that degrades their neighbors' Internet experience. The goal of Comcast's new congestion management system is to enable all users of our network resources to access a "fair share" of that bandwidth, in the interest of ensuring a high-quality online experience for all of Comcast's HSI customers.
Importantly, the new approach is protocol-agnostic; that is, it does not manage congestion by focusing on the use of the specific protocols that place a disproportionate burden on network resources, or any other protocols. Rather, the new approach focuses on managing the traffic of those individuals who are using the most bandwidth at times when network congestion threatens to degrade subscribers' broadband experience and who are contributing disproportionately to such congestion at those points in time.
Specific details about these practices, including relevant threshold information, the type of equipment used, and other particulars, are discussed at some length later in this document. At the outset, however, we present a very high-level, simplified overview of how these practices work. Despite all the detail provided further below, the fundamentals of this approach can be summarized succinctly:
Comcast undertook considerable effort, over the course of many months, to formulate our plans for this congestion management approach, adjusting them, and subjecting them to real-world trials. Market trials were conducted in Chambersburg, PA; Warrenton, VA; Lake City, FL; East Orange, FL; and Colorado Springs, CO, between June and September, 2008. This enabled us to validate the utility of the general approach and collect substantial trial data to test multiple variations and alternative formulations.
| TOC |
Many people have questioned whether congestion should ever exist at all, if an ISP was adding sufficient capacity. There is certainly a relationship between capacity and congestion. But there are two types of congestion that generally present themselves in a network. The first general type of congestion is regularly occurring and is the result of gradually increasing traffic levels up to a point where typical usage peaks cause congestion on a regular basis, and the other which is unpredictable. Comcast, like many ISPs, has a set capacity management process by which capacity additions are automatically triggered based on certain usage trends, which is geared towards bringing additional capacity to the network prior to the onset of regularly occurring congestion. As such, capacity is added when needed and before it presents noticeable effects. This process is in place since capacity additions are not instantaneous and in many cases require significant physical work.
The second general type of congestion is unpredictable congestion, which can occur for a wide range of reasons. One example may be due to current events, where users may be all rushing to access specific content at the exact same time, and where the systems serving that content may not be able to keep up with demand. Another example may be due to a localized disaster, where some network paths have been destroyed or otherwise impaired, and where many users are attempting to communicate with one another at traffic levels significantly above normal.
Thus in both cases, even with continuous upgrades and constant investment in additional capacity, the fact remains that network capacity is not unlimited. A congestion management system, absent superior protocol-based solutions which do not currently exist, can therefore help manage the effects of congestion on users, improving their Internet experience.
| TOC |
It is important to note that the implementation details below and the overall design of the system are matched to traffic patterns that exist on the Internet today and which the authors believe will exist in the near future. While the authors desired to make the system highly adaptable and a good long-term network investment, significant changes in such traffic patterns may necessitate a change in the configuration of the system or, in extreme cases, a different type of system altogether.
To understand exactly how these new congestion management practices work, it is helpful to have a general understanding of how Comcast's HSI network is designed. Comcast's HSI network is what is commonly referred to as a hybrid fiber-coax network, with coaxial cable connecting each subscriber's cable modem to an Optical Node, and fiber optic cables connecting the Optical Node, through distribution hubs, to the Cable Modem Termination System (CMTS), which is also known as a "data node". The CMTSes are then connected to higher-level routers, which in turn are connected to Comcast's Internet backbone facilities. Today, Comcast has over 3,200 CMTSes deployed throughout our network, serving over 15 million HSI subscribers.
Each CMTS has multiple "ports" that handle traffic coming into and leaving the CMTS. In particular, each cable modem deployed on the Comcast HSI network is connected to the CMTS through the ports on the CMTS. These ports can be either "downstream" ports or "upstream" ports, depending on whether they send information to cable modems (downstream) or receive information from cable modems (upstream) attached to the port. (Note that the term "port" as used here generally contemplates single channels on a CMTS, but these statements will apply to virtual channels, also known as "bonded groups", in a DOCSIS 3.0 environment.) Even without channel bonding, multiple channels are usually configured to come out of each physical port. Said another way, there is generally a mapping of multiple channels to each physical port.
Currently, on average, approximately 275 cable modems share the same downstream port and about 100 cable modems share the same upstream port, however this is constantly changing (both numbers generally become smaller over time, based on current DOCSIS technology). Both types of ports can experience congestion that could degrade the broadband experience of our subscribers and, unlike with the previous congestion management practices, both upstream and downstream traffic are subject to management in this new congestion management system.
Based upon the design of the network and traffic patterns observed, the most likely place for congestion to occur is on these CMTS ports. As a result, the congestion management system measures the traffic conditions of CMTS ports, and applies any policy actions to traffic on those ports (rather than some other, more distant segment of the network).
To implement Comcast's new protocol-agnostic congestion management practices, Comcast purchased new hardware and software that was deployed near the Regional Network Routers ("RNRs") that are further upstream in Comcast's network. This new hardware consists of Internet Protocol Detail Record ("IPDR") servers, Congestion Management servers, and PacketCable Multimedia ("PCMM") servers. Further details about each of these pieces of equipment can be found below, in Section 7.4 (Equipment/Software Used and Location). It is important to note here, however, that even though the physical location of these servers is at the RNR, the servers communicate with -- and manage individually -- multiple ports on multiple CMTSes to effectuate the practices described in this document. That is to say, bandwidth usage on one CMTS port will have no effect on whether the congestion management practices described herein are applied to a subscriber on a different CMTS port.
Figure 1 provides a simplified graphical depiction of the network architecture just described:
Figure 1: Simplified Network Diagram Showing High-Level Comcast Network and Servers Relevant to Congestion Management
-------------------------
/ \
| Comcast Internet Backbone |
\ ----
+------------+ --------------------/ \
| Congestion | / \
| Management |<+++GigE++++ +---->| Internet |
| Server | + | | Backbone |
+------------+ + | \ Router /
+ Fiber \ /
+------------+ + | -----
| QoS | + |
| Server |<+++GigE++++ \/
| | + -----
+------------+ + / \
+ / \
+------------+ + | Regional |
| Statistics | +++++++>| Network |
| Collection |<+++GigE++++ | Router |
| Server | \ /
+------------+ +---Fiber------>\ /<------Fiber----+
| ----- |
\/ \/
----- -----
/ \ / \
/ Local \ / Local \
| Market | | Market |
\ Router / \ Router /
+--------->\ /<------------+ \ /
| ----- | ------
| /\ | /\
Fiber | Fiber |
| Fiber | Fiber
| | | |
\/ \/ \/ \/
/------\ /------\ /------\ /------\
| CMTS | | CMTS | | CMTS | | CMTS |
\------/ \------/ \------/ \------/
/\ /\ /\ /\
| | | |
Fiber Fiber Fiber Fiber
| | | |
\/ \/ \/ \/
+---------+ +---------+ +---------+ +---------+
| Optical | | Optical | | Optical | | Optical |
| Node | | Node | | Node | | Node |
+---------+ +---------+ +---------+ +---------+
/\ /\ /\ /\ /\ /\
|| || ||______ || _____|| ||
Coax Coax |__Coax| Coax |Coax__| Coax
|| || || || || ||
\/ \/ \/ \/ \/ \/
+=======+ +=======+ +=======+ +=======+ +=======+ +=======+
= Cable = = Cable = = Cable = = Cable = = Cable = = Cable =
= Modem = = Modem = = Modem = = Modem = = Modem = = Modem =
+=======+ +=======+ +=======+ +=======+ +=======+ +=======+
================================================================
+ Note: This diagram is a simplification of the actual network +
+ and servers, which in actuality includes significant +
+ redundancy and other details too complex to represent here. +
================================================================
| Figure 1 |
Each Comcast HSI subscriber's cable modem has a "bootfile", which is essentially a configuration file that contains certain pieces of information about the subscriber's service to ensure that the service functions properly. (Note: No personal information is included in the bootfile; it only includes information about the service that the subscriber has purchased.) For example, the bootfile contains information about the maximum speed (what we refer to in this document as the "provisioned bandwidth") that a particular modem can achieve based on the tier (personal/residential, commercial, etc.) the customer has purchased. Bootfiles are generally reset from time to time to account for changes in the network and other updates, and this is usually done through a command sent from the network and without the subscriber noticing. In preparation for the transition to this new congestion management system, Comcast sent new bootfiles to our HSI customers' cable modems that created two Quality of Service ("QoS") levels for Internet traffic going to and from the cable modem: (1) "Priority Best Effort" traffic ("PBE"); and (2) "Best Effort" traffic ("BE"). As with previous changes to cable modem bootfiles, the replacement of the old bootfile with the new bootfile requires no active participation by Comcast customers.
Thereafter, all traffic going to or coming from cable modems on the Comcast HSI network is designated as either PBE or BE. PBE is the default status for all Internet traffic coming from or going to a particular cable modem. Traffic is designated BE for a particular cable modem only when both of two conditions are met:
When, and only when, both conditions are met, a user's upstream or downstream traffic (depending on which type of port is in the Near Congestion State) is designated as BE. Then, to the extent that actual congestion occurs, any delay resulting from the congestion will affect BE traffic before it affects PBE traffic.
We now explain the foregoing in greater detail in the following sections.
| TOC |
For a CMTS port to enter the Near Congestion State, traffic flowing to or from that CMTS port must exceed a specified level (the "Port Utilization Threshold") for a specific period of time (the "Port Utilization Duration"). The Port Utilization Threshold on a CMTS port is measured as a percentage of the total aggregate upstream or downstream bandwidth for the particular port during the relevant timeframe. The Port Utilization Duration on the CMTS is measured in minutes.
Values for each of the thresholds which are used as part of this congestion management technique have been tentatively established after an extensive process of lab tests, simulations, technical trials, vendor evaluations, customer feedback, and a third-party consulting analysis. In the same way that specific anti-spam or other network management practices are adjusted to address new issues that arise, it is a near certainty that these values will change over time, as Comcast gathers more data and performs additional analysis resulting from wide-scale use of the new technique. Moreover, as with any large network or software system, software bugs and/or unexpected errors may arise, requiring software patches or other corrective actions. As always, Comcast's decisions on these matters are driven by the marketplace imperative that we deliver the best possible experience to our HSI subscribers.
Given our experience as described above, we determined that a starting point for the upstream Port Utilization Threshold should be 70 percent and the downstream Port Utilization Threshold should be 80 percent. For the Port Utilization Duration, we determined that the starting point should be approximately 15 minutes (although some technical limitations in some newer CMTSes deployed on Comcast's network may make this time period vary slightly). Thus, over any 15-minute period, if an average of more than 70 percent of a port's upstream bandwidth capacity or more than 80 percent of a port's downstream bandwidth capacity is utilized, that port is determined to be in a Near Congestion State.
Based on the trials conducted and operational experience to date, a typical CMTS port on our HSI network is in a Near Congestion State only for relatively small portions of the day, if at all, though there is no way to forecast what will be the busiest time on a particular port on a particular day. Moreover, the trial data and operational experience indicate that, even when a particular port is in a Near Congestion State, the instances where the network actually becomes congested during the Port Utilization Duration are few, and managed users whose packets may be intermittently delayed or dropped during those congested periods perceive little, if any, effect, as discussed below.
| TOC |
Once a particular CMTS port is in a Near Congestion State, the software examines whether any cable modems are consuming bandwidth disproportionately. (Note: Although each cable modem is typically assigned to a particular household, the software does not and cannot actually identify individual users, the number of users sharing a cable modem, or analyze particular users' traffic.) For purposes of this document, we use "cable modem", "user", and "subscriber" interchangeably to mean a subscriber account or user account and not an individual person. For a user to enter an Extended High Consumption State, he or she must consume greater than a certain percentage of his or her provisioned upstream or downstream bandwidth (the "User Consumption Threshold") for a specific length of time (the "User Consumption Duration"). The User Consumption Threshold is measured as a user's consumption of a particular percentage of his or her total provisioned upstream or downstream bandwidth. That bandwidth is the maximum speed that a particular modem can achieve based on the tier (personal/residential, commercial, etc.) the customer has purchased. For example, if a user buys a service with speeds of 50 Mbps downstream and 10 Mbps upstream, then his or her provisioned downstream speed is 50 Mbps and provisioned upstream speed is 10 Mbps. It is also important to note that because the User Consumption Threshold is a percentage of provisioned bandwidth for a particular user account, and not a static value, users of higher speed tiers have correspondingly higher User Consumption Thresholds. Lastly, the User Consumption Duration is measured in minutes.
Following lab tests, simulations, technical trials, customer feedback, vendor evaluations, and an independent third-party consulting analysis, we have determined that the appropriate starting point for the User Consumption Threshold is 70 percent of a subscriber's provisioned upstream or downstream bandwidth, and that the appropriate starting point for the User Consumption Duration is 15 minutes (this has been further validated in our full deployment). That is, when a subscriber uses an average of 70 percent or more of his or her provisioned upstream or downstream bandwidth over a particular 15-minute period, that user is then in an Extended High Consumption State. Therefore, this is a consumption-based threshold and not a peak-speed-based threshold. Thus, the Extended High Consumption State is not tied to whether a user has bursted once or more above this 70% threshold for a brief moment. Instead, it is consumption-based, meaning that a certain bitrate must be exceeded over at least the entire User Consumption Duration.
The User Consumption Thresholds have been set sufficiently high that using the HSI connection for VoIP, gaming, web surfing, or most streaming video cannot alone cause subscribers to our standard-level HSI service to exceed the User Consumption Threshold. For example, while one of Comcast's common HSI service tiers has a provisioned downstream bandwidth of 22 Mbps today, streaming video (even some HD video) from Hulu uses less than 2.5 Mbps, a Vonage or Skype VoIP call uses less than 131 Kbps, and streaming music uses less than 128 kbps (in this example, 70 percent of 22 Mbps is 15.4 Mbps). As noted above, these values are subject to change as necessary in the same way that specific anti-spam or other network management practices are adjusted to address new issues that arise, or should unexpected software bugs or other problems arise.
Based on data collected from the trial markets where the new congestion management practices were tested (further validated in our full deployment), on average less than one-third of one percent of subscribers have had their traffic priority status changed to the BE state on any given day. For example, in Colorado Springs, CO, the largest test market, on any given day in August 2008, an average of 22 users out of 6,016 total subscribers in the trial had their traffic priority status changed to BE at some point during the day.
A user's traffic is released from a BE state when the user's bandwidth consumption drops below 50 percent of his or her provisioned upstream or downstream bandwidth for a period of approximately 15 minutes. These release criteria are intended to minimize (and hopefully prevent) user QoS oscillation, i.e., a situation in which a particular user could cycle repeatedly between BE and PBE. Thus, without this lower release criteria, we were concerned that certain users would oscillate between BE and PBE states for an extended period, without clear benefit to the system and other users, and placing an unnecessary signaling burden on the system. NetForecast, Inc., an independent consultant retained to provide analysis and recommendations regarding Comcast's trials and related congestion management work, suggested this approach, which has worked well in our trials, lab testing, and subsequent national deployment.
Simply put, there are four steps to determining whether the traffic associated with a particular cable modem is designated as PBE or BE:
In cases where a CMTS regularly enters a Near Congestion State, and where congestion subsequently does occur, but where no users match the criteria to be classified in an Extended High Consumption State, this may indicate the congestion observed is regularly occurring, rather than unpredictable congestion. As such, this may be an additional data point in favor of considering whether and when to add capacity.
Figure 2 graphically depicts how this congestion management process works, using an example of a situation where upstream port utilization may be reaching a Near Congestion State (the same diagram, with different values in the appropriate places, could be used to depict the management process for downstream ports, as well):
Figure 2: Upstream Congestion Management Decision Flowchart
/\
+------------+ / \ +---------+ +---------+
| Start | / \ | | / /
| Congestion | / \ | | / /
| Management +-->+ Question +--YES-->| Result |--THEN-->/ ACTION /
| Process | \ #1 / | #1 | / #1 /
| | \ / | | / /
+------------+ \ / +---------+ +---------+
\/ |
| THEN
NO |
| \/
\/ /\
+---------+ / \
| | / \
| | / \
| Result |<-------------NO------------+ Question +
| #2 | \ #2 /
| | \ /
+---------+ \ /
\/
|
YES
|
/\ \/
+---------+ / \ +---------+
| | / \ | |
| | / \ THEN, AT | |
| Result |<--YES--+ Question + <---NEXT ANALYSIS------+ Result |
| #4 | \ #3 / POINT /\ | #3 |
| | \ / | | |
+---------+ \ / | +---------+
\/ |
| |
+------------NO-------------+
KEY TO FIGURES ABOVE:
Question #1: Is the CMTS Upstream Port Utilization at an average
of OVER 70% for OVER 15 minutes?
Result #1: CMTS marked in a Near Congestion State, indicating
congestion *may* occur soon.
Action #1: Search most recent analysis timeframe (approx. 15 mins.)
of IPDR usage data.
Question #2: Are any users consuming an average of OVER 70% of
provisioned upstream bandwidth for OVER 15 minutes?
Result #2: No action taken.
Result #3: Change user's upstream traffic from Priority Best Effort
(PBE) to Best Effort (BE).
Question #3: Is the user in Best Effort (BE) consuming an average
of LESS THAN 50% of provisioned upstream bandwidth
over a period of 15 minutes?
Result #4: Change user's upstream traffic back to Priority Best
Effort (PBE) from Best Effort (BE).
| Figure 2 |
| TOC |
When a CMTS port is in a Near Congestion State and a cable modem connected to that port is in an Extended High Consumption State, that cable modem's traffic is designated as BE. Depending upon the level of utilization on the CMTS port, this designation may or may not result in the user's traffic being delayed or, in extreme cases, dropped before PBE traffic is dropped. This is because of the way that the CMTS handles traffic. Specifically, CMTS ports have what is commonly called a "scheduler" that puts all the packets coming from or going to cable modems on that particular port in a queue and then handles them in turn. A certain number of packets can be processed by the scheduler in any given moment; for each time slot, PBE traffic is given priority access to the available capacity, and BE traffic is processed on a space-available basis.
A rough analogy would be to busses that empty and fill up at incredibly fast speeds. As empty busses arrive at the figurative "bus stop" -- every two milliseconds in this case -- they fill up with as many packets as are waiting for "seats" on the bus, to the limits of the bus' capacity. During non-congested periods, the bus will usually have several empty seats, but, during congested periods, the bus will fill up and packets will have to wait for the next bus. It is in the congested periods that BE packets will be affected. If there is no congestion, packets from a user in a BE state should have little trouble getting on the bus when they arrive at the bus stop. If, on the other hand, there is congestion in a particular instance, the bus may become filled by packets in a PBE state before any BE packets can get on. In that situation, the BE packets would have to wait for the next bus that is not filled by PBE packets. In reality, this all takes place in two-millisecond increments, so even if the packets miss 50 "busses", the delay will only be about one-tenth of a second.
During times of actual network congestion, when packets from BE traffic might be intermittently delayed, there is a variety of effects that could be experienced by a user whose traffic is delayed, depending upon what applications he or she is using. Typically, a user whose traffic is in a BE state during actual congestion may find that a webpage loads sluggishly, a peer-to-peer upload takes somewhat longer to complete, or a VoIP call sounds choppy. Of course, the same thing could happen to the customers on a port that is congested in the absence of any congestion management; the difference here is that the effects of any such delays are shifted toward those who have been placing the greatest burden on the network, instead of being distributed randomly among the users of that port without regard to their consumption levels. As a matter of fact, our studies concluded that the experience of the PBE subscribers improves when this congestion management system is enabled. This conclusion is based on network measurements, such as latency.
NetForecast explored the potential risk of a worst-case scenario for users whose traffic is in a BE state: the possibility of "bandwidth starvation" in the theoretical case where 100 percent of the CMTS bandwidth is taken up by PBE traffic for an extended period of time. In theory, such a condition could mean that a given user whose traffic is designated BE would be unable to effectuate an upload or download (as noted above, both are managed separately) for some period of time. However, when these management techniques were tested, first in company testbeds and then in our real-world trials conducted in the five markets (further validated in our full deployment), such a theoretical condition did not occur. In addition, our experience with the system as fully deployed in our production network demonstrates that these management practices have very modest real-world impacts. In addition, Comcast did not receive a single customer complaint in any of the trial markets that could be traced to this congestion management system, despite having broadly publicized these trials. In our subsequent national deployment into our production network, we still have yet to find a specific complaint that can be traced back to the effect of this congestion management system.
Comcast continues to monitor how user traffic is affected by these new congestion management techniques and will make the adjustments necessary to ensure that all Comcast HSI customers have a high-quality Internet experience.
| TOC |
The above-mentioned functions are carried out using three different types of application servers, supplied by three different vendors. As mentioned above, these servers are installed near Comcast's regional network routers. The exact locations of these servers is not particularly relevant to this document, as this does not change the fact that they manage individual CMTS ports.
The first application server is an IPDR server, which collects relevant cable modem volume usage information from the CMTS, such as how many aggregate upstream or downstream bytes a subscriber uses over a particular period of time. IPDR has been adopted as a standard by many industry organizations and initiatives, such as CableLabs, ATIS, ITU, and 3GPP, among others. The IPDR software deployed was developed by Active Broadband Networks, and is noted as the Statistics Collection Server in Figure 3.
The second application server is the Congestion Management server, which uses Simple Network Management Protocol ("SNMP") [RFC3410] (Case, J., Mundy, R., Partain, D., and B. Stewart, “Introduction and Applicability Statements for Internet-Standard Management Framework,” December 2002.) to measure CMTS port utilization and detect when a port is in a Near Congestion State. When this happens, the Congestion Management server then queries the relevant IPDR data for a list of cable modems meeting the criteria set forth above for being in an Extended High Consumption State. The Congestion Management Server software deployed was developed by Sandvine.
If one or more users meet the criteria to be managed, then the Congestion Management server notifies a third application server, the PCMM application server, as to which users have been in an Extended High Consumption State and whose traffic should be treated as BE. The PCMM servers are responsible for signaling a given CMTS to set the traffic for specific cable modems with a BE QoS, and for tracking and managing the state of such CMTS actions. If no users meet the criteria to be managed, no users will have their traffic managed. The PCMM software deployed was developed by Camiant, and is noted as the QoS Server in Figure 3.
Figure 3 graphically depicts the high-level management flows among the congestion management components on Comcast's network, as described above:
Figure 3: Simplified Diagram Showing High-Level Management Flows Relevant to the System
+---------------+ +---------------+
| Congestion | Instruct QoS Server | QoS |
| Management |******to Change QoS for****>| Server |
| Server | a Device | |
+----+---+------+ +-------+-------+
/\ /\ *
| | Relay Selected *
X +---Statistics: Bytes---+ QoS Action:
| Up/Down by Device | Change from PBE
X +-------+-------+ to BE, or from
| | Statistics | BE to PBE
X | Collection | *
Periodic SNMP | Server | *
Requests to +---------------+ *
Check CMTS Port /\ *
Utilization | *
Levels Statistics Sent *
| Periodically From CMTS *
X | *
| +-----------+-----------+ *
+-X-X-X-X-X-X->| CMTS in Headend |<********
+-----------------------+
H /\ /\ H
H Internet Traffic H
H to/from User H
H \/ \/ H
/+---------------------+\
/ | User's +---------+ |\
/ | Home | Cable | | \
| | Modem | |
============ | +---------+ |
= Notes: = +----------------------+
= ======================================================
= 1-Statistics Collection Servers use IP Detail Records (IPDR). =
= 2-QoS Servers use PacketCable Multimedia (PCMM) =
= to set QoS gates on the CMTS. =
= 3-This figure is a simplification of the actual network and =
= servers, which included redundancies and other complexities =
= not necessary to depict the functional design. =
=================================================================
| Figure 3 |
| TOC |
Comcast started design and development of this new protocol-agnostic congestion management system in March 2008. Comcast shared the design with the IETF and others in the Internet community, as well as with an independent consultant, incorporating feedback we received into the final design. Following lab testing, the system was tested in Comcast's production network in trial markets between June and September 2008. Comcast's production network transition to this new protocol-agnostic congestion management system began in October 2008 and was completed on December 31, 2008.
As described herein, the new approach does not manage congestion by focusing on managing the use of specific protocols. Nor does this approach use TCP "reset packets" [RFC3360] (Floyd, S., “Inappropriate TCP Resets Considered Harmful,” August 2002.). Rather, the system acts such that (1) during periods when a CMTS port is in a Near Congestion State, (2) it identifies the subscribers on that port who have consumed a disproportionate amount of bandwidth over the preceding 15 minutes, (3) it lowers the priority status of those subscribers' traffic to BE status until those subscribers meet the release criteria, and (4) during periods of actual congestion, handles PBE traffic before BE traffic. Comcast's trials and subsequent national deployment indicate that this new congestion management system ensures a quality online experience for all of Comcast's HSI customers.
| TOC |
This system was developed to cope with somewhat "normal" occurrences of congestion that could occur on virtually any IP network. It should also be noted, however, that such a system could also prove particularly useful in the case of "exceptional network utilization" events which existing network usage models do not or cannot accurately predict. Some network operators refer to these exceptional events as 'surges' in utilization, similar to sudden surges in demand in electrical power grids, with which many people may be familiar.
For example, in the case of a severe global pandemic, it may be expected that large swaths of the population may need to work remotely, via their Internet connection. In such a case, a largely unprecedented level of utilization may occur. In such cases, it may be helpful to have a flexible congestion management system that could adapt to this situation and help allocate network resources while additional capacity is being brought online or while a temporary condition persists.
| TOC |
The main limitations of the system include:
| TOC |
There are several new IETF working group efforts which are focused on the question of congestion and its effects, avoiding congestion, managing congestion, and communicating congestion information. This includes the Congestion Exposure (CONEX) working group, the Application Layer Transport Optimization (ALTO) working group, and the Low Extra Delay Background Transport (LEDBAT) working group. Should one or more of these working groups be successful in producing useful work, it is possible that the design or configuration of the system documented here may need to change. For example, this congestion management system does not currently have a way take into accounts differing classes of data transfer, such as that which LEDBAT may specify, which may better yield to other traffic than existing transport protocols. In addition, CONEX may specify methods for this or other systems to signal congestion state or expected congestion to other parts of the network, and/or hosts on either end of a particular network flow. Furthermore, it is conceivable that the result of current or future IETF work could obviate the need for such a congestion management system entirely.
| TOC |
It is important that an ISP secure access to the Congestion Management Servers and the QoS Servers, as well as QoS signaling to the CMTSes, so that unauthorized users and/or hosts cannot make unauthorized changes to QoS settings in the network.
It is also important to secure access to the Statistics Collection Server since this contains IPDR-based byte transfer data which is considered private by end users on an individual basis. In addition this data is considered ISP-proprietary traffic data on an aggregate basis. Access to the Statistics Collection Server should also be secured so that false usage statistics cannot be fed into the system. It is important to note that IPDR data contain a count of bytes sent and bytes received, by cable modem MAC address, over a given interval of time. This data does not contain things such as the source and/or destination Internet address of that data, nor does it contain the protocols used, ports used, etc.
| TOC |
There are no IANA considerations in this document.
NOTE TO RFC EDITOR: PLEASE REMOVE THIS NULL SECTION PRIOR TO PUBLICATION.
| TOC |
The authors wish to acknowledge the hard work of the many people who helped to develop and/or review this document, as well as the people who helped deploy the system in such a short period of time.
The authors also wish to acknowledge the following individuals for performing a detailed review of this document and/or providing comments and feedback that helped to improve and evolve this document:
- Kris Bransom
- Bob Briscoe
- Lars Eggert
- Ari Keranen
- Tero Kivinen
- Matt Mathis
- Stanislav Shalunov
| TOC |
NOTE TO RFC EDITOR: PLEASE REMOVE THIS SECTION PRIOR TO PUBLICATION.
| TOC |
NOTE TO RFC EDITOR: PLEASE REMOVE THIS SECTION PRIOR TO PUBLICATION.
| TOC |
NOTE TO RFC EDITOR: PLEASE REMOVE THIS SECTION PRIOR TO PUBLICATION.
ALSO: Please try to adjust page breaks to keep various ASCII art on one page, as some may break across two pages now.
| TOC |
| TOC |
| Chris Bastian | |
| Comcast Cable Communications | |
| One Comcast Center | |
| 1701 John F. Kennedy Boulevard | |
| Philadelphia, PA 19103 | |
| US | |
| Email: | chris_bastian@cable.comcast.com |
| URI: | http://www.comcast.com |
| Tom Klieber | |
| Comcast Cable Communications | |
| One Comcast Center | |
| 1800 Bishops Gate Drive | |
| Mount Laurel, NJ 08054 | |
| US | |
| Email: | tom_klieber@cable.comcast.com |
| URI: | http://www.comcast.com |
| Jason Livingood | |
| Comcast Cable Communications | |
| One Comcast Center | |
| 1701 John F. Kennedy Boulevard | |
| Philadelphia, PA 19103 | |
| US | |
| Email: | jason_livingood@cable.comcast.com |
| URI: | http://www.comcast.com |
| Jim Mills | |
| Comcast Cable Communications | |
| One Comcast Center | |
| 1800 Bishops Gate Drive | |
| Mount Laurel, NJ 08054 | |
| US | |
| Email: | jim_mills@cable.comcast.com |
| URI: | http://www.comcast.com |
| Richard Woundy | |
| Comcast Cable Communications | |
| 27 Industrial Avenue | |
| Chelmsford, MA 01824 | |
| US | |
| Email: | richard_woundy@cable.comcast.com |
| URI: | http://www.comcast.com |