| Network Working Group | C.D. Donley |
| Internet-Draft | CableLabs |
| Intended status: Informational | L.H. Howard |
| Expires: May 19, 2012 | Time Warner Cable |
| V.K. Kuarsingh | |
| Rogers Communications | |
| J.B. Berg | |
| CableLabs | |
| J.D. Doshi | |
| University of Colorado | |
| November 16, 2011 |
Assessing the Impact of Carrier-Grade NAT on Network Applications
draft-donley-nat444-impacts-03
NAT444 is an IPv4 extension technology being considered by Service Providers to continue offering IPv4 service to customers while transitioning to IPv6. This technology adds an extra Carrier-Grade NAT ("CGN") in the Service Provider network, often resulting in two NATs. CableLabs, Time Warner Cable, and Rogers Communications independently tested the impacts of NAT444 on many popular Internet services using a variety of test scenarios, network topologies, and vendor equipment. This document identifies areas where adding a second layer of NAT disrupts the communication channel for common Internet applications. This document was updated to also include Dual-Stack Lite impacts.
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on May 19, 2012.
Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
IANA and APNIC exhausted their IPv4 address space in 2011. Current projections suggest that RIPE and ARIN may exhaust their free pools of IPv4 addresses in 2012. IPv6 is the solution to the IPv4 depletion problem; however, the transition to IPv6 will not be completed prior to IPv4 exhaustion. NAT444 [I-D.shirasaki-nat444] and Dual-Stack Lite ([RFC6333]) are transition mechanisms that will allow Service Providers to multiplex customers behind a single IPv4 address, which will allow many legacy devices and applications some IPv4 connectivity. While both NAT444 and Dual-Stack Lite do provide basic IPv4 connectivity, they impact a number of advanced applications. This document describes suboptimal behaviors of NAT444 and DS-Lite in our test environments.
In July-August 2010, CableLabs, Time Warner Cable, and Rogers Communications tested the impact of NAT444 on common applications using Carrier Grade NAT (CGN) devices. This testing was focused on a wide array of real time usage scenarios designed to evaluate the user experience over the public Internet using NAT444, in both single ISP and dual ISP environments. The purpose of this testing was to identify applications where the technology either breaks or significantly impacts the user experience. The outcome of the testing revealed that applications such as video streaming, video gaming and peer-to-peer file sharing are impacted by NAT444.
From June – October 2011, CableLabs conducted additional testing of CGN technologies, including both NAT444 and Dual-Stack Lite. The testing focused on working with several vendors including A10, Alcatel-Lucent, and Juniper to optimize the performance of those applications that experienced negative impacts during earlier CGN testing and to expand the testing to DS-Lite.
Applications that were tested included but were not necessarily limited to the following:
The diagrams below depict the general network architecture used for testing NAT444 and Dual Stack-Lite co-existence technologies at CableLabs.
^^^^^^^^ (Internet) vvvvvvvv | | +---------------+ | CGN | +---------------+ | +---------------+ | CMTS | +---------------+ | +---------------+ | CM | +---------------+ | +-------------------------+ | Home Router | +-------------------------+ | +---------------+ | Client | +---------------+
This is a typical case for a client accessing content on the Internet. For this case, we focused on basic web browsing, voice and video chat, instant messaging, video streaming (using YouTube, Google Videos , etc.), Torrent leeching and seeding, FTP, and gaming.
^^^^^^^^ (Internet) vvvvvvvv | | +---------------+ | CGN | +---------------+ | +---------------+ | CMTS | +---------------+ | +---------------+ | CM | +---------------+ | +-------------------------+ | Home Router | +-------------------------+ | | +---------------+ +---------------+ | Client | | Client | +---------------+ +---------------+
This is similar to Case 1, except that two clients are behind the same LSN and in the same home network. This test case was conducted to observe any change in speed in basic web browsing and video streaming.
^^^^^^^^ (Internet) vvvvvvvv | | +---------------+ | CGN | +---------------+ | +---------------+ | CMTS | +---------------+ | ---------------------------------------- | | +---------------+ +---------------+ | CM | | CM | +---------------+ +---------------+ | | +-------------------------+ +-------------------------+ | Home Router | | Home Router | +-------------------------+ +-------------------------+ | | +---------------+ +---------------+ | Client | | Client | +---------------+ +---------------+
In this scenario, the two clients are under the same LSN but behind two different gateways. This simulates connectivity between two residential subscribers on the same ISP. We tested peer-to-peer applications.
^^^^^^^^ ^^^^^^^^
( ISP A ) ( ISP B )
Vvvvvvvv vvvvvvvv
| |
+---------------+ +---------------+
| LSN | | LSN |
+---------------+ +---------------+
| |
+---------------+ +---------------+
| CMTS | | CMTS |
+---------------+ +---------------+
| |
+---------------+ +---------------+
| CM | | CM |
+---------------+ +---------------+
| |
+-------------------------+ +-------------------------+
| Home Router | | Home Router |
+-------------------------+ +-------------------------+
| |
+---------------+ +---------------+
| Client | | Client |
+---------------+ +---------------+
This test case is similar to Case 1 but with the addition of another identical ISP. This topology allows us to test traffic between two residential customers connected across the Internet. We focused on client-to-client applications such as IM and peer-to-peer.
The lab environment was intended to emulate multiple service provider networks with a CGN deployed, and with connectivity to the public IPv4 or IPv6 internet (as dictated by the co-existence technology under test). This was accomplished by configuring a CGN behind multiple CMTSes and setting up multiple home networks for each ISP. Testing involved sending traffic to and from the public internet in both single and dual ISP environments, using both single and multiple home networks. The following equipment was used for testing:
One or more CPE gateway devices were configured in the home network. One or more host devices behind the gateways were also configured in order to test conditions such as multiple users on multiple home networks in the CGN architecture, both in single and dual ISP environments.
The scope of testing was honed down to the specific types of applications and network conditions that demonstrated a high probability of diminishing user experience based on prior testing. The following use cases were tested:
The following CPE devices were used for testing these applications on one or more home networks:
Metrics data that were collected during the course of testing were related to throughput, latency, and jitter. These metrics were evaluated under three conditions:
In our testing, we found no significant differences with respect to latency or jitter when the CGN was in the network versus when it was not present in the network. It should be noted that we did not conduct any performance testing and metrics gathered were limited to single session scenarios. Also, bandwidth was not restricted on the DOCSIS network. Simulated homes shared a single DOCSIS upstream and downstream channel.
Note: Performance testing as defined by CableLabs includes load testing, induction of impairments on the network, etc. This type of testing was out of scope for CGN testing.
The following test scenarios were executed using the aforementioned applications and test equipment:
These test scenarios were executed for both NAT444 and DS-Lite technologies.
The CGN was configured for optimal setting for the specific test being executed for NAT444 or DS-Lite. Individual vendors provided validation of the configuration used for the co-existence technology under test prior to the start of testing. Some NAT444 testing used private [RFC1918] IPv4 space between the CGN and CPE router; other tests used public (non-[RFC1918]) IPv4 space between the CGN and CPE router. With the exception of 6to4 ([RFC3056]) traffic, we observed no difference in test results whether private or public address space was used. 6to4 failed when public space was used between the CGN and CPE router was public, but CPE routers did not initiate 6to4 when private space was used.
CPE gateways and client devices were configured with IPv4 or IPv6 addresses using DHCP or manual configuration as required by each of the devices used in the test.
All devices were brought to operational state. Connectivity of CPE devices to provider network and public Internet were verified prior to start of each test.
IP sniffers and metrics tools were configured as required before starting tests. IP capture and metrics data was collected for all failed test scenarios. Sniffing was configured behind the home routers, north and south of the CMTS, and north and south of the CGN.
The test technician executed test scenarios listed above, for single and dual ISP environments, testing multiple users on multiple home networks, using the applications described above, where applicable to the each specific test scenario. Results checklists were compiled for all tests executed and for each combination of devices tested.
CGN testing revealed that basic services such as e-mail and web browsing worked normally and as expected. However, there were some service affecting issues noted for applications that fall into two categories; dropped service and degraded service. In addition, for some specific applications in which the performance was degraded, throughput, latency and jitter measurements were taken. We observed that performance often differs from vendor to vendor and from test environment to test environment, and the results are somewhat difficult to predict. So as to not become a comparison between different vendor implementations, these results are presented in summary form. When issues were identified, we worked with the vendors involved to confirm the specific issues and explore workarounds. Except where noted, impacts to NAT444 and DS-Lite were similar.
In 2010 testing, we identified that IPv6 transition technologies such as 6to4 [RFC3056] and Teredo [RFC4380]) fail outright or are subject to severe service degradation. We did not repeat transition technology testing in 2011.
Several peer-to-peer applications failed in both the NAT444 and Dual-Stack Lite environments. Specifically, peer-to-peer gaming using Xbox, and peer-to-peer SIP calls using the PJSIP client did not succeed. Many CGN devices use "full cone" NAT so that once the CGN maps a port for outbound services, it will accept incoming connections to that port. However, some applications did not first send outgoing traffic and hence did not open an incoming port through the CGN. Other applications try to open a particular fixed port through the CGN; while service will work for a single subscriber behind the CGN, it fails when multiple subscribers try to use that port.
PJSIP and other SIP software worked when clients used a registration server to initiate calls, provided that the client inside the CGN initiated the traffic first and that only one SIP user was active behind a single IPv4 address at any given time. However, in our testing, we observed that when making a direct client-to-client SIP call across two home networks on a single ISP, or when calling from a single home network across dual ISPs, calls could neither be initiated nor received.
In the case of peer-to-peer gaming between two Xbox 360 users in different home networks on the same ISP, the game could not be connected between the two users. Both users shared an outside IP address, and tried to connect to the same port, causing a connection failure. There are some interesting nuances to this problem. In the case where two users are in the same home network and the scenario is through a single ISP, when the Xbox tries to register with the Xbox server, the server sees that both Xboxes are coming through the same public IP address and directs the devices to connect using their internal IP addresses. So, the connection ultimately gets established directly between both Xboxes via the home gateway, rather than the Xbox server. In the case where there are two Xbox users on two different home networks using a single ISP, and the CGN is configured with only one public IPv4 address, this scenario will not work because the route between the two users cannot be determined. However, if the CGN is configured with two public NAT IP addresses this scenario will work because now there is a unique IP address to communicate with. This is not an ideal solution, however, because it means that there is a one-to-one relationship between IP addresses in the public NAT and the number of Xbox users on each network.
Other peer-to-peer applications that were noted to fail were seeding sessions initiated on Bittorent and uTorrent. In our test, torrent seeding was initiated on a client inside the CGN. Leeching was initiated using a client on the public Internet. It was observed that direct peer-to-peer seeding did not work. However, the torrent session typically redirected the leeching client to a proxy server, in which case the torrent session was set up successfully. Additionally, with the proxy in the network, re-seeding via additional leech clients worked as would be expected for a typical torrent session. Finally, uTorrent tries to use STUN to identify its outside address. In working with vendors, we learned that increasing the STUN timeout to 4 minutes improved uTorrent seeding performance behind a CGN, resulting in the ability for the uTorrent client to open a port and successfully seed content.
There were several applications that diminished the user experience in our environment. We measured these variations in user experience against a baseline IPv4 environment where NAT is not deployed. Specifically, large size file transfers and multiple video streaming sessions initiated on a single client on the same home network proved to be problematic with some home routers behind the CGN.
In our testing, we tried large file transfers from several FTP sites, as well as downloading sizable audio and video files (750MB – 1.4 GB) from the Internet Archive. We observed that when Dual-Stack Lite was implemented for some specific home router and client combinations, the transfer rate was markedly slower. For example, PC1 using one operating system behind the same home router as PC2 using a different operating system yielded a transfer rate of 120Kbps for PC1, versus 250Kbps for PC2. Our conclusion is that varying combinations of home routers and CE client devices may result in a user experience that is less than what the user would expect for typical applications. It is also difficult to predict which combinations of CPE routers and CE devices will produce a reduced experience for the user. We did not analyze the root cause of the divergence in performance across CE devices, as this was beyond the scope of our testing. However, as this issue was specific to Dual-Stack Lite, we suspect that it is related to the MTU.
While video streaming sessions for a single user generally performed well, testing revealed that video streaming sessions such as Microsoft Smooth Streaming technology (i.e. Silverlight) or Netflix might also exhibit some service impacting behavior. In particular, this was observed on one older, yet popular and well-known CPE router where the first session was severely degraded when a second session was initiated in the same home network. Traffic from the first session ceased for 8 s once the second session was initiated. While we are tempted to write this off as a problematic home router, its popularity suggests that home router interactions may cause issues in NAT444 deployments (newer routers that support DS-Lite were not observed to experience this condition). Overall, longer buffering times for video sessions were noted for most client devices behind all types of home routers. However, once the initial buffering was complete, the video streams were consistently smooth. In addition, there were varying degrees as to how well multiple video sessions were displayed on various client devices across the CPE routers tested. Some video playback devices performed better than others.
Since CableLabs completed initial CGN testing in 2010, there have been quantifiable improvements in performance over CGN since that time. These improvements may be categorized as follows:
In terms of content provider updates, we have noted improvements in the overall performance of streaming applications in the CGN environment. Whereas applications such as streaming video were very problematic a year ago with regard to jitter and latency, our most recent testing revealed that there is less of an issue with these conditions, except in some cases when multiple video streaming sessions were initiated on the same client using specific types of home routers. Applications such as MS Smooth Streaming appear to have addressed these issues to some degree.
As far as application updates, use of STUN and/or proxy servers to offset some of the limitations of NAT and tunneling in the network are more evident as workarounds to the peer-to-peer issues. Applications appear to have incorporated other mechanisms for delivering content faster, even if buffering times are somewhat slower and the content is not rendered as quickly.
CGN vendors have also upgraded their devices to mitigate several known issues with specific applications. With regard to addressing peer-to-peer SIP call applications, port reservations appear to be a workaround to the problem. However, this approach has limitations because of there are limited numbers of users that can have port reservations at any given time. For example, one CGN implementation allowed a port reservation to be made on port 5060 (default SIP port) but this was the only port that could be configured for the SIP client. This means that only one user can be granted the port reservation.
There are other challenges that arise when using shared IPv4 address space, as with NAT444. Some of these challenges include:
| Test Scenario (per Test Plan) | Single ISP, Single HN, Single User | Single ISP, Two HN, Single User on Each | Dual ISP, One HN with One User on Each ISP | Dual ISP, One HN+One User on ISP-A, Two HN with One User on Each on ISP-B | Notes |
|---|---|---|---|---|---|
| Video streaming over Netflix | Pass | Pass | Pass | Pass | fails behind one router |
| Video streaming over YouTube | Pass | Pass | Pass | Pass | |
| Video streaming over Joost | Pass | Pass | Pass | Pass | |
| Online gaming with one user | Pass | Pass | Pass | NT | |
| Peer to Peer gaming with two users | Pass | Fail | Pass | NT | fails when both users NAT to same address |
| Bit Torrent uTorrent file seeding | Fail | Fail | Fail | Fail | |
| Bit Torrent uTorrent file leeching | Pass | Pass | Pass | Pass | |
| Pandora internet radio | Pass | Pass | Pass | Pass | |
| FTP server | Pass | Pass | Pass | Pass | |
| Web conferencing GTM | Pass | Pass | Pass | Pass | |
| Social Networking Facebook | Pass | Pass | Pass | Pass | |
| Social Networking Webkinz | Pass | Pass | Pass | Pass | |
| X-Lite for SIP calls with proxy | Pass | Pass | Pass | Pass | |
| X-Lite for SIP calls no proxy | Fail | Fail | Fail | Fail | |
| Skype text chat | Pass | Pass | Pass | Pass | |
| Skype video chat | Pass | Pass | Pass | Pass | |
| Oovoo | Pass | Pass | Pass | Pass | |
| MS Smooth streaming | Pass | Pass | Pass | Pass | |
| Internet Archive video streaming | Pass | Pass | Pass | Pass | |
| Internet Archive audio streaming | Pass | Pass | Pass | Pass | |
| Internet Archive file download | Pass | Pass | Pass | Pass | |
| Iclips | Pass | Pass | Pass | Pass |
| Test Scenario (per Test Plan) | DS-Lite Test Results | Duration of Test Performed | Description of Test Execution | General Observations/Notes |
|---|---|---|---|---|
| Video streaming over Netflix | Pass | 15 | ||
| Video streaming over YouTube | Pass | 10 | ||
| Video streaming over Joost | Pass | 10 | ||
| On line gaming (one user) | Pass | 15 | ||
| Peer to Peer gaming (two users) | Fail | NA | user inside HN1 playing game against user inside HN2 | Users inside both HN are not able to connect. The error shown on console- "The game session is no longer available" |
| Bit Torrent/uTorrent file seeding | Fail | 12 | user on the internet is able to download file using proxy server and not peer-to-peer | |
| Bit Torrent/uTorrent file leeching | Pass | 10 | ||
| Pandora internet radio | Pass | 10 | ||
| FTP server | Pass | 700 Mb | ||
| Web conferencing (GTM) | Pass | 10 | ||
| Social Networking - Facebook | Pass | NA | ||
| Social Networking - Webkinz | Pass | NA | ||
| X-Lite (for SIP calls) (proxy given) | Pass | 10 | ||
| X-Lite (for SIP calls) (proxy not given) | Fail | NA | ||
| Skype text chat | Pass | NA | ||
| Skype video chat | Pass | 20 | ||
| Oovoo | Pass | 15 | ||
| MS Smooth streaming | Pass | 10 | ||
| Internet Archive - video streaming | Pass | 10 | ||
| Internet Archive - audio streaming | Pass | 5 | ||
| Internet Archive - file download | Pass | 80 Mb | ||
| Iclips | Pass | 10 |
The tables below summarize results from 2010 NAT444 testing at CableLabs, Time Warner Cable, and Rogers Communications. They are included for comparison with 2011 results, documented above.
| Test Case | Results | Notes |
|---|---|---|
| Web browsing | pass | |
| pass | ||
| FTP download | pass | performance degraded on very large downloads |
| Bittorrent leeching | pass | |
| Bittorrent seeding | fail | |
| Video streaming | pass | |
| Voice chat | pass | |
| Netflix streaming | pass | |
| Instant Messaging | pass | |
| Ping | pass | |
| Traceroute | pass | |
| Remote desktop | pass | |
| VPN | pass | |
| Xbox live | pass | |
| Xbox online | pass | Blocked by some LSNs. |
| Xbox network test | fail | Your NAT type is moderate. For best online experience you need an open NAT configuration. You should enable UPnP on the router. |
| Nintendo Wii | pass behind one LSN, fail behind another | |
| Playstation 3 | pass | |
| Team Fortress 2 | fail | pass behind one LSN, but performance degraded |
| Starcraft II | pass | |
| World of Warcraft | pass | |
| Call of Duty | pass | performance degraded behind one LSN |
| Slingcatcher | fail | |
| Netflix Party (Xbox) | fail | pass behind one LSN |
| Hulu | pass | performance degraded behind one LSN |
| AIM File Tranfer | pass | performance degraded |
| Webcam | fail | |
| 6to4 | fail | |
| Teredo | fail |
| Test Case | Results | Notes |
|---|---|---|
| Bittorrent leeching | pass | |
| Bittorrent seeding | fail | |
| Video streaming | fail | |
| Voice chat | pass | |
| Netflix streaming | pass | performance severely impacted, eventually failed |
| IM | pass | |
| Limewire leeching | pass | |
| Limewire seeding | fail |
| Test Case | Results | Notes |
|---|---|---|
| Limewire leeching | pass | |
| Limewire seeding | fail | |
| Utorrent leeching | pass | |
| Utorrent seeding | fail |
| Test Case | Results | Notes |
|---|---|---|
| Skype voice call | pass | |
| IM | pass | |
| FTP | fail | |
| Facebook chat | pass | |
| Skype video | pass |
This document has no IANA considerations.
Security considerations are described in [I-D.shirasaki-nat444].
| [I-D.shirasaki-nat444] | Yamagata, I, Shirasaki, Y, Nakagawa, A, Yamaguchi, J and H Ashida, "NAT444", Internet-Draft draft-shirasaki-nat444-02, July 2010. |
| [RFC3056] | Carpenter, B. and K. Moore, "Connection of IPv6 Domains via IPv4 Clouds", RFC 3056, February 2001. |
| [RFC4380] | Huitema, C., "Teredo: Tunneling IPv6 over UDP through Network Address Translations (NATs)", RFC 4380, February 2006. |
| [RFC6333] | Durand, A., Droms, R., Woodyatt, J. and Y. Lee, "Dual-Stack Lite Broadband Deployments Following IPv4 Exhaustion", RFC 6333, August 2011. |
| [RFC1918] | Rekhter, Y., Moskowitz, R., Karrenberg, D., Groot, G. and E. Lear, "Address Allocation for Private Internets", BCP 5, RFC 1918, February 1996. |
| [RFC6269] | Ford, M., Boucadair, M., Durand, A., Levis, P. and P. Roberts, "Issues with IP Address Sharing", RFC 6269, June 2011. |
Thanks to the following people for their testing, guidance, and feedback: