IPv6 versus IPv4 Performance

Yesterday from a post at the Deploy360 website, I learned of Comcast's IPv4 and IPv6 network speed testing tool:

        http://speedtest.comcast.net/

I did a quick test from my laptop in my office and got some very surprising results. The measured IPv6 performance was better than IPv4 by a gigantic margin. With IPv6, I got 822 Mbps download and 667Mbps download throughput. With IPv4, a mere 99Mbps upload and 18Mbps download!

Something seemed fishy, but I had to run off to other work, so I quickly posted the result to Twitter, planning to look into it later.

This generated quite a bit discussion with numerous folks on twitter and elsewhere. My initial speculation was that we do some rate limiting of IPv4 traffic at the Penn border routers for selected areas of the campus, and perhaps this was throttling the IPv4 performance. My other suspicion was that there was something significantly different in the IPv4 and IPv6 routing paths contributing to the difference. The graphic above does show a round-trip time difference of 63ms for the IPv4 path and 32ms for the IPv6 path, which suggests this. Furthermore, if the TCP window is not scaled properly to keep the pipe filled for this path at 63ms (but was for 32ms), then that would decrease throughput also - but not enough to account by itself for the observed difference.

Patrik Falstrom suspected a DPI device or other middlebox causing the problem. The only problem is that we don't have any such middleboxes (unless you consider an IP border router imposing IP address based rate limits a middlebox). In any case, I was leaning towards the rate limits as the cause myself, until I confirmed that those rate limits weren't being applied to any of the traffic from my office network. The rate limits are primarily targeted at the student residential dormitories - without them, our external links typically get overwhelmed with traffic to/from the dorms (most likely due to file sharing, a very common activity on college campuses). The border routers are configured to apply a token bucket rate policer to each individual IPv4 address within the network prefixes that cover the residential networks. Note that this rate limiting is completely application agnostic.  Also note that this scheme cannot scale to IPv6 (a single IPv6 subnet has more than 18 quintillion addresses!), a problem we're ignoring for the time being :-)

Repeat of the test

This morning, I decided to do another test (same laptop), but more carefully, and along with a packet capture. I also explicitly turned off the wireless interface (hmmmm) to make sure that all tests were using the wired gigabit ethernet interface. This time, I got much more reasonable looking results, both address families in the neighborhood of each other: IPv4 853Mbps down, 547Mbps up, and for IPv6 827Mbps down, 730Mbps up. One other difference I notice is that the roundtrip (ping) times to the destination server are 12ms for both IPv4 and IPv6. This is substantially different from yesterday's test (63 and 32ms respectively) despite the fact that I choose the same destination server at Comcast (Washington, DC).

A packet capture reveals that the destination server at Comcast for IPv4 was 68.87.73.52, and for IPv6 was 2001:558:1010:5:68:87:73:52. Are these the same endpoint? Hard to tell, but the fact that the last 4 fields of the IPv6 address spell out the IPv4 address in decimal might be a hint. The traffic streams use TCP port 5050. A traceroute to the IPv4 destination shows the outbound path takes one of Penn's commercial ISP links (Cogent) to New York and then back to Washington/VA. An IPv6 traceroute shows the outbound path goes out via our Internet2 link, the I2 commercial peering service, then Cogent (New York), Level3 (New York), and then Comcast to DC. So the IPv4 and IPv6 paths are substantially different in the forward direction. Harder to tell the path for the return traffic without the aid of some reverse traceroute tools or similar.

Getting a substantial fraction of a gigabit ethernet is not suprising - that's probably the bottleneck bandwidth along the measured path. My laptop has a gigabit ethernet connection to the building network, which in turn has dual 10 Gigabit Ethernet links to a 100 Gig campus core, and then multiple 10Gig links out to commercial ISPs/Internet2 etc. Most tier-1 ISP links and peerings are typically at least 10Gig.

The bandwidth-delay product on these paths is about 1,464 KB (1000Mbps * 12ms). The Comcast endpoint's receive window exceeds this, but my laptop's is slightly undersized, so I could probably do a bit of host tuning to boost the download numbers a bit more.

So, what's the explanation for the strange results I got yesterday? I wish had a packet capture to investigate, but my leading suspicion is that my laptop's wireless adapter (lower bandwidth, shared medium) was used in the IPv4 test, and the wired connection for the IPv6 one. If I have time later, I'll try to reproduce the issue.

--Shumon Huque

---

Addendum (February 9th 2014) - On closer inspection of the packet trace, the speed test appears to use multiple TCP streams in parallel, so scaling the window as high as the bw*delay product of the path isn't necessary.

Internet2 IPv6 Panel recap

A few notes from last month's IPv6 deployment panel at the Fall Internet2 Member Meeting in Philadelphia, which I moderated (October 2nd 2012). Watch the entire video of the session (1 hour 15 minutes) for full details.

I opened the session with a brief review of World IPv6 Launch and some its measurement data, and a mention of other IPv6 deployment measurement projects, including the NIST survey of universities.

ARIN - Mark Kosters, CTO

Mark began his presentation by talking about the current state of IPv4 address depletion. ARIN has 2.87 /8 equivalent address blocks left (note: it's down to 2.79 as of Nov 18th). RIPE and APNIC are almost out, below the one /8 threshold at which a real address rationing stage has already begun - they will give out a maximum of only /20 or /22 sized blocks regardless of how much IPv4 address space you really need. With current projections, ARIN is scheduled to exhaust in August 2013, but events could dramatically change the timeline. For example there are some ISPs in the ARIN region that could easily qualify for /9 allocations. ARIN is in phase 2 of a 4-phase runout plan - details can be seen at https://www.arin.net/resources/request/ipv4_countdown.html. Today, 58% of ARIN's membership has only IPv4 address space, 6% has only IPv6, and 36% have both IPv4 and IPv6 space. Legacy address blocks (ie. allocated to organizations prior to ARIN's existence - this is quite common in the US higher ed community) comprise 45% of the ARIN address space.

The second part of Mark's talk was about IPv6 deployment activites at ARIN itself. He went through a short history of their IPv6 implementation, which dates back to 2003, at which time they had a somewhat creaky, segregated (ie. non dualstack) implementation. By 2008 they migrated to a robust dualstack implementation. All ARIN services today are native, dualstack. Meeting networks are a bit more challenging, where they often have to rely on tunnels (due to hotel carrier limitations).

Comcast - John Brzozowski, Chief IPv6 Architect

Comcast now has IPv6 enabled on 50% of their broadband network footprint. However only 2.5% of the customer base is currently dualstack - of these dualstack users about 65% are using a computer directly connected to the cable modem, and 35% have an IPv6 enabled home router to which devices are attached. Comcast expects this ratio will eventually flip over to 80/20 in favor of home routers as it is in IPv4. To date, their focus has been on residential broadband, but they have pilots for business and commercial service. The Comcast metro ethernet service is IPv6 ready and they'd love to have more customers using it. In terms of traffic, Comcast has seen a 375% increase in IPv6 compared to World IPv6 day (June, 2011), with the majority of the increase occurring between January and June 2012, in the run up to World IPv6 Launch. The majority of the traffic is composed of services like Youtube and Netflix. The 2012 Olympics (streamed by Comcast/NBC in the US) had a noticeable impact - about 6% of this traffic to Comcast customers used IPv6. Comcast continues to work on content and services. Xfinity and comcast.net are IPv6 enabled and they use Akamai as a content delivery network.

They've put in place an extensive measurement and metrics platform to proactively detect any adverse affects on customer experience and take action if needed. Comcast sees advantages to IPv6 beyond the usual address space depletion concerns. An example cited was the fact that once they've allocated an IPv6 subnet, they don't have to be concerned about resizing it again, in marked contrast to their IPv4 deployment - this adds up over time to a significant operational benefit.

Marist College - Eric Kenny

Marist College is located in upstate network with 5,500 students on a 180-acre campus with 50 buildings. They started IPv6 deployment in 2010, and by June 2012 the wired network was mostly done. They plan to deploy IPv6 to the wireless network in the winter of 2012. One big item for them was getting provider independent address space from ARIN (mostly internal process issues). They started with a provider allocated block from NySERNET, but eventually got a /48 from ARIN, and are now wondering if they should have applied for a larger block. They also have a native IPv6 BGP peering with their commercial ISP, Lightower, which was described as an interesting experience, since they were their first IPv6 customer. They use stateless address autoconfiguration, and most devices on their network support IPv6. They haven't yet made much progress with IPv6 enabled network services, apart from DNS. At the current time about 4% of the traffic crossing their border is IPv6 - they expect this number to go up considerably after wireless deployment. One of their biggest concerns has been address tracking and accountability. They've developed their own application to track IPv6 address and MAC address associations.

Louisiana State University - Allie Hopkins

Allie focussed more on the application and process side of things. As part of their rollout process, LSU has had a great dialog with the user community, with extensive outreach and communications via message boards and e-mail lists, and close contact with application developers and deployers. And despite a set of issues (described later), they rate their deployment a success and as can be seen from World IPv6 Launch measurements, they generate a substantial amount of traffic. One of the issues they've had is with their DNS based network registration system which was incapable of working properly with IPv6 enabled hosts. Another issue they encountered was connectivity issues between tagged and untagged VLANs on interfaces, due to the Cisco routers using the same IPv6 link local addresses on them - this was fixed by manually configuring unique addresses on the VLAN interfaces. They've also had ongoing issues with being put on Google's AAAA blacklist.They are aware that they have some pockets of poor IPv6 connectivity within their campus network, but so far they've gotten no help from Google about measurement data that could help them more easily track down these cases.

I was scheduled to do a short presentation about IPv6 at Penn, but I decided to skip it (I'll post the slides later) in the interests of providing more time for audience discussion and questions.

Q&A Portion

Richard Machida (U of Alaska) asked if anyone was looking into deploying IPv6-only networks for any purpose. I answered that we've deployed them only in the lab for testing purposes (longer term, we would investigate whether it's possible to run specific applications that may not need IPv4, like VoIP on them). John says Comcast has no IPv6-only networks, but they do have plenty of IPv6-only devices that sit on dual-stack networks.

There was a question about more details of LSU's network registration system (from what I gathered it's a Netreg or Netreg like system), and whether 802.1x based network access control would be a solution (802.1x authentication happens at the link layer and is IP protocol independent). It probably would, but LSU was not quite prepared to deploy it yet.

There was discussion about what folks were doing about measuring IPv4 vs IPv6 traffic. Comcast went into some more detail about their work. I showed some data from Penn, where over the summer we were approaching 11% inbound and 4% outbound traffic composed of IPv6. We've since dropped down substantially, because we had to take IPv6 off the wireless network (I'm planning to write another blog article detailing why, but the quick summary is that deploying v6 broke IP address mobility, and we're working with our wireless equipment vendor on some possible solutions).

Dan Magorian (Johns Hopkins) asked what cable modem data specifications support IPv6 and what is the state of IPv6 support in the currently deployed field. John's answer is that DOCSIS 3, the current spec supports it pretty completely, almost all current equipment implements it, but there are ways to support IPv6 on older specs too.

---

Addendum: John B was one of the recipients of the Internet Society's Itojun Service Award recently, and presented at IETF'85, for his "for his tireless efforts in providing IPv6 connectivity to cable broadband users across North America and evangelizing the importance of IPv6 deployment globally". Congratulations John!

Shumon Huque

IPv6 panel at Internet2 meeting

Penn is co-hosting the Fall 2012 Internet2 Member meeting in Philadelphia, Oct 1st through the 4th. I'm moderating an IPv6 Deployment Panel at the conference (October 2nd, 1:15pm-2:30pm). Joining me will be John Brzozowski from Comcast, Allie Hopkins from Tulane University, Eric Kenny from Marist College, and Mark Kosters from the American Registry for Internet Numbers (ARIN).

The description says: "Several panelists will provide an update on IPv6 deployment activity and plans at their respective organizations, including both network infrastructure and application services. Other topics might include IPv6 security issues, network monitoring, technical support and training, etc"

John Brzozowski is Comcast's chief IPv6 architect. Comcast is one of the industry's leading adopters of IPv6, and I hope that John will share the latest news about IPv6 developments at Comcast. Mark Kosters is the Chief Technology Officer for ARIN. Allie Hopkins is an IT director at Tulane, but was formerly at Louisiana State University. I expect that Allie will be able to talk about the state of IPv6 deployment at LSU. As you may recall, LSU was on the list of the top IPv6 traffic generating sites during World IPv6 Launch. Eric Kenny is a network engineer from Marist College, which also made that list. I'll meet Eric for the first time at the conference. I've known the other panelists for a while.

In addition to deployment details, some combination of us will try to do a little bit of IPv6 evangelism. I'll also be asking Mark to do the usual ARIN update on the state of IPv4 address depletion.

I hope to see some of you at the conference. The panel session will also be netcast (and most likely archived video will be available to view later). If anyone has suggestions on specific IPv6 related topics the panel should discuss or comment on, feel free to let me know.

References:
* IPv6 at Penn
* IPv6 at LSU
* Comcast IPv6 Information Center
* ARIN: IPv4/IPv6 the bottom line
* ARIN: IPv6 Wiki