Thursday, February 6, 2014

IPv6 versus IPv4 Performance

Yesterday from a post at the Deploy360 website, I learned of Comcast's IPv4 and IPv6 network speed testing tool:

        http://speedtest.comcast.net/

I did a quick test from my laptop in my office and got some very surprising results. The measured IPv6 performance was better than IPv4 by a gigantic margin. With IPv6, I got 822 Mbps download and 667Mbps download throughput. With IPv4, a mere 99Mbps upload and 18Mbps download!


Something seemed fishy, but I had to run off to other work, so I quickly posted the result to Twitter, planning to look into it later.


This generated quite a bit discussion with numerous folks on twitter and elsewhere. My initial speculation was that we do some rate limiting of IPv4 traffic at the Penn border routers for selected areas of the campus, and perhaps this was throttling the IPv4 performance. My other suspicion was that there was something significantly different in the IPv4 and IPv6 routing paths contributing to the difference. The graphic above does show a round-trip time difference of 63ms for the IPv4 path and 32ms for the IPv6 path, which suggests this. Furthermore, if the TCP window is not scaled properly to keep the pipe filled for this path at 63ms (but was for 32ms), then that would decrease throughput also - but not enough to account by itself for the observed difference.

Patrik Falstrom suspected a DPI device or other middlebox causing the problem. The only problem is that we don't have any such middleboxes (unless you consider an IP border router imposing IP address based rate limits a middlebox). In any case, I was leaning towards the rate limits as the cause myself, until I confirmed that those rate limits weren't being applied to any of the traffic from my office network. The rate limits are primarily targeted at the student residential dormitories - without them, our external links typically get overwhelmed with traffic to/from the dorms (most likely due to file sharing, a very common activity on college campuses). The border routers are configured to apply a token bucket rate policer to each individual IPv4 address within the network prefixes that cover the residential networks. Note that this rate limiting is completely application agnostic.  Also note that this scheme cannot scale to IPv6 (a single IPv6 subnet has more than 18 quintillion addresses!), a problem we're ignoring for the time being :-)

Repeat of the test


This morning, I decided to do another test (same laptop), but more carefully, and along with a packet capture. I also explicitly turned off the wireless interface (hmmmm) to make sure that all tests were using the wired gigabit ethernet interface. This time, I got much more reasonable looking results, both address families in the neighborhood of each other: IPv4 853Mbps down, 547Mbps up, and for IPv6 827Mbps down, 730Mbps up. One other difference I notice is that the roundtrip (ping) times to the destination server are 12ms for both IPv4 and IPv6. This is substantially different from yesterday's test (63 and 32ms respectively) despite the fact that I choose the same destination server at Comcast (Washington, DC).


A packet capture reveals that the destination server at Comcast for IPv4 was 68.87.73.52, and for IPv6 was 2001:558:1010:5:68:87:73:52. Are these the same endpoint? Hard to tell, but the fact that the last 4 fields of the IPv6 address spell out the IPv4 address in decimal might be a hint. The traffic streams use TCP port 5050. A traceroute to the IPv4 destination shows the outbound path takes one of Penn's commercial ISP links (Cogent) to New York and then back to Washington/VA. An IPv6 traceroute shows the outbound path goes out via our Internet2 link, the I2 commercial peering service, then Cogent (New York), Level3 (New York), and then Comcast to DC. So the IPv4 and IPv6 paths are substantially different in the forward direction. Harder to tell the path for the return traffic without the aid of some reverse traceroute tools or similar.

Getting a substantial fraction of a gigabit ethernet is not suprising - that's probably the bottleneck bandwidth along the measured path. My laptop has a gigabit ethernet connection to the building network, which in turn has dual 10 Gigabit Ethernet links to a 100 Gig campus core, and then multiple 10Gig links out to commercial ISPs/Internet2 etc. Most tier-1 ISP links and peerings are typically at least 10Gig.

The bandwidth-delay product on these paths is about 1,464 KB (1000Mbps * 12ms). The Comcast endpoint's receive window exceeds this, but my laptop's is slightly undersized, so I could probably do a bit of host tuning to boost the download numbers a bit more.

So, what's the explanation for the strange results I got yesterday? I wish had a packet capture to investigate, but my leading suspicion is that my laptop's wireless adapter (lower bandwidth, shared medium) was used in the IPv4 test, and the wired connection for the IPv6 one. If I have time later, I'll try to reproduce the issue.

--Shumon Huque



Addendum (February 9th 2014) - On closer inspection of the packet trace, the speed test appears to use multiple TCP streams in parallel, so scaling the window as high as the bw*delay product of the path isn't necessary.

Saturday, February 1, 2014

DNSSEC/DANE/TLSA Browser Add-ons

The folks at CZ.NIC (the operators of the Czech Republic's country-code top level domain: .cz) have created a set of web browser add-ons to perform DNSSEC/DANE/TLSA validation. You can read about them and download them from their website:

        https://www.dnssec-validator.cz/

I installed the Firefox web browser plugin and did some quick tests of them on my own website. The plugin installs two new icons on the right side of the browser's location (URL) bar. The first with a key on it shows information about whether the domain name for the website has a valid DNSSEC signature associated with it. The second icon with a lock on it shows information about whether the TLS certificate of the website can be authenticated with a DANE TLSA record. Here are screenshots with my own website ( https://www.huque.com/ ).

In this first screenshot (below), I clicked on the key icon, and it reports that the 'www.huque.com'  domain name has a valid DNSSEC signature.


In the next screenshow (below), I clicked on the lock icon, and it reports that the certificate for www.huque.com has been successfully authenticated by means of a signed TLSA record.


In this case, since this is an HTTPS connection at the standard port (TCP port 443), the plugin looked for the TLSA record at the domain name "_443._tcp.www.huque.com."

    $ dig _443._tcp.www.huque.com. TLSA +noall +answer
   [...]
    _443._tcp.www.huque.com. 7200 IN TLSA 3 0 1 (
                    7EF4BD014E9A4F302FC1EE74FB2D29718C5B0F4CB23B
                    25B267A1D92F0410890B )
    _443._tcp.www.huque.com. 7200 IN RRSIG TLSA 8 5 7200 (
                    20140217205026 20140118205010 14703 huque.com.
                    NsUKFsBAUD4OxrHQ72iB0Oz9mBoMEqL8wMsks56sp2yz
                    3ksXcqGSddooC3jZvGH/4iF6ssD3KRNQVONJqpK246nX
                    jPhxBhM730TKEwMZRw/NRqYanRKyEMhkUy538suej0Pv
                    rK3w8r6tdNF4gXqIM3sQlz9gPY/WOu0zxjezaIk= )


Below is another screenshot for https://www.ietf.org/. In this case, the second icon has a cross marked on it, meaning that no TLSA record was found for this site. Apparently, the IETF is not yet eating its own dogfood. Although see this short slide deck from IETF'87 - there appears to be a proposal to do so.



There are a few configuration options that can be set for the add-on. Here is a view of the settings window:


The plug-in appears to do its own DNS resolution (and validation) by default. But you can also choose to use DNS resolvers configured for your system, or a customer resolver such as the Google public resolver (8.8.8.8).

If you need help creating a TLSA record for your website, I have a web based tool available here:

    https://www.huque.com/bin/gen_tlsa

One thing I should mention, in case you're looking at the configuration of my website: huque.com does not today have a secure delegation (i.e. DS record) published in its parent zone. This is because the registrar I use, Network Solutions, still cannot process requests to install DS records. I did quick check on their website (again) to see if anything's changed. Doesn't appear so:


Instead, I've had DLV record published in the ISC DLV Registry. But there are several big resolver services, like Google DNS, and Comcast, that do not perform lookaside validation, so it's probably time to switch registrars. If anyone has suggestions for competent DNSSEC enabled registrars (with registrar-lock support), I'd be happy to receive them. I hope to make the switch soon.

Dan York from ISOC also has an article on these addons here. (I started writing this before seeing his!)

--Shumon Huque


Addendum (May 2014): my domain huque.com now has a secure delegation from .COM.