Wednesday, July 30, 2014

Key Transparency for DNSSEC?

At the recent IETF meeting in Toronto, there was an interesting discussion in the trans working group on DNSSEC certificate transparency, and there is a (very) preliminary IETF draft (that needs a lot more work):

    http://tools.ietf.org/html/draft-zhang-ct-dnssec-trans

This isn't a new topic. It has been talked about off and on for a number of years. The first time I ran across this was on the "The Right Key" email list in 2012 when measures to detect and counter fraudulently issued PKI (X.509) certificates were being proposed. This ultimately led to the creation of the trans working group, whose main goal is to produce and standardize a transparency system for X.509 certificates, based on the mechanism described in RFC 6962.

Does DNSSEC need a similar transparency system? For X.509 certificates, the threats are well known and documented (I wrote about them a bit in an earlier blog article) - any of the many root certificate authorities, or their intermediates, are capable of issuing a certificate for anyone on the Internet, and it is virtually impossible to know for sure if a fraudulent certificate has been issued. A central, cryptographically verifiable audit log of issued certificates might be able to address this issue (assuming that all CAs participated in it, which is by no means a certain proposition).

For DNSSEC it isn't obvious that a similar mechanism is needed, and whenever this topic comes up, there is a lot of head scratching and bewildered looks from DNS engineers wondering what the possible threat model is. I must admit, I didn't foresee the possible attack, until it was described to me in this exchange on The Right Key list by Paul Hoffman and Ben Laurie:

    http://www.ietf.org/mail-archive/web/therightkey/current/msg00470.html

If you'd like to read the entire email thread, it starts here:

    http://www.ietf.org/mail-archive/web/therightkey/current/msg00452.html

In DNSSEC, the keys for a zone are vouched for by a single parent zone (by means of a signed DS record corresponding to the child's keys). A zone operator can query the parent's DS records himself to verify that the correct DS key is being returned. However, what the zone operator cannot do, is to verify that the parent domain is not selectively responding with false DS records to queries from a targeted set of other victims.

To quote from that thread,

"For example, assume the domain name example.newtld. The owner of example has put DS record A in the newtld zone. If the owner of newtld goes rogue and shows DS record B to a limited number of requests (such as to a particular geographic region or set of network addresses), the party with the private key associated with B can spoof example, and the owner of example would not know unless he could see B."

(Note: for this attack to be useful, in addition to showing the fake DS records, newtld would have to show fake NS records and possibly glue records to redirect the victim to alternate DNS servers with the corresponding DNSKEY records)

If the parent zone is doing this on wide scale, they are likely to get caught and face action. But highly targeted attacks will be very hard to detect. Are these scenarios far fetched? Most DNS top level domains are run by reputable organizations and would likely not risk engaging in such security shenanigans. However, even if we assume they are completely forthright, they are vulnerable to "compelled" attacks by government agencies. In April 2010, Chris Soghoian and Sid Stamm in a paper ("Certified Lies: Detecting and Defeating Government Interception Attacks against SSL") describe such attacks against SSL/TLS certificates with evidence suggesting that they are actually in use. In light of Edward Snowden's NSA revelations, these kinds of compelled attacks (legally compelled or otherwise) are more likely than ever. DNSSEC keys and delegation signer records are just as vulnerable to them.

The title of the IETF draft mentioned at the beginning of this article is "Certificate Transparency for DNSSEC", which is probably a misnomer. The data that would be most valuable to enter into a DNSSEC transparency log are not certificates, but secure entry point keys for zones (e.g. DS records and/or Key Signing Keys). So a more appropriate name might be "Key Transparency for DNSSEC". DNS zones can contain certificates, or hashes corresponding to certificates (e.g. TLSA, CERT, etc records), however there are many other record types that might contain cryptographic keying material (SSHFP, IPSECKEY, and proposals for others in the pipeline). And why not have an audit log for more mundane non-crypto records too, eg. name to address mappings? Individually logging all of these data types for every zone will likely prove to be an infeasible task.

If we limit the scope of the transparency log to DS records, there are still some very significant technical challenges that need to be solved. One is scalability to the world wide DNS. The Certificate Transparency log is being implemented as an append-only log with a Merkle tree, and a DNSSEC log will likely follow the same approach. Quoting RFC 6962:

"The append-only property of each log is technically achieved using Merkle Trees, which can be used to show that any particular version of the log is a superset of any particular previous version. Likewise, Merkle Trees avoid the need to blindly trust logs: if a log attempts to show different things to different people, this can be efficiently detected by comparing tree roots and consistency proofs. Similarly, other misbehaviors of any log (e.g., issuing signed timestamps for certificates they then don't log) can be efficiently detected and proved to the world at large."

Even though DNSSEC is still in a fledgeling state of deployment, we need to design a mechanism that can scale to the entire DNS system and accommodate the expected churn of zone keys (e.g. due to key rollovers etc). A single centralized log may not be able to do this, and alternative models may need to be considered (e.g. limiting the depth of zones that the log will hold; implementing a hierarchy of logs, etc).

Another problem is that a rogue or compelled parent zone can not only return fake DS records, but could also answer authoritatively for names inside the child zone without any referrals to the child zone (a fake in-zone answer). This is harder to protect against, but I can think of a number of possible defenses. At the level of TLDs (top level domains) one possible protection might be to have DNS resolvers treat them as delegation-only and reject all subdomain answers that aren't referrals. Another (probably more promising) approach is for resolvers to employ a query-name minimization algorithm that only reveals the needed labels of the query name to authoritative DNS servers as they traverse the delegation hierarchy. In fact, there is active work going on in the DNS engineering community on such qname minimization schemes and other privacy enhancing extensions to the DNS.

A DNSSEC transparency log, if deployed, could be useful as an audit channel to periodically detect attacks, and for forensics. Performing checks of the log inline with the DNS resolution process may not be practical because of the probably high performance penalty (with the currently proposed log structure), which means attacks could not be detected in real time.

So is it really worth deploying such a system? I'm not yet sure. In the end, DNS is a hierarchical system, and there is always the possibility of being victimized by your parent zone either by error, incompetence, malice, or coercion. Even if we deploy centralized sets of transparency logs, we'd have to them think about how to prevent them from being compromised or co-erced. There are decentralized (non-hierarchical) naming systems out there, like gnunet, namecoin, etc. But they have the usual problem that they are only really used by a small group of very technically savvy users. We should probably take a more serious look at them. But I think the compelled attack is a very real threat, and it's probably worth some serious thought about how to deploy practical defenses against it in the global DNS.

The IETF recently published RFC 7258, declaring that pervasive monitoring is an attack for which all IETF protocols should have technical countermeasures. It may also be time for the IETF to standardize mitigations for highly targeted attacks.

Shumon Huque

Friday, July 18, 2014

Attending IETF90 in Toronto

I'm attending the IETF 90 meeting in Toronto, Canada this coming week. You can read more about the IETF here.

On Sunday I'll be attending the ICANN RSSAC (Root Server System Advisory Committee) meeting. I was recently appointed to the RSSAC Caucus. The RSSAC advises the ICANN community and board on matters relating to the operation, administration, and integrity of the Internet's DNS Root Servers. During the week, I plan to attend IETF working group meetings in various areas (including DNS, Routing,  HTTP, IPv6, and security). I hope to write a report of activities at the meeting after I return.

Shumon Huque

Friday, May 2, 2014

HUQUE.COM has a signed delegation now

(Off topic: I've been at Verisign Labs over a month now - I'll write a separate post describing my experiences at my new job when I have more time. But the short summary is: no regrets so far; lots of interesting technical work I'm getting involved in; it was probably time for me to move on from Penn; and I think I made the right decision).

A few weeks ago, I created a set of test DNS TLSA records in my personal zone (huque.com) to help out some folks participating in the getdns-api hackathon. To correctly authenticate records in the zone, folks needed to have resolvers configured to use the ISC DLV registry, since I hadn't gotten around to setting up a secure delegation in the parent zone (.COM). This finally motivated me to get that done.

Why didn't I do this earlier? Long story, but I've used my own domain for a lot of experiments with DNSSEC configurations over the years (going back to 2007), including testing configurations that I eventually deployed in other larger, production domains. The bigger stumbling block is that the DNS registrar I was using, Network Solutions, does not support DNSSEC (specifically it provides no way for customers to transmit DNSKEY or DS record contents to the TLDs it registers domains in).

So last week, I started the process of changing to a DNSSEC capable registrar. There are a growing number of them out there, and ICANN maintains a list of them. The one I chose is GKG.net.

This was my first time switching registrars - Network Solutions has been my registrar since 1998 when I first registered the domain. At that time Network Solutions both operated the .com DNS top level domain (among others) and registered domains within it. In 2000, Verisign (where I now work) acquired Network Solutions to get into the DNS business, and in 2003 spun it off as a registrar-only business, retaining operation of the .com registry. Unfortunately, as far as I can tell, Network Solutions has announced no plans to support DNSSEC.

The whole process took about 5 days. On April 24th I logged into my network solutions account, unlocked my registration and requested an authorization code to transfer my domain registration. On the 25th I called them on the telephone to confirm the transfer intention and to expedite the delivery of the authorization code (it normally takes them 3 days), which I got shortly after. I then created an account at gkg.net, and initiated the domain transfer request with the supplied code. The charge was US $8.20, which included a 1-year registration period that extended beyond the current expiration date for my registration with Network Solutions. On the 28th, the transfer was successfully completed.

I then logged into the gkg.net account and used their DNSSEC interface to add a DS record for huque.com, which appeared in the .COM zone more or less immediately.

huque.com.      86400 IN DS 40924 8 2 (
                816524EB1C3B7D1315AE8330652DD17909C95DE0533C
                1F2DC023BFFEDB1F5E9B )
huque.com.      86400 IN RRSIG DS 8 2 86400 (
                20140509035959 20140502024959 56657 com.
                UkLdvaDKUcHmqAe8JQyXrxn+luWRKrkjfNzG4/xd/PXy
                zQr03L1ZXNzJHnVp7PZSau2UVfsfz5BmYGN5jepIScPc
                57zd/CnKXTZgucT9ry7dHvkdmxr+UCGY1Zg4LQ0pDyAY
                2avC9Hd2gJKBNJGWfZlGU/KHa1KvRv8fqlNWWQo= )


Below is a screen capture of their DNSSEC interface. You need to supply the parameters from your zone's DNSSEC key: the keytag, algorithm, digest type, and the digest string.


Here's a graphical view of the current chain of trust for my zone from the DNSviz tool. As you can see, there are two secure entry paths into my zone, one via the just created normal delegation path through .COM, and one via the DLV Registry at dlv.isc.org. I will be removing the DLV entries shortly.




Sunday, March 16, 2014

I've left Penn for a new job

After more than 20 years of working at Penn (University of Pennsylvania), I've decided to take a new job as Principal Research Scientist at Verisign Labs, the applied research division of Verisign Inc. You might know that Verisign is one of the world's largest DNS infrastructure providers. It runs the .com, .net, .edu, and .gov DNS top level domains, two of the thirteen DNS root servers (A and J), and performs the critically important root zone management function. Verisign also provides managed DNS, Distributed Denial of Service (DDoS) mitigation, and several other services. (Note: Verisign's certificate services business was sold off to Symantec several years ago).

I've been at Penn for so long, that I startled many of my colleagues with the news of my departure, so I'll say a few things about how this came about. I originally came to Penn in 1989 as an undergraduate. I became a full time IT staff member after I graduated - my first job was the system administrator of the new e-mail server for the School of Arts & Sciences, the largest school within Penn. I completed a Masters degree in Computer Science part-time while working. I later moved to the central IT organization, where I remained until last week, first as a Network Engineer, then as the Lead Engineer, and most recently as an Engineering Director. In addition, I've been a part time Adjunct Faculty in Penn's School of Engineering, teaching a laboratory course on Network Protocols.

I was approached by Allison Mankin and Matt Larson (I've known both of them for a while) about a year ago to see if I'd be interested in considering a research scientist position at Verisign Labs. Matt has since left Verisign to work at Dyn, and Allison is now a Director at Verisign Labs. At the time, I thought this was distinctly a long shot, but over the course of many months, I thought more seriously about the possibility. I visited Verisign Labs in September 2013, did an interview with them, and also met Burt Kaliski, Verisign CTO (Burt is a noted cryptographer and was the founding scientist of RSA Labs, and the developer of the PKCS standards). I kept in touch with Allison since, but it took me until the beginning of this year to finally come to the conclusion that I wanted to take this opportunity, so here I am. My first day at Verisign will be tomorrow (March 17th).

I've enjoyed my job and career at Penn a lot. I've been extensively involved in a very diverse range of technical projects, ranging from software development, systems engineering, and network design. Among others, I was responsible for the design and operation of much of the authentication and security infrastructure at Penn, as well as a variety of other services, like the DNS and DHCP (and many more that I don't have the time to enumerate here). I was the principal architect of IPv6 and DNSSEC deployment at Penn. I was the chief engineer of the MAGPI gigapop and through that role, was involved in R&E regional and national networking activities. Increasingly though, a lot of time is taken up by non-technical activities, and the longer I stay at Penn, the greater the possibility that I'll end up as a full time IT staff manager, which isn't the role I envision for myself. I've always seen my primary role as that of a technologist, and this job change will allow me to continue in that direction.

Verisign Labs appears to have a true applied research agenda, which I find appealing. Obviously DNS and DNSSEC research is an area in which I expect to be working. But there are many other interesting areas of work: routing, IPv6, reputation systems, security protocols, future internet architectures, etc. I'm also looking forward to attending more NANOG and IETF meetings to get myself more plugged into those communities than I've been able to thus far. Verisign Labs also has frequent, productive collaborations with computer science researchers through its university collobaration program.

In the process, I'm moving to the Washington DC metro area (specifically Reston, VA) where Verisign is located, another big change for me!

More later ..

--Shumon Huque.

Thursday, February 6, 2014

IPv6 versus IPv4 Performance

Yesterday from a post at the Deploy360 website, I learned of Comcast's IPv4 and IPv6 network speed testing tool:

        http://speedtest.comcast.net/

I did a quick test from my laptop in my office and got some very surprising results. The measured IPv6 performance was better than IPv4 by a gigantic margin. With IPv6, I got 822 Mbps download and 667Mbps download throughput. With IPv4, a mere 99Mbps upload and 18Mbps download!


Something seemed fishy, but I had to run off to other work, so I quickly posted the result to Twitter, planning to look into it later.


This generated quite a bit discussion with numerous folks on twitter and elsewhere. My initial speculation was that we do some rate limiting of IPv4 traffic at the Penn border routers for selected areas of the campus, and perhaps this was throttling the IPv4 performance. My other suspicion was that there was something significantly different in the IPv4 and IPv6 routing paths contributing to the difference. The graphic above does show a round-trip time difference of 63ms for the IPv4 path and 32ms for the IPv6 path, which suggests this. Furthermore, if the TCP window is not scaled properly to keep the pipe filled for this path at 63ms (but was for 32ms), then that would decrease throughput also - but not enough to account by itself for the observed difference.

Patrik Falstrom suspected a DPI device or other middlebox causing the problem. The only problem is that we don't have any such middleboxes (unless you consider an IP border router imposing IP address based rate limits a middlebox). In any case, I was leaning towards the rate limits as the cause myself, until I confirmed that those rate limits weren't being applied to any of the traffic from my office network. The rate limits are primarily targeted at the student residential dormitories - without them, our external links typically get overwhelmed with traffic to/from the dorms (most likely due to file sharing, a very common activity on college campuses). The border routers are configured to apply a token bucket rate policer to each individual IPv4 address within the network prefixes that cover the residential networks. Note that this rate limiting is completely application agnostic.  Also note that this scheme cannot scale to IPv6 (a single IPv6 subnet has more than 18 quintillion addresses!), a problem we're ignoring for the time being :-)

Repeat of the test


This morning, I decided to do another test (same laptop), but more carefully, and along with a packet capture. I also explicitly turned off the wireless interface (hmmmm) to make sure that all tests were using the wired gigabit ethernet interface. This time, I got much more reasonable looking results, both address families in the neighborhood of each other: IPv4 853Mbps down, 547Mbps up, and for IPv6 827Mbps down, 730Mbps up. One other difference I notice is that the roundtrip (ping) times to the destination server are 12ms for both IPv4 and IPv6. This is substantially different from yesterday's test (63 and 32ms respectively) despite the fact that I choose the same destination server at Comcast (Washington, DC).


A packet capture reveals that the destination server at Comcast for IPv4 was 68.87.73.52, and for IPv6 was 2001:558:1010:5:68:87:73:52. Are these the same endpoint? Hard to tell, but the fact that the last 4 fields of the IPv6 address spell out the IPv4 address in decimal might be a hint. The traffic streams use TCP port 5050. A traceroute to the IPv4 destination shows the outbound path takes one of Penn's commercial ISP links (Cogent) to New York and then back to Washington/VA. An IPv6 traceroute shows the outbound path goes out via our Internet2 link, the I2 commercial peering service, then Cogent (New York), Level3 (New York), and then Comcast to DC. So the IPv4 and IPv6 paths are substantially different in the forward direction. Harder to tell the path for the return traffic without the aid of some reverse traceroute tools or similar.

Getting a substantial fraction of a gigabit ethernet is not suprising - that's probably the bottleneck bandwidth along the measured path. My laptop has a gigabit ethernet connection to the building network, which in turn has dual 10 Gigabit Ethernet links to a 100 Gig campus core, and then multiple 10Gig links out to commercial ISPs/Internet2 etc. Most tier-1 ISP links and peerings are typically at least 10Gig.

The bandwidth-delay product on these paths is about 1,464 KB (1000Mbps * 12ms). The Comcast endpoint's receive window exceeds this, but my laptop's is slightly undersized, so I could probably do a bit of host tuning to boost the download numbers a bit more.

So, what's the explanation for the strange results I got yesterday? I wish had a packet capture to investigate, but my leading suspicion is that my laptop's wireless adapter (lower bandwidth, shared medium) was used in the IPv4 test, and the wired connection for the IPv6 one. If I have time later, I'll try to reproduce the issue.

--Shumon Huque



Addendum (February 9th 2014) - On closer inspection of the packet trace, the speed test appears to use multiple TCP streams in parallel, so scaling the window as high as the bw*delay product of the path isn't necessary.

Saturday, February 1, 2014

DNSSEC/DANE/TLSA Browser Add-ons

The folks at CZ.NIC (the operators of the Czech Republic's country-code top level domain: .cz) have created a set of web browser add-ons to perform DNSSEC/DANE/TLSA validation. You can read about them and download them from their website:

        https://www.dnssec-validator.cz/

I installed the Firefox web browser plugin and did some quick tests of them on my own website. The plugin installs two new icons on the right side of the browser's location (URL) bar. The first with a key on it shows information about whether the domain name for the website has a valid DNSSEC signature associated with it. The second icon with a lock on it shows information about whether the TLS certificate of the website can be authenticated with a DANE TLSA record. Here are screenshots with my own website ( https://www.huque.com/ ).

In this first screenshot (below), I clicked on the key icon, and it reports that the 'www.huque.com'  domain name has a valid DNSSEC signature.


In the next screenshow (below), I clicked on the lock icon, and it reports that the certificate for www.huque.com has been successfully authenticated by means of a signed TLSA record.


In this case, since this is an HTTPS connection at the standard port (TCP port 443), the plugin looked for the TLSA record at the domain name "_443._tcp.www.huque.com."

    $ dig _443._tcp.www.huque.com. TLSA +noall +answer
   [...]
    _443._tcp.www.huque.com. 7200 IN TLSA 3 0 1 (
                    7EF4BD014E9A4F302FC1EE74FB2D29718C5B0F4CB23B
                    25B267A1D92F0410890B )
    _443._tcp.www.huque.com. 7200 IN RRSIG TLSA 8 5 7200 (
                    20140217205026 20140118205010 14703 huque.com.
                    NsUKFsBAUD4OxrHQ72iB0Oz9mBoMEqL8wMsks56sp2yz
                    3ksXcqGSddooC3jZvGH/4iF6ssD3KRNQVONJqpK246nX
                    jPhxBhM730TKEwMZRw/NRqYanRKyEMhkUy538suej0Pv
                    rK3w8r6tdNF4gXqIM3sQlz9gPY/WOu0zxjezaIk= )


Below is another screenshot for https://www.ietf.org/. In this case, the second icon has a cross marked on it, meaning that no TLSA record was found for this site. Apparently, the IETF is not yet eating its own dogfood. Although see this short slide deck from IETF'87 - there appears to be a proposal to do so.



There are a few configuration options that can be set for the add-on. Here is a view of the settings window:


The plug-in appears to do its own DNS resolution (and validation) by default. But you can also choose to use DNS resolvers configured for your system, or a customer resolver such as the Google public resolver (8.8.8.8).

If you need help creating a TLSA record for your website, I have a web based tool available here:

    https://www.huque.com/bin/gen_tlsa

One thing I should mention, in case you're looking at the configuration of my website: huque.com does not today have a secure delegation (i.e. DS record) published in its parent zone. This is because the registrar I use, Network Solutions, still cannot process requests to install DS records. I did quick check on their website (again) to see if anything's changed. Doesn't appear so:


Instead, I've had DLV record published in the ISC DLV Registry. But there are several big resolver services, like Google DNS, and Comcast, that do not perform lookaside validation, so it's probably time to switch registrars. If anyone has suggestions for competent DNSSEC enabled registrars (with registrar-lock support), I'd be happy to receive them. I hope to make the switch soon.

Dan York from ISOC also has an article on these addons here. (I started writing this before seeing his!)

--Shumon Huque


Addendum (May 2014): my domain huque.com now has a secure delegation from .COM.

Wednesday, January 22, 2014

An IPv6 Success Story (Galois)

The following article was contributed by Paul Heinlein, a systems administrator at Galois. Paul attended my full day IPv6 training course at USENIX LISA and just a couple of months later sent me a report of his successful deployment of IPv6. So I asked if he'd like to contribute an article on the topic.

It's great to hear of IPv6 success stories like this. (And of course, I'm glad folks are finding my courses useful). Significantly, his network is already seeing a very substantial amount of IPv6 traffic!

(A note: the version of iptables in Redhat Enterprise Linux6/CentOS 6 has fixed the stateful IPv6 inspection capability. We use it successfully at Penn).

You can find the slides from my IPv6 training course on my website.

-Shumon Huque



An IPv6 Success Story

Paul Heinlein


Galois is a software-engineering firm located in Portland, OR that specializes in the hard problems of computing trust and assurance.

One of our IT goals for 2014 is enabling IPv6 on all our current IPv4 subnets. I was pleased to see Shumon's full-day IPv6 tutorial on the LISA '13 schedule. I hoped he could fill the gaps in my limited
experience with IPv6 and provide me the knowledge I'd need to configure our network hardware and applications.

Full-day technical tutorials can sometimes test my patience, but Shumon's presentation moved quickly while remaining clear. I came away from the day thinking that I had enough basic knowledge to plan and implement our IPv6 rollout.

Returning to Portland, I had to upgrade upgrade a couple core switches and apply to our ISP for a netblock prior to enabling it on our network, but once the hardware was in place, everything went reasonably well.

Shumon's overview had done exactly what I'd hoped it would: allow me to interpret the various bits of vendor-specific and application-specific documentation and assemble a reasonable roll-out plan.

Our DMZ and client networks are now fully IPv6 enabled. (Internal development networks will take a bit more time due to VPN-related complexity.)

Along the way, I learned a couple things Shumon didn't explicitly cover in his presentation.

First, make sure that reverse DNS pointers are in place for any mail server before enabling IPv6. gmail (among others) will reject messages from any mail server without reverse DNS pointers in place.

Second, the ip6tables that ships with RHEL/CentOS 5 has a limited ability to do stateful packet inspection. The generic rule allowing packets from established or related sessions does not work:

   # this is broken in RHEL/CentOS 5
   -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

(We've only got one CentOS 5 machine in our DMZ, so this issue hasn't impacted us in any major way.)

Third, it was a lot of fun! Nearly all our DMZ services (NTP, DNS, Jabber, e-mail, www, ssh, host-based firewalls) needed updated configurations, so I learned a bunch. The day after I did the initial rollout, one of our engineers came into the office, turned off IPv4 on his Mac, and tested what percentage of Google result links he could follow. (He thought about 5%, though he said the queries he chose probably resulted in a higher success rate than might be normal.)

Finally, I've been mildly surprised at the quantity of IPv6 packets traversing our border firewall. Over the past week, IPv6 has comprised 38% of overall inbound traffic and 11% of outbound. The numbers are similar when scoped to the past month.

I'd like to express my thanks to Shumon for the talk and the LISA organizers for putting it on the schedule!

  -- Paul Heinlein <heinlein@galois.com>

Friday, January 17, 2014

World IPv6 Launch Measurements

From the Internet Society's most recent (January 16th 2014) round of IPv6 measurements, here again are the top 100 lists ranked by the two measures of (1) percentage of requests that used IPv6, and (2) total volume of IPv6 requests.

My past articles on this topic:
  * Measurements from December 12th 2013
  * Measurements from  September 17th 2013
  * Measurements from June 6th 2012

ISOC's blog post highlights that Deutsche Telekom, a large German telecommunications carrier, (of which T-Mobile US is a subsidiary*) has been experiencing very substantial IPv6 traffic growth. They've reached a figure of 15.5% of requests composed of IPv6 (to the select set of content providers providing measurement data). In terms of total volume of IPv6 requests, they are in 6th place behind five other large ISPs (See the second ranked list below). Comcast still leads the pack, followed by AT&T, KDDI, Free, and Verizon Wireless.

(*Note: T-Mobile's traffic is counted separately in the measurements from Deutsche Telekom)

Interestingly, while Verizon Wireless has a substantial IPv6 deployment, Verizon FIOS (their fiber-optic landline network) has made no visible progress in IPv6 deployment. Although many people, myself included, have noted this lack of deployment over the years, they were just recently taken sharply to task by folks on the NANOG (North American Network Operators Group) mailing list.

Looking at the rankings by proportion of IPv6 requests from each network, there is a new entry at the top - NYSERNet (a regional R&E network in New York State), coming in at a staggering 99.95%. On the Internet2 IPv6 working group list, I asked Bill Owens of NYSERNet if he wanted to comment. He claims that we shouldn't be too impressed because this is just the small NYSERNet office network and the backbone, with a relatively small population of clients. Regardless, I think it's a noteworthy achievement. As I've mentioned before, a number of universities and other educational organizations still show up prominently in this ranking. Two small american colleges, Gustavus Adolphus in Minnesota at 74.22% and Marist College in upstate New York at 65.77%! And Panamerican University, in Mexico City is at 66.22%.

Penn is slowly inching up at 45%. We'll have IPv6 deployed more extensively on our wireless network pretty soon, at which point I expect our numbers to go up noticeably.


World IPv6 Launch Measurements, by % IPv6 requests:
---------------------------------------------------

   1 NYSERNet 99.95%
   2 SpeedPartner GmbH 95.83%
   3 TOP-IX Consortium 86.99%
   4 Fundacao Parque Tecnologico Itaipu - Brasil 79.92%
   5 DirectVPS 78.63%
   6 interscholz Internet Services GmbH & Co. KG 75.49%
   7 Gustavus Adolphus College 74.22%
   8 Google Fiber 71.87%
   9 Universidad Panamericana 66.22%
  10 Marist College 65.77%
  11 University of Vermont 65.13%
  12 Virginia Tech 65.11%
  13 mur.at - Verein zur Frderung von Netzkwerkkunst 64.97%
  14 Association tetaneutral.net 55.67%
  15 Utility Line Italia srl 53.40%
  16 ThaiSarn 53.23%
  17 DegNet GmbH 52.84%
  18 Trunk Networks Limited 52.60%
  19 Ji?? ?trohalm 52.20%
  20 Alhambra Eidos 52.12%
  21 Louisiana State University 51.93%
  22 Ponto de Presen?a da RNP na Bahia 51.30%
  23 SuperInternet Access Pte Ltd 50.45%
  24 University of New Hampshire 50.44%
  25 Region 7 ESC 50.24%
  26 PREGINET 50.20%
  27 CICA/Junta de Andaluc?a 50.05%
  28 Tanzania Network Information Center 50.00%
  29 Karlsruhe Institute of Technology (KIT) 49.03%
  30 Maxiweb Internet Provider 48.76%
  31 Critical Colocation 48.23%
  32 University of Pennsylvania 45.38%
  33 SPAWAR 44.15%
  34 AMS-IX 42.99%
  35 University of Iowa 42.81%
  36 Kasetsart University 41.99%
  37 DreamHost 41.79%
  38 University of Minnesota 40.80%
  39 Rensselaer Polytechnic Institute 40.74%
  40 Bulgaria NREN 40.57%
  41 Verizon Wireless 40.03%
  42 CZ.NIC 39.02%
  43 guifi.net 37.86%
  44 Hughes Network Systems 35.60%
  45 Sauk Valley Community College 35.52%
  46 Tulane University 34.96%
  47 Free 34.28%
  48 ARNES 33.73%
  49 Greek Research & Technology Network 33.14%
  50 Leibniz Supercomputing Centre 32.71%
  51 Host Virtual, Inc 31.74%
  52 Netwerkvereniging Coloclue 28.41%
  53 Opera Software ASA 27.54%
  54 UNESP 27.26%
  55 UNINETT 27.13%
  56 FCCN 26.00%
  57 VOO 24.12%
  58 RCS & RDS 23.80%
  59 mc.net 23.37%
  60 UFSCar 22.96%
  61 EPT Luxembourg 22.43%
  62 FranTech Solutions 22.03%
  63 AAISP 21.22%
  64 Comcast 20.61%
  65 Chubu Telecommunications 18.90%
  66 Swisscom 18.66%
  67 XS4ALL 17.66%
  68 UFSC - Universidade Federal de Santa Catarina - Brazil 17.50%
  69 LENTEL 16.73%
  70 StarHub 16.28%
  71 Deutsche Telekom AG 15.50%
  72 NIIF/Hungarnet 14.56%
  73 LITNET 14.50%
  74 Red Acad?mica de Centros de Investigaci?n y Universidades Nacionales REACCIUN 13.85%
  75 Indiana University 13.68%
  76 ATT 13.53%
  77 SARENET 13.13%
  78 DMZGlobal 12.97%
  79 NetAssist 12.89%
  80 Academia Sinica Network 12.14%
  81 Cisco 11.95%
  82 Solcon 11.46%
  83 CESNET 11.04%
  84 Funet 10.94%
  85 PT. Wifian Solution 10.19%
  86 FidoNet 10.12%
  87 prgmr.com 9.89%
  88 T-Mobile USA 9.58%
  89 Monash University 9.28%
  90 OVH 9.26%
  91 KDDI 8.94%
  92 Spectrum Networks 8.30%
  93 AMRES - Serbian National Research and Education Network 7.84%
  94 Belnet 7.65%
  95 SWITCH 7.62%
  96 Hurricane Electric 7.58%
  97 M1 Limited 7.22%
  98 RedIRIS 6.96%
  99 Init7 6.93%
 100 CJSC Progressive Technologies 6.71%


World IPv6 Launch Measurements, by volume of IPv6:
--------------------------------------------------


   1 Comcast 20.61%
   2 ATT 13.53%
   3 KDDI 8.94%
   4 Free 34.28%
   5 Verizon Wireless 40.03%
   6 Deutsche Telekom AG 15.50%
   7 Time Warner Cable 3.88%
   8 RCS & RDS 23.80%
   9 Liberty Global 2.43%
  10 Swisscom 18.66%
  11 Telefonica del Peru 4.60%
  12 Hughes Network Systems 35.60%
  13 SoftBank BB 1.46%
  14 Chubu Telecommunications 18.90%
  15 Opera Software ASA 27.54%
  16 VOO 24.12%
  17 XS4ALL 17.66%
  18 StarHub 16.28%
  19 T-Mobile USA 9.58%
  20 Google Fiber 71.87%
  21 China Telecom 0.23%
  22 Forthnet 2.79%
  23 M1 Limited 7.22%
  24 Internode 3.94%
  25 EPT Luxembourg 22.43%
  26 Janet 3.81%
  27 its communications Inc.(iTSCOM) 2.64%
  28 CESNET 11.04%
  29 MediaCat Div./Community Netowork Center Inc. 6.58%
  30 Kasetsart University 41.99%
  31 NTT Communications 4.51%
  32 Belnet 7.65%
  33 ARNES 33.73%
  34 Leibniz Supercomputing Centre 32.71%
  35 LITNET 14.50%
  36 NIIF/Hungarnet 14.56%
  37 OVH 9.26%
  38 Cisco 11.95%
  39 BelWue 5.71%
  40 FCCN 26.00%
  41 UNINETT 27.13%
  42 Altibox AS 0.90%
  43 Monash University 9.28%
  44 SWITCH 7.62%
  45 TUBITAK ULAKBIM / ULAKNET 1.50%
  46 Australian Academic and Research Network (AARNet) 2.00%
  47 Indiana University 13.68%
  48 University of Minnesota 40.80%
  49 AAISP 21.22%
  50 UniNet 3.36%
  51 Xfone 018 0.96%
  52 Ji?? ?trohalm 52.20%
  53 Voxel / Internap 3.51%
  54 GITN Sdn Berhad 2.69%
  55 Louisiana State University 51.93%
  56 University of Pennsylvania 45.38%
  57 SuperCSI 1.69%
  58 JARING Communications Sdn Bhd 0.38%
  59 Solcon 11.46%
  60 CJSC Progressive Technologies 6.71%
  61 Starlink 1.88%
  62 inexio KGaA 2.44%
  63 Virginia Tech 65.11%
  64 green.ch AG 3.26%
  65 Hurricane Electric 7.58%
  66 Dhiraagu 0.60%
  67 Gustavus Adolphus College 74.22%
  68 LENTEL 16.73%
  69 DMZGlobal 12.97%
  70 DegNet GmbH 52.84%
  71 RedIRIS 6.96%
  72 Academia Sinica Network 12.14%
  73 GlobalConnect 1.44%
  74 University of Iowa 42.81%
  75 Funet 10.94%
  76 University of Wisconsin - Madison 6.16%
  77 Storm Internet 3.76%
  78 AMRES - Serbian National Research and Education Network 7.84%
  79 National Informatics Centre 2.32%
  80 SoftLayer Technologies 0.84%
  81 RENATER 4.57%
  82 Karlsruhe Institute of Technology (KIT) 49.03%
  83 guifi.net 37.86%
  84 Init7 6.93%
  85 Host Virtual, Inc 31.74%
  86 Rensselaer Polytechnic Institute 40.74%
  87 Choopa, LLC 0.55%
  88 NetAssist 12.89%
  89 Tulane University 34.96%
  90 DreamHost 41.79%
  91 Bulgaria NREN 40.57%
  92 FranTech Solutions 22.03%
  93 Greek Student Network 0.47%
  94 RESTENA 5.33%
  95 UNESP 27.26%
  96 iway AG 4.06%
  97 FX Networks 0.49%
  98 Tanzania Network Information Center 50.00%
  99 edpnet 1.11%
 100 ADDIX Internet Services 3.05%