DNS Amplification Attacks

There has been a lot of talk recently about DNS amplification attacks (with prominent news reports of high bandwidth attacks targeted at anti-spam services, cloud providers, financial institutions, etc). These are a class of denial of service attack that use DNS servers to emit large amounts of traffic onto unsuspecting victims. The attackers use the forged source addresses of their victims to send a large stream of queries to the DNS servers, and this results in a much larger stream of responses being sent from those DNS servers back to the victim computers - with the aim of overwhelming them or their network connections. Although the DNS servers employed in the attack can become adversely impacted, the primary target of the attacks are the computers whose addresses are being forged.

Attacks that employ IP source address forgery are effective on the Internet today, because the countermeasures that would prevent this forgery are not very widely deployed. BCP 38 (a 13 year old document!) and BCP 84 describe network ingress filtering as a way for ISPs to block forged traffic from their customer networks, but many ISPs fail to employ it. Organizations should also ideally configure their networks not to permit internally generated traffic with forged addresses from crossing their borders - typically a border Access Control List or more granular techniques like unicast reverse path forwarding (URPF, a per-subnet level antispoofing method) can be used to do this, but once again, these are not in very widespread use.

In the past, the attacks commonly employed 'open recursive DNS resolvers' - these are DNS recursive resolvers that will resolve names for any client computer without restriction. When such servers are available, the attacker can use a DNS record of their choosing (possibly under the attackers control) that intentionally generates a very large response. To prevent such abuse, many organizations these days lock down their recursive resolvers so that they answer requests only from clients on their own networks. Penn does this also. There are however public DNS resolver services, like Google DNS, and OpenDNS, that by design are open to the world, and so need to have effective countermeasures in place to deal with these potential attacks. Both Google and OpenDNS say that they do.

There are still a huge number of open recursive DNS resolvers on the Internet, the vast majority of which are probably unintentionally so. The Open DNS Resolver Project has been cataloging them and reports a number in the neighborhood of 25 million!

But authoritative DNS servers (which need to be open to the world) are also quite vulnerable to amplification attacks. In this case, the attacker cannot choose an arbitrary DNS record, but instead must use only records that already exist in the authoritative server's zones. However, DNS responses are usually always amplifying (ie. the size of the response is a lot larger than the size of the request), so it's only a question of the scale of attack that can be achieved.

Interestingly, DNSSEC, a security enhancement to the DNS protocol, makes the amplification problem significantly worse, since DNS responses with cryptographic signatures are much bigger than normal, unsigned DNS responses. Hence lately, we've been seing a lot of these attacks target authoritative servers with zones that are signed with DNSSEC.

Penn was one of the earliest organizations to deploy DNSSEC (August 2009, well before the root and most of the TLDs were signed). We first noticed such attacks on our authoritative servers in July of last year (2012) - at that time we didn't have a good way to counteract these, but our server and network infrastructure was easily able to absorb the attacks, so we didn't take any action - the attacks continued for a few weeks and then disappeared. In late January 2013 though, a new round of amplification attacks happened on a much larger scale that did negatively affect our infrastructure, causing high load on two of our servers and almost saturating the outbound bandwidth on their network connections. By this time, code to implement an experimental response rate limiting countermeasure was available (see Vernon Schryver and Paul Vixie's Response Rate Limiting (RRL) -- implementations are available for popular DNS server software such as BIND, NSD, etc). We deployed these enhancements shortly after the new attacks, and they have effectively addressed the problem for the time being. The RRL code works by keeping track of client requests, and for repeated requests for the same records from the same client addresses it rate limits the responses, either by silently ignoring some requests, or providing small 'truncated' responses. The working assumption is that well behaved DNS resolvers cache responses for the advertised TTL of the record and so should not be making repeated queries for the same record in a short period of time. The truncated responses when encountered will cause well behaved DNS resolvers to retry their query over TCP, which cannot be effectively used in forged address attacks. More details of how this works are available in this technical note describing the operation of RRL. I've heard that the RRL extensions are planned to be incorporated officially into BIND 9.10, although from the recent announcement about ISC's new DNSco subsidiary, it isn't clear whether this feature will be available only to commercial customers.

Any record that produces a large response can be effectively employed in these attacks. Most of the attacks to date though have been using the DNS resource record type 'ANY' (RR type 255). This query, when directed to an authoritative server's zone name, returns all records at the apex of the zone. With DNSSEC, you'll additionally get DNSKEY, RRSIG, and NSEC/NSEC3 records. To give an idea of the scale of the amplification, a DNSSEC-enabled "upenn.edu, ANY" query generates a response that is roughly 88 times as large as the request (a query of about 38 bytes, and a response of 3,344 bytes). The actual amplification ratio of the bits on the wire is less than this because we have to consider the encapsulating headers (L2, IP, and UDP). With Ethernet (14 bytes) and IPv4 (20 bytes) and UDP (8 bytes), a 38-byte DNS query occupys an 80-byte Ethernet frame. The 3,344 byte DNS response packet exceeds the Ethernet MTU and is fragmented into 3 ethernet frames, totalling 3,470 bytes. This yields an amplification ratio of about 40x, so in the absence of rate limiting countermeasures, a 1Mbps stream of query traffic (about 1,500 queries/second) would have produced a 40Mbps stream of traffic directed towards the victim.

Even non-DNSSEC zones produce a quite substantial amplification though, and often it's easily sufficient for an effective attack. From discussion with a number of colleagues at  other institutions, it's clear that non-DNSSEC sites have also been undergoing the same types of attacks and have had to deploy rate limiting countermeasures.

Some people have proposed not answering requests for ANY (and after all ANY was only meant to be a diagnostic tool and not intended for production uses). This might buy time until attackers adapt to using other records. But it could cause collateral damage also. It turns out there is a variety of software that uses ANY for different purposes. For example, the PowerDNS recursor uses it to obtain A and AAAA records from authority servers in one query and response (normally two query/responses would be required).

So, what can be done in the long term? The RRL implementations appear to be working very well at many sites, but attackers will undoubtedly adapt their methods - perhaps by performing more highly distributed attacks across many authoritative servers using a larger set of record types. Some folks at NLnet Labs have written a good paper that discusses some of the issues: Defending against DNS reflection amplification attacks. Also, although RRL has been designed very carefully to minimize collateral damage, there will still be situations in which it might not work very well, eg. when dealing with resolvers behind large scale NATs and CGNs - a problem which might be increasingly common as we approach IPv4 depletion.

There doesn't seem to be much hope (or incentive) for widescale deployment of BCP38 and other methods to reduce the scope of source address forgery.

Ideas have also been proposed in the IETF to enhance the DNS query/response conversation with lightweight authentication cookies, which might thwart most forgery based attacks. See http://tools.ietf.org/html/draft-eastlake-dnsext-cookies-03 for example. But they would require widescale updates to a lot of DNS software to have an effect, and have thus far not gained much traction.

Forcing DNS queries to use TCP is probably a much too heavyweight solution that will impose large costs in terms of DNS response latency and server resource requirements, although the experimental TCP cookie transactions extension (see RFC 6013 - http://tools.ietf.org/html/rfc6013 and this USENIX paper) aims to address some of the issues. It may be necessary to consider a TCP based solution in light of some operational problems observed with UDP and large DNS packets - for example firewalls and other middle boxes that do not pass through IP fragments, or that botch up handling of extension mechanisms like EDNS0 that negotiate the use large UDP/DNS payloads.

Response amplification survey at Internet2 schools

I was interested in knowing what size amplification the ANY query would produce at some other sites, so I wrote a quick program to do this and tabulate the results. I chose the set of 210 Internet2 universities that I already monitor at my dnsstat website. For each zone, I ran an EDNS0 (DO=1) DNS query for ANY at the zone apex via each of the authoritative servers for the zone, and measured the query and response size and the resulting amplification (response_size/query_size). The full set of data can be seen here, but I'll just excerpt some of the entries from the beginning of the file, sorted by descending order of response amplification. As expected the largest amplifications (in the neighborhood of 100x for DNS payloads, 50x for full packets) are all from DNSSEC signed zones. But many non DNSSEC zones produce large amplifications too. The very low amplification ratios for the zones towards the end of the dataset are mostly due to DNS servers that don't understand EDNS0 and return FORMERR (format erorr) responses.

#########################################################################
# DNS response amplification results for ANY query at zone apex.
# Data collected 2013-04-10
# Sorted by descending order of response amplification ratio (last column)
# Amplification is the ratio of the DNS response and request payloads only,
# it doesn't include the encapsulated UDP/IP/L2 etc headers.
# Line Format:
# zone ns_name ns_address query_size response_size amp1 amp2
# amp1 is the ratio of DNS response payload to DNS query payload
# amp2 is the estimated ratio of the entire response packet(s) to request
# packets, assuming an ethernet path MTU.
#########################################################################
ksu.edu nic.kanren.net. 164.113.192.242 36.0 4085.0 113.47 53.99
ksu.edu kic.kanren.net. 164.113.92.250 36.0 4085.0 113.47 53.99
umbc.edu UMBC3.umbc.edu. 130.85.1.3 37.0 4093.0 110.62 53.41
lsu.edu phloem.uoregon.edu. 128.223.32.35 36.0 4016.0 111.56 53.10
lsu.edu bigdog.lsu.edu. 192.16.176.1 36.0 4016.0 111.56 53.10
umbc.edu UMBC5.umbc.edu. 130.85.1.5 37.0 4045.0 109.32 52.80
umbc.edu UMBC4.umbc.edu. 130.85.1.4 37.0 4045.0 109.32 52.80
sdsmt.edu ns5.gratisdns.dk. 85.17.221.46 38.0 4095.0 107.76 52.76
sdsmt.edu ns4.gratisdns.dk. 87.73.3.3 38.0 4095.0 107.76 52.76
sdsmt.edu ns2.gratisdns.dk. 208.43.238.42 38.0 4095.0 107.76 52.76
sdsmt.edu ns1.gratisdns.dk. 109.238.48.13 38.0 4095.0 107.76 52.76
uiowa.edu dns3.uiowa.edu. 128.255.1.27 38.0 4093.0 107.71 52.74
uiowa.edu dns2.uiowa.edu. 128.255.64.26 38.0 4093.0 107.71 52.74
uiowa.edu dns1.uiowa.edu. 128.255.1.26 38.0 4093.0 107.71 52.74
lsu.edu otc-dns2.lsu.edu. 130.39.254.30 36.0 3972.0 110.33 52.54
lsu.edu otc-dns1.lsu.edu. 130.39.3.5 36.0 3972.0 110.33 52.54
ucr.edu adns2.berkeley.edu. 128.32.136.14 36.0 3965.0 110.14 52.45
ucr.edu adns1.berkeley.edu. 128.32.136.3 36.0 3965.0 110.14 52.45
ualr.edu ns4.ualr.edu. 130.184.15.85 37.0 3979.0 107.54 51.96
ualr.edu ns3.ualr.edu. 144.167.5.50 37.0 3979.0 107.54 51.96
ualr.edu ns2.ualr.edu. 144.167.10.1 37.0 3979.0 107.54 51.96
ualr.edu ns.ualr.edu. 144.167.10.48 37.0 3979.0 107.54 51.96
uiowa.edu sns-pb.isc.org. 192.5.4.1 38.0 4013.0 105.61 51.74
sdsmt.edu ns3.gratisdns.dk. 194.0.2.6 38.0 3963.0 104.29 51.11
berkeley.edu ns.v6.berkeley.edu. 128.32.136.6 41.0 4040.0 98.54 50.19
berkeley.edu adns2.berkeley.edu. 128.32.136.14 41.0 4040.0 98.54 50.19
berkeley.edu adns1.berkeley.edu. 128.32.136.3 41.0 4040.0 98.54 50.19
upenn.edu noc3.dccs.upenn.edu. 128.91.251.158 38.0 3866.0 101.74 49.90
berkeley.edu sns-pb.isc.org. 192.5.4.1 41.0 3996.0 97.46 49.66
berkeley.edu phloem.uoregon.edu. 128.223.32.35 41.0 3996.0 97.46 49.66
ksu.edu ns-3.ksu.edu. 129.130.139.150 36.0 3651.0 101.42 48.42
ksu.edu ns-2.ksu.edu. 129.130.139.151 36.0 3651.0 101.42 48.42
ksu.edu ns-1.ksu.edu. 129.130.254.21 36.0 3651.0 101.42 48.42
indiana.edu dns1.iu.edu. 134.68.220.8 40.0 3671.0 91.78 46.30
indiana.edu dns1.illinois.edu. 130.126.2.100 40.0 3671.0 91.78 46.30
indiana.edu dns2.iu.edu. 129.79.1.8 40.0 3655.0 91.38 46.11
mst.edu dns02.srv.mst.edu. 131.151.245.19 36.0 3433.0 95.36 45.63
okstate.edu ns2.cis.okstate.edu. 139.78.200.1 40.0 3599.0 89.97 45.43
okstate.edu ns.cis.okstate.edu. 139.78.100.1 40.0 3599.0 89.97 45.43
mst.edu ns-2.mst.edu. 131.151.247.41 36.0 3417.0 94.92 45.42
mst.edu ns-1.mst.edu. 131.151.247.40 36.0 3417.0 94.92 45.42
mst.edu dns01.srv.mst.edu. 131.151.245.18 36.0 3417.0 94.92 45.42
mst.edu dns03.srv.mst.edu. 131.151.245.20 36.0 3385.0 94.03 45.01
mst.edu ns1.umsl.com. 134.124.31.136 36.0 3369.0 93.58 44.81
ucr.edu ns2.ucr.edu. 138.23.80.20 36.0 3356.0 93.22 44.64
ucr.edu ns1.ucr.edu. 138.23.80.10 36.0 3356.0 93.22 44.64
ksu.edu nic.kanren.net. 2001:49d0:2008:f000::5 36.0 4085.0 113.47 43.58
ksu.edu kic.kanren.net. 2001:49d0:2003:f000::5 36.0 4085.0 113.47 43.58
upenn.edu dns2.udel.edu. 128.175.13.17 38.0 3344.0 88.00 43.38
upenn.edu dns1.udel.edu. 128.175.13.16 38.0 3344.0 88.00 43.38
upenn.edu adns2.upenn.edu. 128.91.254.22 38.0 3344.0 88.00 43.38
upenn.edu sns-pb.isc.org. 192.5.4.1 38.0 3312.0 87.16 42.98
lsu.edu phloem.uoregon.edu. 2001:468:d01:20::80df:2023 36.0 4016.0 111.56 42.88
lsu.edu bigdog.lsu.edu. 2620:105:b050::1 36.0 4016.0 111.56 42.88
[ ... rest of data omitted ...]

Link to full data set.

-- Shumon Huque

ISOC ION Panel: Advancing the Network

"I tend to think of IPv6 & DNSSEC both a little bit like global warming ... something that is developing kind of slowly ... they're both inevitable, it's a just a question of how long it's going to take"
   - Paul Mockapetris.

The Internet Society has posted a video (1 hour 7 minutes) of the ISOC ION panel that I moderated on "Advancing the Network - Where We've Been, Where We're Headed" in San Diego on December 11th:

    http://www.internetsociety.org/deploy360/blog/2013/04/video-advancing-the-network-where-weve-been-where-were-headed-ion-san-diego/

The panelists were Ron Broersma (DREN), Paul Ebersman (Infoblox), John Spence (nephos6), and Paul Mockapetris (Nominum; and inventor of the DNS), and the main topics of discussion were IPv6 and DNSSEC. All the presentations and the subsequent Q&A were quite informative and worth listening to.

One correction I should make to something I said in my introductory remarks. My last slide showed some statistics from the SecSpider DNSSEC zone monitoring project. Since that project relies on user submissions and some amount of crawling, by now, it vastly underestimates the amount of DNSSEC deployment. There aggregate numbers are roughly 275,000 signed zones, whereas the actual number is a lot higher. The netherlands Top Level Domain (.NL) for example has more than 1.4 million signed zones underneath it:

    http://xs.powerdns.com/dnssec-nl-graph/

--Shumon.

IPv6 and DNSSEC at LOPSA-East Conference

I'm teaching 1/2 day courses on IPv6 and DNSSEC at this year's LOPSA-East conference again, being held in New Brunswick, New Jersey, May 3rd-4th 2013.

The IPv6 course is an updated version of the one I did last year at PICC (PICC has since been renamed LOPSA-East) and elsewhere.

Last year I also did a combined course on DNS and DNSSEC. This year's course is focussed on DNSSEC specifically. This will allow me to go into much more detail on how DNSSEC works, how to configure and deploy it (probably with live examples using BIND), etc. I'll also have more time to discuss DANE and application uses of DNSSEC.

Early bird registration discounts for the conference end April 1st. The full schedule of talks and training programs can be seen at:

http://lopsa-east.org/2013/talks/
http://lopsa-east.org/2013/lopsa-east-training/

--Shumon Huque

TIP2013 - Day 1

Sunday, January 13th. Slightly jet lagged, but I was able to sleep intermittently until 4:45am, Hawaii time.

My computer gear on this trip, pictured below, comprises an 11" MacBook Air laptop, iPhone5, iPad2, and a portable 2TB drive. Taking the tiny MacBook Air rather than my normal bulky 15.4" MacBook pro laptop makes a big difference. Not as comfortable to type, but much much lighter. In encountered one unanticipated problem though. As I mentioned in the previous post, I was upgraded to a better room in the Ainahau tower, where I discovered they have no wireless Internet service, just wired ethernet (and the MacBook Air doesn't have an integrated ethernet port). I can't remember the last hotel (in this country) where I didn't have wireless, so it didn't even occur to me to bring the Air's separate USB ethernet dongle. Luckily, my colleague Richard Machida had one with him which he loaned to me a day later.

I took the 7:30am Joint Techs shuttle bus to the University of Hawaii campus. I sat next to Kevin Mayeshiro, from the University of California at Davis - who already knew who I was. It turns out my colleague Colleen Szymanik has spoken to him as a reference for Brocade Networks - Penn is evaluating Brocade as a candidate vendor for the next refresh of our campus core routers. Kevin grew up in Hawaii, so it was good to chat with someone with extensive local knowledge of our venue.

The conference venue is the University of Hawaii's "East West Center", pictured below with the line of folks waiting for registration/bag pickup.

Sunday is customarily a day occupied by tutorial sessions. I attended sessions on BGP Rover, RPKI, and DNSSEC.

Below: Dan Massey form Colorado State University talks about the BGP ROVER route origin verification system.

Below: RPKI (Resource PKI) tutorial/demo session run by Randy Bush of IIJ.

Below: DNSSEC tutorial by Michael Sinatra of ESnet.

In the evening, back in the hotel, I attended the Internet2 Connector's meeting. This is closed meeting of Internet2 and representatives of Internet2 connector networks (GigaPoPs, RONs etc). I usually represent the MAGPI Gigapop, which is a regional R&E Network run by Penn.

-- Shumon Huque

I'm in (so called) Paradise

[Actual date for the photos in this article: Saturday, January 12th 2013]

My colleague, Richard Machida (U of Alaska, Fairbanks), has been blogging from Hawaii, so I felt motivated to do the same. I'll post follow-up articles for each day of the conference.

I'm in Oahu, Hawaii to attend the Winter 2013 Joint Techs conference, hosted by the University of Hawaii in Manoa. Joint Techs is billed as an international conference of networking experts - and is one of the primary technical conferences for the US Research & Education networking community. Whenever in Hawaii, this conference conveniently changes its name to "Techs in Paradise". Also when in Hawaii, it is usually co-located with the APAN (Asia Pacific Advanced Network) meeting - APAN is the Asia Pacific's R&E network/organization. This time, a new format is being tried with a fully merged Joint Techs/APAN conference. As a result, we have quite a large attendance - almost 500 people, stressing the conference facilities a bit.

I chose an aggressive flight schedule that had me leaving Philly at 6:30am and arriving at 1:30pm in Honolulu, if I made the 32 minute (!) connection in San Francisco. Thankfully I did, so I had some daylight to walk around Waikiki a bit and take some photos.

Weather on Oahu so far has been sunny with periods of rain, and temperatures reaching the high 70's (farenheit). Temperature forecast from the local news:

Here are some photos from my flight and from around Waikiki. The complete and growing set, which includes photos from the TIP2013 conference too can be seen on my Google Plus album.

An aerial shot of the Golden Gate bridge and the Presidio Park, from my flight out of San Francisco, shortly after take-off.

 Some aerial shots of Oahu from my flight into Hawaii. The first shows a portion of Eastern Oahu including the Mokapu peninsula and Kailua. If I was on the left side of the plane, I might have been able to get a few photos from directly above Diamond Head, but I saw a lot of clouds in that direction, so it may not have been visible (I have some good shots of that side from a previous trip which I might post later).

The following photo is from directly above Waikiki beach (a detailed crop of a larger photo I took). A large part of Waikiki, Waikiki Beach, and a portion of the Ala Wai canal (which diverts mountain stream water around Waikiki) can be seen.

Diamond Head, probably Hawaii's most famous landmark, is a dormant volcanic crater, created by a relatively recent eruption (ie. geologically speaking; 150,000 years ago). The south western tip, the highest point of the crater rim stands 761 feet high. It's possible to hike all the way to the top (which I've done several times before).

 Another view from Kapiolani Beach Park (a much more tranquil place than Waikiki Beach):

The conference hotel is the Sheraton Princess Kaiulani, which is about a block from the beach. I got upgraded to a "partial ocean view" room due to my SPG gold status, but it isn't really much of a view:

At the last TIP (2008), we occupied a swankier hotel, the Sheraton Waikiki, which was right on the beach, and where I also got upgraded to an ocean view room. Here's a photo from my room at that time - it was designed so that you could seen Diamond Head while you showered from the bathroom (top half of shower curtain was transparent), which was a bit weird. The Sheraton Waikiki has since become even more upscale and I'm told its prices are now out of the range of Joint Techs.

A few other photos:

Link to rest of photos.

Shumon Huque

USENIX LISA 2012 courses

I taught two courses at last month's USENIX LISA 2012 conference in San Diego, California. One on IPv6 and another on DNS and DNSSEC. Each had about 60 attendees and they both went very well.

Slides for the IPv6 course are available at:
http://www.huque.com/~shuque/doc/2012-12-IPv6-Tutorial-huque.pdf

Slides for the DNS and DNSSEC course are available at:
http://www.huque.com/~shuque/doc/2012-12-DNS-DNSSEC-Tutorial-huque.pdf

I also participated in the Internet Society's ION conference, where I moderated a panel on "Advancing the Network: Where We've Been and Where We're Headed". Participants on my panel were Ron Broersma (DREN), Paul Ebersman (InfoBlox), John Spence (Nephos6), and Paul Mockapetris (Nominum, and the inventor of the DNS).

A few photos from my trip are available on my Google Plus page.

--Shumon Huque

Philadelphia skyline photos

My colleague Deke Kassabian posted an older photo of the Philly skyline (that I'd taken a number of years ago) on his Facebook page. So I thought I'd post a more recent set. Since I try not to use Facebook, I've posted the set to my Google Plus page instead.

These were taken in September, on my walk home from work across South Street and the South Street bridge. Equipment: Nikon D90 with Nikon 18-200mm lens.

Internet2 IPv6 Panel recap

A few notes from last month's IPv6 deployment panel at the Fall Internet2 Member Meeting in Philadelphia, which I moderated (October 2nd 2012). Watch the entire video of the session (1 hour 15 minutes) for full details.

I opened the session with a brief review of World IPv6 Launch and some its measurement data, and a mention of other IPv6 deployment measurement projects, including the NIST survey of universities.

ARIN - Mark Kosters, CTO

Mark began his presentation by talking about the current state of IPv4 address depletion. ARIN has 2.87 /8 equivalent address blocks left (note: it's down to 2.79 as of Nov 18th). RIPE and APNIC are almost out, below the one /8 threshold at which a real address rationing stage has already begun - they will give out a maximum of only /20 or /22 sized blocks regardless of how much IPv4 address space you really need. With current projections, ARIN is scheduled to exhaust in August 2013, but events could dramatically change the timeline. For example there are some ISPs in the ARIN region that could easily qualify for /9 allocations. ARIN is in phase 2 of a 4-phase runout plan - details can be seen at https://www.arin.net/resources/request/ipv4_countdown.html. Today, 58% of ARIN's membership has only IPv4 address space, 6% has only IPv6, and 36% have both IPv4 and IPv6 space. Legacy address blocks (ie. allocated to organizations prior to ARIN's existence - this is quite common in the US higher ed community) comprise 45% of the ARIN address space.

The second part of Mark's talk was about IPv6 deployment activites at ARIN itself. He went through a short history of their IPv6 implementation, which dates back to 2003, at which time they had a somewhat creaky, segregated (ie. non dualstack) implementation. By 2008 they migrated to a robust dualstack implementation. All ARIN services today are native, dualstack. Meeting networks are a bit more challenging, where they often have to rely on tunnels (due to hotel carrier limitations).

Comcast - John Brzozowski, Chief IPv6 Architect

Comcast now has IPv6 enabled on 50% of their broadband network footprint. However only 2.5% of the customer base is currently dualstack - of these dualstack users about 65% are using a computer directly connected to the cable modem, and 35% have an IPv6 enabled home router to which devices are attached. Comcast expects this ratio will eventually flip over to 80/20 in favor of home routers as it is in IPv4. To date, their focus has been on residential broadband, but they have pilots for business and commercial service. The Comcast metro ethernet service is IPv6 ready and they'd love to have more customers using it. In terms of traffic, Comcast has seen a 375% increase in IPv6 compared to World IPv6 day (June, 2011), with the majority of the increase occurring between January and June 2012, in the run up to World IPv6 Launch. The majority of the traffic is composed of services like Youtube and Netflix. The 2012 Olympics (streamed by Comcast/NBC in the US) had a noticeable impact - about 6% of this traffic to Comcast customers used IPv6. Comcast continues to work on content and services. Xfinity and comcast.net are IPv6 enabled and they use Akamai as a content delivery network.

They've put in place an extensive measurement and metrics platform to proactively detect any adverse affects on customer experience and take action if needed. Comcast sees advantages to IPv6 beyond the usual address space depletion concerns. An example cited was the fact that once they've allocated an IPv6 subnet, they don't have to be concerned about resizing it again, in marked contrast to their IPv4 deployment - this adds up over time to a significant operational benefit.

Marist College - Eric Kenny

Marist College is located in upstate network with 5,500 students on a 180-acre campus with 50 buildings. They started IPv6 deployment in 2010, and by June 2012 the wired network was mostly done. They plan to deploy IPv6 to the wireless network in the winter of 2012. One big item for them was getting provider independent address space from ARIN (mostly internal process issues). They started with a provider allocated block from NySERNET, but eventually got a /48 from ARIN, and are now wondering if they should have applied for a larger block. They also have a native IPv6 BGP peering with their commercial ISP, Lightower, which was described as an interesting experience, since they were their first IPv6 customer. They use stateless address autoconfiguration, and most devices on their network support IPv6. They haven't yet made much progress with IPv6 enabled network services, apart from DNS. At the current time about 4% of the traffic crossing their border is IPv6 - they expect this number to go up considerably after wireless deployment. One of their biggest concerns has been address tracking and accountability. They've developed their own application to track IPv6 address and MAC address associations.

Louisiana State University - Allie Hopkins

Allie focussed more on the application and process side of things. As part of their rollout process, LSU has had a great dialog with the user community, with extensive outreach and communications via message boards and e-mail lists, and close contact with application developers and deployers. And despite a set of issues (described later), they rate their deployment a success and as can be seen from World IPv6 Launch measurements, they generate a substantial amount of traffic. One of the issues they've had is with their DNS based network registration system which was incapable of working properly with IPv6 enabled hosts. Another issue they encountered was connectivity issues between tagged and untagged VLANs on interfaces, due to the Cisco routers using the same IPv6 link local addresses on them - this was fixed by manually configuring unique addresses on the VLAN interfaces. They've also had ongoing issues with being put on Google's AAAA blacklist.They are aware that they have some pockets of poor IPv6 connectivity within their campus network, but so far they've gotten no help from Google about measurement data that could help them more easily track down these cases.

I was scheduled to do a short presentation about IPv6 at Penn, but I decided to skip it (I'll post the slides later) in the interests of providing more time for audience discussion and questions.

Q&A Portion

Richard Machida (U of Alaska) asked if anyone was looking into deploying IPv6-only networks for any purpose. I answered that we've deployed them only in the lab for testing purposes (longer term, we would investigate whether it's possible to run specific applications that may not need IPv4, like VoIP on them). John says Comcast has no IPv6-only networks, but they do have plenty of IPv6-only devices that sit on dual-stack networks.

There was a question about more details of LSU's network registration system (from what I gathered it's a Netreg or Netreg like system), and whether 802.1x based network access control would be a solution (802.1x authentication happens at the link layer and is IP protocol independent). It probably would, but LSU was not quite prepared to deploy it yet.

There was discussion about what folks were doing about measuring IPv4 vs IPv6 traffic. Comcast went into some more detail about their work. I showed some data from Penn, where over the summer we were approaching 11% inbound and 4% outbound traffic composed of IPv6. We've since dropped down substantially, because we had to take IPv6 off the wireless network (I'm planning to write another blog article detailing why, but the quick summary is that deploying v6 broke IP address mobility, and we're working with our wireless equipment vendor on some possible solutions).

Dan Magorian (Johns Hopkins) asked what cable modem data specifications support IPv6 and what is the state of IPv6 support in the currently deployed field. John's answer is that DOCSIS 3, the current spec supports it pretty completely, almost all current equipment implements it, but there are ways to support IPv6 on older specs too.

---

Addendum: John B was one of the recipients of the Internet Society's Itojun Service Award recently, and presented at IETF'85, for his "for his tireless efforts in providing IPv6 connectivity to cable broadband users across North America and evangelizing the importance of IPv6 deployment globally". Congratulations John!

Shumon Huque

IPv6 and DNSSEC in San Diego

I'll be in San Diego, California in early December for the USENIX LISA 2012 conference. As part of the conference's training program, I'm teaching two courses - a full day on IPv6, and a half-day on DNS and DNSSEC.

The early registration deadline (cheaper rates) is November 19th. Various discounts are available for members of USENIX and LOPSA.

The IPv6 course "Using and Migrating to IPv6", is on Monday, December 10th. This is a revised and expanded version of the half-day course I taught at last year's LISA conference in Boston, Massachusetts, which was well received. I had originally proposed to do a full-day course last year too, but the organizers at the time felt that they wouldn't be able to attract enough attendees for day long session on IPv6. It turned out that my session was packed and the audience was quite engaged - I even ran over by 30 minutes to cover some advanced topics and almost everyone stayed the extra time. Hopefully I'll get a good turnout this time also.

The DNS and DNSSEC course, is on Tuesday morning, December 11th. This will cover the basic DNS protocol as well as the DNS Security Extensions, including practical configuration examples.

Attendee comments and feedback on my last IPv6 and DNS courses (PICC 12) are available if you're interested. In courses like these with audiences with potentially diverse backgrounds, it's rarely possible to please everyone, but my courses generally get almost uniformly positive reviews (so far at least).

Incidentally, "IPv6 and DNSSEC" is one of the themes of the conference this year. Other sessions in the category include Owen DeLong of Hurricane Electric on "IPv6 Address Planning", Scott Rose of NIST on "Progress of DNSSEC deployment in the federal government", and Roland van Rijswijk of SURFnet on "DNSSEC, what every sysadmin should be doing to keep things working".

Vint Cerf is delivering the keynote on "The Internet of Things and Sensors and Actuators", which will surely discuss IPv6.

There are many other interesting sessions. Check out the agenda for the technical program, the training program, and the workshops.



Co-located with LISA, the Internet Society is also hosting its ION Conference on the afternoon of Tuesday, December 11th.  ION is a conference series organized by ISOC's Deploy360 programme, which provides deployment information on advanced technologies like IPv6, DNSSEC, secure routing, etc.

I'm moderating a panel session titled "Advancing the Network: Where We've Been, Where We're Headed" - joining me on the panel are Ron Broersma (DREN), Paul Ebersman (Infoblox), John Spence (Nephos6), and Paul Mockapetris (inventor of the DNS). Another panel focussed on DNSSEC is being run by Dan York. The full agenda is available.

Registration for the ION conference is free, but seats are limited.

Hope to see some of you in San Diego ..

Shumon Huque

DNSSEC and Certificates

DNSSEC is a system to verify the authenticity of DNS data using public key signatures. With increasing deployment of DNSSEC comes the possibility of applications using the DNS to store and retrieve TLS/SSL certificates in an authenticated manner. And possibly obviating the need for public/global certification authorities (CA), and empowering domain owners to issue their own certificates instead.

For those of you who know me a little better, you're probably aware that I've been griping about the inherent deficiencies of the public CA model for years. Certainly my colleagues at Penn have been subjected to many a diatribe on this topic, and occasionally I've done so more publicly (no pun intended).

In brief, the problems are that applications (such as web browsers) are pre-configured to trust a very large number of global CAs. These CAs have absolutely no constraints on the namespace for which they can issue certificates. Any of them can issue a certificate for any of your services (whether you have a business relationship with them or not). And thus, we have what is tantamount to least common denominator security - the collective security of the system is equivalent to the CA with the weakest security and laxest policies. It gets worse - many of these CAs in turn issue subordinate CA certificates for their customers, again with no namespace constraints.

Another issue is that most of these CAs are capable only of issuing certificates with the most basic capabilities. If you need to deploy a certificate with some more exotic feature (eg. an alternate name form or extension), you're basically out of luck.

Incidentally the PKIX specifications do have the capability of constraining the namespace (see RFC 5280 Section 4.2.1.10: Name Constraints extension), but deploying it in the existing public CA model will require a huge amount of work (both technical and political), including deployment of a hierarchical global CA architecture, as well as enhancing client software to validate name constraints (which practically none do today).

Introducing DANE and TLSA

Recent work in the IETF holds some promise to improve things. RFC 6698 (DANE) defines the new DNS record type TLSA, which holds what is called Certificate Association data. In conjunction with DNSSEC signatures, this will permit better and more secure ways for applications to authenticate certificates. The record can be used in 4 different ways (defined by a "Usage" field):

• (Usage 0) Specify authorized public CAs - this can prevent certificates signed by CAs that you didn't explicitly specify from being trusted by client software.
• (Usage 1) Specify which specific server certificate can be trusted. The public CA certificate validation chain all the way to the server certificate must still be validated.
• (Usage 2) Authorize a new non-public CA (eg. a CA that the domain owner operates itself)
• (Usage 3) Directly specify the server certificate in the DNS - no certificate chain from a public CA needs to be used. This is called a "Domain issued certificate"

If you're looking for a way to not use public CAs, Usage 3 is probably the one for you.

The owner name of the TLSA record defines the port, transport protocol, and server host name. Here's an example for the HTTPS service (port 443, TCP) at www.example.com:

  _443._tcp.www.example.com. IN TLSA ( 0 0 1
                                d2abde240d7cd3ee6b4b28c54df034b9
                                7983a1d16e8a410e4561cb106618e971 )

The right hand side ("rdata" or resource data) of the TLSA record contains 3 numbers and the raw certificate data in hexadecimal. The 3 numbers are the usage field (previously described), the selector field (which says whether the certificate data takes into account the full certificate or only the subject's public key), and the matching-type field (which says whether the certificate data is the full content that the selector field specifies, or whether it is a cryptographic hash of it). See RFC 6698 for a more detailed description.

Deploying a TLSA record for my website

After seeing several prominent mentions of DANE at this week's ICANN'45 DNSSEC workshop, (see Dan York and Paul Wouters' presentations for example) I felt motivated to install a TLSA record for my website.

Now billions of users with TLSA capable web browsers and DNSSEC validating resolvers will be able to securely reach my website! Okay, that's a slight exaggeration, but perhaps one day, in the not too distant future!

I wanted to see if I could do this using standard tools already available on most UNIX systems. It wasn't too hard. My site already has an SSL certificate issued by StartCom Ltd, but I want to specify that the server certificate can also be authenticated directly without a CA (TLSA Usage type 3). I'll also use a SHA256 hash (match-type 1) of the full certificate (selector 0) as the certificate association data (so the 3 numbers in the rdata will be 3 0 1).

OpenSSL can be used to easily generate the certificate association data portion. We first convert my PEM encoded certificate (in file huque.crt) into the binary DER encoding (openssl x509 -outform DER) and then pipe the output to "openssl sha256". This generates the 256-bit SHA hash represented as a string of 64 hex digits (each hex digit is 4 bits):

  $ openssl x509 -in huque.crt -outform DER | openssl sha256
  (stdin)= 8cb0fc6c527506a053f4f14c8464bebbd6dede2738d11468dd953d7d6a3021f1

The resulting TLSA DNS record will thus look like:

  _443._tcp.www.huque.com. IN TLSA 3 0 1 ( 
       8cb0fc6c527506a053f4f14c8464bebbd6dede2738d11468dd953d7d6a3021f1 )

One quick nsupdate command  (details omitted) and the record was installed in my DNSSEC signed zone. We can check that the record (and its signature) is present with dig:

$ dig +dnssec +noall +answer +multi _443._tcp.www.huque.com. TLSA

_443._tcp.www.huque.com. 893 IN TLSA 3 0 1 (
                                8CB0FC6C527506A053F4F14C8464BEBBD6DEDE2738D1
                                1468DD953D7D6A3021F1 )
_443._tcp.www.huque.com. 893 IN RRSIG TLSA 8 5 900 (
                                20121117202722 20121018192722 14703 huque.com.
                                OJ4b/O7J+Fh3KVDGG8nH9X5dRSXgl3j7/S/PbB1RczcY
                                fYxdu5C9xZSALCz0MgVFW6Kcne74ou5R/Wd+CDKm7mYY
                                Ga28MrtE1boNLuOTa6kOpFcNjW+UYnMFQuc6S8zhU1G7
                                LH0n3TbM3rJS0wH0u6J4NgtiowZS2VlkwPez3D8= )

(If you're using an older version of BIND/dig that doesn't understand the TLSA record type, you can replace TLSA in the command line above with TYPE52.)

Note: the huque.com zone does not currently have a secure delegation from the .COM TLD. This is mainly because my current DNS registrar (Network Solutions) is incapable of processing DS records, sigh, and I've been too lazy to switch. In the meantime, it has a DLV record published in the DLV registry operated by ISC, so any validating resolver configured to use that registry (and many are) should be able to authenticate records in huque.com.

Not surprisingly, there are already tools out there that make it easy to generate TLSA records. Paul Wouter's hash-slinger package can do it (supposedly integrated into newer Fedora Linux releases). With the "tlsa" program included in that package, we can do this via:

  $ tlsa --create --output rfc --usage 3 --certificate huque.crt www.huque.com
  _443._tcp.www.huque.com. IN TLSA 3 0 1
       8cb0fc6c527506a053f4f14c8464bebbd6dede2738d11468dd953d7d6a3021f1

Obviously DANE needs to be widely adopted by the Internet community for it to be successful, and it's not yet clear how long that will take. And obviously browsers and other application sofware will have to evolve to make use of TLSA records. But preliminary work appears to already be in progress - there are a few browser extensions that try to validate TLSA records for HTTPS websites. I plan to look into those next ..

Incidentally, I'm teaching a half-day DNS and DNSSEC course at the upcoming USENIX LISA conference in San Diego, December 9th-14th. I hope to talk more about DANE and TLSA there.

Shumon Huque