I've left Penn for a new job

After more than 20 years of working at Penn (University of Pennsylvania), I've decided to take a new job as Principal Research Scientist at Verisign Labs, the applied research division of Verisign Inc. You might know that Verisign is one of the world's largest DNS infrastructure providers. It runs the .com, .net, .edu, and .gov DNS top level domains, two of the thirteen DNS root servers (A and J), and performs the critically important root zone management function. Verisign also provides managed DNS, Distributed Denial of Service (DDoS) mitigation, and several other services. (Note: Verisign's certificate services business was sold off to Symantec several years ago).

I've been at Penn for so long, that I startled many of my colleagues with the news of my departure, so I'll say a few things about how this came about. I originally came to Penn in 1989 as an undergraduate. I became a full time IT staff member after I graduated - my first job was the system administrator of the new e-mail server for the School of Arts & Sciences, the largest school within Penn. I completed a Masters degree in Computer Science part-time while working. I later moved to the central IT organization, where I remained until last week, first as a Network Engineer, then as the Lead Engineer, and most recently as an Engineering Director. In addition, I've been a part time Adjunct Faculty in Penn's School of Engineering, teaching a laboratory course on Network Protocols.

I was approached by Allison Mankin and Matt Larson (I've known both of them for a while) about a year ago to see if I'd be interested in considering a research scientist position at Verisign Labs. Matt has since left Verisign to work at Dyn, and Allison is now a Director at Verisign Labs. At the time, I thought this was distinctly a long shot, but over the course of many months, I thought more seriously about the possibility. I visited Verisign Labs in September 2013, did an interview with them, and also met Burt Kaliski, Verisign CTO (Burt is a noted cryptographer and was the founding scientist of RSA Labs, and the developer of the PKCS standards). I kept in touch with Allison since, but it took me until the beginning of this year to finally come to the conclusion that I wanted to take this opportunity, so here I am. My first day at Verisign will be tomorrow (March 17th).

I've enjoyed my job and career at Penn a lot. I've been extensively involved in a very diverse range of technical projects, ranging from software development, systems engineering, and network design. Among others, I was responsible for the design and operation of much of the authentication and security infrastructure at Penn, as well as a variety of other services, like the DNS and DHCP (and many more that I don't have the time to enumerate here). I was the principal architect of IPv6 and DNSSEC deployment at Penn. I was the chief engineer of the MAGPI gigapop and through that role, was involved in R&E regional and national networking activities. Increasingly though, a lot of time is taken up by non-technical activities, and the longer I stay at Penn, the greater the possibility that I'll end up as a full time IT staff manager, which isn't the role I envision for myself. I've always seen my primary role as that of a technologist, and this job change will allow me to continue in that direction.

Verisign Labs appears to have a true applied research agenda, which I find appealing. Obviously DNS and DNSSEC research is an area in which I expect to be working. But there are many other interesting areas of work: routing, IPv6, reputation systems, security protocols, future internet architectures, etc. I'm also looking forward to attending more NANOG and IETF meetings to get myself more plugged into those communities than I've been able to thus far. Verisign Labs also has frequent, productive collaborations with computer science researchers through its university collobaration program.

In the process, I'm moving to the Washington DC metro area (specifically Reston, VA) where Verisign is located, another big change for me!

More later ..

--Shumon Huque.

EDU Top Level Domain statistics

Some DNS Top Level Domain (TLD) operators publish statistics about their DNS zones. Some others have a zone file access program that allows others to examine their data and publish statistics. Frederic Cambus (@fcambus on Twitter) maintains a site called statdns ( http://www.statdns.com/ ) that keeps statistics for several of the TLDs.

The EDU top level domain is conspicuously absent from the statdns site because the operators don't publish any statistics and also don't have a zone file access program in place. The EDU domain has a very complicated operational policy arrangement. It is managed by Educause (a higher education IT consortium), but operated by Verisign, under a contract with the United States Department of Commerce. I recently spoke with colleagues at Educause about current prospects for publishing some statistics or making the zone data available. The good news is that a zone file access program request is in fact in the queue to be approved by the Dept of Commerce. But it's stuck behind a few other requests, so it may still take some time to come to fruition.

In the meantime, to satisfy my own curiosity, I've been looking at other ways to obtain some statistics. In particular I'm interested in seeing how much DNSSEC deployment has happened so far, and how EDU compares with some of the other TLDs in this respect. One way to gain visibility into zone contents is to examine passive DNS databases. A number of folks and organizations run such databases that collect historical information seen from DNS responses at collections of resolvers. By searching records over a period of time in these databases, it's possible to reconstruct a substantial portion of the active records in a zone. I did this for EDU and analyzed the results recently.

The passive DNS database search managed to find about 7,158 second level domains under EDU. Of these, 6955 domains turned out to be valid (the others probably existed at one point but don't any more). EDU is known to have in the neighborhood of 7,000 delegations, so this is most probably a pretty good approximation of the active contents of the zone.

EDU Zone Statistics:

Number of Domains from passive DNS db: 7158
Number of Valid Domains: 6955

Total NS records: 19527
Unique NS records: 9757
Number of (glue) IPv4 address records: 4555
Number of (glue) IPv6 address records: 246

DNSSEC Specific Stats for EDU:

Number of DNSSEC Signed Zones: 94 (1.37%)
Number of NSEC3 Zones: 29 (30.1% of the signed zones)
Number of Zones with DS records: 76
Number of Zones with DLV records at dlv.isc.org: 7

As expected, only a very small fraction (1.37%) of domains have deployed DNSSEC. This compares with about 0.25% in .COM, 0.41% in .NET, and 0.30% in .ORG.

The 94 zones in EDU signed with DNSSEC are:

acadiana.edu
baker.edu
beloit.edu
berkeley.edu
bucknell.edu
cameron.edu
carnegiemellon.edu
catc.edu
chattanoogastate.edu
cltc.edu
cmu.edu
coloradomesa.edu
cookman.edu
csupomona.edu
cuhk.edu
desales.edu
drake.edu
example.edu
fhsu.edu
fhtc.edu
gfcmsu.edu
gsu.edu
gtc.edu
hfg.edu
highlands.edu
indiana.edu
indianatech.edu
internet2.edu
iu.edu
iub.edu
iup.edu
iupui.edu
jhuapl.edu
kestrel.edu
kiropraktik.edu
k-state.edu
ksu.edu
kutztown.edu
lctcs.edu
lsu.edu
ltc.edu
ma.edu
mansfield.edu
mcpherson.edu
merit.edu
mesa.edu
millikin.edu
mnsfld.edu
monmouth.edu
monterey.edu
mst.edu
nau.edu
northcentral.edu
northshorecollege.edu
nwltc.edu
okstate.edu
pacificu.edu
penn.edu
pitt.edu
psc.edu
richland.edu
rockefeller.edu
rose-hulman.edu
scl.edu
sdsmt.edu
sfcollege.edu
shoreline.edu
suu.edu
tbu.edu
tiasnimbas.edu
tilburguniversity.edu
tiss.edu
truman.edu
uaa.edu
ualr.edu
ucaid.edu
ucb.edu
ucberkeley.edu
ucdavis.edu
ucr.edu
uiowa.edu
umbc.edu
uni-stuttgart.edu
unt.edu
untsystem.edu
upenn.edu
upf.edu
usnwc.edu
uwm.edu
uwstout.edu
valencia.edu
waketech.edu
washjeff.edu
weber.edu

The 29 zones that use the NSEC3 variety of DNSSEC are:

csupomona.edu
cuhk.edu
gfcmsu.edu
internet2.edu
jhuapl.edu
kestrel.edu
kiropraktik.edu
k-state.edu
ksu.edu
lsu.edu
ma.edu
mansfield.edu
mcpherson.edu
millikin.edu
mnsfld.edu
pitt.edu
richland.edu
rose-hulman.edu
sdsmt.edu
suu.edu
tiasnimbas.edu
tilburguniversity.edu
ualr.edu
ucaid.edu
ucr.edu
uni-stuttgart.edu
unt.edu
untsystem.edu
washjeff.edu

There are 18 zones that do not have DS records published (not sure why):

beloit.edu
cameron.edu
cookman.edu
iup.edu
kiropraktik.edu
kutztown.edu
mansfield.edu
merit.edu
mnsfld.edu
okstate.edu
shoreline.edu
tbu.edu
tiasnimbas.edu
uaa.edu
usnwc.edu
uwm.edu
uwstout.edu
waketech.edu

There are also 7 zones with DLV records published at ISC's DLV registry, but this set is disjoint with the set that doesn't have DS records:

bucknell.edu
internet2.edu
k-state.edu
ksu.edu
ualr.edu
ucaid.edu
ucr.edu

-- Shumon Huque

TLSA Record Generator

Last year I wrote a blog article on DNSSEC and Certificates. Occasionally I get questions from folks who've tried to follow my instructions to create the content of TLSA records, but have failed because they are using a version of openssl that is too old to generate SHA-256 and SHA-512 hashes.

I've written a small web application to help generate TLSA records. I hope this is of use to some folks:

        https://www.huque.com/bin/gen_tlsa

(I apologize in advance for my rather primitive webpage design skills!)

Here is a screenshot of it in action to generate the TLSA record for my own website:

And the resulting TLSA record that was generated:



-- Shumon Huque

DNS Amplification Attacks

There has been a lot of talk recently about DNS amplification attacks (with prominent news reports of high bandwidth attacks targeted at anti-spam services, cloud providers, financial institutions, etc). These are a class of denial of service attack that use DNS servers to emit large amounts of traffic onto unsuspecting victims. The attackers use the forged source addresses of their victims to send a large stream of queries to the DNS servers, and this results in a much larger stream of responses being sent from those DNS servers back to the victim computers - with the aim of overwhelming them or their network connections. Although the DNS servers employed in the attack can become adversely impacted, the primary target of the attacks are the computers whose addresses are being forged.

Attacks that employ IP source address forgery are effective on the Internet today, because the countermeasures that would prevent this forgery are not very widely deployed. BCP 38 (a 13 year old document!) and BCP 84 describe network ingress filtering as a way for ISPs to block forged traffic from their customer networks, but many ISPs fail to employ it. Organizations should also ideally configure their networks not to permit internally generated traffic with forged addresses from crossing their borders - typically a border Access Control List or more granular techniques like unicast reverse path forwarding (URPF, a per-subnet level antispoofing method) can be used to do this, but once again, these are not in very widespread use.

In the past, the attacks commonly employed 'open recursive DNS resolvers' - these are DNS recursive resolvers that will resolve names for any client computer without restriction. When such servers are available, the attacker can use a DNS record of their choosing (possibly under the attackers control) that intentionally generates a very large response. To prevent such abuse, many organizations these days lock down their recursive resolvers so that they answer requests only from clients on their own networks. Penn does this also. There are however public DNS resolver services, like Google DNS, and OpenDNS, that by design are open to the world, and so need to have effective countermeasures in place to deal with these potential attacks. Both Google and OpenDNS say that they do.

There are still a huge number of open recursive DNS resolvers on the Internet, the vast majority of which are probably unintentionally so. The Open DNS Resolver Project has been cataloging them and reports a number in the neighborhood of 25 million!

But authoritative DNS servers (which need to be open to the world) are also quite vulnerable to amplification attacks. In this case, the attacker cannot choose an arbitrary DNS record, but instead must use only records that already exist in the authoritative server's zones. However, DNS responses are usually always amplifying (ie. the size of the response is a lot larger than the size of the request), so it's only a question of the scale of attack that can be achieved.

Interestingly, DNSSEC, a security enhancement to the DNS protocol, makes the amplification problem significantly worse, since DNS responses with cryptographic signatures are much bigger than normal, unsigned DNS responses. Hence lately, we've been seing a lot of these attacks target authoritative servers with zones that are signed with DNSSEC.

Penn was one of the earliest organizations to deploy DNSSEC (August 2009, well before the root and most of the TLDs were signed). We first noticed such attacks on our authoritative servers in July of last year (2012) - at that time we didn't have a good way to counteract these, but our server and network infrastructure was easily able to absorb the attacks, so we didn't take any action - the attacks continued for a few weeks and then disappeared. In late January 2013 though, a new round of amplification attacks happened on a much larger scale that did negatively affect our infrastructure, causing high load on two of our servers and almost saturating the outbound bandwidth on their network connections. By this time, code to implement an experimental response rate limiting countermeasure was available (see Vernon Schryver and Paul Vixie's Response Rate Limiting (RRL) -- implementations are available for popular DNS server software such as BIND, NSD, etc). We deployed these enhancements shortly after the new attacks, and they have effectively addressed the problem for the time being. The RRL code works by keeping track of client requests, and for repeated requests for the same records from the same client addresses it rate limits the responses, either by silently ignoring some requests, or providing small 'truncated' responses. The working assumption is that well behaved DNS resolvers cache responses for the advertised TTL of the record and so should not be making repeated queries for the same record in a short period of time. The truncated responses when encountered will cause well behaved DNS resolvers to retry their query over TCP, which cannot be effectively used in forged address attacks. More details of how this works are available in this technical note describing the operation of RRL. I've heard that the RRL extensions are planned to be incorporated officially into BIND 9.10, although from the recent announcement about ISC's new DNSco subsidiary, it isn't clear whether this feature will be available only to commercial customers.

Any record that produces a large response can be effectively employed in these attacks. Most of the attacks to date though have been using the DNS resource record type 'ANY' (RR type 255). This query, when directed to an authoritative server's zone name, returns all records at the apex of the zone. With DNSSEC, you'll additionally get DNSKEY, RRSIG, and NSEC/NSEC3 records. To give an idea of the scale of the amplification, a DNSSEC-enabled "upenn.edu, ANY" query generates a response that is roughly 88 times as large as the request (a query of about 38 bytes, and a response of 3,344 bytes). The actual amplification ratio of the bits on the wire is less than this because we have to consider the encapsulating headers (L2, IP, and UDP). With Ethernet (14 bytes) and IPv4 (20 bytes) and UDP (8 bytes), a 38-byte DNS query occupys an 80-byte Ethernet frame. The 3,344 byte DNS response packet exceeds the Ethernet MTU and is fragmented into 3 ethernet frames, totalling 3,470 bytes. This yields an amplification ratio of about 40x, so in the absence of rate limiting countermeasures, a 1Mbps stream of query traffic (about 1,500 queries/second) would have produced a 40Mbps stream of traffic directed towards the victim.

Even non-DNSSEC zones produce a quite substantial amplification though, and often it's easily sufficient for an effective attack. From discussion with a number of colleagues at  other institutions, it's clear that non-DNSSEC sites have also been undergoing the same types of attacks and have had to deploy rate limiting countermeasures.

Some people have proposed not answering requests for ANY (and after all ANY was only meant to be a diagnostic tool and not intended for production uses). This might buy time until attackers adapt to using other records. But it could cause collateral damage also. It turns out there is a variety of software that uses ANY for different purposes. For example, the PowerDNS recursor uses it to obtain A and AAAA records from authority servers in one query and response (normally two query/responses would be required).

So, what can be done in the long term? The RRL implementations appear to be working very well at many sites, but attackers will undoubtedly adapt their methods - perhaps by performing more highly distributed attacks across many authoritative servers using a larger set of record types. Some folks at NLnet Labs have written a good paper that discusses some of the issues: Defending against DNS reflection amplification attacks. Also, although RRL has been designed very carefully to minimize collateral damage, there will still be situations in which it might not work very well, eg. when dealing with resolvers behind large scale NATs and CGNs - a problem which might be increasingly common as we approach IPv4 depletion.

There doesn't seem to be much hope (or incentive) for widescale deployment of BCP38 and other methods to reduce the scope of source address forgery.

Ideas have also been proposed in the IETF to enhance the DNS query/response conversation with lightweight authentication cookies, which might thwart most forgery based attacks. See http://tools.ietf.org/html/draft-eastlake-dnsext-cookies-03 for example. But they would require widescale updates to a lot of DNS software to have an effect, and have thus far not gained much traction.

Forcing DNS queries to use TCP is probably a much too heavyweight solution that will impose large costs in terms of DNS response latency and server resource requirements, although the experimental TCP cookie transactions extension (see RFC 6013 - http://tools.ietf.org/html/rfc6013 and this USENIX paper) aims to address some of the issues. It may be necessary to consider a TCP based solution in light of some operational problems observed with UDP and large DNS packets - for example firewalls and other middle boxes that do not pass through IP fragments, or that botch up handling of extension mechanisms like EDNS0 that negotiate the use large UDP/DNS payloads.

Response amplification survey at Internet2 schools

I was interested in knowing what size amplification the ANY query would produce at some other sites, so I wrote a quick program to do this and tabulate the results. I chose the set of 210 Internet2 universities that I already monitor at my dnsstat website. For each zone, I ran an EDNS0 (DO=1) DNS query for ANY at the zone apex via each of the authoritative servers for the zone, and measured the query and response size and the resulting amplification (response_size/query_size). The full set of data can be seen here, but I'll just excerpt some of the entries from the beginning of the file, sorted by descending order of response amplification. As expected the largest amplifications (in the neighborhood of 100x for DNS payloads, 50x for full packets) are all from DNSSEC signed zones. But many non DNSSEC zones produce large amplifications too. The very low amplification ratios for the zones towards the end of the dataset are mostly due to DNS servers that don't understand EDNS0 and return FORMERR (format erorr) responses.

#########################################################################
# DNS response amplification results for ANY query at zone apex.
# Data collected 2013-04-10
# Sorted by descending order of response amplification ratio (last column)
# Amplification is the ratio of the DNS response and request payloads only,
# it doesn't include the encapsulated UDP/IP/L2 etc headers.
# Line Format:
# zone ns_name ns_address query_size response_size amp1 amp2
# amp1 is the ratio of DNS response payload to DNS query payload
# amp2 is the estimated ratio of the entire response packet(s) to request
# packets, assuming an ethernet path MTU.
#########################################################################
ksu.edu nic.kanren.net. 164.113.192.242 36.0 4085.0 113.47 53.99
ksu.edu kic.kanren.net. 164.113.92.250 36.0 4085.0 113.47 53.99
umbc.edu UMBC3.umbc.edu. 130.85.1.3 37.0 4093.0 110.62 53.41
lsu.edu phloem.uoregon.edu. 128.223.32.35 36.0 4016.0 111.56 53.10
lsu.edu bigdog.lsu.edu. 192.16.176.1 36.0 4016.0 111.56 53.10
umbc.edu UMBC5.umbc.edu. 130.85.1.5 37.0 4045.0 109.32 52.80
umbc.edu UMBC4.umbc.edu. 130.85.1.4 37.0 4045.0 109.32 52.80
sdsmt.edu ns5.gratisdns.dk. 85.17.221.46 38.0 4095.0 107.76 52.76
sdsmt.edu ns4.gratisdns.dk. 87.73.3.3 38.0 4095.0 107.76 52.76
sdsmt.edu ns2.gratisdns.dk. 208.43.238.42 38.0 4095.0 107.76 52.76
sdsmt.edu ns1.gratisdns.dk. 109.238.48.13 38.0 4095.0 107.76 52.76
uiowa.edu dns3.uiowa.edu. 128.255.1.27 38.0 4093.0 107.71 52.74
uiowa.edu dns2.uiowa.edu. 128.255.64.26 38.0 4093.0 107.71 52.74
uiowa.edu dns1.uiowa.edu. 128.255.1.26 38.0 4093.0 107.71 52.74
lsu.edu otc-dns2.lsu.edu. 130.39.254.30 36.0 3972.0 110.33 52.54
lsu.edu otc-dns1.lsu.edu. 130.39.3.5 36.0 3972.0 110.33 52.54
ucr.edu adns2.berkeley.edu. 128.32.136.14 36.0 3965.0 110.14 52.45
ucr.edu adns1.berkeley.edu. 128.32.136.3 36.0 3965.0 110.14 52.45
ualr.edu ns4.ualr.edu. 130.184.15.85 37.0 3979.0 107.54 51.96
ualr.edu ns3.ualr.edu. 144.167.5.50 37.0 3979.0 107.54 51.96
ualr.edu ns2.ualr.edu. 144.167.10.1 37.0 3979.0 107.54 51.96
ualr.edu ns.ualr.edu. 144.167.10.48 37.0 3979.0 107.54 51.96
uiowa.edu sns-pb.isc.org. 192.5.4.1 38.0 4013.0 105.61 51.74
sdsmt.edu ns3.gratisdns.dk. 194.0.2.6 38.0 3963.0 104.29 51.11
berkeley.edu ns.v6.berkeley.edu. 128.32.136.6 41.0 4040.0 98.54 50.19
berkeley.edu adns2.berkeley.edu. 128.32.136.14 41.0 4040.0 98.54 50.19
berkeley.edu adns1.berkeley.edu. 128.32.136.3 41.0 4040.0 98.54 50.19
upenn.edu noc3.dccs.upenn.edu. 128.91.251.158 38.0 3866.0 101.74 49.90
berkeley.edu sns-pb.isc.org. 192.5.4.1 41.0 3996.0 97.46 49.66
berkeley.edu phloem.uoregon.edu. 128.223.32.35 41.0 3996.0 97.46 49.66
ksu.edu ns-3.ksu.edu. 129.130.139.150 36.0 3651.0 101.42 48.42
ksu.edu ns-2.ksu.edu. 129.130.139.151 36.0 3651.0 101.42 48.42
ksu.edu ns-1.ksu.edu. 129.130.254.21 36.0 3651.0 101.42 48.42
indiana.edu dns1.iu.edu. 134.68.220.8 40.0 3671.0 91.78 46.30
indiana.edu dns1.illinois.edu. 130.126.2.100 40.0 3671.0 91.78 46.30
indiana.edu dns2.iu.edu. 129.79.1.8 40.0 3655.0 91.38 46.11
mst.edu dns02.srv.mst.edu. 131.151.245.19 36.0 3433.0 95.36 45.63
okstate.edu ns2.cis.okstate.edu. 139.78.200.1 40.0 3599.0 89.97 45.43
okstate.edu ns.cis.okstate.edu. 139.78.100.1 40.0 3599.0 89.97 45.43
mst.edu ns-2.mst.edu. 131.151.247.41 36.0 3417.0 94.92 45.42
mst.edu ns-1.mst.edu. 131.151.247.40 36.0 3417.0 94.92 45.42
mst.edu dns01.srv.mst.edu. 131.151.245.18 36.0 3417.0 94.92 45.42
mst.edu dns03.srv.mst.edu. 131.151.245.20 36.0 3385.0 94.03 45.01
mst.edu ns1.umsl.com. 134.124.31.136 36.0 3369.0 93.58 44.81
ucr.edu ns2.ucr.edu. 138.23.80.20 36.0 3356.0 93.22 44.64
ucr.edu ns1.ucr.edu. 138.23.80.10 36.0 3356.0 93.22 44.64
ksu.edu nic.kanren.net. 2001:49d0:2008:f000::5 36.0 4085.0 113.47 43.58
ksu.edu kic.kanren.net. 2001:49d0:2003:f000::5 36.0 4085.0 113.47 43.58
upenn.edu dns2.udel.edu. 128.175.13.17 38.0 3344.0 88.00 43.38
upenn.edu dns1.udel.edu. 128.175.13.16 38.0 3344.0 88.00 43.38
upenn.edu adns2.upenn.edu. 128.91.254.22 38.0 3344.0 88.00 43.38
upenn.edu sns-pb.isc.org. 192.5.4.1 38.0 3312.0 87.16 42.98
lsu.edu phloem.uoregon.edu. 2001:468:d01:20::80df:2023 36.0 4016.0 111.56 42.88
lsu.edu bigdog.lsu.edu. 2620:105:b050::1 36.0 4016.0 111.56 42.88
[ ... rest of data omitted ...]

Link to full data set.

-- Shumon Huque

IPv6 and DNSSEC in San Diego

I'll be in San Diego, California in early December for the USENIX LISA 2012 conference. As part of the conference's training program, I'm teaching two courses - a full day on IPv6, and a half-day on DNS and DNSSEC.

The early registration deadline (cheaper rates) is November 19th. Various discounts are available for members of USENIX and LOPSA.

The IPv6 course "Using and Migrating to IPv6", is on Monday, December 10th. This is a revised and expanded version of the half-day course I taught at last year's LISA conference in Boston, Massachusetts, which was well received. I had originally proposed to do a full-day course last year too, but the organizers at the time felt that they wouldn't be able to attract enough attendees for day long session on IPv6. It turned out that my session was packed and the audience was quite engaged - I even ran over by 30 minutes to cover some advanced topics and almost everyone stayed the extra time. Hopefully I'll get a good turnout this time also.

The DNS and DNSSEC course, is on Tuesday morning, December 11th. This will cover the basic DNS protocol as well as the DNS Security Extensions, including practical configuration examples.

Attendee comments and feedback on my last IPv6 and DNS courses (PICC 12) are available if you're interested. In courses like these with audiences with potentially diverse backgrounds, it's rarely possible to please everyone, but my courses generally get almost uniformly positive reviews (so far at least).

Incidentally, "IPv6 and DNSSEC" is one of the themes of the conference this year. Other sessions in the category include Owen DeLong of Hurricane Electric on "IPv6 Address Planning", Scott Rose of NIST on "Progress of DNSSEC deployment in the federal government", and Roland van Rijswijk of SURFnet on "DNSSEC, what every sysadmin should be doing to keep things working".

Vint Cerf is delivering the keynote on "The Internet of Things and Sensors and Actuators", which will surely discuss IPv6.

There are many other interesting sessions. Check out the agenda for the technical program, the training program, and the workshops.



Co-located with LISA, the Internet Society is also hosting its ION Conference on the afternoon of Tuesday, December 11th.  ION is a conference series organized by ISOC's Deploy360 programme, which provides deployment information on advanced technologies like IPv6, DNSSEC, secure routing, etc.

I'm moderating a panel session titled "Advancing the Network: Where We've Been, Where We're Headed" - joining me on the panel are Ron Broersma (DREN), Paul Ebersman (Infoblox), John Spence (Nephos6), and Paul Mockapetris (inventor of the DNS). Another panel focussed on DNSSEC is being run by Dan York. The full agenda is available.

Registration for the ION conference is free, but seats are limited.

Hope to see some of you in San Diego ..

Shumon Huque

A targeted survey of DNSSEC deployment

I've been working on a DNS and DNSSEC monitoring project, which is available at http://www.huque.com/app/dnsstat/

It looks at externally visible features of the authoritative DNS service at a selected set of institutions. The original version monitored the roughly 200 members of Internet2. It was mostly for my own benefit, and to better understand what Penn's peers were doing. But it's grown a bit since then, as others have found it useful and made suggestions about other organizations I should monitor and features I should add.

I'm giving a talk about the project at the Joint Techs conference next week in Palo Alto, California. Bill Owens from NySERNET has given me the most suggestions about the project, so I also recruited him to help me do the talk :-)

The talk will be webcast, and the slides are posted at the conference website. If I manage to get to the last few slides, I hope to talk a bit about application uses of DNSSEC, and in particularly DANE/TLSA which a lot of people are excited about - if DANE sees broad adoption, this might be a compelling enough use to motivate a lot more DNSSEC deployment.

I'll give a quick overview of the project here too, and share some preliminary observations about the data.

In addition to Internet2, I now also monitor ESNet and the set of Department of Energy Labs (ESNet is the co-organizer of the Joint Techs conference, with Internet2), the Ivy League (because Penn often likes to benchmark itself against this group), NySERNET (because Bill Owens asked me to), the set of Internet2 GigaPoPs or regional networks, the US News & World Report ranked top 20 universities, the Times Higher Education ranked top 50 universities, a set of leading Tech companies, and all the DNS top level domains.

The following screenshot shows a summary of the per-category statistics as of this writing. DNSSEC deployment is still in a very fledgling state. Only 14 (6.7%) of the Internet2 members have deployed DNSSEC. There is a much higher ratio of IPv6 enabled DNS servers for every category of institution. My definition of "IPv6-enabled" is that at least one of the published nameservers has an IPv6 address, i.e. it is in theory reachable by an IPv6-only DNS recursive resolver.




The next screenshot shows a tabular view of the DNSSEC enabled domains in Internet2. Six of them have deployed NSEC3 (the version of DNSSEC that provides a defense against zone enumeration) - the number of hash iterations range from 5 to 10, and none use the NSEC3 opt-out feature. Two of them (Kansas State and Oklahoma State) have no secure delegation, which is a bit odd, since .EDU has a sole registrar (Educause) that supports delegation signer records. Kansas State does appear to have a DLV record published though, in dlv.isc.org, the DLV registry operated by Internet Systems Consortium. Oklahoma State appears to be truly an island. Penn was the earliest deployer of DNSSEC, in August 2009. If I recall correctly, Berkeley and LSU followed in 2010.

The table is followed by a summary of DNSSEC algorithms in use for the KSK (key signing key) and ZSK (zone signing key).



The text marked in red indicates a problem detected by the monitor. In this particular table, ualr.edu is showing that 5 of 7 nameserver addresses respond to UDP queries, 5 of 7 to TCP queries, and 1 of 3 IPv6 nameserver addresses respond to queries. Clicking on the domain name takes you to the detail page (in this case, http://www.huque.com/app/dnsstat/detail/ualr.edu/ ) which provides additional information that can identify the problematic servers:

DNS/UDP failed: ns2.ualr.edu., 2620:e8:0:10::1 (<class 'dns.exception.Timeout'>, )
DNS/UDP failed: ns.ualr.edu., 2620:e8:0:10::30 (<class 'dns.exception.Timeout'>, )

In this case the same two IPv6 nameserver addresses account for all the detected failures. If we look at the entire set of domains monitored, this turns out to be a pretty frequent problem. Often a sizable number of IPv6 DNS server addresses do not work. Perhaps folks are still figuring out how to properly monitor IPv6 services (and keep them running).


The rest of my remarks and observations will be without accompanying screenshots. You can check the site out yourself for details, but some things might have changed between when I'm writing this and when you're looking at the project site (the monitor updates the data once per week).

The ESNet community has by far the highest proportion of DNSSEC (9 of 11, or 82%) and IPv6 enabled domains (11 of 11, or 100%!). However it's a small sample size, and the likely reason is that these organizations were subject to the US federal government OMB mandate to get IPv6 and DNSSEC deployed.

Three of the sixteen gigapops have DNSSEC: MAGPI (which is run out of Penn), 3ROX (our neighbor in Pennsylvania), and MERIT. Interestingly, none have secure delegations in their parent zones. For MAGPI, this is because our registrar, Network Solutions can't do DS records yet (last I checked), and perhaps a similar situation holds for 3rox.net. Both MAGPI and 3ROX have DLV records at dlv.isc.org. MERIT could get a secure delegation from EDU, but for some reason they don't have a KSK (i.e. a key defined as a secure entry point). Not sure what's going on there.

The situations in the Ivy League and the US News top 20 don't look particularly good. In both categories, Penn is the only DNSSEC signed domain, although there are a sizable number of IPv6 enabled domains. Looking at the Times Higher Edu top 50, which includes several universities outside the US, is a bit more promising. The five in that category with DNSSEC are UC Berkeley, Cambridge University, Carnegie Mellon, Imperial College, and Penn. None of them use NSEC3.

The Top Level Domains appear to be in reasonably good shape, comparatively speaking. Of the 313 TLDs, 97 or 31.0% have DNSSEC, and 268 or 85.6% have IPv6. (Note to self: I have to decide later if I'm going to expand this category to include the swarm of new gTLDs in the pipeline!)

The "Leading Tech Companies" category looks pretty abysmal. This is my list of selected tech companies. If anyone has suggestions about other companies I should have included, I'd be happy to receive them. Of the 44 companies there now, only Comcast has deployed DNSSEC (Philadelphia appears to be a hotbed of DNSSEC activity!). Only 10 have IPv6 reachable DNS servers. Notably, Google and Facebook have no IPv6 DNS authoritative servers. This means that although their websites are available over IPv6, clients using IPv6-only DNS recursive resolvers will not be able to reach them! Google is also the only one that has no EDNS0 support.

I'll end this article with the output of the project's detail page for Google:

DNS zone details for: google.com (Google)

Date of latest check: July 10, 2012, 9:10 a.m.
Time required for check: 0.38 seconds

Zone Summary information:

• 4 Nameserver records (IPv4=4, IPv6=0)
• 4 Nameserver addresses (IPv4=4, IPv6=0)
• Successful DNS/UDP response: 4 of 4 servers
• Successful DNS/TCP response: 4 of 4 servers
• Successful DNS/IPv4 response: 4 of 4 servers
• Successful DNS/IPv6 response: 0 of 0 servers
• Zone supports DNS over TCP queries on all its servers (4 responses of 4)
• DNS service is NOT accessible over IPv6
• Zone does NOT support DNSSEC (DNS Security Extensions)

Debugging information:

Found nameserver: ns4.google.com.
 Found IPv4 address: 216.239.38.10
Found nameserver: ns3.google.com.
 Found IPv4 address: 216.239.36.10
Found nameserver: ns1.google.com.
 Found IPv4 address: 216.239.32.10
Found nameserver: ns2.google.com.
 Found IPv4 address: 216.239.34.10
NS records 4, IP4 4, IP6 0
NS: ns4.google.com. has no IPv6 address record
NS: ns3.google.com. has no IPv6 address record
NS: ns1.google.com. has no IPv6 address record
NS: ns2.google.com. has no IPv6 address record
Trying DNS/UDP query to ns4.google.com., 216.239.38.10
Trying DNS/UDP query to ns3.google.com., 216.239.36.10
Trying DNS/UDP query to ns1.google.com., 216.239.32.10
Trying DNS/UDP query to ns2.google.com., 216.239.34.10
DNS/UDP success: 4 of 4 servers
Trying DNS/TCP query to ns4.google.com., 216.239.38.10
Trying DNS/TCP query to ns3.google.com., 216.239.36.10
Trying DNS/TCP query to ns1.google.com., 216.239.32.10
Trying DNS/TCP query to ns2.google.com., 216.239.34.10
DNS/TCP success: 4 of 4 servers
Trying EDNS0 SOA query to ns4.google.com., 216.239.38.10
Response from 216.239.38.10 doesn't contain EDNS0 OPT RR
Trying EDNS0 SOA query to ns3.google.com., 216.239.36.10
Response from 216.239.36.10 doesn't contain EDNS0 OPT RR
Trying EDNS0 SOA query to ns1.google.com., 216.239.32.10
Response from 216.239.32.10 doesn't contain EDNS0 OPT RR
Trying EDNS0 SOA query to ns2.google.com., 216.239.34.10
Response from 216.239.34.10 doesn't contain EDNS0 OPT RR
DNS/EDNS0 success: 0 of 4 servers