TIP2013 - Day 1

Sunday, January 13th. Slightly jet lagged, but I was able to sleep intermittently until 4:45am, Hawaii time.

My computer gear on this trip, pictured below, comprises an 11" MacBook Air laptop, iPhone5, iPad2, and a portable 2TB drive. Taking the tiny MacBook Air rather than my normal bulky 15.4" MacBook pro laptop makes a big difference. Not as comfortable to type, but much much lighter. In encountered one unanticipated problem though. As I mentioned in the previous post, I was upgraded to a better room in the Ainahau tower, where I discovered they have no wireless Internet service, just wired ethernet (and the MacBook Air doesn't have an integrated ethernet port). I can't remember the last hotel (in this country) where I didn't have wireless, so it didn't even occur to me to bring the Air's separate USB ethernet dongle. Luckily, my colleague Richard Machida had one with him which he loaned to me a day later.

I took the 7:30am Joint Techs shuttle bus to the University of Hawaii campus. I sat next to Kevin Mayeshiro, from the University of California at Davis - who already knew who I was. It turns out my colleague Colleen Szymanik has spoken to him as a reference for Brocade Networks - Penn is evaluating Brocade as a candidate vendor for the next refresh of our campus core routers. Kevin grew up in Hawaii, so it was good to chat with someone with extensive local knowledge of our venue.

The conference venue is the University of Hawaii's "East West Center", pictured below with the line of folks waiting for registration/bag pickup.

Sunday is customarily a day occupied by tutorial sessions. I attended sessions on BGP Rover, RPKI, and DNSSEC.

Below: Dan Massey form Colorado State University talks about the BGP ROVER route origin verification system.

Below: RPKI (Resource PKI) tutorial/demo session run by Randy Bush of IIJ.

Below: DNSSEC tutorial by Michael Sinatra of ESnet.

In the evening, back in the hotel, I attended the Internet2 Connector's meeting. This is closed meeting of Internet2 and representatives of Internet2 connector networks (GigaPoPs, RONs etc). I usually represent the MAGPI Gigapop, which is a regional R&E Network run by Penn.

-- Shumon Huque

I'm in (so called) Paradise

[Actual date for the photos in this article: Saturday, January 12th 2013]

My colleague, Richard Machida (U of Alaska, Fairbanks), has been blogging from Hawaii, so I felt motivated to do the same. I'll post follow-up articles for each day of the conference.

I'm in Oahu, Hawaii to attend the Winter 2013 Joint Techs conference, hosted by the University of Hawaii in Manoa. Joint Techs is billed as an international conference of networking experts - and is one of the primary technical conferences for the US Research & Education networking community. Whenever in Hawaii, this conference conveniently changes its name to "Techs in Paradise". Also when in Hawaii, it is usually co-located with the APAN (Asia Pacific Advanced Network) meeting - APAN is the Asia Pacific's R&E network/organization. This time, a new format is being tried with a fully merged Joint Techs/APAN conference. As a result, we have quite a large attendance - almost 500 people, stressing the conference facilities a bit.

I chose an aggressive flight schedule that had me leaving Philly at 6:30am and arriving at 1:30pm in Honolulu, if I made the 32 minute (!) connection in San Francisco. Thankfully I did, so I had some daylight to walk around Waikiki a bit and take some photos.

Weather on Oahu so far has been sunny with periods of rain, and temperatures reaching the high 70's (farenheit). Temperature forecast from the local news:

Here are some photos from my flight and from around Waikiki. The complete and growing set, which includes photos from the TIP2013 conference too can be seen on my Google Plus album.

An aerial shot of the Golden Gate bridge and the Presidio Park, from my flight out of San Francisco, shortly after take-off.

 Some aerial shots of Oahu from my flight into Hawaii. The first shows a portion of Eastern Oahu including the Mokapu peninsula and Kailua. If I was on the left side of the plane, I might have been able to get a few photos from directly above Diamond Head, but I saw a lot of clouds in that direction, so it may not have been visible (I have some good shots of that side from a previous trip which I might post later).

The following photo is from directly above Waikiki beach (a detailed crop of a larger photo I took). A large part of Waikiki, Waikiki Beach, and a portion of the Ala Wai canal (which diverts mountain stream water around Waikiki) can be seen.

Diamond Head, probably Hawaii's most famous landmark, is a dormant volcanic crater, created by a relatively recent eruption (ie. geologically speaking; 150,000 years ago). The south western tip, the highest point of the crater rim stands 761 feet high. It's possible to hike all the way to the top (which I've done several times before).

 Another view from Kapiolani Beach Park (a much more tranquil place than Waikiki Beach):

The conference hotel is the Sheraton Princess Kaiulani, which is about a block from the beach. I got upgraded to a "partial ocean view" room due to my SPG gold status, but it isn't really much of a view:

At the last TIP (2008), we occupied a swankier hotel, the Sheraton Waikiki, which was right on the beach, and where I also got upgraded to an ocean view room. Here's a photo from my room at that time - it was designed so that you could seen Diamond Head while you showered from the bathroom (top half of shower curtain was transparent), which was a bit weird. The Sheraton Waikiki has since become even more upscale and I'm told its prices are now out of the range of Joint Techs.

A few other photos:

Link to rest of photos.

Shumon Huque

Stanford Linear Accelerator tour

At the recent Joint Techs conference, our host Stanford University arranged a lunch time tour of the Stanford Linear Accelerator Center (SLAC) for a small group of attendees. I signed up early as I knew it was going to popular with this crowd. SLAC is a 50 GeV electron-positron accelerator operated by Stanford on behalf of the US Dept of Energy.

(SLAC is now officially known as the "SLAC National Accelerator Lab".)

This is the second particle accelerator site I've visited. In the summer of 2007 (also during a Joint Techs conference), I saw Fermilab and the 2 TeV Tevatron, at that time the world's most powerful particle accelerator. My annotated photos from that visit are on flickr. The Tevatron ceased operation in late 2011. One of these days, I hope to visit the Large Hadron Collider at CERN.

Here are some photos from the SLAC tour. The full set is also available.

Bebo White, a noted SLAC computational physicist, was our tour guide. He's an entertaining and jolly character, who by his own admission looks like Santa Claus :-)

How the LINAC works:

Hmm, Science has a long way to go!

SLAC had the first website in North America (the first in the world of course was at CERN, where Tim Berners-Lee worked). Seeing a NeXT Cube brings back memories. That was my main computer for most of my undergraduate years at Penn, when I worked for NeXT as a campus consultant.

Mock-up of accelerator components just outside the end of the Klystron gallery:

Facilities photo/map:

The Klystron gallery along the nearly 2-mile long building that houses the main linac. Klystrons are the key driving force for the accelerator and produce amplified electromagnetic carrier waves that help boost the electrons along the beam line. The Klystrons are to the left of this photo, and sit above the main accelerator beam line which is situated 25-feet beneath the ground.  There are 240 Klystrons in total. The particles are injected at the other end of this tunnel.

Our merry tour group ...

A targeted survey of DNSSEC deployment

I've been working on a DNS and DNSSEC monitoring project, which is available at http://www.huque.com/app/dnsstat/

It looks at externally visible features of the authoritative DNS service at a selected set of institutions. The original version monitored the roughly 200 members of Internet2. It was mostly for my own benefit, and to better understand what Penn's peers were doing. But it's grown a bit since then, as others have found it useful and made suggestions about other organizations I should monitor and features I should add.

I'm giving a talk about the project at the Joint Techs conference next week in Palo Alto, California. Bill Owens from NySERNET has given me the most suggestions about the project, so I also recruited him to help me do the talk :-)

The talk will be webcast, and the slides are posted at the conference website. If I manage to get to the last few slides, I hope to talk a bit about application uses of DNSSEC, and in particularly DANE/TLSA which a lot of people are excited about - if DANE sees broad adoption, this might be a compelling enough use to motivate a lot more DNSSEC deployment.

I'll give a quick overview of the project here too, and share some preliminary observations about the data.

In addition to Internet2, I now also monitor ESNet and the set of Department of Energy Labs (ESNet is the co-organizer of the Joint Techs conference, with Internet2), the Ivy League (because Penn often likes to benchmark itself against this group), NySERNET (because Bill Owens asked me to), the set of Internet2 GigaPoPs or regional networks, the US News & World Report ranked top 20 universities, the Times Higher Education ranked top 50 universities, a set of leading Tech companies, and all the DNS top level domains.

The following screenshot shows a summary of the per-category statistics as of this writing. DNSSEC deployment is still in a very fledgling state. Only 14 (6.7%) of the Internet2 members have deployed DNSSEC. There is a much higher ratio of IPv6 enabled DNS servers for every category of institution. My definition of "IPv6-enabled" is that at least one of the published nameservers has an IPv6 address, i.e. it is in theory reachable by an IPv6-only DNS recursive resolver.




The next screenshot shows a tabular view of the DNSSEC enabled domains in Internet2. Six of them have deployed NSEC3 (the version of DNSSEC that provides a defense against zone enumeration) - the number of hash iterations range from 5 to 10, and none use the NSEC3 opt-out feature. Two of them (Kansas State and Oklahoma State) have no secure delegation, which is a bit odd, since .EDU has a sole registrar (Educause) that supports delegation signer records. Kansas State does appear to have a DLV record published though, in dlv.isc.org, the DLV registry operated by Internet Systems Consortium. Oklahoma State appears to be truly an island. Penn was the earliest deployer of DNSSEC, in August 2009. If I recall correctly, Berkeley and LSU followed in 2010.

The table is followed by a summary of DNSSEC algorithms in use for the KSK (key signing key) and ZSK (zone signing key).



The text marked in red indicates a problem detected by the monitor. In this particular table, ualr.edu is showing that 5 of 7 nameserver addresses respond to UDP queries, 5 of 7 to TCP queries, and 1 of 3 IPv6 nameserver addresses respond to queries. Clicking on the domain name takes you to the detail page (in this case, http://www.huque.com/app/dnsstat/detail/ualr.edu/ ) which provides additional information that can identify the problematic servers:

DNS/UDP failed: ns2.ualr.edu., 2620:e8:0:10::1 (<class 'dns.exception.Timeout'>, )
DNS/UDP failed: ns.ualr.edu., 2620:e8:0:10::30 (<class 'dns.exception.Timeout'>, )

In this case the same two IPv6 nameserver addresses account for all the detected failures. If we look at the entire set of domains monitored, this turns out to be a pretty frequent problem. Often a sizable number of IPv6 DNS server addresses do not work. Perhaps folks are still figuring out how to properly monitor IPv6 services (and keep them running).


The rest of my remarks and observations will be without accompanying screenshots. You can check the site out yourself for details, but some things might have changed between when I'm writing this and when you're looking at the project site (the monitor updates the data once per week).

The ESNet community has by far the highest proportion of DNSSEC (9 of 11, or 82%) and IPv6 enabled domains (11 of 11, or 100%!). However it's a small sample size, and the likely reason is that these organizations were subject to the US federal government OMB mandate to get IPv6 and DNSSEC deployed.

Three of the sixteen gigapops have DNSSEC: MAGPI (which is run out of Penn), 3ROX (our neighbor in Pennsylvania), and MERIT. Interestingly, none have secure delegations in their parent zones. For MAGPI, this is because our registrar, Network Solutions can't do DS records yet (last I checked), and perhaps a similar situation holds for 3rox.net. Both MAGPI and 3ROX have DLV records at dlv.isc.org. MERIT could get a secure delegation from EDU, but for some reason they don't have a KSK (i.e. a key defined as a secure entry point). Not sure what's going on there.

The situations in the Ivy League and the US News top 20 don't look particularly good. In both categories, Penn is the only DNSSEC signed domain, although there are a sizable number of IPv6 enabled domains. Looking at the Times Higher Edu top 50, which includes several universities outside the US, is a bit more promising. The five in that category with DNSSEC are UC Berkeley, Cambridge University, Carnegie Mellon, Imperial College, and Penn. None of them use NSEC3.

The Top Level Domains appear to be in reasonably good shape, comparatively speaking. Of the 313 TLDs, 97 or 31.0% have DNSSEC, and 268 or 85.6% have IPv6. (Note to self: I have to decide later if I'm going to expand this category to include the swarm of new gTLDs in the pipeline!)

The "Leading Tech Companies" category looks pretty abysmal. This is my list of selected tech companies. If anyone has suggestions about other companies I should have included, I'd be happy to receive them. Of the 44 companies there now, only Comcast has deployed DNSSEC (Philadelphia appears to be a hotbed of DNSSEC activity!). Only 10 have IPv6 reachable DNS servers. Notably, Google and Facebook have no IPv6 DNS authoritative servers. This means that although their websites are available over IPv6, clients using IPv6-only DNS recursive resolvers will not be able to reach them! Google is also the only one that has no EDNS0 support.

I'll end this article with the output of the project's detail page for Google:

DNS zone details for: google.com (Google)

Date of latest check: July 10, 2012, 9:10 a.m.
Time required for check: 0.38 seconds

Zone Summary information:

• 4 Nameserver records (IPv4=4, IPv6=0)
• 4 Nameserver addresses (IPv4=4, IPv6=0)
• Successful DNS/UDP response: 4 of 4 servers
• Successful DNS/TCP response: 4 of 4 servers
• Successful DNS/IPv4 response: 4 of 4 servers
• Successful DNS/IPv6 response: 0 of 0 servers
• Zone supports DNS over TCP queries on all its servers (4 responses of 4)
• DNS service is NOT accessible over IPv6
• Zone does NOT support DNSSEC (DNS Security Extensions)

Debugging information:

Found nameserver: ns4.google.com.
 Found IPv4 address: 216.239.38.10
Found nameserver: ns3.google.com.
 Found IPv4 address: 216.239.36.10
Found nameserver: ns1.google.com.
 Found IPv4 address: 216.239.32.10
Found nameserver: ns2.google.com.
 Found IPv4 address: 216.239.34.10
NS records 4, IP4 4, IP6 0
NS: ns4.google.com. has no IPv6 address record
NS: ns3.google.com. has no IPv6 address record
NS: ns1.google.com. has no IPv6 address record
NS: ns2.google.com. has no IPv6 address record
Trying DNS/UDP query to ns4.google.com., 216.239.38.10
Trying DNS/UDP query to ns3.google.com., 216.239.36.10
Trying DNS/UDP query to ns1.google.com., 216.239.32.10
Trying DNS/UDP query to ns2.google.com., 216.239.34.10
DNS/UDP success: 4 of 4 servers
Trying DNS/TCP query to ns4.google.com., 216.239.38.10
Trying DNS/TCP query to ns3.google.com., 216.239.36.10
Trying DNS/TCP query to ns1.google.com., 216.239.32.10
Trying DNS/TCP query to ns2.google.com., 216.239.34.10
DNS/TCP success: 4 of 4 servers
Trying EDNS0 SOA query to ns4.google.com., 216.239.38.10
Response from 216.239.38.10 doesn't contain EDNS0 OPT RR
Trying EDNS0 SOA query to ns3.google.com., 216.239.36.10
Response from 216.239.36.10 doesn't contain EDNS0 OPT RR
Trying EDNS0 SOA query to ns1.google.com., 216.239.32.10
Response from 216.239.32.10 doesn't contain EDNS0 OPT RR
Trying EDNS0 SOA query to ns2.google.com., 216.239.34.10
Response from 216.239.34.10 doesn't contain EDNS0 OPT RR
DNS/EDNS0 success: 0 of 4 servers