Wednesday, May 30, 2012

IPv6 at Penn

World IPv6 Launch (June 6th 2012) is fast approaching, so I thought I'd share some details about IPv6 deployment at the University of Pennsylvania and what we've recently done to prepare for this event.


A quick history

Penn runs a regional network called MAGPI, which connects Research & Education (R&E) institutions in our area (eastern Pennsylvania, New Jersey, and Delaware) to national R&E backbone networks like Internet2. We first deployed IPv6 in the MAGPI network in mid 2002 and soon after, established an external peering with Internet2. At that time, a small number of engineers in the networking department (including myself) typically had our computers directly wired into MAGPI infrastructure to get IPv6 connectivity at desktops and test servers.

IPv6 was introduced more gradually into the Penn campus network infrastructure, starting in 2005. Initially it was enabled only at the border  and core routers, and extended out to only selected IT departmental subnets. In September 2005, Penn hosted the Fall Internet2 member meeting in Philadelphia, where we operated the conference network at the Wyndham Franklin Plaza hotel - this network was fully IPv6 enabled, including support for IPv6 multicast routing. (Incidentally, we are hosting the Fall 2012 Internet2 meeting this October again, so I hope to see some of you there.)

Over the course of the years since, we've been gradually extending IPv6 network connectivity to the rest of the campus, and turning up IPv6 enabled application services where feasible. Needless to say, it is still early days in IPv6 deployment and a huge amount of work remains to be done.

Network Infrastructure

Unlike other IT services at Penn, many of which are highly decentralized, the campus network is mostly run by the central IT organization - this gave us the ability, when needed, to roll out IPv6 to large portions of the network fairly rapidly. Due to many competing priorities and projects, we have mostly not taken advantage of this ability, until quite recently. IPv6 had been deployed on departmental subnets only where it had explicitly been asked for. One of the more interesting cases was the Annenberg School for Communication - they approached the central IT group a few years ago with a need for IPv6 in order to facilitate some collaboration with partners in China who had asked if they'd be able to conduct video conferencing over IPv6. This was the first time we encountered direct external pressure to deploy IPv6. I'm sure it won't be the last.

The one subdivision within the university that does run their own network infrastructure, the School of Engineering & Applied Science, has been an early adopter, and has been running IPv6 in their part of the network since 2007.

In the summer of 2011, we took advantage of the increased interest generated by last year's World IPv6 Day event to extend the deployment of IPv6 to most of the rest of the campus wired network. The one area that was significantly lagging was the wireless network. This was a bit more challenging because of known bugs in our wireless controller vendor's gear (Aruba Networks) which necessitated a code upgrade. That code upgrade did not happen until earlier this year, so we're still in the midst of IPv6 deployment on wireless. As of this writing, 70 wireless subnets (out of roughly 200) have IPv6 available, and we should have the entire wireless network done sometime later this summer.

For the more technically inclined, we run Integrated IS-IS as our interior routing protocol for IPv6, whereas we continue to run OSPF for IPv4. At the time when we were initially testing IPv6, that was clearly the best choice since OSPF version 3 (the new version of OSPF that supports IPv6) was still in a relatively fledgling state of implementation maturity. Also confining IPv6 to a separate routing protocol seemed like a good additional safety measure. We run a single flat Level-2 area for the entire campus. For exterior routing, we have separate BGP peerings over IPv6 transport established with our external peers that carry IPv6 routes only. Our initial deployment used a provider allocated /48 IPv6 block delegated to us by MAGPI. In 2008, we obtained a Provider Independent ("portable") /32 sized IPv6 address block (2607:F470::/32) from the regional registry ARIN, and have mostly renumbered into it.

Currently, Penn's only connection to the IPv6 Internet is via MAGPI and Internet2. But we're planning to turn up IPv6 peering on our direct commercial ISP links (Level3 and Cogent) in the very near future. At least one of them might happen before World IPv6 Launch.

IPv6 enabled servers use statically configured addresses. Clients on campus almost exclusively use stateless address autoconfiguration (including the privacy/temporary address extensions). DHCPv6 has not been an option for us until recently, since we're a 40% Mac campus, and Apple didn't support DHCPv6 until Mac OS X version 10.7 (late summer 2011).  We are developing plans for a possible DHCPv6 service in the future, which I'll elaborate on at a later time.

Application Services

Penn's authoritative DNS service has been IPv6 enabled for many years. The campus DNS resolvers also support DNS queries over IPv6 but since we don't yet run DHCPv6, we don't have a convenient way to hand out their IPv6 addresses. Our homegrown DNS content management system has supported the ability to create AAAA and IPv6 PTR records for a long time also.

A number of departmental web servers, including the School of Engineering & Applied Science, are IPv6 enabled. The Penn central jabber server,, was one of our earlier IPv6 equipped services, and actually sees a high proportion of IPv6 activity. Work is proceeding on many other services.

Some of the most challenging services have been those where components of the service have been outsourced to commercial third parties. The central Penn webserver, is located on the Akamai content delivery network, and Akamai has been slow to deploy IPv6. We successfully worked with Akamai to put the website on IPv6 for last year's world IPv6 day (June 8th 2011), but they were not then prepared to offer it on an ongoing production basis. In April 2012, Akamai finally announced production IPv6 support. As of May 9th, the Penn website is now available over IPv6, hopefully permanently this time.

Akamai uses DNS resolver client addresses to direct users to content servers geographically close to them (although a few other factors including load are also considered by the server selection algorithm). I collected some data with the help of colleagues about where the AAAA record resolves to from various locations. Since we host a cluster of IPv6-enabled Akamai content  servers on our campus network, most of the time, on-campus users of will be directed to these local servers.

One issue we overlooked, is that there is a version of the main Penn website optimized for small form-factor mobile devices ("") which is not on the Akamai CDN, and run by another unit within the IT organization that has not yet deployed IPv6. So, more work remains to get the Penn web presence completely IPv6 ready.

The other challenging service is central e-mail. Penn uses Message Labs (now Symantec Cloud) to scan e-mail for viruses and spam scoring. As a result both inbound and outbound e-mail has to go through Symantec Cloud's servers. We've inquired about IPv6 support for a number of years, but even today, they appear to have no plans to support it. Our latest communication from them (early May 2012) indicates that they have no plans for any IPv6 support for FY13 (their fiscal year starts in April), and that this may change as priorities shift. At some point, we too might be compelled to shift our priorities and end our relationship with Message Labs, and either seek another provider (does Google/Postini do IPv6 yet?) or bring back virus & spam filtering in-house.

For a comparative view of externally visible IPv6 enabled application services deployed at various US universities and other organizations, Mark Prior's IPv6 survey website is a good resource. Of the five services measured there (Web, DNS, Mail, NTP, and Jabber), Penn gets a green box for four - Mail is the missing one because of Symantec Cloud.

Other Projects

From time to time, we've worked with Penn researchers and outside companies on IPv6 related projects. In the fall of 2009, we worked with Alain Durand (then at Comcast) and Roch Guerin (Penn engineering school faculty) on a small trial deployment of Dual Stack Lite; see RFC 6333 for details of this protocol - this was mostly to help Comcast out. It's unlikely that Penn will deploy DSLite in our own production network. We've also worked with Roch and Comcast on an ongoing IPv6 adoption measurement project. Details of this project are available at:

Facilitating Regional Connectivity

As mentioned earlier, Penn enables IPv6 connectivity for regional institutions via the MAGPI GigaPoP and Internet2. Currently, we provide IPv6 connectivity to the following institutions: Princeton University, New Jersey Edge (the state education network for NJ), Lafayette College, and Rutgers University. Of them, Princeton came up first in 2005.

Traffic Measurements

Looking at some recent data, IPv6 traffic traversing the campus border is roughly 3% of the total inbound and about 1% of the total outbound traffic. Internal traffic is probably a slightly higher percentage. We're just starting to deploy better measurement infrastructure for IPv6, so we'll have more comprehensive data in the future. But I'll be writing another article sharing what we have so far next.


Sunday, May 20, 2012

PICC12 conference recap

A brief summary of my visit to the 2012 Professional IT Community Conference (PICC'12) in New Brunswick, New Jersey on May 12th. As mentioned in a previous posting, this is a small, regional IT and system administration conference run by the New Jersey chapter of the League of Professional System Administrators - LOPSA. It's relatively new - this is only the 3rd time it's being held, but attendance was up significantly this year - about 125 attendees, compared to roughly 80 in each of the previous years.

The conference took place on Friday and Saturday, May 11th-12th. Having too many important meetings at work, I had to miss the various sessions on Friday, including unfortunately, the evening keynote by Bill Cheswick on "Rethinking Passwords". I'm pretty sure I've seen a previous version of that talk though, that he gave at Penn's Distributed Systems Lab a few years ago. Actually, I've been thinking and rethinking passwords for years myself, and have many thoughts on the subject, but I'll leave those for a future post. A version of Ches's talk appears to be online at his website. Next year, I hope to attend both days of the PICC conference.

I took the train from Philly early Saturday morning and arrived at the conference venue, the Hyatt Regency hotel in New Brunswick, at 8:30am. My primary agenda for the day was to give two half-day tutorial/training classes on IPv6 and DNS/DNSSEC.

First up was IPv6 (9am-12:30pm). I delivered a revised version of the IPv6 class I taught at the USENIX LISA 2011 conference in Boston in December. Actually, I taught the IPv6 class at PICC last year too, but I was called in at the last moment to do it, and essentially scrambled to put together a set of slides at very short notice. Although that talk was also well received, the LISA talk and this newest revision is a much more thorough and carefully thought out set of slides and material. The class assumes basic knowledge of TCP/IP networking, and goes from IPv6 basics all the way to relatively advanced topics and touches on some of the latest developments in IPv6 too. Some of the slides cover more material than I actually have time to discuss in detail in such a short course. But they are useful as reference material that can be consulted later.

(Actually, I could easily reuse this same slide deck for a full-day or multi-day course ..)

I had about 25 to 30 attendees, which is a pretty good turnout given the number of competing tutorials and sessions. And they were engaged, asking lots of questions. There were several repeat attendees at my talk from the previous year, so I must be doing something right! :-) If you follow me on twitter, you probably already know that I was presented with an IPv6 Buddy (apparently "the essential tool for IPv6 engineers") by Willard Dennis:

The IPv6 Buddy is a USB keyboard to make it easier to type IPv6 addresses. It has keys for all the hex digits, a double-colon ("::") key, and even a period for entering IPv4-mapped IPv6 addresses. It's cute, but the novelty is already beginning to wear off. It will continue to be a nice conversation piece for my office though (not that I don't have enough toys in my office already)!

The afternoon's DNS and DNSSEC class had a similar turnout, and went quite well too. Again, I had a pretty engaged audience, asking lots of questions. This is a new course that I very recently finished designing, and that I delivered for the first time at this conference. Naturally, I'm very interested in feedback on the material. The slides are available here:

I attended the Saturday evening ending keynote, by Rebecca Mercuri, on "The Black Swan and Information Security". Rebecca got her Ph.D. in Computer Science from Penn actually, and is well known for her work in electronic voting. This was an interesting talk, suggesting correctly I think, that the theories in Taleb's book of that name, have very strong parallels in IT and Information Security, and that Black Swan style outlier events are perhaps more normal than people believe. I was also pleasantly amused to hear her make a reference to my DNS talk when discussing risk analysis (specifically the part where I had discussed the pros and cons of routine key rollovers in DNSSEC - see my slides for details).

If I had more time, there were several talks/sessions I would have like to attend. One of these days, I will finally figure out how to find the time to attend Tom Limoncelli's "Time Management" class, but how can I solve this perplexing catch-22! :-) Willard Dennis did a class on GNS3/Dynamips/Dynagen - I'm interested because I use dynamips and dynagen extensively in a class at Penn's Engineering School: TCOM504, where I teach students about routing protocols (IPv4/IPv6, RIP, OSPF, IS-IS, BGP, Multicast, MPLS, etc).

I took the train back to Philly, arriving home around 8:45pm. It was a long but productive day. PICC is a very nice conference. I hope it grows and continues to be successful.

Sunday, May 6, 2012

Help, Blogger and Javascript!

A friend and ex-colleague of mine sent me the following e-mail message:

"Your blog appears to have no content - all I see is blank pages! I'm not sure if that's because it requires JavaScript, or if it's one of those invisible-ink blogs written in lemon juice for secrecy."

As long as I've known him, he's used text-only browsers that (intentionally) don't do javascript, flash, or many of the other infestations of modern webpages.

PS. although the link to my blog is now "", that's just a custom domain name that points to my google/blogger site. If I had more free time on my hand, I might have considered setting up the blog entirely on my server, but that's a project for another day.

His first guess is correct: the google default appearance for new blogger site appears to use Javascript to load content for individual blog articles. And my blog being very new, and largely uncustomized (I just set it up last week), is setup that way. If/when I desire secrecy, I'll use encryption, rather than invisible ink made of lemon juice! :-)

In any case, I'd like my blog to be readable in minimalist browsers. What's the best way to do this? It appears that some blogger sites can be read by them, eg. Examining the source HTML of that site, shows the article content, sans javascript. Maybe I'll try to switch to a blogger template that does that, if I can figure out which one ..

Friday, May 4, 2012

Penn's DNS zone

Some data from a quick analysis of the contents of the  University of Pennsylvania's primary DNS zone (

  Total RR     = 624221
  Total RR     = 159928 (excluding DNSSEC records)
  Total RRsets = 464295
  Total RRsets = 155165 (exluding DNSSEC records)
  Total Names  = 154570
  TTL min, max, avg = 0, 114000, 38562
  RRtype             Count        %
  A                 145684    23.3%
  AAAA                 388     0.1%
  CNAME               9476     1.5%
  DNSKEY                 3     0.0%
  MX                  1363     0.2%
  NS                    10     0.0%
  NSEC              154564    24.8%
  RRSIG             309723    49.6%
  SOA                    1     0.0%
  SRV                 2961     0.5%
  SSHFP                  4     0.0%
  TXT                   41     0.0%
  TYPE65534              3     0.0%

Penn is mostly one big DNS zone (ignoring the reverse DNS zones for now). We don't delegate subdomains to units and divisions within the university. However, we do have a home grown DNS content management system (called "Assignments") that allows units to manage DNS records within their assigned subdomains inside the same zone. Assignments issues updates to the DNS server using the Dynamic Update protocol authenticated with a static transaction signature (TSIG) key. And DNS records are automatically signed (with generation of related DNSSEC metadata) as they are inserted into the zone. Assignments only supports a limited range of DNS record types today, and since we don't have a lot of inhouse programming resources to continue developing it, longer term we hope to be able to migrate away from it to a more capable opensource or commercial system. One of the challenges in doing so is that the current system is very tightly intertwined with a variety of other systems unrelated to DNS (billing, security contacts, machine inventory data, etc). The authoritative DNS servers themselves run ISC's BIND software.

There are 624,221 resource records in total, but almost three quarters of those are DNSSEC metadata records (RRSIG and NSEC). There are many more RRSIG records than NSEC (there is one NSEC per domain name, whereas there is one RRSIG for every RR set). There are 3 DNSKEY records: 1 Key Signing Key (KSK), and 2 Zone Signing Keys (ZSK). Only one of the ZSKs is active and used to sign zone records; the other is pre-published for future key rollovers. The KSK is 2048-bit RSASHA1, the ZSKs are both 1024-bit RSASHA1. The zone was first signed in August 2009, but we had to wait another year for the parent zone (EDU) and the root to be signed. A secure delegation (DS record) for Penn was installed into the EDU zone in August 2010. The 3 private type records (TYPE65534) are used by BIND to record some dynamic signing state. I gave a talk with more details about Penn's DNSSEC deployment plans at the Summer Internet2 Joint Techs conference in July 2009.

Excluding DNSSEC, there are 159,928 records. The vast majority of them are A records (145,684). Compared to a year ago, that represents an 8% increase. By contrast, the number of AAAA records has more than doubled from 180 to 388. Most of the AAAA records are likely IPv6 enabled servers, so that's probably a promising sign of growth. Penn currently uses stateless autoconfiguration for IPv6 client addresses, and such addresses, by and large, are not registered in the DNS.  In the near future, we're planning a limited production rollout of DHCPv6 to some campus subnets. If that happens, pools of DHCPv6 client address ranges will have pre-populated DNS records, which will drive up the AAAA record count significantly.

The TXT records are mostly used for DKIM and SPF. Although the central Penn mail servers do not currently use either, a number of departmental and school mail servers do. E-mail, like many other services at Penn, is highly decentralized.

There are 5 nameserver (NS) records at the zone apex. Two are Penn servers with both IPv4 and IPv6 address records. Two are servers operated by the University of Delaware, with whom we have a reciprocal secondary DNS service arrangement - these have IPv4 addresses only. And the last one is, a dualstack global anycast cloud operated by Internet Systems Consortium. SNS-PB is the public benefit version of ISC's Secondary Name Service.

  ;; [nameserver names]        86400    IN    NS        86400    IN    NS        86400    IN    NS        86400    IN    NS        86400    IN    NS

  ;; [nameserver addresses]        86400    IN    A        86400    IN    A  86400    IN    A  86400    IN    AAAA    2607:f470:1002::2:2  86400    IN    A  86400    IN    AAAA    2607:f470:1003::3:3       7200     IN    A       7200     IN    AAAA    2001:500:2e::1

I'm looking forward to more application use of DNSSEC signed data to verify crypto material. There are a handful of SSHFP records today (mostly my own test systems). Hopefully, we'll see some DANE/TLSA records in the near future.