I've left Penn for a new job

After more than 20 years of working at Penn (University of Pennsylvania), I've decided to take a new job as Principal Research Scientist at Verisign Labs, the applied research division of Verisign Inc. You might know that Verisign is one of the world's largest DNS infrastructure providers. It runs the .com, .net, .edu, and .gov DNS top level domains, two of the thirteen DNS root servers (A and J), and performs the critically important root zone management function. Verisign also provides managed DNS, Distributed Denial of Service (DDoS) mitigation, and several other services. (Note: Verisign's certificate services business was sold off to Symantec several years ago).

I've been at Penn for so long, that I startled many of my colleagues with the news of my departure, so I'll say a few things about how this came about. I originally came to Penn in 1989 as an undergraduate. I became a full time IT staff member after I graduated - my first job was the system administrator of the new e-mail server for the School of Arts & Sciences, the largest school within Penn. I completed a Masters degree in Computer Science part-time while working. I later moved to the central IT organization, where I remained until last week, first as a Network Engineer, then as the Lead Engineer, and most recently as an Engineering Director. In addition, I've been a part time Adjunct Faculty in Penn's School of Engineering, teaching a laboratory course on Network Protocols.

I was approached by Allison Mankin and Matt Larson (I've known both of them for a while) about a year ago to see if I'd be interested in considering a research scientist position at Verisign Labs. Matt has since left Verisign to work at Dyn, and Allison is now a Director at Verisign Labs. At the time, I thought this was distinctly a long shot, but over the course of many months, I thought more seriously about the possibility. I visited Verisign Labs in September 2013, did an interview with them, and also met Burt Kaliski, Verisign CTO (Burt is a noted cryptographer and was the founding scientist of RSA Labs, and the developer of the PKCS standards). I kept in touch with Allison since, but it took me until the beginning of this year to finally come to the conclusion that I wanted to take this opportunity, so here I am. My first day at Verisign will be tomorrow (March 17th).

I've enjoyed my job and career at Penn a lot. I've been extensively involved in a very diverse range of technical projects, ranging from software development, systems engineering, and network design. Among others, I was responsible for the design and operation of much of the authentication and security infrastructure at Penn, as well as a variety of other services, like the DNS and DHCP (and many more that I don't have the time to enumerate here). I was the principal architect of IPv6 and DNSSEC deployment at Penn. I was the chief engineer of the MAGPI gigapop and through that role, was involved in R&E regional and national networking activities. Increasingly though, a lot of time is taken up by non-technical activities, and the longer I stay at Penn, the greater the possibility that I'll end up as a full time IT staff manager, which isn't the role I envision for myself. I've always seen my primary role as that of a technologist, and this job change will allow me to continue in that direction.

Verisign Labs appears to have a true applied research agenda, which I find appealing. Obviously DNS and DNSSEC research is an area in which I expect to be working. But there are many other interesting areas of work: routing, IPv6, reputation systems, security protocols, future internet architectures, etc. I'm also looking forward to attending more NANOG and IETF meetings to get myself more plugged into those communities than I've been able to thus far. Verisign Labs also has frequent, productive collaborations with computer science researchers through its university collobaration program.

In the process, I'm moving to the Washington DC metro area (specifically Reston, VA) where Verisign is located, another big change for me!

More later ..

--Shumon Huque.

IPv6 versus IPv4 Performance

Yesterday from a post at the Deploy360 website, I learned of Comcast's IPv4 and IPv6 network speed testing tool:

        http://speedtest.comcast.net/

I did a quick test from my laptop in my office and got some very surprising results. The measured IPv6 performance was better than IPv4 by a gigantic margin. With IPv6, I got 822 Mbps download and 667Mbps download throughput. With IPv4, a mere 99Mbps upload and 18Mbps download!

Something seemed fishy, but I had to run off to other work, so I quickly posted the result to Twitter, planning to look into it later.

This generated quite a bit discussion with numerous folks on twitter and elsewhere. My initial speculation was that we do some rate limiting of IPv4 traffic at the Penn border routers for selected areas of the campus, and perhaps this was throttling the IPv4 performance. My other suspicion was that there was something significantly different in the IPv4 and IPv6 routing paths contributing to the difference. The graphic above does show a round-trip time difference of 63ms for the IPv4 path and 32ms for the IPv6 path, which suggests this. Furthermore, if the TCP window is not scaled properly to keep the pipe filled for this path at 63ms (but was for 32ms), then that would decrease throughput also - but not enough to account by itself for the observed difference.

Patrik Falstrom suspected a DPI device or other middlebox causing the problem. The only problem is that we don't have any such middleboxes (unless you consider an IP border router imposing IP address based rate limits a middlebox). In any case, I was leaning towards the rate limits as the cause myself, until I confirmed that those rate limits weren't being applied to any of the traffic from my office network. The rate limits are primarily targeted at the student residential dormitories - without them, our external links typically get overwhelmed with traffic to/from the dorms (most likely due to file sharing, a very common activity on college campuses). The border routers are configured to apply a token bucket rate policer to each individual IPv4 address within the network prefixes that cover the residential networks. Note that this rate limiting is completely application agnostic.  Also note that this scheme cannot scale to IPv6 (a single IPv6 subnet has more than 18 quintillion addresses!), a problem we're ignoring for the time being :-)

Repeat of the test

This morning, I decided to do another test (same laptop), but more carefully, and along with a packet capture. I also explicitly turned off the wireless interface (hmmmm) to make sure that all tests were using the wired gigabit ethernet interface. This time, I got much more reasonable looking results, both address families in the neighborhood of each other: IPv4 853Mbps down, 547Mbps up, and for IPv6 827Mbps down, 730Mbps up. One other difference I notice is that the roundtrip (ping) times to the destination server are 12ms for both IPv4 and IPv6. This is substantially different from yesterday's test (63 and 32ms respectively) despite the fact that I choose the same destination server at Comcast (Washington, DC).

A packet capture reveals that the destination server at Comcast for IPv4 was 68.87.73.52, and for IPv6 was 2001:558:1010:5:68:87:73:52. Are these the same endpoint? Hard to tell, but the fact that the last 4 fields of the IPv6 address spell out the IPv4 address in decimal might be a hint. The traffic streams use TCP port 5050. A traceroute to the IPv4 destination shows the outbound path takes one of Penn's commercial ISP links (Cogent) to New York and then back to Washington/VA. An IPv6 traceroute shows the outbound path goes out via our Internet2 link, the I2 commercial peering service, then Cogent (New York), Level3 (New York), and then Comcast to DC. So the IPv4 and IPv6 paths are substantially different in the forward direction. Harder to tell the path for the return traffic without the aid of some reverse traceroute tools or similar.

Getting a substantial fraction of a gigabit ethernet is not suprising - that's probably the bottleneck bandwidth along the measured path. My laptop has a gigabit ethernet connection to the building network, which in turn has dual 10 Gigabit Ethernet links to a 100 Gig campus core, and then multiple 10Gig links out to commercial ISPs/Internet2 etc. Most tier-1 ISP links and peerings are typically at least 10Gig.

The bandwidth-delay product on these paths is about 1,464 KB (1000Mbps * 12ms). The Comcast endpoint's receive window exceeds this, but my laptop's is slightly undersized, so I could probably do a bit of host tuning to boost the download numbers a bit more.

So, what's the explanation for the strange results I got yesterday? I wish had a packet capture to investigate, but my leading suspicion is that my laptop's wireless adapter (lower bandwidth, shared medium) was used in the IPv4 test, and the wired connection for the IPv6 one. If I have time later, I'll try to reproduce the issue.

--Shumon Huque

---

Addendum (February 9th 2014) - On closer inspection of the packet trace, the speed test appears to use multiple TCP streams in parallel, so scaling the window as high as the bw*delay product of the path isn't necessary.

DNS Amplification Attacks

There has been a lot of talk recently about DNS amplification attacks (with prominent news reports of high bandwidth attacks targeted at anti-spam services, cloud providers, financial institutions, etc). These are a class of denial of service attack that use DNS servers to emit large amounts of traffic onto unsuspecting victims. The attackers use the forged source addresses of their victims to send a large stream of queries to the DNS servers, and this results in a much larger stream of responses being sent from those DNS servers back to the victim computers - with the aim of overwhelming them or their network connections. Although the DNS servers employed in the attack can become adversely impacted, the primary target of the attacks are the computers whose addresses are being forged.

Attacks that employ IP source address forgery are effective on the Internet today, because the countermeasures that would prevent this forgery are not very widely deployed. BCP 38 (a 13 year old document!) and BCP 84 describe network ingress filtering as a way for ISPs to block forged traffic from their customer networks, but many ISPs fail to employ it. Organizations should also ideally configure their networks not to permit internally generated traffic with forged addresses from crossing their borders - typically a border Access Control List or more granular techniques like unicast reverse path forwarding (URPF, a per-subnet level antispoofing method) can be used to do this, but once again, these are not in very widespread use.

In the past, the attacks commonly employed 'open recursive DNS resolvers' - these are DNS recursive resolvers that will resolve names for any client computer without restriction. When such servers are available, the attacker can use a DNS record of their choosing (possibly under the attackers control) that intentionally generates a very large response. To prevent such abuse, many organizations these days lock down their recursive resolvers so that they answer requests only from clients on their own networks. Penn does this also. There are however public DNS resolver services, like Google DNS, and OpenDNS, that by design are open to the world, and so need to have effective countermeasures in place to deal with these potential attacks. Both Google and OpenDNS say that they do.

There are still a huge number of open recursive DNS resolvers on the Internet, the vast majority of which are probably unintentionally so. The Open DNS Resolver Project has been cataloging them and reports a number in the neighborhood of 25 million!

But authoritative DNS servers (which need to be open to the world) are also quite vulnerable to amplification attacks. In this case, the attacker cannot choose an arbitrary DNS record, but instead must use only records that already exist in the authoritative server's zones. However, DNS responses are usually always amplifying (ie. the size of the response is a lot larger than the size of the request), so it's only a question of the scale of attack that can be achieved.

Interestingly, DNSSEC, a security enhancement to the DNS protocol, makes the amplification problem significantly worse, since DNS responses with cryptographic signatures are much bigger than normal, unsigned DNS responses. Hence lately, we've been seing a lot of these attacks target authoritative servers with zones that are signed with DNSSEC.

Penn was one of the earliest organizations to deploy DNSSEC (August 2009, well before the root and most of the TLDs were signed). We first noticed such attacks on our authoritative servers in July of last year (2012) - at that time we didn't have a good way to counteract these, but our server and network infrastructure was easily able to absorb the attacks, so we didn't take any action - the attacks continued for a few weeks and then disappeared. In late January 2013 though, a new round of amplification attacks happened on a much larger scale that did negatively affect our infrastructure, causing high load on two of our servers and almost saturating the outbound bandwidth on their network connections. By this time, code to implement an experimental response rate limiting countermeasure was available (see Vernon Schryver and Paul Vixie's Response Rate Limiting (RRL) -- implementations are available for popular DNS server software such as BIND, NSD, etc). We deployed these enhancements shortly after the new attacks, and they have effectively addressed the problem for the time being. The RRL code works by keeping track of client requests, and for repeated requests for the same records from the same client addresses it rate limits the responses, either by silently ignoring some requests, or providing small 'truncated' responses. The working assumption is that well behaved DNS resolvers cache responses for the advertised TTL of the record and so should not be making repeated queries for the same record in a short period of time. The truncated responses when encountered will cause well behaved DNS resolvers to retry their query over TCP, which cannot be effectively used in forged address attacks. More details of how this works are available in this technical note describing the operation of RRL. I've heard that the RRL extensions are planned to be incorporated officially into BIND 9.10, although from the recent announcement about ISC's new DNSco subsidiary, it isn't clear whether this feature will be available only to commercial customers.

Any record that produces a large response can be effectively employed in these attacks. Most of the attacks to date though have been using the DNS resource record type 'ANY' (RR type 255). This query, when directed to an authoritative server's zone name, returns all records at the apex of the zone. With DNSSEC, you'll additionally get DNSKEY, RRSIG, and NSEC/NSEC3 records. To give an idea of the scale of the amplification, a DNSSEC-enabled "upenn.edu, ANY" query generates a response that is roughly 88 times as large as the request (a query of about 38 bytes, and a response of 3,344 bytes). The actual amplification ratio of the bits on the wire is less than this because we have to consider the encapsulating headers (L2, IP, and UDP). With Ethernet (14 bytes) and IPv4 (20 bytes) and UDP (8 bytes), a 38-byte DNS query occupys an 80-byte Ethernet frame. The 3,344 byte DNS response packet exceeds the Ethernet MTU and is fragmented into 3 ethernet frames, totalling 3,470 bytes. This yields an amplification ratio of about 40x, so in the absence of rate limiting countermeasures, a 1Mbps stream of query traffic (about 1,500 queries/second) would have produced a 40Mbps stream of traffic directed towards the victim.

Even non-DNSSEC zones produce a quite substantial amplification though, and often it's easily sufficient for an effective attack. From discussion with a number of colleagues at  other institutions, it's clear that non-DNSSEC sites have also been undergoing the same types of attacks and have had to deploy rate limiting countermeasures.

Some people have proposed not answering requests for ANY (and after all ANY was only meant to be a diagnostic tool and not intended for production uses). This might buy time until attackers adapt to using other records. But it could cause collateral damage also. It turns out there is a variety of software that uses ANY for different purposes. For example, the PowerDNS recursor uses it to obtain A and AAAA records from authority servers in one query and response (normally two query/responses would be required).

So, what can be done in the long term? The RRL implementations appear to be working very well at many sites, but attackers will undoubtedly adapt their methods - perhaps by performing more highly distributed attacks across many authoritative servers using a larger set of record types. Some folks at NLnet Labs have written a good paper that discusses some of the issues: Defending against DNS reflection amplification attacks. Also, although RRL has been designed very carefully to minimize collateral damage, there will still be situations in which it might not work very well, eg. when dealing with resolvers behind large scale NATs and CGNs - a problem which might be increasingly common as we approach IPv4 depletion.

There doesn't seem to be much hope (or incentive) for widescale deployment of BCP38 and other methods to reduce the scope of source address forgery.

Ideas have also been proposed in the IETF to enhance the DNS query/response conversation with lightweight authentication cookies, which might thwart most forgery based attacks. See http://tools.ietf.org/html/draft-eastlake-dnsext-cookies-03 for example. But they would require widescale updates to a lot of DNS software to have an effect, and have thus far not gained much traction.

Forcing DNS queries to use TCP is probably a much too heavyweight solution that will impose large costs in terms of DNS response latency and server resource requirements, although the experimental TCP cookie transactions extension (see RFC 6013 - http://tools.ietf.org/html/rfc6013 and this USENIX paper) aims to address some of the issues. It may be necessary to consider a TCP based solution in light of some operational problems observed with UDP and large DNS packets - for example firewalls and other middle boxes that do not pass through IP fragments, or that botch up handling of extension mechanisms like EDNS0 that negotiate the use large UDP/DNS payloads.

Response amplification survey at Internet2 schools

I was interested in knowing what size amplification the ANY query would produce at some other sites, so I wrote a quick program to do this and tabulate the results. I chose the set of 210 Internet2 universities that I already monitor at my dnsstat website. For each zone, I ran an EDNS0 (DO=1) DNS query for ANY at the zone apex via each of the authoritative servers for the zone, and measured the query and response size and the resulting amplification (response_size/query_size). The full set of data can be seen here, but I'll just excerpt some of the entries from the beginning of the file, sorted by descending order of response amplification. As expected the largest amplifications (in the neighborhood of 100x for DNS payloads, 50x for full packets) are all from DNSSEC signed zones. But many non DNSSEC zones produce large amplifications too. The very low amplification ratios for the zones towards the end of the dataset are mostly due to DNS servers that don't understand EDNS0 and return FORMERR (format erorr) responses.

#########################################################################
# DNS response amplification results for ANY query at zone apex.
# Data collected 2013-04-10
# Sorted by descending order of response amplification ratio (last column)
# Amplification is the ratio of the DNS response and request payloads only,
# it doesn't include the encapsulated UDP/IP/L2 etc headers.
# Line Format:
# zone ns_name ns_address query_size response_size amp1 amp2
# amp1 is the ratio of DNS response payload to DNS query payload
# amp2 is the estimated ratio of the entire response packet(s) to request
# packets, assuming an ethernet path MTU.
#########################################################################
ksu.edu nic.kanren.net. 164.113.192.242 36.0 4085.0 113.47 53.99
ksu.edu kic.kanren.net. 164.113.92.250 36.0 4085.0 113.47 53.99
umbc.edu UMBC3.umbc.edu. 130.85.1.3 37.0 4093.0 110.62 53.41
lsu.edu phloem.uoregon.edu. 128.223.32.35 36.0 4016.0 111.56 53.10
lsu.edu bigdog.lsu.edu. 192.16.176.1 36.0 4016.0 111.56 53.10
umbc.edu UMBC5.umbc.edu. 130.85.1.5 37.0 4045.0 109.32 52.80
umbc.edu UMBC4.umbc.edu. 130.85.1.4 37.0 4045.0 109.32 52.80
sdsmt.edu ns5.gratisdns.dk. 85.17.221.46 38.0 4095.0 107.76 52.76
sdsmt.edu ns4.gratisdns.dk. 87.73.3.3 38.0 4095.0 107.76 52.76
sdsmt.edu ns2.gratisdns.dk. 208.43.238.42 38.0 4095.0 107.76 52.76
sdsmt.edu ns1.gratisdns.dk. 109.238.48.13 38.0 4095.0 107.76 52.76
uiowa.edu dns3.uiowa.edu. 128.255.1.27 38.0 4093.0 107.71 52.74
uiowa.edu dns2.uiowa.edu. 128.255.64.26 38.0 4093.0 107.71 52.74
uiowa.edu dns1.uiowa.edu. 128.255.1.26 38.0 4093.0 107.71 52.74
lsu.edu otc-dns2.lsu.edu. 130.39.254.30 36.0 3972.0 110.33 52.54
lsu.edu otc-dns1.lsu.edu. 130.39.3.5 36.0 3972.0 110.33 52.54
ucr.edu adns2.berkeley.edu. 128.32.136.14 36.0 3965.0 110.14 52.45
ucr.edu adns1.berkeley.edu. 128.32.136.3 36.0 3965.0 110.14 52.45
ualr.edu ns4.ualr.edu. 130.184.15.85 37.0 3979.0 107.54 51.96
ualr.edu ns3.ualr.edu. 144.167.5.50 37.0 3979.0 107.54 51.96
ualr.edu ns2.ualr.edu. 144.167.10.1 37.0 3979.0 107.54 51.96
ualr.edu ns.ualr.edu. 144.167.10.48 37.0 3979.0 107.54 51.96
uiowa.edu sns-pb.isc.org. 192.5.4.1 38.0 4013.0 105.61 51.74
sdsmt.edu ns3.gratisdns.dk. 194.0.2.6 38.0 3963.0 104.29 51.11
berkeley.edu ns.v6.berkeley.edu. 128.32.136.6 41.0 4040.0 98.54 50.19
berkeley.edu adns2.berkeley.edu. 128.32.136.14 41.0 4040.0 98.54 50.19
berkeley.edu adns1.berkeley.edu. 128.32.136.3 41.0 4040.0 98.54 50.19
upenn.edu noc3.dccs.upenn.edu. 128.91.251.158 38.0 3866.0 101.74 49.90
berkeley.edu sns-pb.isc.org. 192.5.4.1 41.0 3996.0 97.46 49.66
berkeley.edu phloem.uoregon.edu. 128.223.32.35 41.0 3996.0 97.46 49.66
ksu.edu ns-3.ksu.edu. 129.130.139.150 36.0 3651.0 101.42 48.42
ksu.edu ns-2.ksu.edu. 129.130.139.151 36.0 3651.0 101.42 48.42
ksu.edu ns-1.ksu.edu. 129.130.254.21 36.0 3651.0 101.42 48.42
indiana.edu dns1.iu.edu. 134.68.220.8 40.0 3671.0 91.78 46.30
indiana.edu dns1.illinois.edu. 130.126.2.100 40.0 3671.0 91.78 46.30
indiana.edu dns2.iu.edu. 129.79.1.8 40.0 3655.0 91.38 46.11
mst.edu dns02.srv.mst.edu. 131.151.245.19 36.0 3433.0 95.36 45.63
okstate.edu ns2.cis.okstate.edu. 139.78.200.1 40.0 3599.0 89.97 45.43
okstate.edu ns.cis.okstate.edu. 139.78.100.1 40.0 3599.0 89.97 45.43
mst.edu ns-2.mst.edu. 131.151.247.41 36.0 3417.0 94.92 45.42
mst.edu ns-1.mst.edu. 131.151.247.40 36.0 3417.0 94.92 45.42
mst.edu dns01.srv.mst.edu. 131.151.245.18 36.0 3417.0 94.92 45.42
mst.edu dns03.srv.mst.edu. 131.151.245.20 36.0 3385.0 94.03 45.01
mst.edu ns1.umsl.com. 134.124.31.136 36.0 3369.0 93.58 44.81
ucr.edu ns2.ucr.edu. 138.23.80.20 36.0 3356.0 93.22 44.64
ucr.edu ns1.ucr.edu. 138.23.80.10 36.0 3356.0 93.22 44.64
ksu.edu nic.kanren.net. 2001:49d0:2008:f000::5 36.0 4085.0 113.47 43.58
ksu.edu kic.kanren.net. 2001:49d0:2003:f000::5 36.0 4085.0 113.47 43.58
upenn.edu dns2.udel.edu. 128.175.13.17 38.0 3344.0 88.00 43.38
upenn.edu dns1.udel.edu. 128.175.13.16 38.0 3344.0 88.00 43.38
upenn.edu adns2.upenn.edu. 128.91.254.22 38.0 3344.0 88.00 43.38
upenn.edu sns-pb.isc.org. 192.5.4.1 38.0 3312.0 87.16 42.98
lsu.edu phloem.uoregon.edu. 2001:468:d01:20::80df:2023 36.0 4016.0 111.56 42.88
lsu.edu bigdog.lsu.edu. 2620:105:b050::1 36.0 4016.0 111.56 42.88
[ ... rest of data omitted ...]

Link to full data set.

-- Shumon Huque

IPv6 at Penn

World IPv6 Launch (June 6th 2012) is fast approaching, so I thought I'd share some details about IPv6 deployment at the University of Pennsylvania and what we've recently done to prepare for this event.

 

A quick history

Penn runs a regional network called MAGPI, which connects Research & Education (R&E) institutions in our area (eastern Pennsylvania, New Jersey, and Delaware) to national R&E backbone networks like Internet2. We first deployed IPv6 in the MAGPI network in mid 2002 and soon after, established an external peering with Internet2. At that time, a small number of engineers in the networking department (including myself) typically had our computers directly wired into MAGPI infrastructure to get IPv6 connectivity at desktops and test servers.

IPv6 was introduced more gradually into the Penn campus network infrastructure, starting in 2005. Initially it was enabled only at the border  and core routers, and extended out to only selected IT departmental subnets. In September 2005, Penn hosted the Fall Internet2 member meeting in Philadelphia, where we operated the conference network at the Wyndham Franklin Plaza hotel - this network was fully IPv6 enabled, including support for IPv6 multicast routing. (Incidentally, we are hosting the Fall 2012 Internet2 meeting this October again, so I hope to see some of you there.)

Over the course of the years since, we've been gradually extending IPv6 network connectivity to the rest of the campus, and turning up IPv6 enabled application services where feasible. Needless to say, it is still early days in IPv6 deployment and a huge amount of work remains to be done.

Network Infrastructure

Unlike other IT services at Penn, many of which are highly decentralized, the campus network is mostly run by the central IT organization - this gave us the ability, when needed, to roll out IPv6 to large portions of the network fairly rapidly. Due to many competing priorities and projects, we have mostly not taken advantage of this ability, until quite recently. IPv6 had been deployed on departmental subnets only where it had explicitly been asked for. One of the more interesting cases was the Annenberg School for Communication - they approached the central IT group a few years ago with a need for IPv6 in order to facilitate some collaboration with partners in China who had asked if they'd be able to conduct video conferencing over IPv6. This was the first time we encountered direct external pressure to deploy IPv6. I'm sure it won't be the last.

The one subdivision within the university that does run their own network infrastructure, the School of Engineering & Applied Science, has been an early adopter, and has been running IPv6 in their part of the network since 2007.

In the summer of 2011, we took advantage of the increased interest generated by last year's World IPv6 Day event to extend the deployment of IPv6 to most of the rest of the campus wired network. The one area that was significantly lagging was the wireless network. This was a bit more challenging because of known bugs in our wireless controller vendor's gear (Aruba Networks) which necessitated a code upgrade. That code upgrade did not happen until earlier this year, so we're still in the midst of IPv6 deployment on wireless. As of this writing, 70 wireless subnets (out of roughly 200) have IPv6 available, and we should have the entire wireless network done sometime later this summer.

For the more technically inclined, we run Integrated IS-IS as our interior routing protocol for IPv6, whereas we continue to run OSPF for IPv4. At the time when we were initially testing IPv6, that was clearly the best choice since OSPF version 3 (the new version of OSPF that supports IPv6) was still in a relatively fledgling state of implementation maturity. Also confining IPv6 to a separate routing protocol seemed like a good additional safety measure. We run a single flat Level-2 area for the entire campus. For exterior routing, we have separate BGP peerings over IPv6 transport established with our external peers that carry IPv6 routes only. Our initial deployment used a provider allocated /48 IPv6 block delegated to us by MAGPI. In 2008, we obtained a Provider Independent ("portable") /32 sized IPv6 address block (2607:F470::/32) from the regional registry ARIN, and have mostly renumbered into it.

Currently, Penn's only connection to the IPv6 Internet is via MAGPI and Internet2. But we're planning to turn up IPv6 peering on our direct commercial ISP links (Level3 and Cogent) in the very near future. At least one of them might happen before World IPv6 Launch.

IPv6 enabled servers use statically configured addresses. Clients on campus almost exclusively use stateless address autoconfiguration (including the privacy/temporary address extensions). DHCPv6 has not been an option for us until recently, since we're a 40% Mac campus, and Apple didn't support DHCPv6 until Mac OS X version 10.7 (late summer 2011).  We are developing plans for a possible DHCPv6 service in the future, which I'll elaborate on at a later time.

Application Services

Penn's authoritative DNS service has been IPv6 enabled for many years. The campus DNS resolvers also support DNS queries over IPv6 but since we don't yet run DHCPv6, we don't have a convenient way to hand out their IPv6 addresses. Our homegrown DNS content management system has supported the ability to create AAAA and IPv6 PTR records for a long time also.

A number of departmental web servers, including the School of Engineering & Applied Science, are IPv6 enabled. The Penn central jabber server, jabber.upenn.edu, was one of our earlier IPv6 equipped services, and actually sees a high proportion of IPv6 activity. Work is proceeding on many other services.

Some of the most challenging services have been those where components of the service have been outsourced to commercial third parties. The central Penn webserver, www.upenn.edu is located on the Akamai content delivery network, and Akamai has been slow to deploy IPv6. We successfully worked with Akamai to put the website on IPv6 for last year's world IPv6 day (June 8th 2011), but they were not then prepared to offer it on an ongoing production basis. In April 2012, Akamai finally announced production IPv6 support. As of May 9th, the Penn website is now available over IPv6, hopefully permanently this time.

Akamai uses DNS resolver client addresses to direct users to content servers geographically close to them (although a few other factors including load are also considered by the server selection algorithm). I collected some data with the help of colleagues about where the www.upenn.edu AAAA record resolves to from various locations. Since we host a cluster of IPv6-enabled Akamai content  servers on our campus network, most of the time, on-campus users of www.upenn.edu will be directed to these local servers.

One issue we overlooked, is that there is a version of the main Penn website optimized for small form-factor mobile devices ("m.upenn.edu") which is not on the Akamai CDN, and run by another unit within the IT organization that has not yet deployed IPv6. So, more work remains to get the Penn web presence completely IPv6 ready.

The other challenging service is central e-mail. Penn uses Message Labs (now Symantec Cloud) to scan e-mail for viruses and spam scoring. As a result both inbound and outbound e-mail has to go through Symantec Cloud's servers. We've inquired about IPv6 support for a number of years, but even today, they appear to have no plans to support it. Our latest communication from them (early May 2012) indicates that they have no plans for any IPv6 support for FY13 (their fiscal year starts in April), and that this may change as priorities shift. At some point, we too might be compelled to shift our priorities and end our relationship with Message Labs, and either seek another provider (does Google/Postini do IPv6 yet?) or bring back virus & spam filtering in-house.

For a comparative view of externally visible IPv6 enabled application services deployed at various US universities and other organizations, Mark Prior's IPv6 survey website is a good resource. Of the five services measured there (Web, DNS, Mail, NTP, and Jabber), Penn gets a green box for four - Mail is the missing one because of Symantec Cloud.

Other Projects

From time to time, we've worked with Penn researchers and outside companies on IPv6 related projects. In the fall of 2009, we worked with Alain Durand (then at Comcast) and Roch Guerin (Penn engineering school faculty) on a small trial deployment of Dual Stack Lite; see RFC 6333 for details of this protocol - this was mostly to help Comcast out. It's unlikely that Penn will deploy DSLite in our own production network. We've also worked with Roch and Comcast on an ongoing IPv6 adoption measurement project. Details of this project are available at: http://mnlab-ipv6.seas.upenn.edu/

Facilitating Regional Connectivity

As mentioned earlier, Penn enables IPv6 connectivity for regional institutions via the MAGPI GigaPoP and Internet2. Currently, we provide IPv6 connectivity to the following institutions: Princeton University, New Jersey Edge (the state education network for NJ), Lafayette College, and Rutgers University. Of them, Princeton came up first in 2005.

Traffic Measurements

Looking at some recent data, IPv6 traffic traversing the campus border is roughly 3% of the total inbound and about 1% of the total outbound traffic. Internal traffic is probably a slightly higher percentage. We're just starting to deploy better measurement infrastructure for IPv6, so we'll have more comprehensive data in the future. But I'll be writing another article sharing what we have so far next.

Links 

• World IPv6 Launch website
• Internet Society's Deploy360 Programme (helping people deploy IPv6, DNSSEC, etc)
• Penn computing's IPv6 page
• Penn IPv6 Strategy Document from March 2009
• An IPv6 Tutorial/Training course I gave recently (PICC 2012)
• IPv6 Campus Deployment Updates Panel, Joint Techs Conference, Feb 2010
• Mark Prior's IPv6 Deployment status survey
• Monitoring IPv6 Content Reachability, Joint Techs Conference, July 2010