World IPv6 Launch (June 6th 2012) is fast approaching, so I thought I'd share some details about IPv6 deployment at the University of Pennsylvania and what we've recently done to prepare for this event.
IPv6 was introduced more gradually into the Penn campus network infrastructure, starting in 2005. Initially it was enabled only at the border and core routers, and extended out to only selected IT departmental subnets. In September 2005, Penn hosted the Fall Internet2 member meeting in Philadelphia, where we operated the conference network at the Wyndham Franklin Plaza hotel - this network was fully IPv6 enabled, including support for IPv6 multicast routing. (Incidentally, we are hosting the Fall 2012 Internet2 meeting this October again, so I hope to see some of you there.)
Over the course of the years since, we've been gradually extending IPv6 network connectivity to the rest of the campus, and turning up IPv6 enabled application services where feasible. Needless to say, it is still early days in IPv6 deployment and a huge amount of work remains to be done.
The one subdivision within the university that does run their own network infrastructure, the School of Engineering & Applied Science, has been an early adopter, and has been running IPv6 in their part of the network since 2007.
In the summer of 2011, we took advantage of the increased interest generated by last year's World IPv6 Day event to extend the deployment of IPv6 to most of the rest of the campus wired network. The one area that was significantly lagging was the wireless network. This was a bit more challenging because of known bugs in our wireless controller vendor's gear (Aruba Networks) which necessitated a code upgrade. That code upgrade did not happen until earlier this year, so we're still in the midst of IPv6 deployment on wireless. As of this writing, 70 wireless subnets (out of roughly 200) have IPv6 available, and we should have the entire wireless network done sometime later this summer.
For the more technically inclined, we run Integrated IS-IS as our interior routing protocol for IPv6, whereas we continue to run OSPF for IPv4. At the time when we were initially testing IPv6, that was clearly the best choice since OSPF version 3 (the new version of OSPF that supports IPv6) was still in a relatively fledgling state of implementation maturity. Also confining IPv6 to a separate routing protocol seemed like a good additional safety measure. We run a single flat Level-2 area for the entire campus. For exterior routing, we have separate BGP peerings over IPv6 transport established with our external peers that carry IPv6 routes only. Our initial deployment used a provider allocated /48 IPv6 block delegated to us by MAGPI. In 2008, we obtained a Provider Independent ("portable") /32 sized IPv6 address block (2607:F470::/32) from the regional registry ARIN, and have mostly renumbered into it.
Currently, Penn's only connection to the IPv6 Internet is via MAGPI and Internet2. But we're planning to turn up IPv6 peering on our direct commercial ISP links (Level3 and Cogent) in the very near future. At least one of them might happen before World IPv6 Launch.
IPv6 enabled servers use statically configured addresses. Clients on campus almost exclusively use stateless address autoconfiguration (including the privacy/temporary address extensions). DHCPv6 has not been an option for us until recently, since we're a 40% Mac campus, and Apple didn't support DHCPv6 until Mac OS X version 10.7 (late summer 2011). We are developing plans for a possible DHCPv6 service in the future, which I'll elaborate on at a later time.
Penn's authoritative DNS service has been IPv6 enabled for many years. The campus DNS resolvers also support DNS queries over IPv6 but since we don't yet run DHCPv6, we don't have a convenient way to hand out their IPv6 addresses. Our homegrown DNS content management system has supported the ability to create AAAA and IPv6 PTR records for a long time also.
A number of departmental web servers, including the School of Engineering & Applied Science, are IPv6 enabled. The Penn central jabber server, jabber.upenn.edu, was one of our earlier IPv6 equipped services, and actually sees a high proportion of IPv6 activity. Work is proceeding on many other services.
Some of the most challenging services have been those where components of the service have been outsourced to commercial third parties. The central Penn webserver, www.upenn.edu is located on the Akamai content delivery network, and Akamai has been slow to deploy IPv6. We successfully worked with Akamai to put the website on IPv6 for last year's world IPv6 day (June 8th 2011), but they were not then prepared to offer it on an ongoing production basis. In April 2012, Akamai finally announced production IPv6 support. As of May 9th, the Penn website is now available over IPv6, hopefully permanently this time.
Akamai uses DNS resolver client addresses to direct users to content servers geographically close to them (although a few other factors including load are also considered by the server selection algorithm). I collected some data with the help of colleagues about where the www.upenn.edu AAAA record resolves to from various locations. Since we host a cluster of IPv6-enabled Akamai content servers on our campus network, most of the time, on-campus users of www.upenn.edu will be directed to these local servers.
One issue we overlooked, is that there is a version of the main Penn website optimized for small form-factor mobile devices ("m.upenn.edu") which is not on the Akamai CDN, and run by another unit within the IT organization that has not yet deployed IPv6. So, more work remains to get the Penn web presence completely IPv6 ready.
The other challenging service is central e-mail. Penn uses Message Labs (now Symantec Cloud) to scan e-mail for viruses and spam scoring. As a result both inbound and outbound e-mail has to go through Symantec Cloud's servers. We've inquired about IPv6 support for a number of years, but even today, they appear to have no plans to support it. Our latest communication from them (early May 2012) indicates that they have no plans for any IPv6 support for FY13 (their fiscal year starts in April), and that this may change as priorities shift. At some point, we too might be compelled to shift our priorities and end our relationship with Message Labs, and either seek another provider (does Google/Postini do IPv6 yet?) or bring back virus & spam filtering in-house.
For a comparative view of externally visible IPv6 enabled application services deployed at various US universities and other organizations, Mark Prior's IPv6 survey website is a good resource. Of the five services measured there (Web, DNS, Mail, NTP, and Jabber), Penn gets a green box for four - Mail is the missing one because of Symantec Cloud.
From time to time, we've worked with Penn researchers and outside companies on IPv6 related projects. In the fall of 2009, we worked with Alain Durand (then at Comcast) and Roch Guerin (Penn engineering school faculty) on a small trial deployment of Dual Stack Lite; see RFC 6333 for details of this protocol - this was mostly to help Comcast out. It's unlikely that Penn will deploy DSLite in our own production network. We've also worked with Roch and Comcast on an ongoing IPv6 adoption measurement project. Details of this project are available at: http://mnlab-ipv6.seas.upenn.edu/
As mentioned earlier, Penn enables IPv6 connectivity for regional institutions via the MAGPI GigaPoP and Internet2. Currently, we provide IPv6 connectivity to the following institutions: Princeton University, New Jersey Edge (the state education network for NJ), Lafayette College, and Rutgers University. Of them, Princeton came up first in 2005.
Looking at some recent data, IPv6 traffic traversing the campus border is roughly 3% of the total inbound and about 1% of the total outbound traffic. Internal traffic is probably a slightly higher percentage. We're just starting to deploy better measurement infrastructure for IPv6, so we'll have more comprehensive data in the future. But I'll be writing another article sharing what we have so far next.
A quick history
Penn runs a regional network called MAGPI, which connects Research & Education (R&E) institutions in our area (eastern Pennsylvania, New Jersey, and Delaware) to national R&E backbone networks like Internet2. We first deployed IPv6 in the MAGPI network in mid 2002 and soon after, established an external peering with Internet2. At that time, a small number of engineers in the networking department (including myself) typically had our computers directly wired into MAGPI infrastructure to get IPv6 connectivity at desktops and test servers.IPv6 was introduced more gradually into the Penn campus network infrastructure, starting in 2005. Initially it was enabled only at the border and core routers, and extended out to only selected IT departmental subnets. In September 2005, Penn hosted the Fall Internet2 member meeting in Philadelphia, where we operated the conference network at the Wyndham Franklin Plaza hotel - this network was fully IPv6 enabled, including support for IPv6 multicast routing. (Incidentally, we are hosting the Fall 2012 Internet2 meeting this October again, so I hope to see some of you there.)
Over the course of the years since, we've been gradually extending IPv6 network connectivity to the rest of the campus, and turning up IPv6 enabled application services where feasible. Needless to say, it is still early days in IPv6 deployment and a huge amount of work remains to be done.
Network Infrastructure
Unlike other IT services at Penn, many of which are highly decentralized, the campus network is mostly run by the central IT organization - this gave us the ability, when needed, to roll out IPv6 to large portions of the network fairly rapidly. Due to many competing priorities and projects, we have mostly not taken advantage of this ability, until quite recently. IPv6 had been deployed on departmental subnets only where it had explicitly been asked for. One of the more interesting cases was the Annenberg School for Communication - they approached the central IT group a few years ago with a need for IPv6 in order to facilitate some collaboration with partners in China who had asked if they'd be able to conduct video conferencing over IPv6. This was the first time we encountered direct external pressure to deploy IPv6. I'm sure it won't be the last.The one subdivision within the university that does run their own network infrastructure, the School of Engineering & Applied Science, has been an early adopter, and has been running IPv6 in their part of the network since 2007.
In the summer of 2011, we took advantage of the increased interest generated by last year's World IPv6 Day event to extend the deployment of IPv6 to most of the rest of the campus wired network. The one area that was significantly lagging was the wireless network. This was a bit more challenging because of known bugs in our wireless controller vendor's gear (Aruba Networks) which necessitated a code upgrade. That code upgrade did not happen until earlier this year, so we're still in the midst of IPv6 deployment on wireless. As of this writing, 70 wireless subnets (out of roughly 200) have IPv6 available, and we should have the entire wireless network done sometime later this summer.
For the more technically inclined, we run Integrated IS-IS as our interior routing protocol for IPv6, whereas we continue to run OSPF for IPv4. At the time when we were initially testing IPv6, that was clearly the best choice since OSPF version 3 (the new version of OSPF that supports IPv6) was still in a relatively fledgling state of implementation maturity. Also confining IPv6 to a separate routing protocol seemed like a good additional safety measure. We run a single flat Level-2 area for the entire campus. For exterior routing, we have separate BGP peerings over IPv6 transport established with our external peers that carry IPv6 routes only. Our initial deployment used a provider allocated /48 IPv6 block delegated to us by MAGPI. In 2008, we obtained a Provider Independent ("portable") /32 sized IPv6 address block (2607:F470::/32) from the regional registry ARIN, and have mostly renumbered into it.
Currently, Penn's only connection to the IPv6 Internet is via MAGPI and Internet2. But we're planning to turn up IPv6 peering on our direct commercial ISP links (Level3 and Cogent) in the very near future. At least one of them might happen before World IPv6 Launch.
IPv6 enabled servers use statically configured addresses. Clients on campus almost exclusively use stateless address autoconfiguration (including the privacy/temporary address extensions). DHCPv6 has not been an option for us until recently, since we're a 40% Mac campus, and Apple didn't support DHCPv6 until Mac OS X version 10.7 (late summer 2011). We are developing plans for a possible DHCPv6 service in the future, which I'll elaborate on at a later time.
Application Services
Penn's authoritative DNS service has been IPv6 enabled for many years. The campus DNS resolvers also support DNS queries over IPv6 but since we don't yet run DHCPv6, we don't have a convenient way to hand out their IPv6 addresses. Our homegrown DNS content management system has supported the ability to create AAAA and IPv6 PTR records for a long time also.
A number of departmental web servers, including the School of Engineering & Applied Science, are IPv6 enabled. The Penn central jabber server, jabber.upenn.edu, was one of our earlier IPv6 equipped services, and actually sees a high proportion of IPv6 activity. Work is proceeding on many other services.
Some of the most challenging services have been those where components of the service have been outsourced to commercial third parties. The central Penn webserver, www.upenn.edu is located on the Akamai content delivery network, and Akamai has been slow to deploy IPv6. We successfully worked with Akamai to put the website on IPv6 for last year's world IPv6 day (June 8th 2011), but they were not then prepared to offer it on an ongoing production basis. In April 2012, Akamai finally announced production IPv6 support. As of May 9th, the Penn website is now available over IPv6, hopefully permanently this time.
Akamai uses DNS resolver client addresses to direct users to content servers geographically close to them (although a few other factors including load are also considered by the server selection algorithm). I collected some data with the help of colleagues about where the www.upenn.edu AAAA record resolves to from various locations. Since we host a cluster of IPv6-enabled Akamai content servers on our campus network, most of the time, on-campus users of www.upenn.edu will be directed to these local servers.
One issue we overlooked, is that there is a version of the main Penn website optimized for small form-factor mobile devices ("m.upenn.edu") which is not on the Akamai CDN, and run by another unit within the IT organization that has not yet deployed IPv6. So, more work remains to get the Penn web presence completely IPv6 ready.
The other challenging service is central e-mail. Penn uses Message Labs (now Symantec Cloud) to scan e-mail for viruses and spam scoring. As a result both inbound and outbound e-mail has to go through Symantec Cloud's servers. We've inquired about IPv6 support for a number of years, but even today, they appear to have no plans to support it. Our latest communication from them (early May 2012) indicates that they have no plans for any IPv6 support for FY13 (their fiscal year starts in April), and that this may change as priorities shift. At some point, we too might be compelled to shift our priorities and end our relationship with Message Labs, and either seek another provider (does Google/Postini do IPv6 yet?) or bring back virus & spam filtering in-house.
For a comparative view of externally visible IPv6 enabled application services deployed at various US universities and other organizations, Mark Prior's IPv6 survey website is a good resource. Of the five services measured there (Web, DNS, Mail, NTP, and Jabber), Penn gets a green box for four - Mail is the missing one because of Symantec Cloud.
Other Projects
From time to time, we've worked with Penn researchers and outside companies on IPv6 related projects. In the fall of 2009, we worked with Alain Durand (then at Comcast) and Roch Guerin (Penn engineering school faculty) on a small trial deployment of Dual Stack Lite; see RFC 6333 for details of this protocol - this was mostly to help Comcast out. It's unlikely that Penn will deploy DSLite in our own production network. We've also worked with Roch and Comcast on an ongoing IPv6 adoption measurement project. Details of this project are available at: http://mnlab-ipv6.seas.upenn.edu/
Facilitating Regional Connectivity
As mentioned earlier, Penn enables IPv6 connectivity for regional institutions via the MAGPI GigaPoP and Internet2. Currently, we provide IPv6 connectivity to the following institutions: Princeton University, New Jersey Edge (the state education network for NJ), Lafayette College, and Rutgers University. Of them, Princeton came up first in 2005.
Traffic Measurements
Looking at some recent data, IPv6 traffic traversing the campus border is roughly 3% of the total inbound and about 1% of the total outbound traffic. Internal traffic is probably a slightly higher percentage. We're just starting to deploy better measurement infrastructure for IPv6, so we'll have more comprehensive data in the future. But I'll be writing another article sharing what we have so far next.
Links
- World IPv6 Launch website
- Internet Society's Deploy360 Programme (helping people deploy IPv6, DNSSEC, etc)
- Penn computing's IPv6 page
- Penn IPv6 Strategy Document from March 2009
- An IPv6 Tutorial/Training course I gave recently (PICC 2012)
- IPv6 Campus Deployment Updates Panel, Joint Techs Conference, Feb 2010
- Mark Prior's IPv6 Deployment status survey
- Monitoring IPv6 Content Reachability, Joint Techs Conference, July 2010
