I've left Penn for a new job

After more than 20 years of working at Penn (University of Pennsylvania), I've decided to take a new job as Principal Research Scientist at Verisign Labs, the applied research division of Verisign Inc. You might know that Verisign is one of the world's largest DNS infrastructure providers. It runs the .com, .net, .edu, and .gov DNS top level domains, two of the thirteen DNS root servers (A and J), and performs the critically important root zone management function. Verisign also provides managed DNS, Distributed Denial of Service (DDoS) mitigation, and several other services. (Note: Verisign's certificate services business was sold off to Symantec several years ago).

I've been at Penn for so long, that I startled many of my colleagues with the news of my departure, so I'll say a few things about how this came about. I originally came to Penn in 1989 as an undergraduate. I became a full time IT staff member after I graduated - my first job was the system administrator of the new e-mail server for the School of Arts & Sciences, the largest school within Penn. I completed a Masters degree in Computer Science part-time while working. I later moved to the central IT organization, where I remained until last week, first as a Network Engineer, then as the Lead Engineer, and most recently as an Engineering Director. In addition, I've been a part time Adjunct Faculty in Penn's School of Engineering, teaching a laboratory course on Network Protocols.

I was approached by Allison Mankin and Matt Larson (I've known both of them for a while) about a year ago to see if I'd be interested in considering a research scientist position at Verisign Labs. Matt has since left Verisign to work at Dyn, and Allison is now a Director at Verisign Labs. At the time, I thought this was distinctly a long shot, but over the course of many months, I thought more seriously about the possibility. I visited Verisign Labs in September 2013, did an interview with them, and also met Burt Kaliski, Verisign CTO (Burt is a noted cryptographer and was the founding scientist of RSA Labs, and the developer of the PKCS standards). I kept in touch with Allison since, but it took me until the beginning of this year to finally come to the conclusion that I wanted to take this opportunity, so here I am. My first day at Verisign will be tomorrow (March 17th).

I've enjoyed my job and career at Penn a lot. I've been extensively involved in a very diverse range of technical projects, ranging from software development, systems engineering, and network design. Among others, I was responsible for the design and operation of much of the authentication and security infrastructure at Penn, as well as a variety of other services, like the DNS and DHCP (and many more that I don't have the time to enumerate here). I was the principal architect of IPv6 and DNSSEC deployment at Penn. I was the chief engineer of the MAGPI gigapop and through that role, was involved in R&E regional and national networking activities. Increasingly though, a lot of time is taken up by non-technical activities, and the longer I stay at Penn, the greater the possibility that I'll end up as a full time IT staff manager, which isn't the role I envision for myself. I've always seen my primary role as that of a technologist, and this job change will allow me to continue in that direction.

Verisign Labs appears to have a true applied research agenda, which I find appealing. Obviously DNS and DNSSEC research is an area in which I expect to be working. But there are many other interesting areas of work: routing, IPv6, reputation systems, security protocols, future internet architectures, etc. I'm also looking forward to attending more NANOG and IETF meetings to get myself more plugged into those communities than I've been able to thus far. Verisign Labs also has frequent, productive collaborations with computer science researchers through its university collobaration program.

In the process, I'm moving to the Washington DC metro area (specifically Reston, VA) where Verisign is located, another big change for me!

More later ..

--Shumon Huque.

World IPv6 Launch Measurements

From the Internet Society's most recent (January 16th 2014) round of IPv6 measurements, here again are the top 100 lists ranked by the two measures of (1) percentage of requests that used IPv6, and (2) total volume of IPv6 requests.

My past articles on this topic:
  * Measurements from December 12th 2013
  * Measurements from  September 17th 2013
  * Measurements from June 6th 2012

ISOC's blog post highlights that Deutsche Telekom, a large German telecommunications carrier, (of which T-Mobile US is a subsidiary*) has been experiencing very substantial IPv6 traffic growth. They've reached a figure of 15.5% of requests composed of IPv6 (to the select set of content providers providing measurement data). In terms of total volume of IPv6 requests, they are in 6th place behind five other large ISPs (See the second ranked list below). Comcast still leads the pack, followed by AT&T, KDDI, Free, and Verizon Wireless.

(*Note: T-Mobile's traffic is counted separately in the measurements from Deutsche Telekom)

Interestingly, while Verizon Wireless has a substantial IPv6 deployment, Verizon FIOS (their fiber-optic landline network) has made no visible progress in IPv6 deployment. Although many people, myself included, have noted this lack of deployment over the years, they were just recently taken sharply to task by folks on the NANOG (North American Network Operators Group) mailing list.

Looking at the rankings by proportion of IPv6 requests from each network, there is a new entry at the top - NYSERNet (a regional R&E network in New York State), coming in at a staggering 99.95%. On the Internet2 IPv6 working group list, I asked Bill Owens of NYSERNet if he wanted to comment. He claims that we shouldn't be too impressed because this is just the small NYSERNet office network and the backbone, with a relatively small population of clients. Regardless, I think it's a noteworthy achievement. As I've mentioned before, a number of universities and other educational organizations still show up prominently in this ranking. Two small american colleges, Gustavus Adolphus in Minnesota at 74.22% and Marist College in upstate New York at 65.77%! And Panamerican University, in Mexico City is at 66.22%.

Penn is slowly inching up at 45%. We'll have IPv6 deployed more extensively on our wireless network pretty soon, at which point I expect our numbers to go up noticeably.


World IPv6 Launch Measurements, by % IPv6 requests:
---------------------------------------------------

   1 NYSERNet 99.95%
   2 SpeedPartner GmbH 95.83%
   3 TOP-IX Consortium 86.99%
   4 Fundacao Parque Tecnologico Itaipu - Brasil 79.92%
   5 DirectVPS 78.63%
   6 interscholz Internet Services GmbH & Co. KG 75.49%
   7 Gustavus Adolphus College 74.22%
   8 Google Fiber 71.87%
   9 Universidad Panamericana 66.22%
  10 Marist College 65.77%
  11 University of Vermont 65.13%
  12 Virginia Tech 65.11%
  13 mur.at - Verein zur Frderung von Netzkwerkkunst 64.97%
  14 Association tetaneutral.net 55.67%
  15 Utility Line Italia srl 53.40%
  16 ThaiSarn 53.23%
  17 DegNet GmbH 52.84%
  18 Trunk Networks Limited 52.60%
  19 Ji?? ?trohalm 52.20%
  20 Alhambra Eidos 52.12%
  21 Louisiana State University 51.93%
  22 Ponto de Presen?a da RNP na Bahia 51.30%
  23 SuperInternet Access Pte Ltd 50.45%
  24 University of New Hampshire 50.44%
  25 Region 7 ESC 50.24%
  26 PREGINET 50.20%
  27 CICA/Junta de Andaluc?a 50.05%
  28 Tanzania Network Information Center 50.00%
  29 Karlsruhe Institute of Technology (KIT) 49.03%
  30 Maxiweb Internet Provider 48.76%
  31 Critical Colocation 48.23%
  32 University of Pennsylvania 45.38%
  33 SPAWAR 44.15%
  34 AMS-IX 42.99%
  35 University of Iowa 42.81%
  36 Kasetsart University 41.99%
  37 DreamHost 41.79%
  38 University of Minnesota 40.80%
  39 Rensselaer Polytechnic Institute 40.74%
  40 Bulgaria NREN 40.57%
  41 Verizon Wireless 40.03%
  42 CZ.NIC 39.02%
  43 guifi.net 37.86%
  44 Hughes Network Systems 35.60%
  45 Sauk Valley Community College 35.52%
  46 Tulane University 34.96%
  47 Free 34.28%
  48 ARNES 33.73%
  49 Greek Research & Technology Network 33.14%
  50 Leibniz Supercomputing Centre 32.71%
  51 Host Virtual, Inc 31.74%
  52 Netwerkvereniging Coloclue 28.41%
  53 Opera Software ASA 27.54%
  54 UNESP 27.26%
  55 UNINETT 27.13%
  56 FCCN 26.00%
  57 VOO 24.12%
  58 RCS & RDS 23.80%
  59 mc.net 23.37%
  60 UFSCar 22.96%
  61 EPT Luxembourg 22.43%
  62 FranTech Solutions 22.03%
  63 AAISP 21.22%
  64 Comcast 20.61%
  65 Chubu Telecommunications 18.90%
  66 Swisscom 18.66%
  67 XS4ALL 17.66%
  68 UFSC - Universidade Federal de Santa Catarina - Brazil 17.50%
  69 LENTEL 16.73%
  70 StarHub 16.28%
  71 Deutsche Telekom AG 15.50%
  72 NIIF/Hungarnet 14.56%
  73 LITNET 14.50%
  74 Red Acad?mica de Centros de Investigaci?n y Universidades Nacionales REACCIUN 13.85%
  75 Indiana University 13.68%
  76 ATT 13.53%
  77 SARENET 13.13%
  78 DMZGlobal 12.97%
  79 NetAssist 12.89%
  80 Academia Sinica Network 12.14%
  81 Cisco 11.95%
  82 Solcon 11.46%
  83 CESNET 11.04%
  84 Funet 10.94%
  85 PT. Wifian Solution 10.19%
  86 FidoNet 10.12%
  87 prgmr.com 9.89%
  88 T-Mobile USA 9.58%
  89 Monash University 9.28%
  90 OVH 9.26%
  91 KDDI 8.94%
  92 Spectrum Networks 8.30%
  93 AMRES - Serbian National Research and Education Network 7.84%
  94 Belnet 7.65%
  95 SWITCH 7.62%
  96 Hurricane Electric 7.58%
  97 M1 Limited 7.22%
  98 RedIRIS 6.96%
  99 Init7 6.93%
 100 CJSC Progressive Technologies 6.71%


World IPv6 Launch Measurements, by volume of IPv6:
--------------------------------------------------

   1 Comcast 20.61%
   2 ATT 13.53%
   3 KDDI 8.94%
   4 Free 34.28%
   5 Verizon Wireless 40.03%
   6 Deutsche Telekom AG 15.50%
   7 Time Warner Cable 3.88%
   8 RCS & RDS 23.80%
   9 Liberty Global 2.43%
  10 Swisscom 18.66%
  11 Telefonica del Peru 4.60%
  12 Hughes Network Systems 35.60%
  13 SoftBank BB 1.46%
  14 Chubu Telecommunications 18.90%
  15 Opera Software ASA 27.54%
  16 VOO 24.12%
  17 XS4ALL 17.66%
  18 StarHub 16.28%
  19 T-Mobile USA 9.58%
  20 Google Fiber 71.87%
  21 China Telecom 0.23%
  22 Forthnet 2.79%
  23 M1 Limited 7.22%
  24 Internode 3.94%
  25 EPT Luxembourg 22.43%
  26 Janet 3.81%
  27 its communications Inc.(iTSCOM) 2.64%
  28 CESNET 11.04%
  29 MediaCat Div./Community Netowork Center Inc. 6.58%
  30 Kasetsart University 41.99%
  31 NTT Communications 4.51%
  32 Belnet 7.65%
  33 ARNES 33.73%
  34 Leibniz Supercomputing Centre 32.71%
  35 LITNET 14.50%
  36 NIIF/Hungarnet 14.56%
  37 OVH 9.26%
  38 Cisco 11.95%
  39 BelWue 5.71%
  40 FCCN 26.00%
  41 UNINETT 27.13%
  42 Altibox AS 0.90%
  43 Monash University 9.28%
  44 SWITCH 7.62%
  45 TUBITAK ULAKBIM / ULAKNET 1.50%
  46 Australian Academic and Research Network (AARNet) 2.00%
  47 Indiana University 13.68%
  48 University of Minnesota 40.80%
  49 AAISP 21.22%
  50 UniNet 3.36%
  51 Xfone 018 0.96%
  52 Ji?? ?trohalm 52.20%
  53 Voxel / Internap 3.51%
  54 GITN Sdn Berhad 2.69%
  55 Louisiana State University 51.93%
  56 University of Pennsylvania 45.38%
  57 SuperCSI 1.69%
  58 JARING Communications Sdn Bhd 0.38%
  59 Solcon 11.46%
  60 CJSC Progressive Technologies 6.71%
  61 Starlink 1.88%
  62 inexio KGaA 2.44%
  63 Virginia Tech 65.11%
  64 green.ch AG 3.26%
  65 Hurricane Electric 7.58%
  66 Dhiraagu 0.60%
  67 Gustavus Adolphus College 74.22%
  68 LENTEL 16.73%
  69 DMZGlobal 12.97%
  70 DegNet GmbH 52.84%
  71 RedIRIS 6.96%
  72 Academia Sinica Network 12.14%
  73 GlobalConnect 1.44%
  74 University of Iowa 42.81%
  75 Funet 10.94%
  76 University of Wisconsin - Madison 6.16%
  77 Storm Internet 3.76%
  78 AMRES - Serbian National Research and Education Network 7.84%
  79 National Informatics Centre 2.32%
  80 SoftLayer Technologies 0.84%
  81 RENATER 4.57%
  82 Karlsruhe Institute of Technology (KIT) 49.03%
  83 guifi.net 37.86%
  84 Init7 6.93%
  85 Host Virtual, Inc 31.74%
  86 Rensselaer Polytechnic Institute 40.74%
  87 Choopa, LLC 0.55%
  88 NetAssist 12.89%
  89 Tulane University 34.96%
  90 DreamHost 41.79%
  91 Bulgaria NREN 40.57%
  92 FranTech Solutions 22.03%
  93 Greek Student Network 0.47%
  94 RESTENA 5.33%
  95 UNESP 27.26%
  96 iway AG 4.06%
  97 FX Networks 0.49%
  98 Tanzania Network Information Center 50.00%
  99 edpnet 1.11%
 100 ADDIX Internet Services 3.05%

Penn wins NSF Campus CyberInfrastructure Award

A while back in a blog article on our 100 Gigabit Ethernet campus upgrades, I mentioned that Penn had applied for a National Science Foundation (NSF) CC-NIE grant to enhance campus network infrastructure for research purposes.

We did in fact win an award. Here's the official notice from NSF. It's about $500,000 which will be used to deploy a dedicated high performance router for researchers and bump up our external connectivity to Internet2 to 100 Gbps. I hope to provide more updates as we begin deploying the necessary pieces of equipment.

--Shumon Huque

An excerpt from the award notice:

ABSTRACT

The University of Pennsylvania's central computing organization is partnering with leading campus researchers in engineering, physics, biology, pathology, genomics, bioinformatics, and computer science to optimize the campus network in support of big data research and high-performance computing. This project establishes a 100 Gbps-capable Science DMZ that is distinct from the general purpose campus network and is engineered for research applications. Additionally, it extends 10 Gbps connectivity to select research projects and increases Penn's connection to Internet2 from 1 Gbps to 100 Gbps, while also extending that connection to the Science DMZ. The project also lays the foundation for further enhancements to research networking infrastructure by extending IPv6 capabilities; upgrading network monitoring tools such as perfSONAR; and enhancing Penn's ability to support experimental networks and network architectures, including OpenFlow and Software Defined Networking.

The project will benefit a range of scientifically meritorious research. It will provide support for the large-scale data transfer, processing, and storage needs of researchers across Penn, while supporting intra- and inter-institutional collaborations and the broad dissemination of research results. Rather than focusing on the logistics of data storage and transfer, researchers will be able to concentrate on the transformation of these data into the information that will drive new discoveries and the creation of new technologies, drugs, therapies, and cures. Network enhancements will also support Penn's commitment to integrating research and education by supporting the network needs of the cross-disciplinary Penn Institute for Computation Science that where faculty actively integrate computation-based research with the training of future generations of STEM researchers.

Singh Nanotechnology Center

The grand opening of Penn's new Singh Center for Nanotechnology is tomorrow (Friday, October 4th). Computing directors in my department got to see it a week early when we held a special meeting in the Forum room.

This post contains a few photos from my visit - of the building exterior, hallways, and conference rooms. The labs weren't open yet. The full set can be found at Google Plus.

The Nanotech center has been featured in some recent articles:

* Philadelphia Inquirer - "Changing Skyline - Inga Saffron"
* The Daily Pennsylvanian
* Philadelphia Inquirer - "Penn going all out for small science"

Building exterior, from Walnut Street close to the 33rd Street intersection, looking east:

This cantilevered section houses a conference room - the Glandt Forum room.

The Forum room, where our meeting was held.

At the edge of the cantilevered section.

Looking westward along Walnut St towards the rest of campus. Directly in front (right side) is the Laboratory for Research on the Structure of Matter (LRSM). To the left is the David Rittenhouse Lab.

View from the green rooftop terrace.

Rooftop terrace.

Hallways.

"We Lost" sculpture by Tony Smith.

Network Engineer Job at Penn

We have an opening for a Senior Network Engineer in the University of Pennsylvania's Engineering group.

        https://jobs.hr.upenn.edu/applicants/Central?quickFind=197742

This position will be part of a small team that works very closely with our Network Operations group, offering final tier escalated support, researching new network designs, architectures & technologies, evaluating new equipment, designing and deploying related software and hardware systems.

The Engineering group is also involved in designing and operating a range of other services, including DNS, DHCP, Authentication & Authorization systems, Voice over IP, etc. So there are opportunities to get involved in many areas.

The candidate for this position will generally need to have a strong networking and programming background, as well as strong familiarity with UNIX and UNIX-like operating systems.

100 Gigabit Ethernet at Penn

This summer, the University of Pennsylvania is upgrading its campus core routing equipment (in fact we're in the midst of this upgrade right now). This is basically an upgrade to the set of large routers that form the center of our network.

The current core topology consists of 5 core routers (and also 2 border routers) interconnected by two independent layer-2 switched 10 Gigabit Ethernet networks. Each of the core routers is located in one of five geographically distributed machine rooms across the campus. A rough diagram is shown below.

This diagram also shows the current external connections to/from the campus network - we have three links (each 10 Gigabit Ethernet) to Internet Service Providers (ISPs). And two connections to MAGPI (the regional Internet2 GigaPoP operated by us), via which we access a 10 Gigabit Ethernet connection to Internet2. The Internet2 connection is shared amongst Penn and other MAGPI customers, which are mostly Research & Education institutions in the geographic area.

The core interconnect is being upgraded to 100 Gigabit Ethernet (a ten-fold increase in link bandwidth). It would be cost prohibitive to fully replicate the current design in 100 Gig (since this equipment is still very expensive) so the interconnect design has been adjusted a bit. Instead of two layer-2 switch fabrics interconnecting the routers, we are deploying the core routers connected in a 100 Gig ring (see the diagram below). When the final design is fully implemented, each core router will have a 10 Gig connection into each of the border routers (this will require some additional upgrades to the border routers, which are expected to happen later this year). The topology redesign has fewer links, and in the final count (summing the bandwidth of all the core facing links), the new core will have about 5 times the aggregate bandwidth of the old one. The maximum (shortest path) edge to edge diameter of the network increases by one routing hop.

100 Gigabit Ethernet is today's state of the art in transmission speed. The next jump up will likely be 400 Gigabit Ethernet for which the IEEE already has a study group launched and several preliminary designs under consideration.

Not depicted in this diagram is the rest of the network towards the end systems. Below the layer of core routers, are smaller routers located at the 200 or so buildings scattered around campus. Each building router is connected to two of the core routers. The building routers feed wiring closets inside the building, which house layer-2 switches that network wallplates are connected to.

In the process of the upgrade, we are also changing router vendors. The current core routers, Cisco 7609 series routers with Sup7203BXL supervisor engines, have served us well. They were originally deployed in the summer of 2005, and have been in operation well past their expected lifetime.

As is our practice, we issued an RFI/P (Request for Information/Purchase) detailing our technical requirements for the next generation routers and solicited responses from the usual suspects, selecting a few vendors whose equipment we bring in for lab testing, followed by a selection.

The product we've selected is the Brocade MLXe series router, specifically the MLXe-16 - this router can support 16 half height, or 8 full height (or a mixture of full/half) line cards, as well as redundant management and switch fabric modules.

A product description of the MLXe series is available at:
http://www.brocade.com/downloads/documents/data_sheets/product_data_sheets/MLX_Series_DS.pdf

The photo below is one of the routers (prior to deployment) in the Vagelos node room (one of 5 machine rooms distributed around campus where we house critical networking equipment and servers).Going from left to right, this chassis has two management modules, one 2-port 100 Gigabit Ethernet card, six 8-port 10 Gigabit Ethernet cards, four switch fabric modules, two more 8-port 10 Gigabit Ethernet cards, and three 24-port Gigabit Ethernet cards.

One of these routers was deployed in production last week. The rest should be up and running by the end of this month or by early July.

(The full set of photos can be seen here on Google Plus)

Shown below is the 2-port 100 Gigabit Ethernet card, partially inserted into the chassis, showing the CFP optical transceiver modules attached.

Unlike preceding generations of ethernet, with 100 Gigabit Ethernet, the transmission technology uses multiple wavelengths in parallel (although there are parallel fiber implementations also). The current IEEE specifications (802.3ba) specify four lanes of 25Gbps. However a number of key vendors in the industry, including Brocade, formed the MSA (Multi Source Agreement) and designed and built a 10x10 (10 lanes of 10Gbps) mechanism of doing 100 Gig, at much lower cost than 4x25Gbps, operating over single mode fiber at distances of 2, 4, or 10km. This is called LR-10 and uses the CFP (C Form factor pluggable) media type.

Pictured below (left) is a Brocade LR10 100 Gigabit Ethernet CFP optical module installed in 100 Gig line card with a single mode fiber connection (LC). On the right is an LR10 CFP module taken out of the router.

Close up of the 8-port 10 Gigabit Ethernet module, and several 24-port Gigabit Ethernet modules. To connect cables to them, we need to install small form factor pluggable transceivers into them, SFP+ for the 10 gig, and SFP for the 1 gig.

Pictured below is one of the five Cisco 7609 routers that will be replaced.

One of the Penn campus border routers, a Juniper M120, is shown below. This is also scheduled to be upgraded in the near future to accommodate 100 Gig and higher density 10 Gig, although the product has not yet been selected/finalized.

Below: A Ciena dense wavelength division multiplexer (DWDM). Penn uses leased metropolitan fiber to reach equipment and carriers in 401 North Broad street, the major carrier hotel in downtown Philadelphia. We have DWDM equipment placed both at the campus and the carrier hotel to carry a mixture of 1 and 10 Gig circuits and connections across this fiber for various purposes. This equipment is also scheduled to be upgraded to allow us to provision 100 Gigabit Ethernet wavelengths between the campus and the carrier hotel.

High Performance Networking for Researchers

Penn is a participant in the National Science Foundation (NSF) funded DYNES (Dynamic Network Systems) project, which provides high bandwidth dedicated point to point circuits between (typically) research labs for specialized applications. Popular uses of this infrastructure today include high energy physics researchers obtaining data from the LHC and other particle accelerator labs, and various NSF GENI network research projects.

Earlier this year, we completed a grant application for the  NSF "Campus Cyberinfrastructure - Network Infrastructure and Engineering (CC-NIE)" program. I spent a large amount of time in March of this year with several Penn colleagues in preparing the application. If Penn does win an award (we'll find out later this year), we will be deploying additional dedicated network infrastructure for campus researchers, bypassing the campus core and with 100 Gbps connectivity out to the Internet2 R&E network. A rough diagram of how this will look is below.

Software Defined Networking

There's a huge amount of buzz about Software Defined Networking (SDN) in the networking industry today, and a number of universities are investigating SDN enabled equipment for deployment in their networks. Of the big router vendors, Brocade does appear to have one of the better SDN/openflow stories thus far. The MLXe series already supports an early version of Openflow (the portion of SDN that allows forwarding tables of switches/routers to be programmed by an external SDN controller).

Penn is building an SDN testbed in our network engineering lab, primarily to investigate its capabilities. For us, SDN is still largely a solution in search of a problem. We run a very simple network by design, whose primary purpose is connectivity and high performance packet delivery. The most probable use case in our future, virtualization of the network, is likely better achieved with a proven technology like MPLS first. But we'll keep an eye on SDN and its evolution. We do want to support research uses of SDN though. Several faculty members in the Computer Science department are interested in SDN, and the NSF CC-NIE grant will allow us to build some SDN enabled network infrastructure separate from the core production network to accommodate their work.

-- Shumon Huque

Internet2 IPv6 Panel recap

A few notes from last month's IPv6 deployment panel at the Fall Internet2 Member Meeting in Philadelphia, which I moderated (October 2nd 2012). Watch the entire video of the session (1 hour 15 minutes) for full details.

I opened the session with a brief review of World IPv6 Launch and some its measurement data, and a mention of other IPv6 deployment measurement projects, including the NIST survey of universities.

ARIN - Mark Kosters, CTO

Mark began his presentation by talking about the current state of IPv4 address depletion. ARIN has 2.87 /8 equivalent address blocks left (note: it's down to 2.79 as of Nov 18th). RIPE and APNIC are almost out, below the one /8 threshold at which a real address rationing stage has already begun - they will give out a maximum of only /20 or /22 sized blocks regardless of how much IPv4 address space you really need. With current projections, ARIN is scheduled to exhaust in August 2013, but events could dramatically change the timeline. For example there are some ISPs in the ARIN region that could easily qualify for /9 allocations. ARIN is in phase 2 of a 4-phase runout plan - details can be seen at https://www.arin.net/resources/request/ipv4_countdown.html. Today, 58% of ARIN's membership has only IPv4 address space, 6% has only IPv6, and 36% have both IPv4 and IPv6 space. Legacy address blocks (ie. allocated to organizations prior to ARIN's existence - this is quite common in the US higher ed community) comprise 45% of the ARIN address space.

The second part of Mark's talk was about IPv6 deployment activites at ARIN itself. He went through a short history of their IPv6 implementation, which dates back to 2003, at which time they had a somewhat creaky, segregated (ie. non dualstack) implementation. By 2008 they migrated to a robust dualstack implementation. All ARIN services today are native, dualstack. Meeting networks are a bit more challenging, where they often have to rely on tunnels (due to hotel carrier limitations).

Comcast - John Brzozowski, Chief IPv6 Architect

Comcast now has IPv6 enabled on 50% of their broadband network footprint. However only 2.5% of the customer base is currently dualstack - of these dualstack users about 65% are using a computer directly connected to the cable modem, and 35% have an IPv6 enabled home router to which devices are attached. Comcast expects this ratio will eventually flip over to 80/20 in favor of home routers as it is in IPv4. To date, their focus has been on residential broadband, but they have pilots for business and commercial service. The Comcast metro ethernet service is IPv6 ready and they'd love to have more customers using it. In terms of traffic, Comcast has seen a 375% increase in IPv6 compared to World IPv6 day (June, 2011), with the majority of the increase occurring between January and June 2012, in the run up to World IPv6 Launch. The majority of the traffic is composed of services like Youtube and Netflix. The 2012 Olympics (streamed by Comcast/NBC in the US) had a noticeable impact - about 6% of this traffic to Comcast customers used IPv6. Comcast continues to work on content and services. Xfinity and comcast.net are IPv6 enabled and they use Akamai as a content delivery network.

They've put in place an extensive measurement and metrics platform to proactively detect any adverse affects on customer experience and take action if needed. Comcast sees advantages to IPv6 beyond the usual address space depletion concerns. An example cited was the fact that once they've allocated an IPv6 subnet, they don't have to be concerned about resizing it again, in marked contrast to their IPv4 deployment - this adds up over time to a significant operational benefit.

Marist College - Eric Kenny

Marist College is located in upstate network with 5,500 students on a 180-acre campus with 50 buildings. They started IPv6 deployment in 2010, and by June 2012 the wired network was mostly done. They plan to deploy IPv6 to the wireless network in the winter of 2012. One big item for them was getting provider independent address space from ARIN (mostly internal process issues). They started with a provider allocated block from NySERNET, but eventually got a /48 from ARIN, and are now wondering if they should have applied for a larger block. They also have a native IPv6 BGP peering with their commercial ISP, Lightower, which was described as an interesting experience, since they were their first IPv6 customer. They use stateless address autoconfiguration, and most devices on their network support IPv6. They haven't yet made much progress with IPv6 enabled network services, apart from DNS. At the current time about 4% of the traffic crossing their border is IPv6 - they expect this number to go up considerably after wireless deployment. One of their biggest concerns has been address tracking and accountability. They've developed their own application to track IPv6 address and MAC address associations.

Louisiana State University - Allie Hopkins

Allie focussed more on the application and process side of things. As part of their rollout process, LSU has had a great dialog with the user community, with extensive outreach and communications via message boards and e-mail lists, and close contact with application developers and deployers. And despite a set of issues (described later), they rate their deployment a success and as can be seen from World IPv6 Launch measurements, they generate a substantial amount of traffic. One of the issues they've had is with their DNS based network registration system which was incapable of working properly with IPv6 enabled hosts. Another issue they encountered was connectivity issues between tagged and untagged VLANs on interfaces, due to the Cisco routers using the same IPv6 link local addresses on them - this was fixed by manually configuring unique addresses on the VLAN interfaces. They've also had ongoing issues with being put on Google's AAAA blacklist.They are aware that they have some pockets of poor IPv6 connectivity within their campus network, but so far they've gotten no help from Google about measurement data that could help them more easily track down these cases.

I was scheduled to do a short presentation about IPv6 at Penn, but I decided to skip it (I'll post the slides later) in the interests of providing more time for audience discussion and questions.

Q&A Portion

Richard Machida (U of Alaska) asked if anyone was looking into deploying IPv6-only networks for any purpose. I answered that we've deployed them only in the lab for testing purposes (longer term, we would investigate whether it's possible to run specific applications that may not need IPv4, like VoIP on them). John says Comcast has no IPv6-only networks, but they do have plenty of IPv6-only devices that sit on dual-stack networks.

There was a question about more details of LSU's network registration system (from what I gathered it's a Netreg or Netreg like system), and whether 802.1x based network access control would be a solution (802.1x authentication happens at the link layer and is IP protocol independent). It probably would, but LSU was not quite prepared to deploy it yet.

There was discussion about what folks were doing about measuring IPv4 vs IPv6 traffic. Comcast went into some more detail about their work. I showed some data from Penn, where over the summer we were approaching 11% inbound and 4% outbound traffic composed of IPv6. We've since dropped down substantially, because we had to take IPv6 off the wireless network (I'm planning to write another blog article detailing why, but the quick summary is that deploying v6 broke IP address mobility, and we're working with our wireless equipment vendor on some possible solutions).

Dan Magorian (Johns Hopkins) asked what cable modem data specifications support IPv6 and what is the state of IPv6 support in the currently deployed field. John's answer is that DOCSIS 3, the current spec supports it pretty completely, almost all current equipment implements it, but there are ways to support IPv6 on older specs too.

---

Addendum: John B was one of the recipients of the Internet Society's Itojun Service Award recently, and presented at IETF'85, for his "for his tireless efforts in providing IPv6 connectivity to cable broadband users across North America and evangelizing the importance of IPv6 deployment globally". Congratulations John!

Shumon Huque

IPv6 panel at Internet2 meeting

Penn is co-hosting the Fall 2012 Internet2 Member meeting in Philadelphia, Oct 1st through the 4th. I'm moderating an IPv6 Deployment Panel at the conference (October 2nd, 1:15pm-2:30pm). Joining me will be John Brzozowski from Comcast, Allie Hopkins from Tulane University, Eric Kenny from Marist College, and Mark Kosters from the American Registry for Internet Numbers (ARIN).

The description says: "Several panelists will provide an update on IPv6 deployment activity and plans at their respective organizations, including both network infrastructure and application services. Other topics might include IPv6 security issues, network monitoring, technical support and training, etc"

John Brzozowski is Comcast's chief IPv6 architect. Comcast is one of the industry's leading adopters of IPv6, and I hope that John will share the latest news about IPv6 developments at Comcast. Mark Kosters is the Chief Technology Officer for ARIN. Allie Hopkins is an IT director at Tulane, but was formerly at Louisiana State University. I expect that Allie will be able to talk about the state of IPv6 deployment at LSU. As you may recall, LSU was on the list of the top IPv6 traffic generating sites during World IPv6 Launch. Eric Kenny is a network engineer from Marist College, which also made that list. I'll meet Eric for the first time at the conference. I've known the other panelists for a while.

In addition to deployment details, some combination of us will try to do a little bit of IPv6 evangelism. I'll also be asking Mark to do the usual ARIN update on the state of IPv4 address depletion.

I hope to see some of you at the conference. The panel session will also be netcast (and most likely archived video will be available to view later). If anyone has suggestions on specific IPv6 related topics the panel should discuss or comment on, feel free to let me know.

References:
* IPv6 at Penn
* IPv6 at LSU
* Comcast IPv6 Information Center
* ARIN: IPv4/IPv6 the bottom line
* ARIN: IPv6 Wiki

Network Engineer openings at Penn

We have two job openings at the University of Pennsylvania for Network Engineers.

1. Network Engineer or Senior Network Engineer
https://jobs.hr.upenn.edu/applicants/Central?quickFind=195121

This position is part of our Network Operations group which deploys and operates Penn's production campus network (data/voice/video), and the area gigapop (MAGPI - http://www.magpi.net/). This is a traditional network engineer position requiring expertise in network protocols at various layers. I expect this job will initially have a strong focus on design and operation of our wireless network.

2. Senior Network Engineer
https://jobs.hr.upenn.edu/applicants/Central?quickFind=195853
http://www.linkedin.com/jobs?viewJob=&jobId=3415100

This position is part of our Network Engineering group (which I lead). This group works very closely with Network Operations, offering final tier escalated support, researching new network designs, architectures, technologies, evaluating new equipment, designing and deploying related software and hardware systems. At Penn, the Network Engineering group also includes folks that run a number of other systems (DNS, DHCP, Authentication, Authorization, etc). We work with many cutting edge things (IPv6, DNSSEC, Openflow, etc). The candidate for this position will generally need to have a strong networking and programming background, as well as strong familiarity with UNIX and UNIX-like operating systems.

IPv6 at Penn

World IPv6 Launch (June 6th 2012) is fast approaching, so I thought I'd share some details about IPv6 deployment at the University of Pennsylvania and what we've recently done to prepare for this event.

 

A quick history

Penn runs a regional network called MAGPI, which connects Research & Education (R&E) institutions in our area (eastern Pennsylvania, New Jersey, and Delaware) to national R&E backbone networks like Internet2. We first deployed IPv6 in the MAGPI network in mid 2002 and soon after, established an external peering with Internet2. At that time, a small number of engineers in the networking department (including myself) typically had our computers directly wired into MAGPI infrastructure to get IPv6 connectivity at desktops and test servers.

IPv6 was introduced more gradually into the Penn campus network infrastructure, starting in 2005. Initially it was enabled only at the border  and core routers, and extended out to only selected IT departmental subnets. In September 2005, Penn hosted the Fall Internet2 member meeting in Philadelphia, where we operated the conference network at the Wyndham Franklin Plaza hotel - this network was fully IPv6 enabled, including support for IPv6 multicast routing. (Incidentally, we are hosting the Fall 2012 Internet2 meeting this October again, so I hope to see some of you there.)

Over the course of the years since, we've been gradually extending IPv6 network connectivity to the rest of the campus, and turning up IPv6 enabled application services where feasible. Needless to say, it is still early days in IPv6 deployment and a huge amount of work remains to be done.

Network Infrastructure

Unlike other IT services at Penn, many of which are highly decentralized, the campus network is mostly run by the central IT organization - this gave us the ability, when needed, to roll out IPv6 to large portions of the network fairly rapidly. Due to many competing priorities and projects, we have mostly not taken advantage of this ability, until quite recently. IPv6 had been deployed on departmental subnets only where it had explicitly been asked for. One of the more interesting cases was the Annenberg School for Communication - they approached the central IT group a few years ago with a need for IPv6 in order to facilitate some collaboration with partners in China who had asked if they'd be able to conduct video conferencing over IPv6. This was the first time we encountered direct external pressure to deploy IPv6. I'm sure it won't be the last.

The one subdivision within the university that does run their own network infrastructure, the School of Engineering & Applied Science, has been an early adopter, and has been running IPv6 in their part of the network since 2007.

In the summer of 2011, we took advantage of the increased interest generated by last year's World IPv6 Day event to extend the deployment of IPv6 to most of the rest of the campus wired network. The one area that was significantly lagging was the wireless network. This was a bit more challenging because of known bugs in our wireless controller vendor's gear (Aruba Networks) which necessitated a code upgrade. That code upgrade did not happen until earlier this year, so we're still in the midst of IPv6 deployment on wireless. As of this writing, 70 wireless subnets (out of roughly 200) have IPv6 available, and we should have the entire wireless network done sometime later this summer.

For the more technically inclined, we run Integrated IS-IS as our interior routing protocol for IPv6, whereas we continue to run OSPF for IPv4. At the time when we were initially testing IPv6, that was clearly the best choice since OSPF version 3 (the new version of OSPF that supports IPv6) was still in a relatively fledgling state of implementation maturity. Also confining IPv6 to a separate routing protocol seemed like a good additional safety measure. We run a single flat Level-2 area for the entire campus. For exterior routing, we have separate BGP peerings over IPv6 transport established with our external peers that carry IPv6 routes only. Our initial deployment used a provider allocated /48 IPv6 block delegated to us by MAGPI. In 2008, we obtained a Provider Independent ("portable") /32 sized IPv6 address block (2607:F470::/32) from the regional registry ARIN, and have mostly renumbered into it.

Currently, Penn's only connection to the IPv6 Internet is via MAGPI and Internet2. But we're planning to turn up IPv6 peering on our direct commercial ISP links (Level3 and Cogent) in the very near future. At least one of them might happen before World IPv6 Launch.

IPv6 enabled servers use statically configured addresses. Clients on campus almost exclusively use stateless address autoconfiguration (including the privacy/temporary address extensions). DHCPv6 has not been an option for us until recently, since we're a 40% Mac campus, and Apple didn't support DHCPv6 until Mac OS X version 10.7 (late summer 2011).  We are developing plans for a possible DHCPv6 service in the future, which I'll elaborate on at a later time.

Application Services

Penn's authoritative DNS service has been IPv6 enabled for many years. The campus DNS resolvers also support DNS queries over IPv6 but since we don't yet run DHCPv6, we don't have a convenient way to hand out their IPv6 addresses. Our homegrown DNS content management system has supported the ability to create AAAA and IPv6 PTR records for a long time also.

A number of departmental web servers, including the School of Engineering & Applied Science, are IPv6 enabled. The Penn central jabber server, jabber.upenn.edu, was one of our earlier IPv6 equipped services, and actually sees a high proportion of IPv6 activity. Work is proceeding on many other services.

Some of the most challenging services have been those where components of the service have been outsourced to commercial third parties. The central Penn webserver, www.upenn.edu is located on the Akamai content delivery network, and Akamai has been slow to deploy IPv6. We successfully worked with Akamai to put the website on IPv6 for last year's world IPv6 day (June 8th 2011), but they were not then prepared to offer it on an ongoing production basis. In April 2012, Akamai finally announced production IPv6 support. As of May 9th, the Penn website is now available over IPv6, hopefully permanently this time.

Akamai uses DNS resolver client addresses to direct users to content servers geographically close to them (although a few other factors including load are also considered by the server selection algorithm). I collected some data with the help of colleagues about where the www.upenn.edu AAAA record resolves to from various locations. Since we host a cluster of IPv6-enabled Akamai content  servers on our campus network, most of the time, on-campus users of www.upenn.edu will be directed to these local servers.

One issue we overlooked, is that there is a version of the main Penn website optimized for small form-factor mobile devices ("m.upenn.edu") which is not on the Akamai CDN, and run by another unit within the IT organization that has not yet deployed IPv6. So, more work remains to get the Penn web presence completely IPv6 ready.

The other challenging service is central e-mail. Penn uses Message Labs (now Symantec Cloud) to scan e-mail for viruses and spam scoring. As a result both inbound and outbound e-mail has to go through Symantec Cloud's servers. We've inquired about IPv6 support for a number of years, but even today, they appear to have no plans to support it. Our latest communication from them (early May 2012) indicates that they have no plans for any IPv6 support for FY13 (their fiscal year starts in April), and that this may change as priorities shift. At some point, we too might be compelled to shift our priorities and end our relationship with Message Labs, and either seek another provider (does Google/Postini do IPv6 yet?) or bring back virus & spam filtering in-house.

For a comparative view of externally visible IPv6 enabled application services deployed at various US universities and other organizations, Mark Prior's IPv6 survey website is a good resource. Of the five services measured there (Web, DNS, Mail, NTP, and Jabber), Penn gets a green box for four - Mail is the missing one because of Symantec Cloud.

Other Projects

From time to time, we've worked with Penn researchers and outside companies on IPv6 related projects. In the fall of 2009, we worked with Alain Durand (then at Comcast) and Roch Guerin (Penn engineering school faculty) on a small trial deployment of Dual Stack Lite; see RFC 6333 for details of this protocol - this was mostly to help Comcast out. It's unlikely that Penn will deploy DSLite in our own production network. We've also worked with Roch and Comcast on an ongoing IPv6 adoption measurement project. Details of this project are available at: http://mnlab-ipv6.seas.upenn.edu/

Facilitating Regional Connectivity

As mentioned earlier, Penn enables IPv6 connectivity for regional institutions via the MAGPI GigaPoP and Internet2. Currently, we provide IPv6 connectivity to the following institutions: Princeton University, New Jersey Edge (the state education network for NJ), Lafayette College, and Rutgers University. Of them, Princeton came up first in 2005.

Traffic Measurements

Looking at some recent data, IPv6 traffic traversing the campus border is roughly 3% of the total inbound and about 1% of the total outbound traffic. Internal traffic is probably a slightly higher percentage. We're just starting to deploy better measurement infrastructure for IPv6, so we'll have more comprehensive data in the future. But I'll be writing another article sharing what we have so far next.

Links 

• World IPv6 Launch website
• Internet Society's Deploy360 Programme (helping people deploy IPv6, DNSSEC, etc)
• Penn computing's IPv6 page
• Penn IPv6 Strategy Document from March 2009
• An IPv6 Tutorial/Training course I gave recently (PICC 2012)
• IPv6 Campus Deployment Updates Panel, Joint Techs Conference, Feb 2010
• Mark Prior's IPv6 Deployment status survey
• Monitoring IPv6 Content Reachability, Joint Techs Conference, July 2010