Monday, June 10, 2013

100 Gigabit Ethernet at Penn

This summer, the University of Pennsylvania is upgrading its campus core routing equipment (in fact we're in the midst of this upgrade right now). This is basically an upgrade to the set of large routers that form the center of our network.

The current core topology consists of 5 core routers (and also 2 border routers) interconnected by two independent layer-2 switched 10 Gigabit Ethernet networks. Each of the core routers is located in one of five geographically distributed machine rooms across the campus. A rough diagram is shown below.

This diagram also shows the current external connections to/from the campus network - we have three links (each 10 Gigabit Ethernet) to Internet Service Providers (ISPs). And two connections to MAGPI (the regional Internet2 GigaPoP operated by us), via which we access a 10 Gigabit Ethernet connection to Internet2. The Internet2 connection is shared amongst Penn and other MAGPI customers, which are mostly Research & Education institutions in the geographic area.

The core interconnect is being upgraded to 100 Gigabit Ethernet (a ten-fold increase in link bandwidth). It would be cost prohibitive to fully replicate the current design in 100 Gig (since this equipment is still very expensive) so the interconnect design has been adjusted a bit. Instead of two layer-2 switch fabrics interconnecting the routers, we are deploying the core routers connected in a 100 Gig ring (see the diagram below). When the final design is fully implemented, each core router will have a 10 Gig connection into each of the border routers (this will require some additional upgrades to the border routers, which are expected to happen later this year). The topology redesign has fewer links, and in the final count (summing the bandwidth of all the core facing links), the new core will have about 5 times the aggregate bandwidth of the old one. The maximum (shortest path) edge to edge diameter of the network increases by one routing hop.

100 Gigabit Ethernet is today's state of the art in transmission speed. The next jump up will likely be 400 Gigabit Ethernet for which the IEEE already has a study group launched and several preliminary designs under consideration.

Not depicted in this diagram is the rest of the network towards the end systems. Below the layer of core routers, are smaller routers located at the 200 or so buildings scattered around campus. Each building router is connected to two of the core routers. The building routers feed wiring closets inside the building, which house layer-2 switches that network wallplates are connected to.

In the process of the upgrade, we are also changing router vendors. The current core routers, Cisco 7609 series routers with Sup7203BXL supervisor engines, have served us well. They were originally deployed in the summer of 2005, and have been in operation well past their expected lifetime.

As is our practice, we issued an RFI/P (Request for Information/Purchase) detailing our technical requirements for the next generation routers and solicited responses from the usual suspects, selecting a few vendors whose equipment we bring in for lab testing, followed by a selection.

The product we've selected is the Brocade MLXe series router, specifically the MLXe-16 - this router can support 16 half height, or 8 full height (or a mixture of full/half) line cards, as well as redundant management and switch fabric modules.

A product description of the MLXe series is available at:

The photo below is one of the routers (prior to deployment) in the Vagelos node room (one of 5 machine rooms distributed around campus where we house critical networking equipment and servers).Going from left to right, this chassis has two management modules, one 2-port 100 Gigabit Ethernet card, six 8-port 10 Gigabit Ethernet cards, four switch fabric modules, two more 8-port 10 Gigabit Ethernet cards, and three 24-port Gigabit Ethernet cards.

One of these routers was deployed in production last week. The rest should be up and running by the end of this month or by early July.

(The full set of photos can be seen here on Google Plus)

Shown below is the 2-port 100 Gigabit Ethernet card, partially inserted into the chassis, showing the CFP optical transceiver modules attached.

Unlike preceding generations of ethernet, with 100 Gigabit Ethernet, the transmission technology uses multiple wavelengths in parallel (although there are parallel fiber implementations also). The current IEEE specifications (802.3ba) specify four lanes of 25Gbps. However a number of key vendors in the industry, including Brocade, formed the MSA (Multi Source Agreement) and designed and built a 10x10 (10 lanes of 10Gbps) mechanism of doing 100 Gig, at much lower cost than 4x25Gbps, operating over single mode fiber at distances of 2, 4, or 10km. This is called LR-10 and uses the CFP (C Form factor pluggable) media type.

Pictured below (left) is a Brocade LR10 100 Gigabit Ethernet CFP optical module installed in 100 Gig line card with a single mode fiber connection (LC). On the right is an LR10 CFP module taken out of the router.

Close up of the 8-port 10 Gigabit Ethernet module, and several 24-port Gigabit Ethernet modules. To connect cables to them, we need to install small form factor pluggable transceivers into them, SFP+ for the 10 gig, and SFP for the 1 gig.

Pictured below is one of the five Cisco 7609 routers that will be replaced.

One of the Penn campus border routers, a Juniper M120, is shown below. This is also scheduled to be upgraded in the near future to accommodate 100 Gig and higher density 10 Gig, although the product has not yet been selected/finalized.

Below: A Ciena dense wavelength division multiplexer (DWDM). Penn uses leased metropolitan fiber to reach equipment and carriers in 401 North Broad street, the major carrier hotel in downtown Philadelphia. We have DWDM equipment placed both at the campus and the carrier hotel to carry a mixture of 1 and 10 Gig circuits and connections across this fiber for various purposes. This equipment is also scheduled to be upgraded to allow us to provision 100 Gigabit Ethernet wavelengths between the campus and the carrier hotel.

High Performance Networking for Researchers

Penn is a participant in the National Science Foundation (NSF) funded DYNES (Dynamic Network Systems) project, which provides high bandwidth dedicated point to point circuits between (typically) research labs for specialized applications. Popular uses of this infrastructure today include high energy physics researchers obtaining data from the LHC and other particle accelerator labs, and various NSF GENI network research projects.

Earlier this year, we completed a grant application for the  NSF "Campus Cyberinfrastructure - Network Infrastructure and Engineering (CC-NIE)" program. I spent a large amount of time in March of this year with several Penn colleagues in preparing the application. If Penn does win an award (we'll find out later this year), we will be deploying additional dedicated network infrastructure for campus researchers, bypassing the campus core and with 100 Gbps connectivity out to the Internet2 R&E network. A rough diagram of how this will look is below.

Software Defined Networking

There's a huge amount of buzz about Software Defined Networking (SDN) in the networking industry today, and a number of universities are investigating SDN enabled equipment for deployment in their networks. Of the big router vendors, Brocade does appear to have one of the better SDN/openflow stories thus far. The MLXe series already supports an early version of Openflow (the portion of SDN that allows forwarding tables of switches/routers to be programmed by an external SDN controller).

Penn is building an SDN testbed in our network engineering lab, primarily to investigate its capabilities. For us, SDN is still largely a solution in search of a problem. We run a very simple network by design, whose primary purpose is connectivity and high performance packet delivery. The most probable use case in our future, virtualization of the network, is likely better achieved with a proven technology like MPLS first. But we'll keep an eye on SDN and its evolution. We do want to support research uses of SDN though. Several faculty members in the Computer Science department are interested in SDN, and the NSF CC-NIE grant will allow us to build some SDN enabled network infrastructure separate from the core production network to accommodate their work.

-- Shumon Huque


  1. I am not a professional in a computer science but I have always thought that such big computers remained in the past. I was so wrong.

  2. Specific Discount and Voucher Codes. Need to get a decent arrangement on your liked things? You've come to the best regions. Here we have stunning vouchers.
    By: anns cottage outlet

  3. Wonderful piece you got here , I feel quite happy after I read your Article because I found out that we still have intelligent informative researchers who can still take time out of their busy schedule to put all this together on Do My Assignment For Me uk

  4. The Juniper M120, one of the Penn campus border routers, is illustrated below. This is also set to be improved in the near future to support Cheap Assignment Help and higher-density 10 Gig, though the product has yet to be chosen/finalized.

  5. Hello guys! Our academy hand-pick the best candidates among those seeking to get a position of an academic writer on our team. Fulfilling all writing assignments from scratch is must-do for every expert, new and already-established alike. Meeting deadlines and staying open to communicating with a customer is another pillar of our writing policy.

  6. In 1973, the Xerox Palo Alto Research Center in California developed Ethernet, which is currently the most extensively used local area network technology. The purpose of its creation was to connect several computer workstations, servers, and printers so that data could be sent between them. This was a significant feat in an era when massive, costly mainframe computers reigned. An Ethernet connection could only transfer 2.85 megabits per second at first (Mbps). The DIX Ethernet standard was established in 1980 by a vendor consortium consisting of Digital Equipment Corporation, Intel, and Xerox. As a result, Ethernet networks were limited to 10 Mbps maximum speeds far into the 1980s. Get similar to the game Twisted Metal in our site.

  7. Therefore, buy term paper from us and free up your time to concentrate on essential commitments. Catch up with your family and friends, pursue your passion outside class and relax while we handle all your assignments.

  8. One of the best courses I have taken till date. Lots of Learning and Very interesting assessment methods. Indeed looking forward to connect with similar minds on What's Future for education. 200-901 - DevNet Associate (DEVASC)

  9. One way to publish your work is to create a blog. You can also create a website and use services like Amazon to sell your books. You can also make money by charging people for an online book publishing. If you want to make money, the best way is to create a book, and then use it to sell other books or make money by charging people for it.

  10. Offer top-quality academic writing pro help UK in all areas, with a focus on assignment assistance, proofreading, and law dissertation assistance. I truly enjoy working with Academic Writing Pro since they are reliable and skilled. The support staff is available around-the-clock. I endorse their offerings. You can get assistance from essay master UK with any problem. Select us, unwind, and let the experts handle your essay and assignments. No matter how challenging, perplexing, or urgent the deadline, our experts are here to help you with your academic assignments.

  11. Organizations may analyse their carbon data, make informed decisions, and lower their greenhouse gas emissions with the help of Net Zero Cloud, a carbon accounting system. For investors and shareholders searching for businesses with a long-term sustainable strategy, becoming a net zero business may be appealing. Investors might be drawn to smaller companies that depend less on fossil fuels since their operations are less likely to be affected by supply issues.

  12. Thank you for sharing amazing server details, I really appreciated .


  13. You have written a very informative article. Shown below is the 2-port 100 Gigabit Ethernet card, partially inserted into the chassis, showing the CFP optical transceiver modules attached. Thanks for sharing all your pictures with us. Now it's time to get Taxi Stratford upon Avon for more information.

  14. A very detailed explanation was provided on this topic. I really appreciate your writing technique, which is impressive and enjoyable to read. Unlike preceding generations of Ethernet, with 100 Gigabit Ethernet, the transmission technology uses multiple wavelengths in parallel. Keep sharing more information about this article. Now it's time to get Best Roofing Services in Wayne NJ for more information.

  15. Digital consultants are professionals who specialize in providing guidance and support for businesses looking to enhance their online presence. They can help with a range of services, including website design

  16. The roofing system plays a crucial role in defining a house as a home, just like the memories we build. Top Roofing Companies .Even if your home is meant to be a haven from the outside world, it doesn't feel like much of one when entering doesn't provide shelter from the rain. With just one call to our roofing company, comfort and tranquility can be restored. We have a prospective workforce team of professionals who have years of industry expertise and who deliver exceptional customer service to resolve your roofing issues.

  17. spotify is the best music streaming platform in the world today. Update the latest version of spotify full apk now and enjoy top music

  18. I value the blog article. Really Great.

  19. 100 Gigabit Ethernet at Penn? Impressive! Cutting-edge technology like this is driving innovation and connectivity in research and business. The future is fast, and Penn is leading the way.

  20. I have shared your blog with friends and they are also impressed. Refer to other blog articles on the topic of games and free apk applications for mobile phones at apkgosu

  21. Wow, this article on "100 Gigabit Ethernet at Penn" is incredibly informative and exciting! As a technology enthusiast, I must say that the advancements in networking speeds never fail to amaze me. The implementation of 100 Gigabit Ethernet at Penn showcases the institution's commitment to staying at the forefront of technology.
    The benefits of this high-speed network are vast. It not only facilitates faster data transmission but also opens up a realm of possibilities for researchers, educators, and students at Penn. With such lightning-fast speeds, collaborative projects and data-intensive research will become more seamless and efficient.
    I am especially thrilled to see how this new infrastructure will revolutionize the educational experience. Students will have access to unprecedented online resources, real-time collaborations, and immersive virtual learning environments. It's a game-changer for education in the digital age.
    Kudos to Penn for taking this technological leap and investing in the future! I can't wait to witness the transformative impact of 100 Gigabit Ethernet on campus life and beyond.

    Un Disputa Acerca de

  22. Saya sangat menyukai mod whatsapp yang luar biasa ini, Anda juga dapat Melihat detailnya

  23. Looking for a Bank Ruptcy Lawyer Near Me, Our skilled team is provide a legal lawyer for your case

  24. Huque's blog is a captivating online platform that offers a unique blend of insightful commentary, engaging narratives, and thought-provoking content. With a diverse range of topics spanning from current events to personal reflections, Huque's blog provides readers with a multifaceted perspective on various issues. violation of protective order virginia

  25. I find myself continually impressed by the depth and quality of your blog posts. Your steadfast dedication to providing valuable information is evident in every article. Your blog remains a dependable source of knowledge and inspiration. I genuinely appreciate your ongoing efforts to produce such insightful content.New Jersey Domestic Violence Attorney and New Jersey Domestic Violence Registry