World IPv6 Launch Measurements

The Internet Society has posted their latest IPv6 measurements (December 12th 2013). Read the section titled "Notes on network operator measurements" to understand how the measurements are being made and which content providers (Google, Facebook, Yahoo!, Akamai) are providing the data.

I've pulled out some of the data, and put together ranked (top 100) lists of networks by two measures: (1) percentage of requests that used IPv6, and (2) total volume of IPv6 requests. As I've done a few times in the past, I'm going to continue periodically writing these entries to have a snapshots in time of IPv6 deployment progress.

Many networks are posting some pretty impressive numbers for IPv6 usage. For the leading network in the %v6 category (the 1st list below), TOP-IX Consortium, an Italian Internet Exchange point has 86% of their requests to the participating content providers using IPv6! Several universities in the US R&E community are doing well too, Gustavus Adolphus College at 74%, Virginia Tech at 62%, University of New Hampshire at 51% among them. The University of Pennsylvania (my own institution) is posting a respectable 40% - we'll have IPv6 fully deployed on our wireless network in early 2014, at which time our numbers should go up substantially. There's an interesting story about why Penn hasn't had IPv6 on its wireless network for so long - I'm planning to write a separate article on that topic in the near future.

In the total volume category (the 2nd list below) Comcast now leads. John Brzozowski, Comcast's chief IPv6 architect, has written a more detailed article on their leadership position in IPv6 deployment. They are followed by several other ISPs: AT&T (US), KDDI (Japan), Free (France), Verizon Wireless, (US) Deutsche Telekom (Germany), RCS & RDS (Romania), Time Warner Cable (US).


World IPv6 Launch Measurements, by % IPv6 requests:
---------------------------------------------------

   1 TOP-IX Consortium    86.27%
   2 Fundacao Parque Tecnologico Itaipu - Brasil    79.65%
   3 DirectVPS         77.66%
   4 ThaiSarn         76.62%
   5 Gustavus Adolphus College   17234    74.17%
   6 Google Fiber      70.22%
   7 Universidad Panamericana    13679    66.84%
   8 mur.at - Verein zur Frderung von Netzkwerkkunst    66.73%
   9 interscholz Internet Services GmbH & Co. KG     66.20%
  10 Virginia Tech     61.69%
  11 Trunk Networks Limited     56.99%
  12 Association tetaneutral.net    56.41%
  13 DegNet GmbH     51.91%
  14 Ponto de PresenC'a da RNP na Bahia     51.75%
  15 Critical Colocation   51.38%
  16 Jiri strohalm    51.29%
  17 SPAWAR     51.12%
  18 ITsjefen AS     51.08%
  19 SuperInternet Access Pte Ltd     50.88%
  20 University of New Hampshire     50.57%
  21 AIMES Grid Services CIC     50.38%
  22 Region 7 ESC     50.27%
  23 PREGINET 50.24%
  24 Maxiweb Internet Provider    50.20%
  25 CICA/Junta de AndalucC-a    50.04%
  26 DreamHost    49.41%
  27 Alhambra Eidos    49.32%
  28 Karlsruhe Institute of Technology (KIT)    48.85%
  29 Sauk Valley Community College     46.13%
  30 University of Minnesota           45.87%
  31 Marist College     45.83%
  32 AMS-IX     44.09%
  33 University of Iowa     42.90%
  34 Kasetsart University     41.31%
  35 Verizon Wireless     40.40%
  36 University of Pennsylvania     40.06%
  37 Bulgaria NREN     38.74%
  38 guifi.net     38.69%
  39 Rensselaer Polytechnic Institute    37.91%
  40 Louisiana State University     35.74%
  41 Leibniz Supercomputing Centre     35.17%
  42 Netwerkvereniging Coloclue     32.66%
  43 Free       31.03%
  44 UNESP       30.15%
  45 NetAssist       29.72%
  46 ARNES       28.98%
  47 Tulane University     28.88%
  48 Utility Line Italia srl     28.19%
  49 Host Virtual, Inc     27.79%
  50 Hughes Network Systems     27.28%
  51 FCCN   26.96%
  52 UFSCar     26.36%
  53 VOO     25.94%
  54 Opera Software ASA     25.69%
  55 Greek Research & Technology Network    24.94%
  56 UNINETT        24.58%
  57 LENTEL         23.48%
  58 Chubu Telecommunications    22.76%
  59 RCS & RDS     22.01%
  60 mc.net     20.20%
  61 Comcast    20.15%
  62 Monash University     19.82%
  63 Swisscom     19.64%
  64 UFSC - Universidade Federal de Santa Catarina - Brazil    19.25%
  65 XS4ALL 18.52%
  66 AAISP  18.13%
  67 SIDN   17.11%
  68 EPT Luxembourg    16.72%
  69 manitu GmbH    15.88%
  70 VentraIP Group (Australia) Pty Ltd    15.73%
  71 LITNET   15.09%
  72 Hurricane Electric    14.88%
  73 ATT     14.82%
  74 NIIF/Hungarnet    14.43%
  75 Funet        12.60%
  76 CESNET        12.33%
  77 Deutsche Telekom AG    12.28%
  78 Indiana University        12.14%
  79 OVH           11.63%
  80 FranTech Solutions    10.98%
  81 Red Academica de Centros de InvestigaciC3n y Universidades Nacionales REACCIUN    10.95%
  82 UniNet       10.48%
  83 Host.MD      10.35%
  84 Belnet       9.73%
  85 Cisco       9.70%
  86 CJSC Progressive Technologies    9.65%
  87 KDDI     8.87%
  88 Academia Sinica Network     8.22%
  89 SARENET  8.01%
  90 DMZGlobal     7.99%
  91 SWITCH     7.86%
  92 Init7     7.70%
  93 MediaCat Div./Community Netowork Center Inc.    7.52%
  94 AMRES - Serbian National Research and Education Network    7.10%
  95 RedIRIS      6.74%
  96 T-Mobile USA      6.49%
  97 Voxel / Internap     6.48%
  98 Defense Research and Engineering Network    6.41%
  99 prgmr.com      6.35%
 100 M1 Limited      6.29%


World IPv6 Launch Measurements, by volume of IPv6:
--------------------------------------------------

   1 Comcast    20.15%
   2 ATT    14.82%
   3 KDDI    8.87%
   4 Free     31.03%
   5 Verizon Wireless     40.40%
   6 Deutsche Telekom AG    12.28%
   7 RCS & RDS          22.01%
   8 Time Warner Cable     4.07%
   9 Liberty Global    2.52%
  10 Telefonica del Peru    5.14%
  11 Swisscom    19.64%
  12 SoftBank BB     1.65%
  13 Hughes Network Systems     27.28%
  14 Chubu Telecommunications     22.76%
  15 Opera Software ASA     25.69%
  16 VOO   25.94%
  17 XS4ALL     18.52%
  18 China Telecom    0.18%
  19 Janet         4.29%
  20 T-Mobile USA    6.49%
  21 Forthnet         3.35%
  22 StarHub        4.81%
  23 University of Minnesota     45.87%
  24 Indiana University        12.14%
  25 CESNET  12.33%
  26 Google Fiber     70.22%
  27 M1 Limited     6.29%
  28 Virginia Tech    61.69%
  29 Internode        4.53%
  30 FCCN        26.96%
  31 EPT Luxembourg     16.72%
  32 Cisco        9.70%
  33 Belnet        9.73%
  34 Louisiana State University     35.74%
  35 RedIRIS   6.74%
  36 UNINETT   24.58%
  37 Leibniz Supercomputing Centre    35.17%
  38 its communications Inc.(iTSCOM)     3.14%
  39 SWITCH     7.86%
  40 NIIF/Hungarnet     14.43%
  41 ARNES     28.98%
  42 MediaCat Div./Community Netowork Center Inc.    7.52%
  43 BelWue     5.87%
  44 LITNET     15.09%
  45 University of Pennsylvania    40.06%
  46 NTT Communications     4.38%
  47 RENATER     4.10%
  48 Kasetsart University     41.31%
  49 University of Iowa     42.90%
  50 TUBITAK ULAKBIM / ULAKNET     1.29%
  51 OVH     11.63%
  52 Tulane University    28.88%
  53 Monash University     19.82%
  54 UNESP  53166     30.15%
  55 AMRES - Serbian National Research and Education Network    7.10%
  56 Rensselaer Polytechnic Institute  37.91%
  57 UFSC - Universidade Federal de Santa Catarina - Brazil    19.25%
  58 University of Wisconsin - Madison      4.84%
  59 Gustavus Adolphus College 74.17%
  60 Funet       12.60%
  61 GARR       1.25%
  62 Marist College     45.83%
  63 SPAWAR     51.12%
  64 SURFnet     1.36%
  65 SuperCSI     2.71%
  66 Altibox AS     0.70%
  67 AAISP       18.13%
  68 Australian Academic and Research Network (AARNet)    2.32%
  69 Karlsruhe Institute of Technology (KIT)  48.85%
  70 Xfone 018       1.11%
  71 UFSCar       26.36%
  72 Voxel / Internap    6.48%
  73 Solcon         5.12%
  74 UniNet         10.48%
  75 CJSC Progressive Technologies    9.65%
  76 CICA/Junta de AndalucC-a     50.04%
  77 Starlink    1.47%
  78 Hurricane Electric     14.88%
  79 Greek Research & Technology Network     24.94%
  80 Jiri strohalm  51.29%
  81 JARING Communications Sdn Bhd    0.27%
  82 Defense Research and Engineering Network    6.41%
  83 GITN Sdn Berhad  3.11%
  84 Dhiraagu         0.85%
  85 Academia Sinica Network    8.22%
  86 green.ch AG         2.83%
  87 Louisiana Optical Network Initiative    5.04%
  88 LENTEL    23.48%
  89 Fundacao Parque Tecnologico Itaipu - Brasil    79.65%
  90 DegNet GmbH         51.91%
  91 National Informatics Centre    2.19%
  92 guifi.net     38.69%
  93 GlobalConnect     1.54%
  94 The Tertiary Education and Research Network of South Africa (TENET)    1.53%
  95 DMZGlobal      7.99%
  96 Init7      7.70%
  97 SoftLayer Technologies    1.40%
  98 Ponto de PresenC'a da RNP na Bahia    51.75%
  99 Storm Internet    5.40%
 100 inexio KGaA    1.15%

EDU Top Level Domain statistics

Some DNS Top Level Domain (TLD) operators publish statistics about their DNS zones. Some others have a zone file access program that allows others to examine their data and publish statistics. Frederic Cambus (@fcambus on Twitter) maintains a site called statdns ( http://www.statdns.com/ ) that keeps statistics for several of the TLDs.

The EDU top level domain is conspicuously absent from the statdns site because the operators don't publish any statistics and also don't have a zone file access program in place. The EDU domain has a very complicated operational policy arrangement. It is managed by Educause (a higher education IT consortium), but operated by Verisign, under a contract with the United States Department of Commerce. I recently spoke with colleagues at Educause about current prospects for publishing some statistics or making the zone data available. The good news is that a zone file access program request is in fact in the queue to be approved by the Dept of Commerce. But it's stuck behind a few other requests, so it may still take some time to come to fruition.

In the meantime, to satisfy my own curiosity, I've been looking at other ways to obtain some statistics. In particular I'm interested in seeing how much DNSSEC deployment has happened so far, and how EDU compares with some of the other TLDs in this respect. One way to gain visibility into zone contents is to examine passive DNS databases. A number of folks and organizations run such databases that collect historical information seen from DNS responses at collections of resolvers. By searching records over a period of time in these databases, it's possible to reconstruct a substantial portion of the active records in a zone. I did this for EDU and analyzed the results recently.

The passive DNS database search managed to find about 7,158 second level domains under EDU. Of these, 6955 domains turned out to be valid (the others probably existed at one point but don't any more). EDU is known to have in the neighborhood of 7,000 delegations, so this is most probably a pretty good approximation of the active contents of the zone.

EDU Zone Statistics:

Number of Domains from passive DNS db: 7158
Number of Valid Domains: 6955

Total NS records: 19527
Unique NS records: 9757
Number of (glue) IPv4 address records: 4555
Number of (glue) IPv6 address records: 246

DNSSEC Specific Stats for EDU:

Number of DNSSEC Signed Zones: 94 (1.37%)
Number of NSEC3 Zones: 29 (30.1% of the signed zones)
Number of Zones with DS records: 76
Number of Zones with DLV records at dlv.isc.org: 7

As expected, only a very small fraction (1.37%) of domains have deployed DNSSEC. This compares with about 0.25% in .COM, 0.41% in .NET, and 0.30% in .ORG.

The 94 zones in EDU signed with DNSSEC are:

acadiana.edu
baker.edu
beloit.edu
berkeley.edu
bucknell.edu
cameron.edu
carnegiemellon.edu
catc.edu
chattanoogastate.edu
cltc.edu
cmu.edu
coloradomesa.edu
cookman.edu
csupomona.edu
cuhk.edu
desales.edu
drake.edu
example.edu
fhsu.edu
fhtc.edu
gfcmsu.edu
gsu.edu
gtc.edu
hfg.edu
highlands.edu
indiana.edu
indianatech.edu
internet2.edu
iu.edu
iub.edu
iup.edu
iupui.edu
jhuapl.edu
kestrel.edu
kiropraktik.edu
k-state.edu
ksu.edu
kutztown.edu
lctcs.edu
lsu.edu
ltc.edu
ma.edu
mansfield.edu
mcpherson.edu
merit.edu
mesa.edu
millikin.edu
mnsfld.edu
monmouth.edu
monterey.edu
mst.edu
nau.edu
northcentral.edu
northshorecollege.edu
nwltc.edu
okstate.edu
pacificu.edu
penn.edu
pitt.edu
psc.edu
richland.edu
rockefeller.edu
rose-hulman.edu
scl.edu
sdsmt.edu
sfcollege.edu
shoreline.edu
suu.edu
tbu.edu
tiasnimbas.edu
tilburguniversity.edu
tiss.edu
truman.edu
uaa.edu
ualr.edu
ucaid.edu
ucb.edu
ucberkeley.edu
ucdavis.edu
ucr.edu
uiowa.edu
umbc.edu
uni-stuttgart.edu
unt.edu
untsystem.edu
upenn.edu
upf.edu
usnwc.edu
uwm.edu
uwstout.edu
valencia.edu
waketech.edu
washjeff.edu
weber.edu

The 29 zones that use the NSEC3 variety of DNSSEC are:

csupomona.edu
cuhk.edu
gfcmsu.edu
internet2.edu
jhuapl.edu
kestrel.edu
kiropraktik.edu
k-state.edu
ksu.edu
lsu.edu
ma.edu
mansfield.edu
mcpherson.edu
millikin.edu
mnsfld.edu
pitt.edu
richland.edu
rose-hulman.edu
sdsmt.edu
suu.edu
tiasnimbas.edu
tilburguniversity.edu
ualr.edu
ucaid.edu
ucr.edu
uni-stuttgart.edu
unt.edu
untsystem.edu
washjeff.edu

There are 18 zones that do not have DS records published (not sure why):

beloit.edu
cameron.edu
cookman.edu
iup.edu
kiropraktik.edu
kutztown.edu
mansfield.edu
merit.edu
mnsfld.edu
okstate.edu
shoreline.edu
tbu.edu
tiasnimbas.edu
uaa.edu
usnwc.edu
uwm.edu
uwstout.edu
waketech.edu

There are also 7 zones with DLV records published at ISC's DLV registry, but this set is disjoint with the set that doesn't have DS records:

bucknell.edu
internet2.edu
k-state.edu
ksu.edu
ualr.edu
ucaid.edu
ucr.edu

-- Shumon Huque

New DNS Top Level Domains

If you follow DNS news, you may know that ICANN has put in place a program to introduce many new generic top level domains (GTLD) into the DNS. I haven't been a fan. ICANN says there is market demand for GTLD expansion (perhaps), and that it allows innovation in the DNS ecosystem (how?). It probably will have an effect of diluting the entrenched market power of the big TLD operators (.com, .org etc), which may be a good thing. But the system may end up being primarily a significant financial windfall for ICANN. Even Esther Dyson (original ICANN chair) has spoken out against the program.

There appear to be some trademark protection mechanisms built in to the new system. But it seems clear that many organizations will rush to defensively register their names under some of the new TLDs. Strictly speaking, DNS domain names have no intended or actual relation to trademarks, but we have to deal with the real world. My university's upper administration has already contacted the IT department to discuss the topic. A while back, we defensively registered "upenn.xxx" to protect against possible reputational damage (and no, I wasn't involved in that decision).

On a more technical note, one interesting and welcome feature of the new GTLDs, is that they must be deployed with DNSSEC. This should significantly increase the proportion of signed top level domains in the DNS. My dnsstat DNS monitoring site has been monitoring the TLDs for a while now, and I just updated it with the latest list of TLDs.

    http://www.huque.com/app/dnsstat/category/tld/

Since late August, 32 new TLDs have been introduced, 27 normal GTLDs, 5 IDN (Internationalized domains) TLDs. But 11 IDN TLDs have also disappeared. That's a net gain of 21 TLDs, bringing the total count to 339.

Some DNSSEC specific stats: 143 (or 42.2%) of the TLDs are signed with DNSSEC. Here's a breakdown of type key and zone signing algorithms in use for the signed TLDs:

Key Signing Keys (KSK):
RSASHA256 (8) = 119 (63.0%)
RSASHA512 (10) = 6 (3.2%)
RSASHA1 (5) = 16 (8.5%)
RSASHA1-NSEC3-SHA1 (7) = 48 (25.4%)

Zone Signing Keys (ZSK):
RSASHA256 (8) = 133 (62.4%)
RSASHA512 (10) = 8 (3.8%)
RSASHA1 (5) = 17 (8.0%)
RSASHA1-NSEC3-SHA1 (7) = 55 (25.8%)

Note: new GTLDs continue to be added, so the numbers in this article might be out of date soon.

Here are the added TLDs so far (as of November 20th 2013):

+ bike
+ camera
+ clothing
+ construction
+ contractors
+ diamonds
+ directory
+ enterprises
+ equipment
+ estate
+ gallery
+ graphics
+ guru
+ holdings
+ kitchen
+ land
+ lighting
+ photography
+ plumbing
+ sexy
+ singles
+ tattoo
+ technology
+ tips
+ today
+ ventures
+ voyage

Here are the new IDN TLDs:

+ xn--80asehdb
+ xn--80aswg
+ xn--mgba3a4f16a
+ xn--ngbc5azd
+ xn--unup4y

Here are the deleted IDN TLDs:

- xn--0zwm56d
- xn--11b5bs3a9aj6g
- xn--80akhbyknj4f
- xn--9t4b11yi5a
- xn--deba0ad
- xn--g6w251d
- xn--hgbk6aj7f53bba
- xn--hlcj6aya9esc7a
- xn--jxalpdlp
- xn--kgbechtv
- xn--zckzah

Note: one IDN TLD (xn--l1acc) has had a severely busted DNSSEC deployment for a while. My monitoring system detects that its DS records in the root of the DNS do not match any DNSKEY records in the zone, and furthermore, the signatures on the DNSKEY records have expired. I hope they get their act together soon.

--Shumon Huque

Penn wins NSF Campus CyberInfrastructure Award

A while back in a blog article on our 100 Gigabit Ethernet campus upgrades, I mentioned that Penn had applied for a National Science Foundation (NSF) CC-NIE grant to enhance campus network infrastructure for research purposes.

We did in fact win an award. Here's the official notice from NSF. It's about $500,000 which will be used to deploy a dedicated high performance router for researchers and bump up our external connectivity to Internet2 to 100 Gbps. I hope to provide more updates as we begin deploying the necessary pieces of equipment.

--Shumon Huque

An excerpt from the award notice:

ABSTRACT

The University of Pennsylvania's central computing organization is partnering with leading campus researchers in engineering, physics, biology, pathology, genomics, bioinformatics, and computer science to optimize the campus network in support of big data research and high-performance computing. This project establishes a 100 Gbps-capable Science DMZ that is distinct from the general purpose campus network and is engineered for research applications. Additionally, it extends 10 Gbps connectivity to select research projects and increases Penn's connection to Internet2 from 1 Gbps to 100 Gbps, while also extending that connection to the Science DMZ. The project also lays the foundation for further enhancements to research networking infrastructure by extending IPv6 capabilities; upgrading network monitoring tools such as perfSONAR; and enhancing Penn's ability to support experimental networks and network architectures, including OpenFlow and Software Defined Networking.

The project will benefit a range of scientifically meritorious research. It will provide support for the large-scale data transfer, processing, and storage needs of researchers across Penn, while supporting intra- and inter-institutional collaborations and the broad dissemination of research results. Rather than focusing on the logistics of data storage and transfer, researchers will be able to concentrate on the transformation of these data into the information that will drive new discoveries and the creation of new technologies, drugs, therapies, and cures. Network enhancements will also support Penn's commitment to integrating research and education by supporting the network needs of the cross-disciplinary Penn Institute for Computation Science that where faculty actively integrate computation-based research with the training of future generations of STEM researchers.

TLSA Record Generator

Last year I wrote a blog article on DNSSEC and Certificates. Occasionally I get questions from folks who've tried to follow my instructions to create the content of TLSA records, but have failed because they are using a version of openssl that is too old to generate SHA-256 and SHA-512 hashes.

I've written a small web application to help generate TLSA records. I hope this is of use to some folks:

        https://www.huque.com/bin/gen_tlsa

(I apologize in advance for my rather primitive webpage design skills!)

Here is a screenshot of it in action to generate the TLSA record for my own website:

And the resulting TLSA record that was generated:



-- Shumon Huque

IPv6 and DNSSEC at LISA in DC

Once again, I'm teaching a couple of courses at the USENIX LISA conference, this time in Washington, DC. The first is a half day course on DNSSEC on Sunday, November 3rd. And the second is a full day course on IPv6 on Monday, November 4th. I hope to see you there if you're interested in learning or talking about these topics. Early bird registration discounts for the conference end on October 22nd (sorry for the short notice).

Matt Simmons (@standaloneSA) interviewed me about both classes: DNSSEC and IPv6.

-- Shumon Huque

Singh Nanotechnology Center

The grand opening of Penn's new Singh Center for Nanotechnology is tomorrow (Friday, October 4th). Computing directors in my department got to see it a week early when we held a special meeting in the Forum room.

This post contains a few photos from my visit - of the building exterior, hallways, and conference rooms. The labs weren't open yet. The full set can be found at Google Plus.

The Nanotech center has been featured in some recent articles:

* Philadelphia Inquirer - "Changing Skyline - Inga Saffron"
* The Daily Pennsylvanian
* Philadelphia Inquirer - "Penn going all out for small science"

Building exterior, from Walnut Street close to the 33rd Street intersection, looking east:

This cantilevered section houses a conference room - the Glandt Forum room.

The Forum room, where our meeting was held.

At the edge of the cantilevered section.

Looking westward along Walnut St towards the rest of campus. Directly in front (right side) is the Laboratory for Research on the Structure of Matter (LRSM). To the left is the David Rittenhouse Lab.

View from the green rooftop terrace.

Rooftop terrace.

Hallways.

"We Lost" sculpture by Tony Smith.

Latest World IPv6 Launch Measurements

The Internet Society recently published results of their latest round (September 17th 2013) of IPv6 measurements. The measurement data is provided by Google, Facebook, Yahoo!, and Akamai. From the description on the website: "We present measurements of network operator participants in World IPv6 Launch, based on data received from major website participants, as described in more detail below. We present a simple average of the data received, and list all networks with measurements from at least two sources, with a simple average above 0.1%."

I find it instructive to sort the results by the percentage of requests from each participating network that are composed of IPv6. This is a pretty good indicator of how extensively these networks have deployed IPv6 to their end users.

Note: the measurements are only done for networks that have signed up as participants in World IPv6 Launch. If you've deployed IPv6 to your users, you should consider registering your network to take part in these measurements.

Here's a ranked list of these networks sorted by percentage of IPv6 requests of the total from each.

     1    interscholz Internet Services GmbH & Co. KG    81.22%
     2    Sauk Valley Community College                  71.23%
     3    ThaiSarn                                       69.41%
     4    Rensselaer Polytechnic Institute               61.25%
     5    Virginia Tech                                  59.54%
     6    Universidad de Carabobo                        58.50%
     7    Sistemas Fratec S.A.                           58.19%
     8    Universidad Panamericana                       57.89%
     9    Bayu Krisnawan                                 56.64%
    10    Dedicated Zone Inc                             56.55%
    11    Google Fiber                                   55.64%
    12    REACCIUN                                       52.41%
    13    NETIS TELECOM                                  52.17%
    14    Gustavus Adolphus College                      46.64%
    15    DreamHost                                      46.32%
    16    Alhambra Eidos                                 45.37%
    17    VOO                                            45.32%
    18    SPAWAR                                         45.28%
    19    Greek Research & Technology Network            43.96%
    20    Karlsruhe Institute of Technology (KIT)        43.51%
    21    AIMES Grid Services CIC                        42.37%
    22    Host Virtual, Inc                              42.24%
    23    ARNES                                          41.90%
    24    FCCN                                           40.95%
    25    Marist College                                 40.89%
    26    guifi.net                                      39.97%
    27    University of Pennsylvania                     38.94%
    28    Zimcom Internet Solutions, Inc                 35.75%
    29    Verizon Wireless                               35.73%
    30    NIIF/Hungarnet                                 29.92%
    31    LITNET                                         29.07%
    32    DirectVPS                                      29.05%
    33    Jiri strohalm                                  28.56%
    34    Hughes Network Systems                         28.00%
    35    DegNet GmbH                                    26.76%
    36    Louisiana State University                     26.61%
    37    University of Minnesota                        26.44%
    38    iway AG                                        25.73%
    39    RedIRIS                                        25.39%
    40    University of Iowa                             22.56%
    41    Universidade Federal de Santa Catarina, Brazil 22.26%
    42    Cisco                                          22.14%
    43    Monash University                              21.82%
    44    Hurricane Electric                             21.80%
    45    RENATER                                        21.55%
    46    TUBITAK ULAKBIM / ULAKNET                      21.46%
    47    Aristotle University of Thessaloniki           20.89%
    48    DataChambers                                   20.28%
    49    UNESP                                          19.85%
    50    Chubu Telecommunications                       19.06%
    51    Swisscom                                       18.83%
    52    Indiana University                             18.08%
    53    Free                                           18.04%
    54    FranTech Solutions                             17.69%
    55    Tulane University                              17.55%
    56    University of New Hampshire                    17.45%
    57    Leibniz Supercomputing Centre                  16.60%
    58    HEAnet                                         16.59%
    59    US Dept of Transportation                      16.55%
    60    GARR                                           16.27%
    61    XS4ALL                                         16.14%
    62    Defense Research and Engineering Network       15.24%
    63    DMZGlobal                                      14.26%
    64    PCextreme B.V.                                 13.80%
    65    RCS & RDS                                      13.25%
    66    PowerTech Information Systems AS               12.24%
    67    SURFnet                                        12.07%
    68    BIT BV                                         11.93%
    69    ATT                                            11.52%
    70    Academia Sinica Network                        11.34%
    71    Honesty Net Solutions (I) Pvt Ltd               9.85%
    72    UNINETT                                         9.48%
    73    Storm Internet                                  9.25%
    74    CESNET                                          9.18%
    75    University Of Lampung                           9.12%
    76    AAISP                                           8.88%
    77    prgmr.com                                       8.61%
    78    KDDI                                            8.49%
    79    Voxel / Internap                                8.29%
    80    Init7                                           8.19%
    81    AMRES - Serbian National R&E Network            8.06%
    82    Comcast                                         7.95%
    83    CJSC Progressive Technologies                   7.92%
    84    MediaCat Div./Community Network Center Inc.     7.17%
    85    green.ch AG                                     7.02%
    86    StarHub                                         6.68%
    87    OVH                                             6.30%
    88    UniNet                                          6.26%
    89    CORPORACION NACIONAL DE TELECOMUNICACIONES      6.02%
    90    National Technical University of Athens         5.91%
    91    Forthnet                                        5.81%
    92    Deutsche Telekom AG                             5.18%
    93    EPT Luxembourg                                  5.11%
    94    Energy Group Networks                           4.62%
    95    M1 Limited                                      4.56%
    96    Internode                                       4.33%
    97    BelWue                                          4.32%
    98    Quonix Networks                                 4.26%
    99    SMELLY BLACK DOG                                4.22%
   100    LENTEL                                          4.15%

DNSSEC Validation in the Internet2 community

As a follow-up to my examination of the ISC DLV registry, I conducted an informal poll of some of my peers in the Internet2 community to find out 1) who is using DNSSEC validation on their resolvers, and 2) who additionally uses the ISC DLV.

A while back I setup a small project to monitor the status of DNS signed zones in Internet2 and few other selected communities. There is no easy way to programmatically determine who is using DNSSEC validation though, so the easiest way is to ask others [1]. I got responses from a number of universities and regional R&E networks. Here's a summary:

Institution                             Uses ISC DLV?
University of Pennsylvania              Yes      
Virginia Tech                           Yes
Univ of California, Los Angeles         No
Univ of Massachusetts, Amherst          No
Kansas Research & Education Network     Yes
Kansas State University                 <unknown>
Fort Hays State University              <unknown>
Louisiana State University              No
Univ of California, Berkeley            Yes
Energy Sciences Network (ESNet)         Yes
Lawrence Berkeley National Lab (LBNL)   <unknown>
North Dakota State University           Yes
Univ of Delaware                        Yes
3ROX (3 Rivers Optical Exchange)        Yes
Pittsburgh Supercomputing Center (PSC)  <some resolvers>
University of Idaho                     No

I'm sure I'm missing others - I'll add to this list as I discover them. If you know of anyone, feel free to let me know!

Footnotes:
[1] Although Geoff Huston and others have conducted some large scale studies of validation use, using a method of buying and analyzing ad impressions at popular websites, directing clients to carefully constructed URLs located in zones with differing DNSSEC signature statuses.

-- Shumon Huque

ISC DLV registry usage

On a LinkedIn forum, Dan York of the Internet Society recently asked a question about who still uses the ISC DNSSEC Lookaside Validation (DLV) registry. While commenting on the discussion, I decided to take a look at the contents of the registry, and I'm sharing some of my findings in this article.

DLV is a method to locate DNSSEC public keys off-path. See RFC 5074 and RFC 4431 for details. It is meant to be an early deployment aid until full deployment of DNSSEC happens. It's useful in situations where the DNSSEC keys for a target zone cannot be obtained by the normal top down traversal of the DNS delegation hierarchy, typically because one or more zones between the target zone and the root aren't signed. Another situation is where a parent zone may be signed but it was not possible for the child zone to have a Delegation Signer (DS) record installed in the parent for some reason - a common one is that the DNS registrar in use did not support the ability to do it.

Internet Systems Consortium (ISC) runs a DLV registry at dlv.isc.org. The basic idea is that if you can't find a DS record for a zone, say "example.org", you append the name of the DLV registry and look for DLV record at "example.org.dlv.isc.org" - the contents of the record are the same as would have been found at the DS record. Validating resolvers are pre-configured with the public key of the dlv.isc.org zone and use it to authenticate the signature associated with the DLV record.

It appears that some large DNS resolver services like Google DNS and Comcast do not use any DLV registries for validation, so only zones that have an intact chain of trust can have their data validated. I'm not sure if ISC publishes any usage statistics for their DLV registry, but from casual discussion with colleagues in the US R&E community over the years, I know quite a number of universities that do have their campus resolvers configured to use it. We use it at the University of Pennsylvania too.While upenn.edu is signed and has a secure delegation in its parent, there are some auxiliary zones that we run, like magpi.net that don't have a secure delegation, and we make use of the ISC DLV registry to publish keys there. In MAGPI's case, the reason is that the registrar we use, Network Solutions, still doesn't support DS records. I suppose it's time to switch registrars, and it's on my todo list!

In modern versions of  resolvers like ISC BIND and Unbound, a mere one line addition to the configuration file will turn this feature on. In fact, some OS distributions, like Fedora Linux already have it turned on in their default configuration.

The ISC DLV zone by design uses NSEC, so it's trivial to write a short program to fully enumerate its contents and look at the data. Here's what I see from a snapshot of the zone taken on August 29th 2013:

  Number of distinct zones:    2760
  Total number of DLV records: 6020

The number of DLV records is higher because most zones have multiple DLV records - their key digests are published with mutiple hashing algorithms (SHA1 and SHA256), and in some cases mutiple keys are published (perhaps key rollovers are in progress). Here's a breakdown of the number of DLV records per zone, and the number of zones with that many records:

  #DLV recs   #Zones
  8                1
  6                3
  4              241
  2             2515

The zone with 8 DLV records (!) incidentally is hysh.jp (4 keys, 2 digests/key).

Looking at the distribution of zones across Top Level Domains (TLD), we see:

  Number of TLDs represented: 111

There are 318 total TLDs at the current time, 116 of which appear to be signed, so that leaves 202 that aren't. I maintain some more detailed statistics of the TLDs at http://www.huque.com/app/dnsstat/category/tld/

Here's the full list of the 111 TLDs represented, sorted by descending order of the number of zones within them that are in the ISC DLV registry.

  arpa 487
  com 456
  org 270
  net 263
  de 185
  info 75
  eu 67
  uk 66
  ch 50
  hu 49
  ro 34
  us 34
  cz 32
  za 31
  pl 31
  fr 29
  ru 28
  ca 28
  it 26
  biz 25
  be 25
  au 25
  nl 24
  jp 22
  id 22
  name 20
  me 20
  mx 19
  tv 18
  at 17
  edu 16
  tw 13
  tk 12
  es 12
  mobi 11
  br 10
  cx 10
  co 8
  is 8
  nu 8
  fi 8
  sk 8
  dk 7
  se 7
  gov 6
  im 6
  ua 6
  am 6
  asia 5
  ws 5
  cc 5
  in 5
  nz 5
  xn--p1ai 5
  pt 4
  gs 3
  do 3
  bz 3
  cn 3
  hr 3
  ms 3
  ve 3
  mil 3
  nf 3
  gm 2
  lc 2
  la 2
  li 2
  th 2
  ph 2
  hn 2
  mu 2
  pro 2
  ar 2
  io 2
  ni 2
  gr 1
  gp 1
  lv 1
  to 1
  tl 1
  lu 1
  tj 1
  tg 1
  ec 1
  rs 1
  re 1
  jobs 1
  cm 1
  int 1
  tm 1
  pe 1
  pn 1
  aero 1
  hk 1
  md 1
  mg 1
  uy 1
  mw 1
  ug 1
  vc 1
  ae 1
  ai 1
  al 1
  vn 1
  as 1
  xxx 1
  kg 1
  sr 1
  st 1
  kr 1

Interestingly of the 2760 zones, 653 of them (almost a quarter!) also have DS records in their parent zones, so technically they don't need to be in the DLV registry at all. This includes three TLDs: th, ua, and kg. I wonder what the motivation for additionally maintaining keys in a DLV registry is. One theoretical reason might be to have an off-path database of keys that could be audited in case of an attack in the normal delegation chain.

Below are the sixteen zones inside .EDU:

  bucknell.edu                 DS exists
  internet2.edu                DS exists
  k-state.edu                  DS exists
  cs.kent.edu                  kent.edu not signed
  ksu.edu                      DS exists
  ai.mit.edu                   mit.edu not signed
  csail.mit.edu                mit.edu not signed
  dlp.mit.edu                  mit.edu not signed
  lcs.mit.edu                  mit.edu not signed
  npitest.psu.edu              psu.edu not signed
  ualr.edu                     DS exists
  ucaid.edu                    DS exists
  cse.ucdavis.edu              DS exists
  math.ucdavis.edu             DS exists
  ucr.edu                      DS exists
  maf.wisc.edu                 wisc.edu not signed

The EDU TLD is signed and has single registrar (Educause) that has supported DNSSEC for a long time. All the second level domains in the list above also have DS records in EDU, so they don't really need to also have DLV records. Most of the third level domains (one at Kent State U, four at MIT, one at Penn State, and one at U of Wisconsin) have parents that are not yet signed, so that makes sense. However, the two third level domains, cse.ucdavis.edu and math.ucdavis.edu have DS records in ucdavis.edu, so don't need DLV records either.
 
Shumon Huque

Network Engineer Job at Penn

We have an opening for a Senior Network Engineer in the University of Pennsylvania's Engineering group.

        https://jobs.hr.upenn.edu/applicants/Central?quickFind=197742

This position will be part of a small team that works very closely with our Network Operations group, offering final tier escalated support, researching new network designs, architectures & technologies, evaluating new equipment, designing and deploying related software and hardware systems.

The Engineering group is also involved in designing and operating a range of other services, including DNS, DHCP, Authentication & Authorization systems, Voice over IP, etc. So there are opportunities to get involved in many areas.

The candidate for this position will generally need to have a strong networking and programming background, as well as strong familiarity with UNIX and UNIX-like operating systems.

LOPSA East Class reviews - IPv6 & DNSSEC

I just received the reviews and attendee feedback for the IPv6 and DNSSEC classes I taught at the recent LOPSA-East conference. So far my recent stint as a technical course instructor at various conferences has been going well. Students are generally very pleased with the courses, and the positive feedback often results in invitations to teach at other venues.

The DNSSEC class is new. At past conferences, I've taught a combined DNS and DNSSEC class. But I've received feedback that many folks would like to see a course focussed on DNSSEC, so I created one. I also incorporated some live demos of setting up DNSSEC, which attendees found to be very useful.

I'll most likely be teaching these classes again at the USENIX LISA conference in Washington, DC later this year.

The possible responses for each question in the feedback survey are "Unsatisfactory", "Missed Some Expectations", "Met Expectations", "Exceeded Expectations", and "Greatly Exceeded Expectations". The data below is only for the (small) subset of the class that offered feedback of course.

IPv6 Course Feedback

==> SA1: Using and Migrating to IPv6 / Huque

Rate this training session: [Description matched the contents of the class]
    * Greatly Exceeded Expectations
    * Exceeded Expectations
    * Greatly Exceeded Expectations
    * Met Expectations
    * Met Expectations
    * Greatly Exceeded Expectations
    * Exceeded Expectations
    * Greatly Exceeded Expectations

Rate this training session: [Class material was useful to my job]
    * Greatly Exceeded Expectations
    * Met Expectations
    * Greatly Exceeded Expectations
    * Met Expectations
    * Met Expectations
    * Exceeded Expectations
    * Exceeded Expectations
    * Greatly Exceeded Expectations

Rate this training session: [Instructor was knowledgeable]
    * Greatly Exceeded Expectations
    * Greatly Exceeded Expectations
    * Greatly Exceeded Expectations
    * Exceeded Expectations
    * Greatly Exceeded Expectations
    * Greatly Exceeded Expectations
    * Greatly Exceeded Expectations
    * Greatly Exceeded Expectations

Rate this training session: [Instructor was able to answer students questions]
    * Greatly Exceeded Expectations
    * Greatly Exceeded Expectations
    * Greatly Exceeded Expectations
    * Greatly Exceeded Expectations
    * Greatly Exceeded Expectations
    * Greatly Exceeded Expectations
    * Greatly Exceeded Expectations
    * Greatly Exceeded Expectations

Rate this training session: [Course material quality]
    * Greatly Exceeded Expectations
    * Exceeded Expectations
    * Greatly Exceeded Expectations
    * Exceeded Expectations
    * Greatly Exceeded Expectations
    * Greatly Exceeded Expectations
    * Exceeded Expectations
    * Greatly Exceeded Expectations

What was the single BEST part of this class?
    * good material, awesome presenter
    * Excellent balance of technical detail with beginner introduction
    * instructor clear experience and ability to communicate topic
    * He has done the work, and could provide many answer from experience
    * Quality of the instructor - good speaker and very knowledgable.
    *
    * The instructor was able to explain a very complex topic clearly & in terms that are directly applicable to my future use of the material.
    * Shumon's depth of knowledge; he adapted what we covered and how fast we were going, on the fly! AWESOME instructor.

Name one aspect of this class that NEEDS IMPROVMENT?
    * needs to be longer
    *
    * wants more time to work in.
    * pacing, which material to emphasis. I thought the first part of the material could have been covered a bit faster, and more time on the meatier issues
    * The class seemed to detail a lot of 'differences between ipv6 and ipv4' and protocol internals in favor of 'how do I actually deal with migration issues'. A better mix would be nice, but I understand that unless you know of the differences, it can be hard to concentrate on implementation.
    *
    * Access to a Lab for a demo might add to the session.
    * nothing, run this class again. MAYBE, talk him into making another class focused on migrating your organization from v4 to v6... so people can do 'intro to ipv6' if they need, then another session on migration strategies.

Should LOPSA offer this class in the furture?
    * Yes
    * Yes
    * Yes
    * Yes
    * Yes
    * Yes
    * Yes
    * Yes

DNSSEC Course Feedback

==> SA4: DNSSEC (DNS Security Extensions) / Huque

Rate this training session: [Description matched the contents of the class]
    * Met Expectations
    * Exceeded Expectations
    * Greatly Exceeded Expectations
    * Exceeded Expectations
    * Greatly Exceeded Expectations

Rate this training session: [Class material was useful to my job]
    * Met Expectations
    * Greatly Exceeded Expectations
    * Greatly Exceeded Expectations
    * Exceeded Expectations
    * Greatly Exceeded Expectations

Rate this training session: [Instructor was knowledgeable]
    * Met Expectations
    * Greatly Exceeded Expectations
    * Greatly Exceeded Expectations
    * Greatly Exceeded Expectations
    * Greatly Exceeded Expectations

Rate this training session: [Instructor was able to answer students questions]
    * Met Expectations
    * Greatly Exceeded Expectations
    * Greatly Exceeded Expectations
    * Greatly Exceeded Expectations
    * Greatly Exceeded Expectations

Rate this training session: [Course material quality]
    * Met Expectations
    * Missed Some Expectations
    * Greatly Exceeded Expectations
    * Exceeded Expectations
    * Greatly Exceeded Expectations

What was the single BEST part of this class?
    * DNSSEC appears do-able
    * My impression of DNSSEC went from theoretically possible to practical in a very short time. Something that is alluded me for quite some time.
    *
    * Best part was seeing the live application of theory in an enterprise environment.
    * Shumon's depth of knowledge; he adapted what we covered and how fast we were going, on the fly! AWESOME instructor. He's so good, we had wonky wifi, and he just ran demos through Bind on his Mac. nice.

Name one aspect of this class that NEEDS IMPROVMENT?
    *
    * Materials were not but will be posted to the presenters site
    *
    * nothing negative to say.
    * nothing, don't change anything, run it again!

Should LOPSA offer this class in the furture?
    * Yes
    * Yes
    * Yes
    * Yes
    * Yes

100 Gigabit Ethernet at Penn

This summer, the University of Pennsylvania is upgrading its campus core routing equipment (in fact we're in the midst of this upgrade right now). This is basically an upgrade to the set of large routers that form the center of our network.

The current core topology consists of 5 core routers (and also 2 border routers) interconnected by two independent layer-2 switched 10 Gigabit Ethernet networks. Each of the core routers is located in one of five geographically distributed machine rooms across the campus. A rough diagram is shown below.

This diagram also shows the current external connections to/from the campus network - we have three links (each 10 Gigabit Ethernet) to Internet Service Providers (ISPs). And two connections to MAGPI (the regional Internet2 GigaPoP operated by us), via which we access a 10 Gigabit Ethernet connection to Internet2. The Internet2 connection is shared amongst Penn and other MAGPI customers, which are mostly Research & Education institutions in the geographic area.

The core interconnect is being upgraded to 100 Gigabit Ethernet (a ten-fold increase in link bandwidth). It would be cost prohibitive to fully replicate the current design in 100 Gig (since this equipment is still very expensive) so the interconnect design has been adjusted a bit. Instead of two layer-2 switch fabrics interconnecting the routers, we are deploying the core routers connected in a 100 Gig ring (see the diagram below). When the final design is fully implemented, each core router will have a 10 Gig connection into each of the border routers (this will require some additional upgrades to the border routers, which are expected to happen later this year). The topology redesign has fewer links, and in the final count (summing the bandwidth of all the core facing links), the new core will have about 5 times the aggregate bandwidth of the old one. The maximum (shortest path) edge to edge diameter of the network increases by one routing hop.

100 Gigabit Ethernet is today's state of the art in transmission speed. The next jump up will likely be 400 Gigabit Ethernet for which the IEEE already has a study group launched and several preliminary designs under consideration.

Not depicted in this diagram is the rest of the network towards the end systems. Below the layer of core routers, are smaller routers located at the 200 or so buildings scattered around campus. Each building router is connected to two of the core routers. The building routers feed wiring closets inside the building, which house layer-2 switches that network wallplates are connected to.

In the process of the upgrade, we are also changing router vendors. The current core routers, Cisco 7609 series routers with Sup7203BXL supervisor engines, have served us well. They were originally deployed in the summer of 2005, and have been in operation well past their expected lifetime.

As is our practice, we issued an RFI/P (Request for Information/Purchase) detailing our technical requirements for the next generation routers and solicited responses from the usual suspects, selecting a few vendors whose equipment we bring in for lab testing, followed by a selection.

The product we've selected is the Brocade MLXe series router, specifically the MLXe-16 - this router can support 16 half height, or 8 full height (or a mixture of full/half) line cards, as well as redundant management and switch fabric modules.

A product description of the MLXe series is available at:
http://www.brocade.com/downloads/documents/data_sheets/product_data_sheets/MLX_Series_DS.pdf

The photo below is one of the routers (prior to deployment) in the Vagelos node room (one of 5 machine rooms distributed around campus where we house critical networking equipment and servers).Going from left to right, this chassis has two management modules, one 2-port 100 Gigabit Ethernet card, six 8-port 10 Gigabit Ethernet cards, four switch fabric modules, two more 8-port 10 Gigabit Ethernet cards, and three 24-port Gigabit Ethernet cards.

One of these routers was deployed in production last week. The rest should be up and running by the end of this month or by early July.

(The full set of photos can be seen here on Google Plus)

Shown below is the 2-port 100 Gigabit Ethernet card, partially inserted into the chassis, showing the CFP optical transceiver modules attached.

Unlike preceding generations of ethernet, with 100 Gigabit Ethernet, the transmission technology uses multiple wavelengths in parallel (although there are parallel fiber implementations also). The current IEEE specifications (802.3ba) specify four lanes of 25Gbps. However a number of key vendors in the industry, including Brocade, formed the MSA (Multi Source Agreement) and designed and built a 10x10 (10 lanes of 10Gbps) mechanism of doing 100 Gig, at much lower cost than 4x25Gbps, operating over single mode fiber at distances of 2, 4, or 10km. This is called LR-10 and uses the CFP (C Form factor pluggable) media type.

Pictured below (left) is a Brocade LR10 100 Gigabit Ethernet CFP optical module installed in 100 Gig line card with a single mode fiber connection (LC). On the right is an LR10 CFP module taken out of the router.

Close up of the 8-port 10 Gigabit Ethernet module, and several 24-port Gigabit Ethernet modules. To connect cables to them, we need to install small form factor pluggable transceivers into them, SFP+ for the 10 gig, and SFP for the 1 gig.

Pictured below is one of the five Cisco 7609 routers that will be replaced.

One of the Penn campus border routers, a Juniper M120, is shown below. This is also scheduled to be upgraded in the near future to accommodate 100 Gig and higher density 10 Gig, although the product has not yet been selected/finalized.

Below: A Ciena dense wavelength division multiplexer (DWDM). Penn uses leased metropolitan fiber to reach equipment and carriers in 401 North Broad street, the major carrier hotel in downtown Philadelphia. We have DWDM equipment placed both at the campus and the carrier hotel to carry a mixture of 1 and 10 Gig circuits and connections across this fiber for various purposes. This equipment is also scheduled to be upgraded to allow us to provision 100 Gigabit Ethernet wavelengths between the campus and the carrier hotel.

High Performance Networking for Researchers

Penn is a participant in the National Science Foundation (NSF) funded DYNES (Dynamic Network Systems) project, which provides high bandwidth dedicated point to point circuits between (typically) research labs for specialized applications. Popular uses of this infrastructure today include high energy physics researchers obtaining data from the LHC and other particle accelerator labs, and various NSF GENI network research projects.

Earlier this year, we completed a grant application for the  NSF "Campus Cyberinfrastructure - Network Infrastructure and Engineering (CC-NIE)" program. I spent a large amount of time in March of this year with several Penn colleagues in preparing the application. If Penn does win an award (we'll find out later this year), we will be deploying additional dedicated network infrastructure for campus researchers, bypassing the campus core and with 100 Gbps connectivity out to the Internet2 R&E network. A rough diagram of how this will look is below.

Software Defined Networking

There's a huge amount of buzz about Software Defined Networking (SDN) in the networking industry today, and a number of universities are investigating SDN enabled equipment for deployment in their networks. Of the big router vendors, Brocade does appear to have one of the better SDN/openflow stories thus far. The MLXe series already supports an early version of Openflow (the portion of SDN that allows forwarding tables of switches/routers to be programmed by an external SDN controller).

Penn is building an SDN testbed in our network engineering lab, primarily to investigate its capabilities. For us, SDN is still largely a solution in search of a problem. We run a very simple network by design, whose primary purpose is connectivity and high performance packet delivery. The most probable use case in our future, virtualization of the network, is likely better achieved with a proven technology like MPLS first. But we'll keep an eye on SDN and its evolution. We do want to support research uses of SDN though. Several faculty members in the Computer Science department are interested in SDN, and the NSF CC-NIE grant will allow us to build some SDN enabled network infrastructure separate from the core production network to accommodate their work.

-- Shumon Huque