Improve Availability of Networks
Improve Availability of Networks
Internet Exchange Points and Their Role in Cyberspace
Abstract and Keywords
The Internet could not exist without Internet exchange points (IXPs). IXPs provide the mechanisms (physical connections in a data center or carrier hotel) that enable Internet service providers (ISPs) to exchange traffic easily and cost effectively. The concept of IXPs was developed in the early 1990s, and IXPs have continued to grow in quantity, location, and size (traffic volume) as the Internet has grown. This chapter provides an overview of IXPs along with their roles in the Internet. The first part identifies the architecture of the Internet. The second deals with the concept of peering (a prerequisite for IXPs). The third and final part discusses IXPs in greater detail. The chapter concludes with a list of organizations which support and contribute to the Internet.
The Internet could not exist without Internet exchange points (IXPs). The concept of IXPs was developed in the early 1990s, and IXPs have continued to grow in quantity, location, and size (traffi c volume) as the Internet has grown. There are, however, very few books or papers written about IXPs. Knowledge of IXPs has long been confined to industry experts. In this chapter, I provide an overview of IXPs along with their roles in the Internet. The first part of this chapter identifies the architecture of the Internet. The second identifies the concept of peering (a prerequisite for IXPs). The third and final part of the chapter identifies IXPs in greater detail. In the conclusion to the chapter I provide a list of organizations which support and contribute to the Internet.
As the name implies, IXPs exchange something, but what? The answer is that Internet service providers (ISPs) exchange their traffic with each other over IXPs. IXPs provide the mechanisms (physical connections in a data center or carrier hotel) that enable ISPs to exchange traffic easily and cost effectively. In other words, it makes business sense for ISPs to exchange traffic over IXPs. To fully understand this point, it is important to first have an understanding of how the Internet works.
As opposed to a traditional telephone network that is owned and administered by a telephone carrier and is therefore considered to be a single autonomous network, the Internet is not a single autonomous network. No one company or organization owns it, and there is no central organization to administer or monitor it. Simply put, the Internet is a collection of ISPs (i.e., a collection of autonomous networks creating a network of multiple autonomous networks). The 36,0471 ISPs  (or autonomous networks) are connected by many different methods, and together they constitute the Internet. As you can imagine, it is difficult to maintain such a network of multiple autonomous networks because each autonomous system (ISP) has different interests, policies, engineering skills, operation skills, and so on. Therefore, it takes an enormous amount of effort by the Internet community to make the Internet work.
Connectivity Options for ISPs
In the previous section, I noted that all ISPs are connected in some way in the Internet. In order for ISPs to connect and exchange their traffic with each other, today’s Internet offers two types of connectivity options for ISPs: transit and peering. The fundamental difference between transit and peering is the scope of reachability. In short, transit provides global reachability (i.e., access to any IP address worldwide) whereas peering provides limited reachability (i.e., access to only a limited subsection of IP addresses). You may wonder why we need both, why having transit only will not (p.92) suffice. I explain that later in this chapter, but before explaining transit and peering in detail and why both are necessary, I first define some basic technical terms.
BGP: T2HE Glue Connecting ISPS
Being connected means an ISP can send and receive (or exchange) its traffic to and from other ISPs. It does not merely mean simple physical connectivity. In the Internet, there is a common language every ISP speaks. This language, more precisely known as communication protocol, is called Border Gateway Protocol (BGP) . BGP provides a mechanism that all ISPs worldwide use to connect to each other to form the Internet. Explaining BGP in detail is beyond the scope of this chapter but I will explain BGP very briefly here.
ASN and routes: Because the Internet is a network of multiple autonomous networks, we need an identifier (or name) for each autonomous network (i.e., a name for each ISP’s network). The identifier used is the autonomous system number (ASN). A route is a collection of addresses. An address is an IP address. For instance, a route can be written as 184.108.40.206/24, which represents 254 usable addresses (from 220.127.116.11 to 18.104.22.168).
Announce or no connection: BGP glues ISPs with ASNs and routes. For example, suppose an ISP (ASN 100) has 1,000 routes. The ISP must announce the routes via BGP to other ISPs, effectively saying something like “We, ASN 100, have 1,000 routes such as 22.214.171.124/24.” This announcement will be propagated throughout the Internet via BGP.
Once another ISP (ASN 200) has received the announcement and updated its routing table so that it effectively reads “We, ASN 200, know ASN 100 has 1,000 routes such as 126.96.36.199/24,” these two ISPs are connected. At this point, ISP (ASN 200) can send traffic to ISP (ASN 100).
Now that you understand how connections between ISPs are made, the following sections explore the concept of transit and peering.
When two ISPs are connected via transit, they are not equal. The relationship is a commercial one (buyer and seller). The key characteristic of transit service is that all routes (usually called full routes) announced by all ISPs (36,047 ISPs) worldwide are provided by the seller to the buyer. Once an ISP has received full routes, it can send and receive traffic to and from anywhere.
When two ISPs are connected via peering, they are basically equal. There is usually no monetary settlement between two peers. In other words, peering is free. A key difference between transit and peering is that transit provides full routes, whereas peering provides partial routes.
Suppose ISP-A (with 1,000 routes in its network) and ISP-B (with 1,500 routes in its network) are peering with each other. ISP-A provides ISP-B with all its routes (1,000 routes) via BGP. In return, ISP-B provides ISP-A with all its routes (1,500 routes) via BGP. Since ISP-A has learned all the routes of ISP-B, ISP-A will be able to send traffic to ISP-B and vice versa. In other words, peering is a relationship whereby two ISPs provide reciprocal access to each other’s network.
The Internet Hierarchy
The Internet is generally composed of three layers: Tier 1 providers, Tier 2 providers, and End Users (enterprises, gaming companies, social networks, content providers, and so on).
Tier 1 providers: Tier 1 providers are at the top of the Internet hierarchy and they can maintain full routes or have global access just by peering with other Tier 1 providers. Generally Tier 1 providers have full mesh interconnections with each other as shown in the schematic diagram in Figure 1.
Note that the number of Tier 1 providers is extremely limited: approximately ten worldwide. As mentioned earlier, as of today 36,047 ISPs are connected, and together they constitute the Internet. This means only 0.03 percent (equal to 10/36,047) of ISPs are Tier1 providers, and the rest are Tier 2 providers. (p.94)
Tier 2 providers: Tier 2 providers are ISPs who peer with other Tier 2 providers, but which still need to purchase transit service from Tier 1 providers (see Figure 2); 99.97 percent of ISPs fall under this category.
Who Peers? Why Peer?
This section focuses on Tier 2 providers; they need peering on a much greater scale than Tier 1 providers. Tier 2 providers can obtain global reachability by simply purchasing full routes from Tier 1 providers and sending all their traffic to the Tier 1 providers. They need to peer for primarily (p.95) three reasons: reduce transit costs, improve performance, and control routing.
Reduce transit costs: Tier 2 providers purchase transit service from Tier 1 providers by megabits per second. This means the more traffic they send to Tier 1 providers, the more they have to pay. On the other hand, peering is free. Considering the ever-increasing price pressure from competition in the market, what would you do if you were a Tier 2 provider to maintain a healthy profit level?
Probably, you would like to peer with as many Tier 2 providers as possible because peering is free. As Figure 3 shows, doing so would reduce the amount of traffic to Tier 1 providers and thus reduce transit costs. For instance, if you have 100 Gbps traffic in your network, your goal may be to send at least 20 percent (or 20 Gbps) of the traffic to peering and send 80 percent (or 80 Gbps) or less of traffic to Tier 1 providers. This would result in a significant cost reduction.
Improve performance: Tier 2 providers purchase full routes from Tier 1 providers. Full routes contain global AS path information. The AS path information is the information the route traffic must take based on the ASN. As Figure 4 shows, however, the AS path may not be optimal. If you are a Tier 2 provider and you want to send traffic to another Tier 2 provider, the destination Tier 2 provider may be multiple-AS paths away, resulting in higher latency. If your customer is a latency-sensitive customer such as an online gaming company, you may lose the customer if the latency of your network is high.
If you peer with a destination Tier 2 provider, the provider will become adjacent to your network (i.e., you will have a direct connection) and latency will be greatly improved.
Control routing: If you are a Tier 2 provider peering with another Tier 2 provider, you have two paths to the destination Tier 2 provider via transit and peering. If one of the paths is not performing well, you can choose the alternate path. In other words, if they do not peer, Tier 2 providers will increase costs, degrade performance, and lose control of routing.
A Brief History of Peering
Beginning with this section, we will look into peering from various perspectives, starting with a business perspective. At the dawn of the Internet, all ISPs were competitors, but also friends. Most ISPs had an open peering policy where they agreed to peer with any other ISP with no prerequisites. At that time, one of the key sayings was “keep local traffic local.” Before peering became common, local traffic from ISP-A to ISP-B was first sent to (p.97) a Tier 1 provider from ISP-A and then the traffic came back to ISP-B via the Tier 1 provider. If ISP-A had been peering with ISP-B, the traffic would have been sent directly from ISP-A to ISP-B. Once ISPs realized the benefits of peering, they were virtually unanimous in supporting peering in general.
As the Internet grew, however, ISPs became more competitive. As a result, peering became a purely business decision. Some ISPs developed peering policies (or peering prerequisites) and started to peer selectively. Some ISPs de-peered, terminating existing peering relationships if they felt a particular peering (or peerings) did not make business sense. As of today, peering continues to be driven by business decisions, and ISPs carefully review and evaluate every single peering relationship periodically.
Peering as a Business Decision
Every peering must make business sense. There are generally three types of peering from a business perspective: symmetrical peering, asymmetrical peering, and no customer peering.
Symmetrical peering: As Figure 5 shows, suppose ISP-A is in talks with ISP-B about creating a peering relationship. After estimating the traffic pattern after peering, ISP-A realizes that the traffic amount from ISP-B to ISP-A would be 1 Gbps, whereas the traffic from ISP-A to ISP-B would be 10 Mbps. This means ISP-B can reduce its transit costs significantly via the peering, but ISP-A cannot. In this case, the peering doesn’t make business sense to ISP-A. If the traffic from ISP-A to ISP-B was 800 Mbps, it would make better business sense for ISP-A. In other words, peering makes sense if peering traffic is symmetrical between the two ISPs.
Asymmetrical peering: As Figure 6 shows, suppose ISP-A is in talks with ISP-B to have a peering relationship and ISP-A is a large content provider (e.g., a content-heavy provider such as an on-demand video provider) and ISP-B is a large regional operator (e.g., an eyeball-heavy provider, such as a local broadband provider). (p.98)
Content-heavy providers would like to push traffic to eyeball-heavy providers (i.e., residential customers) while reducing transit costs. Eyeball-heavy providers would like to pull traffic from content-heavy providers for their residential customers while reducing transit costs.
Obviously the amount of traffic flowing from content-heavy providers to eyeball-heavy providers is always exponentially larger. Does it make sense to peer even though the traffic is not symmetrical? Of course it does. Both content-heavy providers and eyeball-heavy providers could reduce by a significant amount their transit costs by peering with each other.
At the time of this writing, news emerged of an eyeball-heavy provider (specifically, a cable operator) starting to charge a content- (p.99) heavy provider for “peering” in order to protect its own cable television revenue. Peering is a dynamic process.
No customer peering: Suppose ISP-A is in talks with ISP-B to have a peering relationship. If ISP-A is a large ISP who serves North America nationwide and ISP-B is a small local ISP who serves a small city, ISP-B can be a customer of ISP-A. If ISP-A peers with ISP-B, ISP-A loses a revenue opportunity. In other words, ISPs won’t peer with an ISP who is already their customer or potentially their customer.
Bilateral Versus Multilateral Peering
This section looks at peering from the perspective of the peering negotiations involved, specifically looking at two kinds of peerings: bilateral and multilateral.
Bi-lateral peering: This type of peering requires peering negotiation between two ISPs. If ISP-A wishes to peer with ISP-B, ISP-A needs to talk with ISP-B. Both ISPs must agree to establish the peering relationship. If ISP-A wishes to peer with fifty ISPs, ISP-A needs to negotiate with fifty ISPs individually. From a technical perspective, bilateral peering means BGP settings on a border router need to be configured for every single peering relationship. Most ISPs prefer bilateral peering because they can peer selectively based on business decisions.
Multi-lateral peering: Multilateral peering requires no peering negotiation. If ISP-A agrees with multilateral peering, it means ISP-A agrees to peer with all ISPs. From a technical perspective, only one BGP setting is required to peer with all ISPs. Small ISPs or content providers who wish to peer with any ISP prefer multilateral peering.
Private Versus Public Peering
This section looks at peering from the perspective of the physical connections involved.
(p.100) Private peering: Private peering is used when two ISPs (and only two ISPs) peer with each other over a cable. They do not share the cable with other parties. In this case, it is each ISP’s responsibility to find other ISPs to peer with.
Public peering: Public peering is used when an ISP peers with other ISPs on Internet Exchange Points (IXPs). IXPs use a shared fabric such as a layer 2 switch as a platform. Usually, the organization administering the IXP maintains a website providing information such as a list of participants, interface speed options, traffic statistics, hardware platform used, and so on.
Most ISPs prefer public peering at IXPs because they can peer easily and cost effectively.
Ease of peering: When one ISP wants to peer with another ISP privately, the ISP initiating the peering needs to spend a lot of time and effort to find answers to basic questions such as:
Where is the other ISP located? Is it in the same building or a different building?
What is the distance from us to them?
How do we connect?
Are there any local regulations governing running cables?
How much would it cost to build an interconnection?
At an IXP, these questions have been answered already. There is a participant list, so you know who is participating at the IXP. Since other participants have been already connected to the IXP, you care only about how to connect your router to the IXP, which, one way or another, is always achievable. Cost to participate in the IXP is easy to determine. Consequently, it is relatively easy to peer with other ISPs at any IXP.
Cost effectiveness: In private peering, an interconnection via cable is required for each peering. Each ISP will incur a cost to do so. For example, given that current Ethernet ports are either 1 Gbps or 10 Gbps, if an ISP needs to peer with four ISPs via private peering at speed of 2 Gbps, the ISP has to have four 10 GigE ports or eight GigE ports.
On the other hand, at an IXP, the ISP needs only one 10 GigE port to peer with four ISPs at a speed of 2 Gbps on the IXP, because (p.101) 4 × 2 Gbps is less than 10 Gbps, so one 10 GigE connection to the IXP will suffice. At IXPs, peering is cost effective because an ISP can peer with multiple ISPs via a single router interface. In summary, Tier 2 providers need to peer on IXPs to maintain their business competitiveness.
A Brief History of IXPs
The first IXP was Commercial Internet Exchange (CIX, pronounced “kicks”), founded in 1991 by PSINet, AlterNet, and CERFNet. The hardware platform was a Cisco 7010 router located in Palo Alto, California, and managed by a not-for-profit association. CIX was an IXP pioneer and established the basic concept and business model.
Not long after, Metropolitan Area Ethernet (MAE, pronounced “may”) and NSF-sponsored network access points (NAPs) were established by MCI and Sprint. They were managed by organizations for profit. Since then the concept and business model of IXPs has been well understood by the Internet community, and many IXPs have been established. Thanks to the efforts of many researchers and contributors in the Internet community, it is believed that there are somewhere between 300 and 350 IXPs worldwide . However, the number of IXPs and the aggregated traffic on them is still growing.
Depending on countries or political boundaries, IXPs are operated by organizations for profit, non-profit organizations, or governments. Peering policy can be bi-lateral (can peer freely) or mandatory multi-lateral (must peer with all participants).
There are two types of IXPs: layer 2 and layer 3. A layer 2 IXP uses a shared network fabric like Ethernet switches as a platform for performing the exchange (see Figure 7). The IXP is not involved in participant routing. Participants determine who they peer with. They do not have to peer with everyone (bilateral peering).
A layer 3 IXP uses routers as the platform for performing the exchange (see Figure 8). The IXP is involved in participant routing. Participants have limited control over who to peer with, which could potentially create business issues. A layer 3 IXP’s policy is that participants must peer with everyone (multilateral peering). (p.102)
Today, most IXPs select layer 2 Ethernet switches as their platform for many reasons:
They are a proven platform for many years
They support high speeds: 1 G/10 G/100 G
The hardware is relatively cheap
They are easy to operate (no BGP operation)
They offer optional private peering via VLAN
A layer 2 switch platform is a bilateral peering environment. However, some IXPs provide a route server to offer multilateral peering capability in addition to the default bilateral peering environment.
Because there are over three hundred IXPs in the world and they carry a tremendous amount of important Internet traffic, it is important that IXPs are well protected against cyber attacks. Because of this, the IP address range of the public peering network must not be announced by any participants of IXPs to ensure the IP address range is not reachable from anywhere. Periodic vulnerability checks must be performed on IXP platforms to ensure all software is vulnerability free. IXP platforms must be Communications Assistance for Law Enforcement (CALEA) compliant to facilitate investigation processes by law enforcement.
While layer 2 switches are the platform of choice for most IXPs, the platform is not perfect and has several challenging issues, among which are the following:
Ever-increasing traffic: The aggregated traffic on IXPs has grown significantly in the past ten years and it is still growing. In the past ten years, most popular ports transitioned from 100 MB, 1 GB to 10 GB. At large IXPs, it has been truly a challenge to accommodate hundreds of 10 GB port participants. Large IXPs hope that Ethernet switch manufactures continue to increase switch capacity and port density to meet ever-increasing market demands.
Not fully secure: As all participants are in the same broadcast domain, peering traffic can be affected by other participants’ traffi c. Using a private connection (VLAN) per peer is not practical because the number of VLANs is limited to 4,096. As an example of the effect of this limitation, if 120 participants are peering in a meshed configuration, 7,140 VLANs are necessary (= 120 × 120 – 1)/2). It is also difficult to manage such a large number of VLANs.
No MTU optimization: MTU (maximum transmission unit) is the largest layer 2 protocol data unit supported. In the case of Ethernet, the MTU is 1,500 bytes. The 1,500 bytes of MTU were designed in the 1980s when Ethernet’s speed was 10 Mbps. Today, Ethernet’s (p.104) speed is 10 Gbps or faster. It would be more efficient to adopt a larger MTU size ( jumbo frames) when using these higher speeds. However, it is difficult to choose a larger MTU that is agreeable to all participants because every participant’s router or network has a different optimal MTU size. Therefore, most IXPs still use the default MTU size of 1,500 bytes.
No traffic statistics per peer: In order to collect statistics, we rely on SNMP/MIB (Simple Network Management Protocol/Management Information Base). Currently MIB is available per physical switch port. However, this is not convenient because only traffic statistics per physical port can be collected. Since multiple peering sessions will pass through one physical port, it would be beneficial if statistics can be collected per peering session.
While today there is a technology called sFlow to collect statistics per peering session, this is a sampling-based method and thus the statistics are not 100 percent accurate.
Future IXP Architecture
Layer 2 switch platforms are likely to be replaced with newer technologies. One of the possibilities is an MPLS-based IXP. I presented the concept at a NANOG meeting in 2002. The MPLS-based IXP has the potential to overcome several drawbacks of layer 2 switches. Notably, it offers secure tunnel-per-peer, optimal MTU-per-peer, SNMP MIB-per-tunnel, and traffic engineering via RSVP.
This chapter provides a basic degree of information about IXPs and their role in cyberspace, as summarized below:
The Internet is a network of multiple autonomous networks, and is composed of over 35,000 ISPs.
BGP connects ISPs.
ISPs are connected either by transit or by peering relationships.
Peering is a business decision.
Tier 2 providers need to peer on IXPs to reduce costs, increase performance, and control routes. Without IXPs, it is difficult for Tier 2 providers to maintain their business and competitiveness.
There are over 300 IXPs worldwide. The number of IXPs and their aggregated traffic is still growing.
Depending on countries or political boundaries, IXPs are operated by organizations for profit, non-profit organizations, or government.
Most IXPs are based on a layer 2 switch platform. However, there are some challenges as a result of this situation.
In the future, IXP may be based on newer hardware architecture, and MPLS-IXP could be one of the options.
Organizations Supporting the Internet
Due to the multi-autonomous nature of the network, the Internet is difficult to maintain. To help with this, there are numerous organizations supporting the Internet. In other words, the Internet exists because of efforts or contributions by many people. The following is a select list— in no particular order— of organizations that support the Internet in various ways, but it is by no means comprehensive.
Internet Corporation for Assigned Names and Numbers (ICANN): http://www.icann.org/
Internet Assigned Numbers Authority (IANA): http://www.iana.org/
Internet Society (ISOC): http://www.internetsociety.org/
Internet Engineering Task Force (IETF): http://www.ietf.org/
International Telecommunication Unit— Telecommunication Standardization Sector (ITU-T): http://www.itu.int/ITU-T/
American Registry for Internet Numbers (ARIN): https://www.arin.net/
Réseaux IP Européens Network Coordination Centre (RIPE NCC): http://www.ripe.net/
Asian-Pacific Network Information Center (APNIC): http://www.apnic.net/
(p.106) Latin American and Ca rib be an Network Information Center (LAC-NIC): http://lacnic.net/en/index.html
Association Française pour le Nommage Internet en Coopération (AFNIC): http://www.afnic.fr/en/
Internet Software Consortium (ISC): http://www.isc.org/
Asia Pacific Regional Internet Conference on Operational Technologies (APRICOT): http://www.apricot.net/
North American Network Operators’ Group (NANOG): http://www.nanog.org/
African Network Operators’ Group (AFNOG): http://www.afnog.org/
French Network Operators’ Group (FRNOG): http://www.frnog.org/
New Zealand Network Operators’ Group (NZNOG): http://www.nznog.org/
Japan Network Operators’ Group (JANOG): http://www.janog.gr.jp/en/
(1) As of November 2010.