Where are the borders in the Internet?

Let’s continue our journey through the basics of Content Delivery Networks.

Earlier we’ve discussed basics of the Internet, TCP and UDP, and IP. If you’ve read the previous articles, you should understand how the Internet works in general, how your computer sends packets over a network, and how the Internet Protocol guides packets in small networks.

In this article we will tell you about a protocol that works like duct tape for all these computers in the world. BGP. But let’s start with an abstraction that we have not mentioned before.

Autonomous Systems

Let’s imagine that you own a company that wants to have its own “piece of the Internet” to serve some websites, data for your applications, and so on. Where should you start?

Sure, if your company is small, then you don’t want to learn a lot of networking stuff, don’t want to buy networking hardware, and don’t want to hire personnel for maintaining this hardware. It’s easier to buy some space in a cloud, move your infrastructure there, and leave all this headache to professionals. But what if your company is not small? What if you are a CTO of a company that has hundreds of people working on web services, and this company has a huge infrastructure deploying here and there?

Well, in this case you probably want to have more control over the network you’re using. First, to make sure that everything is secured enough. Second, just to be more independent of ISPs and not so vendor-locked.

To accomplish all those goals you may create your own Autonomous System.

First, you should buy a free block of IP addresses. It’s not so easy nowadays due to IPv4 exhausting (we’re not talking about IPv6 in this article). There are two ways:

1. You may go to official registries that will sell you a block of addresses when it will be available. According to RIPE it may not be soon.
2. You may go to private brokers and buy a block of addresses from them.

Second, you should register your Autonomous System (AS) to get an ASN — Autonomous System Number. This is a kind of ID of your AS. Registration is also guided by regional registries like RIPE.

When you buy a block of addresses, and register your AS, you will own your “part of the Internet.” The only thing that you will have to do is to connect your AS to neighbors — others AS that will exchange their routing information with you.

You see, the Internet is not a “big network”. It is actually a big network of big networks. Every AS (there are more than 100k of them today) is managed by some company or a government and it does not matter how exactly they manage their networks. What does matter is a way they communicate with other networks.

Border Gateway Protocol

When you have your AS you want other Internet users to be able to find you over there. Well, you should tell them how they can do it!

As you remember, every computer connected to the Internet has a unique IP address. We might say that your IP address is kind of like your postal address. Every packet you send to the Internet contains your address and the destination address. But instead of saying “Send this letter to 1600 Pennsylvania Avenue, N.W., Washington, DC 20500” those packets say “Send this packet to 2.17.190.190”.

But when you send a letter there is someone in charge of delivering this letter, right? When this person (or, it would be fair to say, this robot) gets your letter they should decide where to send it. Usually there is no direct way to send a letter from one city to another. There are intermediate cities with postal services that get your letter, decide where to send it next, and so on. The process of letter forwarding may look like this:

Or like this:

The final way depends on a huge amount of “metrics.” Postal services in some cities might process more letters than others, the roads between cities also differ one from each other, there might be a problem with the delivery trucks, and so on. So, to decide where you send your letter, the “decision maker” in every intermediate city should see the whole map with postal buildings and their metrics on it.

All this letter forwarding works kind of the same way the Internet does. But instead of cities we have autonomous systems:

And every AS has subnets that it owns:

To make it work, every AS has an “edge router” which works on a “border” of this AS. Such a router processes the traffic that goes inside and outside of the AS. This router should know about all the possible places from which traffic may come and to which this traffic may go. At the same time, this router should somehow tell neighbor edge routers which addresses there are inside the AS. This is what BGP does.

Border Gateway Protocol is one of the most important protocols used over the Internet, but at the same time it is not so popular, because developers don’t deal with it every day. They don’t send or receive BGP packets, don’t set up hardware and software to work with BGP, and so on.

The way routers use BGP is not hard to explain, but there are many details and options, so we won’t break down the protocol itself, its header and mechanics. Instead, let’s see a board overview (but if you are curious — BGP works over TCP).

Let’s say that we have the same network of Autonomous Systems we’ve described earlier. But now, one of the companies that has an AS decided to buy one more block of IPs and it needs to tell any neighbor AS that now it handles those IPs. To do it, this company sends BGP a packet that contains:

1. The address of the new network that now can be reached.
2. The list of AS which should be crossed during the transmission to this network (initially there is only one ASN).
3. The address of the router that is in charge of accepting the incoming traffic.

Every neighbor AS remembers this data and retransmits it to its own neighbors, changing the ASN and the router address.

As you see, every next AS adds itself to the list of AS (which is usually called an AS-path). This leads to the fact that every router that gets this update can decide should it remember the path, or not. Sometimes the advertised path might be longer that this router already knows.

So, yeah, an edge router of an AS has a “BGP table,” which is a huge list of the subnets and the AS paths related to them. For example:

   Network          Next Hop        Path
*> 1.0.0.0/24       213.200.64.93   3257 15169 i
*> 1.0.4.0/24       213.200.64.93   3257 6453 7545 56203 i
*> 1.0.6.0/24       213.200.64.93   3257 174 4826 38803 56203 i
*> 1.0.20.0/23      213.200.64.93   3257 2516 2519 i
*> 1.0.64.0/18      213.200.64.93   3257 2516 7670 18144 i
*> 1.0.128.0/18     213.200.64.93   3257 174 38040 9737 i
*> 1.0.128.0/17     213.200.64.93   3257 38040 9737 9737 i
*> 1.0.129.0/24     213.200.64.93   3257 4651 9737 9737 23969 i
*> 1.0.130.0/24     213.200.64.93   3257 6453 4651 9737 9737 9737 23969 i
*> 1.0.160.0/19     213.200.64.93   3257 2914 38040 9737 i
*> 1.0.192.0/21     213.200.64.93   3257 6453 4651 9737 9737 9737 23969 i
...

When this router receives a new user packet, it checks its destination IP, then looks into the BGP table and decides to which AS this packet should be sent.

Maintaining BGP tables is hard work, because the updates are critical and any wrong movement may initiate a wave of wrong updates, which may lead to the broken network. You’ve probably heard about the long Facebook outage that happened in 2021. That’s exactly what went wrong — they accidentally “told” every neighbor AS that their IPs are not reachable any more. (If you are interested, check this detailed explanation)

So, BGP is truly the duct tape that holds the whole Internet and doesn’t let it fall apart.

Summary

What we’ve explained here is the latest piece of internetworking that we will need to understand CDNs.

Now you know that the Internet is a network of Autonomous Systems. Every AS is maintained by some company which rules it the way this company wants. But every AS has to communicate with other AS to understand how to send and receive traffic. To make this communication easier, every AS uses the Border Gateway Protocol.

The only question we should answer before the end is “How does our computer understand which IP it should use as a destination one when wype uploadcare.com in the address bar?”

This is exactly the topic of the next article.