The worker 10.1.4.21 in network A (AS 64512) sends a query to the database, db-b1, at 10.3.2.6
in network B (AS 64513), prefix 10.3.0.0/16. Two roads join border-a1 to border-b1: through the
providers’ routers p1-edge (203.0.113.9) and p2-edge, or over a direct peering link, 198.51.100.0/30,
where border-b1 is 198.51.100.2. Nobody typed either route, so how did A learn both, and who picked one?
Intent: inside one network, routers compute routes from a shared map. Between networks, each router announces a prefix with the path it took, and policy picks before path length. ICMP is how the network tells a host what went wrong.
Inside a network and between networks#
A network that one team runs is an autonomous system (AS). Inside it, routers share one map and each computes its own shortest paths. Between ASes there is no shared map: border routers tell each other which prefixes they reach, and through which ASes.
Autonomous systems and their numbers#
An AS is a set of routers under one administration and one routing policy, named by an AS number (ASN). The numbers 64512 to 65534 are kept for private use. Like the RFC 1918 addresses of Part 4, private ASNs must be removed before a route reaches the internet.
| Network | ASN | Routers in this part | Role |
|---|---|---|---|
| A | 64512 | gw-a1 10.1.4.1, core-a1, border-a1 | the network of the worker 10.1.4.21, prefix 10.1.4.0/24 |
| B | 64513 | border-b1, db-b1 10.3.2.6 | the database’s network, prefix 10.3.0.0/16 |
| P1 | 64600 | p1-edge | A’s provider |
| P2 | 64601 | p2-edge | B’s provider |
P1 and P2 are connected to each other. A and B also share a direct peering link, 198.51.100.0/30, between border-a1 and border-b1.
Inside: one map, shared by flooding#
Inside A, the routers run OSPF (Open Shortest Path First). Each router tells every other router in its area about its own links: up or down, and the cost the operator set. Every router then holds the same map and computes the shortest paths from itself.
A large network splits into areas joined by a backbone area, so a change is flooded only inside its own area.
Between: announcements over BGP#
Between ASes, routers run BGP (Border Gateway Protocol). A border router tells its neighbour “I
reach 10.3.0.0/16, and the path goes through these ASes”. It says nothing about the links or costs
inside its own network.
Two routers speak BGP over a TCP connection to port 179. UPDATE messages announce new routes and withdraw old ones.
The session stays up while KEEPALIVE or UPDATE messages keep arriving. The hold time is the number of seconds, agreed when the session opens, that may pass without one. When it runs out, the router drops every route it learned over that session.
A session between two ASes is external BGP (eBGP). A session between two routers of the same AS is internal BGP (iBGP).
The table uses two words the next section explains. The AS-PATH is the list of ASes a route has crossed; the NEXT-HOP is the router to send packets to.
| eBGP | iBGP | |
|---|---|---|
| between | routers of two ASes: border-b1 and border-a1 | routers of one AS: border-a1 and core-a1 |
| the AS-PATH | the sender adds its own ASN | left as it is |
| the NEXT-HOP | by default, the sender’s own address on the shared link | left as it is, by default |
| local preference | never sent | sent: how much this AS likes the route |
| the path it runs on | the direct link between the two ASes, unless configured otherwise | any path inside the AS, across several routers |
An everyday picture: OSPF is a floor plan that every router holds. BGP is directions passed on by word of mouth: “reach B through me, I go via P2”. The picture breaks in one place: a router acts on the directions it hears, so a wrong announcement spreads (the route leak, below).
Why two protocols#
| Inside one AS: OSPF | Between ASes: BGP | |
|---|---|---|
| policy | one owner, nothing to negotiate | each AS decides what it accepts and what it announces |
| scale | the links of one network | the whole internet: about 1.09 million IPv4 prefixes in one public table on 2026-10-15 |
| performance | the lowest total cost wins | a route the policy allows wins; length is counted in ASes, not in milliseconds |
| what is shared | every link and its cost | reachable prefixes and the ASes on the path |
In a VPC#
In a VPC you run no OSPF: the provider’s controller writes the routes inside the VPC (Part 4). BGP comes back at the edge, when the VPC connects to your data centre or to another cloud. The link is a VPN or a dedicated line.
There, a managed BGP speaker, often called a cloud router, holds the session. It does not forward packets. The provider’s virtual network does, with the routes the speaker learned.
| The provider does | You still decide |
|---|---|
| runs the BGP speaker and keeps the session up | the ASN at each end: one provider requires a private ASN on its side; yours may be public or private |
| turns learned prefixes into dynamic routes in the VPC route table | which prefixes you announce, and which you accept |
| forwards the packets | the keepalive timer: one provider defaults to 20 s and sets the hold time to three times that |
| picks among learned routes by its own documented order | what wins when a learned route meets your static routes: the longest prefix first, then each provider’s rules. A learned route does not override the VPC’s own subnet routes (Sources) |
What an announcement carries#
An announcement is a prefix plus attributes; together they make a route. Here B’s prefix leaves network B and reaches network A. Two attributes do most of the work: the AS-PATH, the list of ASes crossed, and the NEXT-HOP, the router to send packets to.
| Attribute | What it says | On the route that reaches border-a1 from p1-edge |
|---|---|---|
| prefix | the destination block | 10.3.0.0/16 |
| AS-PATH | the ASes crossed, the latest on the left | 64600 64601 64513 |
| NEXT-HOP | the router to send packets to | p1-edge’s address on its link to border-a1, 203.0.113.9 |
| local preference (LOCAL_PREF) | how much this AS likes the route; higher wins; inside one AS only | added by border-a1 when it passes the route on |
The AS-PATH grows and stops loops#
B sends 10.3.0.0/16 with AS-PATH 64513. P2 passes it on as 64601 64513, and P1 as 64600 64601 64513. Each AS adds its number on the left when it sends the route to another AS.
The list is also a loop check. If P1 sent the route back to P2, p2-edge would find 64601 already in
the path and ignore it. So, with this check on (the default), an announcement does not come back
into an AS it has crossed.
The NEXT-HOP changes at each AS border#
When p1-edge sends the route to border-a1 over eBGP, it puts its own address in the NEXT-HOP.
Packets for 10.3.0.0/16 sent along this route go to p1-edge first.
Inside A, border-a1 passes its pick to core-a1 over iBGP, and by default leaves the NEXT-HOP as it
was. Its pick is the route over the peering (next section). So core-a1 receives border-b1’s address
on the peering link, 198.51.100.2.
The router core-a1 finds the way to that address in its OSPF routes: through border-a1. This is where the two protocols meet: BGP says which exit, OSPF says how to get there.
This works only if A’s OSPF carries the peering link’s subnet, 198.51.100.0/30. If not, core-a1 cannot reach the
NEXT-HOP and sets the route aside. That is why a border router can be configured to put its own
address in the NEXT-HOP instead, which the standard allows.
How a router picks among several routes#
border-a1 now holds two routes to 10.3.0.0/16. It installs one by walking a fixed list of rules
until one route is left. The first rule is policy, not distance.
Unusable routes are dropped first#
Before any rule, a route is set aside if its NEXT-HOP cannot be reached. It is also set aside if the router’s own ASN appears in its AS-PATH.
The order of the rules#
| Rule | Keep the routes with | Who sets the value |
|---|---|---|
| 1 | the highest local preference; for a route from another AS, the value local policy gives it | your policy |
| 2 | the shortest AS-PATH | the ASes on the way |
| 3, 4 | two tie-breakers you will rarely meet: how the route first entered BGP, and a hint from the neighbour AS | the first AS; the neighbour |
| 5 | a route learned over eBGP, rather than over iBGP | the session type |
| 6 | the lowest interior cost to the NEXT-HOP, from OSPF | your OSPF costs |
| 7 | the lowest BGP identifier of the router that sent it | each router |
| 8 | the lowest peer address | each session |
This is the order in the BGP standard. Router products add steps of their own.
The pick at border-a1#
| Route 1 | Route 2 | |
|---|---|---|
| learned from | p1-edge, over eBGP | border-b1, over eBGP on the peering 198.51.100.0/30 |
| AS-PATH | 64600 64601 64513 | 64513 |
| NEXT-HOP | p1-edge, 203.0.113.9 | border-b1, 198.51.100.2 |
| rule 1, local preference | the same: A has set no policy | the same |
| rule 2, AS-PATH length | 3 | 1: wins |
Route 2 wins at rule 2, and rules 3 to 8 are never reached. border-a1 installs it and passes it to core-a1 over iBGP.
Why policy comes first#
The AS-PATH counts networks, not kilometres or milliseconds. One AS can span a continent, so a short AS-PATH is not always a fast one.
Policy comes first because every link has a price, and each AS knows its own. If A wanted traffic for B to go through P1, its operator would give routes from p1-edge a higher local preference. Rule 2 would then never be reached.
Policy: who carries whose traffic#
Every link between two ASes is also a contract: who pays whom, and for which traffic. Policy turns the contract into rules about what each router announces and accepts. The same policies are why a reply can come back by another road.
Customer, provider, peer#
| Your neighbour is (seen from the sender) | Who pays | What you announce to it | In the example |
|---|---|---|---|
| your provider | you pay it | your own prefixes and your customers' | A tells P1 only 10.1.4.0/24 |
| your customer | it pays you | every route you know, so it reaches the whole internet | P1 tells A all its routes |
| your peer | nobody pays for the traffic itself | your own prefixes and your customers' | A tells B only 10.1.4.0/24; B tells A only 10.3.0.0/16 |
These are the usual rules, not a standard: each pair of networks agrees on its own.
A multihomed stub and the route leak#
A pays P1 to carry its traffic to the rest of the internet (transit), and peers with B. It means to carry nobody else’s traffic. Such a network is a multihomed stub: two neighbours, P1 and B, yet every packet crossing its border starts or ends inside it.
A keeps that promise only through what it announces. If border-a1 told P1 “I reach 10.3.0.0/16”,
P1 could send B’s traffic through A and over A’s peering link. A would carry strangers’ traffic at its
own cost.
Sending a route further than it was meant to go is a route leak. A peer’s route passed to your own provider is one of the leak types the standards name. Routers that follow RFC 8212 accept and send nothing over eBGP until you write a policy for each direction; check what yours does.
Hot potato#
When the earlier rules tie, rule 6 picks the exit with the lowest interior cost from this router. This is hot potato routing: like a hot potato, the packet is handed to the next AS as early as possible. Each AS saves its own links and ignores what the rest of the path costs.
Why the reply can come back another way#
Each AS picks its routes by its own rules. A picks a route to 10.3.0.0/16; B picks a route to
10.1.4.0/24. The two choices are not linked.
In this example B’s operator gives routes learned from P2 a higher local preference. Rule 1 then decides before rule 2 is reached. The reply goes through P2 and P1, though the peering path is shorter.
| Direction | Decided at | Path | AS-PATH used |
|---|---|---|---|
request, worker 10.1.4.21 to db-b1 10.3.2.6 | border-a1, rule 2: shortest AS-PATH | gw-a1, core-a1, border-a1, the peering, border-b1 | 64513 |
reply, db-b1 to worker 10.1.4.21 | border-b1, rule 1: local preference | border-b1, p2-edge, p1-edge, border-a1, core-a1, gw-a1 | 64601 64600 64512 |
A traceroute from the worker shows only the request’s path. The round-trip time it prints includes the reply’s longer road.
Anycast: one address, many places#
Anycast announces the same prefix over BGP from several sites, here 192.0.2.0/24, with the
service on 192.0.2.53 at each. Every router keeps the route it likes best, so
each client reaches the site its own network picks. Nothing in the packet says which site it will meet.
“Nearest” here means best by the rules above: policy first, then usually the shortest AS-PATH. It is not measured in kilometres. The DNS root servers use anycast, and a large content network answers DNS for its customers’ sites with anycast addresses (Sources).
A failed site keeps attracting traffic until it withdraws its announcement. Only then do the clients that landed there move to the next best site. DNS clients can also retry another server on their own.
| Traffic | Anycast | Why |
|---|---|---|
| a DNS question and answer over UDP | fits | one packet each way; a route change between two questions does no harm |
| short web requests | fits, with care | a route change breaks only the requests in flight; content networks run this |
| a long TCP session: a replication stream, a large download, a VPN tunnel | risky | after a route change, packets reach a site with no state for the connection, and the session is reset |
ICMP: the network talking back#
ICMP (Internet Control Message Protocol) is how routers and hosts tell a sender what happened to its packet. It rides inside IP as protocol 1, and every message has a type and a code. Ping, traceroute and path MTU discovery are all built on it.
What ICMP says#
An ICMP error carries the IP header of the packet that caused it, and at least the first 8 bytes of its data. Those bytes hold the ports, so the sender’s system can match the error to the right connection.
| Type/code | Name | Sent by | When you meet it |
|---|---|---|---|
| 8/0 | echo | ping | the question ping asks |
| 0/0 | echo reply | the host that was pinged | ping, and traceroute -I, get an answer |
| 3/0, 3/1 | network unreachable, host unreachable | a router | no route, or no host at the end; traceroute prints !N or !H |
| 3/3 | port unreachable | the destination host | nothing listens on that UDP port |
| 3/4 | fragmentation needed and DF (don’t fragment) set | a router | the packet is too big for the next link and may not be split |
| 11/0 | time exceeded in transit | a router | the TTL reached 0 |
ping 10.3.2.6 sends echo, 8/0, and db-b1 answers with echo reply, 0/0. Routers may limit how many
ICMP errors they send. That is one reason a busy router shows * * * in a traceroute (Part 1).
Traceroute line by line#
traceroute -I 10.3.2.6, run on the worker 10.1.4.21, sends echo probes, first with TTL 1, then 2, and so on, three per TTL by
default. Each router lowers the TTL by one. The router that brings it to 0 drops the probe and sends
11/0 back from its own address.
Traceroute prints that address, looks up its name itself, and times the round trip. The message from the router carries no name.
| TTL | Who answers | With | The line shows |
|---|---|---|---|
| 1 | gw-a1, 10.1.4.1 | 11/0 | gw-a1.acme.internal, under 1 ms |
| 2 | core-a1 | 11/0 | core-a1, under 1 ms |
| 3 | border-a1 | 11/0 | border-a1, under 1 ms |
| 4 | border-b1 | 11/0 | border-b1, about 50 ms: the long road between the networks, there and back |
| 5 | db-b1 | 0/0 | db-b1.acme.internal, about 50 ms; the trace stops |
Without -I, Linux traceroute sends UDP probes to port 33434 and up, and db-b1 ends the trace with
3/3 (Part 1). Each line names the router on the way out; the 50 ms includes the answer’s road back through p2-edge, p1-edge and border-a1.
Path MTU discovery#
A link carries packets up to its MTU (Part 4). A sender wants the largest packet the whole path accepts, the path MTU, without packets being split on the way.
| IPv4 | IPv6 | |
|---|---|---|
| who may split a packet | a router, unless the sender set DF (don’t fragment) | only the sender; routers never split |
| a packet too big for the next link | dropped; the router sends 3/4 with that link’s MTU | dropped; the router sends ICMPv6 type 2, Packet Too Big, with that link’s MTU |
| what the sender does | lowers its estimate and sends smaller packets | the same |
| the smallest MTU a link may have | 68 bytes | 1,280 bytes |
An IPv4 host doing path MTU discovery sets DF on every packet. Say the road between A and B crossed a tunnel with an MTU of 1,440 (Part 8).
The database db-b1 sends a 1,500-byte packet, and the router at the tunnel drops it. That router answers 3/4 with 1,440, and db-b1 sends smaller packets from then on.
What breaks when ICMP is blocked#
| Blocked | What you see | Why |
|---|---|---|
| 3/4, or ICMPv6 Packet Too Big | the handshake and small queries work; a big result hangs | the sender never learns the path MTU and resends the same large packet: a black hole, where packets vanish without a word |
| 11/0 | traceroute prints * * * for those hops | the routers’ answers never come back |
| echo, 8/0 and 0/0 | ping fails while the service works | ping and the service use different protocols |
| 3/3, 3/1 | a connected UDP client waits for its timeout instead of failing at once | the error that UDP would hand to the application never arrives |
Blocking all ICMP breaks path MTU discovery. Let 3/4 and ICMPv6 type 2 through every firewall on the path.
In a VPC, the provider’s firewall decides which ICMP reaches your VM. One provider passes ICMP errors that belong to an allowed connection, and another always passes path MTU messages.
A third filters each subnet with a network ACL. That ACL needs a rule for 3/4 when the MTUs differ or the peer is on the internet (Sources).
When it breaks#
| Case | Description | Check with | Why | Fix |
|---|---|---|---|---|
| a VPN that announces too much | the far end of a VPN announces prefixes it learned elsewhere, not only its own; traffic for them now enters the tunnel | the learned routes listed for the BGP session, then the VPC route table’s dynamic routes | no import or export policy filters the session, and a learned route more specific than one of your static routes wins (Part 4); it cannot override the VPC’s subnet routes | accept only the prefixes you expect; announce only your own |
| a peering that flaps | the A-B session goes down and up every few minutes; traffic swings between the peering and the providers | the session’s state and uptime on border-a1 (show bgp summary on an open-source router), and its log | keepalives are lost on the link; after the hold time, which the standard suggests as 90 s, the router drops every route from the session | repair the loss; set the same timers at both ends, since the session uses the smaller hold time |
| a big result that hangs | the connection opens and small queries work; a large result never arrives | ping -M do -s 1472 10.3.2.6 (-M do sets the DF flag; 1,500 bytes with headers), then smaller sizes; tracepath 10.3.2.6 | a link on the path has a smaller MTU, and a firewall drops the 3/4 that says so | let 3/4 and ICMPv6 type 2 through; clamp the TCP MSS, 1,400 in a 1,440 tunnel (Part 3) |
| a reply that comes back another way | the worker’s traceroute shows the peering, but the delay or a firewall log points at P1 | traceroute from both ends: the worker to 10.3.2.6, db-b1 to 10.1.4.21 | each AS picks its own route, and B’s LOCAL_PREF for P2 beats the shorter peering path | nothing, if it works; if you need one road, only B’s operator can change how B chooses, but A can change what B has to choose from, by what it announces and where |
Remember#
- Inside one network, routers share a map (OSPF); between networks, they pass announcements (BGP).
- Each AS adds its number to the AS-PATH, and a router that finds its own number drops the route.
- Policy, LOCAL_PREF, is checked before path length, and path length counts networks, not milliseconds.
- Each side picks its own road, so the reply can come back another way.
- ICMP is the network talking back: traceroute counts on 11/0, and blocking 3/4 makes big transfers hang.
Sources#
Checked 2026-10-16.
- RFC 4271, A Border Gateway Protocol 4 (BGP-4): port 179, the attributes, the decision process (sec. 9.1), the timers (sec. 10)
- RFC 2328, OSPF Version 2: flooding inside an area, the 30-minute refresh, protocol 89
- RFC 6996, Autonomous System (AS) Reservation for Private Use: the private AS blocks
- RFC 1772, Application of the Border Gateway Protocol in the Internet: stub and multihomed ASes
- RFC 7908, Problem Definition and Classification of BGP Route Leaks
- RFC 8212, Default External BGP (EBGP) Route Propagation Behavior without Policies
- RFC 7094, Architectural Considerations of IP Anycast
- RFC 792, Internet Control Message Protocol
- RFC 1812, Requirements for IP Version 4 Routers: TTL expiry, the ICMP error contents, rate limiting
- RFC 1191, Path MTU Discovery
- RFC 4443, ICMPv6: Packet Too Big
- RFC 8200, IPv6 Specification: no fragmentation by routers, the 1,280-byte minimum
- RFC 2923, TCP Problems with Path MTU Discovery: the black hole (sec. 2.1)
- RFC 8201, Path MTU Discovery for IPv6: connections that hang when the messages are blocked
- RFC 1122, Requirements for Internet Hosts: UDP passes ICMP errors to the application
- traceroute(8), Debian, ping(8), Debian, tracepath(8), Debian
- FRRouting, BGP: its route selection steps and the RFC 8212 default
- Potaroo, the active BGP table: the IPv4 prefix count on 2026-10-15
- Cloudflare, How Cloudflare works: anycast addresses for proxied names
- Google Cloud, Cloud Router overview, Create a Cloud Router, BGP timers, learned routes, VPC routes, VPC firewall rules
- AWS, How route priority works, Path MTU Discovery and network ACLs
- Alibaba Cloud, VPC route tables, Security group rules