The worker 10.1.4.21 in network A sends a query to the database, db-b1, at 10.3.2.6 in network
B, prefix 10.3.0.0/16. The packet crosses four routers: gw-a1 10.1.4.1, the subnet’s gateway
(the router that leads out of it), then core-a1, border-a1 and border-b1. None of them knows the whole path, so how does each one decide where to send it?
Intent: a router forwards each packet on its own. It matches the destination against its table, longest prefix first, lowers the TTL and sends the packet on. Subnets, DHCP and NAT decide which addresses those packets carry.
Forwarding and routing#
Each router on the path does two different jobs. Moving one packet from the link it came in on to the link it leaves on is forwarding. Agreeing with the other routers on the table that says which link is routing. The figure shows why the two are separate: a link fails, routing changes one row, and forwarding goes on reading the table without knowing why.
Two jobs at two speeds#
The figure uses Part 5’s network. border-a1 reaches network B two ways: over the peering, a direct link to border-b1, or through the providers’ routers p1-edge (P1) and p2-edge (P2).
| Phase | Who acts | What changes |
|---|---|---|
| Routing | border-b1 and p1-edge each tell border-a1 that they reach 10.3.0.0/16 | border-a1 keeps both routes and chooses the peering: one network on the way (B), against three (P1, P2, B); Part 5 explains the choice |
| Forwarding | the worker 10.1.4.21 sends queries to db-b1; border-a1 reads the chosen row for every packet for 10.3.2.6 | nothing: no routing message is sent, the packets go over the peering to db-b1 |
| Link down | border-a1 loses the peering: the link signal goes, or no message from border-b1 arrives within the hold time (Part 5) | border-a1 marks that route down, chooses the p1-edge route and tells core-a1 |
| Forwarding | the worker’s next query: the same lookup, for the same destination | only the row: the packets now go through p1-edge and p2-edge |
Routing changes the table only when the network changes. Forwarding sends no message of its own: it reads the row that routing left and does not know why the row says what it says.
| Forwarding | Routing | |
|---|---|---|
| scope | one router, one packet | every router of a network |
| input | the packet’s destination address | messages from other routers, or the operator’s settings |
| output | one outgoing link | the rows of the forwarding table |
| when | for every packet, at the speed of the link | the table changes only when the network changes |
| speed | much faster: one lookup per packet | slower: routers exchange messages and recompute |
The table a packet reads is the forwarding table: one row per prefix, each with a next hop, the next router on the way.
An everyday picture: forwarding is a post office worker who reads the zip code and drops the letter in a box. Someone else labelled the boxes, earlier. The picture breaks at a full box: a router does not keep the letter for later but throws it away (Part 1).
Inside a router#
| Part | What it does to a packet |
|---|---|
| input port | receives it and looks up the destination in the forwarding table |
| output queue | holds it until the outgoing link is free; drops it when the queue is full (Part 1) |
| output port | sends it onto the next link |
| routing processor | runs the routing protocols and writes the forwarding table; it does not touch each packet |
A port on a router is a physical interface, not a TCP or UDP port.
Who writes the table#
| Who | How | Example |
|---|---|---|
| you | a static route, typed by hand | on a Linux router, ip route add 10.3.0.0/16 via <address of border-a1> |
| the routers | a routing protocol: OSPF inside a network, BGP between networks (Part 5) | border-a1 announces 10.3.0.0/16 to core-a1 |
| your provider, in a VPC (virtual private cloud) | the provider’s own system, from the subnets you create | the routes inside the VPC |
In a VPC#
In a VPC you do not configure the routers. The provider writes the routes inside the VPC for you.
Some providers write a route per subnet range; another writes one local route for the whole VPC.
You add static routes, for example a default route that sends internet traffic out. On some providers its next hop is a NAT gateway; on another, the NAT rides on a default route to the internet gateway. Dynamic routes arrive from a BGP session (VPN, dedicated line or a transit service), held by the provider’s BGP router (Part 5).
The longest prefix decides among these routes. One provider first checks its policy-based routes and the subnet routes, then compares prefixes among the rest (next section).
The longest prefix wins#
Inside core-a1, the forwarding table holds four rows. Packets arrive from gw-a1 10.1.4.1, and
the address 10.3.2.6 matches three rows. Which row does the router use, and when does it send to
nat-a1, the NAT gateway at 203.0.113.7?
What a prefix says#
A prefix such as 10.3.0.0/16 says: the first 16 bits are fixed, and the other 16 are free. It
covers 65,536 addresses, from 10.3.0.0 to 10.3.255.255.
To test 10.3.2.6, compare its first 16 bits with the prefix. Sixteen bits are the first two
numbers, 10.3 in both, so the address matches.
Every row that matches is a candidate, and the longest prefix wins. 0.0.0.0/0 fixes no bits at
all: it matches every address, so it loses to every other match. It is the default route.
core-a1’s table, walked#
| Prefix | Next hop |
|---|---|
10.1.4.0/24 | gw-a1 |
10.3.0.0/16 | border-a1 |
10.0.0.0/8 | drop |
0.0.0.0/0 | nat-a1 |
| Destination | Rows that match | Longest | core-a1 sends it to |
|---|---|---|---|
10.3.2.6, db-b1 | /16, /8, /0 | 10.3.0.0/16 | border-a1 |
198.51.100.20, a mirror on the internet | /0 | 0.0.0.0/0 | nat-a1 |
10.9.9.9, an address no network uses | /8, /0 | 10.0.0.0/8 | nowhere: dropped |
On a Linux host, ip route get 10.3.2.6 prints the row the kernel picked: the next hop and the
interface.
Why tables are built this way#
One row can cover many networks. 10.3.0.0/16 covers every subnet of network B, so core-a1 needs
no row per subnet. Joining many prefixes into one shorter prefix is called aggregation.
A longer row makes an exception to a shorter one. The row 10.0.0.0/8 covers all of 10.x,
yet 10.1.4.0/24 and 10.3.0.0/16 still win for their own parts.
The /8 row is a blackhole route: a row whose action is to drop. Without it, 10.9.9.9 would match
only 0.0.0.0/0 and leave through nat-a1 for the internet, where private addresses should not go.
| Route type, on Linux | What happens to a matching packet |
|---|---|
blackhole | dropped silently |
unreachable | dropped, and the sender gets ICMP host unreachable |
In a VPC#
The providers’ pages state the same rule: the most specific route that matches wins. Subnet routes get extra protection.
| What you try | What the provider pages say |
|---|---|
| a static route equal to a subnet range, or inside it | one provider refuses to create it |
| a learned route equal to a subnet range, or inside it | the same provider does not install it |
| a static route more specific than a broad learned one | it wins, by the longest prefix |
| a static and a learned route with the same prefix | another provider prefers the static route |
So a broader route never wins against a subnet’s route by the longest-prefix rule (one provider’s policy-based routes are checked first). A more specific route beats a less specific one, and your static routes follow the same rule (Part 5).
The header a router reads#
A router reads a few fields of the IPv4 header and usually changes only the TTL and the
checksum. The figure follows the worker’s packet from 10.1.4.21 to db-b1 10.3.2.6 again, through gw-a1
10.1.4.1 and core-a1, which sends it on by its row 10.3.0.0/16, with its 20-byte header open
below it.
The fields#
The header is 20 bytes when it carries no options.
| Field | Bits | What a router does with it |
|---|---|---|
| version | 4 | reads it: 4 for IPv4 |
| header length | 4 | reads it to find where the data starts; 5 words of 4 bytes, 20 bytes, without options |
| DS field (differentiated services) and ECN (explicit congestion notification) | 6 + 2 | may queue by the DS field; marks ECN when its queue starts to grow, instead of dropping; once named “type of service” |
| total length | 16 | compares it with the next link’s MTU; at most 65,535 bytes |
| identification, flags, fragment offset | 16, 3, 13 | used only when a packet is split; DF is one of the flags |
| TTL | 8 | lowers it by at least one; drops the packet at zero |
| protocol | 8 | a plain router leaves it; a firewall or NAT reads it; the destination hands the data to TCP (6) or UDP (17) |
| header checksum | 16 | checks it on arrival, then recomputes it after changing the TTL |
| source address | 32 | leaves it (a NAT changes it, below) |
| destination address | 32 | the key of the lookup |
A router with a wrong checksum drops the packet. It need not add up the whole header again after the TTL change: it can update the old checksum for the one changed field. In the figure the checksum goes 0xd173, 0xd273, 0xd373, 0xd473, 0xd573 while the TTL falls from 64 to 60: the TTL is the high byte of its 16-bit word, so one less TTL is 0x0100 more in the checksum. The rest of the header stays as the worker wrote it: identification 0x4f2a, the DF flag set, ECN 0, protocol 6 for TCP, in IPv4 version 4.
TTL#
Every router lowers the TTL (time to live) by at least one. A router that brings it to zero drops the packet and tells the sender with an ICMP message (Part 5).
Without it, a packet caught in a routing loop would circle forever. The worker starts at 64, the Linux default, so its packet reaches db-b1 with 60 after four routers. Traceroute counts on this (Parts 1 and 5).
Too big for the next link#
One flag, DF (don’t fragment), decides what happens to a packet that is bigger than the next link. Here one link on the way is too small for the worker’s packet.
Every link has an MTU (maximum transmission unit): the largest packet it carries. Most Ethernet links take 1,500 bytes. A VPC sets its own: one provider defaults to 1,460 and allows 1,300 to 8,896.
Suppose the road from border-a1 in network A to border-b1 runs through a tunnel with an MTU of 1,440. Parts 7 and 8 explain the missing 60 bytes. The worker sends a 1,500-byte packet through gw-a1, core-a1 and border-a1: 20 bytes of header and 1,480 of data.
With DF clear, an IPv4 router may split the packet into fragments. Each fragment carries a copy of the header, and every fragment but the last carries a multiple of 8 data bytes.
| Fragment | Total length | Data bytes | Offset, in 8-byte units | More fragments (MF) |
|---|---|---|---|---|
| 1 | 1,436 | 1,416 | 0 | yes |
| 2 | 84 | 64 | 177 | no |
The first fragment takes the largest multiple of 8 that fits: 1,440 − 20 = 1,420, rounded down to 1,416. The second carries the remaining 64 bytes, starting at byte 1,416, which is 177 units of 8.
A router does not join the fragments it forwards. db-b1 does, matching them by source, destination, protocol and the identification field.
With DF set, border-a1 may not split the packet, so it drops it. It sends the worker ICMP 3/4 (type 3, code 4), fragmentation needed and DF set, with the next link’s MTU, 1,440: the next-hop MTU. Part 5 shows how a sender uses this to find the largest packet a path accepts.
IPv6: the same job, a fixed header#
An IPv6 router does the same job: longest prefix, a counter lowered by one, then the next link. The header is simpler.
| IPv4 | IPv6 | |
|---|---|---|
| address | 32 bits | 128 bits |
| header | 20 bytes or more | 40 bytes, fixed |
| the counter | TTL | hop limit, the same job |
| header checksum | yes, recomputed at each router | none |
| who splits a packet | a router, unless DF is set, or the sender | only the sender |
| smallest MTU a link may have | 68 bytes | 1,280 bytes |
IPv6 is in use today. On 2026-10-13, about 46 % of users reached Google over IPv6; on Saturday 2026-10-10, about 51 %.
A subnet: who you reach without a router#
gw-a1 joins three links, so it has three interfaces and three addresses. Cut the router loose and each link left is an island: a subnet. A host reaches its own island directly and every other island through the gateway.
Addresses belong to interfaces#
An address belongs to an interface, not to a host. A router has one interface per link, usually
with one address on each: gw-a1 has 3 interfaces, 3 addresses. 10.1.4.1 is its address on the
worker’s subnet only; 10.1.5.1 is on 10.1.5.0/24, the subnet of ops-a1; 10.1.255.1 is on the link to core-a1,
10.1.255.2.
A subnet is the set of interfaces that reach each other with no router in between. Its prefix says
which addresses are in it: 10.1.4.0/24 holds 10.1.4.0 to 10.1.4.255. The link between two
routers is a subnet too: 10.1.255.0/30 is an island of two interfaces.
The first address names the subnet. The last one is its broadcast address, the address that means
everyone on the subnet. So a /24 has 254 addresses left for hosts, and a /30 has two.
The host’s own rule#
The worker keeps a two-row table of its own (output trimmed):
$ ip route
default via 10.1.4.1 dev eth0 proto dhcp
10.1.4.0/24 dev eth0 proto kernel scope link src 10.1.4.21It applies the same longest-prefix rule. 10.1.4.22 matches the /24 row, so the worker sends the
packet straight to worker-2 on the link, the same island (Direct). 10.1.5.10 matches only the
default route, so it goes to the gateway, 10.1.4.1, as would 10.3.2.6 or any other island
(Through the gateway).
Through the gateway, the destination in the IP header stays 10.1.5.10. Only the link-layer frame
is addressed to gw-a1 (Part 6).
Private addresses#
Anyone may use these blocks inside a network. Routers on the internet do not carry them, so a host with such an address needs NAT to reach the internet (below).
| Block | Addresses | Defined by | Meant for |
|---|---|---|---|
10.0.0.0/8 | 16,777,216 | RFC 1918 | private networks |
172.16.0.0/12 | 1,048,576 | RFC 1918 | private networks |
192.168.0.0/16 | 65,536 | RFC 1918 | private networks |
100.64.0.0/10 | 4,194,304 | RFC 6598 | shared space between an internet provider’s own NAT and its customers; one cloud provider routes it to its own services |
In a VPC#
A cloud subnet loses some addresses to the provider. One provider keeps four in each subnet range: the first two and the last two. Another keeps five: the first four and the last one.
| Subnet | Usable addresses, classic | Keeps four | Keeps five |
|---|---|---|---|
10.1.4.0/24 | 254 | 252 | 251 |
Both providers put the subnet’s gateway on its second address, as gw-a1 sits on 10.1.4.1.
Getting an address: DHCP#
A new VM boots with no address. Before it can send anything, it must learn its address, its prefix, its gateway and its DNS server. DHCP (Dynamic Host Configuration Protocol) hands them out for a limited time, the lease.
Four messages#
Here gw-a1, 10.1.4.1, answers DHCP for the subnet. Every message carries the same transaction id
(xid), 0x5a1c, so the worker can tell its answers from anyone else’s.
| Message | From | To | Carries |
|---|---|---|---|
| discover | the worker, source 0.0.0.0, UDP port 68 | 255.255.255.255 port 67: everyone on the link | “I need an address” |
| offer | gw-a1, 10.1.4.1, port 67 | the worker, port 68: sent to the offered address, or broadcast if the worker asked for that | address and prefix 10.1.4.21/24, gateway 10.1.4.1, DNS server 10.1.0.53, lease 86,400 s (one day) |
| request | the worker, 0.0.0.0, port 68 | 255.255.255.255 port 67 | “I take 10.1.4.21 from 10.1.4.1” |
| acknowledgement (ACK) | gw-a1, port 67 | the worker, port 68, as the offer | “10.1.4.21 is yours for 86,400 s” |
The worker has no address yet, so it sends from 0.0.0.0 to everyone. worker-2 10.1.4.22 hears both
broadcasts and answers none: it is not a DHCP server. The request is broadcast too: any other server
that made an offer learns that it was not chosen.
The lease and the renewal#
The address is only borrowed. By default, at half the lease, 43,200 s or 12 hours here, the worker
asks 10.1.4.1 to renew it.
When nobody answers#
If no DHCP server answers, a host may give itself an address in 169.254.0.0/16. Such a link-local
address works only on the same link, and no router forwards it.
A VM whose own interface address is 169.254.x.x usually means its DHCP failed. A cloud’s metadata server at 169.254.169.254 is a
different, normal use: a destination, not the VM’s address.
Which server#
ISC DHCP, long a widely used open-source server, ended maintenance at the end of 2022. ISC’s newer server, Kea, replaces it.
NAT: many private addresses behind one#
The worker needs a package from the mirror, a server on the internet that holds a copy of the
packages, at 198.51.100.20 port 443. Its address, 10.1.4.21, is private: no router on the internet
carries a route back to it. nat-a1, the NAT gateway of network A, lends it its own outside address,
203.0.113.7.
The rewrite and its row#
Out: nat-a1 replaces the source 10.1.4.21:40001 with 203.0.113.7:34001 and writes a row. Back:
the mirror answers 203.0.113.7:34001, and nat-a1 uses the row to rewrite the destination to
10.1.4.21:40001.
| Inside address and port | Outside address and port | Destination |
|---|---|---|
10.1.4.21:40001 | 203.0.113.7:34001 | 198.51.100.20:443 |
The port is rewritten too, so many hosts share one outside address, each on its own ports. The
mirror sees one host, 203.0.113.7, and never 10.1.4.21. Both outside addresses here are
documentation addresses (RFC 5737), so they reach nothing real.
An everyday picture: a reception desk sends every letter out under the company’s address, with a reference number. A reply quoting the number goes to the right desk. The picture breaks with a letter that quotes no number: a real desk opens it, a NAT throws it away.
No row, no way in#
A packet from outside with no matching row is dropped: in the figure, the mirror’s Uninvited packet to
203.0.113.7:34002. NAT lets connections out, not in.
To reach a host behind NAT from outside, you need a port forward. It is a fixed row, set by hand, that sends one outside port to one inside host. Two providers’ pages say their NAT gateway accepts no connection opened from outside.
What runs out first#
Ports run out before anything else. A port number is 16 bits, so one outside address has 65,536 ports per protocol. One provider leaves the first 1,024 unused.
| Limit | Value |
|---|---|
| port numbers per address, per protocol | 65,536 |
| one provider, per NAT address | 64,512 TCP ports and 64,512 UDP ports; ports 0 to 1,023 unused |
| the same provider, per VM by default | 64 ports, with its default (static) allocation: 64 simultaneous connections to the same destination address and port; other destinations reuse them |
| another provider’s NAT gateway | 55,000 simultaneous connections per address to each unique destination (address, port, protocol); up to 8 addresses per gateway |
When a VM has used all its ports for one destination, its new connections to that destination fail until old rows expire. The fix is more outside addresses, more ports per VM, or fewer connections (Part 3’s pooling).
A box that changes the packet#
A router that rewrites addresses is a middlebox: a box on the path that does more than a router’s standard job. The end-to-end argument says some functions, such as reliability and security, can be done completely only by the two end hosts.
NAT bends that rule. The address the mirror sees is not the worker’s, and nat-a1 now holds state for every connection. A server behind NAT cannot be reached from outside without a port forward.
When it breaks#
| Case | Description | Check with | Why | Fix |
|---|---|---|---|---|
| one network reachable, another not | the worker reaches worker-2 but not db-b1, or the reverse | ip route get 10.3.2.6 on the worker, then the table of each router on the path | a router on the path has no row for it; or a wrong prefix: with /16, the worker treats all of 10.1.x.x as its own link | add the missing row; fix the prefix length |
| private traffic leaks to the NAT | packets for an unknown 10.x address leave through nat-a1 and die on the internet | ip route get 10.9.9.9 on core-a1 | no row more specific than 0.0.0.0/0 covers them | add the route, or a blackhole route for 10.0.0.0/8 |
a new VM shows 169.254.x.x | the VM has an address but reaches nothing beyond its own link | ip -4 addr on the VM | no DHCP server answered, so the host gave itself a link-local address | make a DHCP server answer on that subnet, then renew the lease |
| connections through the NAT fail under load | new connections time out while old ones work; many workers call the same mirror | the NAT gateway’s port usage and dropped-packet logs | the VM has used all its ports for that destination, or the gateway all its addresses | more NAT addresses or more ports per VM; reuse connections |
| traceroute repeats two hops | the same two routers alternate in the output until the trace gives up | traceroute, then both routers’ tables | a routing loop: each router’s row points to the other; the TTL ends each packet | fix the row that points back |
Remember#
- A router forwards each packet alone: one lookup, TTL minus one, a new checksum, the next link.
- The longest matching prefix wins; the default route
0.0.0.0/0loses to every other match. - A host sends direct inside its subnet, and to its gateway for everything else.
- DHCP lends an address, a gateway and a DNS server for a lease;
169.254.x.xmeans nobody answered. - NAT rewrites the source and keeps a row: replies come in, unsolicited packets do not, and ports run out first.
Sources#
Checked 2026-10-16.
- RFC 791, Internet Protocol: the IPv4 header, fragmentation in 8-byte units, the DF flag, the 68-byte minimum
- RFC 1812, Requirements for IP Version 4 Routers: the most specific route, TTL (sec. 5.3.1), checksum (sec. 5.2.2), no reassembly by routers (sec. 5.2.6), ICMP 3/4
- RFC 1624, Computation of the Internet Checksum via Incremental Update
- RFC 2474, Definition of the Differentiated Services Field, RFC 3168, Explicit Congestion Notification
- RFC 6864, Updated Specification of the IPv4 ID Field
- RFC 1191, Path MTU Discovery
- RFC 1122, Requirements for Internet Hosts: the host sends off-subnet packets to its gateway (sec. 3.3.1)
- RFC 4632, Classless Inter-domain Routing (CIDR): prefix notation and aggregation
- RFC 950, Internet Standard Subnetting Procedure
- RFC 1918, Address Allocation for Private Internets, RFC 6598, Shared Address Space, RFC 6890, Special-Purpose IP Address Registries
- RFC 2131, Dynamic Host Configuration Protocol: the four messages, ports 67 and 68, the xid, the broadcast bit, T1
- RFC 2132, DHCP Options: subnet mask, router, DNS server, lease time
- RFC 3927, Dynamic Configuration of IPv4 Link-Local Addresses
- RFC 3022, Traditional IP Network Address Translator
- RFC 5737, IPv4 Address Blocks Reserved for Documentation
- RFC 3234, Middleboxes: Taxonomy and Issues: the middlebox and the end-to-end principle
- RFC 8200, IPv6 Specification, RFC 4291, IPv6 Addressing Architecture
- Google, IPv6 statistics: the adoption data on 2026-10-10 and 2026-10-13
- ip-route(8), Debian:
ip route get, blackhole and unreachable routes; ip(8), Debian:ip -4 addr - ISC DHCP, Kea, Introduction
- Google Cloud, VPC routes, Subnets, Maximum transmission unit, Cloud NAT overview, Cloud NAT IP addresses and ports, Cloud NAT logs and metrics
- AWS, How route priority works, Route table concepts, Subnet CIDR blocks, NAT gateways, NAT gateway basics
- Alibaba Cloud, VPC route tables