The worker 10.1.4.21 in network A copies a 4 GB table from the database, db-b1, at 10.3.2.6 in
network B. The data crosses border-b1, a 100 Mbps tunnel, border-a1, core-a1 and gw-a1 10.1.4.1, and
both hosts have 10 Gbps network cards and idle CPUs. The copy takes 320 s, about five minutes, and nothing
is broken: where does the time go?
Intent: every hop charges four delays, and distance sets a floor that no faster link lowers. A queue grows sharply as its link nears full load, then drops packets. A copy runs at the rate of its narrowest link, and a traceroute shows which of these you are paying.
Four delays at every hop#
Where does one packet of the copy lose time? It leaves db-b1 10.3.2.6, crosses border-b1, the 100 Mbps tunnel,
border-a1, core-a1 and gw-a1 10.1.4.1, and reaches the worker 10.1.4.21. Look inside border-b1,
where the packet meets the tunnel.
The tunnel is Part 8’s IPsec tunnel, between border-b1’s outside address 203.0.113.20 and
border-a1’s 198.51.100.10, over providers P1 and P2. Part 5 also draws a direct peering link
between A and B. In this part, all traffic between them takes the tunnel.
In the figure, packet 4 of the copy reaches border-b1 while packets 1 to 3 still wait for the tunnel. It pays four delays, one after the other:
| Delay | What sets it | On the copy’s path | What lowers it |
|---|---|---|---|
| processing | the router’s own work: read the header, pick the outgoing link, check the header’s checksum | no figure; it depends on the router (a gateway that encrypts every packet does more work) | a faster router |
| queuing | the packets already waiting for the same link | zero when the queue is empty; different for every packet (next sections) | less traffic, or a faster link |
transmission, L / R | the packet’s length L in bits over the link’s rate R | 12,000 bits at 100 Mbps: 0.12 ms on the tunnel; 1.2 µs on a 10 Gbps LAN link | a faster link |
propagation, d / s | the distance d over the signal’s speed s in the medium | 25 ms over the tunnel’s 5,000 km | only a shorter path |
This part uses 1,500-byte packets (12,000 bits) as a round number. That is about 1,440 bytes of the copy plus the tunnel’s own 60 bytes (Parts 3 and 8).
Their sum is the nodal delay: the time one node adds. Every hop adds its own, from db-b1 to the worker. On this path one term rules: 25 ms of propagation against 0.12 ms of transmission.
Transmission is not propagation#
Transmission delay is the time to put all the bits of a packet onto the wire. It depends on the packet’s length and the link’s rate, never on distance. Propagation delay is the time one bit takes to travel the wire, and it depends on distance alone.
An everyday picture: a delivery truck. Loading time depends on the boxes and the crew, not on where the truck goes. Driving time depends on the road, not on the load.
The picture breaks at the start. A truck leaves when it is full, but a packet’s first bit sets off as soon as it is on the wire.
So a faster link shortens transmission and leaves propagation alone. A tunnel ten times faster cuts 0.12 ms to 0.012 ms per packet; the 25 ms stays.
Distance sets a floor#
Light in vacuum covers 299,792 km a second. In telecom fibre it travels at about two thirds of that. The rule of thumb is 200,000 km a second: 1 ms for every 200 km, one way.
| Fibre between the two ends | One way | Round trip, at least |
|---|---|---|
| 50 km | 0.25 ms | 0.5 ms |
| 1,000 km | 5 ms | 10 ms |
| 5,000 km, networks A and B | 25 ms | 50 ms |
| 10,000 km | 50 ms | 100 ms |
The round-trip time (RTT) is the time from sending a packet to receiving its answer. Between A and B it is 50 ms, almost all of it propagation. No faster link, card or CPU lowers it; only a shorter path does.
A ping far below this floor means something nearer answered: an anycast address, served from a closer site (Part 5). The floor also multiplies. A database client that needs 40 round trips before the first row waits 40 x 50 ms = 2 s, whatever the link rate.
Store and forward: the whole packet first#
In the basic model, a router sends a packet on only after its last bit has arrived. Suppose three
packets of the copy from db-b1 10.3.2.6 for 10.1.4.21 reach border-a1 back to back, as a burst
does. What does the wait cost as they go on to core-a1 and gw-a1 10.1.4.1 over two 10 Gbps links of
network A, link 1 and link 2?
A router that works this way is store-and-forward: it stores the whole packet, checks it, then
forwards it. One packet over one link takes L / R. Over N links of equal rate it takes N L / R,
because every router on the way waits for the whole packet.
Packets in a row overlap. While core-a1 sends packet 1 to gw-a1, border-a1 already sends packet 2 to
core-a1. So P packets over N links take (N + P - 1) L / R.
Here that is (2 + 3 - 1) x 1.2 µs = 4.8 µs, against 6 x 1.2 µs = 7.2 µs without the overlap. The
figure steps through 1.2, 2.4, 3.6 and 4.8 µs.
Propagation is left out, because these links are short.
On the 10 Gbps LAN links this costs microseconds per hop, too little to notice. On the tunnel it is 0.12 ms per packet, since border-a1 must hold each whole packet before it sends it on. Next to the 25 ms of distance, even that is small.
Packets that crossed the tunnel come 0.12 ms apart, so on these links they do not overlap: the overlap needs a burst.
The queue near full load#
Packets reach border-b1’s port to the 100 Mbps tunnel from many senders, in bursts. The ones that find the tunnel busy wait in its output queue, the buffer in front of the link to border-a1. How long they wait depends on how close the tunnel runs to full load.
Traffic intensity#
Traffic intensity is the load on a link: La / R. Here a is the average number of packets arriving per
second, L the bits per packet and R the link’s rate. The tunnel sends 8,333 packets of 1,500 bytes a
second, so 8,333 arriving a second is an intensity of 1.
Traffic intensity La / R | What the queue does |
|---|---|
| near 0 | an arriving packet rarely finds another one waiting; the wait is near zero |
| near 1 | bursts arrive faster than the link drains them, and the average queue gets longer and longer |
| 1 or more | packets arrive at least as fast as they can leave, on average; the queue grows until it is full |
How fast the wait grows#
For random arrivals and random packet lengths, the simplest queue model is M/M/1. Its average wait is
ρ / (1 - ρ) packet times, where ρ is the intensity. If every packet had the same length, the wait
would be half of that.
The shape is the same: it grows sharply near 1. In the table, one packet time on the tunnel is 0.12 ms.
| Load | Packets a second | Average wait, in packet times | Average wait |
|---|---|---|---|
| 0.5 | 4,167 | 1 | 0.12 ms |
| 0.8 | 6,667 | 4 | 0.48 ms |
| 0.95 | 7,917 | 19 | 2.28 ms |
From 0.8 to 0.95 the load grows by about a fifth, and the wait almost five times. The model is a shape, not a forecast: real traffic arrives in bursts, not at random.
An everyday picture: a supermarket checkout. At half load you rarely wait; when the cashier is busy 95 % of the time, a line stands there most of the day. The picture breaks when the line is full: a router does not let a packet wait outside, it throws it away.
When the queue is full#
The buffer holds a fixed number of packets. When it is full, the router drops the packet that arrives: tail drop.
In the figure, two senders feed the queue: db-b1 10.3.2.6, with the copy for the worker 10.1.4.21, and
the api 10.3.2.5, answering worker-2 10.1.4.22. Both workers sit behind border-a1, core-a1 and gw-a1.
Over 1, they send 9,000 packets a second into a tunnel that takes 8,333.
Loss is the normal end of a full queue, not a fault. TCP notices the missing data and sends the segment again (Part 3). Active queue management (AQM) drops or marks packets before the buffer is full, so senders slow down early (RFC 7567).
A big buffer in front of a slow link stays full and adds delay without adding throughput. This is bufferbloat, worst at the consumer edge (RFC 8289). Each packet waiting for the tunnel adds 0.12 ms, so 1,000 waiting packets add 120 ms.
On Debian, fq_codel is the default queue discipline (qdisc): the code that orders and drops packets
in an interface’s queue. The package linux-sysctl-defaults sets
net.core.default_qdisc = fq_codel, with the comment “to fight bufferbloat” (observed on a Debian 13
host).
fq_codel sorts packets into flows and shares the link fairly between them. It starts to drop or mark
packets when the queue has not fallen below 5 ms for 100 ms: its default target and interval.
tc -s qdisc show dev eth0 prints the queue’s drops and its backlog.
Why backbones run half empty#
In 2002, RFC 3439 described network engineers keeping backbones below 50 % utilisation. It called the spare half protection capacity, which gave delay close to the speed of light and loss near zero.
So the queue you meet is usually at the edge of a network, where a fast link meets a slow one, as at border-b1.
Throughput is the narrowest link#
Throughput is the rate at which the worker 10.1.4.21 receives the copy from db-b1 10.3.2.6, in
bits a second. The copy crosses one 100 Mbps tunnel, from border-b1 to border-a1, and 10 Gbps LAN links
everywhere else, through core-a1 and gw-a1 10.1.4.1. Which of them sets its pace?
A copy goes no faster than the slowest link on its path. Its throughput is min(R1, ..., RN), the
rate of the bottleneck link. A copy of F bits then takes about F / min(R).
The table is 4 GB, or 32 Gbit, so the copy takes 32 Gbit / 100 Mbps = 320 s, about five minutes. The 10 Gbps cards change nothing. Headers and the transport’s own limits only add to that time (Part 3).
A link shared by several copies gives each a share. If M copies take equal turns on a link of rate R,
each gets about R / M. With a second copy to worker-2 10.1.4.22, each gets about 50 Mbps and takes
640 s.
In a VPC: the candidates, in order#
In a cloud, the limits the provider sets come first. Check these in this order.
| Candidate | Its limit |
|---|---|
| a VPN tunnel | AWS: up to 1.25 Gbps per standard tunnel. Google Cloud: 250,000 packets a second per tunnel, in and out together, which is 1 to 3 Gbps depending on packet size |
| a per-VM or per-flow cap | Google Cloud: egress of generally 2 Gbps per vCPU, for each VM (traffic inside the VPC network, tunnels included), and 3 Gbps per flow to the internet (a VPN tunnel has its own limit, above; 1 Gbps on H3 and H4D VMs). AWS: 5 Gbps for one flow between instances that are not in the same cluster placement group |
| a NAT gateway | its own limits and ports (Part 4) |
| a shared uplink at the busy hour | about R / M, when M copies share a link of rate R |
| the TCP window over a long path | not a link, but the same cap: a sender has at most one window in flight per round trip. 65,535 bytes every 50 ms is about 10.5 Mbps (Part 3) |
Reading a traceroute#
A traceroute lists the routers on a path and how long each took to answer. It is the quickest way to see which delay you are paying. The worker runs one to db-b1, and border-a1 runs one under the tunnel.
Traceroute sends probes whose TTL lets them go one hop further each round. The router where a probe’s TTL runs out drops it and answers (Part 5 draws how). Traceroute sends three probes per hop by default.
On Linux the probes are UDP by default. The trace ends when the destination answers port unreachable, ICMP type 3, code 3. Each time on a line is a round trip: the four delays out and back, plus the time the router takes to answer.
Through the tunnel#
The worker 10.1.4.21 runs traceroute 10.3.2.6:
| Line | Who answers | Round trips | What it shows |
|---|---|---|---|
| 1 | gw-a1 10.1.4.1 | 0.4, 0.3, 0.5 ms | the worker’s own subnet |
| 2 | core-a1 | 0.5, 0.6, 0.5 ms | inside network A |
| 3 | border-a1 | 0.6, 0.7, 0.6 ms | the tunnel’s near end |
| 4 | border-b1 | 50.3, 50.1, 50.5 ms | the tunnel’s far end: 5,000 km there and back |
| 5 | db-b1 10.3.2.6 | 50.5, 50.7, 50.4 ms | the destination: port unreachable ends the trace |
The tunnel shows as one hop, between line 3 and line 4.
Each of border-a1 and border-b1 lowers the inner packet’s TTL once, as every router does when it forwards. The tunnel adds no decrement of its own. The routers of P1 and P2 only touch the outer header’s TTL, so they never appear (RFC 4301, RFC 2003).
Four patterns#
Under the tunnel, border-a1 runs traceroute 203.0.113.20 from its outside address 198.51.100.10:
| Line | Who answers | Round trips |
|---|---|---|
| 1 | p1-edge | 0.9, 1.0, 0.8 ms |
| 2 | p1-core | * * * |
| 3 | p2-edge | 49.8, 50.0, 49.7 ms |
| 4 | p2-core | 60.4, 58.9, 61.2 ms |
| 5 | border-b1 | 50.1, 50.3, 50.0 ms |
The two traces hold the four patterns you will meet most:
| Pattern | Here | What it means |
|---|---|---|
| several hops under a few ms | lines 1 to 3 of the worker’s trace | inside your own network |
| a jump that stays | line 4 of the worker’s trace; line 3 under the tunnel | a long link: propagation, not a fault |
| a later hop faster than an earlier one | p2-core about 60 ms, then border-b1 50.1 ms | the earlier router was slow to answer; the path is fine |
* * *, with later hops answering | p1-core | a router that does not answer probes; packets still pass through it |
Answering a probe is extra work for a router. Routers may limit how many such ICMP replies they send (RFC 1812 says they should be able to). So a slow or silent hop says little about the traffic that passes through it.
mtr repeats the probes and prints the loss and the times per hop. Loss at one hop that the later hops
do not show is usually that router not answering. Loss that carries on to the last hop usually means
packets are really lost, on the way out or back.
Layers: who reads which header#
Every packet of the copy carries several headers, one per layer. Each device on the path reads only the headers it needs. Which ones, and what does that tell you when something fails?
A protocol fixes the format and the order of the messages two sides exchange. It also fixes what each side does when a message arrives, or when a timer runs out.
The layers and their units#
| Layer, five-layer teaching model | RFC 1122’s layer | Unit | On the copy |
|---|---|---|---|
| application | application | message | the rows of the table, in the database’s own protocol |
| transport | transport | segment | TCP header: from port 5432 to port 40001 |
| network | internet | datagram, or packet | IP header: from 10.3.2.6 to 10.1.4.21 |
| link | link | frame | a frame header for one link, such as Ethernet (Part 6) |
| physical | part of link | bits | the signal on the copper or the fibre |
RFC 1122, the standard for Internet hosts, names four layers, with the physical one folded into the link. It also calls the transport’s unit a message.
This series keeps “message” for the application’s data. It uses “packet” for what the IP layer hands to the link, as in RFC 1122.
At db-b1, each layer wraps what it gets from above in its own header. This is encapsulation. At the worker, each layer takes its header off and passes the rest up.
IP sends each packet on its own and adds no reliability or ordering (RFC 791). A packet may be lost, delivered twice or arrive out of order, and TCP turns that into one ordered stream (Part 3).
Who reads what#
| Device | Reads | Taught in |
|---|---|---|
| a layer-2 switch, such as sw-a1 | the frame header (a layer-3 switch reads the IP header too) | Part 6 |
a router: border-b1, core-a1, gw-a1 10.1.4.1 | the IP header; it puts a new frame header on for each link | Part 4 |
| a NAT gateway or a firewall | the IP header and the ports in the TCP or UDP header | Parts 4 and 8 |
| the two hosts | every header | this part |
In the figure, border-b1 reads 10.1.4.21, picks border-a1 as the next hop and puts a new frame
header on. It also wraps the packet for the tunnel; that wrapping is Part 8’s.
Place a fault by its layer#
If ping to db-b1 answers but port 5432 does not, the path works for ICMP. Look at the transport: a firewall or security group that filters the port, or the database not listening. A switch or a plain router cannot refuse one port.
If nothing answers at all, look at the network layer or below.
Ping uses ICMP, and a firewall may drop it. So a silent ping alone proves nothing; try the port as well (Part 5).
When it breaks#
| Case | Description | Check with | Why | Fix |
|---|---|---|---|---|
| slow across regions | queries between A and B are slow; the same queries inside one network are fast | ping 10.3.2.6, against the distance table | the propagation floor: 1 ms per 200 km, paid on every round trip | fewer round trips per query; move the client closer to the data |
| big copies slow | a 4 GB copy takes 320 s while small queries feel fine | F / R with the narrowest link; the tunnel’s limit and the per-flow caps | the bottleneck link: 100 Mbps here, whatever the cards do | a faster tunnel, a second tunnel with ECMP and several flows, or a path without the tunnel |
| slow at the busy hour | the same copy takes longer at the same hour every day | tc -s qdisc show dev eth0, the interface drop counters | a queue near La / R = 1: the wait grows sharply, then packets drop | more capacity, or move the job to a quiet hour |
| traceroute jumps once and stays | one hop adds about 50 ms, and every later hop keeps it | the distance table | a long link: propagation, not a fault | nothing |
* * * in the middle | one hop shows stars while the later hops answer | mtr, for loss that carries on to the last hop | a router that does not answer probes or limits its replies | nothing, if the last hop shows no loss |
| delay climbs under load | ping times on an edge link rise while a copy runs | ping during the copy, then tc -s qdisc | bufferbloat: a big buffer in front of a slow link stays full | an AQM queue such as fq_codel at the slow link |
Remember#
- Every hop charges four delays: processing, queuing, transmission (
L / R) and propagation (d / s). - Distance sets a floor of 1 ms per 200 km one way; no faster link lowers it.
- The queue grows sharply as load nears 1, and a full queue drops packets, which TCP sends again.
- A copy runs at its narrowest link: 4 GB over 100 Mbps is 320 s, whatever the cards do.
- Read a traceroute for jumps that stay; a slow or silent hop alone is not a fault.
Sources#
Checked 2026-10-16.
- RFC 1122, Requirements for Internet Hosts: the four layers (sec. 1.1.3); segment, message, datagram, packet, frame (sec. 1.3.3)
- RFC 791, Internet Protocol: no reliability or sequencing in IP (sec. 1.2)
- RFC 792, Internet Control Message Protocol: time exceeded (type 11), port unreachable (type 3, code 3)
- RFC 1812, Requirements for IP Version 4 Routers: rate limits on ICMP replies (sec. 4.3.2.8, Rate Limiting)
- RFC 2003, IP Encapsulation within IP: the inner TTL at the tunnel ends (sec. 3.1)
- RFC 4301, Security Architecture for the Internet Protocol: the inner TTL in tunnel mode (sec. 5.1.2.1, the TTL row)
- RFC 7567, IETF Recommendations Regarding Active Queue Management: tail drop and AQM (sec. 2)
- RFC 8289, Controlled Delay Active Queue Management, Experimental: bufferbloat (sec. 1)
- RFC 3439, Some Internet Architectural Guidelines and Philosophy, Informational, 2002: backbones below 50 % as protection capacity (sec. 6)
- RFC 9293, Transmission Control Protocol (TCP): lost segments are sent again (Part 3)
- NIST, speed of light in vacuum
- Wikipedia, Optical fiber: the rule of thumb of 200,000 km a second in fibre
- Wikipedia, M/M/1 queue: the average wait; M/D/1 queue: half of it for packets of one length
- traceroute(8), Debian, mtr(8), Debian, ping(8), Debian
- tc(8), Debian, tc-fq_codel(8), Debian
- AWS, Site-to-Site VPN quotas, Amazon EC2 instance network bandwidth
- Google Cloud, Cloud VPN overview, Network bandwidth