The worker 10.1.4.21 copies a 4 GB table from the database db-b1, 10.3.2.6 port 5432, over a
100 Mbps path, RTT 50 ms. It should take five minutes, but a 64 KB (65,535 bytes) window, far below
the path’s bandwidth-delay product (BDP), holds it to 10.5 Mbps: most of an hour. The network also loses packets on the
way: once the connection is ESTABLISHED, what keeps the bytes in order, and what sets the speed?
Intent: TCP turns a network that loses and reorders packets into one ordered byte stream. It numbers every byte and resends what is not acknowledged. Over a long path, its speed is at most the smaller of two windows, the receiver’s and the sender’s, divided by the round-trip time.
One connection, one header#
IP delivers packets one by one, and it may lose, repeat or reorder them (Part 1). TCP builds an ordered, reliable stream of bytes on top of it. TCP runs only in the two hosts.
Routers forward on the IP header; some also read the ports, to filter, translate or spread flows
(Parts 4, 6 and 8). A TCP connection is named by its two addresses and two ports, the 4-tuple (Part
2’s five values, with the protocol fixed as TCP). Here it is 10.1.4.21:40001 to 10.3.2.6:5432.
The two hosts keep everything else about the connection: what was sent, what arrived, how much room is left. Each segment carries the header fields below.
The figure follows one data segment from db-b1 to the worker. tcpdump -n names its two ends
10.3.2.6.5432 > 10.1.4.21.40001 and the timestamps option TS.
| Field | Size | What it says | Used in |
|---|---|---|---|
| source and destination port | 16 bits each | the two ends of the connection | the 4-tuple: 40001 and 5432 |
| sequence number | 32 bits | the number of the first data byte in this segment | numbering bytes |
| acknowledgment number | 32 bits | the next byte the sender of this segment expects | numbering bytes |
| control bits | 1 bit each: CWR, ECE, URG, ACK, PSH, RST, SYN, FIN, eight in RFC 9293; RFC 9768 (2026) names a ninth, AE | what kind of segment this is; this part uses SYN, ACK, FIN and RST | the handshake, the close |
| window | 16 bits | how many more bytes the sender of this segment can take | flow control |
| options | up to 40 bytes | MSS, window scale, SACK permitted, timestamps | the handshake |
The header is 20 bytes at least. Linux turns timestamps on by default, which adds 12 bytes, so most segments carry 32.
| TCP | UDP | |
|---|---|---|
| gives | an ordered, reliable byte stream; flow and congestion control | single datagrams with ports and a checksum; nothing is resent |
| before data | a handshake, one round trip | nothing |
| header | 20 bytes at least, 32 with timestamps | 8 bytes |
| state | in both hosts, for each connection | none |
| used today by | this copy to db-b1; HTTP/1.1 and HTTP/2 (Part 2); TLS (Part 8) | DNS queries and QUIC, which carries HTTP/3 (Part 2); QUIC does its own resending |
Opening: SYN, SYN-ACK, ACK#
Before any data moves, the worker (10.1.4.21) and db-b1 (10.3.2.6) must agree on starting numbers
and sizes. Three segments do it, and they cost one round trip (RTT), 50 ms here. Part 2 counted this
round trip.
What the two sides agree on#
The worker sends SYN, db-b1 answers SYN-ACK, and the worker sends ACK. Each SYN carries one side’s starting number and its options. Each ACK confirms the other side’s SYN, so both numbers are confirmed after three segments.
| Item | Sent by | Value here | Why |
|---|---|---|---|
| initial sequence number | each side, in its SYN | a 32-bit number from a clock plus a keyed hash of the 4-tuple, so it looks random | an old or forged segment does not fit the new connection |
| MSS, the largest payload in one segment | each side, in its SYN | each offers MSS 1460 from its 1,500-byte link; border-a1 lowers both to 1400 | the tunnel between border-a1 and border-b1 takes about 60 bytes (Part 8) |
| window scale | each side, in its SYN only | window scale 7: the window field counts in units of 2^7 = 128 bytes | the 16-bit field cannot say more than 65,535 bytes |
| SACK permitted | each side, in its SYN | SACK ok on both sides | the receiver may say which later bytes arrived (next section) |
| timestamps | each side, in its SYN, then in every segment | on: Linux’s default | a round-trip sample from every ACK |
With timestamps on, each segment carries 12 bytes less data, 1,388, so the whole packet fits the tunnel’s 1,440; the figures round to 1,400.
Window scale and SACK work only if both SYNs carry them. On the way, each side’s state changes: the worker goes from CLOSED to SYN_SENT to ESTABLISHED. db-b1 goes from LISTEN to SYN_RCVD to ESTABLISHED.
A closed port and a filtered port#
What if nothing listens? Say PostgreSQL on db-b1 is stopped, and the worker sends its SYN to 5432. border-b1 lets it through, since its rules allow 5432 (Part 8).
db-b1’s TCP answers RST at once: nothing listens on 5432. The worker’s connect() fails with
“Connection refused”.
A filtered port is different. The worker sends a SYN to port 5433, and border-b1 drops it by its last rule, “drop everything else”, without a reply (Part 8). The worker cannot tell a dropped SYN from a lost one, so it sends the SYN again when its timer fires.
The standard timer starts at 1 s and doubles each time it fires. Linux 6.5 and later sends the first
five resends 1 s apart, then doubles the gap. Its SYNs leave at 0, 1, 2, 3, 4, 5, 7, 11, 19, 35 and 67
s; after about 131 s, connect() gives up.
| Port on db-b1 | Answer | connect() on the worker | After |
|---|---|---|---|
| open: 5432, PostgreSQL running | SYN-ACK | succeeds | one round trip, 50 ms |
| closed: 5432, PostgreSQL stopped | RST | “Connection refused” | one round trip |
| filtered: 5433, border-b1 drops it | nothing | “Connection timed out” | about 131 s on Linux 6.5 and later |
So an instant refusal means a TCP stack answered RST, usually a host where nothing listens. A hang of about two minutes means the SYN or its answer is dropped. A firewall on the path, a host that is down or a busy server all do that.
Numbering bytes, acknowledging the next one#
Once the connection is open, db-b1 (10.3.2.6) sends the table to the worker (10.1.4.21) as one
stream of bytes, cut into segments of at most 1,400 bytes. Every byte has a number, and every ACK names the next byte the worker expects. That
is how the worker notices a gap and how db-b1 learns what to send again.
Sequence numbers and cumulative ACKs#
A segment carries the number of its first byte. The figure counts from 1, as tcpdump prints
relative numbers; on the wire the count starts from each side’s initial sequence number. db-b1 sends
six segments: 1-1400, 1401-2800, 2801-4200, 4201-5600, 5601-7000 and 7001-8400.
ACK 2801 says “I have every byte before 2801”. One ACK covers everything before it, so a lost ACK does no harm once a later one arrives. This is a cumulative ACK.
The worker need not ACK every segment. It should still send one for at least every second full-sized segment.
An everyday picture: a long letter sent by post, one numbered page per envelope. The reader writes back “send page 3 next”. The picture breaks in one place: TCP numbers bytes, not pages, so the reply says “byte 2801”.
A gap and three duplicate ACKs#
The segment with bytes 2801-4200 is lost. Bytes 4201-5600 still arrive, and the worker keeps them, as the standard says it should. But its ACK can only name the first missing byte, so it repeats ACK 2801.
A repeated ACK is a duplicate ACK, and the worker sends one at once for every segment that arrives early. After three duplicate ACKs, db-b1 sends 2801-4200 again without waiting for its timer. This is fast retransmit.
The resent segment fills the gap, and the worker jumps to ACK 8401: it already held everything after the gap.
SACK: which later bytes arrived#
Each duplicate ACK also carries a SACK block: “I also have 4201-5600”, then 4201-7000 and 4201-8400. So db-b1 resends only the gap, not every byte after it. Both sides agreed to SACK in their SYNs, and Linux turns it on by default.
Linux decides that a segment is lost by time, with RACK. A segment counts as lost once one sent after it is acknowledged and a short wait for reordering has passed. Three duplicate ACKs still count as a sign of loss.
When nothing comes back: the timer#
Duplicate ACKs need later segments to arrive. When nothing comes back from the worker (10.1.4.21),
because the last segment was lost or the path is down, only the retransmission timer is left. db-b1 starts it when it sends, and
sends again when it fires.
How long the sender waits#
The timer is the retransmission timeout (RTO). TCP measures the RTT of its segments and keeps two running values: a smoothed RTT and how much the samples vary. Each new sample moves the first by 1/8 of the difference and the second by 1/4.
The standard sets the RTO to the smoothed RTT plus four times the variation, and at least 1 s. Linux uses a floor of 200 ms instead. Its RTO is the smoothed RTT plus a variation term of at least 200 ms.
In the figure, ACK 1401 comes back 50 ms after bytes 1-1400. Then bytes 1401-2800 are lost twice.
At RTT 50 ms, then, the RTO is about 250 ms (Part 7 uses this number). Before the first sample, the RTO is 1 s. That is why the handshake’s first resend comes after 1 s.
Why the wait doubles#
Each time the standard’s timer fires without an answer, the RTO doubles: 250 ms, 500 ms, 1 s, and so on. A path that answers nothing is often full or down, and sending faster would make it worse. Linux caps the RTO at 120 s.
Linux may first send one probe, a tail loss probe (RFC 8985), before the timer itself fires.
After 15 resends of the same segment, Linux gives up and closes the connection. Its documentation puts that at no less than 924.6 s, about 15 minutes. If ACKs come back sooner, new RTT samples bring the RTO down to about 250 ms again.
After a timeout, db-b1 also starts sending again from one segment (Congestion control, below). ss -tio dst 10.3.2.6 shows rtt (the smoothed value and its variation), rto, and in timer:(on,...) the
number of resends.
Flow control: the receiver’s window#
The worker’s TCP (10.1.4.21) keeps arrived bytes in a receive buffer until the copy program reads
them. Every ACK says how much room is left, the receive window (rwnd). db-b1 (10.3.2.6) never has
more unacknowledged bytes in flight than that.
The buffer and the zero window#
The figure’s buffer holds 11,200 bytes, eight segments, so the numbers stay small: real buffers start at 128 KB and grow to megabytes. After three unread segments, the worker says window 7000; after six, window 2800; after eight, window 0.
An everyday picture: a sink with the tap running faster than the drain. The picture breaks in one place: a sink overflows, while TCP tells the sender to stop before the buffer is full.
At window 0, db-b1 stops, but it keeps sending a small probe now and then. The worker answers each one, so the window that opens is never missed. Without the probes, one lost window update would leave both sides waiting forever.
The window on a long path#
db-b1 can send one window, then must wait for ACKs, which take one round trip to come back. So the copy runs at most at window / RTT.
| Window | Segments of about 1,400 bytes | Throughput at RTT 50 ms | Time for 4 GB |
|---|---|---|---|
| 65,535 bytes, the most without window scale | 46 | 10.5 Mbps | about 51 minutes |
| 625,000 bytes, the bandwidth-delay product | 446 | 100 Mbps, the rate of the path | 320 s, about five minutes |
The bandwidth-delay product (BDP) is the rate of the path times the RTT: 100 Mbps × 50 ms is 625,000 bytes. It is how many bytes the path holds in flight. A smaller window leaves the wire empty for part of every round trip, as in the first figure.
Window scale, agreed in the SYNs, lifts the limit. With a shift of 7, the 16-bit field can say up to about 8 MB; the largest shift, 14, allows 1 GiB.
Linux also grows each connection’s receive buffer by itself, up to the maximum in tcp_rmem. The
kernel’s documentation puts that default between 131,072 bytes and 32 MB, depending on memory. On my-pi
the maximum reads 33554432 bytes, 32 MB.
If a middlebox strips the window scale option from a SYN, neither side scales. The copy then stays at 10.5 Mbps, however large the buffers are.
Congestion control: the sender’s window#
The receive window protects the worker (10.1.4.21); nothing yet protects the routers in between,
border-a1 and border-b1. db-b1 (10.3.2.6) keeps a second limit, the congestion window (cwnd), and sends at most min(cwnd, rwnd) unacknowledged
bytes. Here the worker’s buffer has grown well past 625,000 bytes, so cwnd sets the pace.
Slow start#
Linux starts cwnd at 10 segments. It adds one segment for each segment acknowledged, so cwnd
doubles every round trip. There is no fixed ceiling at the start: on a first connection to db-b1, slow
start runs until the first loss.
For the copy, cwnd goes 10, 20, 40, 80, 160, 320, 640. After six round trips, 300 ms, it passes the
446 segments of the BDP. “Slow” is only the name.
Past the BDP, the extra segments do not go faster. They wait in the queue of the narrowest link, here at border-b1, where the tunnel toward network A runs at 100 Mbps. CUBIC’s slow start may also stop earlier, when it sees the RTT grow.
After a loss#
Reno and CUBIC read a loss as a sign that a queue overflowed somewhere. After three duplicate ACKs,
db-b1 resends the lost segment and cuts cwnd. CUBIC cuts it to 0.7 of its size, Reno to half: here 640 × 0.7 is 448.
Then cwnd grows again, more slowly. CUBIC grows fast at first, flattens near the size where the
last loss happened, then speeds up again to look for more room.
After a timeout, cwnd falls to one segment and slow start begins again. Part 7 shows what that
costs on a radio link.
Which algorithm your host runs#
CUBIC is the default in the Linux, Windows and Apple stacks. sysctl net.ipv4.tcp_congestion_control
prints the Linux default, and my-pi prints cubic. ss -ti names the algorithm for each connection.
| Algorithm | What it reads | On a loss | Status today |
|---|---|---|---|
| Reno | loss | halves cwnd | RFC 5681; always available on Linux |
| CUBIC | loss, and the time since the last loss | cuts cwnd to 0.7 | RFC 9438 (2023); the default in Linux, Windows and Apple |
| BBR | the measured delivery rate, RTT and loss rate, as a model of the path | loss is one input to the model | Google uses it for google.com and YouTube (its 2017 post); the spec, version 3, is an Internet-Draft dated 2026-07-06 |
ECN lets a router mark a packet instead of dropping it, and the receiver echoes the mark to the sender. Linux accepts ECN when the other side asks for it, but does not ask by default.
Closing: FIN, ACK and TIME_WAIT#
When the copy is done, each side closes its own direction with a FIN, and the other side acknowledges
it. That is four segments, or three when db-b1’s ACK rides on its own FIN. The side that closes first,
here the worker, then waits in TIME_WAIT, holding the 4-tuple 10.1.4.21:40001 to
10.3.2.6:5432.
Why the worker waits#
The worker sends FIN and moves to FIN_WAIT_1, then to FIN_WAIT_2 on db-b1’s ACK. db-b1 moves to CLOSE_WAIT, sends its own FIN, waits in LAST_ACK, and ends CLOSED. The worker answers that FIN and enters TIME_WAIT.
The wait has two jobs. A late segment of this connection must not be taken as part of a new one with the same four numbers. And if the last ACK is lost, db-b1 resends its FIN, and the worker can still answer.
The standard asks for twice the longest life of a segment in the network, taken as 2 minutes: 4 minutes in all. Linux waits 60 s, a fixed value in the kernel with no setting. RST ends the connection at once for the side that receives it: no FIN and no TIME_WAIT there.
When TIME_WAIT runs out of ports#
A client that opens a new connection for each query leaves one TIME_WAIT behind per query. From one
address to 10.3.2.6:5432, the worker has 28,232 source ports: 32768-60999, the Linux default range.
Each port stays held for 60 s, so the worker can open about 470 new connections a second (28,232 / 60).
Past that, connect() fails with “Cannot assign requested address”: no port is free.
The fix is a connection pool: a few connections kept open and reused. It is Part 4’s fix for NAT ports too.
In a VPC: idle connections are forgotten#
A cloud NAT or firewall keeps a row for each connection it tracks (Part 8), and removes a row that stays idle. What breaks then depends on which one forgot.
| Provider | What tracks the connection | An idle TCP connection is forgotten after | What is lost | Page |
|---|---|---|---|---|
| Google Cloud | VPC firewall rules | the state stays active only while at least one packet passes every 10 minutes | packets from the far side, until the client sends again | VPC firewall rules |
| Google Cloud | Cloud NAT | 1,200 s (20 minutes) for an established connection, by default | the connection | Tune NAT configuration |
| AWS | security groups | 432,000 s (5 days) by default; 350 s on most Nitro v6 instance types | packets from the far side, until the client sends again | Security group connection tracking |
A NAT that forgets a mapping breaks the connection for good: the reply has no mapping back. A stateful firewall that forgets a row drops only what the row alone allowed, such as a packet from the server after the pause. The client’s next packet matches its allow rule and is tracked again.
AWS advises TCP keepalives more often than every 5 minutes for long-lived connections. Linux sends its first keepalive only after 7,200 s idle, two hours, and only on sockets that ask for keepalives.
So a pooled connection through Cloud NAT, idle for 30 minutes, is broken without either end knowing: its mapping went after 1,200 s. Set keepalives below the shortest idle timeout on the path, in
the pool’s or the driver’s settings, or with TCP_KEEPIDLE on Linux.
When it breaks#
| Case | Description | Check with | Why | Fix |
|---|---|---|---|---|
| connection refused | connect() to db-b1 fails at once with “Connection refused” | ss -ltn on db-b1: no socket listens on 5432 | db-b1’s TCP answered the SYN with RST: PostgreSQL is stopped, or the port is wrong | start the service, or fix the port in the connection string |
| a connect that hangs | the client waits about two minutes, then reports “Connection timed out” | tcpdump -n 'host 10.3.2.6 and tcp port 5432' on the worker: SYNs, 1 s apart and then further, and no answer | the SYN or its answer is dropped: a firewall on the path, a host that is down, a server too busy; Linux resends the SYN for about 131 s | allow TCP 5432 from 10.1.4.0/24 on that firewall (Part 8) |
| a copy at about 10 Mbps on a 100 Mbps path | the 4 GB copy takes about 51 minutes, with no losses | ss -ti dst 10.3.2.6: wscale, rcv_space, cwnd | the window caps the copy at window / RTT: window scale stripped from a SYN, or a fixed small buffer | let the option through; do not set SO_RCVBUF by hand, since it turns off the buffer’s automatic growth |
| a copy that stalls in bursts | the copy stops for a second or more, then goes on | ss -tio dst 10.3.2.6: rto, and the resend count in timer:(on,...) | segments are lost, the timer fires, and cwnd starts again from one segment | find the loss on the path: a full queue or a failing link (Part 1) |
| no free ports | after a burst of short queries, new connections fail with “Cannot assign requested address” | ss -Htan state time-wait dst 10.3.2.6 | wc -l close to 28,232 | every closed connection holds its port for 60 s in TIME_WAIT | a connection pool |
| a pooled connection that hangs after a quiet half hour | the first query after a pause hangs, then fails | the idle timeouts of the NAT and the firewall, against the pool’s idle time; ss -tio shows timer:(keepalive,...) when keepalives are on | a NAT forgot the connection’s mapping (Cloud NAT: 1,200 s), so the reply has no way back | keepalives below the shortest idle timeout, under 5 minutes on AWS |
Remember#
- TCP numbers bytes, and an ACK names the next byte expected: three duplicate ACKs resend a gap at once.
- A silent path waits for the timer: about the RTT plus 200 ms on Linux, doubling each time it fires.
- Throughput is at most window / RTT: 65,535 bytes over 50 ms is 10.5 Mbps.
- The sender keeps a second window,
cwnd: slow start doubles it, and a loss cuts it to 0.7 with CUBIC. - A closed connection holds its port for 60 s, and an idle one may be forgotten: pool connections, set keepalives.
Sources#
Checked 2026-10-16.
- RFC 9293, Transmission Control Protocol (TCP): the header and its eight control bits (sec. 3.1), the states (sec. 3.3.2), initial sequence numbers (sec. 3.4.1), the 2-minute MSL (sec. 3.4.2), the reset (sec. 3.5.2), TIME-WAIT (sec. 3.6.1), keepalives (sec. 3.8.4), zero-window probing and when to ACK (sec. 3.8.6), keeping early segments (sec. 3.10.7); obsoletes RFC 793
- RFC 6298, Computing TCP’s Retransmission Timer: the RTO formula and its gains (sec. 2), backing off (sec. 5)
- RFC 5681, TCP Congestion Control: slow start, ssthresh and the loss window (sec. 3.1), fast retransmit (sec. 3.2)
- RFC 2018, TCP Selective Acknowledgment Options
- RFC 8985, The RACK-TLP Loss Detection Algorithm for TCP
- RFC 7323, TCP Extensions for High Performance: the timestamps option (sec. 1.3), window scale (sec. 2.2, 2.3)
- RFC 6691, TCP Options and Maximum Segment Size
- RFC 6928, Increasing TCP’s Initial Window
- RFC 9438, CUBIC for Fast and Long-Distance Networks: the default stacks (sec. 1), the window curve (sec. 4.2-4.5), the 0.7 decrease (sec. 4.6), slow start (sec. 4.10)
- RFC 3168, The Addition of Explicit Congestion Notification (ECN) to IP
- RFC 768, User Datagram Protocol, RFC 9000, QUIC: A UDP-Based Multiplexed and Secure Transport
- IETF, BBR Congestion Control, draft-ietf-ccwg-bbr-06, Google Cloud blog, TCP BBR congestion control comes to GCP
- Linux kernel, IP Sysctl:
tcp_syn_retries,tcp_syn_linear_timeouts(Linux 6.5 and later), the RTO floor and cap,tcp_retries2,tcp_rmem,tcp_moderate_rcvbuf,tcp_recovery,tcp_ecn,ip_local_port_range - Linux v6.18, include/net/tcp.h:
TCP_INIT_CWND,TCP_TIMEWAIT_LEN,TCP_RTO_MAX,TCP_TIMEOUT_INIT; net/ipv4/Kconfig: CUBIC as the default - tcp(7):
tcp_keepalive_time,TCP_KEEPIDLE; connect(2):ECONNREFUSED,EADDRNOTAVAIL,ETIMEDOUT - ss(8), Debian, tcpdump(8), Debian
- Google Cloud, VPC firewall rules: connection tracking, Tune NAT configuration: the NAT timeouts
- AWS, Amazon EC2 security group connection tracking