The worker 10.1.4.21 sends an IPv4 packet to db-b1 at 10.3.2.6: through sw-a1 to gw-a1 10.1.4.1, then
over a second link to core-a1. The IP destination does not change on the way; the TTL does, and the checksum
that depends on it (Part 4). Yet each link delivers only to a hardware address, a MAC such as
00:00:5e:00:53:01: how does the packet cross one link?
Intent: IP picks the next router; the link moves the packet there inside a frame addressed by MAC. ARP finds that MAC, and a switch learns which port leads to it. A data center connects many such links so that every pair of racks has several equal paths.
A frame carries the packet over one link#
A frame is the envelope one link puts around a packet, addressed to the next device only. Here a link
is everything a frame crosses without a router: the cable of the worker 10.1.4.21, sw-a1, and the
cable of gw-a1 10.1.4.1. The frame
ends with a check value, so the card at the end of a cable can tell when the cable damaged it.
What a frame holds#
The anchor figure showed the idea: one frame per link. gw-a1 throws away the worker’s frame and
builds a new one from its core side, 00:00:5e:00:53:11, to core-a1 at 00:00:5e:00:53:12. The
packet inside keeps its addresses.
| Field | Bytes | Holds | On the worker’s frame |
|---|---|---|---|
| destination MAC | 6 | the next device on this link | gw-a1, 00:00:5e:00:53:01 |
| source MAC | 6 | the sender on this link | the worker, 00:00:5e:00:53:21 |
| type | 2 | what the payload is: 0800 IPv4, 86DD IPv6, 0806 ARP, 8100 a VLAN tag follows | 0800 |
| payload | 46 to 1,500 | the IP packet; a shorter one is padded to 46 | the packet for 10.3.2.6 |
| frame check | 4 | a CRC (cyclic redundancy check) computed over the frame | checked by gw-a1 on arrival |
The 1,500-byte limit on the payload is the MTU of Part 4. A type value of 05DC (1,500) or less is
read as a length instead, from an older way of framing the same field.
An everyday picture: a frame is a courier’s label for one leg of a trip. Each depot puts on a new label, and the address on the parcel stays. The picture breaks here: a label names a depot, but a MAC address names a card, not a place.
MAC addresses#
A MAC address is 48 bits, written as six bytes: 00:00:5e:00:53:21. In a globally unique address,
the first 24 bits or more name the owner of the block: the OUI (organizationally unique identifier).
The owner numbers the rest.
ff:ff:ff:ff:ff:ff, all ones, is the broadcast address: every device on the link accepts that frame.
The addresses in this part come from a block kept for documentation (RFC 9542), so they match no real
card.
A MAC address is flat. An IP prefix says which network a host is in; a MAC address says nothing about where the card is plugged in. That is why a switch has to learn it (below).
A damaged frame is dropped, silently#
The sender’s card computes the frame check as it sends, and the receiving card computes it again and compares. The link uses a CRC because a card computes it in hardware. It also catches more errors than the software checksum of TCP and UDP.
In the figure, segment 6 for db-b1 arrives whole, and gw-a1 sends it on (Phase intact). Segment 7 is hit on the cable between sw-a1 and gw-a1. gw-a1’s card, at the end of the damaged cable, finds a mismatch and throws the frame away without telling the sender (Phase damaged).
The link itself does not repair the loss.
The worker’s TCP gets no acknowledgement (ACK) for segment 7. It sends the segment again after its timer (Part 3; Phase resend in the figure above). UDP data is simply lost.
On Linux, ip -s -s link counts these drops per interface: the crc column under RX errors. The
panel inside gw-a1 shows that counter rising by one.
In a VPC#
The MTU depends on the provider and the settings, and it is not always 1,500. The table shows two providers.
| Provider | MTU inside the VPC | Leaving the VPC |
|---|---|---|
| Google Cloud | default 1,460; you may set 1,300 to 8,896; a Linux VM on a public image learns it through DHCP option 26 | the internet generally uses 1,500, so packets of 1,500 or less avoid most MTU trouble |
| AWS | every instance type supports 1,500; current types support 9,001, called jumbo frames | 1,500 through an internet gateway or a VPN; 8,500 over peering between Regions |
So a 9,001-byte packet that works between two AWS instances will not pass the internet gateway as it is. It is split, or dropped if its DF flag is set. Keep traffic that leaves the VPC at 1,500 or below (When it breaks).
ARP: whose MAC is 10.1.4.1?#
The worker learned gw-a1’s IP address, 10.1.4.1, from DHCP (Part 4). To build the frame it also
needs gw-a1’s MAC, and nobody told it. On a plain Ethernet LAN like this one, it asks the whole link
with ARP (Address Resolution Protocol).
Request and reply#
The request goes in a broadcast frame, and sw-a1 floods it to every port (Phase request). worker-2,
10.1.4.22, sees an address that is not its own and sends no reply. Only gw-a1, which owns 10.1.4.1, answers, and
its reply goes to the worker’s MAC alone (Phase reply).
| ARP field, as RFC 826 names it | In the request | In the reply |
|---|---|---|
| opcode | request | reply |
| hardware address of sender | 00:00:5e:00:53:21, the worker | 00:00:5e:00:53:01, gw-a1 |
| protocol address of sender | 10.1.4.21 | 10.1.4.1 |
| hardware address of target | not known yet, so not set | 00:00:5e:00:53:21 |
| protocol address of target | 10.1.4.1, the address asked about | 10.1.4.21 |
| frame sent to | ff:ff:ff:ff:ff:ff, everyone | 00:00:5e:00:53:21, the worker only |
While it waits, the worker holds the packet for gw-a1. RFC 1122 says a host should hold at least the latest one; Linux keeps up to 3. When the reply arrives, the pair goes into the cache, and the packet leaves (Phase cache).
Only on your own subnet#
The worker asks for gw-a1’s MAC, never for the MAC of 10.3.2.6. A host on another subnet is reached
through the gateway (Part 4), so the frame goes to gw-a1, and gw-a1 builds the next frame.
A request for an address on another subnet normally gets no answer (a router doing proxy ARP is the exception). ARP has a request and a reply, and no error message.
How long the answer is kept#
Linux treats a confirmed entry as valid for at least 15 to 45 seconds, a random value around a base of 30. TCP replies that keep arriving count as proof and extend it.
After that the entry is stale. The next packet still goes out, and if no reply confirms the address within about 5 seconds, Linux asks again, first by unicast.
ip neigh shows the cache and the state of each entry:
$ ip neigh
10.1.4.1 dev eth0 lladdr 00:00:5e:00:53:01 REACHABLE| State | Means |
|---|---|
REACHABLE | confirmed recently; used as it is |
STALE | still valid but not trusted; the next packet starts a check |
DELAY | waiting a few seconds for proof before asking |
FAILED | the checks got no answer |
Other systems use other times. The host requirements standard (RFC 1122) speaks of cache timeouts of about a minute or less.
IPv6#
IPv6 does the same job with Neighbor Discovery. The question goes to a multicast address built from the target’s address, not to everyone on the link.
In a VPC#
In a VPC, broadcast is not supported: Google Cloud says so, and AWS says it does not support broadcast in a VPC. A VM still sends an ARP request for its gateway, but no other VM sees it. In Google Cloud the VPC network answers: “the instance kernel issues ARP requests and the VPC network issues ARP replies”.
A Linux VM there gets a /32 netmask by default. So even traffic to a VM in the same subnet goes to the gateway’s MAC address. The AWS pages used here do not say which device answers.
A switch learns where everyone is#
sw-a1 joins the worker 10.1.4.21, worker-2 10.1.4.22 and gw-a1 10.1.4.1 on ports 1, 2 and 3. It forwards frames by MAC address; it
does not route. Nobody types its table: it fills the table from the frames it sees.
Learn, then forward, flood or filter#
For every frame, sw-a1 first learns from the source, then decides from the destination.
| A frame arrives with | sw-a1 does |
|---|---|
| any source MAC | writes it with the port the frame came in on, and resets its age (learn) |
| a destination in the table, on another port | sends the frame out of that port only (forward) |
a destination not in the table, or ff:ff:ff:ff:ff:ff | sends it out of every other port (flood) |
| a destination on the port the frame came in on | drops it, because the receiver already has it (filter) |
The figure replays the ARP exchange from sw-a1’s side. In Phase flood the table is empty: sw-a1 writes
00:00:5e:00:53:21 on port 1 and floods the request to ports 2 and 3. gw-a1’s reply teaches sw-a1
00:00:5e:00:53:01 on port 3, and goes out of port 1 only (Phase learn).
The worker’s first data frame for gw-a1 then goes from port 1 to port 3 alone (Phase forward). When a switch floods a frame meant for one host, the others get it too. Their cards normally drop a frame whose destination MAC is not their own.
A switch is invisible to hosts. The worker addresses its frame to gw-a1, never to sw-a1. sw-a1 sends no message to hosts about its learning; it shows only in its table.
An everyday picture: a receptionist learns which office each person works in by watching where their visitors come from. For a stranger, the receptionist calls on every floor. The picture breaks here: a receptionist could ask, and a switch never asks; it only watches the frames.
Rows age out#
A row that no frame from its address refreshes is removed once it is too old. A host that moves to another port is learned again on the new one as soon as it sends a frame.
On a Linux bridge, the software switch inside a host, bridge fdb show lists the table. The bridge
setting ageing_time is the age limit, in seconds. On one Linux host checked on 2026-10-16 it was 300
seconds, the value in Phase age of the figure.
Switch or router#
| Switch | Router | |
|---|---|---|
| address it reads | the destination MAC | the destination IP |
| table built by | itself, learning from frames | routing protocols or the operator (Parts 4, 5) |
| setup | none for a plain switch: plug in, and it learns | an address and routes on every interface |
| loops | must not exist: a frame has no TTL, so it would circle forever | survive: the TTL ends a looping packet (Part 4) |
| reach | one broadcast domain | between subnets |
A broadcast domain is the set of devices that one broadcast frame reaches. A router ends it; a switch does not.
Loops and spanning tree#
Two switches joined by two cables form a loop. A flooded frame then goes round and round, copied at each switch. That is a broadcast storm: copies of one frame fill every link until the network stops.
The classic fix is spanning tree. The switches agree on one active path between any two of them and keep the extra cables on standby.
VLANs: one switch, separate networks#
ops-a1, 10.1.5.10, plugs into port 4 of sw-a1 but belongs to another subnet, 10.1.5.0/24. Without
more, the worker’s broadcasts would reach it. A VLAN (virtual LAN) splits one switch into separate
networks, and a trunk carries several VLANs to a second switch, sw-a2, where ops-a2 sits.
Ports in groups#
The operator puts each port of a switch in a VLAN. A port that carries one VLAN, without tags, leads to a host; it is called an access port. Each VLAN is its own broadcast domain.
| Port of sw-a1 | Device | VLAN | Subnet |
|---|---|---|---|
| 1 | the worker, 10.1.4.21 | 10 | 10.1.4.0/24 |
| 2 | worker-2, 10.1.4.22 | 10 | 10.1.4.0/24 |
| 3 | gw-a1, 10.1.4.1 in VLAN 10 and 10.1.5.1 in VLAN 20 | 10 and 20, tagged | both |
| 4 | ops-a1, 10.1.5.10 | 20 | 10.1.5.0/24 |
| trunk | sw-a2, which holds ops-a2, 10.1.5.11 | 10 and 20, tagged | both |
In Phase broadcast in VLAN 10, the worker’s ARP request reaches ports 2 and 3 and the trunk. It never reaches port 4, although ops-a1 is plugged into the same switch.
Between VLANs only a router passes#
The worker cannot reach ops-a1 through the switch alone. Its packet for 10.1.5.10 goes to gw-a1, its
gateway, which routes it into VLAN 20 and sends it back down (Phase between VLANs).
So gw-a1 needs an interface in each VLAN. Here its one port carries both VLANs, tagged, and its address
in VLAN 20 is 10.1.5.1.
A trunk carries its VLANs, tagged#
Between sw-a1 and sw-a2, one cable carries both VLANs. Each frame on it carries a 4-byte 802.1Q tag that names its VLAN. sw-a1 adds the tag as the frame enters the trunk, and sw-a2 removes it before ops-a2 (Phase over the trunk).
| Part of the tag | Bits | Holds |
|---|---|---|
| tag type | 16 | 8100: a VLAN tag follows; it sits after the source MAC |
| priority | 3 | the frame’s class, 0 to 7 |
| one flag | 1 | a bit this part skips |
| VLAN ID | 12 | the VLAN, 10 or 20 here |
The frame’s own type field, 0800 for IPv4, follows the tag. A trunk may also carry one VLAN without a
tag, the native VLAN; both ends must then agree on which one.
Twelve bits give 4,096 values, of which 4,094 can be used as VLAN IDs. That is too few for a cloud provider with many customers (RFC 7348).
Overlays#
A VPC spans many hosts and racks, and a provider runs many customers’ networks at once. Instead of a VLAN tag, an overlay such as VXLAN wraps each frame in a UDP packet. The provider’s own IP network carries it.
VXLAN (RFC 7348) gives each segment a 24-bit ID, which allows up to 16 million segments. The outer packet goes to UDP port 4789 by default.
The Google Cloud and AWS pages used here do not say how their VPCs keep one customer’s traffic apart from another’s. Read VXLAN as an example of the idea, not as what they use.
The data center: many equal paths#
In a data center, each rack has a top-of-rack switch (ToR), here tor-1 and tor-2. Every ToR connects to every spine switch, spine-1 and spine-2. server-1 under tor-1 has two equal paths to server-2 under tor-2: which one does each packet take?
Leaf and spine#
The older design was a tree of ever bigger switches. It reaches its limit: the top switches cannot get enough ports, and traffic between racks must share their few links.
Many large data centers use leaf and spine, a design that repeats one kind of switch in two layers. Every leaf, here the ToR, connects to every spine (RFC 7938); larger sites add a tier.
| Tree | Leaf and spine | |
|---|---|---|
| paths between two racks | a few, all through the top switches | one per spine |
| to grow | replace the top switches with bigger ones | add leaves, and spines up to the number of leaf uplinks |
In the design of RFC 7938 these switches also route. BGP (Part 5) runs between them and fills their tables. In 2014 Facebook (now Meta) described its fabric the same way: routed from the ToR uplinks to the edge, with BGP alone.
ECMP: one flow, one path#
Between two racks there are as many equal paths as there are spines. ECMP (equal-cost multipath) spreads traffic over them per flow, not per packet. tor-1 hashes the header fields that name a flow: the addresses, the protocol (TCP here) and the ports; the result picks one spine.
So every packet of flow 1, port 41001 to 443, takes spine-1 and usually arrives in order (Phase flow 1). Flow 2, port 41002, hashes to spine-2 (Phase flow 2). If one flow’s packets took both spines, some would arrive early, and early segments make the receiver send duplicate ACKs (Part 3).
When spine-1 fails, its links go dead. Data center links are mostly point-to-point fiber, so a dead link is often noticed in milliseconds.
Many BGP routers then close the session over that link at once. tor-1 drops spine-1 from its set, and flow 1 moves to spine-2 (Phase spine down).
Speeds, with their dates#
| Date | Source | Says |
|---|---|---|
| 2022-08-24 | Google, on its Jupiter data center network | servers and storage moving from 40 to 100, 200 and now 400 Gb/s; more than 6 Pb/s of data center bandwidth; no spine layer: optical circuit switches join groups of racks in a direct mesh |
| 2026-10-15 | the IEEE 802.3 Ethernet working group | a task force still at work on 200 Gb/s, 400 Gb/s, 800 Gb/s and 1.6 Tb/s Ethernet |
| 2026 | the Ethernet Alliance roadmap | 100G to 800G links, with 1.6 Tb/s emerging |
In a cloud#
Google Cloud places resources in zones, deployment areas inside a region. Putting VMs in different zones lowers the risk that one infrastructure outage hits them all.
Google names Compute Engine among the services that run on Jupiter. So a VM’s packets cross a data center fabric that the provider builds and runs.
One request through the whole stack#
The worker boots in network A, subnet 10.1.4.0/24, and calls https://api.acme.example in network B,
10.3.0.0/16. A plain HTTPS request over TCP takes six
exchanges before the first byte of the reply, each taught in a part of this series. Every one that
leaves the subnet goes to gw-a1’s MAC first.
| Phase | What the worker sends | IP destination | MAC destination on the first link | Taught in |
|---|---|---|---|---|
| DHCP | discover and request; gw-a1 offers and confirms 10.1.4.21 | 255.255.255.255, from 0.0.0.0 | ff:ff:ff:ff:ff:ff | Part 4 |
| ARP | who has 10.1.4.1? | none: ARP is not carried in IP | ff:ff:ff:ff:ff:ff | this part |
| DNS | the A record for api.acme.example, UDP port 53 | the resolver, 10.1.0.53 | gw-a1, 00:00:5e:00:53:01 | Part 2 |
| TCP | SYN from port 51514 to port 443; the api answers SYN-ACK, the worker ACK | the api, 10.3.2.5 | gw-a1, 00:00:5e:00:53:01 | Part 3 |
| TLS | the handshake | 10.3.2.5 | gw-a1 | Part 8 |
| HTTP | the request | 10.3.2.5 | gw-a1 | Part 2 |
After the DHCP ACK the worker also checks its new address with an ARP probe, one more broadcast (RFC
2131, RFC 5227). The resolver sits on another subnet, 10.1.0.53, so even the DNS query goes to
gw-a1’s MAC.
sw-a1 passes that frame on to gw-a1; the frame says gw-a1, and the packet inside says the resolver. Only the DHCP and ARP phases use broadcast. After them the worker needs no more ARP while its entry for gw-a1 is fresh.
At every router on the way, the frame is rebuilt and the packet keeps its addresses.
When it breaks#
| Case | Description | Check with | Why | Fix |
|---|---|---|---|---|
| two hosts, one address | connections to 10.1.4.21 work, fail, then work again; the MAC behind the address keeps changing in gw-a1’s ARP cache | arping -D 10.1.4.21 from a host on the link: any reply means a card already holds the address | two hosts were given the same address, for example one typed by hand inside the DHCP range | give one host another address; RFC 5227 has a host probe an address with ARP before it uses it |
| a failover that keeps the old MAC | an address moves to a standby host, yet clients keep sending to the old one for a while | ip neigh on a client: the old MAC is still listed for the address | the clients’ ARP caches hold the old pair until it is checked again | the new owner announces the pair: arping -U sends an unsolicited ARP that updates neighbors’ caches |
| a packet bigger than the link | small requests work, big ones hang; often a jumbo-frame VPC talking to the internet | ip link shows each interface’s MTU; ping -M do -s 1472 (Part 5) | a link on the way takes smaller packets than the sender sends (Part 4), and the message that says so is lost or blocked (Part 5) | the same MTU on every interface of a link; 1,500 for traffic that leaves the VPC; let ICMP 3/4 through |
| a link with damaged frames | transfers slow down and TCP resends more; UDP data goes missing | ip -s -s link: the crc count under RX errors keeps growing | a bad cable, port or card flips bits, and each damaged frame is dropped | replace the cable or move to another port; read the counters at both ends |
| a loop between switches | the whole network slows to a stop, and every link is full | the switches’ port counters: the same broadcast frames at a huge rate on every port | two paths between switches and no spanning tree; a frame has no TTL, so a broadcast circles forever | turn spanning tree on, or remove the extra cable |
Remember#
- Each link carries the packet in a new frame: the MAC addresses change at every router, the IP destination does not.
- A damaged frame is dropped and nobody is told: TCP sends again, UDP loses it.
- ARP asks everyone and hears back from one; for another subnet, the frame goes to the gateway’s MAC.
- A switch learns from source addresses and floods what it does not know; a broadcast stops only at a VLAN’s edge or a router.
- Leaf and spine give two racks one path per spine, and ECMP keeps each flow on one of them.
Sources#
Checked 2026-10-16.
- RFC 826, An Ethernet Address Resolution Protocol: the request and reply fields, the broadcast request, the unicast reply
- RFC 894, A Standard for the Transmission of IP Datagrams over Ethernet Networks: data field of 46 to 1,500 bytes, padding, the broadcast address
- RFC 9542, IANA Considerations and IETF Protocol and Documentation Usage for IEEE 802 Parameters: MAC address structure and prefixes, the documentation block
00:00:5e:00:53:00toff - IANA, IEEE 802 Numbers: the type values
0800,86DD,0806,8100and the length range - Wireshark wiki, Ethernet: the 4-byte frame check, a frame with a wrong CRC thrown away
- RFC 3385, Internet Protocol Small Computer System Interface (iSCSI) Cyclic Redundancy Check (CRC)/Checksum Considerations: CRC against checksum
- RFC 1122, Requirements for Internet Hosts: ARP cache timeouts, proxy ARP and the queued packet (sec. 2.3.2)
- RFC 5227, IPv4 Address Conflict Detection: ARP probes and announcements
- RFC 4861, Neighbor Discovery for IPv6: the IPv6 counterpart of ARP
- RFC 7348, Virtual eXtensible Local Area Network (VXLAN): the 12-bit VLAN ID and the 4,094 limit, the tag’s place in the frame, the 24-bit segment ID, UDP port 4789, the frame check at the end of the frame
- RFC 7938, Use of BGP for Routing in Large-Scale Data Centers: tree and leaf-and-spine designs, ECMP paths, broadcast storms, spanning tree, failure detection
- RFC 2992, Analysis of an Equal-Cost Multi-Path Algorithm: hashing the header fields of a flow; flows disrupted when the set changes
- RFC 6437, IPv6 Flow Label Specification: addresses and ports as the input of a load-sharing hash
- RFC 2131, Dynamic Host Configuration Protocol, RFC 2132, DHCP Options (sec. 5.1, interface MTU), RFC 1035, Domain Names: Implementation and Specification, RFC 9293, Transmission Control Protocol, RFC 9846, The Transport Layer Security (TLS) Protocol Version 1.3, RFC 9110, HTTP Semantics: the request walk
- arp(7), Linux manual page:
base_reachable_time,delay_first_probe_time,unres_qlen - Linux v6.18, include/uapi/linux/if_link.h: the
crccounter, packets received with a CRC error - ip-neighbour(8), arping(8), bridge(8), ip-link(8), ip(8), ping(8), Debian
- Wireshark wiki, VLAN: the priority values 0 to 7
- Cisco Meraki, Fundamentals of 802.1Q VLAN Tagging: access and trunk ports, the native VLAN, routing between VLANs
- Google Cloud blog, The evolution of Google’s Jupiter data center network (2022-08-24)
- Facebook engineering, Introducing data center fabric (2014-11-14)
- IEEE 802.3 Ethernet Working Group, Ethernet Alliance, Ethernet roadmap
- Google Cloud, Maximum transmission unit, Compute instance netmasks, Advanced VPC concepts, Regions and zones
- AWS, Network MTU for your EC2 instance, Subnet CIDR blocks