2026-08-27·★ MEMBER·9 min read

Interconnects — How NVLink, PCIe, and Light Set the Limits of Scale

When adding GPUs stops making things faster, the wiring is usually to blame. This piece pins down the bandwidth hierarchy from HBM through NVLink, PCIe, and the inter-node fabric, breaks collective time into 'steps × latency + bytes ÷ bandwidth', and explains why all-to-all hurts and why distance eventually demands light.

#interconnect #nvlink #pcie #infiniband #nccl #collective-communication #hardware

ModalitytextTaskhardware

An analogy: the next desk, the next room, the next building

Asking the colleague beside you "does this look right?" takes seconds. The next room means standing up and walking. Another building means booking a meeting and waiting until tomorrow. The message is the same single line either way — but distance changes the cost by orders of magnitude.

A GPU cluster has exactly this shape. Whether the data you need sits in memory bolted to your own chip, in the GPU next to you in the same server, or in a different server one cable away, the price of the same byte changes. The interconnect is that whole apparatus: the wires, the switches, and the protocols riding on them.

This is also where the difficulty of distributed training ultimately lives. Splitting the computation is easy. Reconciling the split results is the expensive part — and the interconnect is the road that reconciliation has to travel.

Laying out the bandwidth hierarchy

The numbers below come from published specs, quoted only precisely enough to show the order of magnitude. Remember the ratio between tiers, not the absolute figures.

HBM, bolted to the chip. Roughly 2 terabytes per second on A100-class parts. The drawer that was slowest in the GPU memory hierarchy becomes the reference point here. The moment you leave the chip, nothing is faster than this.

GPU to GPU inside one server (NVLink). 600 GB/s per GPU in the A100 generation, 900 GB/s in the H100 generation — both quoted as bidirectional totals. Put an NVSwitch in the middle and all eight GPUs in the chassis can talk to any partner at that rate.

Across servers (PCIe and the network). The usual road off the chip is PCIe: Gen4 x16 gives about 32 GB/s per direction, Gen5 x16 roughly double that. Beyond it, one NDR-generation InfiniBand port carries 400 Gb/s — which is 50 GB/s.

Line them up: HBM in the terabytes-per-second range, NVLink in the hundreds of gigabytes, node-to-node in the tens. Each step outward costs you roughly one order of magnitude.

Don't get fooled by units

Units in this field are not consistent, and that causes real mistakes. Three things to check.

Gb/s and GB/s differ by a factor of eight. A 400 Gb/s port is 50 GB/s. Networking people write bits, memory people write bytes, and both conventions happily coexist in the same document.

One direction, or both summed? NVLink figures are usually bidirectional totals; PCIe figures are usually per direction. Compare them as printed and NVLink looks twice as impressive as it is.

Theoretical or achieved? Between protocol headers and line coding, achieved throughput lands around 70–90% of the theoretical number. Design against measurements, not datasheets.

Connecting everyone makes the cable count explode

If you connect N machines all-to-all with dedicated links, the number of links you need is

L_{\text{full}} = \frac{N(N-1)}{2}

(1)

which says, in words, "you need one cable per pair of participants." $N$ is the machine count and $L_{\text{full}}$ the number of links. Eight machines need 28 cables; 64 machines need 2,016. Multiply the machines by eight and the cables go up more than seventyfold.

So real systems give up on all-to-all wiring. Hang everyone off a switch, arrange them in a ring so each talks only to its neighbours, or bundle them into a tree. A ring needs $N$ links; a tree has $\log N$ levels. That instinct — that the wiring pattern is itself an order-of-magnitude decision — is what pays off below.

FIG 1Read the n on the horizontal axis as a node count. All-to-all wiring is O(n²), a ring is O(n), and a tree's depth is O(log n). Drag n to the right and the three curves stop fitting on one screen; switch the vertical axis to linear and the divergence becomes even starker

Collectives — what distributed training actually puts on the wire

What flows across the network in distributed training isn't point-to-point messages. It is a fixed set of patterns in which every participant takes part at once — collective operations. Two matter most: all-reduce, where everyone's values are summed and everyone ends up holding the result, and all-to-all, where every rank sends different content to every other rank.

Gradient synchronisation in data parallelism is an all-reduce (distributed training from scratch). Once per step, without exception, there is a moment where everyone has to fall into line. How long that moment takes is your ceiling on scaling.