The Memory Wall from Scratch — Why Moving Data Costs More Than Computing

Multiplying two numbers is cheap; delivering them is not. Starting from the physics of charging a wire, we get to why DRAM latency never shrank, the orders of magnitude in the memory hierarchy, Little's law, machine balance and the roofline — and end with a procedure for deciding whether your kernel is compute bound or bandwidth bound.

The expensive part is not the arithmetic

Start by distrusting the phrase "this computation is heavy". Multiplying two numbers is a remarkably cheap operation. What is expensive is getting those two numbers in front of the arithmetic unit.

The reason is physics. A multiply is a local event across a few thousand transistors, with signals travelling microns. Fetching a value from memory means driving a wire that runs to the edge of the chip — often off it. The energy needed to swing that wire to voltage $V$ is

where $C$ is the capacitance hanging off the wire and $V$ is the voltage swing. Put in words: the bill for sending a signal down a wire comes from exactly two things — how much charge that wire insists on holding, and how hard you have to swing the voltage — and the second one is squared. And $C$ grows roughly in proportion to the wire's length, so distance is the price tag. Reading a word from the register next door and reading one from DRAM across the board differ in energy by orders of magnitude. Arithmetic is cheap, movement is expensive: every design decision below rests on that asymmetry.

Why DRAM latency never shrank

Transistor counts rose, clocks rose, arithmetic units multiplied. DRAM latency did not follow at anything like that pace, and the widening gap is the memory wall. The reason is structural.

One DRAM bit is a capacitor and a transistor. To read it you raise the word line and let the cell dump its charge onto the bit line — but hundreds of cells hang off that line, so its capacitance is large and the resulting swing is tiny. A sense amplifier must amplify it before anyone can call it a 0 or a 1. Worse, the read destroys the charge, so the value must be written back and the bit line precharged before the next access.

None of that gets shorter when you shrink the process. A smaller cell holds less charge, which makes the signal weaker and detection harder. Bandwidth is a different story: add I/O pins, raise the transfer clock, stack the dies. Bandwidth can be bought; latency cannot. That asymmetry motivates nearly every trick that follows.

Why nobody builds memory that is both big and fast

So why can't one tier be both? Add capacity and the array occupies more area; wires get longer, resistance and capacitance rise, edges get lazier. At the limit even the speed of light bites: in one cycle at 1 GHz, light travels about 30 cm in vacuum. Big things are, of necessity, far away. Hence the only option: stack a small fast tier on a large slow one.

Top to bottom, latency spans more than six orders of magnitude. On a linear axis the upper tiers simply vanish.

Latency can be hidden; bandwidth cannot

Latency and bandwidth are different quantities. What links them is Little's law, the general queueing result.

$N$ is the number of accesses in flight at once, $\lambda$ is the throughput you want (accesses per second), $L$ is the latency. Read it as: keep flow times round-trip time worth of inventory in the air at all times. It is the same arithmetic a warehouse lives by, which says that the slower your supplier's round trip — and the faster you want to sell — the more orders you have to keep outstanding simply to keep the shelves from going empty. To move 1 TB/s in 64-byte cache lines you need $\lambda \approx 1.6 \times 10^{10}$ accesses per second; at 100 ns latency that is $N = 1600$. Unless 1600 reads are outstanding at every instant, the bandwidth stays unused.

Prefetchers, load queues, and a GPU's tens of thousands of threads all exist to maintain that $N$ — inventory whose purpose is to hide latency. But only latency hides: the volume you can move is fixed by bandwidth. That is where the real argument starts.