The Memory Wall from Scratch — Why Moving Data Costs More Than Computing

Multiplying two numbers is cheap; delivering them is not. Starting from the physics of charging a wire, we get to why DRAM latency never shrank, the orders of magnitude in the memory hierarchy, Little's law, machine balance and the roofline — and end with a procedure for deciding whether your kernel is compute bound or bandwidth bound.

The expensive part is not the arithmetic

Start by distrusting the phrase "this computation is heavy". Multiplying two numbers is a remarkably cheap operation. What is expensive is getting those two numbers in front of the arithmetic unit.

The reason is physics. A multiply is a local event across a few thousand transistors, with signals travelling microns. Fetching a value from memory means driving a wire that runs to the edge of the chip — often off it. The energy needed to swing that wire to voltage $V$ is

where $C$ is the capacitance hanging off the wire and $V$ is the voltage swing. Put in words: the bill for sending a signal down a wire comes from exactly two things — how much charge that wire insists on holding, and how hard you have to swing the voltage — and the second one is squared. And $C$ grows roughly in proportion to the wire's length, so distance is the price tag. Reading a word from the register next door and reading one from DRAM across the board differ in energy by orders of magnitude. Arithmetic is cheap, movement is expensive: every design decision below rests on that asymmetry.

Why DRAM latency never shrank

Transistor counts rose, clocks rose, arithmetic units multiplied. DRAM latency did not follow at anything like that pace, and the widening gap is the memory wall. The reason is structural.

One DRAM bit is a capacitor and a transistor. To read it you raise the word line and let the cell dump its charge onto the bit line — but hundreds of cells hang off that line, so its capacitance is large and the resulting swing is tiny. A sense amplifier must amplify it before anyone can call it a 0 or a 1. Worse, the read destroys the charge, so the value must be written back and the bit line precharged before the next access.

None of that gets shorter when you shrink the process. A smaller cell holds less charge, which makes the signal weaker and detection harder. Bandwidth is a different story: add I/O pins, raise the transfer clock, stack the dies. Bandwidth can be bought; latency cannot. That asymmetry motivates nearly every trick that follows.

Why nobody builds memory that is both big and fast

So why can't one tier be both? Add capacity and the array occupies more area; wires get longer, resistance and capacitance rise, edges get lazier. At the limit even the speed of light bites: in one cycle at 1 GHz, light travels about 30 cm in vacuum. Big things are, of necessity, far away. Hence the only option: stack a small fast tier on a large slow one.

Top to bottom, latency spans more than six orders of magnitude. On a linear axis the upper tiers simply vanish.

Latency can be hidden; bandwidth cannot

Latency and bandwidth are different quantities. What links them is Little's law, the general queueing result.

$N$ is the number of accesses in flight at once, $\lambda$ is the throughput you want (accesses per second), $L$ is the latency. Read it as: keep flow times round-trip time worth of inventory in the air at all times. It is the same arithmetic a warehouse lives by, which says that the slower your supplier's round trip — and the faster you want to sell — the more orders you have to keep outstanding simply to keep the shelves from going empty. To move 1 TB/s in 64-byte cache lines you need $\lambda \approx 1.6 \times 10^{10}$ accesses per second; at 100 ns latency that is $N = 1600$. Unless 1600 reads are outstanding at every instant, the bandwidth stays unused.

Prefetchers, load queues, and a GPU's tens of thousands of threads all exist to maintain that $N$ — inventory whose purpose is to hide latency. But only latency hides: the volume you can move is fixed by bandwidth. That is where the real argument starts.

Machine balance — arithmetic intensity, seen from the hardware

The cost of matrix multiplication defined arithmetic intensity $I$ = FLOPs ÷ bytes moved, and showed GEMM reaching $I \approx n/6$ while GEMV is stuck near $I \approx 0.5$. That is a property of the algorithm. The hardware has a quantity of the same dimension.

This is the machine balance: how many operations this machine can perform in the time it takes to deliver one byte. Stated in words, it counts the multiplies the machine spins through while it waits for a single byte to show up. Its unit is FLOP/byte, the same as $I$, so the two compare directly.

• $I < B_m$ → bandwidth bound. The arithmetic units are idling on data
• $I > B_m$ → compute bound. Bandwidth has headroom to spare

The knee of the roofline — the ridge point — sits exactly at $I = B_m$. And the memory wall is nothing more than the statement that $B_m$ has been rising for decades. Compute grew faster than bandwidth, so code that was once compute bound slid over to the bandwidth-bound side without a single character being edited. General-purpose processors used to sit around $B_m \approx 1$; accelerators with matrix units are in the hundreds. A kernel whose intensity does not reach the hundreds can no longer saturate the arithmetic units.

GEMM fast, GEMV slow — confirmed in orders of magnitude

A matrix-vector product (GEMV) touches each matrix element exactly once, so the lower bound on runtime is simply the time to read the weights.

$P$ is the number of matrix elements, $b$ the bytes per element, $\mathrm{BW}$ the memory bandwidth. Read in words, it states something almost too obvious to write down: you cannot be finished before you have finished reading the matrix, and reading it takes its total size divided by the bandwidth. Compute performance does not appear in this formula. No arithmetic unit, however monstrous, moves that bound; the only ways down are raising bandwidth or shrinking $b$.

GEMM is the mirror image: park a tile in cache and $I$ climbs as far as capacity allows, and past $B_m$ the number of arithmetic units starts to matter again. With the same weight matrix, whether the other operand is one vector or a batch of them decides which resource is the bottleneck.

Why LLM inference is bandwidth bound

Split generation in two. Prefill processes the whole prompt at once, so tokens come in a batch and the work is GEMM — usually compute bound. Decode is the problem: emitting one token requires reading every parameter once. With $N$ parameters at $b$ bytes each, that is $Nb$ bytes per token, so the ceiling on generation speed is

"bytes you can move per second, divided by bytes needed per token". Or, less formally: the model re-reads its entire set of weights to produce each single token, so the generation rate is capped by how many times per second the machine can read that whole set end to end. Again, compute performance is absent. Holding $N = 7 \times 10^9$ in fp16 ($b = 2$) means 14 GB per token; on a machine with 1 TB/s the theoretical ceiling is roughly 70 tokens per second.

Batching by $B$ changes the accounting: the weights are still read once, and that single read now feeds $B$ sequences. Bytes moved stay flat while work multiplies by $B$, so $I$ multiplies by $B$ — GEMV turns into GEMM. Batching is not a speed spell; it is an operation that moves the bottleneck from bandwidth to compute.

That move has two end points. One is the ridge point: once $I$ passes $B_m$ you are compute bound and further batching only stretches latency. The other is the KV cache, which grows with sequence length and must also be re-read every token. At long context the KV reads dominate the weight reads, and the same cache eats the memory you needed to grow the batch.

How this is used in practice

Three steps to tell compute bound from bandwidth bound

1. Derive the bound on paper first. Compute $t_{\text{mem}}$ — bytes moved (weights + activations + KV cache) divided by bandwidth — and $t_{\text{cmp}}$ — FLOPs divided by peak compute. The larger is your theoretical floor. Do this before launching a profiler: without an expectation, you will misread the numbers it gives you.
2. Compare against measurement. If the measured time is within one to two times $t_{\text{mem}}$, you are bandwidth bound and the case is closed. If it is far worse than both theoretical floors, the culprit is the third one — latency bound: insufficient parallelism, synchronization, kernel launch overhead, a Python loop on the host.
3. Confirm with a profiler. On NVIDIA hardware, the Speed of Light section of Nsight Compute (ncu --set full) prints Memory [%] next to Compute (SM) [%]. On CPUs, perf stat -e cache-misses,LLC-load-misses or VTune's Memory Bound metric. In PyTorch, take per-kernel times with torch.profiler and apply steps 1–2 to the top few only.

Read the result as three quadrants: high Memory and low Compute means bandwidth bound (quantization, kernel fusion, layout); high Compute and low Memory means compute bound; both low means latency bound, and that last case is the one people miss.

Raising the batch size works because it splits the cost of moving weights across several requests — at batch 8, weight traffic per token drops eightfold. Continuous batching exists to preserve that split under bursty real-world arrivals. When it stops helping, it is one of three things: you crossed the ridge point, the KV cache exhausted memory, or you hit your latency SLO. Being able to say which one you hit is what determines the next move.

Pitfalls that turn into incidents

• "We swapped in a chip with twice the compute and generation didn't speed up" is the expected outcome if bandwidth stayed the same. Look at bandwidth and capacity before the FLOPS/TOPS on the datasheet.
• Arithmetic intensity depends on the data type. Going fp32 → fp16 doubles $I$. Quantization pays off dramatically only in the bandwidth-bound regime.
• Speeds measured at sizes that fit in cache do not reproduce at production sizes. Benchmark with the real weight footprint.

The standard interview question is this: "A seven-billion-parameter model in fp16, on a machine with 1 TB/s of bandwidth. What is the theoretical minimum time to generate one token?" The answer is 14 GB ÷ 1 TB/s = 14 ms. The point of the question is that you can answer it without ever being told the compute performance.

Summary

• Arithmetic is cheap, movement is expensive — because charging a wire costs $\tfrac{1}{2}CV^2$ and capacitance scales with distance
• DRAM latency is pinned by physics. Bandwidth is purchasable; latency is not
• Latency hides behind $N = \lambda L$ worth of in-flight requests; bandwidth does not hide
• Compare $B_m$ against $I$ and the bottleneck is decided. The memory wall is $B_m$ climbing year after year
• Decode is capped at $\mathrm{tok/s} \le \mathrm{BW}/(Nb)$; batching relocates the bottleneck rather than removing it

Next we look at the same problem from the power side. Why movement is expensive, and why the die cannot simply be paved with arithmetic units, both come down to a single equation.