Why GPUs Are Fast — The Execution Model and the Limits of Parallelism

CPUs and GPUs do not mean the same thing by fast. Where the transistor budget goes, how SIMT bundles 32 threads into a warp, why branch divergence costs you, occupancy and register pressure — and finally Amdahl's law as a way to bound the payoff before you start, plus the profiler counters that tell you when the CPU is the bottleneck.

An analogy: one master, or a hundred ordinary cooks

There are two ways to build the same kitchen. In the first, you hire one master who has the entire workflow memorised, anticipates what comes next, and almost never stands idle. In the second, you line up a hundred ordinary cooks. Each is slower than the master, and at any moment several of them are standing in front of an oven doing nothing. But with a hundred of them, somebody is always working while somebody else waits. No waiting has been eliminated, and yet the second kitchen sends out more plates.

That is the difference between a CPU and a GPU. A CPU is built to finish one task in as little time as possible — latency optimisation. A GPU is built to maximise work completed per unit time — throughput optimisation. They do not define "fast" the same way, so measuring one with the other's yardstick guarantees a misreading.

Where the transistor budget goes

A chip has a finite transistor budget, and design is the decision of where to spend it.

A CPU spends most of it keeping a single instruction stream from stalling: branch predictors, speculative execution, reorder buffers, large caches. None of those perform arithmetic. Every one of them is an investment in "do not let this one thread wait".

The GPU took the opposite bet. It barely predicts and barely speculates, and hands that area to arithmetic units and register files. The register file on a GPU compute unit (an SM, streaming multiprocessor, in NVIDIA's terms) runs to hundreds of kilobytes, and it is not unusual for it to be larger than the L1 cache on the same chip. That is not a demand for storage — it is what it costs to keep the state of thousands of threads resident, with nothing saved and nothing restored. Context switching becomes essentially free, which makes "when a thread stalls on memory, run a different one meanwhile" a workable strategy rather than a wish.

Recall Little's law from the memory wall: filling the bandwidth requires $N = \lambda L$ accesses in flight at all times. A GPU's tens of thousands of threads are exactly that $N$. Instead of shortening latency with caches, it buries latency under parallelism. That is the core of the design.

A warning up front: parallelism only buys a constant

Parallelism buys you a constant factor. Using $p$ processors caps the gain at $p$, and it does not lower the complexity class by a single step. $O(n^2)$ stays $O(n^2)$ on a thousand machines; make $n$ ten times bigger and the hundredfold cost comes right back.

So the order never changes: lower the order, then fix how data moves (the cost of matrix multiplication), then parallelise. Amdahl's law, later in this piece, tells you exactly what ceiling that last step will hit before you commit to it.

SIMT — it looks like threads, but it moves in bundles

If those thousands of threads truly ran independently, nothing would be saved on control logic: you would need one instruction fetch and decode unit per thread.

So the GPU compromises. It groups 32 consecutive threads into a bundle and issues one instruction to all of them. NVIDIA calls the bundle a warp, AMD a wavefront (32 or 64 wide depending on generation). Fetch and decode happen once per bundle; 32 arithmetic lanes sit side by side and apply that one instruction to 32 different pieces of data.

This is SIMT — Single Instruction, Multiple Threads. You write ordinary per-thread code, and the hardware runs it as 32-wide SIMD. It is a compromise struck between writability and efficiency, and most GPU programming pitfalls leak out of that seam. The most important one is the moment the premise breaks: when the bundle no longer agrees on which instruction to run.