CPU Pipelines and Branch Prediction — The Factory Inside One Clock Tick

An instruction appears to finish in a single clock tick because the machine is an assembly line. From the five classic stages, through the three kinds of hazard, to branch prediction, speculative execution, and the day the leftovers of speculation turned into an information leak — Spectre.

Spectre Attacks: Exploiting Speculative Execution

What "one instruction per clock" actually means

The spec sheet says 3.5 GHz: three and a half billion ticks per second. Instructions retire at roughly one per tick, so it is tempting to conclude that an instruction takes 0.3 nanoseconds end to end. It does not. A single instruction needs many ticks to travel from the front of the machine to the back. One result still pops out every tick because many instructions are in flight at once, each in a different stage.

A laundromat makes this concrete. Washing takes 30 minutes, drying 30, folding 30, so one person's laundry always takes 90 minutes, and four people pushed through strictly one at a time need 360. But if the next person starts the washer the moment you move your load to the dryer, the first person still finishes at the 90-minute mark — and after that, someone finishes every 30 minutes. Total: 180.

Two quantities live here and must never be conflated. Latency is how long one load takes (90 minutes); throughput is how many complete per unit time (one per 30 minutes). Pipelining does not shorten latency — the handoffs make it slightly worse — and is adopted anyway because throughput triples. That trade is the whole story.

The five-stage assembly line

The classic textbook machine splits the work of one instruction into five stages.

Between stages sit pipeline registers — small shelves that hold the intermediate result until the next stage picks it up. Because of them, each stage can work from its own shelf without knowing anything about its neighbours. With $N$ instructions, $k$ stages, and $t$ seconds per stage, the total time is:

Read that in words: you wait $k$ stages for the first result, and after that one arrives every stage-time. When $N$ is large the $k-1$ is rounding error and the cost per instruction approaches $t$; without a pipeline it was $kt$, so on paper this is a $k\times$ win. The catch is that splitting the work finer shortens each stage but never shortens the time spent parking data on the shelf, which is why stage counts cannot grow without limit.

Three reasons the line stops

When everything flows, you get one instruction per tick. Whatever interrupts that flow is called a hazard, and there are only three kinds.

Structural hazard — two instructions want the same equipment. With one dryer, somebody waits. It happens when instructions and data come from the same memory, which is why modern CPUs split the instruction cache from the data cache.

Data hazard — one instruction's result is the next instruction's input. The next person is trying to put on clothes the previous person has not finished folding; lining the work up in order is not enough.

Control hazard — nobody knows which instruction to fetch next. The front of the line stalls until a branch resolves, and most of the rest of this article is about that fight.

Data hazards are fixed with wire

Suppose a = b + c is followed by d = a - e. The first instruction finishes computing in EX but does not write its register until WB, so a naive machine has the second one reading a stale value back in ID. The fix is refreshingly direct: run a wire from the output of EX back to the input of EX, handing the result straight to the next instruction instead of waiting for the register file. This is forwarding (bypassing) — passing the clothes directly out of the dryer.

Exactly one combination cannot be wired away: the load-use hazard. A value loaded from memory is not ready until the end of MEM, already too late for the next instruction's EX stage, and you cannot run a wire backwards in time. That one costs a stall — a bubble injected into the pipe. It is why compilers try to put an instruction that does not use the loaded value immediately after a load, and part of why optimized builds reorder your code beyond recognition.

Control hazards — a branch does not know where it is going

The trouble with a branch is the gap in time between fetching and resolving. The fetch stage must produce a "next instruction" every single tick, but the verdict on if (x > 0) is not known until somewhere around EX. For those few ticks, the front of the machine has nothing to feed itself.

And branches are everywhere — roughly one instruction in five to ten. Loop backedges, calls, switch tables, virtual dispatch: programs are essentially made of branches. Stalling at each one would erase the benefit of pipelining entirely.