How AI Accelerators Are Designed — What Actually Separates GPUs, NPUs, and TPUs

GPUs, TPUs, and NPUs are three different answers to one question: how much generality do you trade away for matrix multiplication? A from-scratch tour of systolic arrays, dataflow design, and the co-evolution of hardware and quantization.

In-Datacenter Performance Analysis of a Tensor Processing Unit

Why "AI-specific chips" exist at all

A ChatGPT reply and a smartphone unlocking with your face are, under the hood, doing almost the same thing: multiplying large matrices. Neural network inference is layer after layer of "weight matrix times input," and that is where the vast majority of compute time goes.

A CPU is a Swiss Army knife. To handle branch-heavy programs, operating systems, and everything in between, it spends most of its silicon on overhead — instruction decoding, branch prediction, caches. Matrix multiplication is the exact opposite kind of work: the same trivial operation, repeated an absurd number of times. Worse, the arithmetic cost of matrix multiplication grows roughly with the cube of the matrix size. As models grew, the well-organized generalist stopped being worth its price.

Enter the accelerator. The differences between GPU, TPU, and NPU come down to one trade-off: how much generality each design throws away in exchange for matrix-multiply performance.

An analogy: the genius chef, the hundred-cook kitchen, and the conveyor belt

• A CPU is a single genius chef. Any order, no matter how exotic — but one plate at a time.
• A GPU is a giant kitchen where thousands of line cooks make the same recipe at once. Instructions go out over a loudspeaker, and each cook applies the same steps to different ingredients (data).
• A TPU is a sushi factory's conveyor belt. Ingredients flow past at a fixed rhythm; each station performs one fixed action on whatever arrives and passes it along. Crucially, nobody ever walks back to the refrigerator (memory).
• An NPU is a street food stall: a menu cut down to a few items, kept running on a trickle of gas (power).

The TPU's style is called a systolic array. "Systolic" refers to the heartbeat — data pulses through the chip one beat at a time, which is exactly where the name comes from.

The core intuition: everything reduces to "multiply, then add"

A single neural network output is computed like this:

Read it in words: take each input value $x_i$, multiply it by a weight $W_{ij}$, and add everything up to get the output $y_j$. With the symbols dropped entirely, it says that every incoming number casts a vote, the weight decides how loudly that vote counts, and the output is just the tally. This "multiply, then accumulate" pair is called a MAC (Multiply-ACcumulate) operation, and a matrix multiplication is nothing more than an enormous bundle of MACs.

So accelerator design boils down to exactly two questions: (1) how many MAC units do you tile onto the chip, and (2) how do you feed data to them? In modern silicon, question (2) is the real battlefield — because moving data costs far more than computing with it.

How each architecture answers those two questions is precisely what separates the three families. GPUs hide the cost of data movement with sheer parallelism; TPUs eliminate it through wiring; NPUs shrink it to fit a power budget. Three very different philosophies for running the exact same matrix multiply — and once you see them side by side, every spec sheet starts making sense. Let's open each one up in turn.