The Inference Chip Wars — Inside the Design Philosophies of Groq, Cerebras, and the LPU

Inference-only chips exist because decoding is limited by reading, not by arithmetic. From zero background: the two answers — put everything in SRAM (Groq/LPU) and make the chip a whole wafer (Cerebras) — the software counterattack, and the arithmetic you need to read the market without swallowing vendor benchmarks whole.

Every token, the entire library gets wheeled out again

For years, "AI chip" meant "GPU for training." Lately a different category keeps showing up: silicon built for inference only. Groq's LPU, Cerebras's wafer-scale engine, SambaNova's RDU. The designs differ wildly; the pitch is nearly identical. Same model, faster reply than a GPU.

But it's the same matrix multiplication underneath. So why does "inference-specific" work as a business category at all? One reason: the slowest part of inference isn't the arithmetic.

LLM generation splits into two phases with completely different personalities. Prefill reads the whole prompt at once — big matrix against big matrix, exactly the shape a GPU was built for. Decode emits one token at a time, and that's where it gets ugly: every single token requires reading the model's weights end to end, and each weight you read participates in exactly one multiply-add before you're done with it.

Wheel every book out of the stacks, read one character from each, wheel them all back. Then do it again for the next character. That is decode. What sets the pace isn't how fast the librarian reads — it's the round trip of the cart.

An analogy: the distance to the warehouse decides how fast you answer

To a chip, where data lives is a matter of distance. On-chip SRAM sitting next to the arithmetic units is the shelf at your elbow; HBM stacked on the package is the warehouse across the yard. Reaching for the shelf versus walking to the warehouse differ by orders of magnitude in both latency and energy. We covered that physics in the memory wall.

What makes it interesting is that the shelf and the warehouse are good at opposite things. On-chip SRAM beats HBM on bandwidth by more than a factor of ten — and loses on capacity by two to three orders of magnitude. It takes six transistors to remember one bit, so SRAM eats area.

Boil the design space down and every inference chip is answering one question: can we give up on the slow warehouse and fit everything on the shelf?

The intuition: batch-1 speed comes down to a single division

The ceiling on decode speed is almost anticlimactically simple. Let $P$ be the parameter count, $b$ the bytes per parameter, and $B$ the memory bandwidth:

In plain terms: you cannot go faster than the time it takes to read every weight byte once. Double the bandwidth and the ceiling doubles. Halve the bytes per weight and it doubles. Add arithmetic units and the floor doesn't move a millimeter.

Put numbers in it and it gets concrete. A 70B model at fp16 is roughly 140 GB of weights. On a machine with 3 TB/s of memory bandwidth, that's about 47 ms per token — a ceiling of just over 21 tokens per second. Swap in a chip with twice the advertised FLOPs and those 47 ms don't budge, because what sets the limit is how much you have to haul, not how fast you can multiply.

Notice what's missing from that formula: batch size. However many users you serve simultaneously, the weights get read once. So raising the batch barely changes what any one user feels, while the throughput of the whole box climbs. That structural split is why every argument about inference chips eventually turns into "latency or throughput?"

There's a second term that grows as the conversation does. As context lengthens, each token also has to read the accumulated KV cache on top of the weights. Bytes read per token scale linearly with context length — and across a whole conversation, quadratically.