#gpu
7 articles
01
·Inference & Serving·★ MEMBER·PAPER·11 min read
Surviving GPU Out-of-Memory — Every Cause, Every Fix
`CUDA out of memory` reports only the allocation that happened to fail last, which is almost never the culprit. We count what actually occupies VRAM in five buckets, derive the sixteen-bytes-per-parameter fixed cost of training, and work through the fixes in order of least damage: gradient checkpointing, optimizer compression, offloading, KV cache limits, and fragmentation.
02
·How Transformers Work·★ MEMBER·PAPER·10 min read
FlashAttention from Scratch — The Paradox of Doing More Math to Go Faster
FlashAttention deliberately recomputes the same math in the backward pass. It does strictly more arithmetic and still runs faster — because on a GPU, moving numbers costs more than multiplying them. From counting HBM round trips, through the softmax wall that blocks tiling, to the online-softmax recurrence that breaks it, and the silent backend fallback that bites people in production.
03
·Cloud & Ops·★ MEMBER·10 min read
The Economics of GPU Cloud — Rent, Buy, or Commit
The same GPU carries four prices at once. Put on-demand, committed, spot, and owned hardware on a single equation: the break-even utilization, the true cost of spot once interruptions are priced in, the commitment burn rate, and the line items that never appear on the rate card.
04
·Computer Architecture·★ MEMBER·PAPER·9 min read
The GPU Memory Hierarchy — HBM, SRAM, Registers, and Why Movement Wins
What sets a GPU's speed is not the arithmetic units but where the data sits — registers, shared memory, L2 or HBM — and how many times it is moved. Capacities and bandwidths by order of magnitude, arithmetic intensity and tiling, a roofline per level of the hierarchy, and finally FlashAttention: more FLOPs, less time.
05
·Accelerators·★ MEMBER·PAPER·8 min read
How AI Accelerators Are Designed — What Actually Separates GPUs, NPUs, and TPUs
GPUs, TPUs, and NPUs are three different answers to one question: how much generality do you trade away for matrix multiplication? A from-scratch tour of systolic arrays, dataflow design, and the co-evolution of hardware and quantization.
06
·Parallel & Distributed·★ MEMBER·10 min read
Why GPUs Are Fast — The Execution Model and the Limits of Parallelism
CPUs and GPUs do not mean the same thing by fast. Where the transistor budget goes, how SIMT bundles 32 threads into a warp, why branch divergence costs you, occupancy and register pressure — and finally Amdahl's law as a way to bound the payoff before you start, plus the profiler counters that tell you when the CPU is the bottleneck.
07
·Numerical Computing·★ MEMBER·8 min read
The Cost of Matrix Multiplication — Where Almost All of AI's Compute Goes
Why GEMM is everything: the anatomy of O(n³), memory bandwidth and arithmetic intensity, what actually makes a GPU fast, and the intuition behind tiling — ending with you able to estimate a model's training and inference FLOPs yourself.