#gpu

8 articles

01 ·Paper Deep-Dives·★ MEMBER·PAPER·10 min read Paper Explained: RayOrch — Batching Children Across Parents on GPUs Without Losing Track of Who Owns What Split PDFs into pages and videos into clips, batch them across sources on GPUs, and still route every result back to the right parent in the right order. A from-scratch walkthrough of RayOrch: its vocabulary, state machine, commit rule, and measured results. 02 ·Inference & Serving·FREE·PAPER·11 min read Surviving GPU Out-of-Memory — Every Cause, Every Fix `CUDA out of memory` reports only the allocation that happened to fail last, which is almost never the culprit. We count what actually occupies VRAM in five buckets, derive the sixteen-bytes-per-parameter fixed cost of training, and work through the fixes in order of least damage: gradient checkpointing, optimizer compression, offloading, KV cache limits, and fragmentation. 03 ·How Transformers Work·FREE·PAPER·10 min read FlashAttention from Scratch — The Paradox of Doing More Math to Go Faster FlashAttention deliberately recomputes the same math in the backward pass. It does strictly more arithmetic and still runs faster — because on a GPU, moving numbers costs more than multiplying them. From counting HBM round trips, through the softmax wall that blocks tiling, to the online-softmax recurrence that breaks it, and the silent backend fallback that bites people in production. 04 ·Cloud & Ops·★ MEMBER·10 min read The Economics of GPU Cloud — Rent, Buy, or Commit The same GPU carries four prices at once. Put on-demand, committed, spot, and owned hardware on a single equation: the break-even utilization, the true cost of spot once interruptions are priced in, the commitment burn rate, and the line items that never appear on the rate card. 05 ·Computer Architecture·FREE·PAPER·9 min read The GPU Memory Hierarchy — HBM, SRAM, Registers, and Why Movement Wins What sets a GPU's speed is not the arithmetic units but where the data sits — registers, shared memory, L2 or HBM — and how many times it is moved. Capacities and bandwidths by order of magnitude, arithmetic intensity and tiling, a roofline per level of the hierarchy, and finally FlashAttention: more FLOPs, less time. 06 ·Accelerators·★ MEMBER·PAPER·8 min read How AI Accelerators Are Designed — What Actually Separates GPUs, NPUs, and TPUs GPUs, TPUs, and NPUs are three different answers to one question: how much generality do you trade away for matrix multiplication? A from-scratch tour of systolic arrays, dataflow design, and the co-evolution of hardware and quantization. 07 ·Parallel & Distributed·FREE·10 min read Why GPUs Are Fast — The Execution Model and the Limits of Parallelism CPUs and GPUs do not mean the same thing by fast. Where the transistor budget goes, how SIMT bundles 32 threads into a warp, why branch divergence costs you, occupancy and register pressure — and finally Amdahl's law as a way to bound the payoff before you start, plus the profiler counters that tell you when the CPU is the bottleneck. 08 ·Numerical Computing·FREE·8 min read The Cost of Matrix Multiplication — Where Almost All of AI's Compute Goes Why GEMM is everything: the anatomy of O(n³), memory bandwidth and arithmetic intensity, what actually makes a GPU fast, and the intuition behind tiling — ending with you able to estimate a model's training and inference FLOPs yourself.