Paper walkthrough: Puro-2B — pretraining a 2B model from scratch for $6.9K on consumer GPUs
A team ran 1.4 trillion tokens of pretraining on gaming GPUs and reached Qwen2-1.5B-level quality for roughly $4.4K. Here is the cost structure, the FP8 accounting, the effective learning rate, and the curriculum averaging — from first principles.
Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090
Primary source — what this article is built on
undefined2026-08-27→undefined2026-09-03same month
Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090Kairong Luo, Jiarui Cui, Yaorui Yin et al. · 2026-08-27 · v1arXiv:2608.27370Paper page·PDFundefined
Language model pretraining has become almost synonymous with prohibitive cost, placing it out of reach for much of the academic and open-source communities. Although strong open-source efforts already exist, including open-weight models and open-source training recipes, a cost-efficient, hardware-accessible, and open-source pretraining recipe has long been missing. Even at a small scale, training Llama-3.2-3B costs over \$1.5M, and reproducing SmolLM3-3B needs over \$700K. In this report, we present an open pretraining recipe designed to lower this barrier. Using this recipe, we train a collection of Puro-2B models from scratch on up to 1.4 trillion tokens with FP8 precision on consumer-grade RTX 5090 GPUs. The models in the collection differ in token budgets and selected recipe variants. Our best model is trained at a compute cost of less than \$6.9K and approaches Qwen2.5-1.5B performance under our evaluation protocol. This cost efficiency is enabled by a combination of approaches, including hardware selection, low-precision training, hyperball optimization, curriculum model averaging, and the data recipe. Beyond the recipe itself, we provide two additional results. First, across the Puro-2B collection, we derive a Puro Cost Scaling Law that relates training cost to average model performance; the fitted law suggests that about \$4.4K, less than \$5,090, is sufficient to reach the performance of Qwen2-1.5B. Second, as an end-to-end case study, we examine how pretraining data curricula shape downstream performance after post-training. Such controlled studies are enabled by having access to the full pretraining pipeline rather than model weights alone. We release the full training recipe for Puro-2B, including data, code, and model weights under Apache 2.0 at https://huggingface.co/collections/thu-pacman/puro-2b.
An argument against "pretraining is only for the rich"
The paper's original title is **"Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within 5090" is a pun on the RTX 5090 GPU and a dollar figure.
The abstract makes this case. Pretraining a language model has become synonymous with prohibitive cost, which puts it out of reach for much of the academic and open-source world. Strong open efforts exist — open weights, open training recipes — but a pretraining recipe that is simultaneously cheap, runnable on hardware you can actually buy, and open has been missing. So the authors trained a collection of Puro-2B models from scratch on up to 1.4 trillion tokens, in FP8, on consumer-grade RTX 5090 GPUs. Their best model costs under $6.9K in compute and approaches Qwen2.5-1.5B under their evaluation protocol. Fitting a "Puro Cost Scaling Law" that relates training cost to average performance suggests that about $4.4K — under $5,090 — is enough to reach Qwen2-1.5B. Data, code, and weights are released under Apache 2.0.
Start with the actual numbers
Under the paper's rental-equivalent accounting, reproducing Llama-3.2-3B costs over $1.5M, the open-recipe OLMoE-1B-7B costs $200K, and SmolLM3-3B costs $719K (§1). The Puro-2B production run, by contrast, is Phase 1 on 24 GPUs for 438.84B tokens and Phase 2 on 96 GPUs for 959.99B tokens — 22,514 GPU-hours, about $6.89K, and 17.6 elapsed days in total (§2.1). The lighter variant, which stops Phase 2 at 480B tokens, lands at $4.37K. That is two orders of magnitude of difference, and none of it is magic: each separate contributor to the bill was attacked separately.
A quick sanity check gives you the feel for the scale. Training a 2B-class model on roughly 1.4 trillion tokens works out to about FLOP of theoretical compute; divide that by the measured throughput we will meet shortly (a median 238 TFLOP/s per GPU) and you land near 20,000 GPU-hours — the same order as the 22,514 they actually spent. When your estimate and the measurement differ by an order of magnitude, you have almost always misread either the effective throughput or the token count. The same arithmetic works on your own plan.
Cost decomposes into price times time
Here is theoretical training compute, estimated with the usual convention the paper also uses, where is the parameter count and is the number of training tokens. Stated in words, the formula says there are four separate doors to a lower bill: cut the unit price, stop leaving the GPU idle, cut the token count needed to reach a given quality, and cut the execution time per token. The paper frames it the same way — data selection reduces the required token budget, optimization and curriculum extract more capability per token, FP8 raises hardware utilization, and hardware choice lowers the price of compute (§1).
The first three doors are intuitive. The one that surprises people is that "tokens needed to reach a given quality" depends heavily on learning rate design. The learning rate is the size of your step down the valley: too large and you leap over the bottom and diverge, too small and you never arrive. That design choice converts directly into compute, and therefore into money.
Lever 1: pick a gaming GPU
The authors deliberately chose a consumer RTX 5090 cluster over data-center accelerators (§3.1.1). On peak numbers the 5090 loses badly — BF16 209.5 / FP8 419 TFLOPS versus the H200's 989.5 / 1979 — with only 32 GB of memory and no NVLink. But its effective price is in a different league ($0.31/h against $4.00/h), which gives roughly 2.7× the compute per dollar of an H200. The paper is explicit that this $0.31/h is not a public rental price: it is an estimate obtained by amortizing their own hardware and electricity over five years.
The price you pay is communication. NVIDIA disables PCIe peer-to-peer on non-data-center GPUs at the driver level, forcing inter-GPU traffic through host memory. The authors used a publicly available modified driver plus platform changes (disabling IOMMU and PCIe ACS, adjusting NUMA Per Socket) to enable it, raising bidirectional bandwidth from 32 to 111 GB/s (§3.1.2). On the software side they dropped tensor parallelism, whose collectives fire inside every layer, and used only data and pipeline parallelism — assigning fewer Transformer layers to the pipeline stage that carries the compute-heavy LM head. The result is a median 238 TFLOP/s per GPU and roughly 73% MFU under their mixed-precision convention (§3.1.3).
The same passage tells you where the setup stops working. Dropping tensor parallelism is only an option while the layer weights and optimizer state still fit in 32 GB; past that point the thin interconnect turns directly into an inability to train at all. So the practical boundary on this trick is "as large as pipeline parallelism alone can carry" — which is a good reason not to transplant this paper's conclusion to a 10B-class run.
Lever 2: use blockwise FP8 from step zero
There is no BF16 warm-up and no later precision switch: FP8 is used from random initialization. Only the Transformer linear-layer GEMMs run in FP8; master weights, optimizer state, and core attention stay in BF16/FP32. FP8 is a compute format, not a storage format (§3.2). To stop outliers from destroying the quantization interval, scales are computed for local groups rather than whole tensors — activations and their gradients in groups of 128 along the reduction dimension, weights in 128×128 blocks.
Comments
Sign in to comment