#fp8
3 articles
01
·Large Language Models·★ MEMBER·PAPER·13 min read
Paper walkthrough: Puro-2B — pretraining a 2B model from scratch for $6.9K on consumer GPUs
A team ran 1.4 trillion tokens of pretraining on gaming GPUs and reached Qwen2-1.5B-level quality for roughly $4.4K. Here is the cost structure, the FP8 accounting, the effective learning rate, and the curriculum averaging — from first principles.
02
·Training & Alignment·★ MEMBER·PAPER·10 min read
Mixed Precision Training — Going Faster in fp16/bf16/fp8 Without Breaking
Halve the bits and training gets faster — right up until your gradients quietly turn into zeros or infs. We start from the two ways it breaks, work out what loss scaling actually does, why bf16 made it unnecessary, and what came back with fp8. Ends with the PyTorch AMP knobs and the clipping-order mistake that fails silently.
03
·Accelerators·FREE·9 min read
How Numbers Are Represented — From FP32 to FP8 and INT4
Open up the sign, exponent and mantissa fields and one rule falls out: the exponent buys range, the mantissa buys precision. From there — why bfloat16 was invented, why FP8 ships in two flavours, and what integer quantization actually destroys. Ends with which format to pick for training versus inference, and how to notice degradation before your users do.