#floating-point
2 articles
01
·Numerical Computing·★ MEMBER·10 min read
Numerical Pitfalls — Cancellation, Rounding, and logsumexp
Where "the loss went nan three hours into the run" actually comes from, built up from nothing: how rounding enters, how the condition number amplifies it, and why subtracting two close numbers is so destructive. It all converges on logsumexp — the one trick sitting inside every softmax and cross-entropy implementation.
02
·Accelerators·FREE·9 min read
How Numbers Are Represented — From FP32 to FP8 and INT4
Open up the sign, exponent and mantissa fields and one rule falls out: the exponent buys range, the mantissa buys precision. From there — why bfloat16 was invented, why FP8 ships in two flavours, and what integer quantization actually destroys. Ends with which format to pick for training versus inference, and how to notice degradation before your users do.