#int8

2 articles

01 ·Inference & Serving·FREE·8 min read LLM Quantization from Scratch — Why Losing Precision Doesn't Break It Cut a 16-bit weight down to 4 bits and the model keeps writing. Why doesn't it break? Starting from two numbers — scale and zero point — through the reason weights survive but activations resist (outliers), the fork between PTQ and QAT, what INT8 and INT4 actually do, and the single most common evaluation accident: signing off on perplexity alone. 02 ·Accelerators·FREE·9 min read How Numbers Are Represented — From FP32 to FP8 and INT4 Open up the sign, exponent and mantissa fields and one rule falls out: the exponent buys range, the mantissa buys precision. From there — why bfloat16 was invented, why FP8 ships in two flavours, and what integer quantization actually destroys. Ends with which format to pick for training versus inference, and how to notice degradation before your users do.