mixed

Definition

Combining more than one kind of something. The dominant usage is mixed-precision training: matrix multiplications run in FP16 or BF16 for speed and memory, while a master copy of the weights and certain reductions stay in FP32 to preserve numerical stability. This is standard practice on modern accelerators. Distinguish it from mixture, which in this literature usually refers to how training data sources are blended.

Explainers using this term

• Mixed Precision Training — Going Faster in fp16/bf16/fp8 Without BreakingMixed Precision Training