Textbook › Part I The Physics of Semiconductors
CHAPTER 14
Number Formats and Quantisation
The standard way a computer handles fractional numbers is floating point
. A number is split into a sign, an exponent, and a mantissa. A wider exponent means a wider range of values you can represent; a longer mantissa means more precision.
Number formats commonly used in AI
| Format | Bits | Character |
|---|---|---|
| FP32 | 32 | The traditional standard. The reference precision for training |
| FP16 | 16 | Half precision. Narrow range, so values can run off the scale during training |
| BF16 | 16 | Same exponent width as FP32, with the mantissa cut down. Wide range and easy to train with. The mainstream choice for training today |
| FP8 | 8 | Hardware has started supporting it in recent years. It is beginning to be used for training too |
| INT8 | 8 | Integer. The de facto standard for edge inference |
| INT4 and below | 4 or fewer | In practical use for large language model inference. A constant fight against accuracy loss |
Comments
Sign in to comment