gptq

Definition

A post-training quantization method that compresses the weights of a large language model to three or four bits in a single pass, without retraining. It proceeds layer by layer, rounding weights in order while using approximate second-order information to compensate for the error each rounding introduces, which preserves accuracy far better than naive rounding. Unlike quantization-aware training it touches only finished weights.