Efficient ML #4, Low-bit Quantization: Fighting Outliers
FP4 has only 15 representable values, so how do modern GPUs run inference in 4 bits? The culprit that breaks low-bit quantization is the outlier. From an experiment where a single outlier erases an entire layer, to four cures, group quantization (MXFP4), KL clipping, AdaRound, and Hadamard rotation, we verify each in code, including an experiment where 2-bit accuracy comes back from 53% to 97%.