PTQ works for big models but collapses on small models and low bits (down to 13.6% at 2-bit). The fix is QAT: folding quantization into training itself. We insert fake quantization into the forward pass and punch through round’s zero-gradient problem with the STE (Straight-Through Estimator). If round has zero derivative, how does anything learn? Answered with plots measured directly on an MNIST MLP.
FP4 has only 15 representable values, so how do modern GPUs run inference in 4 bits? The culprit that breaks low-bit quantization is the outlier. From an experiment where a single outlier erases an entire layer, to four cures, group quantization (MXFP4), KL clipping, AdaRound, and Hadamard rotation, we verify each in code, including an experiment where 2-bit accuracy comes back from 53% to 97%.
Starting from LeCun’s 1989 Optimal Brain Damage, this is the story of shrinking a model by cutting weights away (pruning). What to cut, how, and how much; the surprise of compression that survives even a 95% cut thanks to retraining; and the point that you only get a real payoff when the hardware (NVIDIA 2:4) backs you up, all with plots we measured ourselves on an MNIST MLP.
We take the trained 32-bit weights of a neural net and shrink them down to 2–8 bits using two approaches: K-Means (non-uniform) and Linear (uniform, integer arithmetic). From the affine mapping r=S(q−Z) all the way to the compression-ratio vs. accuracy trade-off, with plots measured by actually running the code on an MNIST MLP.