https://blog.caveduck.io/tags/kl-divergence/Recent content in KL-Divergence on Warpspace BlogHugo -- 0.154.5en-USMon, 04 May 2026 00:00:00 +0000https://blog.caveduck.io/posts/low-bit-quantization/Mon, 04 May 2026 00:00:00 +0000https://blog.caveduck.io/posts/low-bit-quantization/FP4 has only 15 representable values, so how do modern GPUs run inference in 4 bits? The culprit that breaks low-bit quantization is the outlier. From an experiment where a single outlier erases an entire layer, to four cures, group quantization (MXFP4), KL clipping, AdaRound, and Hadamard rotation, we verify each in code, including an experiment where 2-bit accuracy comes back from 53% to 97%.