https://blog.caveduck.io/tags/ste/Recent content in STE on Warpspace BlogHugo -- 0.154.5en-USMon, 11 May 2026 00:00:00 +0000https://blog.caveduck.io/posts/quantization-aware-training/Mon, 11 May 2026 00:00:00 +0000https://blog.caveduck.io/posts/quantization-aware-training/PTQ works for big models but collapses on small models and low bits (down to 13.6% at 2-bit). The fix is QAT: folding quantization into training itself. We insert fake quantization into the forward pass and punch through round’s zero-gradient problem with the STE (Straight-Through Estimator). If round has zero derivative, how does anything learn? Answered with plots measured directly on an MNIST MLP.