Efficient ML #1: Deep Learning Data Types
INT8, FP16, BF16, FP8 (E4M3/E5M2), FP4: what do all these data types flooding deep learning model cards actually mean, and how do bits get interpreted as numbers? We take them apart one by one, with a widget where clicking bits updates the formula and value in real time.
Efficient ML #2: Neural Net Quantization
We take the trained 32-bit weights of a neural net and shrink them down to 2–8 bits using two approaches: K-Means (non-uniform) and Linear (uniform, integer arithmetic). From the affine mapping r=S(q−Z) all the way to the compression-ratio vs. accuracy trade-off, with plots measured by actually running the code on an MNIST MLP.
Efficient ML #3: Neural Net Pruning
Starting from LeCun’s 1989 Optimal Brain Damage, this is the story of shrinking a model by cutting weights away (pruning). What to cut, how, and how much; the surprise of compression that survives even a 95% cut thanks to retraining; and the point that you only get a real payoff when the hardware (NVIDIA 2:4) backs you up, all with plots we measured ourselves on an MNIST MLP.
Efficient ML #4, Low-bit Quantization: Fighting Outliers
FP4 has only 15 representable values, so how do modern GPUs run inference in 4 bits? The culprit that breaks low-bit quantization is the outlier. From an experiment where a single outlier erases an entire layer, to four cures, group quantization (MXFP4), KL clipping, AdaRound, and Hadamard rotation, we verify each in code, including an experiment where 2-bit accuracy comes back from 53% to 97%.
Efficient ML #5, QAT: Undoing Quantization Error During Training
PTQ works for big models but collapses on small models and low bits (down to 13.6% at 2-bit). The fix is QAT: folding quantization into training itself. We insert fake quantization into the forward pass and punch through round’s zero-gradient problem with the STE (Straight-Through Estimator). If round has zero derivative, how does anything learn? Answered with plots measured directly on an MNIST MLP.