Warpspace runs a series of experiments to make models smaller so we can run Caveduck.io on-device and at low cost. This is the first post (#1, data types) of that Efficient ML series. The follow-ups - #2 Quantization and #3 Pruning - take the data types covered here, shrink an actual model, and measure the results firsthand.
Why did we end up with so many data types?
Just a few years ago, float32 alone was enough for all the numbers in deep learning. But look at papers or model cards these days and you get a flood of data types: FP16, BF16, FP8 (E4M3/E5M2), INT8, INT4, FP4, NF4… From the names alone it’s hard to tell what’s what, or why there are so many.
The reason is simple: low-bit arithmetic is just cheap. On a 45nm process, the energy cost per operation looks roughly like this.
| Operation | Energy (pJ) |
|---|---|
| 8-bit int ADD | 0.03 |
| 32-bit int ADD | 0.1 |
| 32-bit float ADD | 0.9 |
| 8-bit int MULT | 0.2 |
| 32-bit float MULT | 3.7 |
An 8-bit integer multiply is about 18x cheaper than a 32-bit float multiply, and addition is 30x cheaper. On top of that, the cost of reading a single value from memory is hundreds of times more expensive than one multiply or add. So for the same model, the fewer bits you pack a number into, the more power, memory, and bandwidth you save. All those data types above are ultimately different answers to the question: “how many bits, and where do we spend them?”
The catch is that fewer bits means fewer distinct numbers you can represent, and you lose that much precision or range. So each format’s character comes down to how it splits its bits among sign, exponent, and fraction. This post takes apart those allocation rules — how a bit pattern actually gets interpreted as a number — one at a time.
For each data type, I’ve included a widget where you can click the bits directly. Press a 0 or 1 to flip it and the formula and result update in real time. Rather than reading an explanation with your eyes, actually poking at the bits will give you a much faster feel for it.
1. Integers
Unsigned Integer
The simplest one. Each of the $n$ bits carries a place value of $2^k$, and you sum the place values of every bit that’s on ($=1$).
$$\text{value} = \sum_{i=0}^{n-1} b_i \cdot 2^i$$The representable range is $[0,\ 2^n - 1]$. For 8 bits, that’s 0 to 255.
Try pressing the bits in the widget below. The default 00110001 is $2^5 + 2^4 + 2^0 = 49$.
Signed Integer
To represent negative numbers, you need a sign. There are two approaches.
Sign-Magnitude — use the leading bit as the sign ($0$=positive, $1$=negative) and the rest to represent the magnitude. Intuitive, but it has a fatal flaw: 00000000 and 10000000 are both zero (+0 and −0). Two zeros make the hardware more of a hassle. The range is $[-2^{n-1}+1,\ 2^{n-1}-1]$.
Two’s Complement — what modern computers actually use. The idea is simple: make the leading bit’s place value negative. That is, the MSB’s weight is $-2^{n-1}$ rather than $+2^{n-1}$.
$$\text{value} = -b_{n-1}\cdot 2^{n-1} + \sum_{i=0}^{n-2} b_i \cdot 2^i$$This way there’s only one zero, 00000000, and the range $[-2^{n-1},\ 2^{n-1}-1]$ reaches one step further on the negative side. Below, 11001111 is $-2^7 + 2^6 + 2^3 + 2^2 + 2^1 + 2^0 = -49$. Try turning off the MSB and see how it changes.
2. Fixed-Point
Integers alone can’t hold fractional values. The easiest extension is to fix the position of the decimal point. You just read the bit string as a two’s complement integer, then scale it by a fixed $2^{-f}$.
$$\text{value} = (\text{value read as integer}) \times 2^{-f}$$Below, 8 bits are split into a 4-bit integer part and a 4-bit fractional part ($f=4$). Each bit’s weight runs $2^3, 2^2, \dots, 2^0, 2^{-1}, \dots, 2^{-4}$. Read as an integer, 00110001 is 49, and $49 \times 2^{-4} = 3.0625$.
Fixed-point is simple but its limits are clear. Because the decimal point is fixed, the range of magnitudes it can represent is narrow. It struggles to handle very large and very small numbers at the same time. Solving that problem is the job of our next protagonist: floating-point.
3. Floating-Point — IEEE 754
Floating-point is exactly what the name says: the decimal point floats. Think of it as the binary version of scientific notation ($1.5 \times 10^3$). The bits are split into three parts.
- Sign — 1 bit
- Exponent — sets the scale of the magnitude → determines the range
- Fraction — sets the fine detail → determines the precision
The formula for interpreting a normal number is this.
$$\text{value} = (-1)^{\text{sign}} \times (1 + \text{Fraction}) \times 2^{\text{Exponent} - \text{bias}}$$Here bias is an offset that lets the exponent represent negative values too, equal to $2^{e-1}-1$ ($e$ = number of exponent bits). And note the $(1 + \cdots)$ in front of the fraction: a normal number always has an implicit 1 at the front (this is called the “implicit leading 1”).
Hold on — a note on terminology: Fraction / Mantissa / Significand
The terminology here gets mixed up a bit across the literature, so it’s easy to confuse. The proper name for the whole $1.\text{Fraction}$ in the formula above — the significant-digits part — is the Significand.
- Fraction — refers only to the fractional part actually stored in the bits. The bits shaded yellow in the widget are exactly this. It’s the sub-decimal value $0.\text{b}_1\text{b}_2\cdots$.
- Significand — the whole $1.\text{Fraction}$ including the implicit 1. This is the significant value actually multiplied in.
- Mantissa — historically the term for the fractional part of a logarithm table, but in floating-point it’s usually used loosely to mean the same thing as Fraction. The IEEE 754 standard itself uses “Significand” as the official term and discourages “mantissa,” but in the field “mantissa” is still common.
To summarize: Significand = 1 + Fraction, and when people casually say “mantissa” they usually mean the stored Fraction. In this post we’ll call the field stored in the bits the Fraction.
Below is 32-bit single precision (FP32). Sign 1 + exponent 8 + fraction 23 = 32 bits, with bias 127. The default represents $0.265625 = (1 + 0.0625) \times 2^{125-127}$. Try pressing the exponent bits one at a time and watch the value jump by a factor of 2 each time.
Special values: 0, infinity, NaN, and subnormals
There’s one odd thing about the formula. Because of $(1 + \text{Fraction})$, a normal number can’t represent 0. So IEEE 754 uses the exponent field as a special signal.
| Exponent | Fraction = 0 | Fraction ≠ 0 | Interpretation |
|---|---|---|---|
00…0 (=0) | $\pm 0$ | subnormal | $(-1)^s \times \text{Fraction} \times 2^{1-\text{bias}}$ |
00…1 ~ 11…0 | normal | normal | $(-1)^s \times (1+\text{Fraction}) \times 2^{\text{Exp}-\text{bias}}$ |
11…1 (=max) | $\pm\infty$ | NaN | — |
The key is when the exponent is all zeros. In that case you drop the implicit 1 ($1+\text{Fraction} \to \text{Fraction}$) and fix the exponent at $2^{1-\text{bias}}$. What this produces are subnormal (denormal) numbers, which densely fill in the very small values near 0. Conversely, when the exponent is all ones, you get infinity (fraction 0) and NaN (fraction ≠ 0).
Try building these yourself in the FP32 widget above.
- Set the exponent to all ones (
0 11111111 0…0) →+∞ - From there, turn on any fraction bit →
NaN - All zeros →
0 - Leave the exponent at 0 and turn on a few fraction bits → a tiny subnormal value
The wider the exponent, the wider the range; the wider the fraction, the higher the precision. Exponent → Range, Fraction → Precision. This one line is the core trade-off behind the design of every low-precision format that follows.
4. Half the size: FP16 and BF16
FP32 is accurate, but it eats a full 32 bits. Deep learning often doesn’t need that much precision, so we use 16-bit formats. Same 16 bits, but the split comes down to where you allocate them.
FP16 (IEEE 754 Half Precision)
Exponent 5 + fraction 10. Bias 15. An allocation that invests more in precision (the fraction). Below is $1\,10001\,1100000000$, i.e. $-(1+0.75)\times 2^{17-15} = -7.0$.
BF16 (Google Brain Float)
Exponent 8 + fraction 7. Bias 127. It has the same total bit count as FP16, but keeps the same 8-bit exponent as FP32. That means its range is identical to FP32, at the cost of precision. It’s popular because it doesn’t have to worry about overflow in situations where the scale of values swings wildly, like gradients during training.
Below is $2.5 = (1 + 0.25)\times 2^{1}$ represented in BF16 ($0\,10000000\,0100000$).
Get a feel for the difference in exponent bit count across the two widgets. BF16 has 8 exponent cells, so it reaches very large/small numbers, but with only 7 fraction cells its values are sparse. FP16 is the opposite.
5. Going lower: FP8 and FP4
FP8 (E4M3 / E5M2)
8-bit floating-point is supported by the latest hardware (e.g. Nvidia Hopper/Blackwell). Two allocations are used almost like standards.
- E4M3 — exponent 4 + fraction 3. Precision-first. Mainly used for weights and activations in the forward pass. It has no INF and uses only
S.1111.111as NaN. The largest representable normal value is $448$. - E5M2 — exponent 5 + fraction 2. Range-first. Used for large-scale values like gradients in the backward pass. It has INF and NaN like IEEE.
Starting with E4M3. Bias 7. The default 0 0111 000 is $(1+0)\times 2^{7-7} = 1.0$.
E5M2. Bias 15. With 5 exponent cells it holds a much wider range, but with only 2 fraction cells the spacing between values is coarse.
INT4 and FP4
The extreme end. With 4 bits, you can represent only 16 values. How those 16 are laid out differs from format to format.
INT4 — two’s complement integer. Placed at uniform spacing from $-8$ to $7$.
FP4 distributes its values differently depending on the exponent/fraction split. The more exponent bits, the more the values pack in near 0 and spread out toward the edges, i.e. non-uniformly.
- E1M2 — exponent 1 + fraction 2. Closest to an integer (bias 0).
0111= $(1+0.75)\times 2^{1-0} = 3.5$.
- E2M1 — exponent 2 + fraction 1 (bias 1).
0111= $(1+0.5)\times 2^{3-1} = 6$.
- E3M0 — exponent 3 + fraction 0 (bias 3). With no fraction, it effectively represents only powers of 2.
0111= $(1+0)\times 2^{7-3} = 16$. Press the random button and watch the values spread out exponentially, like $\dots, 4, 8, 16$.
Plotting the “16 representable values” of these four formats on a single number line makes the differences in character jump right out. INT4 is evenly spaced; FP4 packs values near 0 and spreads them at the edges the more exponent bits you add.
Here’s one interesting observation. The weight distribution of a neural network is usually clustered near 0 with long tails. If so, then rather than the INT approach of laying out values uniformly, an FP-style format that densely fills the region near 0 can fit the actual values better with the same 4 bits. Which format suits which distribution — that’s why choosing a data type leads directly to accuracy. (We measure this observation on a real model in the quantization part.)
Wrapping up
That’s the story of what the data types you run into in deep learning actually mean. To summarize:
- Integer / fixed-point — values at uniform spacing. Simple, but narrow range.
- Floating-point — split its bits between the exponent for range and the fraction for precision. Dense near 0, sparse at the edges.
- Reducing bits = reducing the number of representable values. From FP32’s roughly 4.3 billion down to FP4’s 16.
Now when you see names like FP16, BF16, FP8 E4M3, or INT4, you’ll be able to picture the bit layout in your head. In deep learning, “which data type should I use?” is no longer a trivial implementation detail but a design choice that decides how small and fast — yet how accurate — you can run a model.
In the next post, we take these data types as our weapons and actually shrink an already-trained 32-bit model down to 2–8 bits — running the code ourselves to measure how much accuracy is preserved and how much the size shrinks.
👉 Neural Net Quantization: Shrinking a 32-bit model down to 2 bits