Half precision halves the memory and doubles the throughput, and silently rounds small gradients to zero. Watch it happen, then fix it.
Overview
Why bother
Modern accelerators run half-precision mat times a few times faster than single-precision ones, and half-precision tensors take half the memory. That means bigger batches, bigger models, and a real speed-up on the same hardware.
The catch is the format. fp32 has 8 exponent bits and 23 mantissa bits. fp16 has 5 and 10, giving a smallest normal value around 10⁻⁵. And a max of 65,504.
Both ends of that range are a problem. The small end is the one that bites.
Mixed Precision and Loss Scaling
This module needs JavaScript: the
numbers are computed in the page rather than recorded.
Worth knowing
fp16 has 5 exponent bits and 10 mantissa bits. Its smallest representable value is about 6×10−8.
Gradients are routinely smaller than that. They round to exactly zero, and the parameter never moves.
Loss scaling multiplies the loss before the backward pass, shifting every gradient up into the representable range.
The optimiser unscales before stepping, so the update is unchanged. The scaling exists only to survive the trip through fp16.
Mixed Precision and Loss Scaling
Training in half the bits, the failure that causes. The one-line trick that drops it.
Watch the gradients disappear
The histogram is grad magnitudes spread over a few orders of magnitude, which is what a real network makes. The two marked lines are fp16's floor and ceiling.
With the loss scale at 2⁰.. That is, no scaling — the readout reports how many of the 3,000 grads round to zero. Those params receive no update at all. Not a small update. None.
This is the failure that made early fp16 training unreliable. The model trains, the loss goes down, and a fraction of the network is silently frozen.
Loss scaling
The fix is almost embarrassingly simple. Times the loss by a large constant before the backward pass. By the chain rule each grad is times'd by the same constant, so the whole spread shifts up into the representable range. Then divide by that constant before the tuner steps.
Drag the loss scale upward and watch the underflow count fall to zero.
The update is in math unchanged. The scaling exists purely so the numbers live the trip through fp16. It's a change of units, not a change of method.
Push it too far and the readout starts reporting overflow: grads that have become infinite. That is a failure too, and it's where the auto version comes from.
Dynamic loss scaling
No one tunes the constant by hand. The standard method starts high, times by two when a stretch of steps ends without overflow, and halves it and skips the step when an infinity or NaN appears.
Skipping is the key part: an overflowed grad isn't clipped or repaired, it's discarded, because there's no way to recover what it should have been. A few skipped steps early in training cost nothing.
This is what torch.cuda.amp.GradScaler does, and it's why mixed precision is now a decorator rather than a project.
What stays in fp32
"Mixed" is the operative word. In a standard setup:
Weights are kept in an fp32 master copy. Updates are often far smaller than the weights themselves, and adding a tiny number to a large one in fp16 rounds to no change at all.
mat times and convolutions run in fp16. Here The speed is.
Reductions — sums, means, softmax, norming statistics — build up in fp32, because adding many small numbers is just where limited mantissa bits go wrong.
Loss is found in fp32.
bfloat16
The newer option has 8 exponent bits and 7 mantissa bits — the same range as fp32, with less precision.
That trade drops this entire page. Range was the problem. Precision, for grads, mostly wasn't. bfloat16 needs no loss scaling. This Is why it is the default on hardware that supports it, and why large-model training largely stopped talking about GradScaler.
The range that makes loss scaling necessary
The failure is fully about fp16's floor. fp16 carries 5 exponent bits, giving a smallest normal value around 2^-14 ≈ 6.1 × 10^-5 and a max of 65,504. Real grads routinely sit *below* that floor:
quantity
magnitude
in fp16
a typical small gradient
~1 × 10^-7
rounds to zero
the same, times a loss scale of 4096
~4 × 10^-4
represented fine
A grad that rounds to zero isn't a small update, it's *no* update — the param freezes, silently, while the loss still falls and the run looks healthy. Loss scaling fixes it in one line: times the loss by a constant before the backward pass, so by the chain rule each grad is times'd too and the whole spread shifts up into the representable range. Divide it back out before the tuner steps. The update is in math unchanged — it's a change of units, not of method. Push the constant too high and grads overflow to infinity instead. This Is why the standard scaler is *dynamic*: start high, double it after stretches without overflow, and halve it and skip the step when an infinity appears. bfloat16 sidesteps the whole page by keeping fp32's 8 exponent bits — same range, less precision — so it needs no scaling at all.
The other half of "mixed" is what on purpose stays in fp32. Weights keep an fp32 master copy, because an update is often far smaller than the weight it adjusts and adding a tiny number to a large one in fp16 rounds to no change at all. Reductions — sums, means, softmax, norming statistics — build up in fp32, because adding many small numbers is just where ten mantissa bits go wrong and give the mysterious NaNs that half-precision softmax is known for. The fp16 lives where the speed is: the mat times and convolutions the tensor cores accelerate. Layers that are memory-bound rather than find-bound gain little, which is why the speed-up is real but never uniform across a model.
Where it goes wrong with Mixed Precision and Loss Scaling
fp16 without loss scaling. Silent underflow, and a partly frozen model.
A fixed scale. Use the dynamic version.
Reductions in fp16. Softmax and layer norm in half precision give mysterious NaNs.
given a speed-up. It comes from the tensor cores. Layers that are memory-bound rather than find-bound gain little. Profile before expecting a win.
Questions people ask about Mixed Precision and Loss Scaling
What is mixed precision training? Running most operations in 16-bit floats while keeping a 32-bit master copy of the weights. It saves memory and speeds up GPUs with tensor cores.
Why is loss scaling needed? Small gradients underflow to zero in fp16. Multiplying the loss by a large factor shifts the gradients into representable range, then they are divided back.
What is the difference between fp16 and bfloat16? bfloat16 keeps the exponent range of fp32 but fewer mantissa bits, so it rarely overflows and usually needs no loss scaling. fp16 has more precision but a narrow range.
Which operations stay in fp32? Reductions, softmax, loss computation and the optimiser updates. They are sensitive to rounding and overflow.
What are signs that it is going wrong? NaN losses, loss scale collapsing to a tiny value, or accuracy dropping below the fp32 baseline.
Will it change my accuracy? Usually not. With a master copy and loss scaling, results match fp32 within normal run-to-run noise.
Half the bits, and the two things that break
float16 halves memory and doubles throughput on tensor cores. It also has a much narrower range than float32. Grads live near the bottom of that range -- which is the entire reason loss scaling exists.
example_01.pyNumPy
import numpy as np
def limits(dt):
f = np.finfo(dt)
return f.max, f.tiny, f.eps
print("%12s %8s %14s %16s %14s"
% ("type", "bytes", "largest", "smallest normal", "precision"))
for dt in (np.float64, np.float32, np.float16):
mx, tiny, eps = limits(dt)
print("%12s %8d %14.3e %16.3e %14.3e"
% (np.dtype(dt).name, np.dtype(dt).itemsize, mx, tiny, eps))
print()
print("float16 runs out of range at 65504. below its smallest NORMAL value")
print("of 6.1e-05 it keeps going in subnormals, with steadily fewer digits,")
print("until it reaches zero at %.2e:" % float(np.finfo(np.float16).smallest_subnormal))
for v in (1e-4, 1e-5, 1e-6, 1e-7, 3e-8, 1e-8):
print(" %.0e as float16 -> %.3e%s"
% (v, float(np.float16(v)),
" <- gone" if np.float16(v) == 0 else ""))
print()
print("that bottom edge is the problem: gradients routinely live there.")
Output
The next block shows what happens when a realistic gradient distribution meets that floor, and how loss scaling fixes it.
example_02.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
grads = np.abs(rng.lognormal(mean=-17.5, sigma=1.6, size=200_000))
print("a realistic gradient distribution meets float16:")
print(" gradient magnitudes: median %.3e, 5th percentile %.3e"
% (np.median(grads), np.percentile(grads, 5)))
lost = (grads.astype(np.float16) == 0).mean()
print(" fraction that become EXACTLY ZERO when cast to float16: %.1f%%"
% (100 * lost))
print(" those gradients are not small any more. they are gone, and the")
print(" weights they belonged to will not update at all.")
print()
print("LOSS SCALING is the fix, and it is almost embarrassingly simple:")
print("multiply the loss by a large constant before the backward pass, then")
print("divide the gradients by the same constant before the update.")
print()
for scale in (1, 256, 4096, 65536, 1_048_576):
scaled = (grads * scale).astype(np.float16)
survived = (scaled != 0).mean()
overflowed = (~np.isfinite(scaled)).mean()
print(" scale %9d: %6.1f%% of gradients survive, %5.2f%% overflow to inf"
% (scale, 100 * survived, 100 * overflowed))
print()
print("the whole gradient distribution slides up into float16's usable range.")
print("dividing by the same constant afterwards restores the true values")
print("exactly -- multiplication by a power of two only changes the exponent,")
print("so it costs no precision at all:")
x = np.float16(0.000123)
print(" 0.000123 as float16 : %.9f" % float(x))
print(" x * 1024 / 1024 back again : %.9f" % float(np.float16(x * 1024) / 1024))
Output
The next block shows why the scale must be dynamic, and the second failure mode: accumulation.
example_03.pyNumPy
import numpy as np
print("too large a scale overflows, which is why the scale is DYNAMIC:")
print(" start high (2^16), and on any step that produces inf or nan:")
print(" skip the update entirely, halve the scale")
print(" after N good steps in a row: double it")
print(" the scale ends up tracking the largest value your gradients")
print(" currently reach, and it moves as training changes them.")
print()
print("the second thing that breaks is accumulation. adding many small")
print("numbers to a large one in float16 loses them completely:")
acc16 = np.float16(1024.0)
small = np.float16(0.4)
for _ in range(64):
acc16 = np.float16(acc16 + small)
print(" float16: 1024 + 0.4 sixty-four times = %.4f (should be %.1f)"
% (float(acc16), 1024 + 64 * 0.4))
acc32 = np.float32(1024.0)
for _ in range(64):
acc32 = np.float32(acc32 + np.float32(0.4))
print(" float32: the same sum = %.4f" % float(acc32))
print(" 0.4 is below the spacing between representable float16 values near")
print(" 1024, so each addition rounds straight back to where it started.")
print()
print("that is why mixed precision keeps a MASTER COPY of the weights in")
print("float32. the forward and backward passes run in float16 for speed;")
print("the optimiser update -- which is exactly this kind of accumulation --")
print("happens in float32, and the float16 weights are refreshed from it.")
Output
The next block shows how bfloat16 sidesteps the range problem entirely.
example_04.pyNumPy
import numpy as np
print("bfloat16 solves the range problem differently -- same 16 bits, more")
print("of them spent on the exponent:")
print("%14s %10s %10s %16s" % ("format", "exponent", "mantissa", "largest"))
print("%14s %10d %10d %16.3e" % ("float32", 8, 23, np.finfo(np.float32).max))
print("%14s %10d %10d %16.3e" % ("float16", 5, 10, np.finfo(np.float16).max))
print("%14s %10s %10s %16s" % ("bfloat16", "8", "7", "~3.4e+38"))
print(" bfloat16 has float32's exponent range, so it does not underflow and")
print(" needs no loss scaling. it pays with precision -- 7 mantissa bits")
print(" against 10 -- which in practice matters far less than the range.")
print(" that is why it is the default on hardware that supports it.")
print()
print("what you actually get: roughly half the activation memory, so bigger")
print("batches; and 2 to 8 times the matmul throughput on tensor cores.")
print("what you should check: that your loss curve matches the float32 run")
print("for the first few hundred steps. if it diverges, the scale is wrong.")
Output
Check yourself
0 of 3
Answer without scrolling back up.
What does loss scaling change about the parameter update?
It is a change of units, not a change of algorithm. Multiply the loss, every gradient scales by the chain rule, divide before stepping.
What does dynamic loss scaling do when a gradient overflows?
There is no way to recover what an overflowed gradient should have been, so it is discarded rather than repaired. A few skipped steps early cost nothing.
Why does bfloat16 not need loss scaling?
bfloat16 trades mantissa bits for exponent bits - less precision, same range. For gradients, precision mostly was not the issue.
Cheat sheet
Mixed Precision and Loss Scaling
Train in float16, see the underflow that zeros out small gradients, then fix it with a loss scale that fits in one extra line of code.
DEEP LEARNING · vizlearn.in/deep_learning/mixed_precision_training.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.