Gradient Clipping

What clipping does

if ||g|| > threshold: g ← g · (threshold / ||g||)

The gradient's direction is unchanged — it still points the way steepest descent says to go — only its length is capped. This is norm clipping, the common form; value clipping (capping each component independently) is cruder and used less often.

Capping the size of a step

Gradient clipping is a single guard rail: if the gradient is larger than a threshold, scale it down before applying the update.

if ‖g‖ > c:  g ← g × c / ‖g‖

Note what that does and does not change. The direction is preserved exactly — every component is scaled by the same factor. Only the length is limited. That is why clipping by norm is the standard, and clipping each value independently is not: per-value clipping distorts the direction, so the update no longer points downhill.

loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()

The call must come after backward() and before step(). Placing it anywhere else does nothing, silently.

The failure it prevents

A single unusually large gradient can destroy a model in one step. Suppose the gradient norm is normally around 0.5 and one batch produces 4,000 — a rare example, an outlier, an unlucky interaction. With a learning rate of 1e-3 that is a weight change of 4.0 in one step, which pushes the weights far outside the region where the loss is meaningful. The next forward pass produces inf, the loss becomes NaN, and every subsequent update propagates the NaN through the whole model.

Nothing recovers from that. The run is over, and if it happened at hour six of an eight-hour job, the checkpoint is what saves you.

Clipping makes that impossible. The worst case becomes a step of size learning_rate × threshold, whatever the gradient was.

Choosing the threshold

The useful way to pick it is to measure rather than guess. Log the gradient norm for a few hundred steps without clipping, look at the distribution, and set the threshold somewhere above the typical value but below the outliers.

total = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1e9)
# returns the norm BEFORE clipping, so this logs without actually clipping

Then watch how often it triggers. Occasional clipping — a few percent of steps — is exactly right: the rare spike is caught and normal steps pass through untouched. Clipping on every step means the threshold is too low and you are throttling all learning, which shows up as slow, stalled convergence.

Common values: 1.0 for transformers and most language models, 5.0 for recurrent networks, 0.5 when training is particularly unstable. These are conventions rather than derived numbers.

Cap the step, keep the direction

One bad batch can produce a gradient large enough to destroy a working model in a single update. Clipping bounds the damage, and the difference between the two ways of doing it matters.

import numpy as np

rng = np.random.default_rng(0)

g = np.array([3.0, -4.0, 12.0, 0.5])
print("a gradient:", g)
print("its L2 norm: %.4f" % np.linalg.norm(g))
print()

def clip_norm(grad, max_norm):
    n = np.linalg.norm(grad)
    return grad * min(1.0, max_norm / (n + 1e-12)), n

def clip_value(grad, limit):
    return np.clip(grad, -limit, limit)

print("clip by NORM to 5.0 -- rescale the whole vector:")
c, n = clip_norm(g, 5.0)
print("  scale factor %.4f" % (5.0 / n))
print("  result", np.round(c, 4), " norm %.4f" % np.linalg.norm(c))
print("  direction preserved:", np.allclose(c / np.linalg.norm(c), g / np.linalg.norm(g)))
print()
print("clip by VALUE to 5.0 -- truncate each element:")
cv = clip_value(g, 5.0)
print("  result", cv, " norm %.4f" % np.linalg.norm(cv))
print("  direction preserved:",
      np.allclose(cv / np.linalg.norm(cv), g / np.linalg.norm(g)))
print()
print("clipping by value CHANGED THE DIRECTION. the 12.0 carried %.0f percent"
      % (100 * 12.0 / np.abs(g).sum()))
print("of this gradient's total magnitude, and it was cut to 5.0 while the")
print("0.5 was left alone, so the update now points somewhere the gradient")
print("never pointed.")
print("that is why clip_grad_norm_ is the standard and clipping by value is")
print("a niche tool.")
print()

print("what it is protecting you from. training a small model where one")
print("batch contains an outlier:")
w = np.array([1.0, 1.0])
X = rng.normal(size=(40, 2))
y = X @ np.array([2.0, -1.0]) + rng.normal(0, 0.1, 40)
X_bad = X.copy(); y_bad = y.copy()
X_bad[0] *= 300                      # one corrupted row
y_bad[0] *= 300

for label, Xb, yb, clip in (("clean batch      ", X, y, None),
                            ("corrupted batch  ", X_bad, y_bad, None),
                            ("corrupted + clip ", X_bad, y_bad, 1.0)):
    ww = np.array([1.0, 1.0])
    g2 = 2 * Xb.T @ (Xb @ ww - yb) / len(Xb)
    norm = np.linalg.norm(g2)
    if clip:
        g2 = g2 * min(1.0, clip / norm)
    after = ww - 0.01 * g2
    print("  %s gradient norm %12.2f -> weights %s"
          % (label, norm, np.round(after, 4)))
print("  the corrupted batch moves the weights by %s in one step."
      % np.round(np.abs(np.array([1.0, 1.0])
                        - (np.array([1.0, 1.0]) - 0.01 * 2 * X_bad.T
                           @ (X_bad @ np.array([1.0, 1.0]) - y_bad) / 40)), 1))
print("  with clipping it moves by at most lr * max_norm = %.4f in total."
      % (0.01 * 1.0))
print()

print("that bound is the real guarantee, and it is worth stating plainly:")
for lr, mn in ((0.01, 1.0), (0.001, 5.0), (0.1, 0.5)):
    print("  lr=%.3f, max_norm=%.1f -> no single update can move the weights"
          % (lr, mn))
    print("     further than %.4f, whatever the data does." % (lr * mn))
print()

print("how to choose max_norm -- measure, do not guess. log the norms first:")
norms = [np.linalg.norm(2 * X[i:i+8].T @ (X[i:i+8] @ np.array([0.5, 0.5])
                                          - y[i:i+8]) / 8) for i in range(0, 32, 8)]
print("  observed batch gradient norms:", np.round(norms, 4))
print("  a common choice is around the 90th percentile of the normal range,")
print("  here about %.2f. set it far below that and you are clipping every"
      % np.percentile(norms, 90))
print("  step, which silently turns your optimiser into normalised SGD.")
print()
print("where it is not optional: RNNs and LSTMs, where the same weight matrix")
print("is applied at every timestep so gradients genuinely can explode")
print("exponentially. transformers use it routinely too, usually at 1.0.")

Guided experiments

1. Step twice normally. The weight descends the parabola exactly as gradient descent should.
2. Step through the spike, unclipped. Uncheck Clip Gradients first. Step 3's gradient is 50x normal, and the update sends w far past the minimum — sometimes far enough that the next raw gradient is even larger, a genuine divergence.
3. Reset, turn clipping back on, step through again. The spike is capped at the clip norm before it is applied, and the weight takes a bounded step instead of an unbounded one.
4. Lower the clip norm. Smaller thresholds cap harder — including, if set low enough, ordinary gradients that were never a problem. It is a trade between safety and full-speed learning.

Worth remembering

Gradient clipping rescales an oversized gradient down to a maximum norm before the optimiser applies it, preserving direction while bounding step size. It is cheap insurance against the rare exploding gradient that would otherwise throw training off course, and it is standard practice in RNNs and very deep networks, where long chains of multiplication make the occasional huge gradient close to inevitable.

Where the spikes come from

Clipping treats a symptom, and it is worth knowing the causes, because sometimes the cause is fixable.

Recurrent depth. Backpropagating through 100 time steps multiplies 100 Jacobian factors. If their product exceeds 1, it compounds — the original motivation for clipping.

A learning rate that is too high makes weights large, which makes activations large, which makes gradients large. If clipping fires constantly, lower the learning rate before raising the threshold.

An unlucky batch. One example with an extreme feature value or a mislabelled target produces a genuinely enormous gradient. Scaling the inputs and checking the labels both reduce this.

Missing normalisation. Without batch or layer normalisation, activations can drift into ranges where gradients explode.

A loss that can spike. Cross-entropy on a confidently wrong prediction, or a division by something near zero, produces a large loss and a large gradient.

So the sensible order is: scale the inputs, normalise the activations, set the learning rate correctly, add warm-up — and clip as the final safety net rather than the first response.

Clipping and mixed precision

One interaction worth knowing. Mixed-precision training scales the loss up before the backward pass, to stop small gradients underflowing in 16-bit. That means the gradients you see are also scaled.

Clipping must therefore happen after unscaling, or the threshold means something different from what you intended:

scaler.scale(loss).backward()
scaler.unscale_(optimizer)                                  # remove the loss scale
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)     # now clip
scaler.step(optimizer)
scaler.update()

Getting this order wrong is a real bug: with a loss scale of 65,536, a threshold of 1.0 applied before unscaling effectively clips at 1/65,536 and training stops learning entirely.

Questions people ask

Does clipping slow training? Only if the threshold is too low, in which case you are discarding real gradient signal on most steps. Log the trigger rate.

Norm or value clipping? Norm, almost always — it preserves direction. Value clipping is occasionally used in reinforcement learning.

Is clipping the same as gradient normalisation? No. Clipping only intervenes above a threshold; normalisation would rescale every gradient to a fixed length, which discards magnitude information the optimiser uses.

Can clipping hide a real problem? Yes — a model that only trains with aggressive clipping usually has a learning rate, initialisation or data problem underneath.

Do I need it with Adam? Adam's per-parameter scaling already limits step sizes somewhat, but it does not prevent a single enormous gradient from dominating the moving averages. Clipping is still standard with Adam in transformers.

What threshold for a new model? Start at 1.0, measure the trigger rate, and adjust.

Recap in one screen

• If the gradient's norm exceeds a threshold, scale it down — direction preserved, length capped.
• One enormous gradient can produce NaN weights and end a run; clipping makes that impossible.
• Call it between backward() and step(), and after unscaling under mixed precision.
• Occasional triggering is healthy; constant triggering means the threshold is too low or the learning rate too high.
• Essential for recurrent models, standard in transformers, usually unnecessary in plain CNNs.