Gradient Clipping

Context first

Most gradients in training are reasonably sized. At times — a badly scaled batch, a long recurrent chain, a numerically unstable loss — one gradient is enormous. Applied right, w − lr · g can throw a weight far outside the region the optimiser was making sane progress in. And the next few steps are spent recovering rather than learning. Gradient clipping stops this by capping the gradient's norm before the update: if the norm exceeds a threshold, each part is scaled down by the same factor so the way is preserved but the step size is bounded. The threshold is a hyperparameter, often between 0.5 and 5.0. The first transformer paper used 1.0. Clipping is most key in recurrent networks. There gradients pass through a chain of matrix multiplications at each time step and can grow exponentially — the exploding gradient problem. Without clipping, a single bad batch can undo thousands of steps of progress. With it, the worst that happens is a capped step in roughly the right way. In PyTorch the call is torch.nn.utils.clip_grad_norm_(net.params(), max_norm), placed between loss.backward() and optimizer.step(). Clipping does not fix the root cause of instability — proper weight initialisation, layer normalisation. And learning-rate warmup do that — but it is cheap insurance against the rare catastrophic gradient. This page trains a small RNN with and without clipping, plots the gradient norms at each step. And shows the loss diverging without the safety valve.

What clipping does

if ||g|| > threshold: g ← g · (threshold / ||g||)

The gradient's way is unchanged — it still points the way steepest descent says to go — just its length is capped. This is norm clipping, the common form. Value clipping (capping each part independently) is cruder and used less often. The threshold is a hyperparameter, often set between 0.5 and 5.0 depending on the design. Too low and clipping fires on nearly each step, slowing convergence. Too high and the rare dangerous gradients are not caught. The standard practice is to monitor the gradient norm in training and set the threshold above the typical value but well below the spikes. In PyTorch the call is torch.nn.utils.clip_grad_norm_(net.params(), max_norm), placed between loss.backward() and optimizer.step(). Gradient clipping is most key in recurrent networks. There the chain of multiplications through time steps can amplify gradients exponentially (the exploding gradient problem). Transformers also clip by default — the first paper used a threshold of 1.0. Clipping does not fix the root cause of instability. It just stops the worst symptom. Proper weight initialisation, layer normalisation, and learning-rate warmup address the underlying issues. This page trains a small RNN with and without clipping, plots the gradient norms at each step. And shows the training loss diverging without the safety valve.

Capping the size of a step

Gradient clipping is a single guard rail: if the gradient is more than a threshold, scale it down before applying the update.

if ‖g‖ > c:  g ← g × c / ‖g‖

Note what that does and does not change. The way is preserved just — each part is scaled by the same factor. Just the length is limited. That is why clipping by norm is the standard. And clipping each value independently is not: per-value clipping distorts the way. So the update no longer points downhill.

loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()

The call must come after backward() and before step(). Placing it anywhere else does nothing, silently.

The failure it stops

A single unusually large gradient can destroy a net in one step. Suppose the gradient norm is often around 0.5 and one batch gives 4,000 — a rare case, an outlier, an unlucky interaction. With a learning rate of 1e-3 that is a weight change of 4.0 in one step. It pushes the weights far outside the region where the loss is meaningful. The next forward pass gives inf, the loss turns NaN. And each later update propagates the NaN through the whole net.

Nothing recovers from that. The run is over. And if it happened at hour six of an eight-hour job, the checkpoint is what saves you.

Clipping makes that impossible. The worst case turns a step of size learning_rate × threshold, whatever the gradient was.

Choosing the threshold

The useful way to pick it is to measure rather than guess. Log the gradient norm for a few hundred steps without clipping, look at the distribution. And set the threshold somewhere above the typical value but below the outliers.

total = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1e9)
# returns the norm BEFORE clipping, so this logs without actually clipping

Then watch how often it triggers. Occasional clipping — a few percent of steps — is just right: the rare spike is caught and normal steps pass through untouched. Clipping on each step means the threshold is too low and you are throttling all learning. It shows up as slow, stalled convergence.

Common vals: 1.0 for transformers and most language nets, 5.0 for recurrent networks, 0.5 when training is very unstable. These are conventions rather than derived counts.

Cap the step, keep the way

One bad batch can give a gradient large enough to destroy a working net in a single update. Clipping bounds the damage. And the gap between the two ways of doing it matters.

import numpy as np

rng = np.random.default_rng(0)

g = np.array([3.0, -4.0, 12.0, 0.5])
print("a gradient:", g)
print("its L2 norm: %.4f" % np.linalg.norm(g))
print()

def clip_norm(grad, max_norm):
    n = np.linalg.norm(grad)
    return grad * min(1.0, max_norm / (n + 1e-12)), n

def clip_value(grad, limit):
    return np.clip(grad, -limit, limit)

print("clip by NORM to 5.0 -- rescale the whole vector:")
c, n = clip_norm(g, 5.0)
print("  scale factor %.4f" % (5.0 / n))
print("  result", np.round(c, 4), " norm %.4f" % np.linalg.norm(c))
print("  direction preserved:", np.allclose(c / np.linalg.norm(c), g / np.linalg.norm(g)))
print()
print("clip by VALUE to 5.0 -- truncate each element:")
cv = clip_value(g, 5.0)
print("  result", cv, " norm %.4f" % np.linalg.norm(cv))
print("  direction preserved:",
      np.allclose(cv / np.linalg.norm(cv), g / np.linalg.norm(g)))
print()
print("clipping by value CHANGED THE DIRECTION. the 12.0 carried %.0f percent"
      % (100 * 12.0 / np.abs(g).sum()))
print("of this gradient's total magnitude, and it was cut to 5.0 while the")
print("0.5 was left alone, so the update now points somewhere the gradient")
print("never pointed.")
print("that is why clip_grad_norm_ is the standard and clipping by value is")
print("a niche tool.")
print()

Building on the same setup, the next block shows the left cases.

import numpy as np

print("what it is protecting you from. training a small model where one")
print("batch contains an outlier:")
w = np.array([1.0, 1.0])
X = rng.normal(size=(40, 2))
y = X @ np.array([2.0, -1.0]) + rng.normal(0, 0.1, 40)
X_bad = X.copy(); y_bad = y.copy()
X_bad[0] *= 300                      # one corrupted row
y_bad[0] *= 300

for label, Xb, yb, clip in (("clean batch      ", X, y, None),
                            ("corrupted batch  ", X_bad, y_bad, None),
                            ("corrupted + clip ", X_bad, y_bad, 1.0)):
    ww = np.array([1.0, 1.0])
    g2 = 2 * Xb.T @ (Xb @ ww - yb) / len(Xb)
    norm = np.linalg.norm(g2)
    if clip:
        g2 = g2 * min(1.0, clip / norm)
    after = ww - 0.01 * g2
    print("  %s gradient norm %12.2f -> weights %s"
          % (label, norm, np.round(after, 4)))
print("  the corrupted batch moves the weights by %s in one step."
      % np.round(np.abs(np.array([1.0, 1.0])

The next block covers the left steps.

                        - (np.array([1.0, 1.0]) - 0.01 * 2 * X_bad.T
                           @ (X_bad @ np.array([1.0, 1.0]) - y_bad) / 40)), 1))
print("  with clipping it moves by at most lr * max_norm = %.4f in total."
      % (0.01 * 1.0))
print()

print("that bound is the real guarantee, and it is worth stating plainly:")
for lr, mn in ((0.01, 1.0), (0.001, 5.0), (0.1, 0.5)):
    print("  lr=%.3f, max_norm=%.1f -> no single update can move the weights"
          % (lr, mn))
    print("     further than %.4f, whatever the data does." % (lr * mn))
print()

print("how to choose max_norm -- measure, do not guess. log the norms first:")
norms = [np.linalg.norm(2 * X[i:i+8].T @ (X[i:i+8] @ np.array([0.5, 0.5])
                                          - y[i:i+8]) / 8) for i in range(0, 32, 8)]
print("  observed batch gradient norms:", np.round(norms, 4))
print("  a common choice is around the 90th percentile of the normal range,")
print("  here about %.2f. set it far below that and you are clipping every"
      % np.percentile(norms, 90))
print("  step, which silently turns your optimiser into normalised SGD.")
print()
print("where it is not optional: RNNs and LSTMs, where the same weight matrix")
print("is applied at every timestep so gradients genuinely can explode")
print("exponentially. transformers use it routinely too, usually at 1.0.")

Guided experiments

1. Step twice often. The weight descends the parabola just as gradient descent should.
2. Step through the spike, unclipped. Uncheck Clip Gradients first. Step 3's gradient is 50x normal. And the update sends w far past the min — sometimes far enough that the next raw gradient is still more, a genuine divergence.
3. Reset, turn clipping back on, step through again. The spike is capped at the clip norm before it is applied. And the weight takes a bounded step instead of an unbounded one.
4. Less the clip norm. Less thresholds cap harder — with, if set low enough, ordinary gradients that were never a problem. It is a trade between safety and full-speed learning.

Worth remembering

Gradient clipping rescales an oversized gradient down to a max norm before the optimiser applies it, preserving way while bounding step size. It is cheap insurance against the rare exploding gradient that would otherwise throw training off course. And it is standard practice in RNNs and very deep networks. There long chains of multiplication make the occasional huge gradient close to inevitable.

Where the spikes come from

Clipping treats a symptom, and it is worth knowing the causes, since sometimes the cause is fixable.

Recurrent depth. Backpropagating through 100 time steps multiplies 100 Jacobian factors. If their product exceeds 1, it compounds — the first motivation for clipping.

A learning rate that is too high makes weights large. It makes activations large, which makes gradients large. If clipping fires constantly, less the learning rate before raising the threshold.

An unlucky batch. One case with an extreme feature value or a mislabelled target gives a genuinely enormous gradient. Scaling the inputs and checking the labels both reduce this.

Missing normalisation. Without batch or layer normalisation, activations can drift into ranges where gradients explode.

A loss that can spike. Cross-entropy on a confidently wrong prediction. Or a division by something near zero, gives a large loss and a large gradient.

So the sensible order is: scale the inputs, normalise the activations, set the learning rate correctly, add warm-up — and clip as the final safety net rather than the first response.

Clipping and mixed precision

One interaction worth knowing. Mixed-precision training scales the loss up before the backward pass, to stop small gradients underflowing in 16-bit. That means the gradients you see are also scaled.

Clipping must therefore happen after unscaling. Or the threshold means something other from what you intended:

scaler.scale(loss).backward()
scaler.unscale_(optimizer)                                  # remove the loss scale
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)     # now clip
scaler.step(optimizer)
scaler.update()

Getting this order wrong is a real bug: with a loss scale of 65,536, a threshold of 1.0 applied before unscaling effectively clips at 1/65,536 and training stops learning fully.

Questions people ask

Does clipping slow training? Just if the threshold is too low, in which case you are discarding real gradient signal on most steps. Log the trigger rate.

Norm or value clipping? Norm, almost always — it preserves way. Value clipping is at times used in reinforcement learning.

Is clipping the same as gradient normalisation? No. Clipping just intervenes above a threshold. Normalisation would rescale each gradient to a fixed length. It discards magnitude information the optimiser uses.

Can clipping hide a real problem? Yes — a net that just trains with aggressive clipping often has a learning rate, initialisation or data problem underneath.

Do I need it with Adam? Adam's per-param scaling now limits step sizes somewhat. But it does not stop a single enormous gradient from dominating the moving averages. Clipping is still standard with Adam in transformers.

What threshold for a new net? Start at 1.0, measure the trigger rate, and adjust.

Recap in one screen

• If the gradient's norm exceeds a threshold, scale it down — way preserved, length capped.
• One enormous gradient can give NaN weights and end a run. Clipping makes that impossible.
• Call it between backward() and step(), and after unscaling under mixed precision.
• Occasional triggering is healthy. Constant triggering means the threshold is too low or the learning rate too high.
• Key for recurrent nets, standard in transformers, often unneeded in plain CNNs.