Optimizers & The 3D Loss Landscape

Saddles matter more than local minima

The folk explanation of training failure is "it got stuck in a local minimum". In high dimensions that is mostly wrong. For a point to be a local minimum, the surface must curve upward in every direction at once — and with millions of parameters that is vanishingly unlikely. Saddle points, where it curves up in some directions and down in others, are enormously more common.

This is a practical difference. A local minimum is a trap. A saddle is not — there is a way down, and an optimiser with momentum carries enough velocity from earlier steps to keep moving across the flat region until it finds one.

The loss surface as a landscape

Picture two weights on the horizontal axes and the loss on the vertical one. Training is a ball rolling downhill on that surface, and the optimiser decides how the ball moves.

Real networks have millions of weights, so the surface is millions of dimensions rather than two. The picture is still useful, because the features that make optimisation hard are visible in three dimensions:

• Ravines — steep in one direction, nearly flat in another. Plain gradient descent zigzags across the steep direction and crawls along the flat one.
• Saddle points — downhill in some directions, uphill in others, with a gradient of zero at the centre. In high dimensions these vastly outnumber local minima, because it is improbable for millions of curvature signs to agree.
• Plateaus — large flat regions where gradients are tiny and progress stalls.
• Sharp versus flat minima — both fit the training data; flat ones tend to generalise better, because a small shift in the weights (or in the data) changes the loss less.

How each optimiser behaves on that landscape

Plain SGD takes a step directly downhill, scaled by the learning rate. In a ravine that means oscillating across the narrow direction while barely advancing along the valley floor.

Momentum accumulates velocity, so the oscillations partly cancel and the consistent downhill direction builds speed. It is the difference between a ball on a rough surface and one with mass.

Adam gives every weight its own effective step size, dividing by the square root of its recent squared gradients. Weights on flat directions get larger steps and weights on steep ones get smaller, which reshapes the ravine into something closer to a bowl.

That plateau row explains a real advantage of adaptive methods: dividing by a small accumulated gradient magnitude increases the step, so Adam keeps moving where SGD would sit still.

Saddle points, not local minima

The old fear was that gradient descent would get stuck in bad local minima. High-dimensional geometry says otherwise.

For a stationary point to be a local minimum, the curvature must be positive in every direction. With ten million parameters, that requires ten million signs to agree, which is vanishingly unlikely. Almost every point where the gradient vanishes is a saddle — downhill in at least one direction.

That is good news, because saddles are escapable. The stochastic noise from mini-batches is usually enough to push the model off the ridge and onto a descending path, which is one reason mini-batch training generalises better than full-batch.

It also reframes what training is doing: not searching for a unique global minimum, but descending through a landscape of many near-equivalent good solutions, most of which are fine.

Try this above

1. Run SGD (Red) alone and watch it stall when it reaches a flat region — near-zero gradient means near-zero step.
2. Switch to Momentum (Blue) from the same start. It coasts across the flat section on accumulated velocity.
3. Select All (Compare) and note the arrival order, then change Learning Rate and see whether the order holds. It often does not.
4. Slow Sim Speed right down at the moment each optimiser crosses the ridge — that is where the differences are widest.

What usually goes wrong

Over-reading the picture. This surface has two parameters. A real network has millions, and its loss landscape has properties that genuinely do not exist in 3D. Use this to build intuition about momentum and adaptive step sizes; do not use it to conclude anything about how many minima a real network has.Judging an optimiser from one starting point. Change where the run begins and the ranking often changes with it. A single trajectory is an anecdote.

In one line

The shape of the surface, not the cleverness of the algorithm, decides which optimiser looks good.

Sharp and flat minima

Two solutions can fit the training data equally well and behave very differently on new data.

A sharp minimum is a narrow crevice: move the weights slightly and the loss rises steeply. Since the test distribution differs slightly from the training one, that sensitivity translates into worse test performance.

A flat minimum is a broad basin: nearby weights are almost as good, so a small mismatch between train and test costs little.

Several observed behaviours follow from this:

• Small batches generalise slightly better because their gradient noise makes it hard to settle into a narrow crevice.
• Very large batches sometimes generalise worse, and techniques like LARS and longer warm-up exist to counter it.
• Weight averaging (SWA) averages weights from several late-training points, landing nearer the centre of a basin than any individual point.
• Sharpness-aware minimisation (SAM) explicitly penalises sharpness by taking a step towards the worst nearby point before updating.

Visualising a real loss surface

You can plot the landscape of an actual network, and the pictures are informative.

Take a trained model's weights, pick two random directions in weight space, and evaluate the loss on a grid of small perturbations along them. Plotting that grid gives a two-dimensional slice through a million-dimensional surface.

What such plots reliably show: networks without skip connections have chaotic, spiky surfaces; networks with them are dramatically smoother. That is a fairly direct visual explanation of why residual connections make deep networks trainable, and it is one of the more satisfying results in the area.

The caveat is that a two-dimensional slice of a million-dimensional surface can be misleading. Treat these plots as intuition, not proof.

Four surfaces that break four different optimisers

A bowl, a ravine, a saddle and a plateau. Each one defeats a different optimiser, and running all four on all four shows why there is no single right answer.

import numpy as np

def bowl(w):
    return 0.5 * (w ** 2).sum(), w

def ravine(w):
    c = np.array([400.0, 1.0])       # 400x steeper across than along
    return 0.5 * (c * w ** 2).sum(), c * w

def saddle(w):
    # up in one direction, down in the other, flat at the origin
    return w[0] ** 2 - w[1] ** 2, np.array([2 * w[0], -2 * w[1]])

def plateau(w):
    # tanh saturates: enormous region with almost no gradient
    s = np.tanh(w)
    return (s ** 2).sum(), 2 * s * (1 - s ** 2)

SURFACES = [("bowl", bowl, np.array([3.0, 3.0])),
            ("ravine (400:1)", ravine, np.array([-2.0, -2.0])),
            ("saddle", saddle, np.array([0.001, 0.001])),
            ("plateau (tanh)", plateau, np.array([3.0, 3.0]))]

def sgd(grad_fn, w, steps, lr):
    for _ in range(steps):
        w = w - lr * grad_fn(w)[1]
    return w

def momentum(grad_fn, w, steps, lr, beta=0.9):
    v = np.zeros_like(w)
    for _ in range(steps):
        v = beta * v + grad_fn(w)[1]
        w = w - lr * v
    return w

def rmsprop(grad_fn, w, steps, lr, beta=0.9):
    s = np.zeros_like(w)
    for _ in range(steps):
        g = grad_fn(w)[1]
        s = beta * s + (1 - beta) * g ** 2
        w = w - lr * g / (np.sqrt(s) + 1e-8)
    return w

def adam(grad_fn, w, steps, lr, b1=0.9, b2=0.999):
    m = np.zeros_like(w); v = np.zeros_like(w)
    for t in range(1, steps + 1):
        g = grad_fn(w)[1]
        m = b1 * m + (1 - b1) * g
        v = b2 * v + (1 - b2) * g ** 2
        w = w - lr * (m / (1 - b1 ** t)) / (np.sqrt(v / (1 - b2 ** t)) + 1e-8)
    return w

OPTS = [("SGD", sgd, 0.004), ("momentum", momentum, 0.002),
        ("RMSProp", rmsprop, 0.02), ("Adam", adam, 0.02)]

print("four surfaces, four optimisers, 500 steps each.")
print("the number is the loss reached -- lower is better, except on the")
print("saddle where the interesting question is whether it ESCAPED.")
print()
print("%-18s %14s %14s %14s %14s"
      % ("surface", *[n for n, _, _ in OPTS]))
for sname, fn, start in SURFACES:
    row = []
    for oname, opt, lr in OPTS:
        w = opt(fn, start.copy(), 500, lr)
        loss = fn(w)[0]
        row.append("%.3e" % loss if np.isfinite(loss) else "diverged")
    print("%-18s %14s %14s %14s %14s" % (sname, *row))
print()

print("read it surface by surface.")
print()
print("BOWL -- everything works. this is the surface people picture, and it")
print("is the one that tells you nothing about an optimiser.")
print()

print("RAVINE -- 400x steeper across than along. SGD's step is capped by the")
print("steep direction (anything above 2/400 = %.4f diverges), so the"
      % (2 / 400))
print("shallow one crawls at that same tiny rate:")
for step in (0, 100, 300, 500):
    ww = sgd(ravine, np.array([-2.0, -2.0]), step, 0.004)
    print("   SGD after %3d steps: %s" % (step, np.round(ww, 5)))
print("   the first coordinate is solved almost immediately; the second is")
print("   still travelling at step 500. RMSProp and Adam rescale per")
print("   coordinate, which is exactly the fix.")
print()

print("SADDLE -- the gradient at the origin is EXACTLY zero, and it is not a")
print("minimum. starting at (0.001, 0.001):")
for oname, opt, lr in OPTS:
    w = opt(saddle, np.array([0.001, 0.001]), 500, lr)
    print("   %-10s ended at %s (loss %.3e) -- %s"
          % (oname, np.round(w, 4), saddle(w)[0],
             "escaped" if abs(w[1]) > 0.5 else "still stuck near the origin"))
print("   escaping means running away DOWN the second coordinate. an")
print("   optimiser that normalises its step size gets out fastest, because")
print("   even a tiny gradient produces a full-sized step.")
print()

print("PLATEAU -- tanh saturates, so far from the origin the gradient is")
print("almost zero even though the loss is high:")
for r in (1.0, 2.0, 3.0, 4.0):
    g = plateau(np.array([r, r]))[1]
    print("   at distance %.0f: loss %.4f, gradient magnitude %.3e"
          % (r, plateau(np.array([r, r]))[0], np.linalg.norm(g)))
print("   SGD's step is proportional to that gradient, so it barely moves.")
print("   this is the same problem as a vanishing gradient, and it is why")
print("   saturating activations were abandoned in hidden layers.")
print()

print("the summary that actually transfers:")
print("  a real loss surface is all four of these in different directions at")
print("  once, and which one dominates changes during training.")
print("  adaptive methods (Adam, RMSProp) handle ravines and plateaus well")
print("  because they rescale by the gradient's own recent magnitude.")
print("  momentum handles ravines by accumulating the consistent direction")
print("  and cancelling the oscillating one.")
print("  plain SGD handles none of them, and still generalises slightly")
print("  better than Adam on many vision problems -- which is why it has")
print("  not gone away.")

Questions people ask

Does the optimiser find the global minimum? No, and it does not need to. High-dimensional landscapes have vast numbers of near-equivalent good solutions.

Why does momentum help? It averages recent gradients, so consistent directions accumulate and oscillations cancel — exactly what a ravine needs.

Are local minima a problem? Rarely. Saddle points and plateaus are the real obstacles in high dimensions.

Should I use a large batch to get a cleaner gradient? A cleaner gradient is not automatically better — the noise helps escape saddles and avoid sharp minima. Scale the learning rate if you do increase the batch.

What is the difference between the learning rate and momentum? The learning rate is how far to step; momentum is how much of the previous direction to keep.

Can I see whether my minimum is flat? Approximately — perturb the weights slightly and measure how much the loss rises. Large increases mean a sharp minimum.

Recap in one screen

• Loss landscapes have ravines, plateaus and saddles; the optimiser's job is to navigate them.
• Momentum damps oscillation across a ravine; adaptive methods rescale each direction.
• In high dimensions almost every zero-gradient point is a saddle, not a local minimum — and noise escapes them.
• Flat minima generalise better than sharp ones, which is why small batches and weight averaging help.
• Skip connections visibly smooth the surface, which is a direct explanation of why they work.