Optimizers & The 3D Loss Landscape

What the terrain is made of

• Ravines — steep in one way, nearly flat in another. Plain SGD ricochets between the walls.
• Plateaus — large regions where the gradient is tiny. Progress crawls since there is almost nothing to follow.
• Saddle points — the gradient is close to zero but you are not at a min: the surface curves up in some ways and down in others.

At a saddle point, plain SGD stalls since the gradient is near zero in each way — momentum. Or a per-param scaling like Adam, is what pushes through. A ravine (a long narrow valley) is the other classic failure mode: the gradient points across the valley rather than along it. So plain SGD oscillates side to side instead of progressing toward the min. Momentum damps the oscillation by averaging past gradients. And RMSprop scales each param's learning rate by the recent magnitude of its gradient. It shrinks the step across the valley and lets the step along it dominate. Adam mixes both: it carries a momentum term and a per-param scaling term, with bias correction for the first few steps. The 3D loss surface makes these gaps visible in a way that 2D curves cannot — you can see the trajectory spiralling into a bowl, overshooting a ridge, or threading through a saddle. This page plots three loss surfaces (a bowl, a ravine, and a saddle), runs SGD, momentum, RMSprop, and Adam on each, and shows the trajectory and convergence speed side by side.

Saddles matter more than local minima

The folk explanation of training failure is "it got stuck in a local min". In high dimensions that is mostly wrong. For a point to be a local min, the surface must curve upward in each way at once — and with millions of params that is vanishingly unlikely. Saddle points, where it curves up in some ways and down in others, are enormously more common.

This is a real gap. A local min is a trap. A saddle is not — there is a way down. And an optimiser with momentum carries enough velocity from earlier steps to keep moving across the flat region until it finds one.

The loss surface as a landscape

Picture two weights on the horizontal axes and the loss on the vertical one. Training is a ball rolling downhill on that surface. And the optimiser decides how the ball moves.

Real networks have millions of weights. So the surface is millions of dimensions rather than two. The picture is still useful, since the features that make optimisation hard are visible in three dimensions:

• Ravines — steep in one way, nearly flat in another. Plain gradient descent zigzags across the steep way and crawls along the flat one.
• Saddle points — downhill in some ways, uphill in others, with a gradient of zero at the centre. In high dimensions these vastly outnumber local minima, since it is improbable for millions of curvature signs to agree.
• Plateaus — large flat regions where gradients are tiny and progress stalls.
• Sharp versus flat minima — both fit the training data. Flat ones tend to generalise better, since a small shift in the weights (or in the data) changes the loss less.

How each optimiser behaves on that landscape

Plain SGD takes a step right downhill, scaled by the learning rate. In a ravine that means oscillating across the narrow way while barely advancing along the valley floor.

Momentum accumulates velocity, so the oscillations partly cancel and the consistent downhill way builds speed. It is the gap between a ball on a rough surface and one with mass.

Adam gives each weight its own effective step size, dividing by the square root of its recent squared gradients. Weights on flat ways get more steps and weights on steep ones get less. It reshapes the ravine into something closer to a bowl.

That plateau row explains a real advantage of adaptive ways: dividing by a small accumulated gradient magnitude raises the step. So Adam keeps moving where SGD would sit still.

Saddle points, not local minima

The old fear was that gradient descent would get stuck in bad local minima. High-dimensional geometry says otherwise.

For a stationary point to be a local min, the curvature must be positive in each way. With ten million params, that needs ten million signs to agree. It is vanishingly unlikely. Almost each point where the gradient vanishes is a saddle — downhill in at least one way.

That is good news, since saddles are escapable. The stochastic noise from mini-batches is often enough to push the net off the ridge and onto a descending path. It is one reason mini-batch training generalises better than full-batch.

It also reframes what training is doing: not searching for a unique global min. But descending through a field of many near-equivalent good solutions, most of which are fine.

Try this above

1. Run SGD (Red) alone and watch it stall when it reaches a flat region — near-zero gradient means near-zero step.
2. Switch to Momentum (Blue) from the same start. It coasts across the flat section on accumulated velocity.
3. Select All (Check) and note the arrival order, then change Learning Rate and see whether the order holds. It often does not.
4. Slow Sim Speed right down at the moment each optimiser crosses the ridge — that is where the gaps are widest.

What often goes wrong

Over-reading the picture. This surface has two params. A real network has millions. And its loss landscape has properties that genuinely do not exist in 3D. Use this to build intuition about momentum and adaptive step sizes. Do not use it to conclude anything about how many minima a real network has.Judging an optimiser from one starting point. Change where the run begins and the ranking often changes with it. A single trajectory is an anecdote.

In one line

The shape of the surface, not the cleverness of the algorithm, decides which optimiser looks good.

Sharp and flat minima

Two solutions can fit the training data equally well and behave very differently on new data.

A sharp min is a narrow crevice: move the weights slightly and the loss rises steeply. Since the test distribution differs slightly from the training one, that sensitivity translates into worse test performance.

A flat min is a broad basin: nearby weights are almost as good. So a small mismatch between train and test costs little.

A few observed behaviours follow from this:

• Small batches generalise slightly better since their gradient noise makes it hard to settle into a narrow crevice.
• Very large batches sometimes generalise worse. And techniques like LARS and longer warm-up exist to counter it.
• Weight averaging (SWA) averages weights from a few late-training points, landing nearer the centre of a basin than any individual point.
• Sharpness-aware minimisation (SAM) clearly penalises sharpness by taking a step towards the worst nearby point before updating.

Visualising a real loss surface

You can plot the field of an actual network. And the pictures are informative.

Take a trained net's weights, pick two random ways in weight space. And test the loss on a grid of small perturbations along them. Plotting that grid gives a two-dimensional slice through a million-dimensional surface.

What such plots reliably show: networks without skip connections have chaotic, spiky surfaces. Networks with them are dramatically smoother. That is a fairly direct visual explanation of why residual connections make deep networks trainable. And it is one of the more satisfying results in the area.

The caveat is that a two-dimensional slice of a million-dimensional surface can be misleading. Treat these plots as intuition, not proof.

Four surfaces that break four other optimisers

A bowl, a ravine, a saddle and a plateau. Each one defeats a other optimiser. And running all four on all four shows why there is no single right answer.

import numpy as np
def bowl(w): return 0.5*(w**2).sum(), w
def ravine(w):
    c = np.array([400.,1.]); return 0.5*(c*w**2).sum(), c*w
def saddle(w): return w[0]**2-w[1]**2, np.array([2*w[0],-2*w[1]])
def plateau(w):
    s = np.tanh(w); return (s**2).sum(), 2*s*(1-s**2)
def sgd(fn,w,n,lr):
    for _ in range(n): w = w - lr*fn(w)[1]
    return w
def momentum(fn,w,n,lr,beta=0.9):
    v = np.zeros_like(w)
    for _ in range(n): v = beta*v+fn(w)[1]; w = w-lr*v
    return w
def rmsprop(fn,w,n,lr,beta=0.9):
    s = np.zeros_like(w)
    for _ in range(n):
        g = fn(w)[1]; s = beta*s+(1-beta)*g**2; w = w-lr*g/(np.sqrt(s)+1e-8)
    return w
def adam(fn,w,n,lr,b1=0.9,b2=0.999):
    m = np.zeros_like(w); v = np.zeros_like(w)
    for t in range(1,n+1):
        g = fn(w)[1]; m = b1*m+(1-b1)*g; v = b2*v+(1-b2)*g**2
        w = w-lr*(m/(1-b1**t))/(np.sqrt(v/(1-b2**t))+1e-8)
    return w
SURFS = [("bowl",bowl,[3.,3.]), ("ravine 400:1",ravine,[-2.,-2.]),
         ("saddle",saddle,[.001,.001]), ("plateau",plateau,[3.,3.])]
OPTS = [("SGD",sgd,.004),("momentum",momentum,.002),
        ("RMSProp",rmsprop,.02),("Adam",adam,.02)]
print("Four surfaces, four optimisers, 500 steps each (loss reached):")
print("%-16s %12s %12s %12s %12s" % ("surface",*[n for n,_,_ in OPTS]))
for sn,fn,s0 in SURFS:
    vals = []
    for _,opt,lr in OPTS:
        loss = fn(opt(fn, np.array(s0), 500, lr))[0]
        vals.append("%.3e" % loss if np.isfinite(loss) else "diverged")
    print("%-16s %12s %12s %12s %12s" % (sn, *vals))

Now read the table surface by surface to see what each one reveals about optimiser design.

import numpy as np

def ravine(w):
    c = np.array([400.0, 1.0])
    return 0.5 * (c * w**2).sum(), c * w

def sgd(fn, w, n, lr):
    for _ in range(n): w = w - lr * fn(w)[1]
    return w

print("BOWL -- everything works. it tells you nothing about an optimiser.")
print()
print("RAVINE -- 400x steeper across than along. SGD's step is capped by")
print("the steep direction (above 2/400 = %.4f it diverges), so the" % (2/400))
print("shallow direction crawls at that same tiny rate:")
for step in (0, 100, 300, 500):
    ww = sgd(ravine, np.array([-2.0, -2.0]), step, 0.004)
    print("   SGD after %3d steps: %s" % (step, np.round(ww, 5)))
print("   the first coordinate is solved fast; the second is still")
print("   travelling at step 500. RMSProp and Adam rescale per coordinate.")

The saddle and plateau are the surfaces that split adaptive ways from plain SGD.

import numpy as np
def saddle(w): return w[0]**2-w[1]**2, np.array([2*w[0],-2*w[1]])
def plateau(w):
    s = np.tanh(w); return (s**2).sum(), 2*s*(1-s**2)
def sgd(fn,w,n,lr):
    for _ in range(n): w = w-lr*fn(w)[1]
    return w
def momentum(fn,w,n,lr,beta=0.9):
    v = np.zeros_like(w)
    for _ in range(n): v = beta*v+fn(w)[1]; w = w-lr*v
    return w
def rmsprop(fn,w,n,lr,beta=0.9):
    s = np.zeros_like(w)
    for _ in range(n):
        g = fn(w)[1]; s = beta*s+(1-beta)*g**2; w = w-lr*g/(np.sqrt(s)+1e-8)
    return w
def adam(fn,w,n,lr,b1=0.9,b2=0.999):
    m = np.zeros_like(w); v = np.zeros_like(w)
    for t in range(1,n+1):
        g = fn(w)[1]; m = b1*m+(1-b1)*g; v = b2*v+(1-b2)*g**2
        w = w-lr*(m/(1-b1**t))/(np.sqrt(v/(1-b2**t))+1e-8)
    return w
OPTS = [("SGD",sgd,.004),("momentum",momentum,.002),
        ("RMSProp",rmsprop,.02),("Adam",adam,.02)]

print("SADDLE -- gradient is zero at the origin, yet it is not a minimum:")
for on, opt, lr in OPTS:
    w = opt(saddle, np.array([.001,.001]), 500, lr)
    print("   %-10s -> %s  %s" % (on, np.round(w,4), "escaped" if abs(w[1])>.5 else "stuck"))
print("   normalising step size lets a tiny gradient produce a full step.")
print()
print("PLATEAU -- tanh saturates, gradient vanishes far from origin:")
for r in (1., 2., 3., 4.):
    g = plateau(np.array([r,r]))[1]
    print("   dist %.0f: loss %.4f, |grad| %.3e" % (r, plateau(np.array([r,r]))[0], np.linalg.norm(g)))
print("   SGD barely moves. same problem as vanishing gradients.")
print()
print("SUMMARY: a real surface is all four at once. adaptive methods rescale")
print("by gradient magnitude; momentum accumulates the consistent direction.")
print("plain SGD handles none, yet generalises better on many vision tasks.")

Questions people ask

Does the optimiser find the global min? No. And it does not need to. High-dimensional landscapes have vast counts of near-equivalent good solutions.

Why does momentum help? It averages recent gradients. So consistent ways accumulate and oscillations cancel — just what a ravine needs.

Are local minima a problem? Rarely. Saddle points and plateaus are the real obstacles in high dimensions.

Should I use a large batch to get a cleaner gradient? A cleaner gradient is not automatically better — the noise helps escape saddles and avoid sharp minima. Scale the learning rate if you do increase the batch.

What is the gap between the learning rate and momentum? The learning rate is how far to step. Momentum is how much of the last way to keep.

Can I see whether my min is flat? About — perturb the weights slightly and measure how much the loss rises. Large raises mean a sharp min.

Recap in one screen

• Loss landscapes have ravines, plateaus and saddles. The optimiser's job is to navigate them.
• Momentum damps oscillation across a ravine. Adaptive ways rescale each way.
• In high dimensions almost each zero-gradient point is a saddle, not a local min — and noise escapes them.
• Flat minima generalise better than sharp ones. It is why small batches and weight averaging help.
• Skip connections visibly smooth the surface. It is a direct explanation of why they work.