Overfitting vs. Underfitting

The bias-variance decomposition

Expected error on unseen data splits into three parts:

error = bias² + variance + irreducible noise

Capacity trades the first two against each other: adding it lowers bias and raises variance. The best model sits where their sum is smallest, not where either is minimised.

Telling them apart from two numbers

The diagnosis needs only the training and validation loss, compared against each other and against what is achievable:

• Both high, close together — underfitting. The model cannot represent the pattern. Add capacity, train longer, or reduce regularisation.
• Training low, validation much higher — overfitting. The model has memorised. Add data, add regularisation, or reduce capacity.
• Both low, close together — this is what you want.
• Training loss above validation loss — usually not a paradox but an artefact: dropout and batch norm are active during training and disabled at validation, so the training number is measured on a handicapped model.

Fixes, in the order worth trying

For overfitting, more data is the only fix with no downside, and augmentation is the cheap approximation to it. After that: early stopping, which costs nothing; then dropout or weight decay; and only then a smaller model, since shrinking capacity is the bluntest option.

For underfitting, first confirm the model is training at all — a learning rate that is too high or too low looks exactly like insufficient capacity. Then widen or deepen the network, remove regularisation, train for longer, or improve the features. A model that cannot overfit a small subset of the training data has a bug, not a capacity problem, and that is the fastest test to run.

Two ways to be wrong

Underfitting is a model too simple to capture the pattern. Training error is high, validation error is high, and the two are similar. The model has not learned enough.

Overfitting is a model that learned the training data including its noise. Training error is very low, validation error is much higher, and the gap is the symptom. The model learned too much of the wrong thing.

The diagnosis needs only two numbers:

That last row is worth naming: validation better than training usually means augmentation is applied only to training data, dropout is inflating training loss, or the split leaked.

Reading the training curve

Plot both losses against epochs and the shape tells you what is happening.

Both falling, still close: keep training.

Both flat and high from early on: underfitting. The model, the features or the learning rate is the problem.

Training falling, validation flattening then rising: the classic overfitting signature. The point where validation turns up is where the model stopped generalising and started memorising, and it is exactly where early stopping should trigger.

Validation noisy but not trending: normal. Judge the trend over several epochs, not one.

A related check: if training accuracy reaches 100%, the model has memorised the training set. That is not automatically bad — large networks often do this and still generalise — but the validation curve is the only thing that tells you which case you are in.

Fixing underfitting

The model needs more capacity or better inputs.

• A bigger model — more layers, more units.
• Better features. On tabular data this usually matters more than the architecture.
• Train longer. Check the curve is genuinely flat and not still descending.
• Less regularisation — lower dropout, less weight decay.
• A better learning rate. A rate far too low looks exactly like underfitting.
• Check for bugs. Wrong labels, a broken loss, or gradients not reaching the parameters all present as underfitting.

The sanity check that settles it: try to overfit a tiny subset — 20 examples — deliberately. A correct model and training loop should reach near-zero loss on 20 examples within a few hundred steps. If it cannot, the problem is a bug, not capacity, and no amount of tuning will help.

Try this above

1. Set Model Complexity to its minimum. The fitted curve cannot follow the data no matter how many Target Epochs you allow — more training will never fix underfitting.
2. Push complexity to maximum and watch the curve thread every single point, including the ones that are clearly noise.
3. Now raise Data Noise and repeat both. The high-complexity model gets dramatically worse; the low-complexity one barely changes, because it was never paying attention to the noise.
4. Use Sim Speed to slow the fit and watch the moment the curve starts chasing individual points.

Which lever to pull

They need opposite treatment, which is why diagnosing correctly matters more than any individual fix.

• Underfitting — more capacity, better features, train longer, reduce regularisation.
• Overfitting — more data, less capacity, more regularisation (dropout, weight decay), early stopping.

What usually goes wrong

Treating overfitting by training less. Early stopping helps, but if a model overfits after two epochs the real problem is capacity against data volume, not epoch count.Adding data to fix underfitting. More data does not help a model that lacks the capacity to represent the pattern in the first place. It is the standard remedy for overfitting and close to useless for the opposite.

In one line

Too little capacity misses the signal; too much memorises the noise — and the cures point in opposite directions.

Fixing overfitting

In rough order of effectiveness:

More data. The only fix that reduces variance without adding bias, and the reason "get more data" is such a reliable answer.

Data augmentation. Effectively more data, and often the single biggest improvement on image tasks with a few thousand examples.

Early stopping. Free, and it directly targets the moment overfitting begins. Keep the best checkpoint, not the last.

Weight decay. Penalises large weights, pulling the model towards simpler solutions. Always worth having on.

Dropout. Standard in dense layers and transformers, less used in modern convolutional networks.

A smaller model. Effective and blunt; try the regularisers first, since a smaller model also lowers the achievable ceiling.

Transfer learning. With a few hundred examples, fine-tuning a pretrained model beats training a smaller one from scratch by a wide margin.

Label smoothing and mixup. Soften the targets so the model is not pushed towards absolute confidence.

The modern complication: double descent

The classical picture — a U-shaped test error, with a sweet spot of complexity in the middle — is incomplete for very large models.

As model size grows, test error falls, then rises around the point where the model can exactly interpolate the training data, and then falls again as the model grows larger still. Very large networks often generalise well despite fitting their training data perfectly, which classical theory did not predict.

Two practical implications. Training accuracy of 100% is not by itself evidence of a problem — look at validation. And with modern architectures, "make the model smaller" is a less automatic response than it used to be; more data, augmentation and regularisation usually beat shrinking the network.

The classical U-curve is still real. It is now understood as the first half of a longer story.

Both failures, on one dial

Capacity swept from far too little to far too much, with train and test error side by side. The two failures are opposite ends of one axis, and the gap between the two numbers is the diagnosis.

import numpy as np

rng = np.random.default_rng(0)
NOISE = 0.35

def make(n):
    x = rng.uniform(-3, 3, n)
    return x, np.sin(1.6 * x) + 0.25 * x ** 2 + rng.normal(0, NOISE, n)

x_tr, y_tr = make(20)
x_te, y_te = make(2000)
print("20 training points from a smooth curve, plus noise of sd %.2f." % NOISE)
print("2000 test points from the same curve.")
print()
print("no model can score better than %.4f on test data, because that is the"
      % NOISE ** 2)
print("variance of the noise itself. that number is the floor.")
print()

def fit(deg, xa, ya):
    A = np.vander(xa, deg + 1)
    w = np.linalg.lstsq(A, ya, rcond=None)[0]
    return lambda xb: np.vander(xb, deg + 1) @ w, w

print("%8s %8s %12s %12s %10s   %s"
      % ("degree", "params", "train MSE", "test MSE", "gap", "verdict"))
for deg in (0, 1, 2, 4, 8, 14, 19):
    f, w = fit(deg, x_tr, y_tr)
    tr = ((f(x_tr) - y_tr) ** 2).mean()
    te = ((f(x_te) - y_te) ** 2).mean()
    if tr > 1.5 * NOISE ** 2:
        v = "underfitting"
    elif tr < 0.4 * NOISE ** 2:
        v = "overfitting -- fitting the noise"
    else:
        v = "about right"
    print("%8d %8d %12.4f %12s %10s   %s"
          % (deg, deg + 1, tr,
             "%.4f" % te if te < 1e6 else "%.2e" % te,
             "%.4f" % (te - tr) if te < 1e6 else "huge", v))
best = min(range(0, 20), key=lambda d: ((fit(d, x_tr, y_tr)[0](x_te) - y_te) ** 2).mean())
print()
print("lowest test error is at degree %d. note that the 'verdict' column is a"
      % best)
print("heuristic on the TRAINING error alone -- it flags degree 8 even though")
print("degree 8 happens to test better than degree 4. that is the point of it:")
print("a training error below the noise floor is a warning worth acting on,")
print("not a verdict. only test error settles the question.")
print()
print("degree 0 predicts one number for everything. degree 19 has 20")
print("parameters for 20 points, so it passes through every one of them")
print("exactly -- training error %.2e -- and is useless between them."
      % ((fit(19, x_tr, y_tr)[0](x_tr) - y_tr) ** 2).mean())
print()

print("the coefficients tell you the same story on their own:")
for deg in (2, 8, 19):
    _, w = fit(deg, x_tr, y_tr)
    print("  degree %2d: largest coefficient %.3e" % (deg, np.abs(w).max()))
print("  enormous coefficients that cancel each other out is what memorising")
print("  looks like from the inside. it is also exactly what weight decay")
print("  and L2 regularisation are built to prevent.")
print()

print("ridge regularisation on the degree-19 fit -- same model, penalised:")
A = np.vander(x_tr, 20)
for lam in (0.0, 1e-8, 1e-4, 1e-1, 10.0):
    w = np.linalg.solve(A.T @ A + lam * np.eye(20), A.T @ y_tr)
    tr = ((np.vander(x_tr, 20) @ w - y_tr) ** 2).mean()
    te = ((np.vander(x_te, 20) @ w - y_te) ** 2).mean()
    print("  lambda %8.0e: train %8.4f  test %12s  max|w| %.2e"
          % (lam, tr, "%.4f" % te if te < 1e6 else "%.2e" % te, np.abs(w).max()))
print("  the penalty cut the largest coefficient from 6.1e+02 to 5.9e-01 and")
print("  the test error by four orders of magnitude. it still did not rescue")
print("  the fit -- 20 parameters on 20 points is past saving by a penalty")
print("  alone. what it did do is stop the coefficients exploding, without")
print("  making the model any smaller.")
print()

print("and the other lever -- the same degree-19 model, given more points")
print("(each row draws a fresh training sample, so the 20-point row differs")
print("from the table above):")
for n in (20, 50, 200, 2000):
    xb, yb = make(n)
    f, _ = fit(19, xb, yb)
    tr = ((f(xb) - yb) ** 2).mean()
    te = ((f(x_te) - y_te) ** 2).mean()
    print("  %4d training points: train %8.4f  test %12s"
          % (n, tr, "%.4f" % te if te < 1e6 else "%.2e" % te))
print("  20 parameters overfit 20 points and are fine on 2000. capacity is")
print("  never too large in the abstract -- only relative to your data.")
print()

print("all of this transfers directly to a neural network. the dial is width,")
print("depth, and epochs rather than polynomial degree, and the two fixes are")
print("the same two: more data, or less freedom to use the capacity you have")
print("-- weight decay, dropout, early stopping.")
print()
print("the diagnosis, in one line each:")
print("  training error high        -> underfitting. a bigger model, longer")
print("                                training, better features. more data")
print("                                will NOT help.")
print("  training low, test high    -> overfitting. more data, or regularise.")
print("  both low and close together-> done.")
print()
print("which is why you print both numbers every epoch. one loss curve cannot")
print("tell you which of the two problems you have.")

Questions people ask

How big a train/validation gap is acceptable? There is no universal number. What matters is whether validation performance is good enough and stable. A 2% gap on a hard task may be fine; a 30% gap rarely is.

Can a model overfit and underfit at once? Yes — high bias and high variance together, usually from a badly chosen model plus insufficient data. Both errors are large and the gap is also large.

Does more data always help? With overfitting, yes. With underfitting, no — a linear model fitted to a million points is still linear.

Should I stop as soon as validation loss rises? Use patience — wait several epochs, since validation loss is noisy and can dip and recover.

Is 100% training accuracy bad? Not necessarily, for large models. The validation curve is the evidence that matters.

How do I know if it is a bug rather than underfitting? Try to overfit 20 examples. If you cannot, it is a bug.

Recap in one screen

• Underfitting: both errors high and similar. Overfitting: training low, validation much higher.
• The gap measures variance; the level of the training error measures bias.
• Fix underfitting with capacity, features, longer training and less regularisation.
• Fix overfitting with more data, augmentation, early stopping, weight decay and dropout.
• Always try to overfit a tiny subset first — failing that test means a bug, not a modelling problem.