Overfitting vs Underfitting

Overfitting vs Underfitting: A Practical Guide

Both failures look like poor performance and need opposite fixes. Telling them apart is a question of whether the model can even represent the pattern, or whether it has memorised the noise around it.

A concrete demonstration

Fit ten data points that follow a gentle curve with a little noise on top.

Degree 1 (a straight line) cannot bend at all. It misses the shape all over — high error on training data and new data. Underfitting.
Degree 9 through ten points passes just through each one. Training error is zero. And between the points the curve swings wildly to vals the real pattern never gives. Overfitting.
Degree 3 follows the trend and ignores the jitter. Slightly worse on the training set than degree 9, considerably better on anything new.

The degree-9 net is not "more exact". It has memorised the noise, and noise does not repeat. The diagnostic is one extra column: the test score. If training score is high and test score is much less, the net is overfitting — it learned the training set's noise. If both scores are low, the net is underfitting — it cannot represent the pattern at all. The fixes are opposite: overfitting needs less capacity (less degree, more regularisation, more data, dropout), underfitting needs more capacity (more degree, more features, less regularisation). Applying the wrong fix makes things worse. And a single accuracy count cannot tell you which way to go. The bias-variance tradeoff is the theoretical frame: bias drives underfitting (the net cannot flex enough), variance drives overfitting (the net flexes too much). Learning curves — plotting both scores against training-set size — show whether more data will help (it closes an overfitting gap but cannot fix an underfitting plateau). This page fits polynomials of degree 1, 3. And 9 to the same noisy data, prints both scores. And shows the learning curves for each.

The bias-variance decomposition

Expected error on unseen data splits into three parts:

error = bias² + variance + irreducible noise

Bias is error from wrong assumptions — a net too simple to represent the real relationship. High bias is underfitting. And it shows up as poor performance on training and validation data.Variance is sensitivity to the particular training sample — a net flexible enough to fit noise. High variance is overfitting: excellent training performance, poor validation performance.Irreducible noise is the part no net can remove. If the same inputs sometimes give other outputs, no amount of capacity will fix it. And a validation loss that flattens above zero is often this rather than a failure.

Capacity trades the first two against each other: adding it lowers bias and raises variance. The best net sits where their sum is smallest, not where either is minimised.

Telling them apart from two counts

The diagnosis needs just the training and validation loss, compared against each other and against what is achievable:

Both high, close together — underfitting. The net cannot represent the pattern. Add capacity, train longer, or reduce regularisation.
Training low, validation much more — overfitting. The net has memorised. Add data, add regularisation, or reduce capacity.
Both low, close together — this is what you want.
Training loss above validation loss — often not a paradox but an artefact: dropout and batch norm are active in training and disabled at validation. So the training count is measured on a handicapped net.

Fixes, in the order worth trying

For overfitting, more data is the just fix with no downside. And augmentation is the cheap approximation to it. After that: early stopping, which costs nothing. Then dropout or weight decay. And just then a less net, since shrinking capacity is the bluntest option.

For underfitting, first confirm the net is training at all — a learning rate that is too high or too low looks just like not enough capacity. Then widen or deepen the network, remove regularisation, train for longer. Or improve the features. A net that cannot overfit a small subset of the training data has a bug, not a capacity problem. And that is the fastest test to run.

Two ways to be wrong

Underfitting is a net too simple to capture the pattern. Training error is high, validation error is high. And the two are close. The net has not learned enough.

Overfitting is a net that learned the training data with its noise. Training error is very low, validation error is much more. And the gap is the symptom. The net learned too much of the wrong thing.

The diagnosis needs just two counts:

Training error	Validation error	Diagnosis
High	High, similar	Underfitting
Low	Much higher	Overfitting
Low	Low, similar	A good fit
High	Lower than training	A bug or a leak

That last row is worth naming: validation better than training often means augmentation is applied just to training data, dropout is inflating training loss. Or the split leaked.

Reading the training curve

Plot both losses against epochs and the shape tells you what is happening.

Both falling, still close: keep training.

Both flat and high from early on: underfitting. The net, the features or the learning rate is the problem.

Training falling, validation flattening then rising: the classic overfitting signature. The point where validation turns up is where the net stopped generalising and started memorising. And it is just where early stopping should trigger.

Validation noisy but not trending: normal. Judge the trend over a few epochs, not one.

A tied check: if training accuracy reaches 100%, the net has memorised the training set. That is not automatically bad — large networks often do this and still generalise — but the validation curve is the just thing that tells you which case you are in.

Fixing underfitting

The net needs more capacity or better inputs.

A bigger net — more layers, more units.
Better features. On tabular data this often matters more than the design.
Train longer. Check the curve is genuinely flat and not still descending.
Less regularisation — less dropout, less weight decay.
A better learning rate. A rate far too low looks just like underfitting.
Check for bugs. Wrong labels, a broken loss. Or gradients not reaching the params all present as underfitting.

The sanity check that settles it: try to overfit a tiny subset — 20 cases — deliberately. A correct net and training loop should reach near-zero loss on 20 cases within a few hundred steps. If it cannot, the problem is a bug, not capacity. And no amount of tuning will help.

Try this above

Set Net Complexity to its min. The fitted curve cannot follow the data no matter how many Target Epochs you let — more training will never fix underfitting.
Push complexity to max and watch the curve thread each single point, with the ones that are clearly noise.
Now raise Data Noise and repeat both. The high-complexity net gets dramatically worse. The low-complexity one barely changes, since it was never paying attention to the noise.
Use Sim Speed to slow the fit and watch the moment the curve starts chasing individual points.

Which lever to pull

They need opposite treatment, which is why diagnosing correctly matters more than any individual fix.

Underfitting — more capacity, better features, train longer, reduce regularisation.
Overfitting — more data, less capacity, more regularisation (dropout, weight decay), early stopping.

What often goes wrong

Treating overfitting by training less. Early stopping helps. But if a net overfits after two epochs the real problem is capacity against data volume, not epoch count.Adding data to fix underfitting. More data does not help a net that lacks the capacity to represent the pattern in the first place. It is the standard remedy for overfitting and close to useless for the opposite.

In one line

Too little capacity misses the signal. Too much memorises the noise — and the cures point in opposite ways.

Fixing overfitting

In rough order of effectiveness:

More data. The just fix that reduces variance without adding bias. And the reason "get more data" is such a reliable answer.

Data augmentation. Effectively more data. And often the single biggest improvement on pic tasks with a few thousand cases.

Early stopping. Free, and it right targets the moment overfitting begins. Keep the best checkpoint, not the last.

Weight decay. Penalises large weights, pulling the net towards simpler solutions. Always worth having on.

Dropout. Standard in dense layers and transformers, less used in modern convolutional networks.

A less net. Effective and blunt. Try the regularisers first, since a less net also lowers the achievable ceiling.

Transfer learning. With a few hundred cases, fine-tuning a pretrained net beats training a less one from scratch by a wide margin.

Label smoothing and mixup. Soften the targets so the net is not pushed towards absolute confidence.

The modern complication: double descent

The classical picture — a U-shaped test error, with a sweet spot of complexity in the middle — is incomplete for very large nets.

As net size grows, test error falls, then rises around the point where the net can just interpolate the training data. And then falls again as the net grows more still. Very large networks often generalise well despite fitting their training data perfectly. It classical theory did not predict.

Two real implications. Training accuracy of 100% is not by it evidence of a problem — look at validation. And with modern designs, "make the net less" is a less automatic response than it used to be. More data, augmentation and regularisation often beat shrinking the network.

The classical U-curve is still real. It is now understood as the first half of a longer story.

Both failures, on one dial

Capacity swept from far too little to far too much, with train and test error side by side. The two failures are opposite ends of one axis. And the gap between the two counts is the diagnosis.

example_01.pyNumPy

import numpy as np

rng = np.random.default_rng(0)
NOISE = 0.35

def make(n):
    x = rng.uniform(-3, 3, n)
    return x, np.sin(1.6 * x) + 0.25 * x ** 2 + rng.normal(0, NOISE, n)

x_tr, y_tr = make(20)
x_te, y_te = make(2000)
print("20 training points from a smooth curve, plus noise of sd %.2f." % NOISE)
print("2000 test points from the same curve.")
print()
print("no model can score better than %.4f on test data, because that is the"
      % NOISE ** 2)
print("variance of the noise itself. that number is the floor.")
print()

def fit(deg, xa, ya):
    A = np.vander(xa, deg + 1)
    w = np.linalg.lstsq(A, ya, rcond=None)[0]
    return lambda xb: np.vander(xb, deg + 1) @ w, w

Output

Building on the same setup, the next block shows the left cases.

example_02.pyNumPy

import numpy as np

print("%8s %8s %12s %12s %10s   %s"
      % ("degree", "params", "train MSE", "test MSE", "gap", "verdict"))
for deg in (0, 1, 2, 4, 8, 14, 19):
    f, w = fit(deg, x_tr, y_tr)
    tr = ((f(x_tr) - y_tr) ** 2).mean()
    te = ((f(x_te) - y_te) ** 2).mean()
    if tr > 1.5 * NOISE ** 2:
        v = "underfitting"
    elif tr < 0.4 * NOISE ** 2:
        v = "overfitting -- fitting the noise"
    else:
        v = "about right"
    print("%8d %8d %12.4f %12s %10s   %s"
          % (deg, deg + 1, tr,
             "%.4f" % te if te < 1e6 else "%.2e" % te,
             "%.4f" % (te - tr) if te < 1e6 else "huge", v))
best = min(range(0, 20), key=lambda d: ((fit(d, x_tr, y_tr)[0](x_te) - y_te) ** 2).mean())
print()
print("lowest test error is at degree %d. note that the 'verdict' column is a"
      % best)
print("heuristic on the TRAINING error alone -- it flags degree 8 even though")
print("degree 8 happens to test better than degree 4. that is the point of it:")
print("a training error below the noise floor is a warning worth acting on,")
print("not a verdict. only test error settles the question.")
print()
print("degree 0 predicts one number for everything. degree 19 has 20")
print("parameters for 20 points, so it passes through every one of them")
print("exactly -- training error %.2e -- and is useless between them."
      % ((fit(19, x_tr, y_tr)[0](x_tr) - y_tr) ** 2).mean())
print()

Output

The next block covers the left steps.

example_03.pyNumPy

import numpy as np

print("the coefficients tell you the same story on their own:")
for deg in (2, 8, 19):
    _, w = fit(deg, x_tr, y_tr)
    print("  degree %2d: largest coefficient %.3e" % (deg, np.abs(w).max()))
print("  enormous coefficients that cancel each other out is what memorising")
print("  looks like from the inside. it is also exactly what weight decay")
print("  and L2 regularisation are built to prevent.")
print()

print("ridge regularisation on the degree-19 fit -- same model, penalised:")
A = np.vander(x_tr, 20)
for lam in (0.0, 1e-8, 1e-4, 1e-1, 10.0):
    w = np.linalg.solve(A.T @ A + lam * np.eye(20), A.T @ y_tr)
    tr = ((np.vander(x_tr, 20) @ w - y_tr) ** 2).mean()
    te = ((np.vander(x_te, 20) @ w - y_te) ** 2).mean()
    print("  lambda %8.0e: train %8.4f  test %12s  max|w| %.2e"
          % (lam, tr, "%.4f" % te if te < 1e6 else "%.2e" % te, np.abs(w).max()))
print("  the penalty cut the largest coefficient from 6.1e+02 to 5.9e-01 and")
print("  the test error by four orders of magnitude. it still did not rescue")
print("  the fit -- 20 parameters on 20 points is past saving by a penalty")
print("  alone. what it did do is stop the coefficients exploding, without")
print("  making the model any smaller.")
print()

Output

The next block continues with the left demonstrations.

example_04.pyNumPy

import numpy as np

print("and the other lever -- the same degree-19 model, given more points")
print("(each row draws a fresh training sample, so the 20-point row differs")
print("from the table above):")
for n in (20, 50, 200, 2000):
    xb, yb = make(n)
    f, _ = fit(19, xb, yb)
    tr = ((f(xb) - yb) ** 2).mean()
    te = ((f(x_te) - y_te) ** 2).mean()
    print("  %4d training points: train %8.4f  test %12s"
          % (n, tr, "%.4f" % te if te < 1e6 else "%.2e" % te))
print("  20 parameters overfit 20 points and are fine on 2000. capacity is")
print("  never too large in the abstract -- only relative to your data.")
print()

print("all of this transfers directly to a neural network. the dial is width,")
print("depth, and epochs rather than polynomial degree, and the two fixes are")
print("the same two: more data, or less freedom to use the capacity you have")
print("-- weight decay, dropout, early stopping.")
print()
print("the diagnosis, in one line each:")
print("  training error high        -> underfitting. a bigger model, longer")
print("                                training, better features. more data")
print("                                will NOT help.")
print("  training low, test high    -> overfitting. more data, or regularise.")
print("  both low and close together-> done.")
print()
print("which is why you print both numbers every epoch. one loss curve cannot")
print("tell you which of the two problems you have.")

Output

Questions people ask

How big a train/validation gap is acceptable? There is no universal count. What matters is whether validation performance is good enough and stable. A 2% gap on a hard task may be fine. A 30% gap rarely is.

Can a net overfit and underfit at once? Yes — high bias and high variance together, often from a badly chosen net plus not enough data. Both errors are large and the gap is also large.

Does more data always help? With overfitting, yes. With underfitting, no — a linear net fitted to a million points is still linear.

Should I stop as soon as validation loss rises? Use patience — wait a few epochs, since validation loss is noisy and can dip and recover.

Is 100% training accuracy bad? Not necessarily, for large nets. The validation curve is the evidence that matters.

How do I know if it is a bug rather than underfitting? Try to overfit 20 cases. If you cannot, it is a bug.

Recap in one screen

Underfitting: both errors high and close. Overfitting: training low, validation much more.
The gap measures variance. The level of the training error measures bias.
Fix underfitting with capacity, features, longer training and less regularisation.
Fix overfitting with more data, augmentation, early stopping, weight decay and dropout.
Always try to overfit a tiny subset first — failing that test means a bug, not a modelling problem.

Overfitting vs. Underfitting

UNDERFITTING

Train Error (MSE)

Validation Error (MSE)

Overview

A concrete demonstration