The Bias-Variance Tradeoff

Core Concepts: Bias vs. Variance

Every machine learning model's error can be decomposed into three parts: Bias, Variance, and Irreducible Error. We'll focus on the first two.

High Bias (Underfitting)

High Variance (Overfitting)

The dartboard picture

The quickest way to feel the difference is to imagine four players throwing darts at a bullseye. Every throw is a model trained on a slightly different sample of data, and the bullseye is the truth you are trying to predict.

• Low bias, low variance. All the darts land in a tight group on the bullseye. This is the model you want and rarely get.
• Low bias, high variance. The darts scatter widely, but their average lands on the bullseye. Each individual model is unreliable, even though the method is aimed correctly.
• High bias, low variance. A tight group, but sitting in the top-left corner. The method is consistent and consistently wrong.
• High bias, high variance. Scattered and off-centre. Everything that can go wrong has.

Bias is how far the average throw sits from the bullseye — error that comes from the model being too simple to represent the real pattern. Variance is how spread out the throws are — error that comes from the model reacting to the particular rows it happened to be trained on.

The word "tradeoff" is there because the usual ways of reducing one increase the other. Give a model more freedom and it can bend towards the truth (less bias) but it also starts bending towards the noise (more variance).

The three pieces every error is made of

Take a model, train it many times on different samples, and the expected squared error on a new point splits cleanly into three parts:

Total error = Bias² + Variance + Irreducible noise

• Bias² — the systematic miss. A straight line trying to follow a curve has high bias no matter how much data you give it.
• Variance — the instability. Train the same method on two random halves of your data; if the two models disagree wildly on the same input, that gap is variance.
• Irreducible noise — the part of the target nobody can predict. Two houses with identical features sell for different prices because one seller was in a hurry. No model, however clever, removes this term.

That last one matters more than it sounds. If 15% of the variation in your target is genuine randomness, a model that reaches 85% is not underperforming — it has hit the ceiling. Chasing the remaining 15% is how people end up fitting noise and calling it progress.

Underfitting and overfitting, in symptoms

Underfitting and overfitting are just bias and variance seen from the outside, through the two error numbers you actually have.

The gap between the two numbers is the practical measure of variance, and the level of the training error is the practical measure of bias. Nothing more sophisticated is needed to make the call.

What actually moves each one

Once you know which problem you have, the fixes barely overlap, which is why the diagnosis is worth doing carefully.

Too much bias, and you want more flexibility:

• Use a more expressive model — a polynomial instead of a line, a gradient-boosted forest instead of a single shallow tree.
• Add features, or better features. Interactions and ratios often help more than a fancier algorithm.
• Reduce regularisation — a smaller penalty term, a deeper tree, more training rounds.

Too much variance, and you want more discipline:

• Get more training data. This is the only fix that reduces variance without adding bias, and it is why "more data" is such a reliable answer.
• Increase regularisation — L1/L2 penalties, dropout, a shallower tree, early stopping.
• Remove noisy or barely-useful features.
• Average many models together. Bagging and random forests exist almost entirely to cancel variance out: each tree is unstable, but the vote across hundreds of them is not.

Notice that "more data" appears under variance and not under bias. A straight line fitted to ten million points is still a straight line. If your model is underfitting, another million rows will not help.

Interpreting the Visualization

This lab gives you direct control over a model's bias and variance to see how they affect the fit.

• The Sliders: The Bias slider controls the model's rigidity. High bias forces a simpler model (like a straight line). The Variance slider controls the model's flexibility or "wiggliness." High variance allows the model to contort itself to fit every data point.
• The Data: The blue and orange dots represent your dataset, sampled from a "true" underlying function (the dashed line), with some random noise added.
• The Goal: Your goal is to find a balance that minimizes the Total Error (Val MSE). This is the ultimate measure of your model's performance on unseen data.
• The "Resample Data" Button: This is the most important button! Clicking it simulates getting a completely new dataset from the same underlying source. It powerfully demonstrates the concept of variance: a high-variance model will change drastically with each new sample, while a low-variance model will remain stable.

Decompose the error into its two halves

Train the same model on 250 different samples of the same problem, then measure how wrong it is on average and how much it moves between samples. Those are bias and variance, measured rather than described.

import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline

TRUTH = lambda x: np.sin(1.5 * x) + 0.3 * x
x_test = np.linspace(0.2, 5.8, 120).reshape(-1, 1)
truth_test = TRUTH(x_test.ravel())

def decompose(make_model, runs=250, n=40, noise=0.6):
    rng = np.random.default_rng(0)
    preds = np.empty((runs, len(x_test)))
    for r in range(runs):
        x = rng.uniform(0, 6, n).reshape(-1, 1)
        y = TRUTH(x.ravel()) + rng.normal(0, noise, n)
        preds[r] = make_model().fit(x, y).predict(x_test)
    bias2 = ((preds.mean(axis=0) - truth_test) ** 2).mean()
    return bias2, preds.var(axis=0).mean()

def poly(d):
    return lambda: make_pipeline(PolynomialFeatures(d), LinearRegression())

print("truth: sin(1.5x) + 0.3x.  each sample is 40 noisy points from it.")
print("250 samples, one model refit on every one of them.")
print()
print("%-24s %10s %10s %10s" % ("model", "bias^2", "variance", "total"))
rows = []
for name, mk in ([("polynomial deg %d" % d, poly(d)) for d in (1, 3, 5, 7, 9)]
                 + [("tree, depth 2",   lambda: DecisionTreeRegressor(max_depth=2)),
                    ("tree, depth 5",   lambda: DecisionTreeRegressor(max_depth=5)),
                    ("tree, unlimited", lambda: DecisionTreeRegressor())]):
    b, v = decompose(mk)
    rows.append((name, b + v))
    print("%-24s %10.4f %10.4f %10.4f" % (name, b, v, b + v))
print()
print("lowest total error: %s (%.4f)" % min(rows, key=lambda r: r[1]))
print()
print("read the two columns against each other:")
print("  bias^2 falls as the model gets more flexible -- it can finally reach")
print("         the shape of the truth. deg 1 cannot bend at all, so it is stuck.")
print("  variance rises as it gets more flexible -- it starts fitting the noise,")
print("         and the noise is different in every sample.")
print("  the total is a U. finding the bottom of it is most of what tuning is.")
print()
print("bias^2 is how far the AVERAGE prediction sits from the truth. more data")
print("will not fix it, because the model cannot express the shape it needs.")
print("variance is how much one prediction moves when the training sample")
print("changes. that error does shrink with more data, or with less capacity.")
print()
print("the trees show the same arc: depth 2 is biased, unlimited depth has")
print("almost none and pays for it in variance. averaging many deep trees is")
print("exactly what a random forest does,")
print("and it is why bagging helps trees far more than it helps a straight line.")

Try it yourself

Try these experiments to build a strong intuition.

1. Simulate High Bias (Underfitting): Set the Bias slider to a high value (e.g., 90%) and Variance to a low value (e.g., 10%). Notice that the model is a very simple curve that fails to capture the true trend. The Training Error is high. Now, click Resample Data several times. The model's prediction curve barely moves. This is the definition of low variance but high bias.
2. Simulate High Variance (Overfitting): Set Bias to a low value (e.g., 10%) and Variance to a high value (e.g., 90%). The model will weave through the training points almost perfectly, resulting in a very low Training Error. However, the Variance Gap will be huge, indicating poor generalization. Now, click Resample Data repeatedly. The prediction curve will change wildly each time, chasing the noise in the new sample. This is a high-variance model.
3. Find the "Sweet Spot": Start with both sliders in the middle. Try to adjust them to find the lowest possible Total Error (Val MSE). You'll likely find this "sweet spot" somewhere in the middle, where the model is complex enough to capture the trend but not so complex that it memorizes the noise. This is the essence of the tradeoff.

Questions people ask

Is high variance always bad? Not if you are going to average it away. A single deep decision tree has terrible variance, which is exactly why a random forest of five hundred of them works so well — the individual instabilities are independent enough to cancel.

Where does the "sweet spot" sit? At the model complexity where test error is lowest, which is generally past the point where training error stops improving noticeably. You find it by measuring, not by reasoning — a validation curve across a complexity setting is the standard way.

Does deep learning break the tradeoff? Very large networks show a strange pattern called double descent: test error falls, rises around the point where the model can exactly memorise the training set, and then falls again as the model grows even bigger. The classic U-shaped curve is still real, it is just the first half of a longer story.

How do I measure variance without training a hundred models? Cross-validation gives you a cheap estimate. Look at the spread of scores across the folds, not only the average: a mean of 0.85 with a standard deviation of 0.02 is a very different model from a mean of 0.85 with a standard deviation of 0.12.

Does regularisation add bias on purpose? Yes, and that is the point. It deliberately pulls the model towards simpler solutions, accepting a small systematic error in exchange for a large drop in instability. Ridge regression is the clearest example: it is biased by construction and usually beats unbiased least squares on new data.

Recap in one screen

• Bias is being wrong on average; variance is being unstable from sample to sample; noise is the part nobody gets.
• High training error means bias. A big gap between training and test error means variance.
• More data fixes variance, never bias. More flexibility fixes bias and costs variance.
• Regularisation, averaging and early stopping are the everyday tools for trading a little bias for a lot less variance.
• The goal is the lowest total error, not zero of either — and never zero of the noise term.

Worth remembering

• The Goal is Generalization: A low training error is meaningless if the validation error is high. The goal is always to build a model that performs well on data it has never seen before.
• Underfitting (High Bias): Your model is too simple. Symptoms: High training error and high validation error.
• Overfitting (High Variance): Your model is too complex. Symptoms: Very low training error but high validation error. The gap between them is large.
• The Tradeoff is Real: As you decrease bias by making your model more complex (e.g., adding more features or layers), you almost always increase its variance. The art of machine learning is finding the right level of complexity for the given data.