Maximum Likelihood Estimation

Quick Context

You have some data and a family of distributions that might have produced it. Which member of the family should you pick?

Maximum likelihood gives one answer, and it is almost embarrassingly reasonable: pick the distribution that makes the data you actually observed as unsurprising as possible. Nothing more sophisticated than that.

It is worth learning properly because it is not one technique among many. Mean squared error, log loss and cross-entropy are all maximum likelihood in disguise, each for a different assumption about the noise.

Likelihood is not probability

The two words get used interchangeably in conversation and they are not the same thing.

• Probability fixes the distribution and asks about the data. "Given this coin is fair, how likely is 7 heads in 10?"
• Likelihood fixes the data and asks about the distribution. "Given I observed 7 heads in 10, how well does 'fair' explain it?"

The data never changes in this module — the points stay exactly where they are. What moves is the candidate curve, and the likelihood is a statement about the curve, not about the points.

The calculation

For each observed point, read off the density of the candidate distribution at that point. Multiply them all together:

L(μ, σ) = p(x₁) × p(x₂) × … × p(xₙ)

The stems in the top panel are the individual densities. A candidate positioned over the bulk of the data has tall stems everywhere; one shifted away has some tall and many nearly zero, and one near-zero factor drags the whole product down.

In practice nobody computes that product, because multiplying a few hundred numbers below 1 underflows to exactly zero in floating point. Taking the logarithm turns the product into a sum:

log L = log p(x₁) + log p(x₂) + … + log p(xₙ)

The logarithm is monotonic, so whatever maximises the sum also maximises the product. Nothing is lost, and the arithmetic becomes stable.

Interactive Exploration Guide

1. Search by hand. Drag the Guessed Mean slider slowly across its range and watch the log-likelihood readout rise to a peak and fall away. There is exactly one best answer and you can feel where it is.
2. Check the punchline. Click Jump to the MLE and compare Your Guess against Sample Mean. They are the same number. The maximum-likelihood estimate of a normal mean is the sample mean — the formula you already knew, derived rather than assumed.Run this
3. Watch it climb. Click Climb to the Maximum and watch the curve slide into place while log-likelihood rises. This is gradient ascent on the likelihood, which is exactly what fitting a model does.Run this
4. Get the spread wrong. Set the Guessed Spread slider to 6. The curve flattens, every stem shortens, and log-likelihood falls — a distribution too vague to commit to anything explains the data badly. Now set it to 0.3: the curve is so narrow that points even slightly off-centre get almost no density, and it falls again. The best spread is in between.Run this
5. Break the arithmetic. Set the Sample Size slider to its maximum and watch Raw Likelihood. It collapses toward zero and Underflowed flips to yes, while the log-likelihood carries on working perfectly. This is not a rounding detail — it is the entire reason every implementation works in log space.Run this
6. See small samples mislead. Set the Sample Size slider to 2 and press New Data a few times. The MLE jumps around wildly. Maximum likelihood is not magic; with little data it happily returns a confident, wrong answer.Run this

Why your loss function looks the way it does

This is the part worth carrying away. Take maximum likelihood and change only the assumed noise:

• Assume normal noise around the prediction, and maximising log-likelihood turns out to be identical to minimising the sum of squared errors. Squaring in least squares is not an arbitrary choice of penalty — it falls out of the exponent in the normal density.
• Assume a Bernoulli outcome, and the same principle gives log loss, which is what logistic regression minimises.
• Assume a categorical outcome, and you get cross-entropy, the loss of essentially every classifier.

So "minimising the loss" and "maximising the likelihood" are usually the same sentence. Negative log-likelihood is the loss; the minus sign is there only because optimisers are written to go downhill.

Where it goes wrong

• It overfits happily. Maximum likelihood picks whatever explains the observed data best, with no preference for simplicity. Regularisation is what you add to stop it, and it corresponds to putting a prior on the parameters.
• The MLE of variance is biased. It divides by n, while the unbiased estimator divides by n−1. Maximum likelihood is not guaranteed to be unbiased, only to maximise likelihood.
• The model family matters more than the fit. Fitting a normal to data that is not remotely normal gives you the best possible normal, and it is still the wrong answer. The method chooses within a family, never between families.
• Likelihoods are not probabilities of the parameters. "The likelihood of μ = 3 is high" does not mean μ = 3 is probable. Turning one into the other requires a prior, which is what Bayesian inference does with Bayes' theorem.

Key Takeaway

Maximum likelihood picks the distribution that makes the observed data least surprising, by holding the data fixed and varying the parameters — the reverse of a probability question. The likelihood is the product of each point's density, and because that product underflows to zero on any real sample, it is always computed as a sum of logarithms instead, which is safe because the logarithm is monotonic. For a normal distribution the answer comes out as exactly the sample mean and sample spread, and changing only the assumed noise turns the same principle into squared error, log loss or cross-entropy — which is why minimising a loss and maximising a likelihood are usually the same thing.