Maximum Likelihood Estimation

Likelihood is not probability

The two words get used interchangeably in conversation and they are not the same thing.

• Probability fixes the distribution and asks about the data. "Given this coin is fair, how likely is 7 heads in 10?"
• Likelihood fixes the data and asks about the distribution. "Given I observed 7 heads in 10, how well does 'fair' explain it?"

The data never changes in this module — the points stay exactly where they are. What moves is the candidate curve, and the likelihood is a statement about the curve, not about the points.

The calculation

For each observed point, read off the density of the candidate distribution at that point. Multiply them all together:

L(μ, σ) = p(x₁) × p(x₂) × … × p(xₙ)

The stems in the top panel are the individual densities. A candidate positioned over the bulk of the data has tall stems everywhere; one shifted away has some tall and many nearly zero, and one near-zero factor drags the whole product down.

In practice nobody computes that product, because multiplying a few hundred numbers below 1 underflows to exactly zero in floating point. Taking the logarithm turns the product into a sum:

log L = log p(x₁) + log p(x₂) + … + log p(xₙ)

The logarithm is monotonic, so whatever maximises the sum also maximises the product. Nothing is lost, and the arithmetic becomes stable.

Which parameters make the data least surprising?

Maximum likelihood estimation turns "fit a model" into a precise question: of all the parameter values I could choose, which makes the data I actually observed most probable?

Flip a coin 10 times and get 7 heads. For any bias p, the probability of that outcome is:

L(p) = C(10,7) × p⁷ × (1−p)³

Evaluate it at a few values:

The maximum is at p = 0.7, which is exactly the observed proportion. That is not a coincidence — for a binomial, the maximum likelihood estimate is always the sample proportion, and it can be derived in three lines of calculus.

Note the shift in perspective. A probability treats the parameters as fixed and asks about the data. A likelihood treats the data as fixed and asks about the parameters. Same formula, read in the other direction.

Logs, and why the estimate is always found that way

The likelihood of n independent observations is a product of n terms, each below 1. For n = 1,000 that underflows to exactly zero in floating point, and the calculation is destroyed before it starts.

Taking logarithms turns the product into a sum:

log L(θ) = Σ log P(xᵢ | θ)

Because log is monotonic, whatever maximises the log-likelihood also maximises the likelihood — so nothing is lost, and everything becomes numerically stable and easy to differentiate.

By convention optimisers minimise, so what is actually minimised is the negative log-likelihood. That phrase, and the abbreviation NLL, appear throughout machine learning for exactly this reason.

Why your loss function is what it is

This is the payoff, and it is a genuinely unifying idea: the standard loss functions are negative log-likelihoods under different assumptions about the noise.

So choosing MSE is not a neutral default — it is an assumption that errors are symmetric, bell-shaped and constant in spread. When that is wrong (heavy tails, skew, counts), the loss is fighting the data, and switching to the likelihood that matches the process usually beats any amount of model tuning.

The same framework explains regularisation. Adding a prior over the parameters and maximising the posterior instead of the likelihood — maximum a posteriori estimation — produces ridge regression from a normal prior and lasso from a Laplace prior. The penalty term is the log of the prior.

Exploration guide

1. Search by hand. Drag the Guessed Mean slider slowly across its range and watch the log-likelihood readout rise to a peak and fall away. There is exactly one best answer and you can feel where it is.
2. Check the punchline. Click Jump to the MLE and compare Your Guess against Sample Mean. They are the same number. The maximum-likelihood estimate of a normal mean is the sample mean — the formula you already knew, derived rather than assumed.
3. Watch it climb. Click Climb to the Maximum and watch the curve slide into place while log-likelihood rises. This is gradient ascent on the likelihood, which is exactly what fitting a model does.
4. Get the spread wrong. Set the Guessed Spread slider to 6. The curve flattens, every stem shortens, and log-likelihood falls — a distribution too vague to commit to anything explains the data badly. Now set it to 0.3: the curve is so narrow that points even slightly off-centre get almost no density, and it falls again. The best spread is in between.
5. Break the arithmetic. Set the Sample Size slider to its maximum and watch Raw Likelihood. It collapses toward zero and Underflowed flips to yes, while the log-likelihood carries on working perfectly. This is not a rounding detail — it is the entire reason every implementation works in log space.
6. See small samples mislead. Set the Sample Size slider to 2 and press New Data a few times. The MLE jumps around wildly. Maximum likelihood is not magic; with little data it happily returns a confident, wrong answer.

Why your loss function looks the way it does

This is the part worth carrying away. Take maximum likelihood and change only the assumed noise:

• Assume normal noise around the prediction, and maximising log-likelihood turns out to be identical to minimising the sum of squared errors. Squaring in least squares is not an arbitrary choice of penalty — it falls out of the exponent in the normal density.
• Assume a Bernoulli outcome, and the same principle gives log loss, which is what logistic regression minimises.
• Assume a categorical outcome, and you get cross-entropy, the loss of essentially every classifier.

So "minimising the loss" and "maximising the likelihood" are usually the same sentence. Negative log-likelihood is the loss; the minus sign is there only because optimisers are written to go downhill.

Where it goes wrong

• It overfits happily. Maximum likelihood picks whatever explains the observed data best, with no preference for simplicity. Regularisation is what you add to stop it, and it corresponds to putting a prior on the parameters.
• The MLE of variance is biased. It divides by n, while the unbiased estimator divides by n−1. Maximum likelihood is not guaranteed to be unbiased, only to maximise likelihood.
• The model family matters more than the fit. Fitting a normal to data that is not remotely normal gives you the best possible normal, and it is still the wrong answer. The method chooses within a family, never between families.
• Likelihoods are not probabilities of the parameters. "The likelihood of μ = 3 is high" does not mean μ = 3 is probable. Turning one into the other requires a prior, which is what Bayesian inference does with Bayes' theorem.

Search for the parameter that explains the data

Score every candidate parameter by how likely it makes the data you actually saw, then check the winner against the closed-form answer.

import numpy as np

rng = np.random.default_rng(0)

TRUE_P = 0.65
flips = (rng.random(2000) < TRUE_P).astype(int)
heads, n = flips.sum(), len(flips)
print("observed %d heads in %d flips" % (heads, n))
print()

def log_likelihood(p, heads=heads, n=n):
    return heads * np.log(p) + (n - heads) * np.log(1 - p)

print("%8s %18s" % ("p", "log-likelihood"))
for p in (0.30, 0.50, 0.60, 0.65, 0.70, 0.80):
    print("%8.2f %18.4f" % (p, log_likelihood(p)))
print("        every other p makes the data we saw less likely.")
print()

grid = np.linspace(0.001, 0.999, 20_000)
best = grid[np.argmax([log_likelihood(p) for p in grid])]
print("grid search best p    : %.4f" % best)
print("closed form (heads/n) : %.4f" % (heads / n))
print("true value            : %.4f" % TRUE_P)
print()
print("the closed form is not a shortcut for this case -- it is what you get")
print("by setting the derivative of the log-likelihood to zero.")
print()

print("the estimate tightens with more data:")
for size in (20, 100, 1000, 10_000):
    ests = [(rng.random(size) < TRUE_P).mean() for _ in range(2000)]
    print("  %6d flips: estimate %.4f +/- %.4f" % (size, np.mean(ests), np.std(ests)))
print()

print("same idea for a normal distribution:")
data = rng.normal(50.0, 8.0, 500)
print("  MLE mean %8.4f   vs sample mean %8.4f" % (data.mean(), data.mean()))
print("  MLE sd   %8.4f   vs unbiased sd %8.4f"
      % (data.std(ddof=0), data.std(ddof=1)))
print("  the MLE for sd divides by n, not n-1, so it is biased slightly low.")
print()
print("and why the log at all: the raw likelihood of %d flips is" % n)
print("  %.3e -- indistinguishable from zero on any hardware."
      % np.exp(log_likelihood(best)))

Worth remembering

Maximum likelihood picks the distribution that makes the observed data least surprising, by holding the data fixed and varying the parameters — the reverse of a probability question. The likelihood is the product of each point's density, and because that product underflows to zero on any real sample, it is always computed as a sum of logarithms instead, which is safe because the logarithm is monotonic. For a normal distribution the answer comes out as exactly the sample mean and sample spread, and changing only the assumed noise turns the same principle into squared error, log loss or cross-entropy — which is why minimising a loss and maximising a likelihood are usually the same thing.

Properties, and the one caveat

Maximum likelihood estimators have attractive theoretical properties, which is why the method is the default across statistics:

• Consistent. With enough data, the estimate converges to the true parameter value.
• Asymptotically efficient. No other consistent estimator has smaller variance in the large-sample limit.
• Invariant. If θ-hat is the MLE of θ, then g(θ-hat) is the MLE of g(θ) — so you can estimate on whichever scale is convenient.

The caveat is that they are not necessarily unbiased in small samples. The classic example is variance: the maximum likelihood estimate divides by n and systematically underestimates the true variance, which is why the sample variance divides by n−1 instead. That correction is a deliberate departure from the MLE.

The other practical caveat is overfitting. Maximising likelihood alone will happily choose parameters that explain the training data perfectly and generalise poorly — a mixture model can put a component on a single point with vanishing variance and drive the likelihood to infinity. Regularisation, priors, and held-out validation are the answers.

Finding the maximum in practice

Some models have a closed form: the mean for a normal, the proportion for a binomial, the sample mean for a Poisson rate. Most do not, and the maximum is found numerically.

from scipy.optimize import minimize
import numpy as np

def neg_log_likelihood(params, data):
    mu, log_sigma = params                 # optimise log sigma to keep it positive
    sigma = np.exp(log_sigma)
    return -np.sum(-0.5 * np.log(2 * np.pi * sigma**2)
                   - (data - mu)**2 / (2 * sigma**2))

result = minimize(neg_log_likelihood, x0=[0.0, 0.0], args=(data,))

Two habits from that snippet generalise. Optimise unconstrained transforms of constrained parameters — log of a standard deviation, logit of a probability — so the optimiser cannot wander into invalid values. And write the negative log-likelihood, because every optimiser minimises.

Training a neural network is the same procedure at scale: the loss is a negative log-likelihood, and gradient descent is the numerical maximiser.

Questions people ask

What is the difference between probability and likelihood? Probability fixes the parameters and asks about data; likelihood fixes the data and asks about parameters. The function is the same; what varies is different.

Why maximise the log instead of the likelihood? Products of many small numbers underflow, and sums are easier to differentiate. The maximum is in the same place.

Is MLE the same as least squares? Least squares is the MLE when the errors are assumed normal with constant variance. Change the noise assumption and you get a different loss.

How does this relate to Bayesian estimation? MLE uses only the data. MAP adds a prior and maximises the posterior. Full Bayesian inference keeps the whole posterior distribution rather than one point.

Can likelihood exceed 1? For continuous data, yes — likelihoods are densities, not probabilities. Only the location of the maximum matters.

What if the likelihood has several peaks? Then the optimiser may land on a local one. Multiple random restarts are the standard defence, and it is exactly why k-means and mixture models use n_init.

Recap in one screen

• MLE picks the parameters that make the observed data most probable.
• Work with the log-likelihood: products become sums, and underflow disappears.
• Optimisers minimise, so the quantity in code is the negative log-likelihood.
• MSE, MAE and cross-entropy are all negative log-likelihoods under different noise assumptions.
• Adding a prior turns MLE into MAP, and recovers ridge and lasso as special cases.
• Consistent and efficient in large samples; biased and prone to overfitting in small ones.