Maximum Likelihood Estimation
Slide a candidate distribution over fixed data and watch the likelihood peak. This is where almost every loss function in machine learning comes from.
Candidate
Data and Candidate
log L = 0.00Each stem is one point's likelihood under the current candidate. Taller is better.
Log-Likelihood as the Mean Moves
Spread is held fixed. The peak sits exactly at the sample mean.
Fit
The Answer
For a normal distribution the maximum-likelihood mean is exactly the sample mean, and the maximum-likelihood spread is exactly the sample standard deviation. The formulas you already knew are the answer to this optimisation.
Maximum Likelihood Estimation
Maximum likelihood chooses parameter values that make the observed data most probable under a model.
Why use log-likelihood?
Because logarithm is increasing, maximising L and maximising log L give the same parameter estimate. The log converts products into sums and is more numerically manageable:
An MLE may be biased in finite samples, may not be unique, and can lie on a boundary or fail to exist without suitable assumptions. See logarithms, probability models, and cross-entropy.
Find the estimate for coin flips
Suppose 8 of 10 independent flips are heads. Under a Bernoulli model with unknown head probability p, the likelihood is L(p)=p⁸(1−p)². It asks: for each candidate value of p, how plausible is this observed sequence under the model? At p=0.5 the likelihood is 0.5¹⁰; at p=0.8 it is 0.8⁸0.2², which is larger. Differentiating the log-likelihood, 8log p+2log(1−p), gives the maximum at p̂=8/10=0.8. The estimate is a consequence of the Bernoulli model and these data; it is not proof that the true probability is exactly 0.8.
Likelihood values are not posterior probabilities: they need not sum or integrate to one across parameter values. A prior distribution and Bayes' rule are needed for a posterior. MLE also says nothing by itself about precision; a small sample can produce a highly uncertain estimate. In more complex models, numerical optimisation may find a local maximum, and different parameter values can yield the same maximum. Report the model, estimate, and an uncertainty measure rather than treating the maximizing value as the whole result.
Try this above
Move the model parameter while holding the observations fixed. Find where the likelihood peaks, then compare it with the log-likelihood curve.
For normally distributed observations with known variance, the MLE of the mean is the sample mean because that value maximises the product of the normal densities. This illustrates that the same statistic can arise from a model-based optimisation, not from a universal rule. If observations are dependent, the likelihood must include that dependence; multiplying marginal probabilities would be wrong. If the model is misspecified, the MLE may still converge to a best-fitting parameter within the wrong model rather than the real-world quantity someone intended.
An MLE is not invariant to every reparameterisation in the same numeric way: if φ=g(θ) is one-to-one, the maximizing value maps to g(θ̂), but a density-based likelihood for transformed parameters includes a Jacobian factor when interpreted as a probability density. This distinction matters when comparing likelihoods across parameterisations. The data likelihood is not normalised over θ like a posterior; Bayesian inference adds a prior and normalises over parameter values.
Sources and next steps
- OpenStax: Introductory Statistics 2e introduces likelihood-based estimation.
- Continue with confidence intervals to describe uncertainty around an estimate.
Overview
Build the likelihood
Given observations x₁,…,xₙ and a model with parameter θ, the likelihood is L(θ) = ∏ᵢ p(xᵢ|θ), assuming the observations are independent under that model. The maximum-likelihood estimate is the θ that maximises this function. Likelihood is a function of the parameter for fixed observed data; it is not the probability that the parameter is true.
For n Bernoulli observations with k successes, L(p) = pᵏ(1−p)ⁿ⁻ᵏ. Maximising it gives p̂ = k/n: the observed success fraction. The estimate follows from the chosen model and sample; another model can yield a different estimate.