Clusters as overlapping distributions rather than hard groups, fitted by alternating between guessing and re-estimating.
Overview
Hard labels throw information away
K-means gives each point just one cluster. For a point sitting squarely inside a group that is fine. For a point in the share between two groups it's a fabrication: the method had almost no reason to prefer one over the other, and the output records none of that doubt.
A Gaussian mixture keeps the doubt. It models the data as having come from a few Gaussian spreads mixed together, and reports, for each point, the chance that it came from each one.
Raise the share control above and watch the readout. As the two groups merge, the number of points assigned with less than 90% trust climbs — and each one of those is a point k-means would have labelled without hesitation.
Gaussian Mixture Models
This module needs JavaScript: the
numbers are computed in the page rather than recorded.
Worth knowing
A GMM assumes the data came from several Gaussians mixed together, and fits their means, widths and weights.
Each point gets a probability of belonging to each component, not a single label.
EM alternates: given the current parameters, compute responsibilities; given responsibilities, re-estimate the parameters.
K-means is the special case where every component is spherical, equally sized, and every responsibility is 0 or 1.
Gaussian Mixture Models
What clustering looks like when a point is allowed to belong partly to two groups.
The model
Three params per component: a mean, a width, and a mixing weight saying what fraction of the data that component accounts for. The density is their weighted sum, and the two curves in the chart are the fitted components.
Fitting is a chicken-and-egg problem. Knowing which points belong to which component would make guessing the params easy, and knowing the params would make assigning points easy. Neither is known.
Expectation-Maximisation
EM resolves it by alternating, starting from a guess:
E step. With the current params, find each point's job — the chance that each component made it.
M step. With those jobs, re-guess each component's mean, width and weight, weighting each point by how much it belongs.
Repeat. Each iteration provably doesn't fall the likelihood, so the process settles.
Drag the iterations control from 1 upward and watch the curves settle. The early steps move a great deal. The last twenty barely move at all, which is what settling looks like.
The ensure is only about *local* optima. EM settles to a local max that depends on the starting point. This Is why setups run it a few times from different initialisations and keep the best. Scikit-learn's n_init exists for this.
Why it beats k-means, when it does
Elliptical clusters. With a full covariance mat per component, a GMM fits stretched and tilted clusters. K-means silently assumes spheres, so an elongated cluster gets cut in half.
Different sizes. Mixing weights let one component account for 80% of the data and another for 20%. K-means has no such notion and tends toward equal-sized clusters.
Soft set. handy in itself when the output feeds a thing downstream that can use a chance.
A likelihood. Being a proper probabilistic model, a GMM can score how well it shows the data, which makes BIC and AIC available for choosing the number of components — a principled option to the elbow.
K-means is just the limiting case: spherical components of equal weight, with jobs forced to 0 or 1.
The E-step, worked
The soft set the whole method turns on is one math. Two components, N(0,1) with weight 0.6 and N(3,1) with weight 0.4, and a point to assign: its job to each is the weighted density there, normed so the two sum to 1.
point
r(component 1)
r(component 2)
x = 0.5
0.968
0.032
x = 1.5
0.600
0.400
At x = 0.5 the point sits deep in the first component and is assigned to it with 97% trust — but k-means would have recorded the same hard label with no hint of the 3% doubt. At x = 1.5, halfway between the means, the jobs fall back to the mixing weights themselves: the data gives almost no reason to prefer either component, and the GMM says so where k-means would still pick one. That preserved doubt is the whole point, and it's why the number of low-trust sets climbs as the components share. The M-step then re-guesses each mean, width and weight using each point weighted by its job, and alternating the two provably never falls the likelihood — which is also why a GMM can report one, through BIC, and choose its own number of components where k-means can't.
What a GMM buys over k-means, when the data suits it, is three things at once: elliptical clusters, because a full covariance per component fits stretched and tilted shapes k-means cuts in half. Unequal sizes, because the mixing weights let one component own 80% of the data. And the soft set itself, handy when a thing downstream can act on a chance. K-means is just the limiting case — spherical components of equal weight with jobs forced to 0 or 1 — which is why it fabricates the trust a GMM keeps. The costs are the assumption of Gaussian components (crescents defeat it) and settling only to a local optimum, which is why libraries run a few initialisations and keep the best.
The other failure to watch is a singularity: a component can collapse onto a single point, driving its width toward zero and the likelihood toward infinity. A small constant added to each covariance. On by default in most libraries — blocks it, and it's the same smoothing instinct that stops too many components fitting the noise. This Is what BIC's fine on param count guards against when choosing how many components to keep.
Where it goes wrong with Gaussian Mixture Models
given Gaussian. If the clusters are crescents, a mixture of Gaussians is the wrong model, and it will fit two Gaussians to them anyway.
Singularities. A component can collapse onto a single point, driving its width to zero and the likelihood to infinity. Smoothing — a small constant added to the covariance. Blocks it, and is on by default in most libraries.
One run from one start. Local optima are real. Use a few initialisations.
Too many components. With enough Gaussians you can fit any of it, including the noise. BIC punishes param count for this reason.
Questions people ask about Gaussian Mixture Models
How is a GMM different from k-means? A GMM gives each point a probability of belonging to every cluster, and each cluster has its own shape through a covariance matrix. K-means gives hard labels and assumes round clusters.
How do I choose the number of components? Fit models for a range of component counts and pick the lowest BIC or AIC. BIC penalises complexity more heavily, so it tends to give simpler models.
Which covariance type should I use? Full covariance fits any ellipse but needs more data. Diagonal, spherical or tied reduce parameters, which helps when you have few points or many features.
Why does my GMM give different results each run? EM starts from random initial values and only finds a local optimum. Set n_init higher and fix random_state.
Can a GMM be used for anomaly detection? Yes. Points with a very low density under the fitted mixture are unusual, so you can threshold score_samples.
What is the E-step and M-step? The E-step computes how responsible each component is for each point. The M-step updates means, covariances and weights using those responsibilities, and the two repeat until the likelihood stops improving.
Soft clusters, and shapes k-means cannot fit
A GMM gives each point a chance of belonging to each cluster. Fits ellipses rather than spheres. Both differences show up on the same data.
example_01.pyscikit-learn
import numpy as np
from sklearn.mixture import GaussianMixture
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.metrics import adjusted_rand_score
X, y = make_blobs(n_samples=600, centers=3, cluster_std=0.8,
center_box=(-14.0, 14.0), random_state=0)
X = X @ np.array([[1.0, 0.0], [1.2, 0.35]])
gm = GaussianMixture(n_components=3, random_state=0).fit(X)
km = KMeans(n_clusters=3, n_init=10, random_state=0).fit(X)
print("three well-separated blobs, sheared into long tilted ellipses:")
print(" k-means agreement %.4f" % adjusted_rand_score(y, km.labels_))
print(" GMM agreement %.4f" % adjusted_rand_score(y, gm.predict(X)))
print(" k-means can only draw spheres, so it cuts across the grain")
print(" of the ellipses and splits them in the wrong places.")
print()
print("what a GMM stores per cluster -- weight, centre, covariance:")
for c in range(3):
print(" cluster %d weight %.4f centre %s" % (c, gm.weights_[c],
np.round(gm.means_[c], 3)))
print(" covariance %s" % np.round(gm.covariances_[c], 3).tolist())
print()
p = gm.predict_proba(X)
conf = p.max(axis=1)
print("every point gets a probability per cluster, not just a label:")
order = np.argsort(conf)
print(" least certain (border points):")
for i in order[:4]:
print(" %s -> %s" % (np.round(X[i], 2), np.round(p[i], 4)))
print(" most certain (deep inside a component):")
for i in order[-3:]:
print(" %s -> %s" % (np.round(X[i], 2), np.round(p[i], 4)))
print(" %d of %d points are less than 90%% sure." % ((conf < 0.9).sum(), len(X)))
Output
The next block compares covariance types and uses BIC to pick the number of components.
example_02.pyscikit-learn
import numpy as np
from sklearn.mixture import GaussianMixture
from sklearn.datasets import make_blobs
from sklearn.metrics import adjusted_rand_score
X, y = make_blobs(n_samples=600, centers=3, cluster_std=0.8,
center_box=(-14.0, 14.0), random_state=0)
X = X @ np.array([[1.0, 0.0], [1.2, 0.35]])
print("covariance_type trades flexibility for parameters:")
for ct in ("spherical", "diag", "tied", "full"):
m = GaussianMixture(n_components=3, covariance_type=ct, random_state=0).fit(X)
print(" %-10s agreement %.4f BIC %9.1f parameters %d"
% (ct, adjusted_rand_score(y, m.predict(X)), m.bic(X),
m._n_parameters()))
print(" 'spherical' is soft k-means. 'full' fits any ellipse.")
print(" 'tied' wins on BIC here because all three ellipses share one")
print(" shape, so it gets full's accuracy for spherical's parameter count.")
print()
print("BIC picks the number of components, which k-means cannot do:")
print("%6s %12s %12s" % ("k", "BIC", "AIC"))
for k in range(1, 7):
m = GaussianMixture(n_components=k, random_state=0).fit(X)
print("%6d %12.1f %12.1f" % (k, m.bic(X), m.aic(X)))
best = min(range(1, 7), key=lambda k: GaussianMixture(n_components=k,
random_state=0).fit(X).bic(X))
print(" lowest BIC at k=%d, and the data really has 3 components." % best)
print(" BIC is not infallible -- on overlapping components it under-counts.")
print(" it works because a GMM is a probability model: BIC compares how")
print(" well each k explains the data against how many parameters it spent.")
print()
print("the catch: with enough components a GMM puts one tiny component")
print("on a handful of points and drives variance toward zero, sending")
print("the likelihood to infinity. reg_covar exists to stop that.")
Output
Check yourself
0 of 3
Answer without scrolling back up.
What does a GMM report for each point?
That is the point: a point in the overlap between two groups records its doubt, where k-means would label it without hesitation.
What do the two steps of EM do?
It resolves a chicken-and-egg problem by alternating, and each iteration provably does not decrease the likelihood - though only to a local maximum.
K-means is the special case of a GMM where:
Which is why a GMM handles elliptical and unequally sized clusters that k-means cuts in half or splits evenly.
Cheat sheet
Gaussian Mixture Models
Fit a Gaussian mixture and watch each point split its membership across clusters. Step through EM iteration by iteration and compare to hard k-means.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.