Covariance and Correlation

Quick Context

Variance describes how one variable spreads out. Covariance is the same idea for two: when x is above its mean, is y usually above its mean too?

Correlation is covariance with the units removed. That single change is what makes it comparable across datasets — and, as we will see, still leaves it blind to anything that is not a straight line.

Covariance, one point at a time

cov(x, y) = (1/n) Σ (xᵢ − x̄)(yᵢ − ȳ)

For each point, take how far x is from its mean, multiply by how far y is from its mean, and average. The sign of each product is all that matters:

• Both above their means, or both below → the product is positive.
• One above, one below → the product is negative.

The scatter above colours points by exactly this. Green points are pushing covariance up; orange are pulling it down. A cloud that leans up-and-right has mostly green; one that leans down-and-right has mostly orange; a shapeless cloud has both in equal measure and they cancel.

Why correlation exists

Covariance has a fatal flaw for reporting: it carries the units of both variables multiplied together. Measure height in metres and weight in kilograms and you get one number; switch height to centimetres and the same data gives a number a hundred times larger. Nothing about the relationship changed.

Dividing by both standard deviations cancels the units out:

r = cov(x, y) / ( σₓ σᵧ )

The result is always between −1 and 1, whatever the data is measured in. That is the entire difference between the two quantities: correlation is standardised covariance.

Interactive Exploration Guide

1. Prove the units problem. With the Pattern on Linear, drag the X Unit Scale slider from 1 to 10. Covariance grows roughly tenfold and so does the standard deviation of X — but correlation does not move. This is the whole reason r is what gets reported.Run this
2. Sweep the strength. Drag the Target Correlation slider from −1 to 1 and watch the cloud tilt from down-right, through shapeless, to up-right. At ±1 the points collapse onto a single line.Run this
3. Meet the big failure. Set the Pattern to U-shaped curve. Every point sits on a clean parabola — y is completely determined by x — yet r sits near zero. Correlation measures straight-line association only, and a symmetric curve has none.Run this
4. Watch one point take over. Set the Pattern to One extreme outlier. A single far-off point drags r away from what the bulk of the data is doing. Correlation is not robust.Run this
5. See small samples lie. Set the Pattern to No relationship and the Sample Size slider to 10, then press Resample a few times. r bounces around, sometimes reaching 0.5 or beyond, purely by chance. Now set the sample size to 400 and resample again — it stays near zero.Run this
6. Read the ellipse. With Show Spread Ellipse on, notice that its long axis tilts with the correlation and its axes stay perpendicular. Those axes are the eigenvectors of the covariance matrix, which is precisely what PCA computes.

Three ways it misleads

• Correlation is not causation. The oldest warning in statistics and still the most ignored. Ice cream sales correlate with drownings; the summer heat causes both. A third variable driving two others is called confounding and produces textbook-perfect correlations.
• Correlation is not dependence. The U-shaped preset shows a perfectly deterministic relationship with r near zero. "Uncorrelated" and "independent" mean the same thing only for normally distributed data; in general, independence implies zero correlation, but never the reverse.
• A single number hides the shape. Anscombe's quartet is four datasets with identical means, variances and correlations that look nothing alike when plotted. Always plot the scatter.

Where it matters in machine learning

• PCA is the eigendecomposition of the covariance matrix. The ellipse you can see above is the object it operates on.
• Multicollinearity — highly correlated features make linear regression coefficients unstable and hard to interpret, which is one of the problems Ridge was designed to relieve.
• Feature selection by correlation with the target will silently discard any feature whose relationship is non-linear, which is exactly the failure the U-shaped preset demonstrates.
• r² is the fraction of variance in one variable explained by a straight-line fit on the other — the same quantity linear regression reports.

What usually goes wrong

• Comparing covariances across datasets. They are in different units and are not comparable. Compare correlations instead.
• Trusting r on a small sample. With ten points a correlation of 0.5 is unremarkable noise. Report a confidence interval or a sample size alongside it.
• Using Pearson's r on ranked or heavily skewed data. Spearman's rank correlation handles monotonic-but-not-linear relationships and is far less sensitive to outliers.
• Reading r = 0 as "no relationship". It means no linear relationship. Plot it before concluding anything.

Key Takeaway

Covariance averages the product of each variable's deviation from its own mean, so it is positive when two variables move together and negative when they move oppositely — but it carries the units of both, which makes its magnitude meaningless on its own. Correlation divides by both standard deviations to strip the units out, giving a number always between −1 and 1 that is unchanged by rescaling. Both measure straight-line association only: a perfect U-shaped relationship gives a correlation near zero, a single outlier can dominate the result, and small samples produce large correlations by chance, so the scatter plot is never optional.