Covariance and Correlation

Covariance, one point at a time

cov(x, y) = (1/n) Σ (xᵢ − x̄)(yᵢ − ȳ)

For each point, take how far x is from its mean, multiply by how far y is from its mean, and average. The sign of each product is all that matters:

• Both above their means, or both below → the product is positive.
• One above, one below → the product is negative.

The scatter above colours points by exactly this. Green points are pushing covariance up; orange are pulling it down. A cloud that leans up-and-right has mostly green; one that leans down-and-right has mostly orange; a shapeless cloud has both in equal measure and they cancel.

Why correlation exists

Covariance has a fatal flaw for reporting: it carries the units of both variables multiplied together. Measure height in metres and weight in kilograms and you get one number; switch height to centimetres and the same data gives a number a hundred times larger. Nothing about the relationship changed.

Dividing by both standard deviations cancels the units out:

r = cov(x, y) / ( σₓ σᵧ )

The result is always between −1 and 1, whatever the data is measured in. That is the entire difference between the two quantities: correlation is standardised covariance.

Do two variables move together?

Covariance measures whether two variables tend to rise and fall together.

cov(X, Y) = Σ(x − x̄)(y − ȳ) / (n − 1)

The logic is in the product. When both variables are above their means, the product is positive. When both are below, negative times negative is positive again. When one is above and the other below, the product is negative. Add them up and the sign tells you the direction of the relationship.

The problem is the size. Covariance is in the product of the two units — "pound-square-metres" for price against floor area — so the number itself is uninterpretable and changes if you switch from metres to centimetres.

Correlation fixes that by dividing out both standard deviations:

r = cov(X, Y) / (σₓ σₖ)

The result is always between −1 and 1, unitless, and comparable across any pair of variables.

Change the units, watch one move

The same two variables, with height measured in metres and then in centimetres. One number survives the change and one does not.

import numpy as np

rng = np.random.default_rng(0)
height_m = rng.normal(1.7, 0.1, 500)          # metres
weight_kg = 40 * height_m + rng.normal(0, 5, 500)

for label, h in [("height in metres", height_m),
                 ("height in cm", height_m * 100)]:
    c = np.cov(h, weight_kg)[0, 1]
    r = np.corrcoef(h, weight_kg)[0, 1]
    print("%-18s covariance %10.3f   correlation %6.3f" % (label, c, r))
print()
print("the relationship did not change. covariance is in units of")
print("height times weight, so it moves with the units; correlation")
print("divides them out and does not.")

What correlation cannot see

Two limitations matter more than any other, and both cause real mistakes.

It only measures straight-line relationships. Take y = x² over the range −5 to 5. The relationship is perfect and deterministic, and the correlation is 0, because the rising half cancels the falling half. Anscombe's quartet makes the same point: four datasets with identical correlations and utterly different shapes. Always plot the scatter before trusting an r.

Correlation is not causation. Three explanations produce the same r: X causes Y, Y causes X, or a third variable causes both. Ice cream sales correlate with drownings because both follow temperature. Establishing causation requires an intervention — a randomised experiment — or careful causal modelling. No amount of correlation supplies it.

A third, quieter limitation: correlation is sensitive to outliers. A single extreme point can create an r of 0.8 in data that is otherwise a shapeless cloud, or destroy a genuine relationship. Spearman's rank correlation, which correlates the ranks rather than the values, is the robust alternative and also detects monotonic non-linear relationships.

Correlation matrices, and multicollinearity

With several features, the correlation matrix shows every pair at once, and it is one of the first things worth plotting on a new dataset.

import pandas as pd
corr = df.corr(numeric_only=True)          # Pearson by default
corr = df.corr(method="spearman")          # rank-based, robust

Two things to look for. High correlation between a feature and the target is promising. High correlation between two features is a warning: multicollinearity makes linear model coefficients unstable and uninterpretable, flipping sign when a row is added, even though the predictions stay fine.

The responses are to drop one of the pair, combine them, use PCA, or move to ridge regression, which handles correlated features gracefully by shrinking them together.

Tree-based models are largely unaffected in their predictions, but their feature importances get split arbitrarily between correlated columns, making both look less important than either really is.

Things to try

1. Prove the units problem. With the Pattern on Linear, drag the X Unit Scale slider from 1 to 10. Covariance grows roughly tenfold and so does the standard deviation of X — but correlation does not move. This is the whole reason r is what gets reported.
2. Sweep the strength. Drag the Target Correlation slider from −1 to 1 and watch the cloud tilt from down-right, through shapeless, to up-right. At ±1 the points collapse onto a single line.
3. Meet the big failure. Set the Pattern to U-shaped curve. Every point sits on a clean parabola — y is completely determined by x — yet r sits near zero. Correlation measures straight-line association only, and a symmetric curve has none.
4. Watch one point take over. Set the Pattern to One extreme outlier. A single far-off point drags r away from what the bulk of the data is doing. Correlation is not robust.
5. See small samples lie. Set the Pattern to No relationship and the Sample Size slider to 10, then press Resample a few times. r bounces around, sometimes reaching 0.5 or beyond, purely by chance. Now set the sample size to 400 and resample again — it stays near zero.
6. Read the ellipse. With Show Spread Ellipse on, notice that its long axis tilts with the correlation and its axes stay perpendicular. Those axes are the eigenvectors of the covariance matrix, which is precisely what PCA computes.

Three ways it misleads

• Correlation is not causation. The oldest warning in statistics and still the most ignored. Ice cream sales correlate with drownings; the summer heat causes both. A third variable driving two others is called confounding and produces textbook-perfect correlations.
• Correlation is not dependence. The U-shaped preset shows a perfectly deterministic relationship with r near zero. "Uncorrelated" and "independent" mean the same thing only for normally distributed data; in general, independence implies zero correlation, but never the reverse.
• A single number hides the shape. Anscombe's quartet is four datasets with identical means, variances and correlations that look nothing alike when plotted. Always plot the scatter.

Where it matters in machine learning

• PCA is the eigendecomposition of the covariance matrix. The ellipse you can see above is the object it operates on.
• Multicollinearity — highly correlated features make linear regression coefficients unstable and hard to interpret, which is one of the problems Ridge was designed to relieve.
• Feature selection by correlation with the target will silently discard any feature whose relationship is non-linear, which is exactly the failure the U-shaped preset demonstrates.
• r² is the fraction of variance in one variable explained by a straight-line fit on the other — the same quantity linear regression reports.

Traps worth knowing

• Comparing covariances across datasets. They are in different units and are not comparable. Compare correlations instead.
• Trusting r on a small sample. With ten points a correlation of 0.5 is unremarkable noise. Report a confidence interval or a sample size alongside it.
• Using Pearson's r on ranked or heavily skewed data. Spearman's rank correlation handles monotonic-but-not-linear relationships and is far less sensitive to outliers.
• Reading r = 0 as "no relationship". It means no linear relationship. Plot it before concluding anything.

Where that leaves you

Covariance averages the product of each variable's deviation from its own mean, so it is positive when two variables move together and negative when they move oppositely — but it carries the units of both, which makes its magnitude meaningless on its own. Correlation divides by both standard deviations to strip the units out, giving a number always between −1 and 1 that is unchanged by rescaling. Both measure straight-line association only: a perfect U-shaped relationship gives a correlation near zero, a single outlier can dominate the result, and small samples produce large correlations by chance, so the scatter plot is never optional.

The covariance matrix

For several variables, covariances are collected into a square matrix: variances on the diagonal, covariances off it. It is symmetric, because cov(X, Y) = cov(Y, X).

This object is more important than it first appears:

• PCA is the eigendecomposition of the covariance matrix. Its eigenvectors are the principal components and its eigenvalues are the variances along them.
• A multivariate normal distribution is defined by a mean vector and a covariance matrix, and the matrix determines the shape and orientation of the resulting ellipse.
• Mahalanobis distance uses its inverse to measure distance in units of the data's own spread, which is why it detects outliers that per-feature thresholds miss.
• Portfolio theory uses it to compute the variance of a weighted combination, which is how diversification is quantified.

Standardising the data first turns the covariance matrix into the correlation matrix — the same information, expressed in comparable units. That is exactly why PCA is normally run on standardised data.

Choosing the right coefficient

A practical routine on a new dataset: compute Pearson and Spearman side by side. A large gap between them is a signal — either the relationship is non-linear, or outliers are distorting the Pearson value. Either way it points you at a scatter plot worth looking at.

Questions people ask

What does r = 0 mean? No linear relationship. There may be a perfect non-linear one.

Is a correlation of 0.5 strong? It depends entirely on the field. In physics it would be weak; in behavioural research it is substantial. Interpret against the domain, not a fixed table.

How is r related to R²? For simple linear regression with one predictor, R² is exactly r squared. So r = 0.7 means about half the variance is explained.

Can correlation be used with categorical data? Not Pearson. Use Cramér's V for two categoricals, or the point-biserial correlation for one binary and one continuous.

Why does covariance have such odd units? Because it is a product of two deviations. That is the entire reason correlation exists.

Does a high correlation with the target mean the feature is useful? It is a good sign, but a feature that is uncorrelated alone can be very useful in combination with others, and a highly correlated feature can be a leak from the future. Check both.

Recap in one screen

• Covariance gives the direction of a linear relationship, in uninterpretable units.
• Correlation divides by both standard deviations, giving a unitless number from −1 to 1.
• r = 0 means no linear relationship, not no relationship — always look at the scatter plot.
• Correlation never establishes causation; a confounder produces the same number.
• Spearman is the robust, monotonic alternative; mutual information catches any dependence.
• The covariance matrix is what PCA decomposes and what defines a multivariate normal.