Projections

Quick Context

Projection answers one question: of all the points on the line through a, which is closest to b? The answer is b's shadow, and the line from b down to that shadow is perpendicular. Those two facts — closest point, perpendicular error — are the same fact, and almost every fitting method in machine learning is built on it.

The formula, and where it comes from

projₐ(b) = ( (a · b) / (a · a) ) a

The projection has to lie on the line, so it must be some multiple t·a. The residual b − t·a has to be perpendicular to a, so its dot product with a is zero. Solve a · (b − t·a) = 0 for t and you get t = (a · b) / ‖a‖². That is the whole derivation.

Two things worth noticing. The length of a cancels out — only its direction matters, which is why doubling a leaves the shadow exactly where it was. And t can be negative, which simply means the shadow falls on the far side of the origin.

The scalar projection is (a · b) / ‖a‖, the signed length of the shadow. The vector projection is that length pointed along a. Mixing the two up is the most common slip here.

Interactive Exploration Guide

1. Read the default. a = (3, 1), b = (2, 3). The dot product is 9, ‖a‖² is 10, so t = 0.900 and the shadow lands at (2.70, 0.90) — short of b, and on the line.
2. Confirm the right angle. proj · r reads 0.000. Drag anything you like and it stays 0.000, because that orthogonality is what defines the projection rather than a coincidence of these numbers.
3. Stretch a. Double a to (6, 2). t halves to 0.450 and the shadow stays exactly where it was, at (2.70, 0.90). Only the direction of a was ever used.
4. Make the shadow vanish. Point b at right angles to a — with a = (3, 1), try b = (-1, 3). The dot product is 0, so the projection collapses to the origin and the residual is the whole of b. a explains nothing about b.
5. Make the residual vanish. Point b along a instead. Now the shadow is b itself and ‖r‖ is 0 — a explains all of b.
6. Go negative. Swing b behind the origin relative to a. t goes negative and the shadow points the other way down the line, which is exactly what a negative dot product means.

Why this is the root of least squares

Fitting a line to data means solving Xw = y when there is no exact solution: y almost never lies in the space that the columns of X can reach. The best you can do is find the point in that space closest to y — which is the projection of y onto the column space of X.

The residual has to be perpendicular to every column, so Xᵀ(y − Xw) = 0, which rearranges into the normal equations XᵀXw = Xᵀy. That is where the closed-form solution of ordinary least squares comes from: not calculus, geometry. The same picture on this page, in as many dimensions as you have features.

It is also why the residuals of a fitted linear model are uncorrelated with its predictors by construction, and why "the errors look structured" means your model is missing a direction rather than being unlucky.

Where else it shows up

• PCA. Projecting points onto a direction and asking which direction keeps the most spread — the residuals on that page are these residuals.
• Gram-Schmidt. Building an orthogonal basis by repeatedly subtracting off the projection onto what you already have.
• Cosine similarity. The projection of a unit vector onto another unit vector is the cosine of the angle between them.
• Attention. A dot product against a key vector is a projection, scaled — it measures how much of the query points along that key.

Key Takeaway

The projection of b onto a is ((a·b)/‖a‖²)a: the point on the line through a that is closest to b, and the only one whose residual is perpendicular to a. Only a's direction matters, so scaling a changes nothing; the scale factor goes negative when the shadow falls behind the origin, zero when the vectors are perpendicular, and equals b exactly when they are parallel. Least squares is this picture with a subspace in place of a line — the fitted values are a projection of y, the residual is orthogonal to every predictor, and the normal equations are just that orthogonality written down.