Projections

The formula, and where it comes from

projₐ(b) = ( (a · b) / (a · a) ) a

The projection has to lie on the line, so it must be some multiple t·a. The residual b − t·a has to be perpendicular to a, so its dot product with a is zero. Solve a · (b − t·a) = 0 for t and you get t = (a · b) / ‖a‖². That is the whole derivation.

Two things worth noticing. The length of a cancels out — only its direction matters, which is why doubling a leaves the shadow exactly where it was. And t can be negative, which simply means the shadow falls on the far side of the origin.

The scalar projection is (a · b) / ‖a‖, the signed length of the shadow. The vector projection is that length pointed along a. Mixing the two up is the most common slip here.

Casting a shadow onto a direction

A projection answers: how much of vector a lies along the direction of vector b? Picture the sun directly overhead of b and a casting a shadow onto it.

proj_b(a) = ( (a · b) / (b · b) ) b

The fraction is a number saying how far along b to go; multiplying by b turns it back into a vector.

For a = [3, 4] and b = [1, 0] (the x-axis):

• a · b = 3, b · b = 1, so the projection is 3 × [1, 0] = [3, 0].

Which is exactly what you would expect: the shadow of [3, 4] on the x-axis is [3, 0], and the part left over, [0, 4], is perpendicular to b.

That decomposition is the point of the whole topic. Any vector splits uniquely into a part along a direction and a part at right angles to it:

a = proj_b(a) + (a − proj_b(a))

The second term is the residual, and it is perpendicular to b by construction.

Least squares is a projection

Linear regression fits a line by minimising squared residuals. Geometrically it is projecting.

The target vector y lives in n-dimensional space (one dimension per data point). The predictions the model can produce — every possible combination of the feature columns — form a subspace, usually of far lower dimension. Almost certainly, y is not in that subspace, which is why a perfect fit is impossible.

The best approximation is the point in the subspace closest to y, and that is exactly the projection of y onto it. The residual is what is left over, and the defining property is that it is perpendicular to every feature column:

Xᵀ(y − Xβ) = 0  →  β = (XᵀX)⁻¹Xᵀy

That is the normal equation, derived without calculus. "Normal" here means perpendicular — the equation is stating that the residual is orthogonal to the columns of X.

This also explains a fact that puzzles people: the residuals of a fitted least-squares model sum to zero (when an intercept is included), because the constant column is one of the directions the residual must be perpendicular to.

Orthogonality, and why it makes things easy

Two vectors are orthogonal when their dot product is zero — no shared direction at all.

An orthonormal basis is a set of mutually orthogonal unit vectors, and it makes projection trivial: the component along each basis vector is just the dot product, with no division needed. Decomposing a vector becomes a list of dot products.

This is why so much of applied linear algebra is spent constructing orthogonal bases:

• PCA produces orthogonal components, so each carries information the others do not.
• QR decomposition factors a matrix into an orthogonal part and a triangular part, and is the numerically stable way to solve least squares — better than forming XᵀX and inverting it.
• Fourier and wavelet transforms project a signal onto an orthogonal basis of frequencies.
• Gram-Schmidt is the classic procedure for turning any independent set into an orthonormal one, by repeatedly subtracting projections.

Split a vector in two

The projection and the residual, and the three properties that make least squares work.

import numpy as np

a = np.array([4.0, 3.0])       # what we project
b = np.array([1.0, 0.0])       # onto this direction

proj = (a @ b) / (b @ b) * b
residual = a - proj

print("a        :", a)
print("onto b   :", proj)
print("residual :", residual)
print()
print("the residual is perpendicular to b: dot =", round(residual @ b, 12))
print("and the parts add back up:", np.allclose(proj + residual, a))
print()
print("|a|^2 = |proj|^2 + |residual|^2 :",
      np.allclose(a @ a, proj @ proj + residual @ residual))
print("which is Pythagoras, and is why least squares works.")

Things to try

1. Read the default. a = (3, 1), b = (2, 3). The dot product is 9, ‖a‖² is 10, so t = 0.900 and the shadow lands at (2.70, 0.90) — short of b, and on the line.
2. Confirm the right angle. proj · r reads 0.000. Drag anything you like and it stays 0.000, because that orthogonality is what defines the projection rather than a coincidence of these numbers.
3. Stretch a. Double a to (6, 2). t halves to 0.450 and the shadow stays exactly where it was, at (2.70, 0.90). Only the direction of a was ever used.
4. Make the shadow vanish. Point b at right angles to a — with a = (3, 1), try b = (-1, 3). The dot product is 0, so the projection collapses to the origin and the residual is the whole of b. a explains nothing about b.
5. Make the residual vanish. Point b along a instead. Now the shadow is b itself and ‖r‖ is 0 — a explains all of b.
6. Go negative. Swing b behind the origin relative to a. t goes negative and the shadow points the other way down the line, which is exactly what a negative dot product means.

Why this is the root of least squares

Fitting a line to data means solving Xw = y when there is no exact solution: y almost never lies in the space that the columns of X can reach. The best you can do is find the point in that space closest to y — which is the projection of y onto the column space of X.

The residual has to be perpendicular to every column, so Xᵀ(y − Xw) = 0, which rearranges into the normal equations XᵀXw = Xᵀy. That is where the closed-form solution of ordinary least squares comes from: not calculus, geometry. The same picture on this page, in as many dimensions as you have features.

It is also why the residuals of a fitted linear model are uncorrelated with its predictors by construction, and why "the errors look structured" means your model is missing a direction rather than being unlucky.

Where else it shows up

• PCA. Projecting points onto a direction and asking which direction keeps the most spread — the residuals on that page are these residuals.
• Gram-Schmidt. Building an orthogonal basis by repeatedly subtracting off the projection onto what you already have.
• Cosine similarity. The projection of a unit vector onto another unit vector is the cosine of the angle between them.
• Attention. A dot product against a key vector is a projection, scaled — it measures how much of the query points along that key.

The short of it

The projection of b onto a is ((a·b)/‖a‖²)a: the point on the line through a that is closest to b, and the only one whose residual is perpendicular to a. Only a's direction matters, so scaling a changes nothing; the scale factor goes negative when the shadow falls behind the origin, zero when the vectors are perpendicular, and equals b exactly when they are parallel. Least squares is this picture with a subspace in place of a line — the fitted values are a projection of y, the residual is orthogonal to every predictor, and the normal equations are just that orthogonality written down.

Projections in machine learning

Dimensionality reduction. PCA projects data onto the subspace spanned by the top principal components. Everything perpendicular to that subspace is discarded, and the explained variance ratio measures how much of the data survived the projection.

Attention. Each token is projected into query, key and value spaces by learned matrices — three different views of the same vector, chosen so the comparison the model needs becomes a simple dot product.

Embeddings. Projecting high-dimensional embeddings down to two or three dimensions is how they are visualised; the projection is lossy by construction, which is why t-SNE and UMAP plots should be read for structure, not for exact distances.

Regularised regression. Ridge regression shrinks the projection along low-variance directions more than along high-variance ones, which is exactly why it stabilises collinear problems.

Gradient projection. Constrained optimisation keeps parameters on a permitted surface by projecting each step back onto it after taking it.

The projection matrix

Projecting onto a subspace can be written as a single matrix, applied to any vector:

P = X (XᵀX)⁻¹ Xᵀ

In statistics this is called the hat matrix, because it puts the hat on y: ŷ = Py.

It has two properties that identify any projection matrix. It is idempotent — P² = P, because projecting something that is already in the subspace changes nothing. And it is symmetric.

Its diagonal entries are the leverage values, measuring how much each observation influences its own fitted value. A point with high leverage sits far from the centre of the feature space and can pull the whole fit towards itself, which is why leverage is one of the standard regression diagnostics alongside residuals.

Questions people ask

What is the difference between projection and rotation? A rotation preserves lengths and is reversible; a projection discards a component and is not.

Can I project onto more than one direction? Yes — onto any subspace. Project onto each vector of an orthonormal basis and add the pieces.

Why is the residual perpendicular? Because if it had any component along the subspace, you could move within the subspace and get closer — so the closest point is precisely the one where nothing is left pointing that way.

Is projection the same as dropping a column? Only in the special case of projecting onto an axis. General projections combine features rather than discarding them.

How does this relate to cosine similarity? The scalar projection of a onto b is |a| cos θ. Divide by |a| and you have the cosine itself.

Why use QR instead of the normal equation? Numerical stability. Forming XᵀX squares the condition number, so QR (or SVD) gives more accurate coefficients on ill-conditioned data.

Recap in one screen

• A projection is the shadow of one vector onto a direction or subspace.
• Every vector splits into a projected part and a perpendicular residual.
• Least squares projects the targets onto the space of achievable predictions, which is where the normal equation comes from.
• Orthonormal bases make projection a set of dot products, which is why PCA, QR and Fourier methods build them.
• The hat matrix is the projection in matrix form, and its diagonal gives each point's leverage.