Projections
Shine a light straight down onto the line through a. The shadow b casts is the projection, and what is left over is always at right angles to it.
The Two Vectors
or drag either arrowhead on the plot
The Shadow And What Is Left
—Green is a, blue is b, orange is the projection of b onto a.
The Projection
The Residual
That last number is zero for every a and b you can choose. The residual is what a cannot explain, and it is always perpendicular to a.
Projections
A projection keeps the component of a vector that lies in a chosen subspace and discards the rest.
Project onto a subspace
With orthonormal basis columns in Q, projection onto their span is QQᵀv. The residual is perpendicular to every column of Q. Least-squares fitting uses this geometry: it projects observed data onto the column space of a design matrix. QR decomposition makes this calculation stable without explicitly forming the projection matrix; see QR decomposition and orthogonality.
A projection is idempotent: projecting twice gives the same result as once. It is not generally a rotation or a scaling; components outside the selected subspace are removed.
Projection explains least-squares fitting
The projection is the closest point in the chosen line or subspace to v under Euclidean distance. For a line generated by a, the residual r=v−projₐ(v) is perpendicular to a. This perpendicularity characterises the closest point: moving a little along the line cannot reduce the squared distance. For a subspace with orthonormal basis Q, the projected vector is QQᵀv, and Qᵀr=0 expresses perpendicularity to every basis direction.
In least squares, a design matrix X has columns spanning all predictions the model can produce. The fitted vector Xβ is the projection of observed y onto that column space, so the residual y−Xβ is orthogonal to every column of X. This gives the normal equations XᵀXβ=Xᵀy. QR factorisation is generally a more stable way to solve the least-squares problem than explicitly forming XᵀX. Projection matrices are symmetric and idempotent (P²=P); they do not usually preserve the length of vectors because the discarded component shortens the result.
Try this above
Move the vector and projection direction. Check that the residual stays perpendicular to the direction while the projected point moves along it.
Projection is also a way to decompose a vector into explained and unexplained parts: v=proj(v)+r, with the two parts perpendicular. The squared lengths add, ‖v‖²=‖proj(v)‖²+‖r‖², by the Pythagorean theorem. In regression, the projection is the fitted response and the residual is what the model's column space cannot explain. A small residual does not establish that the model is appropriate; residual patterns can reveal curvature, unequal variance, or missing structure.
For a line generated by a, rescaling a does not change the projected vector: replacing a with ca changes both numerator and denominator by c², which cancels. This is a useful check that projection depends on the subspace, not the arbitrary length of its basis vector. For a non-orthonormal basis matrix, QQᵀ is not generally the projection; use A(AᵀA)⁻¹Aᵀ when columns are independent, or a QR/SVD method for better numerical stability.
Sources and next steps
- MIT OpenCourseWare: Linear Algebra covers projection matrices and least squares.
- See dot products for the scalar calculation behind a projection.
Overview
Project onto a direction
For nonzero vector a, the projection of v onto a is the closest point to v on the line spanned by a. Its scalar coordinate along a is (a · v)/(a · a); multiplying by a gives the projected vector:
For a = (1, 2) and v = (3, 1), a · v = 5 and a · a = 5, so the projection is (1, 2). The residual v − projₐ(v) = (2, −1) is perpendicular to a: their dot product is zero. If a = 0, projection onto its direction is undefined.