Rank and Linear Independence
Two columns reach a whole plane — until one of them becomes a multiple of the other, and everything you can reach falls onto a single line.
The Two Columns
every dot is some combination s·c₁ + t·c₂
The Span
—Green and blue are the columns. The dots are everything they can build between them.
Rank
The Dependency
Two independent columns reach every point in the plane, so any target can be hit and there is exactly one way to hit it.
Rank and Linear Independence
Rank counts how many independent directions remain in a matrix's columns or rows.
What rank reveals
Rank tells you the number of independent constraints or representable directions. A low-rank data matrix has redundancy that can sometimes be compressed; singular value decomposition reveals the dominant directions. Rank also predicts whether Ax = b has a unique solution, many solutions, or no solution, but the answer depends on consistency and the number of unknowns as well as rank.
Exact rank is sensitive to tiny floating-point noise. Numerical software therefore uses a tolerance when deciding which singular values count as zero. A nearly dependent matrix may be formally full-rank yet poorly conditioned.
Connect rank to solving Ax=b
For an m×n matrix, rank is the number of independent columns, equivalently the dimension of its column space; it is also the number of independent rows. If the columns are independent, no coefficient vector other than zero maps to zero. In A=[[1,2],[2,4]], the second column doubles the first, so there is only one independent direction and rank is 1.
Rank helps classify linear systems, but rank alone does not give the answer. Ax=b is consistent only if b lies in A's column space. If it is consistent, the number of free parameters is n−rank(A); a unique solution requires full column rank (rank n). A square matrix is invertible exactly when its rank equals its size. For rectangular systems, the determinant is unavailable, while rank still applies. Numerical software estimates rank by treating singular values below a tolerance as zero, so changing the tolerance can change the reported numerical rank. A matrix that is technically full rank but nearly dependent can still produce unstable solutions; see SVD.
Try this above
Make one vector a multiple of another and watch the rank drop. Then perturb one component slightly; notice how exact dependence differs from near-dependence.
The rank-nullity theorem states rank(A)+nullity(A)=n for an m×n matrix, where n is the number of columns and nullity is the dimension of solutions to Ax=0. Each independent null direction represents a change to x that leaves Ax unchanged. This explains why rank deficiency creates non-unique parameter vectors even when predictions are the same. In data fitting, multiple coefficient vectors can represent an identical output when features are redundant; removing or regularising redundancy can improve interpretability and numerical stability.
Sources and next steps
- MIT OpenCourseWare: Linear Algebra develops rank, column spaces, and independence.
- See basis and span for how independent vectors form coordinates.
Overview
Independence means no redundancy
Vectors are linearly independent when the only combination that sums to zero uses all-zero coefficients. If one vector can be built from the others, it adds no new direction. The rank of a matrix is the dimension of its column space; it also equals the dimension of its row space.
For A = [[1, 2], [2, 4]], the second column is twice the first, so the columns are dependent and rank(A) = 1. The map can produce points on a line, but not fill the plane. For a square matrix, full rank is equivalent to a nonzero determinant; see determinants.