Rank and Linear Independence

Quick Context

Two columns are linearly independent when neither one can be built out of the other. When they are, the combinations s·c₁ + t·c₂ sweep out the whole plane. When one is a multiple of the other, everything you can build lies on a single line, no matter how hard you pull on s and t.

The rank is the count of genuinely independent directions — the dimension of what the columns can reach. Two columns can have rank 2, rank 1, or rank 0 if both are zero. They can never have rank 3, which is the first useful fact: rank is capped by the number of rows as well as the number of columns.

Ways of saying the same thing

For a square matrix, all of these are one statement:

• the columns are linearly dependent;
• the determinant is zero;
• the rank is less than the number of columns;
• the matrix has no inverse;
• Ax = 0 has a solution other than x = 0;
• at least one eigenvalue is zero.

Column rank always equals row rank, which is a genuinely surprising theorem: the number of independent columns and the number of independent rows are the same number, whatever shape the matrix is.

Interactive Exploration Guide

1. Start independent. c₁ = (2, 1), c₂ = (1, 2). The determinant is 3, the rank is 2, and the reachable dots fill the plane in a lattice.
2. Collapse it. Press Make Column 2 A Multiple Of Column 1. c₂ becomes 1.5×c₁ = (3.0, 1.5), the determinant goes to exactly 0 and every dot falls onto one line. Adding a second column bought you nothing.Run this
3. Nudge it back. Move Column 2 y by a single notch. The rank jumps straight back to 2 — rank is a count, so it does not degrade gradually, it steps. That is also why "nearly dependent" needs a different measure entirely.Run this
4. Try the near-miss. Set c₂ to (3.0, 1.6) instead: the determinant is 0.20, technically rank 2, and the lattice is stretched into a nearly-flat sliver. This is what collinear features look like, and why a fit on them is unstable.
5. Zero a column. Set both of c₂'s components to 0. Rank 1 again: a zero column contributes no direction at all.
6. Zero both. Rank 0, and the only reachable point is the origin.

Where this bites in practice

• The dummy variable trap. One-hot encode a category into k columns and keep an intercept, and the k dummies sum to the intercept column exactly. The design matrix is rank deficient, XᵀX is singular, and ordinary least squares has no unique solution. Dropping one dummy fixes it.
• Duplicated features. Height in cm and height in inches are one direction written twice. The rank does not go up; the instability does.
• More features than rows. With p > n the rank is at most n, so the columns are guaranteed dependent and the model can fit the training data perfectly for reasons that have nothing to do with the data. Regularisation is the standard answer.
• Low-rank approximation. Keeping only the strongest few directions of a matrix is what PCA does, and what the "low rank" in LoRA refers to: a big weight update approximated by the product of two thin matrices, because the update did not need every direction.

Key Takeaway

Rank counts the independent directions a matrix's columns actually provide, which is the dimension of everything they can reach. Full rank means the span is as large as it could be and the matrix is invertible; a drop in rank means one column was a combination of the others, the determinant is zero, and both the inverse and the unique solution to Ax = b disappear together. Rank is a count and therefore steps rather than slides, so "nearly dependent" is a separate and more practical concern — it is what collinear features, duplicated columns and the dummy variable trap all produce, and why low-rank structure is something worth exploiting on purpose.