Every matrix, however ugly, is a rotation then a stretch then another rotation. Step through the three and watch it happen.
Overview
What every matrix does
Take the unit circle and apply any 2×2 mat. The result is always an ellipse. Not at times — always, for each mat there's.
That is a strong claim, and the SVD is the reason it holds. each mat can be written as three ops in sequence:
A = U Sigma V'
Read right to left. This Is the order they apply:
V' rotates. A rotation does nothing to a circle. This Is why stage 2 in the visualisation looks the same to stage 1. But watch the two spokes, which do move. It's choosing which directions are about to be stretched.
Sigma stretches along the axes, by the singular values. This is where the circle turns an ellipse.
U rotates the ellipse into its final orientation.
Step through the four stages above and the claim stops being abstract. What you set the four mat entries to, the shape at stage 3 is an axis-aligned ellipse, each time.
Singular Value Decomposition
This module needs JavaScript: the
numbers are computed in the page rather than recorded.
Worth knowing
Any matrix factors as A = U Σ V': rotate, stretch along the axes, rotate again.
The singular values in Σ are the stretch factors, always non-negative and always in descending order.
Unlike eigenvectors, an SVD exists for every matrix — non-square ones included.
The condition number is the largest singular value divided by the smallest. Large means the matrix is near-singular.
Singular Value Decomposition
The factorisation that works on each mat. The one worth knowing if you only learn one.
The singular values
They're the lengths of the ellipse's semi-axes, always non-minus, and usually listed largest first.
The largest is how much the mat can stretch any vector — its gain in the worst case.
The smallest is how much it can squash one.
Their ratio is the rule number, and it's the single most handy number in the readout. A rule number near 1 means the mat treats all directions roughly alike. A large one means it nearly flattens some direction, and solving any of it involving that mat will amplify error along it. Drag the entries until the two singular values are far apart and watch the ellipse become a sliver.
A zero singular value means the mat truly flattens a direction: the ellipse collapses to a line segment, the mat is singular, and it has no inverse.
Why not eigenvectors
The eigendecomposition is the more famous factorisation, and the SVD is the more handy one, for three reasons.
It always exists. Eigendecomposition needs a square mat, and even then not each square mat has one. The SVD exists for each mat of each shape, including a 1000×3.
Its bases are unit-perp. Eigenvectors need not be at right angles, and when they're nearly parallel the breakdown is by number fragile. U and V are unit-perp by build.
Its values are real and non-minus. Eigenvals of a real mat can be complex. Singular values never are.
For a even plus-definite mat the two coincide. This Is why they are so often confused.
Low-rank approximation
The property that makes the SVD indispensable: keep only the largest *k* singular values, set the rest to zero, and times back. The result is provably the best rank-*k* guess of the first mat, in the least-squares sense. Not a good one — the best.
That single fact underwrites a surprising amount:
PCA is the SVD of the mean-centred data mat. The principal components are the right singular vectors and the shown variances are the squared singular values, which is why any PCA setup you look inside is calling an SVD.
Image compression keeps the top singular values of the pixel mat.
Latent semantic analysis factors a term-doc mat and reads the retained dims as topics.
Recommender systems factor a sparse user-item mat into a low-rank product.
Noise reduction works on the assumption that signal focuses in the large singular values and noise spreads through the small ones.
Singular values by hand, from A'A
You rarely find an SVD on paper. But Seeing where the singular values come from drops the mystery. The singular values of A are the square roots of the eigenvals of A'A. Take A = [[3, 0], [4, 5]]:
A even 2×2[[a, b], [b, a]] has eigenvals a + b and a − b, here 45 and 5. So the singular values are:
singular value
from
σ₁ = √45 ≈ 6.708
the largest stretch A applies
σ₂ = √5 ≈ 2.236
the smallest
Their ratio, √(45/5) = 3, is the rule number: A stretches its most sensitive direction three times more than its least. That is why the unit circle turns an ellipse with semi-axes 6.708 and 2.236, and why the singular values are always real and non-minus. They're square roots of eigenvals of A'A, which is even positive semi-definite for any A. Eigenvals of A itself could be complex or minus. The SVD's numbers never are, which is half the reason it's the factorisation to reach for.
This is the machinery under PCA. This Is nothing more than the SVD of the mean-centred data: the principal directions are the right singular vectors and the variance each shows is its singular value squared. Keeping only the largest few — setting the small singular values to zero and timesing back — gives the provably best low-rank guess of the data. This Is the same op behind image compression, latent-semantic analysis and recommender factorisations. The small singular values are where noise focuses, so dropping them often sharpens the signal rather than degrading it.
Where it goes wrong with Singular Value Decomposition
Losing to centre before PCA. Without subtracting the mean, the first component points at the mean rather than at the direction of greatest variance.
Reading a rule number as an error. It's a sensitivity: it says how much input error can be amplified, not how much has been.
given singular values are eigenvals. They're the eigenvals of A'A, square-rooted. Not the same object, and not equal except in special cases.
finding the full SVD when you want the top few. For a large sparse mat a truncated solver is orders of magnitude cheaper.
Questions people ask about Singular Value Decomposition
What does the SVD actually give me? It factors any matrix as U times S times V transpose: a rotation, a stretch along the axes, and another rotation. The singular values in S say how much stretch each direction gets.
How is SVD different from eigendecomposition? SVD works on any matrix, including non-square ones, and uses orthogonal factors. Eigendecomposition needs a square matrix and may not have orthogonal eigenvectors.
How do I get the best low-rank approximation? Keep the k largest singular values and zero the rest. The Eckart-Young theorem says no other rank-k matrix is closer in the Frobenius or spectral norm.
How does SVD relate to PCA? PCA on centred data is the SVD of that data matrix. The right singular vectors are the principal directions and squared singular values give the variances.
What does a tiny singular value tell me? The matrix is nearly rank-deficient in that direction. A large ratio of biggest to smallest signals an ill-conditioned problem.
Is SVD expensive? Full SVD costs about the cube of the smaller dimension. For large matrices use truncated or randomised SVD to get only the top k components.
Rebuild a matrix from its biggest pieces
SVD splits any mat into a stack of rank-one layers sorted by importance. Keeping only the first few is what compression and PCA really do.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
# a matrix that is really rank 2, with a little noise on top
A = np.outer(rng.normal(size=40), rng.normal(size=25)) * 5.0
A += np.outer(rng.normal(size=40), rng.normal(size=25)) * 2.0
A += rng.normal(size=(40, 25)) * 0.05
U, s, Vt = np.linalg.svd(A, full_matrices=False)
print("A is %d x %d" % A.shape)
print("first 8 singular values:", np.round(s[:8], 4))
print()
print("each is the 'size' of one rank-one layer. two are large, the rest are noise.")
print()
total = (s ** 2).sum()
print("%5s %14s %14s %12s" % ("k", "kept variance", "reconstr. err", "numbers stored"))
for k in (1, 2, 3, 5, 10, 25):
Ak = (U[:, :k] * s[:k]) @ Vt[:k]
kept = (s[:k] ** 2).sum() / total
err = np.linalg.norm(A - Ak) / np.linalg.norm(A)
stored = k * (A.shape[0] + A.shape[1] + 1)
print("%5d %13.4f%% %14.6f %12d" % (k, 100 * kept, err, stored))
print("%5s %13s %14s %12d" % ("full", "100%", "0", A.size))
print()
print("k=2 stores %d numbers instead of %d and loses almost nothing."
% (2 * (40 + 25 + 1), A.size))
print()
print("singular values are never negative, and U and V are orthonormal:")
print(" U.T @ U close to I:", np.allclose(U.T @ U, np.eye(U.shape[1])))
print(" V.T @ V close to I:", np.allclose(Vt @ Vt.T, np.eye(Vt.shape[0])))
Output
Check yourself
0 of 3
Answer without scrolling back up.
What shape does the unit circle become under any 2x2 matrix?
Always an ellipse - that is what the SVD asserts. Rotate, stretch along the axes, rotate again, and stage 3 is an axis-aligned ellipse for every matrix.
What does a large condition number tell you?
It is the largest singular value over the smallest. Near 1 means all directions are treated alike; large means the ellipse is a sliver.
Why is the SVD preferred over an eigendecomposition?
Eigendecomposition needs a square matrix and does not always exist even then; eigenvectors need not be perpendicular. Singular values are always real and non-negative.
Cheat sheet
Singular Value Decomposition
Factor any matrix into U, Sigma and V-transpose. Truncate to k components and watch the low-rank reconstruction improve as k increases.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.