Eigenvalues and Eigenvectors

The definition, and what it is really saying

A v = λ v

Read it as an equation of behaviours rather than symbols. The left-hand side is "apply the whole transformation to v". The right-hand side is "just scale v by a number". A vector satisfying this is one for which the entire matrix — all four entries, all that rotating and shearing — collapses into a single multiplication.

That is why they matter. Along an eigenvector, a complicated transformation becomes arithmetic.

Finding them

Rearranging Av = λv gives (A − λI)v = 0, which has a non-zero solution only when the matrix A − λI is singular — that is, when its determinant is zero:

det(A − λI) = 0

For a 2×2 matrix that is a quadratic in λ, so there are two eigenvalues (possibly repeated, possibly complex). For n×n it is a degree-n polynomial, and solving it by hand stops being sensible almost immediately.

import numpy as np

A = np.array([[3, 1], [0, 2]])
values, vectors = np.linalg.eig(A)

values          # [3., 2.]
vectors[:, 0]   # the eigenvector for lambda = 3

Two facts worth carrying: the eigenvalues sum to the trace (the diagonal total) and multiply to the determinant. Both are quick sanity checks on a computed result.

The directions a transformation does not turn

Multiply a matrix by a vector and the vector usually moves and rotates. For a few special directions, it does not rotate at all — it only stretches or shrinks.

A v = λ v

v is an eigenvector: a direction the transformation leaves pointing the same way. λ is its eigenvalue: how much it is stretched along that direction.

A worked example. For A = [[3, 1], [0, 2]], try v = [1, 0]:

A [1, 0]ᵀ = [3, 0]ᵀ = 3 × [1, 0]ᵀ

Same direction, three times as long. So [1, 0] is an eigenvector with eigenvalue 3.

Reading the eigenvalues tells you what the transformation does:

Why this is the heart of PCA

PCA is exactly the eigendecomposition of the covariance matrix.

The covariance matrix describes how the data spreads. Its eigenvectors are the directions of that spread, and its eigenvalues are how much variance lies along each one. Sort by eigenvalue, keep the largest few, and you have compressed the data while preserving as much variation as possible.

That is the whole algorithm. The explained-variance percentages that PCA reports are the eigenvalues divided by their total.

The same decomposition powers other tools:

• Spectral clustering uses the eigenvectors of a graph's Laplacian to find groups that k-means cannot.
• PageRank is the principal eigenvector of the web's link matrix.
• Vibration and stability analysis reads resonant modes as eigenvectors and their frequencies as eigenvalues.
• Recurrent network stability depends on the largest eigenvalue of the recurrent weight matrix: above 1 the state explodes, below 1 it vanishes — the direct cause of exploding and vanishing gradients through time.

Symmetric matrices, and why they are pleasant

Covariance matrices, correlation matrices and Gram matrices are all symmetric, and symmetric matrices have unusually good properties:

• All eigenvalues are real, with no complex numbers to worry about.
• Eigenvectors for distinct eigenvalues are orthogonal — at right angles.
• They can be written as A = QΛQᵀ with Q orthogonal, so the inverse of Q is just its transpose.

That orthogonality is why principal components are uncorrelated with each other, which is the property that makes them useful as replacement features.

Use np.linalg.eigh rather than eig for symmetric matrices: it is faster, more accurate, and returns the eigenvalues in sorted order.

See a vector survive the transform

For most vectors a matrix changes both direction and length. For an eigenvector it only changes the length, and the eigenvalue is by how much.

import numpy as np

A = np.array([[2.0, 1.0], [1.0, 2.0]])
values, vectors = np.linalg.eig(A)

print("eigenvalues :", values.round(4))
print("eigenvectors (columns):"); print(vectors.round(4))
print()
for i in range(2):
    v = vectors[:, i]
    print("A @ v%d = %s" % (i, (A @ v).round(4)))
    print("  %.2f * v%d = %s   <- the same vector, only scaled"
          % (values[i], i, (values[i] * v).round(4)))
print()
print("an eigenvector is a direction the matrix does not rotate.")

Guided experiments

1. Hunt for them by hand. Drag the Probe Angle slider slowly and watch the "turned by" readout. For most angles the orange image points somewhere different from the white probe; at two angles it drops to zero and the two arrows lie on top of each other. Those are the eigenvectors, found the hard way.
2. Let it find them. Click Snap to an Eigenvector and the probe jumps to a direction where the turn is exactly 0°. Click again to jump to the other one.
3. See the whole picture. Click Sweep All Directions and watch the probe rotate through a full circle while the image swings around at a different rate, crossing it exactly twice.
4. Check the identities. Drag any entry slider and watch the cross-checks panel. trace always equals λ₁+λ₂ and det always equals λ₁·λ₂, for every matrix you can reach.
5. Make them perpendicular. Set the Preset to Symmetric. The two eigenvectors are now at exactly 90° to each other. Any matrix with b = c has this property, and it is the reason PCA produces axes at right angles.
6. Make every direction an eigenvector. Set the Preset to Uniform scale. Now nothing rotates at all, whatever the probe angle, because a uniform scale stretches every direction by the same factor.
7. Take them away entirely. Set the Preset to Rotation. Real eigenvalues vanish and Real? reads no. A rotation turns every direction, so there is nothing left for it to leave alone.

Why symmetric matrices are special

When b = c the matrix is symmetric, and three things become guaranteed rather than lucky: the eigenvalues are always real, the eigenvectors are always perpendicular, and the matrix can always be diagonalised.

This is not a curiosity. Covariance matrices are symmetric by construction, because the covariance of x with y is the same number as the covariance of y with x. PCA is precisely the eigendecomposition of a covariance matrix, and the guarantee above is what makes its principal components perpendicular and its variances real, positive numbers. The whole method rests on this property.

Where else they turn up

• PCA — eigenvectors of the covariance matrix are the directions of greatest variance; the eigenvalues are how much variance each one carries.
• PageRank — the ranking is the dominant eigenvector of the link matrix.
• Stability — repeatedly applying a matrix makes vectors blow up if the largest |λ| exceeds 1 and shrink to nothing if it is below. This is the exploding-and-vanishing-gradient story in a different vocabulary.
• Conditioning — the ratio of the largest to smallest eigenvalue tells you how stretched the loss surface is, which is what makes plain gradient descent zig-zag.

Traps worth knowing

• Expecting every matrix to have real eigenvectors. Rotations have none. A negative discriminant is not an error.
• Forgetting eigenvectors have no fixed length. If v is an eigenvector so is 2v and −v. Only the direction is determined, which is why implementations return unit vectors and the sign can differ between libraries.
• Assuming perpendicularity in general. That is guaranteed only for symmetric matrices. The Generic preset shows two eigenvectors that are not at right angles.
• Running PCA on unscaled features. The covariance matrix is dominated by whichever feature has the largest units, so the first component just points at it. Standardise first.

In one line

An eigenvector is a direction a transformation does not rotate, and its eigenvalue is the factor it is stretched by, so along an eigenvector the whole matrix collapses into a single multiplication. For a 2×2 matrix they follow from the trace and determinant alone, and always satisfy λ₁+λ₂ = trace and λ₁λ₂ = det; a negative discriminant means no real eigenvectors exist, which is exactly the case for a rotation. Symmetric matrices are guaranteed real eigenvalues and perpendicular eigenvectors, and since covariance matrices are symmetric, that guarantee is what PCA is built on.

Eigen, singular values, and which to use

Eigendecomposition needs a square matrix. Real data is rarely square — it is n samples by p features.

Singular value decomposition generalises the idea to any matrix:

A = U Σ Vᵀ

The singular values in Σ play the role of eigenvalues, and V's columns are directions in feature space. For a covariance matrix the two coincide: the singular values are the square roots of the eigenvalues.

In practice, PCA implementations use SVD on the centred data matrix rather than forming the covariance matrix and decomposing it, because it is numerically more stable and avoids squaring the condition number. Scikit-learn's PCA does exactly this.

SVD also does jobs eigendecomposition cannot: low-rank approximation for compression, matrix completion for recommender systems, and pseudo-inverses for least squares problems with no unique solution.

Practical notes

• Scale before decomposing. Eigenvalues of a covariance matrix depend on units, so an unscaled column with a large range dominates the first component.
• Signs are arbitrary. An eigenvector and its negation describe the same direction; do not read meaning into which one a library returns.
• Order is not guaranteed by np.linalg.eig. Sort explicitly, or use eigh for symmetric matrices.
• Repeated eigenvalues mean the corresponding eigenvectors are not unique — any rotation within that subspace works, so individual component directions become meaningless.
• Very small eigenvalues indicate near-collinear features and an ill-conditioned matrix, where solving linear systems becomes numerically unreliable.

Questions people ask

Can a matrix have no eigenvectors? Over the real numbers, yes — a pure rotation turns every direction, so it has no real eigenvectors. Over the complex numbers, every square matrix has them.

What does a zero eigenvalue mean? The transformation collapses that direction entirely. The matrix is singular and has no inverse.

How many eigenvalues does an n×n matrix have? Exactly n, counting repeats and complex values.

Why are eigenvectors usually normalised? Because any scalar multiple of an eigenvector is also an eigenvector; fixing the length to 1 makes the answer unique up to sign.

Is this only for PCA? No — spectral clustering, PageRank, graph analysis, differential equations, quantum mechanics and network stability all use the same decomposition.

How does this relate to the determinant? The determinant is the product of the eigenvalues, which is why a zero eigenvalue means a zero determinant and a singular matrix.

Recap in one screen

• An eigenvector is a direction a matrix does not rotate; its eigenvalue is the factor it is stretched by.
• Solve det(A - λI) = 0 for the eigenvalues, then find each eigenvector — or call a library.
• The eigenvalues sum to the trace and multiply to the determinant.
• PCA is the eigendecomposition of the covariance matrix: eigenvectors are components, eigenvalues are explained variance.
• Symmetric matrices have real eigenvalues and orthogonal eigenvectors, which is why components are uncorrelated.
• SVD generalises the idea to non-square matrices and is what implementations actually use.