Principal Component Analysis

What it actually computes

Centre the data, then build its covariance matrix Σ. The variance of the data measured along any unit direction u is

var(u) = uᵀ Σ u

PCA asks which u makes that as large as possible. The answer is the top eigenvector of Σ, and the variance it achieves is its eigenvalue λ₁. The second component is the best remaining direction perpendicular to the first, and so on.

That is the whole algorithm. The explained variance ratio of a component is its eigenvalue divided by the sum of all of them — the share of the total spread that component accounts for.

Squashing a shadow without losing the shape

Imagine holding a teapot in front of a lamp and looking at its shadow on the wall. The teapot is three-dimensional; the shadow is two-dimensional. Rotate the teapot and the shadow changes: from one angle you see the spout and handle clearly, from another you see a featureless blob.

PCA is the search for the angle that produces the most informative shadow. Formally, it finds the directions along which the data varies most, and projects onto them. Directions where the data barely varies are the ones you can drop without losing much.

"Varies most" is the key phrase, and it comes with a built-in assumption: that variance equals information. That is usually a reasonable assumption and occasionally a bad one — a low-variance feature can still be the one that predicts your target. PCA is unsupervised; it never looks at the target, so it cannot know.

What the components actually are

Each principal component is a weighted combination of your original features. If you have height, weight and shoe size, the first component might be:

PC1 = 0.58×height + 0.60×weight + 0.55×shoe size

All three weights are positive and similar, so PC1 is essentially "overall body size" — a single number that captures most of the variation across all three columns. A second component might be 0.7×height − 0.7×weight, which reads as "tall and light versus short and heavy", a build axis.

Three properties are guaranteed by construction:

• Ordered. PC1 captures the most variance, PC2 the next most, and so on. This is why keeping the first few is a sensible way to compress.
• Uncorrelated. Every component is at right angles to the others, so they carry non-overlapping information. This alone solves multicollinearity.
• Complete. With all components kept, no information is lost — you have rotated the data, not reduced it. Reduction only happens when you discard the tail.

Reading the explained variance

The output you make decisions from is explained_variance_ratio_ — the share of total variance each component accounts for.

Three ways to choose how many to keep, in descending order of usefulness:

1. Set a target. "Keep enough components for 95% of the variance." In scikit-learn that is literally PCA(n_components=0.95).
2. Look for the elbow in a scree plot — the point where the bars stop dropping sharply. Here that is around PC3.
3. Keep 2 or 3 if the purpose is a plot a human will look at.

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

pipe = make_pipeline(StandardScaler(), PCA(n_components=0.95))
X_small = pipe.fit_transform(X)

pca = pipe[-1]
print(pca.n_components_)                        # how many it kept
print(pca.explained_variance_ratio_.cumsum())   # the table above

Scaling first is mandatory, not advisory

PCA maximises variance, and variance is measured in the units of the data. A salary column measured in pounds has a variance in the billions; an age column has a variance in the hundreds. Without scaling, PC1 will be "salary" — not because salary matters most, but because pounds are small units.

Standardise every column to mean 0 and standard deviation 1 first, and each feature gets an equal chance to contribute. This turns PCA on the covariance matrix into PCA on the correlation matrix, which is what almost everyone means when they say PCA.

The only time to skip it is when all your features are genuinely in the same units and their relative magnitudes are meaningful — pixel intensities, or repeated measurements of the same quantity.

One more ordering rule: fit the scaler and the PCA on training data only, then apply both to test data. Fitting PCA on the whole dataset lets the test rows influence the directions, which is a subtle and very common leak.

Where the variance actually went

PCA rotates the data so the first axis carries the most spread. Here is the rotation, the variance it captured, and the reconstruction you get by throwing the rest away.

import numpy as np
from sklearn.decomposition import PCA
from sklearn.datasets import make_classification
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

rng = np.random.default_rng(0)
# three real directions, dressed up as twelve columns
latent = rng.normal(size=(800, 3)) * [4.0, 2.0, 0.5]
mix = rng.normal(size=(3, 12))
X = latent @ mix + rng.normal(0, 0.15, (800, 12))

pca = PCA().fit(X)
print("twelve columns, but only three real directions underneath.")
print()
print("%6s %14s %14s" % ("comp", "variance %", "cumulative %"))
cum = 0
for i, r in enumerate(pca.explained_variance_ratio_[:8]):
    cum += r
    print("%6d %13.4f%% %13.4f%%" % (i + 1, 100 * r, 100 * cum))
print("  components 4 onward carry %.4f%% between them."
      % (100 * pca.explained_variance_ratio_[3:].sum()))
print()

print("reconstruct from the first k components and measure what was lost:")
for k in (1, 2, 3, 4, 8, 12):
    p = PCA(n_components=k).fit(X)
    back = p.inverse_transform(p.transform(X))
    err = np.linalg.norm(X - back) / np.linalg.norm(X)
    print("  k=%2d  kept %7.4f%%  reconstruction error %.6f"
          % (k, 100 * p.explained_variance_ratio_.sum(), err))
print()

print("the components are orthogonal, by construction:")
C = pca.components_[:3]
print(np.round(C @ C.T, 10) + 0.0)
print()
print("and the transformed columns are uncorrelated:")
Z = PCA(n_components=3).fit_transform(X)
print(np.round(np.corrcoef(Z.T), 10) + 0.0)
print()

print("PCA is not feature selection -- every component uses every column:")
print("  component 1 loadings:", np.round(pca.components_[0][:6], 4), "...")
print("  you cannot point at a column and say it was dropped.")
print()

print("and it must be scaled first, or it just finds the biggest units:")
Xu = X.copy(); Xu[:, 0] *= 1000
print("  unscaled, component 1 variance share: %.4f"
      % PCA().fit(Xu).explained_variance_ratio_[0])
print("  scaled,   component 1 variance share: %.4f"
      % PCA().fit(StandardScaler().fit_transform(Xu)).explained_variance_ratio_[0])
print()

Xc, yc = make_classification(n_samples=1500, n_features=40, n_informative=6,
                             flip_y=0.10, random_state=0)
print("as a preprocessing step, on 1500 rows and 40 columns:")
for k in (2, 5, 10, 40):
    s = cross_val_score(make_pipeline(StandardScaler(), PCA(n_components=k),
                                      LogisticRegression(max_iter=2000)),
                        Xc, yc, cv=5).mean()
    print("  %2d components -> cv %.4f" % (k, s))
print("  no PCA at all  -> cv %.4f"
      % cross_val_score(make_pipeline(StandardScaler(),
                                      LogisticRegression(max_iter=2000)),
                        Xc, yc, cv=5).mean())
print()
print("PCA never helped here, and it is worth being clear about why. it keeps")
print("the directions with the most VARIANCE. nothing guarantees those are the")
print("directions that predict your label -- it never looked at the label.")
print()

Xw, yw = make_classification(n_samples=120, n_features=400, n_informative=8,
                             flip_y=0.05, random_state=0)
print("now the case it is built for -- 120 rows and 400 columns:")
print("  no PCA         -> cv %.4f"
      % cross_val_score(make_pipeline(StandardScaler(),
                                      LogisticRegression(max_iter=5000)),
                        Xw, yw, cv=5).mean())
for k in (5, 20, 60):
    print("  %2d components -> cv %.4f"
          % (k, cross_val_score(make_pipeline(StandardScaler(),
                                              PCA(n_components=k),
                                              LogisticRegression(max_iter=5000)),
                                Xw, yw, cv=5).mean()))
print()
print("with more columns than rows there is enough room to fit noise exactly,")
print("and cutting the space down is the thing that stops it. that is when")
print("PCA earns its place: as compression that happens to regularise.")

Guided tour

1. Start off-axis. The projection line opens flat at 0° while the cloud is tilted at 30°, so the line is capturing far less than it could. Read Kept.
2. Sweep the angle. Drag Projection Angle from 0° to 180°. Captured variance rises to exactly one peak and falls to exactly one trough. The peak is PC1, the trough is PC2, and they are 90° apart — always.
3. Land on it. Press Align to PC1. Kept now equals the explained variance ratio, and no other angle beats it.
4. Make the second component worthless. Set Spread Across It to 0. The cloud collapses onto a line, λ₂ goes to zero and PC1 explains 100% — two columns of data carrying one dimension of information.
5. Now make it useless. Set both spreads equal instead. The cloud becomes a round blob, the two eigenvalues come out close together and PC1 explains only a little over half — with no preferred direction to find, reducing to 1D costs you nearly half of everything.
6. Watch the residuals. With Project Onto The Line on, the dashed stubs are the distances thrown away. Sweep the angle again: PCA is the line that makes those stubs shortest, which is the same thing as capturing the most variance.
7. Break it by hand. Drag a single point far away from the rest. One outlier can swing PC1 noticeably, because variance squares distances and squares are unforgiving.

Two ways to say the same thing

PC1 is usually introduced as "the direction of maximum variance". It is equally true that PC1 is the line minimising the squared perpendicular distance from every point — the dashed stubs on the plot. Both descriptions pick out the same line, because total variance is fixed: whatever the projection does not keep, the residuals hold.

Note the perpendicular distance. That is what makes PCA different from least-squares regression, which minimises vertical distance to a target column. PCA has no target column; it treats every feature symmetrically, which is why it is unsupervised.

How many components to keep

Sort the eigenvalues, take the running total of their explained variance ratios, and stop at a threshold you can defend — 95% is the common one. Plotting the ratios in order gives the scree plot, and the "elbow" where it flattens is the informal version of the same decision.

On real high-dimensional data the pay-off can be dramatic: images, spectra and survey responses often carry most of their variance in a small fraction of their components, because the raw measurements are heavily correlated to begin with.

Failure modes

• Not standardising first. PCA maximises variance, and variance carries units. Leave income in rupees alongside age in years and the first component will be income, whatever the data means. Standardise unless every column is already on the same scale.
• Fitting on the whole dataset before splitting. Fit PCA on the training split and apply that same transform to test data. Fitting on everything leaks the test set's structure into your features.
• Expecting the components to mean something. A component is a weighted blend of your original columns. It sometimes has a readable interpretation and often does not, and reaching for one is where a lot of nonsense gets written.
• Using it to pick features for a classifier. PCA keeps the directions with the most variance, not the ones that separate your classes. A low-variance direction can be exactly the one that matters, and PCA will drop it without hesitation.
• Forgetting outliers move it. A single extreme point can dominate the covariance matrix. Look at the data first.

Summing up

PCA rotates the axes to the directions the data actually varies along: PC1 is the direction of greatest variance, each later component is the best remaining direction perpendicular to the ones before it, and they are the eigenvectors of the covariance matrix with the eigenvalues as the variance each one carries. The explained variance ratio tells you exactly what a reduction costs, which turns dropping dimensions from a guess into a measured trade. Standardise your columns first, fit on the training split only, and remember that PCA maximises variance rather than usefulness — it has never seen your labels.

What it is used for

• Visualising high-dimensional data. Project 200 columns down to 2 and plot them. Clusters, outliers and groupings become visible in a way no table ever makes them.
• Speeding up training. Fewer columns means faster fitting, sometimes dramatically. A classic pipeline is PCA to 50 components followed by an SVM or KNN, both of which suffer badly in high dimensions.
• Removing multicollinearity. Components are uncorrelated by construction, so a linear model on components has no collinearity problem at all.
• Noise reduction. Small-variance components are frequently measurement noise. Dropping them and reconstructing the data often produces a cleaner version — this is how PCA-based image denoising works.
• Compression. Storing 50 components instead of 500 features is a tenfold saving, with a controlled and measurable loss.
• Anomaly detection. Reconstruct each row from the kept components and measure the error. Rows that reconstruct badly do not fit the dominant structure of the data.

The costs you accept

Interpretability is the big one. Your model no longer uses "income" and "age"; it uses "0.4×income + 0.3×age − 0.2×tenure". You can inspect the loadings to interpret a component, but you can no longer report a coefficient per original feature, which matters in regulated settings.

PCA is also linear. It rotates and projects; it cannot unroll a curved manifold. Data shaped like a spiral or a Swiss roll is not helped by PCA, and t-SNE, UMAP or a kernel PCA is the right tool. And because it is unsupervised, it can discard exactly the low-variance direction that separated your classes — if the goal is classification, compare against linear discriminant analysis, which does look at the labels.

Finally, outliers pull the components towards themselves, because they contribute enormous variance. Clean or clip them before you fit.

Questions people ask

How many components should I keep? Enough for 90–95% of the variance for a modelling pipeline; two or three for a plot. Where accuracy is what matters, treat it as a hyperparameter and cross-validate it.

Can I interpret a component? Sometimes. Look at the largest loadings and see whether they tell a coherent story — "all the size measurements load positively" is interpretable, an arbitrary mix of fifteen features is not.

Does PCA remove outliers? No, and outliers distort it. Handle them first.

Is PCA the same as factor analysis? No. PCA finds directions of maximum variance with no underlying model; factor analysis posits latent factors generating the observed variables and separates shared variance from noise.

Can I use it on categorical data? Not sensibly — variance of one-hot columns is not what PCA assumes. Use multiple correspondence analysis, or an embedding.

Why do the signs of the components flip between runs? Because a component and its negation describe the same axis. The sign is arbitrary and carries no meaning; do not read anything into it.

Recap in one screen

• PCA rotates the data onto new axes ordered by how much variance they capture.
• Keep the first few axes to compress; drop the tail to remove noise.
• Components are uncorrelated combinations of your original features, not features themselves.
• Standardise first, and fit on training data only.
• Linear and unsupervised: it cannot unroll curved structure and does not know your target.