Principal Component Analysis

Quick Context

Real datasets have columns that repeat each other. Height in centimetres and height in inches carry one fact between them; so, more subtly, do a customer's number of orders and their total spend. PCA finds the directions the data actually varies along, which is rarely the directions your columns happen to be written in.

The result is a new set of axes, ordered by how much spread each one carries. Keep the first few and you have compressed the data with a known and measurable amount of loss — which is the part that separates PCA from simply deleting columns and hoping.

What it actually computes

Centre the data, then build its covariance matrix Σ. The variance of the data measured along any unit direction u is

var(u) = uᵀ Σ u

PCA asks which u makes that as large as possible. The answer is the top eigenvector of Σ, and the variance it achieves is its eigenvalue λ₁. The second component is the best remaining direction perpendicular to the first, and so on.

That is the whole algorithm. The explained variance ratio of a component is its eigenvalue divided by the sum of all of them — the share of the total spread that component accounts for.

Interactive Exploration Guide

1. Start off-axis. The projection line opens flat at 0° while the cloud is tilted at 30°, so the line is capturing far less than it could. Read Kept.Run this
2. Sweep the angle. Drag Projection Angle from 0° to 180°. Captured variance rises to exactly one peak and falls to exactly one trough. The peak is PC1, the trough is PC2, and they are 90° apart — always.Run this
3. Land on it. Press Align to PC1. Kept now equals the explained variance ratio, and no other angle beats it.Run this
4. Make the second component worthless. Set Spread Across It to 0. The cloud collapses onto a line, λ₂ goes to zero and PC1 explains 100% — two columns of data carrying one dimension of information.Run this
5. Now make it useless. Set both spreads equal instead. The cloud becomes a round blob, the two eigenvalues come out close together and PC1 explains only a little over half — with no preferred direction to find, reducing to 1D costs you nearly half of everything.
6. Watch the residuals. With Project Onto The Line on, the dashed stubs are the distances thrown away. Sweep the angle again: PCA is the line that makes those stubs shortest, which is the same thing as capturing the most variance.
7. Break it by hand. Drag a single point far away from the rest. One outlier can swing PC1 noticeably, because variance squares distances and squares are unforgiving.

Two ways to say the same thing

PC1 is usually introduced as "the direction of maximum variance". It is equally true that PC1 is the line minimising the squared perpendicular distance from every point — the dashed stubs on the plot. Both descriptions pick out the same line, because total variance is fixed: whatever the projection does not keep, the residuals hold.

Note the perpendicular distance. That is what makes PCA different from least-squares regression, which minimises vertical distance to a target column. PCA has no target column; it treats every feature symmetrically, which is why it is unsupervised.

How many components to keep

Sort the eigenvalues, take the running total of their explained variance ratios, and stop at a threshold you can defend — 95% is the common one. Plotting the ratios in order gives the scree plot, and the "elbow" where it flattens is the informal version of the same decision.

On real high-dimensional data the pay-off can be dramatic: images, spectra and survey responses often carry most of their variance in a small fraction of their components, because the raw measurements are heavily correlated to begin with.

What usually goes wrong

• Not standardising first. PCA maximises variance, and variance carries units. Leave income in rupees alongside age in years and the first component will be income, whatever the data means. Standardise unless every column is already on the same scale.
• Fitting on the whole dataset before splitting. Fit PCA on the training split and apply that same transform to test data. Fitting on everything leaks the test set's structure into your features.
• Expecting the components to mean something. A component is a weighted blend of your original columns. It sometimes has a readable interpretation and often does not, and reaching for one is where a lot of nonsense gets written.
• Using it to pick features for a classifier. PCA keeps the directions with the most variance, not the ones that separate your classes. A low-variance direction can be exactly the one that matters, and PCA will drop it without hesitation.
• Forgetting outliers move it. A single extreme point can dominate the covariance matrix. Look at the data first.

Key Takeaway

PCA rotates the axes to the directions the data actually varies along: PC1 is the direction of greatest variance, each later component is the best remaining direction perpendicular to the ones before it, and they are the eigenvectors of the covariance matrix with the eigenvalues as the variance each one carries. The explained variance ratio tells you exactly what a reduction costs, which turns dropping dimensions from a guess into a measured trade. Standardise your columns first, fit on the training split only, and remember that PCA maximises variance rather than usefulness — it has never seen your labels.