Principal Component Analysis

What PCA computes

PCA first centres each feature by subtracting its mean. It then finds directions that maximise the variance of the projected observations. If the centred data has covariance matrix Σ, the variance along a unit direction u is:

variance along u = uᵀΣu, subject to uᵀu = 1

The direction with the largest value is the first principal component, or PC1. It is an eigenvector of Σ; its eigenvalue is the variance captured along that direction. PC2 captures the most remaining variance while staying perpendicular to PC1. Later components continue the same way.

For example, suppose two centred features have covariance matrix [[4, 3], [3, 4]]. The directions (1, 1) and (1, −1) have variances 7 and 1. PC1 therefore explains 7/(7 + 1) = 87.5% of the total variance. Keeping only PC1 reduces two values per row to one, while discarding the remaining 12.5% of variance.

Try the projection visualiser

1. Set Projection Angle to 0°. The line misses the cloud’s main direction, so the readout shows less captured variance.
2. Move it toward 30°. The captured variance reaches a maximum when the line aligns with PC1. The perpendicular direction is PC2; the two axes are orthogonal.

The visualiser shows a two-dimensional case. In higher dimensions, the same rule applies: sort components by eigenvalue, then choose how many to keep. A cumulative explained-variance plot can guide compression, but a threshold such as 95% is a heuristic, not a universal requirement.

Scaling and data leakage

PCA is sensitive to feature units because it maximises variance. If one column is measured in thousands and another in fractions, the large-scale column may dominate. Standardising features is often appropriate when units differ and you want them to contribute on comparable scales. It is not automatic: when raw variance has meaning or features already share a meaningful scale, standardising can remove useful information.

Fit both the scaler and PCA on training data only, then apply those fitted transformations to validation and test data. Fitting them on the full dataset lets held-out rows influence the components, which leaks information. A scikit-learn pipeline keeps those steps together; see feature scaling and the official PCA documentation.

When PCA helps, and what it cannot tell you

PCA can reduce storage, speed up some downstream models, and provide a compact view of high-dimensional numeric data. It is useful when features are correlated and retaining most of their total variance is a sensible objective.

Variance is not the same as predictive value. A low-variance direction may separate classes, so PCA can discard a feature that matters to the target. The components are weighted combinations of the original features; inspect their loadings before assigning them a meaning. PCA is also linear and sensitive to outliers, which can strongly affect covariance. It does not remove outliers or discover arbitrary nonlinear structure. Compare its goal with t-SNE and UMAP when the task is visualising nonlinear neighbourhoods.

Sources and next steps

See the scikit-learn PCA guide for implementation details. For the variance and eigenvector ideas behind the method, review covariance and correlation and eigenvalues and eigenvectors.