Vector Norms

The three you will meet

L1 = |x₁| + |x₂| + …

Add up the absolute values. Called the Manhattan or taxicab norm, because it is the distance you walk on a street grid — you cannot cut the corner.

L2 = √(x₁² + x₂² + …)

Ordinary straight-line distance, from Pythagoras. This is the default: when someone writes ‖x‖ with no subscript, they mean this.

L∞ = max(|x₁|, |x₂|, …)

Just the largest component. Everything else is ignored entirely.

All three are the special cases p = 1, 2 and ∞ of one family, Lp = (Σ|xᵢ|ᵖ)^1/p, which is what the general p slider sweeps through.

The shapes are the point

Draw every vector whose norm equals exactly 1 and you get that norm's unit ball. This is where the three stop being interchangeable:

• L2 gives a circle. Perfectly round — no direction is special.
• L1 gives a diamond, with sharp corners sitting exactly on the axes.
• L∞ gives a square, with flat sides.

Those corners on the L1 diamond are not decoration. They are the entire reason Lasso produces exact zeros and Ridge does not.

Several ways to measure length

A norm measures the size of a vector, and the choice of norm changes what "size" means.

For x = [3, −4]:

• L1 = 3 + 4 = 7
• L2 = √(9 + 16) = 5
• L∞ = max(3, 4) = 4
• L0 = 2 non-zero entries

L2 is the everyday default — it is the straight-line length and the one that corresponds to ordinary geometry. L1 sums the magnitudes and is more forgiving of a single large component. L∞ only looks at the worst one.

The unit circles explain everything

Draw the set of vectors with norm exactly 1 under each measure and you get three different shapes:

• L2: a circle.
• L1: a diamond, with corners on the axes.
• L∞: a square.

Those corners are not a curiosity — they are the entire reason lasso produces sparse models.

Optimisation under a norm constraint means finding where the loss function's contours first touch the constraint shape. On a smooth circle, the touching point is almost never on an axis, so no coefficient becomes exactly zero: that is ridge, which shrinks everything without eliminating anything.

On a diamond, the corners stick out and lie exactly on the axes. Contours are far more likely to touch a corner — and touching a corner means one coefficient is exactly zero. That is lasso, and it is why L1 regularisation selects features while L2 does not.

Norms as regularisation

loss = prediction error + λ × (norm of the weights)

The same choice appears in deep learning under a different name: weight decay is L2 regularisation applied to the network's parameters, and it remains one of the few regularisers used in essentially every serious training recipe.

The scaling requirement follows directly. The penalty is applied to raw coefficient sizes, so a feature measured in thousands naturally has a tiny coefficient and is barely penalised, while one measured in units is penalised heavily. Standardise before regularising, always.

Norms elsewhere in training

Gradient clipping. Compute the L2 norm of the whole gradient, and if it exceeds a threshold, rescale it down. This is standard in recurrent networks and transformers, where occasional enormous gradients would otherwise destroy the weights in a single step.

Monitoring. Logging the gradient norm per layer is the fastest diagnosis of vanishing or exploding gradients, and it costs nothing.

Normalisation. Dividing a vector by its L2 norm gives a unit vector, which is what makes dot products equal cosine similarity — the standard preprocessing step in every embedding pipeline.

Loss functions. Mean squared error is a squared L2 norm of the residual vector; mean absolute error is an L1 norm. That is precisely why MSE is dominated by outliers and MAE is not.

Four norms on one vector

And then two vectors with identical L2 and different L1 — which is the whole reason L1 regularisation produces zeros and L2 does not.

import numpy as np

v = np.array([3.0, -4.0, 0.0, 12.0])

print("L1 (sum of absolutes) :", np.linalg.norm(v, 1))
print("L2 (Euclidean length) :", np.linalg.norm(v, 2))
print("L-inf (largest)       :", np.linalg.norm(v, np.inf))
print("L0 (count of non-zero):", int(np.count_nonzero(v)))
print()
print("two vectors of the same L2 length:")
for w in (np.array([1.0, 1.0]), np.array([1.414, 0.0])):
    print("  %-14s L1 %.3f  L2 %.3f" % (str(w), np.linalg.norm(w, 1),
                                        np.linalg.norm(w, 2)))
print()
print("equal L2, different L1 - L1 prefers the one with a zero in it.")

Guided tour

1. Confirm the ordering. Set Component X to 3 and Component Y to 4. L2 reads exactly 5 — the 3-4-5 triangle — while L1 reads 7 and L∞ reads 4. The rule L∞ ≤ L2 ≤ L1 holds for every vector you can drag to.
2. Find where they agree. Set Component Y to 0. Now all three read the same number, because a vector lying on an axis has nothing for them to disagree about.
3. Find where they disagree most. Set both components equal. L1 is now the largest it can be relative to L2 — the diagonal is exactly where the diamond is furthest inside the circle.
4. Sweep the family. Drag the General p slider from 1 to 8 and watch the Lp ball morph from diamond to circle to square. There is nothing special about 1 and 2 except that they are useful and easy to differentiate.
5. See sparsity happen. Tick Sparsity Demo. A line of equally good solutions appears, and each ball grows until it first touches that line. The L1 diamond almost always makes contact at a corner — where one coordinate is exactly zero — while the L2 circle touches at a smooth point with both coordinates non-zero.
6. Try to break it. With the demo on, drag the sliders to change the line's angle. The L1 solution stays pinned to an axis across most angles; only when the line is nearly parallel to a diamond edge does it slide off the corner.

Why the corners cause zeros

Regularised regression solves a constrained problem: find the best fit whose coefficient vector has norm below some budget. Geometrically, you inflate the norm's unit ball until it first touches the set of best-fitting solutions.

A circle is smooth, so it will touch that set at whatever point happens to be nearest — a generic point, with every coordinate non-zero. That is Ridge: everything shrinks, nothing disappears.

A diamond is mostly corner. Its corners stick out further along the axes than its edges do anywhere else, so contact is disproportionately likely to happen at a corner — and a corner is exactly a point where a coordinate equals zero. That is Lasso: coefficients are deleted, not merely shrunk.

The same fact shows up in the calculus. The derivative of x² is 2x, which fades to nothing as x approaches zero; the derivative of |x| is ±1 all the way down. Smooth pressure that weakens versus constant pressure that does not — geometry and calculus telling the same story.

Where each shows up

• L2 — Euclidean distance in KNN and k-means, weight decay, gradient clipping, and the length in almost every geometric argument.
• L1 — Lasso and feature selection, and mean absolute error, which is more robust to outliers than squared error for the same reason.
• L∞ — worst-case bounds and adversarial robustness, where "no single pixel changed by more than ε" is exactly an L∞ constraint.

Where this goes wrong

• Assuming ‖x‖ means L2. It usually does, but not always. In a paper about sparsity it may well not.
• Forgetting to scale features first. A norm adds up components, so a feature measured in thousands dominates the total. Regularising unscaled features penalises them by their units rather than their importance.
• Calling L0 a norm. Counting non-zero entries is not a norm — it fails the scaling rule, since doubling a vector does not double the count. It is also what Lasso approximates, because optimising it directly is intractable.
• Expecting L1 zeros to be stable. With correlated features the corner that gets touched can flip between similar datasets. The sparsity is real; which particular coefficient survives is not always meaningful.

What to remember

A norm answers "how big is this vector", and L1, L2 and L∞ give three different answers — the sum of absolute values, the straight-line length, and the largest component — always ordered L∞ ≤ L2 ≤ L1, agreeing only on the axes. What separates them in practice is the shape of their unit ball: L2's circle is smooth and touches a solution set at a generic point, while L1's diamond is mostly corner and its corners sit on the axes, where a coordinate is exactly zero. That single geometric fact is why Lasso deletes features and Ridge only shrinks them.

Matrix norms

Norms extend from vectors to matrices, and two appear regularly.

The Frobenius norm treats the matrix as one long vector and takes the L2 norm of all its entries. It is what weight decay penalises in a neural network layer, and what np.linalg.norm(W) returns by default.

The spectral norm is the largest singular value — the greatest factor by which the matrix can stretch any vector. It bounds how much a layer can amplify its input, which makes it the natural measure for stability. Spectral normalisation, used in GAN training, constrains exactly this quantity to keep the discriminator well-behaved.

import numpy as np

np.linalg.norm(v)               # L2 of a vector
np.linalg.norm(v, 1)            # L1
np.linalg.norm(v, np.inf)       # L-infinity
np.linalg.norm(W)               # Frobenius, for a matrix
np.linalg.norm(W, 2)            # spectral (largest singular value)

What makes something a norm

Three conditions: it is zero only for the zero vector, scaling the vector scales the norm by the same factor, and the triangle inequality holds — ‖a + b‖ ≤ ‖a‖ + ‖b‖, the formal statement that a detour is never shorter.

The "L0 norm" fails the second condition (doubling a vector does not double the count of non-zero entries), which is why it is not really a norm. It is also non-differentiable and combinatorially hard to optimise, and L1 is used as its practical, convex stand-in — the reason lasso exists in the form it does.

Questions people ask

Which norm should I use for regularisation? L2 by default. L1 when you want feature selection or a genuinely sparse model. Elastic net when features are correlated and you want both.

Why does L1 produce zeros and L2 does not? Because the L1 constraint region has corners on the axes, and optimisation is disproportionately likely to land on one.

Is weight decay the same as L2 regularisation? They coincide for plain SGD. For Adam they differ, which is why AdamW — which decouples the decay from the adaptive scaling — became the standard optimiser for transformers.

What does normalising a vector mean? Dividing by its norm so the length becomes 1, keeping only the direction.

Which norm for distance between embeddings? Cosine, or equivalently L2 on normalised vectors — the ranking is the same.

Does the norm choice affect KNN? Yes. L1 is often more stable in high dimensions, and switching metric can change which neighbours are selected.

Recap in one screen

• A norm measures a vector's size; L1 sums magnitudes, L2 is straight-line length, L∞ takes the largest component.
• The shape of the unit ball explains regularisation: a diamond's corners give lasso its sparsity, a circle's smoothness gives ridge its shrinkage.
• Standardise features before applying any norm-based penalty.
• Gradient clipping, embedding normalisation, MSE and MAE are all norms in disguise.
• Frobenius and spectral norms extend the idea to matrices, and bound how much a layer can amplify.