Vector Norms

Quick Context

"How big is this vector?" sounds like it has one answer. It has several, and machine learning uses at least three of them routinely — often in the same model.

A norm is any function that takes a vector and returns a non-negative size, is zero only for the zero vector, scales properly when you multiply the vector by a constant, and obeys the triangle inequality. Several functions qualify, and they disagree.

The three you will meet

L1 = |x₁| + |x₂| + …

Add up the absolute values. Called the Manhattan or taxicab norm, because it is the distance you walk on a street grid — you cannot cut the corner.

L2 = √(x₁² + x₂² + …)

Ordinary straight-line distance, from Pythagoras. This is the default: when someone writes ‖x‖ with no subscript, they mean this.

L∞ = max(|x₁|, |x₂|, …)

Just the largest component. Everything else is ignored entirely.

All three are the special cases p = 1, 2 and ∞ of one family, Lp = (Σ|xᵢ|ᵖ)^1/p, which is what the general p slider sweeps through.

The shapes are the point

Draw every vector whose norm equals exactly 1 and you get that norm's unit ball. This is where the three stop being interchangeable:

• L2 gives a circle. Perfectly round — no direction is special.
• L1 gives a diamond, with sharp corners sitting exactly on the axes.
• L∞ gives a square, with flat sides.

Those corners on the L1 diamond are not decoration. They are the entire reason Lasso produces exact zeros and Ridge does not.

Interactive Exploration Guide

1. Confirm the ordering. Set Component X to 3 and Component Y to 4. L2 reads exactly 5 — the 3-4-5 triangle — while L1 reads 7 and L∞ reads 4. The rule L∞ ≤ L2 ≤ L1 holds for every vector you can drag to.Run this
2. Find where they agree. Set Component Y to 0. Now all three read the same number, because a vector lying on an axis has nothing for them to disagree about.Run this
3. Find where they disagree most. Set both components equal. L1 is now the largest it can be relative to L2 — the diagonal is exactly where the diamond is furthest inside the circle.
4. Sweep the family. Drag the General p slider from 1 to 8 and watch the Lp ball morph from diamond to circle to square. There is nothing special about 1 and 2 except that they are useful and easy to differentiate.Run this
5. See sparsity happen. Tick Sparsity Demo. A line of equally good solutions appears, and each ball grows until it first touches that line. The L1 diamond almost always makes contact at a corner — where one coordinate is exactly zero — while the L2 circle touches at a smooth point with both coordinates non-zero.
6. Try to break it. With the demo on, drag the sliders to change the line's angle. The L1 solution stays pinned to an axis across most angles; only when the line is nearly parallel to a diamond edge does it slide off the corner.

Why the corners cause zeros

Regularised regression solves a constrained problem: find the best fit whose coefficient vector has norm below some budget. Geometrically, you inflate the norm's unit ball until it first touches the set of best-fitting solutions.

A circle is smooth, so it will touch that set at whatever point happens to be nearest — a generic point, with every coordinate non-zero. That is Ridge: everything shrinks, nothing disappears.

A diamond is mostly corner. Its corners stick out further along the axes than its edges do anywhere else, so contact is disproportionately likely to happen at a corner — and a corner is exactly a point where a coordinate equals zero. That is Lasso: coefficients are deleted, not merely shrunk.

The same fact shows up in the calculus. The derivative of x² is 2x, which fades to nothing as x approaches zero; the derivative of |x| is ±1 all the way down. Smooth pressure that weakens versus constant pressure that does not — geometry and calculus telling the same story.

Where each shows up

• L2 — Euclidean distance in KNN and k-means, weight decay, gradient clipping, and the length in almost every geometric argument.
• L1 — Lasso and feature selection, and mean absolute error, which is more robust to outliers than squared error for the same reason.
• L∞ — worst-case bounds and adversarial robustness, where "no single pixel changed by more than ε" is exactly an L∞ constraint.

What usually goes wrong

• Assuming ‖x‖ means L2. It usually does, but not always. In a paper about sparsity it may well not.
• Forgetting to scale features first. A norm adds up components, so a feature measured in thousands dominates the total. Regularising unscaled features penalises them by their units rather than their importance.
• Calling L0 a norm. Counting non-zero entries is not a norm — it fails the scaling rule, since doubling a vector does not double the count. It is also what Lasso approximates, because optimising it directly is intractable.
• Expecting L1 zeros to be stable. With correlated features the corner that gets touched can flip between similar datasets. The sparsity is real; which particular coefficient survives is not always meaningful.

Key Takeaway

A norm answers "how big is this vector", and L1, L2 and L∞ give three different answers — the sum of absolute values, the straight-line length, and the largest component — always ordered L∞ ≤ L2 ≤ L1, agreeing only on the axes. What separates them in practice is the shape of their unit ball: L2's circle is smooth and touches a solution set at a generic point, while L1's diamond is mostly corner and its corners sit on the axes, where a coordinate is exactly zero. That single geometric fact is why Lasso deletes features and Ridge only shrinks them.