Ridge and Lasso Regression

Quick Context

Give a degree-12 polynomial eighteen noisy points and it will pass through nearly all of them. The fit looks superb on the data it has seen, and it is worthless: between the points the curve swings violently, because the only way to hit every point is to use enormous coefficients that cancel each other out.

Regularisation attacks this directly. Instead of asking the model only to fit the data, you ask it to fit the data and keep its coefficients small, and you decide how much the second part matters.

One extra term

Ordinary least squares minimises the squared error alone. Ridge and Lasso add a penalty on the size of the weights:

Ridge:   minimise   SSE + λ · Σ w_i²

Lasso:   minimise   SSE + λ · Σ |w_i|

λ (lambda) is the dial. At λ = 0 both reduce exactly to OLS. As λ grows the penalty dominates and every coefficient is pushed toward zero, until at very large λ the model gives up and predicts a flat line through the mean.

The only difference between the two is whether you square the weights or take their absolute value. That one change has a consequence out of all proportion to its size.

Why Lasso zeroes and Ridge does not

Look at how each penalty behaves as a coefficient approaches zero. The derivative of w² is 2w, which itself goes to zero — so the closer a Ridge coefficient gets to zero, the weaker the force pushing it further. It approaches zero and never arrives.

The derivative of |w| is a constant ±1, no matter how small w is. Lasso pushes with the same strength all the way down, so a coefficient whose contribution to the fit is worth less than that constant push gets driven to exactly zero and stays there.

That is why Lasso performs feature selection and Ridge does not. Watch the coefficient bars above: under Ridge they all shrink together and stay non-zero; under Lasso they vanish one by one.

Interactive Exploration Guide

1. See the problem first. Set the Penalty Type to None (plain OLS) and set the Polynomial Degree slider to 12. The curve whips between the training points, train MSE is almost zero, and test MSE is enormous. That gap is overfitting, made visible.Run this
2. Apply a little Ridge. Switch Penalty Type to Ridge (L2) and set the Lambda slider to -3. The curve calms down immediately and test MSE drops, while train MSE rises slightly. That trade is the entire point of regularisation.Run this
3. Overdo it. Set the Lambda slider to its maximum, 2. The model is now so heavily penalised that it flattens toward a straight line and underfits — both errors climb together. Regularisation is a dial, not a switch.Run this
4. Watch Lasso delete. Switch Penalty Type to Lasso (L1) and set the Lambda slider to -1. Count the grey bars: coefficients have gone to exactly zero, and the non-zero count falls. Ridge at the same lambda leaves every bar standing.Run this
5. Starve it of data. Set the Training Points slider to 8 with degree 12 and no penalty. With fewer points than parameters the fit is wild. Add the penalty back and it becomes usable again — this is why regularisation matters most when data is scarce.Run this
6. Remove the noise. Set the Noise slider to 0. Now the high-degree fit is fine without any penalty, because there is no noise left to memorise. Overfitting is a response to noise, not to flexibility on its own.Run this

Choosing between them

• Ridge when you believe most features carry some signal, and especially when features are correlated. Ridge spreads weight across a correlated group rather than picking one arbitrarily.
• Lasso when you suspect most features are irrelevant and you want the model to tell you which. The zeros are the output you care about, as much as the predictions are.
• Elastic Net, which adds both penalties, when you want Lasso's selection but have correlated groups that Lasso would otherwise split arbitrarily.

In practice λ is not chosen by eye. It is selected by cross-validation: fit at many values, keep the one with the best held-out error.

What usually goes wrong

• Forgetting to scale first. The penalty acts on raw coefficient magnitudes, and a coefficient's size depends on its feature's units. A feature measured in metres gets a coefficient a thousand times larger than the same feature in millimetres, so it absorbs a thousand times more penalty. Standardise before regularising; this is not optional.
• Penalising the intercept. The bias term should not be shrunk — doing so drags predictions toward zero rather than toward the mean. Every good implementation excludes it; if you write your own, remember to.
• Reading Lasso's zeros as truth. With correlated features Lasso picks one from a group more or less arbitrarily, and a small change in the data can flip which one survives. Its selection is useful, not stable.
• Tuning lambda on the test set. That makes the test score optimistic and no longer an estimate of anything. Use a validation split or cross-validation, and keep the test set for the end.

Key Takeaway

Both methods add a penalty on coefficient size to the loss, trading a little training accuracy for a lot of stability, with lambda controlling how hard that trade is pushed. Ridge squares the weights, so its pressure fades as a coefficient nears zero and everything merely shrinks; Lasso uses absolute values, so its pressure stays constant and weak coefficients are driven to exactly zero, which makes it a feature selector as well as a regulariser. Scale your features first, never penalise the intercept, and pick lambda by cross-validation rather than by eye.