Ridge and Lasso Regression
Turn the penalty up and watch a wild polynomial calm down. Ridge shrinks every coefficient; Lasso deletes them.
Parameters
log scale, 1e-6 to 1e+2
The Fit
degree 9Coefficients
Bar height is |coefficient|, on a log scale. Grey means driven to exactly zero.
Error
Model Size
Overview
Regularisation discourages a regression model from using unnecessarily large coefficients. Ridge uses a squared penalty; Lasso uses an absolute-value penalty. That difference affects whether fitted coefficients can become exactly zero.
What the penalties change
Ordinary least squares minimises the sum of squared residuals. Ridge and Lasso add a penalty controlled by λ:
Ridge: SSE + λ Σ wᵢ²
Lasso: SSE + λ Σ |wᵢ|
At λ = 0, both reduce to ordinary least squares. As λ grows, large coefficients become more costly and the fit is pulled toward smaller coefficients. In the extreme, predictions lose their feature-dependent variation.
Ridge’s smooth squared penalty usually shrinks coefficients but leaves them nonzero. Lasso’s absolute-value penalty has a sharp point at zero; the optimum can land exactly there. Lasso can therefore produce a sparse model, though which features survive may be unstable when predictors are strongly correlated.
See the effect on a flexible fit
- Set Cost Type to None and Polynomial Degree to 12. A high-degree curve can fit the training points closely while behaving erratically between them.
- Switch to Ridge and increase Lambda. The curve becomes smoother as coefficients shrink; training error may rise while test error falls.
The visualiser uses a small synthetic dataset. In a real problem, select λ using validation data or cross-validation—not the final test set.
Which one should you choose?
| Method | Useful when | Watch for |
|---|---|---|
| Ridge (L2) | Many features have some signal, especially when predictors are correlated | Coefficients shrink together and are rarely exactly zero. |
| Lasso (L1) | A sparse coefficient vector is useful and many features may be irrelevant | It can select one of several correlated features arbitrarily. |
| Elastic Net | You want sparsity but also need to handle correlated groups | It combines L1 and L2 penalties and has an additional mixing parameter. |
Scale numeric features before applying coefficient penalties when their units differ. Otherwise the same real-world effect can require different coefficient sizes and receive different penalty strength. Fit scaling inside each training fold; see feature scaling.
Limits and interpretation
Regularisation can reduce overfitting, but it does not guarantee better predictions. A nonzero coefficient is not proof that a feature is important, and a zero Lasso coefficient does not prove the feature has no relationship with the target. With correlated predictors, multiple coefficient patterns may predict similarly. Check performance on held-out data and interpret coefficients in the context of the feature scale and model assumptions.
Sources and next steps
See scikit-learn’s linear-model documentation for Ridge and Lasso estimators. Continue with cross-validation to select regularisation strength.