Ridge and Lasso Regression

One extra term

Ordinary least squares minimises the squared error alone. Ridge and Lasso add a penalty on the size of the weights:

Ridge:   minimise   SSE + λ · Σ w_i²

Lasso:   minimise   SSE + λ · Σ |w_i|

λ (lambda) is the dial. At λ = 0 both reduce exactly to OLS. As λ grows the penalty dominates and every coefficient is pushed toward zero, until at very large λ the model gives up and predicts a flat line through the mean.

The only difference between the two is whether you square the weights or take their absolute value. That one change has a consequence out of all proportion to its size.

Why Lasso zeroes and Ridge does not

Look at how each penalty behaves as a coefficient approaches zero. The derivative of w² is 2w, which itself goes to zero — so the closer a Ridge coefficient gets to zero, the weaker the force pushing it further. It approaches zero and never arrives.

The derivative of |w| is a constant ±1, no matter how small w is. Lasso pushes with the same strength all the way down, so a coefficient whose contribution to the fit is worth less than that constant push gets driven to exactly zero and stays there.

That is why Lasso performs feature selection and Ridge does not. Watch the coefficient bars above: under Ridge they all shrink together and stay non-zero; under Lasso they vanish one by one.

Why a penalty helps at all

Plain least squares has one instruction: make the training errors as small as possible. With many features, or with features that are nearly duplicates of each other, it will do exactly that by assigning some enormous coefficients that happen to cancel each other out — +4,300 on one column, −4,290 on a near-copy.

Those coefficients fit the training data beautifully and mean nothing. Move one input slightly and the prediction swings wildly, because two huge numbers are no longer cancelling.

Regularisation adds a second instruction: and keep the coefficients small. The model now minimises

error on the training data  +  λ × (size of the coefficients)

λ (called alpha in scikit-learn) sets the exchange rate between the two. At λ = 0 you are back to ordinary least squares. As λ grows, the model becomes progressively less willing to buy a small error reduction with a large coefficient, and at very large λ every coefficient is pushed to nearly zero and the model predicts the mean.

Deliberately accepting some bias to remove a lot of variance is the whole trade, and on data with many correlated features it is nearly always worth making.

The one difference that changes everything

The two methods differ only in how "size of the coefficients" is measured.

• Ridge (L2) penalises the sum of squared coefficients.
• Lasso (L1) penalises the sum of absolute coefficients.

That small change produces a large behavioural difference. Squaring means the penalty on a coefficient of 0.1 is 0.01 — almost nothing — so ridge has very little incentive to push a small coefficient all the way to zero. It shrinks everything smoothly and keeps every feature.

With absolute values, the penalty on 0.1 is 0.1, and the pressure to remove it entirely is exactly as strong as the pressure on any other unit of coefficient. So lasso pushes weak coefficients to exactly zero, and the features attached to them drop out of the model completely.

That makes lasso a feature selector as well as a regulariser. Fit it with a moderate alpha on 200 columns and you may be left with 30 non-zero coefficients and a model you can actually read.

The correlated-features row is the one that decides most real cases. Ridge splits the weight across a group of correlated columns, which is stable and sensible. Lasso arbitrarily keeps one and zeroes the others, and which one it keeps can change with a small change in the data — unstable, though the predictions stay similar.

Elastic Net uses both penalties with a mixing ratio, and it is the standard answer when you want lasso's sparsity without its instability on correlated groups.

Two penalties, two very different results

Ridge shrinks every coefficient toward zero; lasso sends most of them exactly to zero. Watching both on the same correlated data shows why that difference matters.

import numpy as np
from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(0)
n, p = 120, 60
X = rng.normal(size=(n, p))
X[:, 1] = X[:, 0] + rng.normal(0, 0.05, n)     # a near-duplicate column
true = np.zeros(p); true[[0, 5, 12, 30]] = [4.0, -3.0, 2.5, 1.5]
y = X @ true + rng.normal(0, 1.5, n)
print("%d rows, %d columns, and only 4 of those columns matter." % (n, p))
print()

def report(name, model):
    m = make_pipeline(StandardScaler(), model)
    cv = cross_val_score(m, X, y, cv=5).mean()
    m.fit(X, y)
    coef = m[-1].coef_
    nz = (np.abs(coef) > 1e-6).sum()
    print("%-22s cv R2 %7.4f   nonzero %3d/%d   largest |coef| %7.3f"
          % (name, cv, nz, p, np.abs(coef).max()))
    return coef

c_ols   = report("plain least squares", LinearRegression())
c_ridge = report("ridge, alpha=10",     Ridge(alpha=10.0))
c_lasso = report("lasso, alpha=0.3",    Lasso(alpha=0.3, max_iter=10000))
c_en    = report("elastic net",         ElasticNet(alpha=0.3, l1_ratio=0.5,
                                                   max_iter=10000))
print()
print("what each one did to the four real coefficients and one noise column:")
print("%8s %10s %10s %10s %10s" % ("column", "OLS", "ridge", "lasso", "truth"))
for i in (0, 5, 12, 30, 7):
    label = "col %d" % i + (" *" if true[i] else "")
    print("%8s %10.3f %10.3f %10.3f %10.3f"
          % (label, c_ols[i], c_ridge[i], c_lasso[i], true[i]))
print("  (* marks a column that genuinely matters. col 7 is pure noise.)")
print()

print("columns 0 and 1 are 99 percent the same variable, and only column 0")
print("is real. watch what each does with that pair as the penalty tightens:")
print("%-24s %9s %9s %9s" % ("", "col 0", "col 1", "col0+col1"))
for name, c in (("OLS (no penalty)", c_ols),):
    print("%-24s %9.3f %9.3f %9.3f" % (name, c[0], c[1], c[0] + c[1]))
for a in (0.3, 1.0):
    for kind, mk in (("ridge",
                      lambda: Ridge(alpha=a)),
                     ("lasso",
                      lambda: Lasso(alpha=a, max_iter=10000))):
        c = make_pipeline(StandardScaler(), mk()).fit(X, y)[-1].coef_
        print("%-24s %9.3f %9.3f %9.3f"
              % ("%s, alpha=%.1f" % (kind, a), c[0], c[1], c[0] + c[1]))
print()
print("ridge keeps both and splits the credit evenly -- that is what squaring")
print("the coefficients rewards. lasso keeps the total roughly right too, but")
print("as alpha rises it stops hedging and puts everything on one column.")
print("neither total is far off. only ridge gives you the same SPLIT if you")
print("refit on slightly different rows, which is why lasso coefficients are")
print("a poor thing to interpret when your columns are correlated.")
print()

print("the alpha sweep, which is the only hyperparameter either has:")
print("%10s %10s %8s %10s %8s" % ("alpha", "ridge cv", "|w|max", "lasso cv", "nonzero"))
for a in (0.01, 0.1, 1.0, 10.0, 100.0):
    r = make_pipeline(StandardScaler(), Ridge(alpha=a)).fit(X, y)
    l = make_pipeline(StandardScaler(), Lasso(alpha=a, max_iter=10000)).fit(X, y)
    print("%10.2f %10.4f %8.3f %10.4f %8d"
          % (a,
             cross_val_score(make_pipeline(StandardScaler(), Ridge(alpha=a)),
                             X, y, cv=5).mean(),
             np.abs(r[-1].coef_).max(),
             cross_val_score(make_pipeline(StandardScaler(),
                                           Lasso(alpha=a, max_iter=10000)),
                             X, y, cv=5).mean(),
             (np.abs(l[-1].coef_) > 1e-6).sum()))
print()
print("alpha=0 is plain OLS. alpha to infinity sends everything to zero and")
print("predicts the mean. everything useful happens in between, and the only")
print("way to find it is to cross-validate -- RidgeCV and LassoCV do exactly")
print("that loop for you.")
print()
print("and both need scaling. the penalty charges by coefficient size, so a")
print("column measured in millimetres gets a smaller coefficient and a smaller")
print("penalty than the same column in metres.")

Guided tour

1. See the problem first. Set the Penalty Type to None (plain OLS) and set the Polynomial Degree slider to 12. The curve whips between the training points, train MSE is almost zero, and test MSE is enormous. That gap is overfitting, made visible.
2. Apply a little Ridge. Switch Penalty Type to Ridge (L2) and set the Lambda slider to -3. The curve calms down immediately and test MSE drops, while train MSE rises slightly. That trade is the entire point of regularisation.
3. Overdo it. Set the Lambda slider to its maximum, 2. The model is now so heavily penalised that it flattens toward a straight line and underfits — both errors climb together. Regularisation is a dial, not a switch.
4. Watch Lasso delete. Switch Penalty Type to Lasso (L1) and set the Lambda slider to -1. Count the grey bars: coefficients have gone to exactly zero, and the non-zero count falls. Ridge at the same lambda leaves every bar standing.
5. Starve it of data. Set the Training Points slider to 8 with degree 12 and no penalty. With fewer points than parameters the fit is wild. Add the penalty back and it becomes usable again — this is why regularisation matters most when data is scarce.
6. Remove the noise. Set the Noise slider to 0. Now the high-degree fit is fine without any penalty, because there is no noise left to memorise. Overfitting is a response to noise, not to flexibility on its own.

Choosing between them

• Ridge when you believe most features carry some signal, and especially when features are correlated. Ridge spreads weight across a correlated group rather than picking one arbitrarily.
• Lasso when you suspect most features are irrelevant and you want the model to tell you which. The zeros are the output you care about, as much as the predictions are.
• Elastic Net, which adds both penalties, when you want Lasso's selection but have correlated groups that Lasso would otherwise split arbitrarily.

In practice λ is not chosen by eye. It is selected by cross-validation: fit at many values, keep the one with the best held-out error.

Failure modes

• Forgetting to scale first. The penalty acts on raw coefficient magnitudes, and a coefficient's size depends on its feature's units. A feature measured in metres gets a coefficient a thousand times larger than the same feature in millimetres, so it absorbs a thousand times more penalty. Standardise before regularising; this is not optional.
• Penalising the intercept. The bias term should not be shrunk — doing so drags predictions toward zero rather than toward the mean. Every good implementation excludes it; if you write your own, remember to.
• Reading Lasso's zeros as truth. With correlated features Lasso picks one from a group more or less arbitrarily, and a small change in the data can flip which one survives. Its selection is useful, not stable.
• Tuning lambda on the test set. That makes the test score optimistic and no longer an estimate of anything. Use a validation split or cross-validation, and keep the test set for the end.

Where that leaves you

Both methods add a penalty on coefficient size to the loss, trading a little training accuracy for a lot of stability, with lambda controlling how hard that trade is pushed. Ridge squares the weights, so its pressure fades as a coefficient nears zero and everything merely shrinks; Lasso uses absolute values, so its pressure stays constant and weak coefficients are driven to exactly zero, which makes it a feature selector as well as a regulariser. Scale your features first, never penalise the intercept, and pick lambda by cross-validation rather than by eye.

Choosing alpha, and the scaling requirement

Alpha is not a setting you reason your way to — it is chosen by cross-validation, and scikit-learn provides estimators that do exactly that.

from sklearn.linear_model import RidgeCV, LassoCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
import numpy as np

alphas = np.logspace(-3, 3, 25)      # 0.001 to 1000, log-spaced

ridge = make_pipeline(StandardScaler(), RidgeCV(alphas=alphas, cv=5))
lasso = make_pipeline(StandardScaler(), LassoCV(alphas=alphas, cv=5))

ridge.fit(X_train, y_train)
print(ridge[-1].alpha_)               # the value cross-validation chose

Search alpha on a logarithmic scale — the interesting range spans orders of magnitude, and a linear grid wastes almost all its points.

Scaling is mandatory. The penalty is applied to raw coefficient values, so a feature measured in thousands naturally has a tiny coefficient and is barely penalised, while a feature measured in units has a large one and is penalised heavily. Standardise first and the penalty falls evenly. Skipping this is the most common way to get nonsense out of a regularised model.

Note also that the intercept is not penalised, and should not be — shrinking it towards zero would mean insisting the target is centred at zero.

Reading the coefficient path

The most informative plot in this topic is the coefficient path: alpha on a log x-axis, coefficient values on the y-axis, one line per feature.

For ridge, the lines converge smoothly towards zero as alpha grows, but never touch it. For lasso, lines hit zero one after another and stay there — and the order in which they drop out is a ranking of how much each feature was contributing. The last few survivors are the features carrying the signal.

Two things to look for. A coefficient that changes sign as alpha varies is a warning about collinearity, not a finding. And a large gap between the alpha with the lowest cross-validation error and the "one standard error" alpha — the largest alpha within one standard error of the best — is a chance to buy a much simpler model for almost no accuracy. That heuristic is standard practice in statistics and underused elsewhere.

Questions people ask

Which should I use by default? Ridge, if you have no strong reason to prefer otherwise. It is stable, has a closed-form solution, and does not throw away information. Move to lasso when you specifically want a smaller feature set, and to elastic net when you want both.

Does regularisation work with logistic regression? Yes, identically, and scikit-learn applies L2 by default with C = 1/alpha. Everything here transfers.

Can lasso select more features than there are rows? No — lasso can select at most n features from a dataset with n rows. Elastic net does not have this limitation, which is another reason it is preferred in wide, short datasets such as genomics.

Should I trust lasso's feature selection? As a ranking, broadly. As a definitive statement about which features matter, no — refit with different random splits and see how stable the selected set is. Stability selection formalises this.

Is regularisation the same as feature selection? Lasso happens to do both. Ridge does not select anything; it keeps every feature and reduces their influence.

What if alpha comes out at essentially zero? Then your data does not need regularisation — plenty of rows, few features, low collinearity. That is a legitimate outcome and worth noticing.

Recap in one screen

• Both methods add a penalty on coefficient size to the usual squared-error objective.
• Ridge squares the coefficients: smooth shrinkage, nothing reaches zero, correlated features share the weight.
• Lasso uses absolute values: weak coefficients hit exactly zero, so the model selects features.
• Elastic net mixes the two and is the safe default on correlated, wide data.
• Standardise your features, search alpha on a log scale by cross-validation, and read the coefficient path.