Logistic Regression

The two steps

First the linear score, exactly as in ordinary least squares:

z = w₁x₁ + w₂x₂ + b

Then the sigmoid, which turns that score into a probability:

p = 1 / (1 + e^−z)

Sigmoid has three properties that matter. It is 0.5 exactly when z is 0. It approaches 1 as z grows and 0 as z falls, without ever reaching either. And it is smooth everywhere, so it has a gradient everywhere — which is what lets the model be trained by gradient descent at all.

Work one through by hand

Take weights w = [1.0, 1.0] and bias b = −10, the values this page starts on. Feed it the point (4, 4):

z = (1.0 × 4) + (1.0 × 4) − 10 = −2

p = 1 / (1 + e²) = 0.12

So the model gives that point a 12% chance of being class B — a fairly confident vote for class A. Now move to (6, 6): z becomes +2 and p becomes 0.88, the mirror image. The point (5, 5) sits exactly on the boundary, where z is 0 and p is 0.50.

Where the boundary comes from

The model predicts class B whenever p is above the threshold. With the usual threshold of 0.5, that happens exactly when z is above 0. So the decision boundary is the set of points where w₁x₁ + w₂x₂ + b = 0 — a straight line, and in higher dimensions a flat plane.

This is worth being precise about, because it is the model's main limitation. The probabilities curve; the boundary does not. Logistic regression can express "confidence falls off gradually as you move away from the line", but it cannot express a boundary that bends.

A worked prediction, end to end

Predict whether a student passes an exam from hours studied. Suppose the fitted model is:

z = 1.2 × hours − 4.0

A student who studied 4 hours: z = 1.2×4 − 4.0 = 0.8. Push that through the sigmoid:

p = 1 / (1 + e^−0.8) = 1 / (1 + 0.449) = 0.69

A 69% chance of passing. At 2 hours, z = −1.6 and p = 0.17. At 3.33 hours, z = 0 and p = 0.50 exactly — that is the decision boundary, the point where the model is perfectly undecided.

Notice what the coefficient 1.2 means. It is not "one more hour adds 1.2 to the probability" — probabilities do not work like that. It means one more hour adds 1.2 to the log-odds, which multiplies the odds by e^1.2 = 3.3. So each extra hour makes passing 3.3 times more likely in odds terms, whether you were at bad odds or good ones.

That is the trick logistic regression pulls off: the relationship is a straight line in log-odds space, which becomes an S-curve in probability space. Straight lines are easy to fit; S-curves are what probabilities actually look like.

Reading the coefficients in odds

Because of that relationship, every coefficient has a clean interpretation once you exponentiate it.

This is why logistic regression survives in medicine, credit scoring and any regulated setting. "Smoking multiplies the odds of this outcome by 2.4, holding age and weight constant" is a sentence a clinician, a regulator and a court can all work with. A gradient-boosted ensemble may be more accurate and cannot produce that sentence.

Two cautions. Odds ratios are not probability ratios — doubling the odds from 1:100 barely moves the probability, while doubling from 1:1 moves it a lot. And these readings assume the other features are held constant, which requires that they are not near-duplicates of each other.

Regularisation, and the defaults that surprise people

Scikit-learn's LogisticRegression applies L2 regularisation by default, with C=1.0. This catches people out: coefficients come back smaller than the maths textbook predicts, because the model is being penalised for large ones.

C is the inverse of the penalty strength. Small C means heavy regularisation and small, cautious coefficients; large C means little regularisation and coefficients free to grow. Turning it off entirely (penalty=None) is available but rarely wise.

Regularisation matters more here than in linear regression because of perfect separation. If some feature separates the classes exactly — every student above 5 hours passed, every student below failed — the likelihood keeps improving as the coefficient grows towards infinity. Unregularised fitting either fails to converge or returns absurd coefficients like 4,000. The L2 penalty stops this by making enormous coefficients expensive.

from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

model = make_pipeline(
    StandardScaler(),                       # so the penalty falls evenly
    LogisticRegression(C=1.0, max_iter=1000, class_weight="balanced"),
)
model.fit(X_train, y_train)
model.predict_proba(X_test)[:, 1]           # probabilities, not labels

Scaling is required once regularisation is on: the penalty is applied to raw coefficient sizes, so a feature measured in thousands gets an artificially tiny coefficient and is effectively regularised out of the model.

Multi-class, and what to use instead

For more than two classes there are two schemes. One-vs-rest trains one binary model per class and takes the highest score. Multinomial (softmax) regression fits all classes jointly with a softmax over their scores, which produces probabilities that sum to 1 and is generally the better choice; modern scikit-learn uses it automatically for multi-class problems.

And a short honest comparison of when to reach elsewhere:

That last row is underrated. Add squared terms, interactions or splines and a logistic model handles curved boundaries perfectly well — it is linear in its parameters, not in your data.

A line, then a squash

Logistic regression is linear regression's output pushed through a sigmoid. Computing the probability by hand from the coefficients shows there is nothing else in the box.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, log_loss

X, y = make_classification(n_samples=3000, n_features=4, n_informative=3,
                           n_redundant=0, n_clusters_per_class=1,
                           flip_y=0.05, class_sep=1.4, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
model = LogisticRegression().fit(Xtr, ytr)

print("coefficients", np.round(model.coef_[0], 4))
print("intercept   %.4f" % model.intercept_[0])
print()

sigmoid = lambda z: 1 / (1 + np.exp(-z))
row = Xte[0]
z = row @ model.coef_[0] + model.intercept_[0]
print("one row, done by hand:")
print("  features     ", np.round(row, 4))
print("  w . x + b  = %.6f     <- this is just linear regression" % z)
print("  sigmoid(z) = %.6f     <- squashed into (0, 1)" % sigmoid(z))
print("  sklearn says %.6f" % model.predict_proba([row])[0, 1])
print()

print("the sigmoid, at a few points:")
for zz in (-6, -2, -0.5, 0, 0.5, 2, 6):
    print("   z=%5.1f -> %.6f" % (zz, sigmoid(zz)))
print("  z=0 is the decision boundary, where the model is exactly undecided.")
print()

print("what the coefficients mean -- multiply the ODDS, not the probability:")
for i, c in enumerate(model.coef_[0]):
    print("   feature %d: +1 unit multiplies the odds by %.4f" % (i, np.exp(c)))
print()
p = model.predict_proba(Xte)[:, 1]
big = int(np.argmax(np.abs(model.coef_[0])))
print("   take feature %d, the strongest one (odds x %.4f):"
      % (big, np.exp(model.coef_[0][big])))
for base in (0.10, 0.50, 0.90):
    odds = base / (1 - base)
    new = odds * np.exp(model.coef_[0][big])
    print("     a row at p=%.2f moves to p=%.4f, a change of %+.4f"
          % (base, new / (1 + new), new / (1 + new) - base))
print("   one coefficient, three different-sized changes. that is why you")
print("   cannot read a logistic coefficient as 'adds N% to the chance' --")
print("   it adds a constant amount to the LOG ODDS, not to the probability.")
print()

print("it is trained on log loss, not accuracy:")
print("  accuracy %.4f, log loss %.4f" % (accuracy_score(yte, model.predict(Xte)),
                                          log_loss(yte, p)))
print("  log loss punishes confident mistakes specifically:")
for prob, truth in ((0.99, 1), (0.60, 1), (0.40, 1), (0.01, 1)):
    print("    said %.2f, answer was %d -> loss %.4f"
          % (prob, truth, -np.log(prob if truth else 1 - prob)))

Things to try

1. Let it find the answer. Click Fit with Gradient Descent and watch the boundary swing into position while log loss falls. The weights that arrive are the ones that make the observed labels most likely.
2. Break it deliberately. Set the Weight w1 slider to -3. The boundary flips its orientation and accuracy collapses, because the model now reads the first feature backwards.
3. Separate the classes. Set the Class Overlap slider to 0.3 and fit again. With clean separation the shading turns sharply from blue to orange — the model is confident everywhere except a thin band.
4. Then make it hard. Set the Class Overlap slider to 3. Fit again and the shading becomes a soft gradient: the model is honestly uncertain across most of the plot, and log loss stays high no matter what the weights do.
5. Move the threshold, not the model. Set the Threshold slider to 0.9. The weights and the probabilities do not change at all — only the line at which a probability becomes a prediction. Watch false positives fall and false negatives rise.
6. Read a single point. Drag the white probe across the boundary and watch the probability pass through 0.50. That number is what the model actually outputs; the class label is just a comparison against the threshold.

Why not squared error?

Logistic regression is trained with log loss (also called binary cross-entropy), not the squared error used for linear regression:

loss = −[ y · log(p) + (1 − y) · log(1 − p) ]

There are two reasons. The first is practical: squared error applied to a sigmoid produces a non-convex surface with local minima that gradient descent can get stuck in, while log loss is convex and has a single global optimum. The second is about incentives. Log loss goes to infinity as a confident prediction turns out to be wrong, so a model that says 0.99 and is wrong is punished enormously. Squared error caps that penalty at 1, which lets a badly calibrated classifier off far too lightly.

The link to neural networks

A single logistic regression is a single neuron. The weighted sum is the dot product, the sigmoid is the activation, and the training signal is the same gradient descent step. Stack these units into layers and you have a neural network; that is why the perceptron module looks so familiar after this one.

The reverse is also true and worth remembering: if your network has one layer and a sigmoid output, you have not built anything more powerful than logistic regression, no matter how it is implemented.

Traps worth knowing

• Reading coefficients as probabilities. A weight of 2.0 does not mean "2% more likely". Weights act on the log-odds scale; exponentiate one to get an odds ratio, which is the only interpretation that holds.
• Unscaled features. The weights are fitted jointly, so a feature measured in thousands and one measured in decimals produce wildly different coefficient magnitudes and a badly conditioned optimisation. Scale first.
• Perfect separation. If a straight line separates the classes exactly, the likelihood is maximised by pushing the weights to infinity. Training does not converge and the coefficients blow up. Regularisation — see Ridge and Lasso — is the standard fix.
• Leaving the threshold at 0.5. That default is only right when the classes are balanced and the two kinds of error cost the same. On an imbalanced problem it is usually the wrong place to stand; pick the threshold from the ROC curve instead.

In one line

Logistic regression computes a linear score and squashes it through a sigmoid to get a probability, which makes the decision boundary a straight line and the output something you can actually reason about. It is trained with log loss because that surface is convex and punishes confident mistakes properly. It remains the sensible first classifier to try on any problem: it is fast, it rarely overfits, its coefficients can be inspected, and it gives you a calibrated probability rather than a bare label — and if it is beaten by something more complex, you at least know by how much.

Questions people ask

Why is it called regression if it classifies? Because it regresses on the log-odds, which is a continuous quantity. The classification comes afterwards, from applying a threshold.

Can I use it for probabilities directly? Yes, and this is one of its strengths — logistic regression is usually better calibrated out of the box than most alternatives. Check with a reliability diagram before betting on it.

What if my classes are imbalanced? Use class_weight="balanced" and move the threshold away from 0.5. The model handles imbalance more gracefully than most, but the default threshold still assumes symmetric costs.

My model will not converge. What now? Scale the features, raise max_iter, and check for perfectly separating features. Those three cover nearly every case.

Do I need to remove correlated features? For prediction, no. For interpreting individual coefficients, yes — collinearity makes them unstable, and the "holding others constant" reading stops being meaningful.

How is this related to a neural network? A logistic regression is exactly one neuron with a sigmoid activation. Stack them into layers and you have a network; the loss function and the gradient descent are the same machinery.

Recap in one screen

• Compute a weighted sum, squash it through a sigmoid, get a probability, apply a threshold.
• Coefficients are additive in log-odds, so exponentiate them to read multiplicative effects on the odds.
• The boundary is a straight line (or plane) — curved boundaries need engineered features.
• L2 regularisation is on by default and is what protects you from perfect separation.
• Scale your features whenever regularisation is active.
• Well calibrated, fully interpretable, and the direct ancestor of the neural network.