Logistic Regression

Quick Context

Linear regression predicts a number. If you need a yes or no, you cannot use it directly: it happily predicts −4 or 17, and neither is a probability. Logistic regression fixes this with one extra step. It computes the same weighted sum, then passes the result through a function that squashes any real number into the range 0 to 1.

That is the entire model. It is called regression because the inside is a linear regression, and it is used for classification because the outside is a probability.

The two steps

First the linear score, exactly as in ordinary least squares:

z = w₁x₁ + w₂x₂ + b

Then the sigmoid, which turns that score into a probability:

p = 1 / (1 + e^−z)

Sigmoid has three properties that matter. It is 0.5 exactly when z is 0. It approaches 1 as z grows and 0 as z falls, without ever reaching either. And it is smooth everywhere, so it has a gradient everywhere — which is what lets the model be trained by gradient descent at all.

Work one through by hand

Take weights w = [1.0, 1.0] and bias b = −10, the values this page starts on. Feed it the point (4, 4):

z = (1.0 × 4) + (1.0 × 4) − 10 = −2

p = 1 / (1 + e²) = 0.12

So the model gives that point a 12% chance of being class B — a fairly confident vote for class A. Now move to (6, 6): z becomes +2 and p becomes 0.88, the mirror image. The point (5, 5) sits exactly on the boundary, where z is 0 and p is 0.50.

Where the boundary comes from

The model predicts class B whenever p is above the threshold. With the usual threshold of 0.5, that happens exactly when z is above 0. So the decision boundary is the set of points where w₁x₁ + w₂x₂ + b = 0 — a straight line, and in higher dimensions a flat plane.

This is worth being precise about, because it is the model's main limitation. The probabilities curve; the boundary does not. Logistic regression can express "confidence falls off gradually as you move away from the line", but it cannot express a boundary that bends.

Interactive Exploration Guide

1. Let it find the answer. Click Fit with Gradient Descent and watch the boundary swing into position while log loss falls. The weights that arrive are the ones that make the observed labels most likely.Run this
2. Break it deliberately. Set the Weight w1 slider to -3. The boundary flips its orientation and accuracy collapses, because the model now reads the first feature backwards.Run this
3. Separate the classes. Set the Class Overlap slider to 0.3 and fit again. With clean separation the shading turns sharply from blue to orange — the model is confident everywhere except a thin band.Run this
4. Then make it hard. Set the Class Overlap slider to 3. Fit again and the shading becomes a soft gradient: the model is honestly uncertain across most of the plot, and log loss stays high no matter what the weights do.Run this
5. Move the threshold, not the model. Set the Threshold slider to 0.9. The weights and the probabilities do not change at all — only the line at which a probability becomes a prediction. Watch false positives fall and false negatives rise.Run this
6. Read a single point. Drag the white probe across the boundary and watch the probability pass through 0.50. That number is what the model actually outputs; the class label is just a comparison against the threshold.

Why not squared error?

Logistic regression is trained with log loss (also called binary cross-entropy), not the squared error used for linear regression:

loss = −[ y · log(p) + (1 − y) · log(1 − p) ]

There are two reasons. The first is practical: squared error applied to a sigmoid produces a non-convex surface with local minima that gradient descent can get stuck in, while log loss is convex and has a single global optimum. The second is about incentives. Log loss goes to infinity as a confident prediction turns out to be wrong, so a model that says 0.99 and is wrong is punished enormously. Squared error caps that penalty at 1, which lets a badly calibrated classifier off far too lightly.

The link to neural networks

A single logistic regression is a single neuron. The weighted sum is the dot product, the sigmoid is the activation, and the training signal is the same gradient descent step. Stack these units into layers and you have a neural network; that is why the perceptron module looks so familiar after this one.

The reverse is also true and worth remembering: if your network has one layer and a sigmoid output, you have not built anything more powerful than logistic regression, no matter how it is implemented.

What usually goes wrong

• Reading coefficients as probabilities. A weight of 2.0 does not mean "2% more likely". Weights act on the log-odds scale; exponentiate one to get an odds ratio, which is the only interpretation that holds.
• Unscaled features. The weights are fitted jointly, so a feature measured in thousands and one measured in decimals produce wildly different coefficient magnitudes and a badly conditioned optimisation. Scale first.
• Perfect separation. If a straight line separates the classes exactly, the likelihood is maximised by pushing the weights to infinity. Training does not converge and the coefficients blow up. Regularisation — see Ridge and Lasso — is the standard fix.
• Leaving the threshold at 0.5. That default is only right when the classes are balanced and the two kinds of error cost the same. On an imbalanced problem it is usually the wrong place to stand; pick the threshold from the ROC curve instead.

Key Takeaway

Logistic regression computes a linear score and squashes it through a sigmoid to get a probability, which makes the decision boundary a straight line and the output something you can actually reason about. It is trained with log loss because that surface is convex and punishes confident mistakes properly. It remains the sensible first classifier to try on any problem: it is fast, it rarely overfits, its coefficients can be inspected, and it gives you a calibrated probability rather than a bare label — and if it is beaten by something more complex, you at least know by how much.