Logistic Regression

From a score to a probability

For features x₁, …, xₚ, the model first calculates a linear score:

z = w₁x₁ + … + wₚxₚ + b

It maps that score through the sigmoid function:

p = 1 / (1 + e^−z)

Equivalently, log(p / (1 − p)) = z. The weights therefore act on log-odds, not directly on probability. A threshold—often 0.5 by default—turns p into a predicted class. Changing the threshold changes labels without changing the fitted probabilities.

Work through one prediction

Use weights w₁ = w₂ = 1 and bias b = −10, as in the visualiser. For the point (4, 4), z = 4 + 4 − 10 = −2, so p = sigmoid(−2) ≈ 0.119. The model assigns about a 12% probability to class B. At (6, 6), z = 2 and p ≈ 0.881. At (5, 5), z = 0 and p = 0.5, exactly on the boundary for a 0.5 threshold.

Explore the model

1. Click Fit with Gradient Descent and watch the weights and loss change as the model fits the displayed labels.
2. Set Weight w1 to −3. The first feature now pushes the score in the opposite direction, so the boundary rotates and predictions may worsen.
3. Change Threshold while keeping the fitted weights fixed. The probabilities stay the same; the class boundary moves, trading false positives against false negatives.

Regularisation and feature scale

Scikit-learn’s LogisticRegression uses L2 regularisation by default, with C=1.0. C is the inverse of regularisation strength: smaller values penalise large coefficients more strongly. Regularisation helps keep estimates finite and reduces overfitting; see the official LogisticRegression documentation for solver and penalty options.

Because the penalty acts on coefficient sizes, feature units matter. A change of units changes the coefficient needed to represent the same effect and therefore changes how strongly it is penalised. Scale numeric features when their units differ, and fit the scaler only on each training split. A pipeline keeps that order intact:

~~~python from sklearn.linear_model import LogisticRegression from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler

model = make_pipeline( StandardScaler(), LogisticRegression(C=1.0, max_iter=1000), ) model.fit(X_train, y_train) probabilities = model.predict_proba(X_test)[:, 1] ~~~

Assumptions and common mistakes

• Reading a coefficient as a probability change. A one-unit increase in xⱼ changes the log-odds by wⱼ, with other features held fixed. The corresponding odds ratio is exp(wⱼ); the probability change depends on the starting point.
• Treating 0.5 as a universal threshold. It is appropriate only under particular probability and error-cost assumptions. Choose a threshold on validation data when false-positive and false-negative costs differ; see threshold tuning.
• Expecting a curved boundary from raw features. The boundary is linear in the supplied features. Nonlinear patterns require transformed features or a different model.
• Interpreting coefficients with strongly correlated inputs. Predictions may remain useful while individual coefficients become unstable and difficult to interpret.

For a closer look at how probabilities compare with decisions, continue to probability calibration and the confusion matrix.