Softmax and Cross-Entropy

What softmax does

p_i = e^z_i / Σ_j e^z_j

Exponentiate every score, then divide by the total. Exponentiating makes everything positive; dividing by the sum makes it add to 1. Those two steps are the whole function.

Two properties are worth holding on to. It is monotonic: the largest logit always becomes the largest probability, so softmax never changes which class wins. And it is shift-invariant: adding the same constant to every logit leaves the probabilities completely unchanged, because the constant factors out of the numerator and denominator together.

What cross-entropy does

For a single correct label, the loss collapses to one term:

loss = −log( p_true )

Every other class drops out, because the true distribution is 1 for the right answer and 0 everywhere else. So the loss depends on one number: the probability the model gave to the correct class.

Look at the curve. At p = 1 the loss is 0. At p = 0.5 it is 0.69. At p = 0.1 it is 2.3. As p approaches 0 it goes to infinity. That asymptote is the point: a model that is confidently wrong is punished without limit, while one that hedges is only mildly penalised.

From scores to probabilities

A classifier's final layer outputs one unbounded number per class — a logit. Softmax turns those into probabilities:

pᵢ = e^zᵢ / Σ_j e^zᵎ

Worked through on logits [2.0, 1.0, 0.1]:

• Exponentiate: e² = 7.39, e¹ = 2.72, e⁰·¹ = 1.11. Sum = 11.22.
• Divide: 0.659, 0.242, 0.099. They sum to 1.

Two properties do the work. Exponentiating makes everything positive, which probabilities must be. And dividing by the total makes them sum to 1.

The exponential also exaggerates differences. A logit gap of 1.0 became a probability ratio of 2.7 to 1. A gap of 5 becomes 148 to 1. That sharpening is why a small change in the final layer can flip a prediction decisively, and why temperature scaling — dividing the logits by a constant before the softmax — is such an effective way to soften or sharpen a model's confidence.

The loss that goes with it

Cross-entropy measures how far the predicted distribution is from the truth. With a one-hot target the sum collapses to a single term:

loss = −log(probability assigned to the correct class)

The penalty grows without bound as the prediction approaches zero, so a confidently wrong model is punished far harder than an uncertain one. Squared error caps the penalty at 1 and gives confident mistakes only a gentle nudge, which is exactly why classification uses cross-entropy.

A useful sanity check at initialisation: with C balanced classes, the expected starting loss is ln(C). For 10 classes that is 2.30. Seeing that number in the first few steps means the model is initialised sensibly; seeing something wildly different suggests a bug in the labels or the loss.

Why the two are always paired

Softmax and cross-entropy are combined for a mathematical reason that matters in practice. The derivative of the pair, with respect to the logits, is startlingly simple:

∂L/∂zᵢ = pᵢ − yᵢ

Predicted probability minus the true label. Nothing else. No sigmoid derivative to shrink the gradient, no saturation term.

Compare with sigmoid plus squared error, where the gradient includes a(1 − a) — a factor that goes to zero exactly when the model is confidently wrong. That model learns slowest in the cases it most needs to fix. The softmax and cross-entropy pairing has no such term, and that is why it became the standard.

It also explains why frameworks fuse them. nn.CrossEntropyLoss applies log-softmax and the negative log-likelihood in one numerically stable step, and computes that clean gradient directly.

Why they are always implemented together

Softmax turns scores into probabilities and cross-entropy scores them. Computed separately both overflow; computed together the gradient collapses to one subtraction.

import numpy as np

logits = np.array([2.0, 1.0, 0.1, -1.0])
print("raw scores from the last layer:", logits)
print("softmax makes them a distribution -- positive, and summing to 1:")
e = np.exp(logits)
p = e / e.sum()
print("  exp(z)      ", np.round(e, 4))
print("  exp(z)/sum  ", np.round(p, 6), " sum %.10f" % p.sum())
print()

print("it is shift-invariant, which is the fact the implementation relies on:")
for shift in (0.0, 100.0, -50.0):
    z = logits + shift
    zz = z - z.max()
    q = np.exp(zz) / np.exp(zz).sum()
    print("  add %+7.1f to every score -> %s" % (shift, np.round(q, 6)))
print("  the same distribution every time. only DIFFERENCES between scores")
print("  matter, never their absolute size.")
print()

print("which is what stops it overflowing. the naive version:")
big = np.array([1000.0, 999.0, 998.0])
naive = np.exp(big) / np.exp(big).sum()
print("  exp([1000, 999, 998]) = %s  <- inf/inf" % np.exp(big))
print("  naive softmax          = %s" % naive)
stable = np.exp(big - big.max()) / np.exp(big - big.max()).sum()
print("  subtract the max first = %s" % np.round(stable, 6))
print("  every library does that subtraction. it changes nothing")
print("  mathematically and everything numerically.")
print()

print("cross-entropy over that distribution, per possible answer:")
for c in range(4):
    print("  true class %d -> loss = -log(%.6f) = %.6f" % (c, p[c], -np.log(p[c])))
print()

print("the same trap on the loss side. a confident wrong answer:")
z = np.array([400.0, -400.0])
q = np.exp(z - z.max()) / np.exp(z - z.max()).sum()
print("  softmax([400, -400]) = %s" % q)
print("  the second entry underflowed to exactly %s" % q[1])
print("  so -log of it is %s, and the gradient is meaningless"
      % ("inf" if q[1] == 0 else "%.4f" % -np.log(q[1])))
print("  combining the two steps avoids ever forming that number:")
print("  log_softmax(z)[1] = z[1] - logsumexp(z) = %.4f"
      % (z[1] - (z.max() + np.log(np.exp(z - z.max()).sum()))))
print("  that is why frameworks give you cross_entropy(LOGITS, target) and")
print("  warn you not to apply softmax yourself first.")
print()

print("and now the reason it is elegant rather than merely stable.")
print("the gradient of (softmax + cross-entropy) with respect to the logits:")
true_class = 0
onehot = np.zeros(4); onehot[true_class] = 1
analytic = p - onehot

def loss_of(z):
    zz = z - z.max()
    pp = np.exp(zz) / np.exp(zz).sum()
    return -np.log(pp[true_class])

h = 1e-6
numeric = np.array([(loss_of(logits + h * np.eye(4)[i])
                     - loss_of(logits - h * np.eye(4)[i])) / (2 * h)
                    for i in range(4)])
print("  analytic  (p - onehot):", np.round(analytic, 8))
print("  numeric               :", np.round(numeric, 8))
print("  max difference: %.2e" % np.abs(analytic - numeric).max())
print()
print("the whole derivative is 'predicted minus actual'. no softmax")
print("derivative, no chain rule term, no jacobian -- they cancel exactly.")
print("that cancellation is why this pairing is the default and why any")
print("other loss on top of a softmax is a worse deal than it looks.")

Guided tour

1. Watch the gap do the work. Set the Score for cat slider to 6. Its probability shoots up and every other class is squeezed toward zero. Softmax cares about the differences between logits, not their absolute size.
2. Prove shift invariance. Click Add 100 to Every Score. The logits all become enormous and the probabilities do not move at all. This is not a curiosity — it is what lets a real implementation subtract the maximum before exponentiating, which is the only thing standing between you and numerical overflow.
3. Punish a confident mistake. Set the True Label to fish, then set the Score for fish slider to -6. The model is now confidently wrong and the loss climbs steeply. Accuracy would record this as one error; cross-entropy records how badly.
4. Flatten it out. Set the Temperature slider to its maximum, 4. Dividing the logits by a large temperature shrinks the gaps between them, and the distribution tends toward uniform. This is exactly the temperature knob on a language model.
5. Sharpen it. Set the Temperature slider to 0.1. The gaps are magnified and the distribution collapses onto the single highest logit — effectively an argmax. Low temperature means safe and repetitive; high means varied and eventually incoherent.
6. Read the gradient. With the true label set, look at the gradient panel. The entry for the correct class is negative and the rest are positive: training pushes the right logit up and the others down, by an amount equal to how wrong the probability was.

Why they are paired

Take the derivative of cross-entropy with respect to the raw logits and almost everything cancels:

∂loss / ∂z_i = p_i − y_i

Predicted minus actual. No exponentials, no division, no chain of awkward terms — the softmax derivative and the log derivative annihilate each other exactly. It is numerically stable, it is one subtraction, and it is why every framework fuses the two into a single operation (CrossEntropyLoss in PyTorch takes raw logits, not probabilities).

That fusion is also a very common bug: applying softmax yourself and then passing the result to a loss that expects logits applies softmax twice, which flattens the distribution and quietly cripples training.

Where you meet it next

Softmax is not only an output layer. It is the normalising step inside attention: attention scores are logits, and softmax turns them into weights that sum to 1 so they can average the values. Every property above — monotonic, shift-invariant, temperature-controllable — applies there unchanged.

Where this goes wrong

• Applying softmax twice. As above. If your loss function is named for logits, give it logits.
• Using it for multi-label problems. Softmax forces the outputs to compete — they must sum to 1. If an example can belong to several classes at once, you want independent sigmoids and binary cross-entropy instead.
• Reading probabilities as calibrated confidence. Modern networks are systematically overconfident; a 0.99 often means rather less than 99%. Calibration is a separate step.
• Forgetting the max subtraction. Writing softmax by hand and exponentiating a logit of 1000 gives infinity, then NaN. Subtract the maximum first; it changes nothing mathematically and everything numerically.

Summing up

Softmax exponentiates the raw scores and divides by their total, turning any set of logits into a probability distribution — monotonic, so the winner never changes, and shift-invariant, which is what makes it numerically safe. Cross-entropy then reduces to minus the log of the probability given to the true class, so a confident mistake is punished without limit while a hedge is not. Paired, their gradient with respect to the logits collapses to predicted minus actual, which is why frameworks fuse them into one operation and why you should never apply softmax before a loss that expects logits.

Numerical stability, and the bug everyone hits

e^1000 is infinity in floating point, and once an infinity appears the result is NaN. Since logits are unbounded, this is a real risk.

The fix is to subtract the maximum logit before exponentiating. Because the constant cancels between numerator and denominator, the result is unchanged, but the largest exponent becomes e⁰ = 1:

import numpy as np

def softmax(z):
    z = z - np.max(z)         # shift: mathematically identical, numerically safe
    e = np.exp(z)
    return e / e.sum()

Every framework does this internally, which leads to the practical rule: pass logits to the loss function, never probabilities.

import torch.nn as nn

criterion = nn.CrossEntropyLoss()      # expects raw logits
logits = model(x)                      # no softmax in the model
loss = criterion(logits, targets)      # integer class indices, not one-hot

Applying softmax in the model and then using CrossEntropyLoss applies it twice. The model still trains — badly, slowly, with no error message. It is one of the most common bugs in deep learning code.

For binary problems, BCEWithLogitsLoss is the equivalent fused pair.

Variants worth knowing

Sigmoid instead of softmax for multi-label problems, where several classes can be true at once. Softmax forces the outputs to compete for a fixed total, which is wrong there.

Label smoothing replaces the one-hot target with, say, 0.9 on the true class and the remaining 0.1 spread across the others. The model is no longer pushed towards infinite confidence, which improves calibration and usually test accuracy.

Temperature scaling divides logits by T before the softmax. T > 1 softens the distribution; T < 1 sharpens it. Fitting a single T on held-out data is the standard, cheap way to calibrate a network's confidence, and it changes no predictions' ranking.

Focal loss down-weights examples the model already gets right, focusing training on the hard ones. Designed for object detection, where background vastly outnumbers objects.

Questions people ask

Why exponentiate rather than just normalise the logits? Because logits can be negative, and dividing negatives by a sum does not give probabilities. Exponentiating guarantees positivity and gives a smooth, differentiable function.

Should my model end with softmax? No — return logits and let the loss apply it. Apply softmax explicitly only at inference, when you want to display probabilities.

Are softmax outputs real probabilities? They are a distribution, and typically an overconfident one. Check with a reliability diagram and apply temperature scaling if the numbers matter.

What loss for two classes? Either a single sigmoid output with binary cross-entropy, or two logits with softmax. Both work; the sigmoid version has one fewer parameter.

Why is my loss exactly ln(number of classes) and not falling? The model is predicting uniformly and learning nothing — check the learning rate, the label alignment and whether gradients are reaching the parameters.

Does softmax have parameters? No. It is a fixed function applied to the final layer's outputs.

Recap in one screen

• Softmax exponentiates the logits and normalises, turning scores into a distribution that sums to 1.
• The exponential exaggerates differences, which is why temperature scaling is an effective confidence dial.
• Cross-entropy is −log(probability of the correct class), punishing confident mistakes without limit.
• Together their gradient is simply predicted − actual, with no saturating term.
• Subtract the maximum logit for stability, and always hand logits to the fused loss rather than probabilities.