Softmax and Cross-Entropy

Quick Context

A network's final layer gives one raw count per class. These are called logits. And they can be anything: 8.2, −3.1, 0.0. They are not chances — they do not sit between 0 and 1 and they do not add up to anything in particular.

Softmax turns them into a chance spread. Cross-entropy then measures how wrong that spread is. The two are almost always used together. And the reason is more interesting than convention.

Softmax exponentiates each logit and divides by the sum of all the exponentiated logits. The result is a vector of positive counts that sum to 1. A valid chance spread. The largest logit always gets the largest chance. So softmax never changes which class wins. It just tells you how confident the net is.

Cross-entropy measures the gap between two chance spreads: the net's prediction and the true label. For a single correct class (the usual case), the loss simplifies to the negative log of the chance given to the correct class. If the net is confident and right, that chance is near 1 and the loss is near 0. If the net is confident and wrong, the chance is near 0 and the loss explodes toward infinity.

The gradient of the mixed softmax-cross-entropy loss has a remarkably clean form: for each class, the gradient is simply the predicted chance minus the target (1 for the correct class, 0 for all else). This means the update pushes the correct class's chance up and each other class's chance down, by an amount proportional to the current error. No second derivatives, no complicated chain rule. Which is one reason this combination became the universal default for sorting.

In practice, frameworks compute softmax and cross-entropy together in a single fused step (PyTorch's CrossEntropyLoss takes raw logits, not chances) since the fused version is numerically stable. Computing softmax apart and then taking the log can give NaN or Inf when logits are large. The fused version subtracts the max logit first. It cancels out in math but stops overflow.

What softmax does

p_i = e^z_i / Σ_j e^z_j

Exponentiate each score, then divide by the total. Exponentiating makes all positive. Dividing by the sum makes it add to 1. Those two steps are the whole function.

Two properties are worth holding on to. It is monotonic: the largest logit always turns the largest chance. So softmax never changes which class wins. And it is shift-invariant: adding the same constant to each logit leaves the chances fully unchanged, since the constant factors out of the numerator and denominator together.

What cross-entropy does

For a single correct label, the loss collapses to one term:

loss = −log( p_true )

Each other class drops out, since the true spread is 1 for the right answer and 0 all over else. So the loss depends on one count: the chance the net gave to the correct class.

Look at the curve. At p = 1 the loss is 0. At p = 0.5 it is 0.69. At p = 0.1 it is 2.3. As p ways 0 it goes to infinity. That asymptote is the point: a net that is confidently wrong is punished without limit, while one that hedges is just mildly penalised.

From scores to chances

A sorter's final layer outputs one unbounded count per class — a logit. Softmax turns those into chances:

pᵢ = e^zᵢ / Σ_j e^zᵎ

Worked through on logits [2.0, 1.0, 0.1]:

• Exponentiate: e² = 7.39, e¹ = 2.72, e⁰·¹ = 1.11. Sum = 11.22.
• Divide: 0.659, 0.242, 0.099. They sum to 1.

Two properties do the work. Exponentiating makes all positive, which chances must be. And dividing by the total makes them sum to 1.

The exponential also exaggerates gaps. A logit gap of 1.0 became a chance ratio of 2.7 to 1. A gap of 5 turns 148 to 1. That sharpening is why a small change in the final layer can flip a prediction decisively. And why temperature scaling — dividing the logits by a constant before the softmax. Is such an effective way to soften or sharpen a net's confidence.

The loss that goes with it

Cross-entropy measures how far the predicted spread is from the truth. With a one-hot target the sum collapses to a single term:

loss = −log(chance given to the correct class)

The cost grows without bound as the prediction ways zero. So a confidently wrong net is punished far harder than an uncertain one. Squared error caps the cost at 1 and gives confident mistakes just a gentle nudge. It is just why sorting uses cross-entropy.

A useful sanity check at setup: with C balanced classes, the expected starting loss is ln(C). For 10 classes that is 2.30. Seeing that count in the first few steps means the net is set up sensibly. Seeing something wildly other suggests a bug in the labels or the loss.

Why the two are always paired

Softmax and cross-entropy are mixed for a math-based reason that matters in practice. The derivative of the pair, with respect to the logits, is startlingly simple:

∂L/∂zᵢ = pᵢ − yᵢ

Predicted chance minus the true label. Nothing else. No sigmoid derivative to shrink the gradient, no saturation term.

Check with sigmoid plus squared error. There the gradient includes a(1 − a). A factor that goes to zero just when the net is confidently wrong. That net learns slowest in the cases it most needs to fix. The softmax and cross-entropy pairing has no such term. And that is why it became the standard.

It also explains why frameworks fuse them. nn.CrossEntropyLoss applies log-softmax and the negative log-likelihood in one numerically stable step. And computes that clean gradient right.

Why they are always built together

Softmax turns scores into chances and cross-entropy scores them. Computed apart both overflow. Computed together the gradient collapses to one subtraction.

import numpy as np

logits = np.array([2.0, 1.0, 0.1, -1.0])
print("raw scores from the last layer:", logits)
print("softmax makes them a distribution -- positive, and summing to 1:")
e = np.exp(logits)
p = e / e.sum()
print("  exp(z)      ", np.round(e, 4))
print("  exp(z)/sum  ", np.round(p, 6), " sum %.10f" % p.sum())
print()

print("it is shift-invariant, which is the fact the implementation relies on:")
for shift in (0.0, 100.0, -50.0):
    z = logits + shift
    zz = z - z.max()
    q = np.exp(zz) / np.exp(zz).sum()
    print("  add %+7.1f to every score -> %s" % (shift, np.round(q, 6)))
print("  the same distribution every time. only DIFFERENCES between scores")
print("  matter, never their absolute size.")
print()

print("which is what stops it overflowing. the naive version:")
big = np.array([1000.0, 999.0, 998.0])
naive = np.exp(big) / np.exp(big).sum()
print("  exp([1000, 999, 998]) = %s  <- inf/inf" % np.exp(big))
print("  naive softmax          = %s" % naive)
stable = np.exp(big - big.max()) / np.exp(big - big.max()).sum()
print("  subtract the max first = %s" % np.round(stable, 6))
print("  every library does that subtraction. it changes nothing")
print("  mathematically and everything numerically.")

Cross-entropy measures how far the predicted spread lands from the true label. And the mixed loss has the same overflow trap that softmax does on its own.

import numpy as np

logits = np.array([2.0, 1.0, 0.1, -1.0])
e = np.exp(logits - logits.max())
p = e / e.sum()

print("cross-entropy over that distribution, per possible answer:")
for c in range(4):
    print("  true class %d -> loss = -log(%.6f) = %.6f" % (c, p[c], -np.log(p[c])))
print()

print("the same trap on the loss side. a confident wrong answer:")
z = np.array([400.0, -400.0])
q = np.exp(z - z.max()) / np.exp(z - z.max()).sum()
print("  softmax([400, -400]) = %s" % q)
print("  the second entry underflowed to exactly %s" % q[1])
print("  so -log of it is %s, and the gradient is meaningless"
      % ("inf" if q[1] == 0 else "%.4f" % -np.log(q[1])))
print("  combining the two steps avoids ever forming that number:")
print("  log_softmax(z)[1] = z[1] - logsumexp(z) = %.4f"
      % (z[1] - (z.max() + np.log(np.exp(z - z.max()).sum()))))
print("  that is why frameworks give you cross_entropy(LOGITS, target) and")
print("  warn you not to apply softmax yourself first.")

The gradient of the mixed softmax-plus-cross-entropy simplifies to a single subtraction. It a numerical check confirms.

import numpy as np

logits = np.array([2.0, 1.0, 0.1, -1.0])
e = np.exp(logits - logits.max())
p = e / e.sum()

true_class = 0
onehot = np.zeros(4); onehot[true_class] = 1
analytic = p - onehot

def loss_of(z):
    zz = z - z.max()
    pp = np.exp(zz) / np.exp(zz).sum()
    return -np.log(pp[true_class])

h = 1e-6
numeric = np.array([(loss_of(logits + h * np.eye(4)[i])
                     - loss_of(logits - h * np.eye(4)[i])) / (2 * h)
                    for i in range(4)])
print("analytic  (p - onehot):", np.round(analytic, 8))
print("numeric               :", np.round(numeric, 8))
print("max difference: %.2e" % np.abs(analytic - numeric).max())
print()
print("the whole derivative is 'predicted minus actual'. no softmax")
print("derivative, no chain rule term, no jacobian -- they cancel exactly.")
print("that cancellation is why this pairing is the default and why any")
print("other loss on top of a softmax is a worse deal than it looks.")

Guided tour

1. Watch the gap do the work. Set the Score for cat slider to 6. Its chance shoots up and each other class is squeezed toward zero. Softmax cares about the gaps between logits, not their absolute size.
2. Prove shift invariance. Click Add 100 to Each Score. The logits all turn enormous and the chances do not move at all. This is not a curiosity. It is what lets a real build subtract the max before exponentiating. It is the just thing standing between you and numerical overflow.
3. Punish a confident mistake. Set the True Label to fish, then set the Score for fish slider to -6. The net is now confidently wrong and the loss climbs steeply. Score would record this as one error. Cross-entropy records how badly.
4. Flatten it out. Set the Temperature slider to its max, 4. Dividing the logits by a large temperature shrinks the gaps between them. And the spread tends toward uniform. This is just the temperature knob on a language net.
5. Sharpen it. Set the Temperature slider to 0.1. The gaps are magnified and the spread collapses onto the single highest logit — effectively an argmax. Low temperature means safe and repetitive. High means varied and eventually incoherent.
6. Read the gradient. With the true label set, look at the gradient panel. The entry for the correct class is negative and the rest are positive: training pushes the right logit up and the others down, by an amount equal to how wrong the chance was.

Why they are paired

Take the derivative of cross-entropy with respect to the raw logits and almost all cancels:

∂loss / ∂z_i = p_i − y_i

Predicted minus actual. No exponentials, no division, no chain of awkward terms. The softmax derivative and the log derivative annihilate each other just. It is numerically stable, it is one subtraction. And it is why each framework fuses the two into a single step (CrossEntropyLoss in PyTorch takes raw logits, not chances).

That fusion is also a very common bug: applying softmax yourself and then passing the result to a loss that expects logits applies softmax twice. It flattens the spread and quietly cripples training.

Where you meet it next

Softmax is not just an output layer. It is the scaling step inside attention: attention scores are logits. And softmax turns them into weights that sum to 1 so they can average the vals. Each property above — monotonic, shift-invariant, temperature-controllable — applies there unchanged.

Where this goes wrong

• Applying softmax twice. As above. If your loss function is named for logits, give it logits.
• With it for multi-label problems. Softmax forces the outputs to compete. They must sum to 1. If an case can belong to a few classes at once, you want free sigmoids and binary cross-entropy instead.
• Reading chances as calibrated confidence. Modern networks are step by step too sure. A 0.99 often means rather less than 99%. Tuning is a split step.
• Forgetting the max subtraction. Writing softmax by hand and exponentiating a logit of 1000 gives infinity, then NaN. Subtract the max first. It changes nothing in math and all numerically.

Summing up

Softmax exponentiates the raw scores and divides by their total, turning any set of logits into a chance spread — monotonic. So the winner never changes. And shift-invariant, which is what makes it numerically safe. Cross-entropy then reduces to minus the log of the chance given to the true class. So a confident mistake is punished without limit while a hedge is not. Paired, their gradient with respect to the logits collapses to predicted minus actual. It is why frameworks fuse them into one step and why you should never apply softmax before a loss that expects logits.

Numerical stability, and the bug everyone hits

e^1000 is infinity in floating point. And once an infinity appears the result is NaN. Since logits are unbounded, this is a real risk.

The fix is to subtract the max logit before exponentiating. Since the constant cancels between numerator and denominator, the result is unchanged. But the largest exponent turns e⁰ = 1:

import numpy as np

def softmax(z):
    z = z - np.max(z)         # shift: mathematically identical, numerically safe
    e = np.exp(z)
    return e / e.sum()

Each framework does this internally. It leads to the real rule: pass logits to the loss function, never chances.

import torch.nn as nn

criterion = nn.CrossEntropyLoss()      # expects raw logits
logits = model(x)                      # no softmax in the model
loss = criterion(logits, targets)      # integer class indices, not one-hot

Applying softmax in the net and then with CrossEntropyLoss applies it twice. The net still trains — badly, slowly, with no error message. It is one of the most common bugs in deep learning code.

For binary problems, BCEWithLogitsLoss is the equivalent fused pair.

Variants worth knowing

Sigmoid instead of softmax for multi-label problems. There a few classes can be true at once. Softmax forces the outputs to compete for a fixed total. It is wrong there.

Label smoothing replaces the one-hot target with, say, 0.9 on the true class and the left 0.1 spread across the others. The net is no longer pushed towards infinite confidence. It improves tuning and often test score.

Temperature scaling divides logits by T before the softmax. T > 1 softens the spread. T < 1 sharpens it. Fitting a single T on held-out data is the standard, cheap way to calibrate a network's confidence. And it changes no predictions' ranking.

Focal loss down-weights cases the net now gets right, focusing training on the hard ones. Designed for object detection, where background vastly outnumbers objects.

Questions people ask

Why exponentiate rather than just scale the logits? Since logits can be negative. And dividing negatives by a sum does not give chances. Exponentiating guarantees positivity and gives a smooth, differentiable function.

Should my net end with softmax? No. Give logits and let the loss apply it. Apply softmax clearly just at test time, when you want to display chances.

Are softmax outputs real chances? They are a spread. And often an too sure one. Check with a reliability diagram and apply temperature scaling if the counts matter.

What loss for two classes? Either a single sigmoid output with binary cross-entropy. Or two logits with softmax. Both work; the sigmoid version has one fewer param.

Why is my loss just ln(count of classes) and not falling? The net is predicting uniformly and learning nothing. Check the learning rate, the label alignment and whether gradients are reaching the params.

Does softmax have params? No. It is a fixed function applied to the final layer's outputs.

Recap in one screen

• Softmax exponentiates the logits and scales, turning scores into a spread that sums to 1.
• The exponential exaggerates gaps. It is why temperature scaling is an effective confidence dial.
• Cross-entropy is −log(chance of the correct class), punishing confident mistakes without limit.
• Together their gradient is simply predicted − actual, with no saturating term.
• Subtract the max logit for stability. And always hand logits to the fused loss rather than chances.