Softmax and Cross-Entropy

Quick Context

A network's final layer produces one raw number per class. These are called logits, and they can be anything: 8.2, −3.1, 0.0. They are not probabilities — they do not sit between 0 and 1 and they do not add up to anything in particular.

Softmax converts them into a probability distribution. Cross-entropy then measures how wrong that distribution is. The two are almost always used together, and the reason is more interesting than convention.

What softmax does

p_i = e^z_i / Σ_j e^z_j

Exponentiate every score, then divide by the total. Exponentiating makes everything positive; dividing by the sum makes it add to 1. Those two steps are the whole function.

Two properties are worth holding on to. It is monotonic: the largest logit always becomes the largest probability, so softmax never changes which class wins. And it is shift-invariant: adding the same constant to every logit leaves the probabilities completely unchanged, because the constant factors out of the numerator and denominator together.

What cross-entropy does

For a single correct label, the loss collapses to one term:

loss = −log( p_true )

Every other class drops out, because the true distribution is 1 for the right answer and 0 everywhere else. So the loss depends on one number: the probability the model gave to the correct class.

Look at the curve. At p = 1 the loss is 0. At p = 0.5 it is 0.69. At p = 0.1 it is 2.3. As p approaches 0 it goes to infinity. That asymptote is the point: a model that is confidently wrong is punished without limit, while one that hedges is only mildly penalised.

Interactive Exploration Guide

1. Watch the gap do the work. Set the Score for cat slider to 6. Its probability shoots up and every other class is squeezed toward zero. Softmax cares about the differences between logits, not their absolute size.Run this
2. Prove shift invariance. Click Add 100 to Every Score. The logits all become enormous and the probabilities do not move at all. This is not a curiosity — it is what lets a real implementation subtract the maximum before exponentiating, which is the only thing standing between you and numerical overflow.Run this
3. Punish a confident mistake. Set the True Label to fish, then set the Score for fish slider to -6. The model is now confidently wrong and the loss climbs steeply. Accuracy would record this as one error; cross-entropy records how badly.Run this
4. Flatten it out. Set the Temperature slider to its maximum, 4. Dividing the logits by a large temperature shrinks the gaps between them, and the distribution tends toward uniform. This is exactly the temperature knob on a language model.Run this
5. Sharpen it. Set the Temperature slider to 0.1. The gaps are magnified and the distribution collapses onto the single highest logit — effectively an argmax. Low temperature means safe and repetitive; high means varied and eventually incoherent.Run this
6. Read the gradient. With the true label set, look at the gradient panel. The entry for the correct class is negative and the rest are positive: training pushes the right logit up and the others down, by an amount equal to how wrong the probability was.

Why they are paired

Take the derivative of cross-entropy with respect to the raw logits and almost everything cancels:

∂loss / ∂z_i = p_i − y_i

Predicted minus actual. No exponentials, no division, no chain of awkward terms — the softmax derivative and the log derivative annihilate each other exactly. It is numerically stable, it is one subtraction, and it is why every framework fuses the two into a single operation (CrossEntropyLoss in PyTorch takes raw logits, not probabilities).

That fusion is also a very common bug: applying softmax yourself and then passing the result to a loss that expects logits applies softmax twice, which flattens the distribution and quietly cripples training.

Where you meet it next

Softmax is not only an output layer. It is the normalising step inside attention: attention scores are logits, and softmax turns them into weights that sum to 1 so they can average the values. Every property above — monotonic, shift-invariant, temperature-controllable — applies there unchanged.

What usually goes wrong

• Applying softmax twice. As above. If your loss function is named for logits, give it logits.
• Using it for multi-label problems. Softmax forces the outputs to compete — they must sum to 1. If an example can belong to several classes at once, you want independent sigmoids and binary cross-entropy instead.
• Reading probabilities as calibrated confidence. Modern networks are systematically overconfident; a 0.99 often means rather less than 99%. Calibration is a separate step.
• Forgetting the max subtraction. Writing softmax by hand and exponentiating a logit of 1000 gives infinity, then NaN. Subtract the maximum first; it changes nothing mathematically and everything numerically.

Key Takeaway

Softmax exponentiates the raw scores and divides by their total, turning any set of logits into a probability distribution — monotonic, so the winner never changes, and shift-invariant, which is what makes it numerically safe. Cross-entropy then reduces to minus the log of the probability given to the true class, so a confident mistake is punished without limit while a hedge is not. Paired, their gradient with respect to the logits collapses to predicted minus actual, which is why frameworks fuse them into one operation and why you should never apply softmax before a loss that expects logits.