How Loss is Calculated

What a loss function has to do

A loss collapses a whole batch of predictions into one scalar. And it has to do so in a way that is differentiable. Otherwise there is nothing to descend. Which function you pick depends on what you are predicting.

• Regression — mean squared error: MSE = mean((y − ŷ)²)
• Sorting — cross-entropy: −Σ y log(ŷ)

MSE penalises large errors disproportionately since of the squaring: an error of 10 adds 100, while two errors of 5 add just 50 mixed. Cross-entropy penalises confident wrong predictions most: predicting 0.01 for a true class costs far more than predicting 0.4. Both are differentiable all over that matters. It is the non-negotiable requirement. Score (the fraction of correct predictions) is the count people care about. But its gradient is zero almost all over. So it cannot be used for training. The loss is computed per sample, then averaged over the batch. That average is the single scalar that backpropagation differentiates with respect to each param. For regression, MAE and Huber loss are options when outlier robustness matters. For sorting with more than two classes, cross-entropy transfers naturally: the sum runs over all classes. And just the true class adds (the rest have y=0). This page computes both losses on a small case, shows how each gradient changes as the prediction moves. And covers the relationship between cross-entropy and the softmax output.

Both, with real counts

MSE. Predictions [2.5, 0.0, 2.0] against targets [3.0, −0.5, 2.0]. The errors are 0.5, 0.5 and 0. Squared: 0.25, 0.25, 0. Mean: 0.167. Squaring means one error of 2.0 costs more than four errors of 0.5.Cross-entropy. If the net gives the correct class a chance of 0.7, the loss is −ln(0.7) = 0.357. If it gives that class just 0.05, the loss is −ln(0.05) = 3.00. Roughly eight times worse. Being confidently wrong is punished far harder than being uncertain.

One count that says how wrong the net is

Training needs a single scalar to minimise. The loss function is what gives it: check each prediction with its target, score the discrepancy. And average over the batch.

All downstream depends on that one count. The gradient is its derivative, the solver follows the gradient. And the net turns whatever cuts it. Choosing the loss is therefore choosing what the net will tune for. A modelling decision, not a technical detail.

The standard choices:

Worked through, both kinds

Regression. Predictions [2.5, 0.0, 2.1] against targets [3.0, −0.5, 2.0]:

• Errors: −0.5, 0.5, 0.1
• Squared: 0.25, 0.25, 0.01 → MSE = 0.51/3 = 0.17
• Absolute: 0.5, 0.5, 0.1 → MAE = 1.1/3 = 0.367

Sorting. Three samples, true classes 0, 1, 2. And the net gives the correct class chances 0.7, 0.2, 0.9:

• Losses: −log(0.7) = 0.357, −log(0.2) = 1.609, −log(0.9) = 0.105
• Mean = 0.690

The second sample adds most of the total. And that is deliberate. Cross-entropy punishes confident mistakes without limit. So the net's attention goes where it is most wrong.

A useful check: at setup, cross-entropy over C balanced classes should be about ln(C). 2.30 for 10 classes. Seeing that in the first steps means the setup is sane.

Choosing between squared and absolute error

Both measure regression error and they encode other opinions about outliers.

Squaring makes a single large miss dominate: an error of 10 adds 100, while twenty errors of 1 add 20 in total. The fitted net will distort it to reduce that one point.

Absolute error treats each unit of error equally. So it is reliable — and it has a corner at zero. It makes its gradient constant in magnitude no matter how close you are.

Huber loss is the standard compromise: squared within a threshold δ of zero, linear beyond it. Smooth gradients near the optimum, bounded influence from outliers. It is the default regression loss in a few detection frameworks for just this reason.

There is a deeper reading. Cutting squared error estimates the conditional mean. Cutting absolute error estimates the conditional median. So the choice of loss is a choice about which summary of the target spread you want predicted.

Try this above

1. Raise Batch Size and watch the reported loss steady. Averaging over more samples reduces the sample-to-sample swing without changing what is being measured.
2. Increase Dropout and note training loss climbing. That is expected — you are deliberately handicapping the network in training.
3. Add hidden capacity with Hidden and watch how quickly loss falls across Epochs.

What often goes wrong

Checking loss vals that are not comparable. A loss summed over a batch and a loss averaged over a batch differ by a factor of the batch size. Two runs with other batch sizes or other reduction settings are not measuring the same thing. And neither is a run that changed loss function mid-test.MSE on a sorter. It technically works and trains badly. When the net is confidently wrong, cross-entropy gives a large gradient and MSE gives a small one. So the net is slowest to fix just the errors that matter most.Unscaled regression targets. Squared error on vals around 300,000 gives enormous gradients. Scale the target. Or the first update will destroy the weights.

In one line

The loss is the just thing the network is tuning. If it does not measure what you care about, nothing downstream can fix that.

Reduction, weighting and masking

Three details of how a batch's losses are mixed matter in practice.

Reduction. mean averages over the batch and is the default. sum totals them, which makes the effective learning rate depend on batch size. none returns per-case losses. It is what you need for custom weighting or for finding the hardest cases.

Class weighting. On imbalanced data, weighting each class's contribution by the inverse of its frequency stops the majority class dominating the average:

weights = torch.tensor([1.0, 12.0])          # positive class is 12x rarer
criterion = nn.CrossEntropyLoss(weight=weights)

Masking. Padded positions in a sequence. And pixels marked "unlabelled" in a segmentation mask, must be excluded. ignore_index does this:

criterion = nn.CrossEntropyLoss(ignore_index=-100)     # skip these positions

Forgetting to mask is a real and quiet bug: the net spends a large share of its capacity learning to predict padding tokens. And the reported loss is not comparable to anyone else's.

Training loss versus the metric you report

These are other things and should be let to differ.

The loss must be differentiable so gradients exist. Score, F1, IoU and mAP are not differentiable. They are step functions of the predictions, with zero gradient almost all over. You cannot tune them right.

So the standard arrangement is: train on a smooth surrogate, test on the metric you care about. Train on cross-entropy, report F1. Train on Dice-plus-cross-entropy, report IoU.

The two can disagree, and that is informative rather than alarming. Validation loss rising while score stays flat often means the net is turning too sure on the cases it now gets right. A tuning problem rather than a sorting one. Early stopping on the metric you care about, rather than on the loss, is often the better choice.

Some metrics have differentiable relaxations — soft-Dice, soft-IoU. And the CIoU family for bounding boxes. And with them closes part of the gap between what you tune and what you measure.

Each loss, computed by hand

MSE, MAE, Huber, binary and categorical cross-entropy on the same predictions -- so you can see what each one punishes and how hard.

import numpy as np

truth = np.array([3.0, 3.0, 3.0, 3.0])
preds = np.array([3.1, 3.5, 4.0, 8.0])
print("regression losses. truth is 3.0 in every row:")
print("%12s %10s %10s %10s %10s" % ("prediction", "error", "squared", "absolute", "huber"))
def huber(e, d=1.0):
    return np.where(np.abs(e) <= d, 0.5 * e ** 2, d * (np.abs(e) - 0.5 * d))
for p in preds:
    e = p - 3.0
    print("%12.1f %10.1f %10.4f %10.4f %10.4f" % (p, e, e ** 2, abs(e), huber(e)))
print()
e = preds - truth
print("  MSE   %.4f   -- the 8.0 row alone contributes %.1f%% of it"
      % ((e ** 2).mean(), 100 * e[-1] ** 2 / (e ** 2).sum()))
print("  MAE   %.4f   -- that row contributes %.1f%%"
      % (np.abs(e).mean(), 100 * abs(e[-1]) / np.abs(e).sum()))
print("  Huber %.4f   -- squared near zero, linear far out: %.1f%%"
      % (huber(e).mean(), 100 * huber(e)[-1] / huber(e).sum()))
print()
print("that is the entire choice. MSE says one bad miss is worse than many")
print("small ones. MAE says they are the same. Huber says the first thing")
print("near the middle and the second thing in the tail.")
print()

print("binary cross-entropy -- the loss for a yes/no answer:")
print("%10s %10s %12s" % ("said", "truth", "loss"))
for p, t in ((0.99, 1), (0.90, 1), (0.60, 1), (0.50, 1), (0.10, 1), (0.01, 1)):
    print("%10.2f %10d %12.4f" % (p, t, -np.log(p)))
print("  the loss for being right is small but never zero. the loss for being")
print("  confidently wrong has no ceiling -- at p=0.001 it is %.4f."
      % -np.log(0.001))
print()
print("  it is symmetric, so predicting 0.01 when the answer is 0 is cheap:")
for p, t in ((0.01, 0), (0.50, 0), (0.99, 0)):
    print("%10.2f %10d %12.4f" % (p, t, -np.log(1 - p)))
print()

Building on the same setup, the next block shows the left cases.

import numpy as np

print("categorical cross-entropy over 4 classes -- only the true class counts:")
logits = np.array([2.0, 1.0, 0.1, -1.0])
probs = np.exp(logits - logits.max()); probs /= probs.sum()
print("  logits      ", np.round(logits, 2))
print("  softmax     ", np.round(probs, 4), " sums to", round(probs.sum(), 10))
for true_class in range(4):
    onehot = np.zeros(4); onehot[true_class] = 1
    print("    if the answer is class %d: loss = -log(%.4f) = %.4f"
          % (true_class, probs[true_class], -np.log(probs[true_class])))
print("  the full formula sums over all classes, but the one-hot target")
print("  zeroes every term except the true one. that is why implementations")
print("  just index into the probability vector.")
print()

print("why the loss must be differentiable -- accuracy is not:")
print("%10s %12s %12s" % ("nudge", "accuracy", "cross-entropy"))
p = np.array([0.30, 0.40, 0.45, 0.48])
t = np.array([1, 1, 1, 1])
for shift in (0.00, 0.01, 0.02, 0.03, 0.04, 0.05, 0.10):
    q = np.clip(p + shift, 1e-9, 1 - 1e-9)
    acc = ((q > 0.5).astype(int) == t).mean()
    ce = -(t * np.log(q) + (1 - t) * np.log(1 - q)).mean()
    print("%10s %12.4f %12.4f" % ("+%.2f" % shift, acc, ce))
print()
print("every nudge makes the model measurably better, and accuracy notices")
print("exactly twice -- when a prediction crosses 0.50. in between it is")
print("perfectly flat, which means its gradient is zero, which means gradient")
print("descent has nothing to follow.")
print()
print("cross-entropy moves on every single nudge. that is not a nicety, it is")
print("the requirement: you optimise a loss and you REPORT a metric, and they")
print("are usually not the same function.")

Questions people ask

Why is my loss NaN? A log(0), a division by zero. Or a learning rate high enough to send the weights to infinity. Use the framework's fused losses, which guard the logarithm.

Can I add a few losses together? Yes. And it is common in multi-task nets. Weight them, and be aware the weights matter: a term ten times more dominates training no matter weight.

Should the loss go to zero? Not on real data. Irreducible noise means there is a floor. And reaching zero on training data often means memorisation.

Why does my validation loss rise while score holds? Growing overconfidence. Consider label smoothing. And stop on the metric rather than the loss.

How do I handle imbalanced classes? Class weights. Or focal loss, which down-weights the easy cases the majority class gives.

Do I apply softmax before the loss? No — pass logits. The fused loss applies it internally and stably. Applying it twice is a common silent bug.

Recap in one screen

• The loss reduces the whole batch to one count. And the net turns whatever cuts it.
• MSE for regression, cross-entropy for sorting. Huber when outliers are present.
• Squared error predicts the conditional mean. Absolute error predicts the median.
• Mask padded positions and weight rare classes. Or the average is measuring the wrong thing.
• Train on a differentiable surrogate, test on the metric that matters. And expect them to disagree at times.