How Loss is Calculated

Both, with real numbers

MSE. Predictions [2.5, 0.0, 2.0] against targets [3.0, −0.5, 2.0]. The errors are 0.5, 0.5 and 0. Squared: 0.25, 0.25, 0. Mean: 0.167. Squaring means one error of 2.0 costs more than four errors of 0.5.Cross-entropy. If the model gives the correct class a probability of 0.7, the loss is −ln(0.7) = 0.357. If it gives that class only 0.05, the loss is −ln(0.05) = 3.00 — roughly eight times worse. Being confidently wrong is punished far harder than being uncertain.

One number that says how wrong the model is

Training needs a single scalar to minimise. The loss function is what produces it: compare each prediction with its target, score the discrepancy, and average over the batch.

Everything downstream depends on that one number. The gradient is its derivative, the optimiser follows the gradient, and the model becomes whatever minimises it. Choosing the loss is therefore choosing what the model will optimise for — a modelling decision, not a technical detail.

The standard choices:

Worked through, both kinds

Regression. Predictions [2.5, 0.0, 2.1] against targets [3.0, −0.5, 2.0]:

• Errors: −0.5, 0.5, 0.1
• Squared: 0.25, 0.25, 0.01 → MSE = 0.51/3 = 0.17
• Absolute: 0.5, 0.5, 0.1 → MAE = 1.1/3 = 0.367

Classification. Three samples, true classes 0, 1, 2, and the model gives the correct class probabilities 0.7, 0.2, 0.9:

• Losses: −log(0.7) = 0.357, −log(0.2) = 1.609, −log(0.9) = 0.105
• Mean = 0.690

The second sample contributes most of the total, and that is deliberate. Cross-entropy punishes confident mistakes without limit, so the model's attention goes where it is most wrong.

A useful check: at initialisation, cross-entropy over C balanced classes should be about ln(C) — 2.30 for 10 classes. Seeing that in the first steps means the setup is sane.

Choosing between squared and absolute error

Both measure regression error and they encode different opinions about outliers.

Squaring makes a single large miss dominate: an error of 10 contributes 100, while twenty errors of 1 contribute 20 in total. The fitted model will distort itself to reduce that one point.

Absolute error treats every unit of error equally, so it is robust — and it has a corner at zero, which makes its gradient constant in magnitude regardless of how close you are.

Huber loss is the standard compromise: squared within a threshold δ of zero, linear beyond it. Smooth gradients near the optimum, bounded influence from outliers. It is the default regression loss in several detection frameworks for exactly this reason.

There is a deeper reading. Minimising squared error estimates the conditional mean; minimising absolute error estimates the conditional median. So the choice of loss is a choice about which summary of the target distribution you want predicted.

Try this above

1. Raise Batch Size and watch the reported loss steady. Averaging over more samples reduces the sample-to-sample swing without changing what is being measured.
2. Increase Dropout and note training loss climbing. That is expected — you are deliberately handicapping the network during training.
3. Add hidden capacity with Hidden and watch how quickly loss falls across Epochs.

What usually goes wrong

Comparing loss values that are not comparable. A loss summed over a batch and a loss averaged over a batch differ by a factor of the batch size. Two runs with different batch sizes or different reduction settings are not measuring the same thing, and neither is a run that changed loss function mid-experiment.MSE on a classifier. It technically works and trains badly. When the model is confidently wrong, cross-entropy produces a large gradient and MSE produces a small one, so the model is slowest to fix exactly the errors that matter most.Unscaled regression targets. Squared error on values around 300,000 produces enormous gradients. Scale the target, or the first update will destroy the weights.

In one line

The loss is the only thing the network is optimising — if it does not measure what you care about, nothing downstream can fix that.

Reduction, weighting and masking

Three details of how a batch's losses are combined matter in practice.

Reduction. mean averages over the batch and is the default; sum totals them, which makes the effective learning rate depend on batch size; none returns per-example losses, which is what you need for custom weighting or for finding the hardest examples.

Class weighting. On imbalanced data, weighting each class's contribution by the inverse of its frequency stops the majority class dominating the average:

weights = torch.tensor([1.0, 12.0])          # positive class is 12x rarer
criterion = nn.CrossEntropyLoss(weight=weights)

Masking. Padded positions in a sequence, and pixels marked "unlabelled" in a segmentation mask, must be excluded. ignore_index does this:

criterion = nn.CrossEntropyLoss(ignore_index=-100)     # skip these positions

Forgetting to mask is a real and quiet bug: the model spends a large share of its capacity learning to predict padding tokens, and the reported loss is not comparable to anyone else's.

Training loss versus the metric you report

These are different things and should be allowed to differ.

The loss must be differentiable so gradients exist. Accuracy, F1, IoU and mAP are not differentiable — they are step functions of the predictions, with zero gradient almost everywhere. You cannot optimise them directly.

So the standard arrangement is: train on a smooth surrogate, evaluate on the metric you care about. Train on cross-entropy, report F1. Train on Dice-plus-cross-entropy, report IoU.

The two can disagree, and that is informative rather than alarming. Validation loss rising while accuracy stays flat usually means the model is becoming overconfident on the examples it already gets right — a calibration problem rather than a classification one. Early stopping on the metric you care about, rather than on the loss, is often the better choice.

Some metrics have differentiable relaxations — soft-Dice, soft-IoU, and the CIoU family for bounding boxes — and using them closes part of the gap between what you optimise and what you measure.

Every loss, computed by hand

MSE, MAE, Huber, binary and categorical cross-entropy on the same predictions -- so you can see what each one punishes and how hard.

import numpy as np

truth = np.array([3.0, 3.0, 3.0, 3.0])
preds = np.array([3.1, 3.5, 4.0, 8.0])
print("regression losses. truth is 3.0 in every row:")
print("%12s %10s %10s %10s %10s" % ("prediction", "error", "squared", "absolute", "huber"))
def huber(e, d=1.0):
    return np.where(np.abs(e) <= d, 0.5 * e ** 2, d * (np.abs(e) - 0.5 * d))
for p in preds:
    e = p - 3.0
    print("%12.1f %10.1f %10.4f %10.4f %10.4f" % (p, e, e ** 2, abs(e), huber(e)))
print()
e = preds - truth
print("  MSE   %.4f   -- the 8.0 row alone contributes %.1f%% of it"
      % ((e ** 2).mean(), 100 * e[-1] ** 2 / (e ** 2).sum()))
print("  MAE   %.4f   -- that row contributes %.1f%%"
      % (np.abs(e).mean(), 100 * abs(e[-1]) / np.abs(e).sum()))
print("  Huber %.4f   -- squared near zero, linear far out: %.1f%%"
      % (huber(e).mean(), 100 * huber(e)[-1] / huber(e).sum()))
print()
print("that is the entire choice. MSE says one bad miss is worse than many")
print("small ones. MAE says they are the same. Huber says the first thing")
print("near the middle and the second thing in the tail.")
print()

print("binary cross-entropy -- the loss for a yes/no answer:")
print("%10s %10s %12s" % ("said", "truth", "loss"))
for p, t in ((0.99, 1), (0.90, 1), (0.60, 1), (0.50, 1), (0.10, 1), (0.01, 1)):
    print("%10.2f %10d %12.4f" % (p, t, -np.log(p)))
print("  the loss for being right is small but never zero. the loss for being")
print("  confidently wrong has no ceiling -- at p=0.001 it is %.4f."
      % -np.log(0.001))
print()
print("  it is symmetric, so predicting 0.01 when the answer is 0 is cheap:")
for p, t in ((0.01, 0), (0.50, 0), (0.99, 0)):
    print("%10.2f %10d %12.4f" % (p, t, -np.log(1 - p)))
print()

print("categorical cross-entropy over 4 classes -- only the true class counts:")
logits = np.array([2.0, 1.0, 0.1, -1.0])
probs = np.exp(logits - logits.max()); probs /= probs.sum()
print("  logits      ", np.round(logits, 2))
print("  softmax     ", np.round(probs, 4), " sums to", round(probs.sum(), 10))
for true_class in range(4):
    onehot = np.zeros(4); onehot[true_class] = 1
    print("    if the answer is class %d: loss = -log(%.4f) = %.4f"
          % (true_class, probs[true_class], -np.log(probs[true_class])))
print("  the full formula sums over all classes, but the one-hot target")
print("  zeroes every term except the true one. that is why implementations")
print("  just index into the probability vector.")
print()

print("why the loss must be differentiable -- accuracy is not:")
print("%10s %12s %12s" % ("nudge", "accuracy", "cross-entropy"))
p = np.array([0.30, 0.40, 0.45, 0.48])
t = np.array([1, 1, 1, 1])
for shift in (0.00, 0.01, 0.02, 0.03, 0.04, 0.05, 0.10):
    q = np.clip(p + shift, 1e-9, 1 - 1e-9)
    acc = ((q > 0.5).astype(int) == t).mean()
    ce = -(t * np.log(q) + (1 - t) * np.log(1 - q)).mean()
    print("%10s %12.4f %12.4f" % ("+%.2f" % shift, acc, ce))
print()
print("every nudge makes the model measurably better, and accuracy notices")
print("exactly twice -- when a prediction crosses 0.50. in between it is")
print("perfectly flat, which means its gradient is zero, which means gradient")
print("descent has nothing to follow.")
print()
print("cross-entropy moves on every single nudge. that is not a nicety, it is")
print("the requirement: you optimise a loss and you REPORT a metric, and they")
print("are usually not the same function.")

Questions people ask

Why is my loss NaN? A log(0), a division by zero, or a learning rate high enough to send the weights to infinity. Use the framework's fused losses, which guard the logarithm.

Can I add several losses together? Yes, and it is common in multi-task models. Weight them, and be aware the weights matter: a term ten times larger dominates training regardless of importance.

Should the loss go to zero? Not on real data. Irreducible noise means there is a floor, and reaching zero on training data usually means memorisation.

Why does my validation loss rise while accuracy holds? Growing overconfidence. Consider label smoothing, and stop on the metric rather than the loss.

How do I handle imbalanced classes? Class weights, or focal loss, which down-weights the easy examples the majority class provides.

Do I apply softmax before the loss? No — pass logits. The fused loss applies it internally and stably. Applying it twice is a common silent bug.

Recap in one screen

• The loss reduces the whole batch to one number, and the model becomes whatever minimises it.
• MSE for regression, cross-entropy for classification; Huber when outliers are present.
• Squared error predicts the conditional mean; absolute error predicts the median.
• Mask padded positions and weight rare classes, or the average is measuring the wrong thing.
• Train on a differentiable surrogate, evaluate on the metric that matters, and expect them to disagree occasionally.