Build and understand deep learning architectures explicitly designed to predict continuous numerical values.
All else — hidden layers, ReLU, backpropagation, the optimiser — is unchanged from a classifier.
That simplicity is both the appeal and the trap. Since the change is small, people adapt a classifier by swapping the output layer and the loss, run it. And move on. The mistakes that follow are set to regression and do not give error messages.
The first is forgetting to scale the target. A classifier's output is always between 0 and 1. So the internal activations stay in a reasonable range. A regression target of 500,000 means the loss starts enormous and the gradients explode on the first step. Normalising the target to zero mean and unit variance, then reversing the transform on the predictions, avoids this fully.
The second is with too many hidden units for a smooth function. A regression surface is often simpler than a classification edge. A network with 256 hidden units fitting a relationship that is nearly linear will memorise the training noise and predict jagged curves. Start small: one hidden layer of 16 or 32 units is enough for many real-world regression tasks. And you can always add capacity if the validation loss plateaus too high.
The third is testing with the wrong metric. Accuracy does not exist for regression. Mean squared error penalises large errors heavily. Mean absolute error treats all errors equally. R-squared tells you how much variance the net explains relative to predicting the mean. Each answers a other question. And reporting the wrong one hides the net's real behaviour.
The output layer, the loss, the target scaling and the test metric are the four things a regression network needs you to get right. All else transfers from classification unchanged.
Build and understand deep learning architectures explicitly designed to predict continuous numerical values.
A regression network is a classifier with two changes: nothing squashes the output, and the loss measures distance instead of disagreement.
All else — hidden layers, ReLU, backpropagation, the optimiser — is unchanged from a classifier.
That simplicity is both the appeal and the trap. Since the change is small, people adapt a classifier by swapping the output layer and the loss, run it. And move on. The mistakes that follow are set to regression and do not give error messages.
The first is forgetting to scale the target. A classifier's output is always between 0 and 1. So the internal activations stay in a reasonable range. A regression target of 500,000 means the loss starts enormous and the gradients explode on the first step. Normalising the target to zero mean and unit variance, then reversing the transform on the predictions, avoids this fully.
The second is with too many hidden units for a smooth function. A regression surface is often simpler than a classification edge. A network with 256 hidden units fitting a relationship that is nearly linear will memorise the training noise and predict jagged curves. Start small: one hidden layer of 16 or 32 units is enough for many real-world regression tasks. And you can always add capacity if the validation loss plateaus too high.
The third is testing with the wrong metric. Accuracy does not exist for regression. Mean squared error penalises large errors heavily. Mean absolute error treats all errors equally. R-squared tells you how much variance the net explains relative to predicting the mean. Each answers a other question. And reporting the wrong one hides the net's real behaviour.
The output layer, the loss, the target scaling and the test metric are the four things a regression network needs you to get right. All else transfers from classification unchanged.
Predicting house price from three features: 3 inputs → 16 hidden with ReLU → 1 linear output. That is (3×16 + 16) + (16×1 + 1) = 81 params.
The output neuron is where regression and classification part ways. Leave a sigmoid on it and the largest value the network can ever predict is 1.0 — each house is priced somewhere between nothing and one. It is a surprisingly easy mistake to make when adapting a classifier.
Use a few output neurons when you have a few continuous targets: predicting an (x, y, z) position is three linear outputs and one shared body.
A classification network ends in softmax or sigmoid since the output must be a probability, bounded in [0, 1]. A regression network predicts an unbounded quantity — a price, a temperature, a duration — so squashing the output would put a ceiling on what it can ever predict.
The final layer is therefore left linear: a plain weighted sum with no activation applied. Putting a sigmoid there caps each prediction at 1. And putting a ReLU there makes negative predictions impossible — at times what you want for a strictly positive target. And a silent bug otherwise.
The loss encodes what kind of error you care about. And the two standard choices behave very differently:
These are not interchangeable: minimising MSE fits the conditional mean of the target, minimising MAE fits the median. On a skewed target those are other counts. And the choice of loss is really a choice about which one you want.
Input scaling matters as much as ever. Target scaling is the extra consideration set to regression.
If house prices range over hundreds of thousands, the initial loss is in the billions, the gradients are enormous. And the first update destroys the initialisation. Standardising the target — or predicting its logarithm — brings the loss into a sane range.
Two things then have to happen, and forgetting either is a common bug:
Invert the transformation before reporting. An RMSE computed on standardised targets is in standard deviations, not pounds. And is not comparable to anyone else's count.
Fit the target scaler on training data just, just as for the inputs.
Predicting log(y) deserves particular mention. It suits multiplicative targets — prices, counts, durations — since it turns proportional errors into absolute ones. It is often what matters. Note that exponentiating a predicted mean of logs gives a geometric mean. It is slightly below the arithmetic mean. For most applications that bias is acceptable. And there are corrections if it is not.
A regression network is a classification network with three changes: one output unit instead of many, no activation on that output. And a regression loss.
| Classification | Regression | |
|---|---|---|
| Output units | One per class | One (or one per target) |
| Output activation | Softmax / sigmoid | None |
| Loss | Cross-entropy | MSE, MAE or Huber |
| Metric | Accuracy, F1 | RMSE, MAE, R² |
"No activation" is the part people get wrong. Applying ReLU to the output caps predictions at zero and below. It silently breaks any target that can be negative. Applying sigmoid caps them at 1. Leave it linear unless you have a set reason — and if the target is genuinely non-negative, predicting its logarithm is often better than clamping with ReLU.
model = nn.Sequential(
nn.Linear(n_features, 128), nn.ReLU(),
nn.Linear(128, 64), nn.ReLU(),
nn.Linear(64, 1), # no activation
)
criterion = nn.MSELoss() # or nn.HuberLoss()MSE squares the errors, so one large miss dominates. Minimising it predicts the conditional mean of the target.
MAE treats each unit of error equally and is reliable to outliers. Minimising it predicts the conditional median.
Huber is squared near zero and linear beyond a threshold δ — smooth gradients near the optimum, bounded influence from outliers. Often the best default when the data has any outliers at all.
Two more, for set needs. Quantile loss predicts a chosen percentile rather than the centre. It is how prediction intervals are produced — train one head for the 10th percentile and one for the 90th and you have an interval. And Poisson or Tweedie loss suits counts and heavily zero-inflated targets. There MSE assumes a symmetry the data does not have.
A squashing activation left on the output. Caps the prediction range and flattens the gradient for any target outside it. So the net cannot learn its way out.Unscaled targets. Squared error on vals near 300,000 gives gradients around 1011. The first update destroys the weights and the loss turns NaN. Standardise the target and invert the scaling afterwards.Reporting accuracy. There is no such thing for continuous output. Report MAE or RMSE in the target's own units. So "off by £18,000 on average" is something a reader can really judge.
Classifier body, linear head, distance-based loss — and scale your targets.
A network trained on house sizes from 50 to 200 square metres will happily give a count for 900. That count is fiction.
Networks interpolate well and extrapolate unpredictably. Outside the range of the training data there is no signal constraining the function. And the shape it takes there is an artefact of the design and the initialisation rather than anything learned.
Three replies:
Note that tree-based nets have the same limitation in a more visible form: a forest cannot predict outside the range of its training targets at all. Linear nets extrapolate confidently and wrongly. There is no free lunch here — just awareness.
For tabular regression, often not. Gradient-boosted trees often match or beat a neural network on rows-and-columns data while needing less tuning, less data and no GPU. This has been tested repeatedly.
Neural networks earn their place in regression when:
Otherwise, fit a gradient-boosted net first and treat it as the baseline the network has to beat.
Three habits make the counts meaningful:
Quote a baseline. "RMSE 28.5" means nothing alone. Against 61.0 for predicting the mean, it is a result.
Report the error in the target's units. "Often wrong by £20k on a £250k house" is actionable. A standardised count is not.
Plot the residuals against the predictions. A shapeless band around zero is healthy. A curve means a missing non-linear relationship. A funnel widening to the right means the error grows with the target — predict the logarithm instead. A cluster far from the rest is an outlier group worth investigating.
A regression network differs from a classifier in the last layer, the loss, the metric and the target scaling. Each one is shown here, along with what breaks if you skip it.
Building on the same setup, the next block shows the left cases.
The next block covers the left steps.
The next block continues with the left demonstrations.
Should the output layer have an activation? No, for unbounded targets. Never ReLU unless negatives are genuinely impossible and you have considered a log transform instead.
MSE or MAE? MSE if large errors are disproportionately costly. MAE if outliers are present and should not dominate. Huber as the default compromise.
Do I need to scale the target? For networks, often yes — and remember to invert before reporting.
How do I get prediction intervals? Quantile regression with two extra output heads. Or Monte Carlo dropout, or an ensemble's spread.
Can one network predict a few targets? Yes — a few output units and a summed loss. Weight the terms, since a target with a more scale will otherwise dominate.
Why is my net predicting nearly the same value for all? It has collapsed to predicting the mean. It is what happens when the features carry little signal, the learning rate is too low. Or the target was not scaled.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What is meant by “Check the input range at prediction time” here?
What is meant by “Net the right quantity” here?
Adapt a neural network to predict continuous values, and see how its output layer and loss differ from a classifier's.