Build and understand deep learning architectures explicitly designed to predict continuous numerical values.
Everything else — hidden layers, ReLU, backpropagation, the optimiser — is unchanged from a classifier.
A regression network is a classifier with two changes: nothing squashes the output, and the loss measures distance instead of disagreement.
Predicting house price from three features: 3 inputs → 16 hidden with ReLU → 1 linear output. That is (3×16 + 16) + (16×1 + 1) = 81 parameters.
The output neuron is where regression and classification part ways. Leave a sigmoid on it and the largest value the network can ever predict is 1.0 — every house is priced somewhere between nothing and one. It is a surprisingly easy mistake to make when adapting a classifier.
Use several output neurons when you have several continuous targets: predicting an (x, y, z) position is three linear outputs and one shared body.
A classification network ends in softmax or sigmoid because the output must be a probability, bounded in [0, 1]. A regression network predicts an unbounded quantity — a price, a temperature, a duration — so squashing the output would put a ceiling on what it can ever predict.
The final layer is therefore left linear: a plain weighted sum with no activation applied. Putting a sigmoid there caps every prediction at 1, and putting a ReLU there makes negative predictions impossible — occasionally what you want for a strictly positive target, and a silent bug otherwise.
The loss encodes what kind of error you care about, and the two standard choices behave very differently:
These are not interchangeable: minimising MSE fits the conditional mean of the target, minimising MAE fits the median. On a skewed target those are different numbers, and the choice of loss is really a choice about which one you want.
Input scaling matters as much as ever. Target scaling is the additional consideration specific to regression.
If house prices range over hundreds of thousands, the initial loss is in the billions, the gradients are enormous, and the first update destroys the initialisation. Standardising the target — or predicting its logarithm — brings the loss into a sane range.
Two things then have to happen, and forgetting either is a common bug:
Invert the transformation before reporting. An RMSE computed on standardised targets is in standard deviations, not pounds, and is not comparable to anyone else's number.
Fit the target scaler on training data only, exactly as for the inputs.
Predicting log(y) deserves particular mention. It suits multiplicative targets — prices, counts, durations — because it turns proportional errors into absolute ones, which is usually what matters. Note that exponentiating a predicted mean of logs gives a geometric mean, which is slightly below the arithmetic mean; for most applications that bias is acceptable, and there are corrections if it is not.
A regression network is a classification network with three changes: one output unit instead of many, no activation on that output, and a regression loss.
| Classification | Regression | |
|---|---|---|
| Output units | One per class | One (or one per target) |
| Output activation | Softmax / sigmoid | None |
| Loss | Cross-entropy | MSE, MAE or Huber |
| Metric | Accuracy, F1 | RMSE, MAE, R² |
"No activation" is the part people get wrong. Applying ReLU to the output caps predictions at zero and below, which silently breaks any target that can be negative. Applying sigmoid caps them at 1. Leave it linear unless you have a specific reason — and if the target is genuinely non-negative, predicting its logarithm is usually better than clamping with ReLU.
model = nn.Sequential(
nn.Linear(n_features, 128), nn.ReLU(),
nn.Linear(128, 64), nn.ReLU(),
nn.Linear(64, 1), # no activation
)
criterion = nn.MSELoss() # or nn.HuberLoss()MSE squares the errors, so one large miss dominates. Minimising it predicts the conditional mean of the target.
MAE treats every unit of error equally and is robust to outliers. Minimising it predicts the conditional median.
Huber is squared near zero and linear beyond a threshold δ — smooth gradients near the optimum, bounded influence from outliers. Usually the best default when the data has any outliers at all.
Two more, for specific needs. Quantile loss predicts a chosen percentile rather than the centre, which is how prediction intervals are produced — train one head for the 10th percentile and one for the 90th and you have an interval. And Poisson or Tweedie loss suits counts and heavily zero-inflated targets, where MSE assumes a symmetry the data does not have.
A squashing activation left on the output. Caps the prediction range and flattens the gradient for any target outside it, so the model cannot learn its way out.Unscaled targets. Squared error on values near 300,000 produces gradients around 1011. The first update destroys the weights and the loss becomes NaN. Standardise the target and invert the scaling afterwards.Reporting accuracy. There is no such thing for continuous output. Report MAE or RMSE in the target's own units, so "off by £18,000 on average" is something a reader can actually judge.
Classifier body, linear head, distance-based loss — and scale your targets.
A network trained on house sizes from 50 to 200 square metres will happily produce a number for 900. That number is fiction.
Networks interpolate well and extrapolate unpredictably. Outside the range of the training data there is no signal constraining the function, and the shape it takes there is an artefact of the architecture and the initialisation rather than anything learned.
Three responses:
Note that tree-based models have the same limitation in a more visible form: a forest cannot predict outside the range of its training targets at all. Linear models extrapolate confidently and wrongly. There is no free lunch here — only awareness.
For tabular regression, often not. Gradient-boosted trees usually match or beat a neural network on rows-and-columns data while needing less tuning, less data and no GPU. This has been tested repeatedly.
Neural networks earn their place in regression when:
Otherwise, fit a gradient-boosted model first and treat it as the baseline the network has to beat.
Three habits make the numbers meaningful:
Quote a baseline. "RMSE 28.5" means nothing alone. Against 61.0 for predicting the mean, it is a result.
Report the error in the target's units. "Typically wrong by £20k on a £250k house" is actionable; a standardised number is not.
Plot the residuals against the predictions. A shapeless band around zero is healthy. A curve means a missing non-linear relationship. A funnel widening to the right means the error grows with the target — predict the logarithm instead. A cluster far from the rest is an outlier group worth investigating.
A regression network differs from a classifier in the last layer, the loss, the metric and the target scaling. Each one is shown here, along with what breaks if you skip it.
Should the output layer have an activation? No, for unbounded targets. Never ReLU unless negatives are genuinely impossible and you have considered a log transform instead.
MSE or MAE? MSE if large errors are disproportionately costly; MAE if outliers are present and should not dominate; Huber as the default compromise.
Do I need to scale the target? For networks, usually yes — and remember to invert before reporting.
How do I get prediction intervals? Quantile regression with two extra output heads, or Monte Carlo dropout, or an ensemble's spread.
Can one network predict several targets? Yes — several output units and a summed loss. Weight the terms, since a target with a larger scale will otherwise dominate.
Why is my model predicting nearly the same value for everything? It has collapsed to predicting the mean, which is what happens when the features carry little signal, the learning rate is too low, or the target was not scaled.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What is meant by “Check the input range at prediction time” here?
and flag or refuse inputs outside it. This is a production concern that is easy to overlook.
What is meant by “Model the right quantity” here?
Predicting price-per-square-metre, or a change rather than a level, often extrapolates far better than predicting the raw level.
Predicting house price from three features: 3 inputs → 16 hidden with ReLU → 1 linear output. That is (3×16 + 16) + (16×1 + 1) = 81 parameters.