Evaluation Metrics for Regression

The Four Key Metrics

Each metric tells a slightly different story about your model's errors. The interactive plot above shows the true data points (green dots), the model's prediction line (orange), and the errors, or residuals (dashed red lines), which are the distances between each true point and the line.

1. Mean Absolute Error (MAE)

2. Mean Squared Error (MSE)

3. Root Mean Squared Error (RMSE)

4. R-Squared ($R^2$)

Five houses, four metrics

Definitions blur together until you put numbers through them. Here are five predicted and actual house prices, in thousands of pounds:

• MAE = (10 + 15 + 5 + 10 + 60) / 5 = 20. On average the model is £20k out.
• MSE = (100 + 225 + 25 + 100 + 3600) / 5 = 810. In squared thousands of pounds, which is not a unit anyone can picture.
• RMSE = √810 = 28.5. Back in thousands of pounds, and noticeably larger than the MAE.
• MAPE = mean of (10/200, 15/250, 5/180, 10/300, 60/400) = mean of 5%, 6%, 2.8%, 3.3%, 15% = 6.4%.

The gap between MAE 20 and RMSE 28.5 is the whole lesson. Four houses are predicted within £15k; one is out by £60k. Squaring turns that single house into 3600 of the 4050 total — 89% of the MSE comes from one row out of five. RMSE is loudly telling you there is a bad miss somewhere; MAE quietly averages it in.

So a rule you can use immediately: if RMSE is much larger than MAE, you have outliers. Go and look at them before you touch the model.

R², and what "explained variance" means

R² answers a different question from the others: not "how wrong am I?" but "how much better am I than the laziest possible model?"

That lazy model is predicting the mean of the target for every row. Compute its total squared error, compute your model's, and:

R² = 1 − (your squared error / the mean-predictor's squared error)

• R² = 1 — perfect predictions.
• R² = 0 — exactly as good as always guessing the average. Your model has added nothing.
• R² < 0 — worse than guessing the average. Entirely possible on test data, and always worth investigating rather than reporting.

The trap is that R² never falls when you add a feature to a linear model, even a column of random numbers, because the extra freedom can only reduce training error. Adjusted R² corrects for this by penalising the number of predictors, and is the one to quote when comparing models with different numbers of features.

The other trap is comparing R² across datasets. An R² of 0.4 might be superb in a noisy domain like human behaviour and embarrassing in a controlled physical measurement. It is a comparison against the mean of this dataset, so it means nothing across two.

Choosing by the shape of the mistake you can tolerate

Each metric encodes an opinion about which errors matter.

Two practical notes. MAPE is undefined when an actual value is zero and blows up when one is near zero — for demand forecasting where zeros are common, use MAE or a symmetric variant instead. And MSE is the loss most regressors minimise internally, so evaluating with MSE tells you how well the optimiser did its stated job, not necessarily how useful the model is.

Reporting numbers people can act on

A metric on its own is a number without a verdict. Three habits make it a decision.

Quote a baseline beside it. "RMSE 28.5" means nothing alone. "RMSE 28.5, against 61.0 for predicting the average and 34.2 for last year's model" is a result. Build the dumb baseline first, every time.

Say what the error means in the target's units. "Typically wrong by £20k on a £250k house" is a sentence a business owner can price. Percentages help here, which is one of the few genuine arguments for MAPE.

Plot the residuals. The single most informative thing you can do with a regression model costs two lines of code: plot predicted values against errors. A shapeless cloud around zero is healthy. A curve means the model is missing a non-linear relationship. A funnel widening to the right means the error grows with the size of the target, and predicting the logarithm often fixes it. A handful of points far from the rest are the outliers your RMSE has been complaining about.

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np

y_true = [200, 250, 180, 300, 400]
y_pred = [190, 265, 175, 290, 340]

mae  = mean_absolute_error(y_true, y_pred)             # 20.0
rmse = np.sqrt(mean_squared_error(y_true, y_pred))     # 28.46
r2   = r2_score(y_true, y_pred)                        # 0.87

baseline = np.full_like(y_true, np.mean(y_true), dtype=float)
print(mean_absolute_error(y_true, baseline))           # the number to beat

Five metrics, and the one case that separates them

MAE, RMSE, R^2, MAPE and median absolute error on the same predictions -- then one outlier is introduced, and only some of them notice.

import numpy as np
from sklearn.metrics import (mean_absolute_error, mean_squared_error, r2_score,
                             median_absolute_error,
                             mean_absolute_percentage_error)

truth = np.array([100.0, 200.0, 300.0, 400.0, 500.0, 600.0])
a = truth + np.array([25, -25, 25, -25, 25, -25])      # spread evenly
b = truth + np.array([0, 0, 0, 0, 0, 150])             # all in one place

print("two sets of predictions with the SAME total absolute error:")
print("  A: off by 25 on every row      -> total |error| %d" % np.abs(a - truth).sum())
print("  B: perfect except one row, off by 150 -> total |error| %d"
      % np.abs(b - truth).sum())
print()
print("%-26s %10s %10s" % ("", "A", "B"))
for name, fn in (("MAE", mean_absolute_error),
                 ("RMSE", lambda t, p: np.sqrt(mean_squared_error(t, p))),
                 ("median abs error", median_absolute_error),
                 ("R^2", r2_score),
                 ("MAPE", mean_absolute_percentage_error)):
    print("%-26s %10.4f %10.4f" % (name, fn(truth, a), fn(truth, b)))
print()
print("MAE says they are identical, because they are: same total error.")
print("RMSE says B is worse, because it squares before averaging and one big")
print("miss squares up much faster than six small ones.")
print("median absolute error says B is nearly perfect, because four of its six")
print("rows are exact and the median ignores the tail entirely.")
print()
print("none of them is wrong. they answer different questions:")
print("  MAE   -- what is a typical miss, in the units of the thing?")
print("  RMSE  -- how bad is it when it goes wrong?")
print("  median-- what does the usual case look like, ignoring disasters?")
print()

print("MAPE has a trap of its own -- it divides by the truth:")
small = np.array([1.0, 2.0, 1000.0])
pred = np.array([2.0, 3.0, 1001.0])
print("  truth", small, "predictions", pred, "-- every row off by exactly 1")
print("  MAE  %.4f" % mean_absolute_error(small, pred))
print("  MAPE %.4f  <- the two small rows dominate completely"
      % mean_absolute_percentage_error(small, pred))
print("  and a single zero in the truth makes it infinite.")
print()

print("R^2 compares you against predicting the mean, every time:")
rng = np.random.default_rng(0)
y = rng.normal(50, 10, 400)
for name, p in (("predict the mean", np.full(400, y.mean())),
                ("predict the mean + noise", np.full(400, y.mean()) + rng.normal(0, 5, 400)),
                ("a decent model", y + rng.normal(0, 4, 400)),
                ("a bad model", y.mean() - (y - y.mean()))):
    print("  %-26s R^2 %8.4f   RMSE %7.4f"
          % (name, r2_score(y, p), np.sqrt(mean_squared_error(y, p))))
print("  R^2 = 0 means 'no better than the mean'. it can go negative, and a")
print("  negative R^2 means you would have done better predicting one number.")
print()
print("R^2 is the only one on this list that is unitless, which makes it the")
print("only one you can compare across different problems -- and the only one")
print("that tells you nothing about whether an error of that size matters.")
print("Report RMSE or MAE next to it, in the units your reader thinks in.")

Try it yourself

Use the controls and the interactive plot to build a strong intuition for these metrics.

1. The Impact of a Single Outlier: Start with the "Good Linear Fit" dataset. Drag one of the green data points far away from the prediction line. Watch the metrics change. You'll notice that MSE and RMSE increase dramatically compared to MAE. This is because they square the error, heavily penalizing this large deviation (outlier).
2. Good Fit vs. Poor Fit: Use the "Initial Dataset" dropdown. Switch between "Good Linear Fit" and "Poor Fit (High Bias)". Observe how all error metrics (MAE, MSE, RMSE) are much higher for the poor fit. Now look at the $R^2$ score. For the good fit, it should be high (e.g., > 0.8), while for the poor fit, it will be very low, possibly even negative, indicating the model is useless.
3. Manually Improving the Fit: Select the "Poor Fit" dataset. The orange line will be far from the data. Now, click and drag the handles on the ends of the orange line to move it closer to the green data points. As you manually improve the fit, watch all the error metrics decrease and the $R^2$ score increase towards 1.0.
4. The Effect of Noise: Select the "Good Linear Fit" and use the "Noise Level" slider. As you increase the noise, the data points become more scattered. Notice that even with the best possible line, the error metrics will rise, and the maximum achievable $R^2$ score will fall. This shows that the inherent randomness in data puts a limit on how well any model can perform.

Which Metric Should You Use?

• Use MAE when you want a simple, interpretable metric that is not sensitive to outliers. For example, forecasting sales where a few large errors are acceptable.
• Use RMSE when large errors are particularly undesirable and you want to penalize them more. It's the most common metric for regression tasks. For example, predicting house prices where a large error could be very costly.
• Use $R^2$ when you want to understand the proportion of variance your model explains. It's great for communicating the overall "goodness of fit" of your model to a non-technical audience.

Questions people ask

Which metric should I optimise during training? Usually MSE or Huber, because they are smooth and differentiable. You can train on MSE and still report MAE — the training loss and the reporting metric do not have to match, and often should not.

My R² is negative on the test set. What happened? The model is doing worse than predicting the average. Common causes: severe overfitting, a distribution shift between train and test, or a preprocessing step fitted on the wrong data.

Is RMSE always bigger than MAE? Yes, or equal — they are only equal when every error has the same magnitude. The ratio between them is a quick outlier detector.

How do I compare models predicting different targets? Not with RMSE, which lives in the target's units. Use MAPE or R², or normalise RMSE by the target's mean or range.

Should I remove outliers to improve my metrics? Only if they are genuine errors — a price of £1 for a house, a sensor reading of 999. Removing real but inconvenient values makes your metric look better and your model worse in exactly the cases that matter.

Can I use accuracy for regression? Not directly, but "within £10k" style tolerance accuracy is a legitimate business metric and is often the one stakeholders actually care about. Report it alongside, not instead of, a proper error measure.

Recap in one screen

• MAE is the average miss in the target's own units; it treats all errors equally.
• MSE and RMSE square the errors, so a few big misses dominate; RMSE brings the units back.
• RMSE much larger than MAE is a reliable signal that outliers are present.
• R² compares your model to predicting the mean; below zero it has lost to the mean.
• MAPE is intuitive as a percentage but breaks near zero.
• Always report a baseline, always plot the residuals.