Four numbers that answer different questions, and the one property that decides which of them you should be reporting.
Overview
The metric is a statement about what matters
A regression model returns numbers. Judging those numbers requires a metric. The choice of metric is not a formality. Different metrics reward different behaviour, and a model that looks good on one can look poor on another.
scikit-learn ships with a dozen regression metrics. Each one condenses the list of residuals into a single number. The residuals are the same in each case: predicted minus actual, for every row. What changes is how those residuals are combined.
Understanding the main three -- MAE, RMSE and R-squared -- is enough for most work. Each answers a slightly different question. MAE asks how far off the average guess is. RMSE asks the same thing but penalises large misses more heavily. R-squared asks how much better the model is than always guessing the mean.
Choosing the wrong metric does not break anything. It just gives you a number that does not match what you actually care about. A model tuned on RMSE will tolerate many small errors to avoid one large one. A model tuned on MAE will spread the errors more evenly. Neither is better in the abstract. The right choice depends on what a mistake costs in your application.
each regression metric summarises the same thing. The gaps between guesses and actuals. And they differ in how they weigh those gaps. That weighting isn't a technical detail. It's a claim about which errors are worse. The claim should come from the problem rather than from habit.
If being wrong by 10 twice is just as bad as being wrong by 20 once, you want MAE. If one large miss is disproportionately damaging, you want RMSE. If a 10% error on a small value matters as much as a 10% error on a large one, you want a percentage metric. These are different situations and they truly occur. This Is why the library ships a few dozen metrics rather than one.
Worth knowing
MAE is the average absolute error, in the target's own units - the one to quote to a person.
RMSE squares before averaging, so it punishes a few large errors far more than many small ones.
R² is unitless and compares the model against predicting the mean; it says nothing about whether the error is tolerable.
MAPE is a percentage, which is easy to explain and undefined when any true value is zero.
Always compare against DummyRegressor - a score with no baseline is not interpretable.
In scoring= strings, error metrics carry a neg_ prefix because the helpers maximise.
Regression Metrics: A Practical Guide
R-squared is the default and rarely the right answer. Choosing the metric is choosing which mistakes you care about.
The four, on the same predictions
Same errors, four summaries of them - and they do not rank models identically.
Two sets of predictions wrong by the same total amount, and only one metric notices the difference.
example_02.pyscikit-learn
from sklearn.metrics import mean_absolute_error, mean_squared_error
import numpy as np
y_true = np.array([100, 120, 140, 160, 180])
# Both predictions are wrong by 75 in total, spread differently.
spread = np.array([115, 135, 155, 175, 195]) # 15 out, five times
one_bad = np.array([100, 120, 140, 160, 105]) # 75 out, once
for label, pred in [("15 out, five times", spread), ("75 out, once", one_bad)]:
err = np.abs(y_true - pred)
mae = mean_absolute_error(y_true, pred)
rmse = np.sqrt(mean_squared_error(y_true, pred))
print("%-20s total error %3d MAE %5.2f RMSE %5.2f"
% (label, err.sum(), mae, rmse))
Output
MAE is in the units of the target
Which is what makes it the number to put in front of somebody who has to decide whether the model is good enough.
example_03.pyscikit-learn
from sklearn.metrics import mean_absolute_error, r2_score
import numpy as np
y_true = np.array([100000, 120000, 140000])
y_pred = np.array([104000, 117000, 149000])
mae = mean_absolute_error(y_true, y_pred)
print("R2 :", round(r2_score(y_true, y_pred), 3), "- unitless")
print("MAE:", round(mae, 1), "- pounds, the same units as the target")
print()
print("only one of those answers 'is being wrong by this much acceptable?'")
Output
MAPE, and the division it cannot survive
A percentage error is easy to explain and undefined the moment a true value is zero.
example_04.pyscikit-learn
from sklearn.metrics import mean_absolute_percentage_error
import numpy as np
y_true = np.array([100.0, 200.0, 400.0])
y_pred = np.array([110.0, 210.0, 410.0])
print("all 10 out, MAPE =", round(mean_absolute_percentage_error(y_true, y_pred), 4))
print(" the same absolute error is a bigger share of a small number")
print()
with_zero = np.array([0.0, 200.0, 400.0])
print("with a zero in y_true, MAPE =",
mean_absolute_percentage_error(with_zero, y_pred))
print(" not an error, just a meaningless number - check for zeros yourself")
Output
Compare against the laziest possible model
A score means nothing until you know what doing nothing would have scored.
example_05.pyscikit-learn
from sklearn.dummy import DummyRegressor
from sklearn.linear_model import LinearRegression
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
X, y = make_regression(n_samples=200, n_features=4, noise=25.0, random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)
dummy = DummyRegressor(strategy="mean").fit(X_tr, y_tr)
model = LinearRegression().fit(X_tr, y_tr)
print("always predict the mean, R2 :", round(dummy.score(X_te, y_te), 4))
print("linear regression, R2 :", round(model.score(X_te, y_te), 4))
print()
print("the baseline is what 'good' has to beat.")
Output
Naming a metric for the cross-validation helpers
And the reason half of them are spelled with a neg_ prefix.
example_06.pyscikit-learn
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LinearRegression
from sklearn.datasets import make_regression
import numpy as np
X, y = make_regression(n_samples=200, n_features=4, noise=25.0, random_state=0)
model = LinearRegression()
r2 = cross_val_score(model, X, y, cv=5, scoring="r2")
mae = cross_val_score(model, X, y, cv=5, scoring="neg_mean_absolute_error")
print("r2 :", np.round(r2, 3))
print("neg_mean_absolute_error:", np.round(mae, 1))
print()
print("the 'neg_' prefix exists because these helpers always maximise,")
print("so an error metric is negated to make bigger mean better.")
Output
MAE: the average size of the mistake
Mean abs error is the mean of the abs differences. Nothing is squared. Each error adds in proportion to its size.
Its great virtue is that it's expressed in the units of the target. If you're predicting house prices in pounds, an MAE of 5,333 means the common guess is out by about five thousand pounds. That sentence can be said to somebody who has never heard of a regression. They can tell you whether it's acceptable.
That is a truly key property. Most of the value of a model is decided by a person who has to judge whether the errors are tolerable. They can't judge that from a unitless number. MAE is the default worth reaching for when the answer will be reported to anyone.
It's also solid: a single wild outlier moves it by its own size and no more.
RMSE: the same idea, with big errors amplified
Root mean squared error squares each error, means. Takes the square root. The squaring is the whole difference.
Squaring makes a large error count for much more than a few small ones adding to the same total. The editors on this page show it just. Two sets of guesses wrong by 75 in total have the same MAE of 15. RMSE of 15.00 and 33.54 in turn. The second concentrated its error in one place, and RMSE says so.
Whether that is wanted depends fully on the use. For predicting delivery times, being an hour late once may be far worse than being twelve minutes late five times. The customer notices the hour. For predicting daily sales, the total is what matters and the distribution of errors isn't above all key.
The square root at the end puts the number back into the units of the target. RMSE is close to MAE in magnitude. RMSE is always at least as large as MAE. The gap between them tells you how unevenly the errors are spread. Equal means uniform, a wide gap means a few large misses.
MSE without the square root is the same ranking in squared units. This Are uninterpretable. It appears because it's what the fitting cuts and what differentiates cleanly, not because anyone should report it.
R-squared: a comparison, not a measurement
R² asks how much of the variation in the target the model accounts for, checked against a model that always predicts the mean.
1.0 is perfect, 0.0 is no better than the mean. Minus means worse than the mean. Which is common on a test set and always a signal rather than a curiosity.
Two things make it common. It's unitless. It can be checked across problems with different targets. And it has a built-in baseline. So A value tells you a thing without a separate check run.
Two things make it dangerous. It says nothing about the size of the errors. An R² of 0.87 is compatible with being wrong by five pounds or five thousand. And it depends on the variance of the target in the test set. This Means the same model scores differently on two test sets drawn from the same data. A narrow test set makes the model look worse simply because the mean baseline is harder to beat.
Use it for checking models on the same data. Don't use it to decide whether a model is good enough for a purpose. Don't compare it across datasets.
MAPE, and why it is both loved and distrusted
Mean abs percentage error expresses each error as a fraction of the true value and means those.
Its appeal is obvious: "we are often 6% out" is a sentence anyone understands. It's close across products, regions or scales in a way an abs number isn't.
Its problems are equally real. It's undefined when any true value is zero. Rather than raising, scikit-learn returns an huge number. The editor above shows 1.65e+17. Which will pass through an blending step and give nonsense no one notices. It's skewed. Over-predicting is penalised differently from under-predicting. This Biases model pick towards forecasting low. And it explodes for true values that are merely small rather than zero. A few near-zero rows can dominate the mean fully.
If percentages are what the audience wants, MAPE is usable on strictly plus data with no small values. Or else, quote MAE alongside the mean of the target and let the reader do the division.
Always compare against doing nothing
DummyRegressor predicts the mean, or the median, or a constant you choose, skipping the features fully. Fitting one takes a line and it's the most valuable line in an test.
no metric is readable in alone. An R² of 0.4 could be an great result on a truly noisy problem or a terrible one on an easy problem. An MAE of 5,000 is meaningless without knowing that the mean baseline scores 12,000. Or 5,200, in which case the model has bought almost nothing for its cost.
That last case is more common than people expect. It never shows up unless the baseline is run. A model that beats the mean by 4% isn't always worth deploying, keeping and showing. The number that tells you so takes two lines to give.
The neg_ prefix
cross_val_score and each search helper are written to raise. They assume bigger is better. That the same code can rank any metric without knowing which one it's.
Error metrics are the wrong way round for that. Smaller is better. Rather than special-case them, scikit-learn negates them: neg_mean_absolute_error returns -15 where MAE would return 15. Raising -15 towards zero is the same as cutting 15.
So the values come back minus and you flip the sign to read them. It's startling the first time. It's the reason scoring="mean_absolute_error" raises an error naming the valid options. sorted(sklearn.metrics.get_scorer_names()) lists all of them. This Is faster than guessing the spelling.
Errors that are not symmetric
All four metrics on this page treat over- and under-guess as equally bad. Often they aren't.
Under-predicting demand means empty shelves and lost sales. Over-predicting means stock that has to be written off. Under-guessing a journey time annoys a customer. Over-guessing loses the booking. Under-forecasting a load causes an outage. Over-forecasting costs money. In each one of those the two directions have different prices. A even metric will happily choose a model that is wrong in the costly direction.
Two ways to handle it. Write a custom scorer with make_scorer. This Takes a function of (y_true, y_pred) and a greater_is_better flag. Put the real cost in that function. If a unit of over-guess costs three times a unit of under-guess, say so arithmetically. Or use quantile regression, where the model is fitted to predict a chosen percentile rather than the mean. Fitting the 80th percentile gives guesses that are on purpose high most of the time. This Is what you want when running out is worse than having spare.
GradientBoostingRegressor(loss="quantile", alpha=0.8) does the second right. The skew then lives in the model rather than only in the report.
Reading errors as a distribution
A single averaged number hides all about the shape of the errors. The shape is often where the info is.
Two habits repay the minute they cost. Plot residuals against guesses. A shapeless cloud means the model has extracted what it can. An arc means a missing non-linearity. A fan that widens to the right means the errors grow with the size of the target. This Often argues for predicting the logarithm instead.
Look at the worst cases. Sorting by abs error and reading the ten worst rows tells you more about a model than any summary statistic. often they've a thing in common. A category the training data barely covered, a period with an unusual event, rows with a missing field imputed to the median. And that commonality is a feature waiting to be added or a subset the model shouldn't be trusted on.
Neither of these appears in a metric, and both change what you do next.
Which metric should I optimise during training? often squared error. It's smooth and fast to fit. You can report a different one - fitting and reporting need not use the same metric, and often shouldn't.
Is RMSE always bigger than MAE? Yes, or equal when each error is the same size. The ratio between them measures how uneven the errors are.
How do I see each valid scoring name?sorted(sklearn.metrics.get_scorer_names()). This Is quicker than guessing at the spelling.
Should I report one metric or a few? a few. Always with the baseline beside them. MAE answers "how wrong, often", RMSE answers "are there occasional disasters". The dummy answers "checked to what". Any one of the three alone is easy to misread, and together they take one extra line to give.
My R-squared is high and the guesses look bad. How? Almost always a target with large variance. Here Beating the mean is easy and still leaves errors that matter in abs terms. Look at MAE next to the mean of the target.
Things to try with Regression Metrics
Run the second editor. Same total error, same MAE, RMSE more than twice as large. That single check is the whole arg for choosing between them.
Add an outlier. In the first editor, change one guess to 400 and watch which metrics move most.
Beat the baseline by less. In the fifth editor, raise noise to 200 and see how much of the model's edge lives.
Get the spelling wrong. Ask for scoring="mae" and read the error - it lists each valid name.
Questions people ask about Regression Metrics
Which metric should I report: MAE, RMSE or R-squared? Report MAE for an average error in the target's own units and RMSE if large misses matter more. R-squared helps compare against a mean baseline.
Why is RMSE larger than MAE? RMSE squares errors before averaging, so big mistakes count more. The gap between the two hints at how heavy the tail of errors is.
Why do scores show up negative in cross_val_score? Scorers follow a greater-is-better rule, so errors are returned with a neg_ prefix. Flip the sign to read the error.
When does MAPE fail? When true values are zero or close to it, because it divides by them. It also penalises over- and under-prediction unequally.
What counts as a good R-squared? It depends on the field. Compare against a dummy model on the same data rather than against a fixed number.
Should I look at more than one number? Yes. A residual plot or an error histogram shows patterns, such as errors growing with the target, that a single metric hides.
Where this leaves you
MAE for reporting to a person, RMSE when large errors are disproportionately costly, R² for checking models on the same data. A dummy baseline underneath all three. Which one you lead with is a choice about the problem, not about the model.
Check yourself
0 of 4
Answer without scrolling back up.
Which metric is expressed in the target's own units?
MAE is an average of absolute differences, so predicting pounds gives an MAE in pounds. RMSE is too, since the square root undoes the squaring.
Two models have the same total absolute error. Which metric distinguishes them?
Squaring makes RMSE larger when the error is concentrated in a few large misses rather than spread evenly.
What does a negative R-squared mean?
R-squared is measured against the mean baseline, and a model can do worse than that - commonly on a test set after overfitting.
Why is it spelled neg_mean_absolute_error?
Negating lets one code path rank every metric by taking the largest, without knowing whether bigger or smaller is better for that metric.
Cheat sheet
Regression Metrics
Compare MAE, MSE, RMSE and R-squared on the same predictions, and see why R-squared alone rarely tells you whether a model is good enough.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.