One split gives you one number and no idea how much to trust it. Five splits tell you both.
Overview
The problem with one split
A single train/test split gives you one score. That score depends on which rows happened to land in each set. A different random seed would give a different number. Cross-validation addresses this by running multiple splits and reporting all the scores.
The idea is straightforward. Divide the data into k equal parts. Train on k-1 of them, test on the remaining one. Repeat k times, each time holding out a different part. The result is k scores instead of one.
This gives you a mean and a spread. The mean is a better estimate of how the model performs in general. The spread tells you how sensitive the score is to which rows were held out. A tight spread means the model is stable. A wide spread means the result depends heavily on the particular data.
Cross-validation does not produce a final trained model. It produces an estimate of how well a model of this type, trained on this much data, tends to perform. You still fit a final model on all the data after deciding which approach to use.
A held-out test set answers the question honestly. It answers it once, on one given random pick of rows.
Run the split again with a different seed and the score moves. The editors on this page show six splits of the same data giving scores from 0.9737 to 0.9912. A spread of nearly two points, on an easy dataset with a few hundred rows. On a smaller or harder dataset the spread is much wider.
That variation isn't noise you can skip. If you compare two models on one split each and they differ by a point, you've learned nothing. The difference is well inside the range a single split makes by chance. Reporting the winner would be reporting which model got the friendlier rows.
Cross-validation fixes this by not choosing. It splits the data into k parts, holds each one out in turn, fits on the rest. Returns k scores. So each row is tested on just once and trained on k-1 times.
Worth knowing
cross_val_score returns one score per fold - the spread matters as much as the mean.
For a classifier, cv=5 uses StratifiedKFold automatically; for a regressor, plain KFold.
Every preprocessing step must sit inside a Pipeline, or it is fitted on the fold it is about to be tested on.
cross_validate takes several metrics at once and can return training scores, which is how overfitting becomes visible.
The models fitted during cross-validation are discarded - it estimates a procedure, it does not produce a model.
TimeSeriesSplit for ordered data, GroupKFold when rows share an entity that must not straddle the split.
Cross-Validation: A Practical Guide
A single train/test split gives you one number drawn from a spread you never see. Cross-validation shows you the spread.
Five splits instead of one
The same model fitted five times, each time holding a different fifth back.
example_01.pyscikit-learn
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_val_score
import numpy as np
X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
scores = cross_val_score(model, X, y, cv=5)
print("one score per fold:", np.round(scores, 4))
print("mean :", round(scores.mean(), 4))
print("std :", round(scores.std(), 4))
Output
Why one number was never enough
Six different single splits of the same data, and the score you would have reported depends on which one you happened to run.
example_02.pyscikit-learn
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
import numpy as np
X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
scores = []
for seed in range(6):
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2,
random_state=seed, stratify=y)
scores.append(model.fit(X_tr, y_tr).score(X_te, y_te))
print("six single splits:", np.round(scores, 4))
print("lowest %.4f, highest %.4f - a spread of %.4f"
% (min(scores), max(scores), max(scores) - min(scores)))
print()
print("any one of those could have been 'the' test score.")
Output
Stratified folds, and what plain folds do instead
Watch the third plain fold: it contains none of the minority class at all.
example_03.pyscikit-learn
from sklearn.model_selection import KFold, StratifiedKFold
import numpy as np
y = np.array([0] * 15 + [1] * 5) # 75/25, five in the minority
print("KFold (splits by position, ignoring the classes):")
for _, te in KFold(n_splits=5, shuffle=True, random_state=0).split(y, y):
print(" test labels:", y[te], " ones:", int(y[te].sum()))
print()
print("StratifiedKFold (each fold mirrors the whole):")
for _, te in StratifiedKFold(n_splits=5, shuffle=True, random_state=0).split(y, y):
print(" test labels:", y[te], " ones:", int(y[te].sum()))
Output
Several metrics, and the training score too
cross_validate is the fuller version, and the train column is how you see overfitting.
example_04.pyscikit-learn
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_validate
import numpy as np
X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
out = cross_validate(model, X, y, cv=5,
scoring=["accuracy", "precision", "recall"],
return_train_score=True)
for key in sorted(out):
print("%-22s %s" % (key, np.round(out[key], 3)))
Output
Preprocessing outside the loop invents skill
Pure noise, nothing to learn, and one of these two numbers says the model found something.
example_05.pyscikit-learn
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score
import numpy as np
# Pure noise: there is nothing to learn, so honest accuracy is about 0.5.
rng = np.random.RandomState(0)
X = rng.normal(size=(120, 2000))
y = rng.randint(0, 2, size=120)
# WRONG: choose the best features using every row, then cross-validate.
picked = SelectKBest(f_classif, k=20).fit_transform(X, y)
wrong = cross_val_score(LogisticRegression(max_iter=1000), picked, y, cv=5)
# RIGHT: the selection happens inside each fold, on the training part only.
right = cross_val_score(
make_pipeline(SelectKBest(f_classif, k=20),
LogisticRegression(max_iter=1000)), X, y, cv=5)
print("selected before splitting :", round(wrong.mean(), 3))
print("selected inside the folds :", round(right.mean(), 3))
print()
print("the data is random. one of those numbers is a hallucination.")
Output
The other splitters
cv= takes a number or a splitter object, and the object is how you handle time and groups.
example_06.pyscikit-learn
from sklearn.model_selection import LeaveOneOut, KFold, TimeSeriesSplit
import numpy as np
X = np.arange(6).reshape(-1, 1)
print("KFold(3) :", [list(te) for _, te in KFold(3).split(X)])
print("LeaveOneOut :", [list(te) for _, te in LeaveOneOut().split(X)])
print()
print("TimeSeriesSplit(3) - train always precedes test:")
for tr, te in TimeSeriesSplit(n_splits=3).split(X):
print(" train", list(tr), "-> test", list(te))
Output
What the k numbers tell you
The mean is the headline, and the standard deviation is the part people skip.
A mean of 0.98 with a standard deviation of 0.006 is a stable, believable result. A mean of 0.98 with a standard deviation of 0.09 says the model works well on some subsets and badly on others. This Is a fully different situation and often means either very little data or a subgroup the model handles poorly.
Two models whose ranges share heavily aren't distinguishable by this evidence, what their means. That single habit. Looking at the spread before believing a difference. Blocks a large share of the wasted effort in applied machine learning.
cross_val_score returns the array. Printing it rather than only its mean costs nothing and is where the info is.
How many folds
cv=5 is the usual default and cv=10 the other common choice. The trade between them is straightforward.
More folds mean each model trains on more data. Each is closer to the model you'd build dataset. The estimate is less pessimistic. More folds also mean more fits. Ten-fold takes twice if five-fold. And the training sets share more. This Makes the scores more linked and the standard deviation an underestimate of the true variability.
The extreme is LeaveOneOut. Here k equals the number of samples. each model trains on all but one row. This Is as close to the full-data model as possible. It costs n fits. It's worth it only on very small datasets. Its variance estimate is poor for the same link reason.
Five is a reasonable default for most work. Ten when the dataset is small enough that the extra fits are cheap and you want the training sets larger. RepeatedStratifiedKFold runs the whole thing a few times with different shuffles when you need a more solid guess of the spread.
Stratification happens by default, and only for classifiers
Pass an int as cv and scikit-learn chooses the splitter for you: StratifiedKFold when the estimator is a classifier, plain KFold or else.
That default is a good one and worth understanding rather than leaning on blindly. The editor above shows what plain folds do to an imbalanced target. With five minority samples across five folds, the counts come out 2, 1, 0, 1, 1. One fold holds none of the class at all. The recall found on it's undefined and the score for that fold is meaningless. Stratified folds give just one to each.
Note that stratification is on the target. When the thing that must stay weighed is a thing else. A site, a batch, a demographic group. You need to pass a splitter object rather than an int.
The rule that makes it honest
Cross-validation is only honest if all learned from data is learned inside the loop.
That includes the model, obviously. It also includes the scaler's mean and standard deviation, the imputer's median, the encoder's list of categories, the vectoriser's vocab, and. Most damagingly — any feature pick.
The fifth editor makes the cost concrete. The data is pure random noise with no tie between features and target. The honest fit is 0.5. Selecting the twenty "best" features using the whole dataset and then cross-validating gives 0.825. The pick looked at the held-out rows, found the columns that happened to correlate with the target in those rows too. Handed the model a shortcut. Doing the pick inside the folds gives 0.45. This Is the truth.
That isn't a small distortion, and it's the single most common way published results turn out to be wrong. With 2000 pick features and 120 samples, some columns correlate with the target by chance. Choosing them using all the data is choosing them partly on the test rows.
Pipeline is the mechanism that blocks this. A pipeline is a single estimator as far as cross_val_score is concerned. Its fit. Including each transformer inside it. Runs on the training part of each fold only. Using one isn't a style preference. It's what makes the number mean what it says.
What cross-validation does not give you
It doesn't give you a model. The k models fitted during the process are scored and thrown away.
What it gives you is an guess of how well the method performs. This prep, this estimator, these settings, on data like this. Once you've decided the method is good enough, you fit it once more on all the data. That is the model you keep. cross_val_predict returns the out-of-fold guesses if you want them for inspection. It's by name not a way of making a model either.
It also doesn't replace a final held-out test set when you've been using cross-validation to make choices. Checking twenty setting settings by cross-validation and reporting the best one's score is optimistic for the same reason as picking the best of twenty test scores. The winner was selected for doing well on those given folds. Nested cross-validation is the rigorous answer. A separate untouched test set is the hands-on one.
The splitters worth knowing
cv accepts an int or a splitter object. The objects are how you say a thing the int can't.
StratifiedKFold and KFold are the defaults. Passing them by name lets you set shuffle=True and a random_state. This The int form doesn't. Note that KFold does not shuffle by default. Data arriving in a sorted order makes folds that are each a contiguous block. At times a disaster no one notices.
TimeSeriesSplit trains on the past and tests on what follows, never the reverse, with a training set that grows each time. It's the only correct choice for sorted data.
GroupKFold and StratifiedGroupKFold take a groups array and keep each row of a group on one side. Use them when a few rows show the same patient, customer, doc or device. Or else the model recognises the entity rather than learning the pattern. The score is inflated by an amount nothing reveals.
ShuffleSplit draws random train/test pairs without the each-row-tested-once ensure. This Is handy when you want many guesses from a large dataset cheaply.
What it costs, and when that matters
Cross-validation fits the model k times. It costs roughly k times a single fit. For a logistic regression on a few thousand rows that is imperceptible. For a large ensemble on a large dataset it can turn a two-minute test into twenty.
Three ways to keep it affordable. n_jobs=-1 runs the folds in parallel across cores. Since the folds are free this is close to free speed on any machine with more than one. Fewer folds. Three rather than ten. When you're exploring rather than reporting. And a subsample of the data while you're iterating, with the full run reserved for the result you intend to quote.
The mistake worth avoiding is skipping cross-validation because it's slow and going back to a single split. A single split isn't faster in any way that matters. It's the same fit once. It buys a number you can't calibrate. If the budget is truly tight, three folds on the full data beats one split each time.
Note also that cross_validate reports fit_time and score_time per fold. This Is the cheapest way to find out where the time is really going before optimising any of it.
Reading the training scores
return_train_score=True adds a column that is worth the extra work. The gap between train and test is the clearest diagnostic in the whole library.
Train high, test high, small gap. The model has learned a thing that generalises. Nothing to fix.
Train very high, test much lower. Overfitting. The model has stored patterns exact to the training rows. The response is more smoothing, a simpler model, or more data.
Train low, test low, small gap. Underfitting. The model can't capture the structure even on data it has seen. The response is the opposite. A more flexible model, better features, less smoothing.
Train lower than test. Unusual and worth investigating rather than celebrating. It happens legitimately when smoothing or dropout is active during training but not scoring. Illegitimately when the folds aren't close.
The two failure modes need opposite treatments. This Is why guessing between them wastes so much time. One extra arg tells you which one you've.
Does KFold shuffle by default? No. Data arriving sorted makes folds that are each a contiguous block. This Can be a silent disaster. Pass a splitter object with shuffle=True when the order might mean a thing.
Can I cross-validate a pipeline? Yes. You should. A pipeline is a single estimator to cross_val_score. This Is just what keeps its transformers inside the folds.
Why is my cross-validated score lower than my test score? often because each fold trains on less data than a single 80/20 split does. The estimate is slightly pessimistic. A large gap suggests the split was lucky.
Things to try with Cross-Validation
Run the second editor. Six seeds, six answers. That spread is the reason the rest of the page exists.
Look at the spread. In the first editor, print scores.max() - scores.min() and compare it against the difference between two models you're considering.
Run the leakage demo. The data is random. Sit with that the wrong version reports 0.825.
Watch overfitting appear. In the fourth editor, swap the pipeline for DecisionTreeClassifier() and compare the train and test columns.
Questions people ask about Cross-Validation
How many folds should I use? Five or ten is the usual choice. More folds cost more time and reduce bias slightly, while fewer are quicker but noisier.
What does a large spread between fold scores mean? The estimate is unstable. The data may be small or uneven, so report the mean together with the standard deviation.
Does cross_val_score train a final model? No. It fits throwaway copies on each fold and returns scores. You still call fit on the full training set afterwards.
Should preprocessing be inside the folds? Yes. Put scalers and imputers in a pipeline so they are fitted on each training fold only, which prevents leakage.
When is plain KFold wrong? With grouped rows, time series or imbalanced classes. Use GroupKFold, TimeSeriesSplit or StratifiedKFold instead.
Do I still need a test set? Yes. Cross-validation guides choices during development, and a final untouched test set gives the report number.
Where this leaves you
cross_val_score for a quick guess with its spread, cross_validate when you want a few metrics or the training scores, a Pipeline around all that learns. A splitter object when the data has time or groups in it. The mean is the headline. The spread is what tells you whether to believe it.
Check yourself
0 of 4
Answer without scrolling back up.
What does cross_val_score return?
An array with one entry per fold. The spread across them matters as much as the mean, and the fitted models are discarded.
Why must preprocessing go inside a Pipeline for cross-validation?
Anything fitted before the split has seen every fold. With feature selection on noise this took a true 0.45 to a reported 0.825.
You pass cv=5 with a classifier. Which splitter is used?
scikit-learn picks StratifiedKFold for classifiers and plain KFold otherwise. Pass a splitter object when you need shuffling or a fixed seed.
After cross-validating, which model do you deploy?
Cross-validation estimates how a procedure performs. Once you accept the procedure, refit it once on everything - the k models are thrown away.
Cheat sheet
Cross-Validation
Replace a single train/test split with cross-validation to see the variance behind every score and pick a model you can trust.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.