Interactive explorer to visualize K-Fold splitting and validation logic.
Overview
What a single split cannot tell you
Hold out 20% of the data, train on the rest, and you get an accuracy figure. Change the random seed and you get a different one — sometimes several points different on a small dataset.
Neither number is wrong; both are estimates from a sample of test rows, and a sample of 200 rows carries real sampling error. With one split you have no way to tell whether a two-point difference between two models is a genuine improvement or the luck of the partition.
Configuration
5
Higher k creates more folds but requires more training iterations.
Visualization
Fold: - / -
Train Data
Test Fold
Results
Ready to start. Click 'Run Next Fold'
Average Accuracy
--%
K-Fold Cross Validation: A Practical Guide
One train-test split gives you one number, and that number depends on which rows happened to land in the test set. K-fold cross-validation uses every row for testing exactly once and reports the spread as well as the average.
How k-fold works
Split the data into k equally sized folds. Then run k separate experiments: hold out fold 1 and train on folds 2…k, hold out fold 2 and train on the rest, and so on. Each fold serves as the test set exactly once, and each row is predicted exactly once by a model that never saw it.
You end up with k scores. Report the mean as the performance estimate and the standard deviation as a measure of how stable it is. The standard deviation is the part a single split cannot give you, and it is often the more informative number: a model scoring 0.82 ± 0.01 is a very different proposition from one scoring 0.82 ± 0.09.
k = 5 or k = 10 are the standard choices. Larger k means more training data per fold (less pessimistic bias) but k model fits and more correlated training sets. The extreme, k = n, is leave-one-out: nearly unbiased, high variance, and usually far too expensive.
Why one number is not an answer
Suppose you split 1,000 rows into 800 for training and 200 for testing, and the model scores 87%. Report that and you have said something slightly dishonest, because you have not said how much of the 87% was luck.
Shuffle the data differently and try again: 84%. Again: 89%. Again: 86%. The model did not change. What changed is which awkward rows happened to land in the test set.
The spread across those runs is the thing a single split hides. A model that scores 87% ± 1% and a model that scores 87% ± 6% are not equally trustworthy, and only the second one is going to embarrass you in production. Cross-validation exists to put the ± on the number.
It also fixes a quieter waste. With one split, 20% of your data only ever gets tested on and 80% only ever gets trained on. When data is expensive — medical records, labelled defects, anything a human had to annotate — throwing away a fifth of it is a real cost. K-fold uses every row for both jobs, just never at the same time.
A worked example with 100 rows
Take 100 rows and k = 5. Cross-validation shuffles them, cuts them into five blocks of 20, and runs five separate experiments:
Run
Trained on
Tested on
Score
1
Blocks 2,3,4,5
Block 1
0.86
2
Blocks 1,3,4,5
Block 2
0.91
3
Blocks 1,2,4,5
Block 3
0.79
4
Blocks 1,2,3,5
Block 4
0.88
5
Blocks 1,2,3,4
Block 5
0.84
The headline is the mean: (0.86 + 0.91 + 0.79 + 0.88 + 0.84) / 5 = 0.856. The standard deviation is about 0.04, so the honest summary is "roughly 0.86, give or take 0.04".
Block 3 is worth a look on its own. A fold that scores well below the others is not noise to be averaged away — it usually means those twenty rows contain something the others do not. A different time period, a different hospital, a different product line. Open them.
Note what five folds cost: five models trained instead of one. That is the whole downside, and it is why k = 5 or k = 10 are the standard choices. Ten folds give a slightly better estimate for double the compute, and beyond that the returns are tiny.
Choosing k, and the leave-one-out extreme
k = 5 — the everyday default. Each model trains on 80% of the data, and you pay five times the training cost.
k = 10 — the classic recommendation. Each model sees 90% of the data, so the estimate is a little less pessimistic, and the variance across folds is usually smaller.
k = n (leave-one-out) — every single row takes a turn as the test set. Almost unbiased, but you train n models, and the folds are so similar to each other that the estimate can be surprisingly unstable. Reserve it for very small datasets.
There is a real tension here. Small k means each model trains on less data, so every fold's score is pessimistic compared to the model you will finally ship on 100% of the data. Large k reduces that pessimism but makes the training sets nearly identical, which means the fold scores stop being independent estimates. Five and ten are compromises that have survived decades of practice.
For very small datasets, repeated k-fold is often the better move: run 5-fold cross-validation five times with different shuffles and average the twenty-five scores. You get a much steadier estimate without the pathologies of leave-one-out.
Nested cross-validation, and why the ordinary kind can lie
Here is the trap that catches most people once. You use cross-validation to pick hyperparameters — try thirty settings, keep whichever scored best — and then report that best score as the model's performance.
That number is optimistic, and sometimes badly so. You searched thirty options against those exact folds and kept the luckiest. The score now includes the luck.
The fix is nested cross-validation. An outer loop splits off a test fold and never touches it for tuning. Inside each outer training set, an inner cross-validation picks the hyperparameters. The outer fold then scores that tuned model on data no part of the tuning ever saw.
from sklearn.model_selection import GridSearchCV, cross_val_score, KFold
inner = KFold(n_splits=5, shuffle=True, random_state=0)
outer = KFold(n_splits=5, shuffle=True, random_state=1)
search = GridSearchCV(model, param_grid, cv=inner) # picks the settings
scores = cross_val_score(search, X, y, cv=outer) # scores them honestly
print(scores.mean(), scores.std())
It costs k×k model fits, so it is not free. Use it when you are reporting a number someone will make a decision on. For day-to-day iteration, plain k-fold plus one genuinely untouched hold-out set is the pragmatic compromise.
The same discipline applies to preprocessing. Fit your scaler, your imputer and your feature selector inside each fold, by putting them in a Pipeline. Scaling the whole dataset before splitting leaks the test folds' means and standard deviations into training, and quietly inflates every score you produce.
One split lies; five splits argue
The same model and the same data, scored by many different single splits and then by cross-validation. The spread between splits is the point.
example_01.pyscikit-learn
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import (train_test_split, cross_val_score,
KFold, StratifiedKFold)
X, y = make_classification(n_samples=900, n_features=15, n_informative=5,
flip_y=0.15, random_state=0)
model = RandomForestClassifier(n_estimators=120, random_state=0)
single = []
for seed in range(12):
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25, random_state=seed)
single.append(model.fit(Xtr, ytr).score(Xte, yte))
print("twelve single 75/25 splits of the same data:")
print(" ", np.round(single, 4))
print(" worst %.4f, best %.4f, a spread of %.4f"
% (min(single), max(single), max(single) - min(single)))
print()
print("pick the seed you like and you can report either number honestly.")
print("that is the problem cross-validation solves.")
print()
scores = cross_val_score(model, X, y, cv=StratifiedKFold(5, shuffle=True,
random_state=0))
print("5-fold cross-validation:")
print(" fold scores", np.round(scores, 4))
print(" mean %.4f, sd %.4f" % (scores.mean(), scores.std()))
print()
print("every row is used for testing exactly once, and for training four times.")
print()
print("more folds means more training data per fold and more compute:")
for k in (3, 5, 10):
s = cross_val_score(model, X, y, cv=StratifiedKFold(k, shuffle=True,
random_state=0))
print(" k=%2d train rows per fold %4d mean %.4f sd %.4f"
% (k, int(len(X) * (k - 1) / k), s.mean(), s.std()))
print()
print("note which way the spread goes: more folds means a NOISIER set of fold")
print("scores, because each fold is scored on fewer rows. the mean gets more")
print("reliable while the individual numbers get less so.")
print()
print("stratified folds keep the class balance in each fold, which matters more")
print("the rarer the positive class is. it is the default for classifiers in")
print("scikit-learn, and you should not turn it off without a reason.")
Output
Guided experiments
Watch each fold take its turn. Set Folds (k) to 5 and press Run Next Fold repeatedly. A different block becomes the test set each time, and the remaining blocks train.
Compare the fold scores. Press Auto-Run All and look at the individual results, not just the average. The spread between the best and worst fold is exactly the uncertainty a single split would have hidden.
Increase k. Set Folds (k) to 10 and run again. Each model trains on 90% of the data instead of 80%, and there are twice as many fits to pay for.
Shuffle first. Press Shuffle and re-run. If the data arrived sorted by class, unshuffled folds can be wildly unrepresentative — which is why stratification exists.
Stratified, grouped, and time series
Plain k-fold splits at random, which is wrong in three common situations:
Imbalanced classes — a random fold might contain almost none of the minority class. Stratified k-fold preserves the class proportions in every fold, and should be the default for classification.
Grouped data — multiple rows per patient, user or session. If rows from the same group land in both train and test, the model has effectively seen the answer. GroupKFold keeps each group entirely on one side.
Time series — random folds train on the future to predict the past, which is leakage in its purest form. Use a forward-chaining split where each fold trains only on data preceding its test window.
Where this goes wrong
Preprocessing before splitting. Fitting a scaler, imputer or feature selector on the whole dataset leaks test information into training and inflates every fold. Put the preprocessing inside a Pipeline so it is refitted within each fold.
Random folds on time series. Produces excellent scores and a model that fails in production.
Reporting only the mean. The standard deviation is what tells you whether a difference between two models is meaningful.
Tuning hyperparameters on the same CV used to report the score. Selecting on those folds means the reported number is optimistic; nested cross-validation is the correct fix.
Forgetting to shuffle sorted data. Folds become unrepresentative before stratification can help.
What to remember
K-fold cross-validation trains k models, each holding out a different fold, so every row is tested once and you get a mean and a spread rather than one seed-dependent number. Use stratified folds for classification, grouped folds when rows share a subject, and forward-chaining for time series. Whatever the variant, fit every preprocessing step inside the fold — doing it beforehand leaks the test set and quietly invalidates the whole exercise.
Questions people ask
Do I still need a separate test set? Yes, if you can afford one. Cross-validation is for choosing between options; a final untouched hold-out is for the number you tell other people. Every decision you make while looking at the cross-validation scores erodes their independence a little.
Which model do I actually deploy? Not any of the k fold models. Once cross-validation has told you which method and which settings to use, retrain on all the data and ship that. The fold scores were an estimate of how that final model will behave.
Should I shuffle? Usually yes — data that arrived sorted by class or by date makes unshuffled folds wildly unrepresentative. The exception is time series, where shuffling destroys the point entirely.
What if the folds disagree a lot? Treat it as a finding. Wide spread means your dataset is small, heterogeneous, or contains groups the split is cutting through. Look at the worst fold's rows before you look at a different algorithm.
Is cross-validation useful for deep learning? Rarely, and only for small datasets. Training a large network five times is often unaffordable, and with hundreds of thousands of examples a single well-constructed validation set is stable enough.
Can I cross-validate on imbalanced data? Yes, with StratifiedKFold, which keeps each fold's class proportions the same as the whole. Without it, a rare class can be absent from a fold entirely and the score for that fold becomes meaningless.
Recap in one screen
One split gives one number; cross-validation gives a mean and a spread, and the spread is the part you were missing.
k-fold trains k models, testing each row exactly once. k = 5 or 10 is the normal choice.
Stratify for classification, group for repeated subjects, and never shuffle time series.
Fit every preprocessing step inside the fold, using a pipeline, or you leak.
Tuning on the same folds you report inflates the result — nest the loops when the number matters.
Retrain on everything before you deploy.
Check yourself
0 of 3
Answer without scrolling back up.
What problem does k-fold cross-validation solve?
Every row gets to be in the test set exactly once, so the score is averaged over k splits instead of resting on one lucky or unlucky partition.
For an imbalanced dataset, which variant should you reach for?
Stratification preserves the class ratio in every fold. Without it a rare class can be absent from a fold entirely, making that fold's score meaningless.
What is the main cost of cross-validation?
10-fold means ten training runs. That is usually fine for classical models and often prohibitive for large deep networks, which is why they typically use a single held-out set.
Cheat sheet
K-Fold Cross Validation
Hold out 20% of the data, train on the rest, and you get an accuracy figure. Change the random seed and you get a different one — sometimes several points different on a small dataset.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.