Visualize how datasets are partitioned for model training and evaluation.
Overview
Why Split Data? The Core Principle
Imagine you're studying for an exam. If you study using the exact same questions that will be on the final test, you might get a perfect score. But does that mean you've truly learned the material? Not necessarily. You've just memorized the answers. Machine learning models can do the same thing—a phenomenon called overfitting.
To build a model that performs well on new, unseen data, we must split our dataset into distinct sets, each with a specific purpose.
Ratios
70%
15%
15%
Total100%
Dataset (n=100)
Train
Validation
Test
Distribution
Training Set
70
Model Learning
Validation Set
15
Hyperparameter Tuning
Test Set
15
Final Evaluation
A Guide to Splitting Data: Train, Validation, and Test Sets
Understand the most critical step in building a reliable machine learning model.
The Training Set
This is the largest part of your dataset, typically 60-80%. The model uses this data to learn the underlying patterns, relationships, and features. It's the "study material" for your model. The model sees both the input features and the correct answers (labels) and adjusts its internal parameters to make accurate predictions.
The Validation Set
This is a smaller portion of the data (around 10-20%) that the model does not learn from directly. Instead, we use it to tune the model's hyperparameters. Hyperparameters are the model's configuration settings, like the learning rate in a neural network or the number of trees in a random forest. We train the model on the training set and then evaluate its performance on the validation set. We repeat this process with different hyperparameter settings and choose the combination that performs best on the validation data. This is like taking practice quizzes to see which study techniques are working best.
The Test Set
This final piece of the dataset (around 10-20%) is the "final exam." The model has never seen this data before—not during training and not during hyperparameter tuning. We use the test set only once, at the very end, to get an unbiased estimate of how our final, tuned model will perform in the real world. This score tells us how well the model is expected to generalize to new, unseen data.
Why the split exists at all
A model's score on the data it was trained on answers the wrong question. It measures how well the model memorised, and memorising is easy — a lookup table scores 100% and predicts nothing.
The question you actually care about is: how will this behave on data it has never seen? The only honest way to answer it is to hide some data from the model, then reveal it once, at the end.
The mechanics are simple. Shuffle the rows, take some fraction — 20% is the usual default — and set it aside. Train on the rest. Score on the part you set aside.
Everything difficult about this is in the word "hide". The test set has to be genuinely untouched: not used to choose features, not used to tune a threshold, not used to decide when to stop training, and not peeked at more than a handful of times over a project's life. Every look leaks a little information from the test set into your decisions, and by the twentieth look you are fitting the test set with your own judgement.
Three splits, three jobs
Once you start tuning anything, two splits are not enough. The standard arrangement is three:
Training set (~60–80%) — the model fits its parameters here.
Validation set (~10–20%) — you compare models, choose hyperparameters, pick a threshold, decide when to stop. Look at this as often as you like.
Test set (~10–20%) — touched once, at the end, to produce the number you report.
The distinction between the second and third is the one people collapse, and it is the one that causes inflated results. If you chose your settings by looking at a set, its score is no longer an unbiased estimate of anything — it has become part of the training procedure. Cross-validation can replace the fixed validation set, but it does not replace the final untouched test set.
How big should the test set be? Big enough that the score is stable. Rough guide: a few thousand rows in the test set gives a confidence interval of a percentage point or two on accuracy; a few hundred gives you five or more. With 10 million rows you can happily hold out 1% and still measure precisely.
The splits that are not random
Random splitting assumes every row is independent. When that assumption is false, a random split produces a score that looks superb and does not survive contact with reality.
Time series. Never shuffle. Train on the past, test on the future, exactly as the model will be used. A shuffled split lets the model see Thursday while predicting Wednesday, which is not a skill it will have in production.
Grouped data. Multiple rows per patient, per user, per device. Split by group, not by row — otherwise the same patient appears on both sides and the model recognises the individual rather than the condition. GroupKFold and GroupShuffleSplit do this.
Imbalanced classes. Use stratify=y so both sides keep the same class proportions. Without it, a rare class can be missing from the test set entirely.
Duplicates and near-duplicates. Deduplicate before splitting. Two copies of the same record landing on opposite sides is a leak with a friendly face.
random_state is not a formality. Without it, every run reshuffles, your metrics wander, and you cannot tell whether a change helped or the split moved.
Experiment with the Visualization
The interactive panel demonstrates these concepts visually:
Adjust the Ratios: Use the sliders to change the percentages of the train, validation, and test sets. Notice how the counts and the colors in the data grid update instantly. A common split is 70% train, 15% validation, and 15% test.
The Importance of Shuffling: Click the "Shuffle" button. The data points are randomly reassigned to the different sets. This is a crucial step in practice. If your data is ordered (e.g., by date), failing to shuffle could mean your test set contains only the most recent data, leading to a biased and inaccurate evaluation of your model's performance.
Two-Way vs. Three-Way Split: Set the validation slider to its minimum. This simulates a simple train-test split, which is common for basic models that don't require hyperparameter tuning. However, for most modern machine learning tasks, the three-way split is the gold standard.
The Golden Rule: No Data Leakage
The most important rule in data splitting is to prevent data leakage. This happens when information from outside the training set is used to create the model. The most common form of leakage is "peeking" at the test set. If you use the test set to make any decisions—like choosing which model to use or which hyperparameters to tune—you have contaminated it. Your final evaluation will be overly optimistic, and your model will likely fail in the real world. The test set must remain a sacred, untouched resource until the very end.
Leakage: the failure this is meant to prevent
Data leakage is when information from outside the training set reaches the model, and it is the single most common reason a model that scored 0.95 in a notebook scores 0.6 in production. The split is your main defence, and it only works if the ordering of operations is right.
The classic mistakes:
Scaling before splitting.StandardScaler().fit(X) on the full dataset computes means that include the test rows. Fit on training data only, then transform the rest.
Imputing before splitting. Same problem: the median you fill with was computed partly from test data.
Feature selection before splitting. Choosing the top 20 features by their correlation with the target, across the whole dataset, is choosing them with the test answers in hand.
Oversampling before splitting. SMOTE creates synthetic points from real ones; if it runs first, synthetic relatives of test rows end up in training.
A feature that encodes the future. A days_since_last_payment column computed at export time, or a customer status field updated after the event you are predicting. No split protects you from this — only reading the column definitions does.
The structural fix for the first four is a Pipeline, which forces every preprocessing step to be fitted inside the training fold. It is not a style preference; it is the mechanism that makes the split mean something.
Why the split has to happen first
The same model scored on data it has seen and data it has not. The gap is the entire reason the split exists.
example_01.pyscikit-learn
import numpy as np
from sklearn.datasets import make_classification
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
X, y = make_classification(n_samples=2000, n_features=20, n_informative=6,
flip_y=0.15, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
print("train %d rows, test %d rows" % (len(Xtr), len(Xte)))
print()
tree = DecisionTreeClassifier(random_state=0).fit(Xtr, ytr)
print("a tree grown to full depth:")
print(" accuracy on data it trained on : %.4f" % accuracy_score(ytr, tree.predict(Xtr)))
print(" accuracy on data it never saw : %.4f" % accuracy_score(yte, tree.predict(Xte)))
print(" it memorised the training set. the second number is the real one.")
print()
print("depth-limited, the gap closes and the honest score goes up:")
print("%8s %12s %12s" % ("depth", "train", "test"))
for d in (2, 3, 5, 8, 12, None):
t = DecisionTreeClassifier(max_depth=d, random_state=0).fit(Xtr, ytr)
print("%8s %12.4f %12.4f"
% (d, accuracy_score(ytr, t.predict(Xtr)), accuracy_score(yte, t.predict(Xte))))
print()
print("the split size is a trade, and small test sets are noisy:")
for frac in (0.1, 0.2, 0.3, 0.5):
scores = []
for seed in range(30):
a, b, c, d2 = train_test_split(X, y, test_size=frac, random_state=seed)
m = DecisionTreeClassifier(max_depth=5, random_state=0).fit(a, c)
scores.append(accuracy_score(d2, m.predict(b)))
print(" test_size=%.1f (%4d rows): mean %.4f, spread across seeds %.4f"
% (frac, int(frac * len(X)), np.mean(scores), np.std(scores)))
print()
print("a 10% test set moves around nearly twice as much as a 50% one. that")
print("spread is why one split is a weak estimate, and why cross-validation exists.")
Output
Questions people ask
What split ratio should I use? 80/20 for small datasets, 70/15/15 when you need a validation set, and much smaller test fractions when you have millions of rows. Absolute size matters more than the percentage.
Should I retrain on all the data before deploying? Usually yes, once the evaluation is finished and you have decided on the model. More data is better, and you already have your honest estimate from the held-out run.
What if my test score is much better than my training score? Suspect a bug. Common causes: the test set is easier by accident, augmentation is applied only to training, or the split leaked in a way that made test rows familiar.
Do I need a split if I use cross-validation? Cross-validation covers the validation role well. Keeping one final untouched test set is still the right practice for any number you report externally.
How often can I look at the test set? As few times as you can manage. Once is the ideal. Each additional look, followed by a change, converts a little of it into training data.
Is random_state=42 special? Only culturally. Any fixed number gives reproducibility; the point is that it is fixed, not which number it is.
Recap in one screen
Score on data the model has never seen, or you are measuring memorisation.
Train fits, validation chooses, test reports — and test gets looked at once.
Stratify for imbalance, group for repeated subjects, and split chronologically for time series.
Every preprocessing step belongs inside the training fold; use a pipeline to enforce it.
Set the random seed, deduplicate first, and be suspicious of a test score that looks too good.
A checklist before you trust a score
Run through these six questions whenever a model's test number is about to leave your machine:
Was the test set touched during development? Feature choice, threshold selection, early stopping and "just checking" all count.
Was every preprocessing step fitted inside the training data? Scalers, imputers, encoders, feature selectors and samplers.
Could a row in test have a near-duplicate in training? Repeated customers, augmented copies, the same document scraped twice.
Does the split respect time? If predictions will be made about the future, the test set must be the future.
Does any feature contain information unavailable at prediction time? Fields updated after the outcome are the classic case, and no split protects against them.
Is the test set big enough for the difference you are claiming? A 1% improvement measured on 200 rows is noise.
Every one of these has ended a project's credibility somewhere. The first three are mechanical and a pipeline enforces them. The last three need someone to read the column definitions and think, which is why they are the ones that survive into production.
Check yourself
0 of 3
Answer without scrolling back up.
Why hold back a test set at all?
Training accuracy measures memorisation as much as learning. The only honest estimate of future performance comes from data the model has never been fitted on.
You scale your features using statistics from the whole dataset, then split. What have you done?
The scaler saw the test set's mean and range, so the test score is no longer clean. Fit the scaler on the training split only, then apply it to the test split.
Your test set is tiny - say 20 rows. What is the main problem?
With 20 rows, one extra mistake moves accuracy by five whole percentage points. The estimate has such wide error bars that it cannot distinguish a good model from a mediocre one.
Cheat sheet
Train-Test Split Method
Imagine you're studying for an exam. If you study using the exact same questions that will be on the final test, you might get a perfect score. But does that mean you've truly learned the material? Not necessarily. You've just memorized the answers. Machine learning models can do the same thing—a phenomenon called overfitting.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.