vizlearn
Home / Preprocessing

Train, Validation & Test Split

Machine LearningBy Updated

Visualize how datasets are partitioned for model training and evaluation.

Ratios

70%
15%
15%
Total 100%

Dataset (n=100)

Train
Validation
Test

Distribution

Training Set
70
Model Learning
Validation Set
15
Hyperparameter Tuning
Test Set
15
Final Evaluation

Overview

A model can memorize its training examples, so its training score does not tell us how well it handles new cases. A held-out set estimates that generalization. The split must match the intended prediction task: random rows for independent examples, groups for repeated entities, and time-ordered data for forecasting.

The three roles

  • Training set: fits model parameters and preprocessing steps.
  • Validation set: compares models, selects hyperparameters, and sets a decision threshold.
  • Test set: gives a final estimate after those choices are complete.

A 70/15/15 split is one starting point, not a rule. With little data, cross-validation can make better use of training examples; keep a final test set if you need an independent final check. For a large dataset, a single held-out validation set may be adequate.

Read the visualization

The grid assigns each row to Train, Validation, or Test. Change the three proportions and the counts update; press Shuffle to see a different assignment. Try 70% Train, 15% Validation, and 15% Test, then reduce Test: more training data can help fit the model, but the final estimate becomes noisier when fewer examples remain for evaluation.

Shuffling is appropriate only when rows are exchangeable. It can place near-duplicates or related records in both train and test, making the evaluation too easy. For customers with repeated purchases, split by customer. For future prediction, train on earlier dates and test on later dates. Stratification can preserve class proportions in a random classification split, but it does not fix grouping or time leakage.

Keep information on the right side of the split

Split first, then fit transformations using training data only. A scaler, imputer, feature selector, or resampler must not learn from validation or test rows. In cross-validation, fit each transformation separately inside each training fold; a scikit-learn Pipeline helps enforce that order.

The test set is for the final report. Repeatedly checking it while changing features, thresholds, or models leaks information through those decisions. If you use its results to choose a change, it has become validation data; collect a new test set for an unbiased final estimate.

Worked example: predict next month’s churn

Suppose records include customers observed from January through December, and the goal is to predict January churn. Randomly splitting customer-month rows can put December for one customer in training and an earlier month for the same customer in test. It can also train on dates later than the period being evaluated.

Instead, define the prediction date and information available then. Train on earlier months and evaluate on a later time window. If the deployment goal is generalizing to entirely new customers, also hold out customers by group. The split should reproduce the deployment question; there is no universally correct ratio or shuffle setting.

Common mistakes with Train-Test Split

  • Using the test set to tune: this makes the reported score optimistic.
  • Preprocessing before splitting: test-set statistics enter training.
  • Randomly splitting time series: future information can leak into the past.
  • Ignoring repeated entities: related records across splits inflate performance.
  • Treating one score as precise: small test sets have substantial sampling uncertainty.

Sources and next steps

Predict, then reveal

About to run: set Train to its minimum (10). Before it does — what happens to the readout?

Committing to an answer first is the point — the reveal runs the experiment on the visualisation above and reads the real value back, so nothing here is scripted.

Check yourself

0 of 3

Answer without scrolling back up.

  1. Why hold back a test set at all?

  2. You scale your features using statistics from the whole dataset, then split. What have you done?

  3. Your test set is tiny - say 20 rows. What is the main problem?

Cheat sheet

Train-Test Split Method

See why testing on data you trained on flatters a model, and how holding out a test set gives you an honest score.

MACHINE LEARNING · vizlearn.in/machine_learning/train_test_split.html

More in Machine Learning

Machine LearningFeature ScalingMachine LearningData LeakageMachine LearningHandling Missing ValuesMachine LearningOutliers and Influence

About the author

Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.

About VizLearnGitHubLinkedInContact