Every module in this track, applied once to one dataset, in the order you would actually do it.
Overview
The order
Look. Split. Build. Check. Tune. Report.
Most of what goes wrong in applied machine learning is this sequence performed in a different order — tuning before checking a baseline, splitting after preprocessing, choosing a metric after seeing the scores. Each of those produces a number that cannot be trusted, and none of them produces an error.
The dataset here is small and synthetic: four columns of mixed type, 8% missing values in one of them, and an 8% positive rate. That combination is deliberately ordinary, because the point is the procedure rather than the result.
Worth knowing
The editors on this page run in order and share one interpreter - it is a single script, split into steps.
Look, split, build, check, tune, report. The order matters more than any individual choice in it.
Everything that learns from data lives inside the pipeline, so the folds and the test set stay honest.
The metric is chosen from the class balance, and the baseline is fitted before the model is believed.
The threshold is a parameter: chosen on the development set, saved with the model.
The test set is opened once, after every decision has been made.
A Complete Workflow: A Practical Guide
Twenty-two modules, applied to one small dataset in the order you would actually do it. Nothing here is new; the sequence is the point.
Look at the data first
Four counts, before any modelling decision is made.
example_01.pyscikit-learn
Output
Split, and put the test set away
Stratified, because 8% is not enough to leave to chance.
example_02.pyscikit-learn
Output
Build the pipeline
Two branches, four learned transformations, one estimator - and nothing fitted outside it.
example_03.pyscikit-learn
Output
Compare against doing nothing
The baseline first, and a metric chosen for an 8% positive rate.
example_04.pyscikit-learn
Output
Tune the model and the preprocessing together
Eight random samples, cross-validated, optimising the metric that matters.
example_05.pyscikit-learn
Output
Choose the threshold, then open the test set
Once. The numbers this prints are the ones you are entitled to report.
example_06.pyscikit-learn
Output
1. Look
Four counts, before any decision: the shape, the dtypes, the missing values per column, and the class balance.
Each one determines a later choice. The dtypes say which columns need encoding and which need scaling. The missing count says whether an imputer is needed and in which branch. The class balance — 8% here — rules out accuracy as a metric and argues for a stratified split.
This step takes four lines and is the one most often skipped. Every decision that follows is made in the dark without it.
2. Split
Immediately, and stratified.
The test set is set aside now, before anything has been fitted, scaled, imputed or selected. That is what makes the number at the end mean something, and it is the single most important line in the script.
Stratifying matters at 8%: an unstratified 20% test set could easily land with a class rate of 5% or 11%, and every metric computed from it would be shifted by an amount nothing reveals.
What remains is the development set, and everything up to the final step uses only that. Where this module says "development set", a larger project would cross-validate within it rather than splitting again.
3. Build the pipeline
Two branches, because the columns need different treatment.
Numeric columns get an imputer — with add_indicator=True, since a missing charge may itself be informative — and then a scaler, because the model is regularised and a penalty on coefficient size needs comparable units. Categorical columns get one-hot encoding with handle_unknown="ignore", so a region absent from a training fold does not crash the run.
class_weight="balanced" goes on the model, because the positives are rare.
The important property is that nothing is fitted outside this object. The imputer's median, the scaler's mean, the encoder's category list and the model's coefficients are all learned inside fit, which means they are all refitted on the training part of every cross-validation fold. The leakage module measured what happens otherwise.
Printing the feature names after preprocessing is a cheap sanity check: ten columns from four, and reading them confirms the missing indicator survived and the categories expanded as expected.
4. Check against a baseline
Two numbers, and the first one is the dummy.
DummyClassifier(strategy="most_frequent") scores an average precision of 0.082 — exactly the base rate, which is what average precision does for a model with no skill. The pipeline scores 0.478. That gap is the evidence that the model has learned something, and without the first number the second is uninterpretable.
The metrics are chosen for the problem rather than accepted from the default. Average precision because positives are rare and ROC AUC would flatter both models. Recall and precision alongside it, because the single summary hides the trade.
make_scorer(precision_score, zero_division=0) appears because the dummy never predicts a positive, which makes precision undefined. Saying what to return is better than five warnings in the output.
5. Tune
A randomised search over the model's regularisation and the preprocessing's imputation strategy, scored on average precision.
Two things are worth noticing. The search covers a preprocessing choice as well as a model one, which is possible only because the preprocessing is inside the pipeline. And scoring="average_precision" is passed explicitly, because the default would optimise accuracy — and on 8% positives, accuracy prefers a model that predicts nothing.
best_score_ comes out at 0.4829. That number is optimistic: it is the best of eight tries on the same folds, and the hyperparameter-search module measured how much that inflates things. It is a guide for choosing, not a result to report.
6. Choose the threshold, then report
The threshold is a parameter, so it is chosen on the development set — here by maximising F1 along the precision-recall curve, though a cost calculation would be better if the costs were known.
Then the test set is opened, once. Average precision on it is 0.5283, and the classification report gives the per-class picture at the chosen threshold: recall 0.667 and precision 0.468 on the positive class, from 33 positives in 400 rows.
Those numbers are the deliverable. They are honest because nothing about them was chosen after seeing them.
Finally the model is saved with its threshold. A pipeline pickled alone would be loaded elsewhere and used through predict(), silently reverting to 0.5 — which the thresholds module showed can be the difference between catching most of the positives and almost none.
What is missing, and would come next
This is a complete workflow, not a complete project.
A second model family. Gradient boosting would very likely beat this linear one, and the whole script would change by one line, because the pipeline interface does not care what the final estimator is.
More features. The learning-curve check from the overfitting module would say whether more data helps, and feature work usually beats tuning when it does not.
The timing question. Every column here was known at prediction time by construction. On real data that has to be asked of each one, and it is the check no code performs.
Monitoring. A model is fitted on the past and used on the future, and the distributions drift. What is measured after deployment matters as much as anything measured before it.
The checklist, without the code
The same procedure, stated so it can be applied to a dataset that looks nothing like this one.
Before modelling. Count the rows and columns. List the dtypes. Count missing values per column. Count the classes, or describe the target's distribution. Ask, of every column, whether it would be known at prediction time.
Splitting. Stratify for classification. Group when several rows share an entity. Order forward when the data has a time. Split before anything learns.
Building. One pipeline. Numeric branch: impute, then scale if the model needs it. Categorical branch: impute, then encode with handle_unknown="ignore". Nothing fitted outside.
Checking. Fit a dummy. Choose a metric from the class balance and the cost of each mistake. Cross-validate with return_train_score=True and read the gap.
Tuning. Randomised search over a logarithmic range for regularisation, including preprocessing choices, scored on the metric you chose. Check the winner is not at the edge of the range. Read cv_results_ and treat settings within one standard deviation as tied.
Reporting. Choose the threshold on development data. Open the test set once. Report the metric, the per-class breakdown, the counts behind the rates, and the baseline beside them. Save the model with its threshold and the library version.
Nothing on that list is difficult. The discipline is doing it in that order every time, including when the first result already looks good — which is exactly when the temptation to skip to the end is strongest.
When the result is disappointing
The procedure gives an honest number, and the honest number is often lower than hoped. What to do next depends on a diagnosis the workflow has already produced.
Train and test both low. Underfitting. More flexible model, better features, less regularisation. Tuning will not help.
Train high, test low. Overfitting. More regularisation, a simpler model, or more data — and the learning curve says whether more data would work.
Both reasonable, but not good enough. The features do not carry enough signal. This is the most common case and the least welcome, because the answer is domain work rather than modelling: new features, a different unit of analysis, a reframed target.
Barely better than the dummy. Consider that the problem may not be predictable from these features at all. That is a legitimate finding, and reporting it early is far more valuable than six weeks of tuning towards it.
The workflow above is what makes each of those diagnoses available. A script that fits one model and prints one accuracy supports none of them.
Why is the test average precision higher than the cross-validated one? Fold variation. With 33 positives in the test set the estimate is noisy, and a small gap either way carries no information.
Should I refit on everything before deploying? Yes, once the procedure is settled and the test score is recorded. More data gives a slightly better model - but the score you report is the one from the test set, not from the refit.
Where would a second model go? In the same search: {"clf": [LogisticRegression(), RandomForestClassifier()]} puts the choice itself in the grid, evaluated by the same cross-validation.
Is pickle the right way to save it? For your own short-lived use, yes. Record the scikit-learn version beside it, because a pickle is not portable across versions, and prefer skops for anything shared.
Where the track has taken you
Twenty-three modules, and the shape of them was deliberate.
The API came first, because fit, predict and transform are the part that transfers to every estimator in the library and to every one written after it. Learning the interface once is most of learning scikit-learn.
Honest numbers came second and took four modules, which is more than the estimators got. That ordering reflects where the mistakes actually are. A model is a few lines; knowing whether its score is real is the difficult part, and splitting, cross-validation, overfitting and the search's optimism are all facets of the same question.
Preparing data took six, because that is where leakage lives. Scaling, encoding, imputation and column routing are individually simple, and every one of them learns something from data — which is why they belong inside a pipeline and why the pipeline is not a convenience.
The models came last and took the fewest. That is the honest proportion: choosing between logistic regression, a forest and a booster matters far less than everything before it, and the library makes the choice a one-line change precisely so that it can.
What you can do now is take a table you have never seen, work out what it needs, build a pipeline that cannot cheat, and produce a number you would defend. The remaining skill is domain knowledge about the columns, which no library supplies.
Things to try
Run the editors in order. They share one interpreter and build on each other, like a script.
Swap the model. Change LogisticRegression to RandomForestClassifier in step three and re-run everything after it. Nothing else needs to change.
Remove the indicator. Drop add_indicator=True and compare the cross-validated average precision.
Break the order. Fit the imputer on all of X before the split, and watch the test score improve for the wrong reason.
Where this leaves you
Look at the data, split it away, put everything that learns inside a pipeline, compare against a baseline on a metric you chose deliberately, tune with cross-validation, pick the threshold from the cost, and open the test set once. That is the whole of it, and the order is what makes the last number worth reporting.
Check yourself
0 of 4
Answer without scrolling back up.
When should the test set be split off?
Anything fitted before the split - a median, a scaler, a category list - has seen the test rows, and the final number is no longer honest.
Why fit a DummyClassifier at all?
The dummy scored 0.082 average precision, exactly the base rate. Without it, 0.478 is a number with no scale.
Why pass scoring="average_precision" to the search?
The metric decides which candidate wins. Leaving it at the default tunes towards ignoring the rare class.
Why save the threshold alongside the model?
The threshold is part of the decision the model makes. Loaded without it, the served model silently uses a default nobody chose.
Cheat sheet
A Complete Workflow
Most of what goes wrong in applied machine learning is this sequence performed in a different order — tuning before checking a baseline, splitting after preprocessing, choosing a metric after seeing the scores. Each of those produces a number that cannot be trusted, and none of them produces an error.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.