Every module in this track, applied once to one dataset, in the order you would actually do it.
Overview
The order
This page puts the earlier modules together into one end-to-end workflow. The dataset is small and deliberately messy: mixed types, missing values and imbalanced classes. The point is the sequence of steps, not the final score.
Each step in a machine-learning project depends on the one before it. Exploring the data tells you what preprocessing to apply. Splitting comes before preprocessing so the test set stays untouched. A baseline model tells you whether a tuned model is worth the effort. Evaluation on the test set comes last, and happens exactly once.
Skipping a step or reordering the sequence introduces errors that do not raise exceptions. Scaling before splitting leaks test-set statistics. Tuning without a baseline means you cannot tell whether the improvement came from the model or from the hyperparameters. Evaluating on the test set more than once turns it into a validation set.
The six editors on this page are one continuous script. They share a single interpreter. Each one uses variables that the previous one created. Running them out of order raises a NameError. That constraint is intentional. It mirrors the dependency structure of a real project.
Look. Split. Build. Check. Tune. Report.
Most of what goes wrong in applied machine learning is this sequence performed in a different order. Tuning before checking a baseline, splitting after prep, choosing a metric after seeing the scores. Each of those makes a number that can't be trusted. None of them makes an error.
The dataset here is small and synthetic: four columns of mixed type, 8% missing values in one of them. An 8% positive rate. That combination is on purpose plain. The point is the method rather than the result.
The six editors below are one script, not six examples. They share a single interpreter and each one uses what the last one defined. Run them in order from the top. Starting in the middle raises NameError for a variable an earlier step was supposed to create.
Worth knowing
The editors on this page run in order and share one interpreter - it is a single script, split into steps.
Look, split, build, check, tune, report. The order matters more than any individual choice in it.
Everything that learns from data lives inside the pipeline, so the folds and the test set stay honest.
The metric is chosen from the class balance, and the baseline is fitted before the model is believed.
The threshold is a parameter: chosen on the development set, saved with the model.
The test set is opened once, after every decision has been made.
A Complete Workflow: A Practical Guide
Twenty-two modules, applied to one small dataset in the order you'd really do it. Nothing here is new. The sequence is the point.
Look at the data first
Four counts, before any modelling decision is made.
Stratified, because 8% is not enough to leave to chance.
example_02.pyscikit-learn
from sklearn.model_selection import train_test_split
X_dev, X_test, y_dev, y_test = train_test_split(
X, y, test_size=0.2, random_state=0, stratify=y)
print("development set:", X_dev.shape, " churn %.3f" % y_dev.mean())
print("test set :", X_test.shape, " churn %.3f" % y_test.mean())
print()
print("the test set is now put away. everything that follows uses")
print("only the development set, until the very last step.")
Output
Build the pipeline
Two branches, four learned transformations, one estimator - and nothing fitted outside it.
example_03.pyscikit-learn
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
numeric = ["tenure", "monthly"]
categorical = ["plan", "region"]
pre = ColumnTransformer([
("num", make_pipeline(SimpleImputer(strategy="median", add_indicator=True),
StandardScaler()), numeric),
("cat", OneHotEncoder(handle_unknown="ignore"), categorical),
])
pipe = Pipeline([("pre", pre),
("clf", LogisticRegression(max_iter=1000,
class_weight="balanced"))])
pipe.fit(X_dev, y_dev)
print("features after preprocessing:", pipe[:-1].transform(X_dev).shape[1])
print("names:", list(pipe.named_steps["pre"].get_feature_names_out()))
Output
Compare against doing nothing
The baseline first, and a metric chosen for an 8% positive rate.
example_04.pyscikit-learn
from sklearn.dummy import DummyClassifier
from sklearn.model_selection import cross_validate
from sklearn.metrics import make_scorer, precision_score
# precision is undefined when nothing is predicted positive, which the
# dummy never does - make_scorer says what to return instead of warning.
scoring = {"AP": "average_precision",
"recall": "recall",
"precision": make_scorer(precision_score, zero_division=0)}
for label, model in [("dummy", DummyClassifier(strategy="most_frequent")),
("pipeline", pipe)]:
out = cross_validate(model, X_dev, y_dev, cv=5, scoring=scoring)
print("%-10s" % label, " ".join("%s %.3f" % (s, out["test_" + s].mean())
for s in scoring))
print()
print("average precision is the summary to watch here - positives are 8%,")
print("so the dummy scores the base rate and ROC AUC would flatter both.")
Output
Tune the model and the preprocessing together
Eight random samples, cross-validated, optimising the metric that matters.
example_05.pyscikit-learn
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import loguniform
grid = {"clf__C": loguniform(1e-2, 1e2),
"pre__num__simpleimputer__strategy": ["mean", "median"]}
search = RandomizedSearchCV(pipe, grid, n_iter=8, cv=5,
scoring="average_precision",
random_state=0).fit(X_dev, y_dev)
print("best parameters:")
for k, v in sorted(search.best_params_.items()):
print(" %-38s %s" % (k, round(v, 4) if isinstance(v, float) else v))
print("best cross-validated AP: %.4f" % search.best_score_)
print()
print("this number is optimistic - it is the best of eight tries.")
Output
Choose the threshold, then open the test set
Once. The numbers this prints are the ones you are entitled to report.
example_06.pyscikit-learn
from sklearn.metrics import (average_precision_score, precision_recall_curve,
classification_report)
import numpy as np, pickle
model = search.best_estimator_ # already refitted on all of X_dev
# Choose the threshold on the development set, never on the test set.
dev_proba = model.predict_proba(X_dev)[:, 1]
p, r, t = precision_recall_curve(y_dev, dev_proba)
f1 = 2 * p[:-1] * r[:-1] / np.clip(p[:-1] + r[:-1], 1e-12, None)
threshold = float(t[int(np.argmax(f1))])
print("threshold chosen on the development set: %.3f" % threshold)
# The test set, opened once.
test_proba = model.predict_proba(X_test)[:, 1]
print()
print("average precision on the untouched test set: %.4f"
% average_precision_score(y_test, test_proba))
print()
print(classification_report(y_test, (test_proba >= threshold).astype(int),
digits=3, zero_division=0))
blob = pickle.dumps({"model": model, "threshold": threshold})
print("saved %d bytes - the pipeline and its threshold together" % len(blob))
Output
1. Look
Four counts, before any choice: the shape, the dtypes, the missing values per column, and the class balance.
Each one finds a later choice. The dtypes say which columns need encoding and which need scaling. The missing count says whether an imputer is needed and in which branch. The class balance. 8% here. Rules out fit as a metric and argues for a stratified split.
This step takes four lines and is the one most often skipped. each choice that follows is made in the dark without it.
2. Split
Right away, and stratified.
The test set is set aside now, before any of it has been fitted, scaled, imputed or selected. That is what makes the number at the end mean a thing. It's the single most key line in the script.
Stratifying matters at 8%: an unstratified 20% test set could easily land with a class rate of 5% or 11%. Each metric found from it would be shifted by an amount nothing reveals.
What remains is the development set. All up to the final step uses only that. Where this module says "development set", a larger project would cross-validate within it rather than splitting again.
3. Build the pipeline
Two branches, because the columns need different treatment.
Numeric columns get an imputer. With add_indicator=True. A missing charge may itself be helpful — and then a scaler. The model is smoothed and a fine on coeff size needs close units. cat columns get one-hot encoding with handle_unknown="skip". A region absent from a training fold doesn't crash the run.
class_weight="weighed" goes on the model, because the positives are rare.
The key property is that nothing is fitted outside this object. The imputer's median, the scaler's mean, the encoder's category list and the model's coeffs are all learned inside fit. This Means they're all refitted on the training part of each cross-validation fold. The leakage module measured what happens or else.
Printing the feature names after prep is a cheap sanity check. Ten columns from four. Reading them confirms the missing indicator survived and the categories expanded as expected.
4. Check against a baseline
Two numbers, and the first one is the dummy.
DummyClassifier(strategy="most_frequent") scores an mean precision of 0.082. Just the base rate. This Is what mean precision does for a model with no skill. The pipeline scores 0.478. That gap is the evidence that the model has learned a thing. Without the first number the second is uninterpretable.
The metrics are chosen for the problem rather than accepted from the default. Mean precision because positives are rare and ROC AUC would flatter both models. Recall and precision alongside it. The single summary hides the trade.
make_scorer(precision_score, zero_division=0) appears because the dummy never predicts a plus. This Makes precision undefined. Saying what to return is better than five warnings in the output.
5. Tune
A randomised search over the model's smoothing and the prep's imputation strategy, scored on average precision.
Two things are worth noticing. The search covers a prep choice and a model one. This Is possible only because the prep is inside the pipeline. And scoring="average_precision" is passed by name. The default would optimise fit. And on 8% positives, fit prefers a model that predicts nothing.
best_score_ comes out at 0.4829. That number is optimistic. It's the best of eight tries on the same folds. The setting-search module measured how much that inflates things. It's a guide for choosing, not a result to report.
6. Choose the threshold, then report
The threshold is a param. So It's chosen on the development set. Here by raising F1 along the precision-recall curve, though a cost math would be better if the costs were known.
Then the test set is opened, once. Mean precision on it's 0.5283. The classification report gives the per-class picture at the chosen threshold. Recall 0.667 and precision 0.468 on the positive class, from 33 positives in 400 rows.
Those numbers are the deliverable. They're honest because nothing about them was chosen after seeing them.
Finally the model is saved with its threshold. A pipeline pickled alone would be loaded elsewhere and used through predict(), silently reverting to 0.5. Which the thresholds module showed can be the difference between catching most of the positives and almost none.
What is missing, and would come next
This is a complete workflow, not a complete project.
A second model family. Grad boosting would very likely beat this linear one. The whole script would change by one line. The pipeline interface doesn't care what the final estimator is.
More features. The learning-curve check from the overfitting module would say whether more data helps. Feature work often beats tuning when it doesn't.
The timing question. each column here was known at guess time by build. On real data that has to be asked of each one. It's the check no code performs.
Monitoring. A model is fitted on the past and used on the future. The spreads drift. What is measured after deployment matters as much as any of it measured before it.
The checklist, without the code
The same method, stated so it can be applied to a dataset that looks nothing like this one.
Before modelling. Count the rows and columns. List the dtypes. Count missing values per column. Count the classes, or show the target's spread. Ask, of each column, whether it would be known at guess time.
Splitting. Stratify for classification. Group when a few rows share an entity. Order forward when the data has a time. Split before any of it learns.
Building. One pipeline. Numeric branch: impute, then scale if the model needs it. cat branch: impute, then encode with handle_unknown="skip". Nothing fitted outside.
Checking. Fit a dummy. Choose a metric from the class balance and the cost of each mistake. Cross-validate with return_train_score=True and read the gap.
Tuning. Randomised search over a logarithmic range for smoothing, including prep choices, scored on the metric you chose. Check the winner isn't at the edge of the range. Read cv_results_ and treat settings within one standard deviation as tied.
Reporting. Choose the threshold on development data. Open the test set once. Report the metric, the per-class breakdown, the counts behind the rates. The baseline beside them. Save the model with its threshold and the library version.
Nothing on that list is hard. The discipline is doing it in that order each time, including when the first result already looks good. Which is just when the temptation to skip to the end is strongest.
When the result is disappointing
The method gives an honest number. The honest number is often lower than hoped. What to do next depends on a diagnosis the workflow has already made.
Train and test both low. Underfitting. More flexible model, better features, less smoothing. Tuning won't help.
Train high, test low. Overfitting. More smoothing, a simpler model, or more data. And the learning curve says whether more data would work.
Both reasonable, but not good enough. The features don't carry enough signal. This is the most common case and the least welcome. The answer is domain work rather than modelling. New features, a different unit of analysis, a reframed target.
Barely better than the dummy. Consider that the problem may not be predictable from these features at all. That is a legitimate finding. Reporting it early is far more valuable than six weeks of tuning towards it.
The workflow above is what makes each of those diagnoses available. A script that fits one model and prints one fit supports none of them.
Why is the test mean precision higher than the cross-validated one? Fold variation. With 33 positives in the test set the estimate is noisy. A small gap either way carries no info.
Should I refit on all before deploying? Yes, once the method is settled and the test score is recorded. More data gives a slightly better model - but the score you report is the one from the test set, not from the refit.
Where would a second model go? In the same search: {"clf": [LogisticRegression(), RandomForestClassifier()]} puts the choice itself in the grid, tested by the same cross-validation.
Is pickle the right way to save it? For your own short-lived use, yes. Record the scikit-learn version beside it. A pickle isn't portable across versions. Prefer skops for any of it shared.
Where the track has taken you
Twenty-three modules, and the shape of them was deliberate.
The API came first, because fit, predict and transform are the part that transfers to each estimator in the library and to each one written after it. Learning the interface once is most of learning scikit-learn.
Honest numbers came second and took four modules. This Is more than the estimators got. That sorting reflects where the mistakes really are. A model is a few lines. Knowing whether its score is real is the hard part. Splitting, cross-validation, overfitting and the search's optimism are all facets of the same question.
Preparing data took six, because that is where leakage lives. Scaling, encoding, imputation and column routing are on their own simple. Each one of them learns a thing from data. Which is why they belong inside a pipeline and why the pipeline isn't a convenience.
The models came last and took the fewest. That is the honest proportion. Choosing between logistic regression, a forest and a booster matters far less than all before it. The library makes the choice a one-line change just so that it can.
What you can do now is take a table you've never seen, work out what it needs, build a pipeline that can't cheat. Give a number you'd defend. The left skill is domain knowledge about the columns. This No library supplies.
Things to try with A Complete Workflow
Swap the model. Change LogisticRegression to RandomForestClassifier in step three and re-run all after it. Nothing else needs to change.
Drop the indicator. Drop add_indicator=True and compare the cross-validated mean precision.
Break the order. Fit the imputer on all of X before the split. Watch the test score improve for the wrong reason.
Questions people ask about A Complete Workflow
What order should the steps follow? Look at the data, split it, build a pipeline, compare with a baseline, tune with cross-validation, choose the threshold and report on the test set once.
Why fit a baseline first? A dummy model sets the floor. Without it a score has no scale.
When do I touch the test set? Once, at the end. Using it to guide choices turns it into a second validation set.
Why put everything in one pipeline? So every learned step is refitted inside each fold and the same steps run on new data.
What if the result is disappointing? Check for data problems, try a better baseline comparison and look at errors by group before reaching for a fancier model.
How do I ship the model? Save the fitted pipeline and the chosen threshold together, then record the library versions.
Where this leaves you
Look at the data, split it away, put all that learns inside a pipeline, compare against a baseline on a metric you chose on purpose, tune with cross-validation, pick the threshold from the cost. Open the test set once. That is the whole of it. The order is what makes the last number worth reporting.
Check yourself
0 of 4
Answer without scrolling back up.
When should the test set be split off?
Anything fitted before the split - a median, a scaler, a category list - has seen the test rows, and the final number is no longer honest.
Why fit a DummyClassifier at all?
The dummy scored 0.082 average precision, exactly the base rate. Without it, 0.478 is a number with no scale.
Why pass scoring="average_precision" to the search?
The metric decides which candidate wins. Leaving it at the default tunes towards ignoring the rare class.
Why save the threshold alongside the model?
The threshold is part of the decision the model makes. Loaded without it, the served model silently uses a default nobody chose.
Cheat sheet
A Complete Workflow
Follow one dataset through loading, splitting, scaling, encoding, fitting and scoring to see every scikit-learn step in context.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.