Letting cross-validation choose the settings - and understanding why the number it reports for the winner is too high.
Overview
What is being searched
Every estimator has parameters that you set before calling fit. These are hyperparameters. The number of neighbours in KNN, the regularisation strength in logistic regression, the maximum depth of a tree. The model does not learn them from the data. You choose them.
Choosing well matters. The right hyperparameters can double a model's score. The wrong ones can make a good algorithm look useless. Manual tuning works for one or two parameters, but it is slow, hard to reproduce, and easy to fool yourself with.
scikit-learn provides two search tools: GridSearchCV and RandomizedSearchCV. Both try a set of hyperparameter values, evaluate each combination with cross-validation, and keep the best. The difference is how they choose the combinations. Grid search tries every combination on a predefined list. Random search samples from distributions.
Grid search is thorough but expensive. The number of fits is the product of all the list lengths times the number of folds. Three parameters with five values each and five folds means 375 fits. Random search is cheaper and often finds a good setting faster, because it does not waste fits on parameters that turn out not to matter.
Both search objects follow the estimator interface. They have fit, predict and score. You can drop them into a pipeline or use them inside nested cross-validation.
Settings are the settings you pass to the constructor: C, max_depth, n_neighbors, alpha. They aren't learned from the data during fit. A thing outside the fit has to choose them.
Choosing by hand means changing one, re-running. Keeping what scored better. Which is a search, done slowly and without a record. The search tools do the same thing systematically, with cross-validation at each setting so the check isn't decided by one lucky split.
GridSearchCV takes a dict of params to lists of values and tries each combination. The number of fits is the product of the list lengths times the number of folds. It grows alarmingly. Three params with five values each and five folds is 375 fits.
Both search objects are themselves estimators. They've fit, predict and score. A search can go inside a pipeline, inside cross_val_score, or anywhere else an estimator goes. Which is what makes nested cross-validation a one-liner.
Worth knowing
GridSearchCV tries every combination; the number of fits is the product of the grid times the folds.
RandomizedSearchCV samples n_iter combinations, which scales to large spaces and usually finds nearly as good a setting.
The search refits the winner on all the data - best_estimator_ is ready to use.
best_score_ is the best of many tries on the same folds, so it is optimistic; a held-out test set or nested CV gives the honest number.
Read cv_results_, not just the winner - settings within one standard deviation of each other are tied.
scoring= defaults to accuracy, and changing it can change which setting wins.
Hyperparameter Search: A Practical Guide
Settings have to be chosen by reading rather than by taste. The tools that do it are simple. The number they report is the part that needs care.
The search, and what it leaves you with
Four settings, five folds, twenty fits - and a model already refitted on everything.
example_01.pyscikit-learn
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import load_wine
X, y = load_wine(return_X_y=True)
pipe = Pipeline([("scale", StandardScaler()),
("clf", LogisticRegression(max_iter=2000))])
search = GridSearchCV(pipe, {"clf__C": [0.01, 0.1, 1, 10]}, cv=5).fit(X, y)
print("best parameters:", search.best_params_)
print("best cv score :", round(search.best_score_, 4))
print("fits performed :", 4 * 5, "(4 settings x 5 folds)")
print()
print("best_estimator_ has already been refitted on all the data:")
print(" ", type(search.best_estimator_).__name__,
"->", search.best_estimator_.predict(X[:5]))
Output
Read the whole table, not just the winner
The top two differ in the fourth decimal, and the standard deviation is two hundred times that.
example_02.pyscikit-learn
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import load_wine
X, y = load_wine(return_X_y=True)
pipe = Pipeline([("scale", StandardScaler()),
("clf", LogisticRegression(max_iter=2000))])
search = GridSearchCV(pipe, {"clf__C": [0.01, 0.1, 1, 10]}, cv=5).fit(X, y)
r = search.cv_results_
print("%10s %10s %8s %6s" % ("C", "mean", "std", "rank"))
for c, m, s, k in zip(r["param_clf__C"], r["mean_test_score"],
r["std_test_score"], r["rank_test_score"]):
print("%10s %10.4f %8.4f %6d" % (c, m, s, k))
print()
print("the top two are within one standard deviation of each other,")
print("which means the ranking between them is not evidence.")
Output
Random search finds nearly the same thing, faster
Six samples against sixteen combinations.
example_03.pyscikit-learn
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from scipy.stats import randint
import time
X, y = make_classification(n_samples=300, n_features=10, n_informative=4,
random_state=0)
rf = RandomForestClassifier(n_estimators=20, random_state=0)
grid = {"max_depth": [2, 4, 6, 8], "min_samples_leaf": [1, 3, 5, 7]}
t = time.time()
g = GridSearchCV(rf, grid, cv=3).fit(X, y)
grid_time = time.time() - t
dist = {"max_depth": randint(2, 9), "min_samples_leaf": randint(1, 8)}
t = time.time()
r = RandomizedSearchCV(rf, dist, n_iter=6, cv=3, random_state=0).fit(X, y)
rand_time = time.time() - t
print("grid : %2d combinations, %.2fs, best %.4f"
% (16, grid_time, g.best_score_))
print("randomized : 6 samples, %.2fs, best %.4f"
% (rand_time, r.best_score_))
Output
Tuning the preprocessing too
Including whether a step happens at all.
example_04.pyscikit-learn
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_wine
X, y = load_wine(return_X_y=True)
pipe = Pipeline([("scale", StandardScaler()),
("pca", PCA()),
("clf", LogisticRegression(max_iter=2000))])
# The preprocessing is tuned alongside the model - including whether
# the PCA step happens at all.
grid = {"pca": ["passthrough", PCA(n_components=4), PCA(n_components=8)],
"clf__C": [0.1, 1.0]}
search = GridSearchCV(pipe, grid, cv=5).fit(X, y)
print("best:", {k: str(v) for k, v in search.best_params_.items()})
print("score:", round(search.best_score_, 4))
Output
best_score_ is optimistic
Random labels, so the truth is 0.5. The search reports considerably more than that.
example_05.pyscikit-learn
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.ensemble import RandomForestClassifier
import numpy as np
# Random labels: the honest score is 0.5, whatever the search reports.
rng = np.random.RandomState(0)
X = rng.normal(size=(120, 20))
y = rng.randint(0, 2, size=120)
grid = {"max_depth": [1, 2, 3, 5, 8]}
search = GridSearchCV(RandomForestClassifier(n_estimators=20, random_state=0),
grid, cv=5)
inner = search.fit(X, y).best_score_
outer = cross_val_score(search, X, y, cv=5).mean()
print("best_score_ from the search :", round(inner, 3))
print("nested cross-validation :", round(outer, 3))
print("the truth (the labels are random): 0.5")
print()
print("best_score_ is the best of five tries on the same folds,")
print("so it is optimistic by construction.")
Output
The metric decides the winner
One grid, three metrics, and they do not all choose the same setting.
example_06.pyscikit-learn
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=800, n_features=10, weights=[0.93],
random_state=0)
grid = {"class_weight": [None, "balanced"]}
for metric in ("accuracy", "f1", "balanced_accuracy"):
s = GridSearchCV(LogisticRegression(max_iter=2000), grid, cv=5,
scoring=metric).fit(X, y)
print("%-19s picks class_weight=%-9s (%.4f)"
% (metric, str(s.best_params_["class_weight"]), s.best_score_))
print()
print("two of the three keep the default and the third does not.")
print("scoring= is a decision about the problem, not a formality.")
Output
Grid or random
RandomizedSearchCV samples n_iter combinations rather than trying all of them, drawing from lists or from spreads such as scipy.stats.randint and loguniform.
It's often the better choice, for a reason that isn't obvious. In a common search most params barely matter and one or two dominate. A grid spends its budget evenly. With two params and sixteen combinations it tries only four distinct values of the one that matters. Random search with sixteen samples tries sixteen distinct values of it. Each sample varies all.
The editor above shows six random samples getting within 0.003 of a sixteen-combination grid in 40% of the time. This Is the usual shape of the result.
Grid search remains right when the space is truly small and you want each combination on record. Two or three params with a handful of sensible values each.
HalvingGridSearchCV and HalvingRandomSearchCV are the middle path: they test many picks on a small subset of the data, discard the worst. Give the survivors more resources. On a large dataset the saving is substantial.
The number it reports is too high
This is the part that matters most. It's easy to miss because all about the method looks careful.
best_score_ is the cross-validated score of the winning setting. It's the max of many cross-validated scores found on the same folds. Taking a max over noisy guesses selects for luck and quality. The winner is partly the setting that happened to suit those given folds.
The fifth editor makes the size of the effect visible. The labels are random. The honest fit is 0.5. The search reports 0.642, having picked the best of five depths on the same five folds. Nested cross-validation. Where the whole search is repeated inside each outer fold. Reports 0.575. Even that isn't quite 0.5 with only 120 rows.
So best_score_ is a pick artefact. The more settings you try the worse it gets. The remedies are a test set held out before the search and touched once afterwards, or nested cross-validation when you need the guess itself to be trustworthy. Reporting best_score_ as the model's speed is one of the most common ways published numbers turn out to be optimistic.
Read the table
cv_results_ holds each setting with its mean score, standard deviation and rank. Reading it rather than only best_params_ changes what you conclude oddly often.
The second editor is a clean example: C=0.1 scores 0.9833 and C=1 scores 0.9832, with standard deviations around 0.02. The winner beat the runner-up by one ten-thousandth, against noise two hundred times larger. Declaring C=0.1 the best value is reading a coin flip.
Two habits follow. Compare the gap against the standard deviation. Any of it inside one is a tie. And prefer the simpler setting among ties. More smoothing, shallower trees, fewer components. When two settings are the same on the evidence, the one less likely to have fitted noise is the better bet.
std_test_score also flags an unreliable search: large standard deviations mean the folds disagree. No ranking found from them deserves trust.
Searching the whole pipeline
Because a pipeline is an estimator, a search can tune any of it inside it using the double-underscore path — clf__C, pre__num__imputer__strategy.
That makes prep choices measurable rather than assumed. Mean or median imputation, scaled or not, how many PCA components, whether to drop the first one-hot column. All of them can go in the grid. The cross-validation that tests them is the same honest one that tests the model.
A step can be tuned in or out fully by including "passthrough" among its values. This The fourth editor uses to discover that PCA doesn't help on that data. The same trick swaps whole models: {"clf": [LogisticRegression(), RandomForestClassifier()]} puts the model choice itself in the grid.
The key property is that all of this happens inside the folds. A search over prep done by hand, outside cross-validation, is the leak the earlier modules measured.
The metric decides the answer
scoring defaults to fit for classifiers and R² for regressors. Leaving it there means optimising fit whether or not fit is what you care about.
The last editor searches class_weight on imbalanced data under three metrics. Fit and F1 keep the default. Weighed fit chooses "weighed". Same data, same grid, different winner. Because the metrics disagree about what a good model is. This Is just what the metrics modules said they would do.
So scoring= is where the choice about which mistakes matter enters the tuning process. Choosing it on purpose isn't optional if the classes are imbalanced or the costs are skewed.
refit can name one metric to refit on when a few are being found. This Is how you report a panel of metrics while selecting on one.
Keeping it affordable
n_jobs=-1 parallelises across cores, and search is close to just right parallel.
Search coarsely first — powers of ten — then refine around what region wins. A logarithmic grid over C from 0.001 to 1000 in one pass tells you more than a fine linear grid in the wrong place.
Reduce the folds to 3 while exploring and raise them for the final check. And note that the params worth searching are few: C or alpha for linear models, max_depth and min_samples_leaf for trees, learning_rate and n_estimators for boosting. Searching all a model exposes wastes budget on params that don't move the score.
Which parameters are worth the budget
Searching all an estimator exposes wastes most of the fits on settings that don't move the score. A short list of what really matters, per model family.
Linear models.C or alpha, over a logarithmic range. That is very nearly all. The fine type (l1 against l2) is worth one check when feature pick is wanted.
Trees.max_depth and min_samples_leaf, or ccp_alpha instead of both. The criterion isn't worth searching. The choice-trees module measured the three within half a point of each other.
Random forests.max_features first, then min_samples_leaf on noisy data. n_estimators isn't a search param. Set it as high as you can afford, since it can't overfit.
Grad boosting.learning_rate and n_estimators together. They trade against each other, plus max_depth in a narrow range of about 3 to 8. This family repays tuning more than any other on this list. Is the reason to keep budget in reserve.
k-NN.n_neighbors, and weights as a two-value check.
SVM.C and gamma, both logarithmic. This pair truly needs a two-dimensional search because they interact strongly.
The general shape: one or two params carry nearly all the improvement. They're often the ones controlling smoothing. Finding them for a new estimator is a matter of reading which param the docs discusses at length.
What a search cannot fix
Worth stating, because tuning is where effort goes when a model disappoints. It's often the wrong place to spend it.
A search moves a model within its family. It can't make a linear model represent a curve, can't make a tree extrapolate. Can't make up for for features that don't carry the signal. If the learning curve has settled and the model is underfitting, no combination of settings closes the gap. The answer is different features or a different family.
Nor can it fix a metric that measures the wrong thing, a split that leaks, or labels that are wrong. A search will optimise what it's pointed at, including a leak. Report a confident number for it.
The order that wastes least effort: get the split and the metric right, get the features right, pick a model family that can represent the structure. Tune last. Tuning first is the most common way to spend a week moving a score by half a point.
How many combinations is reasonable? What finishes in the time you've. Twenty random samples over a sensible range beats an exhaustive grid over the wrong one.
Can I search two models at once? Yes - put the estimator itself in the grid: {"clf": [LogisticRegression(), RandomForestClassifier()]}, with each model's own params guarded by the step name.
Does refit cost much? One extra fit on the full data. This Is negligible next to the search itself. refit=False skips it if you only want the table.
What is error_score for? Some combinations are invalid and raise. The default passes the error. error_score=np.nan records the failure and lets the search continue.
Ranges that are worth searching over
Choosing the values is most of the skill, and two habits cover it.
Search logarithmically for any of it that scales multiplicatively. Smoothing strengths, learning rates, gamma. The notable range for C spans six orders of magnitude. A linear grid from 1 to 10 explores almost none of it. np.logspace(-3, 3, 7) gives 0.001 to 1000 in seven values. scipy.stats.loguniform(1e-3, 1e3) is the equal for random search.
Search linearly for any of it that counts things. Tree depth, number of neighbours, min samples in a leaf. These are small ints and a linear range over a plausible span is right.
Then check whether the winner sits at the edge of the range. If the best C is the largest value tried, the range was too narrow and the search hasn't found the optimum. It has found the boundary. Extending and re-running is a one-line change and is the difference between a tuned model and one that stopped where you happened to stop looking.
The corollary is that a first pass should be on purpose wide and coarse. Powers of ten across the whole plausible range, three folds, few picks. That locates the region in a few seconds. The refined search can then spend its budget a place worth spending it.
Things to try with Hyperparameter Search
Read the table. The second editor's top two settings are separated by 0.0001. Decide for yourself whether that is a result.
Watch the optimism. The fifth editor searches random labels and reports 0.642 for a thing whose truth is 0.5.
Tune a step away. In the fourth editor, note that passthrough won. The PCA step wasn't helping.
Change the metric. In the last editor, add "recall" and see which setting it prefers.
Questions people ask about Hyperparameter Search
GridSearchCV or RandomizedSearchCV? Grid search tries every combination and suits small spaces. Randomized search samples a fixed number and covers large spaces cheaper.
Why is best_score_ too optimistic? It is the maximum over many noisy trials, so it is biased upwards. Confirm on a test set or with nested cross-validation.
Should I search over a whole pipeline? Yes. Passing the pipeline keeps preprocessing inside each fold, and you can tune its steps with the double-underscore names.
How do I choose the scoring metric? Pick the metric that matches the real goal and pass it as scoring. The default may favour something you do not care about.
How do I keep a search affordable? Use fewer folds, a smaller range, n_jobs=-1 and HalvingRandomSearchCV, which drops weak candidates early.
Where can I read all the results? In cv_results_, which converts to a DataFrame and shows scores and spread for every candidate.
Where this leaves you
Random search over a sensible range, with a pipeline so the prep is tuned honestly, a scoring chosen for the problem, cv_results_ read rather than skimmed. The winner's score treated as optimistic until a held-out set says or else.
Check yourself
0 of 4
Answer without scrolling back up.
Why is best_score_ optimistic?
Taking a maximum selects for luck as well as quality. On random labels a search reported 0.642 where the truth was 0.5.
When is RandomizedSearchCV usually better than GridSearchCV?
A grid spends its budget evenly across parameters that mostly do not matter. Random sampling varies everything at once.
Two settings differ by 0.0001 with a standard deviation of 0.02. What should you conclude?
A gap far inside one standard deviation is noise. Among ties, the more regularised or simpler setting is the safer bet.
What does scoring= default to for a classifier?
Which means an unset scoring silently optimises accuracy - and on imbalanced data that can pick a different winner than the metric you care about.
Cheat sheet
Hyperparameter Search
Search for the best hyperparameters with GridSearchCV and RandomizedSearchCV, and measure each combination fairly with cross-validation.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.