Letting cross-validation choose the settings - and understanding why the number it reports for the winner is too high.
Overview
What is being searched
Hyperparameters are the settings you pass to the constructor: C, max_depth, n_neighbors, alpha. They are not learned from the data during fit, so something outside the fit has to choose them.
Choosing by hand means changing one, re-running, and keeping what scored better — which is a search, done slowly and without a record. The search tools do the same thing systematically, with cross-validation at each setting so the comparison is not decided by one lucky split.
GridSearchCV takes a dictionary of parameters to lists of values and tries every combination. The number of fits is the product of the list lengths times the number of folds, and it grows alarmingly: three parameters with five values each and five folds is 375 fits.
Both search objects are themselves estimators. They have fit, predict and score, so a search can go inside a pipeline, inside cross_val_score, or anywhere else an estimator goes — which is what makes nested cross-validation a one-liner.
Worth knowing
GridSearchCV tries every combination; the number of fits is the product of the grid times the folds.
RandomizedSearchCV samples n_iter combinations, which scales to large spaces and usually finds nearly as good a setting.
The search refits the winner on all the data - best_estimator_ is ready to use.
best_score_ is the best of many tries on the same folds, so it is optimistic; a held-out test set or nested CV gives the honest number.
Read cv_results_, not just the winner - settings within one standard deviation of each other are tied.
scoring= defaults to accuracy, and changing it can change which setting wins.
Hyperparameter Search: A Practical Guide
Hyperparameters have to be chosen by measurement rather than by taste. The tools that do it are simple; the number they report is the part that needs care.
The search, and what it leaves you with
Four settings, five folds, twenty fits - and a model already refitted on everything.
example_01.pyscikit-learn
Output
Read the whole table, not just the winner
The top two differ in the fourth decimal, and the standard deviation is two hundred times that.
example_02.pyscikit-learn
Output
Random search finds nearly the same thing, faster
Six samples against sixteen combinations.
example_03.pyscikit-learn
Output
Tuning the preprocessing too
Including whether a step happens at all.
example_04.pyscikit-learn
Output
best_score_ is optimistic
Random labels, so the truth is 0.5. The search reports considerably more than that.
example_05.pyscikit-learn
Output
The metric decides the winner
One grid, three metrics, and they do not all choose the same setting.
example_06.pyscikit-learn
Output
Grid or random
RandomizedSearchCV samples n_iter combinations rather than trying all of them, drawing from lists or from distributions such as scipy.stats.randint and loguniform.
It is usually the better choice, for a reason that is not obvious. In a typical search most parameters barely matter and one or two dominate. A grid spends its budget evenly, so with two parameters and sixteen combinations it tries only four distinct values of the one that matters. Random search with sixteen samples tries sixteen distinct values of it, because every sample varies everything.
The editor above shows six random samples getting within 0.003 of a sixteen-combination grid in 40% of the time, which is the usual shape of the result.
Grid search remains right when the space is genuinely small and you want every combination on record — two or three parameters with a handful of sensible values each.
HalvingGridSearchCV and HalvingRandomSearchCV are the middle path: they evaluate many candidates on a small subset of the data, discard the worst, and give the survivors more resources. On a large dataset the saving is substantial.
The number it reports is too high
This is the part that matters most, and it is easy to miss because everything about the procedure looks careful.
best_score_ is the cross-validated score of the winning setting. It is the maximum of many cross-validated scores computed on the same folds. Taking a maximum over noisy estimates selects for luck as well as quality: the winner is partly the setting that happened to suit those particular folds.
The fifth editor makes the size of the effect visible. The labels are random, so the honest accuracy is 0.5. The search reports 0.642, having picked the best of five depths on the same five folds. Nested cross-validation — where the whole search is repeated inside each outer fold — reports 0.575, and even that is not quite 0.5 with only 120 rows.
So best_score_ is a selection artefact, and the more settings you try the worse it gets. The remedies are a test set held out before the search and touched once afterwards, or nested cross-validation when you need the estimate itself to be trustworthy. Reporting best_score_ as the model's performance is one of the most common ways published numbers turn out to be optimistic.
Read the table
cv_results_ holds every setting with its mean score, standard deviation and rank. Reading it rather than only best_params_ changes what you conclude surprisingly often.
The second editor is a clean example: C=0.1 scores 0.9833 and C=1 scores 0.9832, with standard deviations around 0.02. The winner beat the runner-up by one ten-thousandth, against noise two hundred times larger. Declaring C=0.1 the best value is reading a coin flip.
Two habits follow. Compare the gap against the standard deviation — anything inside one is a tie. And prefer the simpler setting among ties: more regularisation, shallower trees, fewer components. When two settings are indistinguishable on the evidence, the one less likely to have fitted noise is the better bet.
std_test_score also flags an unreliable search: large standard deviations mean the folds disagree, and no ranking computed from them deserves confidence.
Searching the whole pipeline
Because a pipeline is an estimator, a search can tune anything inside it using the double-underscore path — clf__C, pre__num__imputer__strategy.
That makes preprocessing choices measurable rather than assumed. Mean or median imputation, scaled or not, how many PCA components, whether to drop the first one-hot column: all of them can go in the grid, and the cross-validation that evaluates them is the same honest one that evaluates the model.
A step can be tuned in or out entirely by including "passthrough" among its values, which the fourth editor uses to discover that PCA does not help on that data. The same trick swaps whole models: {"clf": [LogisticRegression(), RandomForestClassifier()]} puts the model choice itself in the grid.
The important property is that all of this happens inside the folds. A search over preprocessing done by hand, outside cross-validation, is the leak the earlier modules measured.
The metric decides the answer
scoring defaults to accuracy for classifiers and R² for regressors, and leaving it there means optimising accuracy whether or not accuracy is what you care about.
The last editor searches class_weight on imbalanced data under three metrics. Accuracy and F1 keep the default; balanced accuracy chooses "balanced". Same data, same grid, different winner — because the metrics disagree about what a good model is, which is exactly what the metrics modules said they would do.
So scoring= is where the decision about which mistakes matter enters the tuning process. Choosing it deliberately is not optional if the classes are imbalanced or the costs are asymmetric.
refit can name one metric to refit on when several are being computed, which is how you report a panel of metrics while selecting on one.
Keeping it affordable
n_jobs=-1 parallelises across cores, and search is close to perfectly parallel.
Search coarsely first — powers of ten — then refine around whatever region wins. A logarithmic grid over C from 0.001 to 1000 in one pass tells you more than a fine linear grid in the wrong place.
Reduce the folds to 3 while exploring and raise them for the final comparison. And remember that the parameters worth searching are few: C or alpha for linear models, max_depth and min_samples_leaf for trees, learning_rate and n_estimators for boosting. Searching everything a model exposes wastes budget on parameters that do not move the score.
Which parameters are worth the budget
Searching everything an estimator exposes wastes most of the fits on settings that do not move the score. A short list of what actually matters, per model family.
Linear models.C or alpha, over a logarithmic range, and that is very nearly all. The penalty type (l1 against l2) is worth one comparison when feature selection is desirable.
Trees.max_depth and min_samples_leaf, or ccp_alpha instead of both. The criterion is not worth searching — the decision-trees module measured the three within half a point of each other.
Random forests.max_features first, then min_samples_leaf on noisy data. n_estimators is not a search parameter: set it as high as you can afford, since it cannot overfit.
Gradient boosting.learning_rate and n_estimators together, since they trade against each other, plus max_depth in a narrow range of about 3 to 8. This family repays tuning more than any other on this list, and is the reason to keep budget in reserve.
k-NN.n_neighbors, and weights as a two-value comparison.
SVM.C and gamma, both logarithmic, and this pair genuinely needs a two-dimensional search because they interact strongly.
The general shape: one or two parameters carry nearly all the improvement, and they are usually the ones controlling regularisation. Finding them for a new estimator is a matter of reading which parameter the documentation discusses at length.
What a search cannot fix
Worth stating, because tuning is where effort goes when a model disappoints, and it is often the wrong place to spend it.
A search moves a model within its family. It cannot make a linear model represent a curve, cannot make a tree extrapolate, and cannot compensate for features that do not carry the signal. If the learning curve has converged and the model is underfitting, no combination of hyperparameters closes the gap — the answer is different features or a different family.
Nor can it fix a metric that measures the wrong thing, a split that leaks, or labels that are wrong. A search will optimise whatever it is pointed at, including a leak, and report a confident number for it.
The order that wastes least effort: get the split and the metric right, get the features right, pick a model family that can represent the structure, and tune last. Tuning first is the most common way to spend a week moving a score by half a point.
How many combinations is reasonable? Whatever finishes in the time you have. Twenty random samples over a sensible range beats an exhaustive grid over the wrong one.
Can I search two models at once? Yes - put the estimator itself in the grid: {"clf": [LogisticRegression(), RandomForestClassifier()]}, with each model's own parameters guarded by the step name.
Does refit cost much? One extra fit on the full data, which is negligible next to the search itself. refit=False skips it if you only want the table.
What is error_score for? Some combinations are invalid and raise. The default propagates the error; error_score=np.nan records the failure and lets the search continue.
Ranges that are worth searching over
Choosing the values is most of the skill, and two habits cover it.
Search logarithmically for anything that scales multiplicatively. Regularisation strengths, learning rates, gamma. The interesting range for C spans six orders of magnitude, and a linear grid from 1 to 10 explores almost none of it. np.logspace(-3, 3, 7) gives 0.001 to 1000 in seven values, and scipy.stats.loguniform(1e-3, 1e3) is the equivalent for random search.
Search linearly for anything that counts things. Tree depth, number of neighbours, minimum samples in a leaf. These are small integers and a linear range over a plausible span is right.
Then check whether the winner sits at the edge of the range. If the best C is the largest value tried, the range was too narrow and the search has not found the optimum — it has found the boundary. Extending and re-running is a one-line change and is the difference between a tuned model and one that stopped where you happened to stop looking.
The corollary is that a first pass should be deliberately wide and coarse. Powers of ten across the whole plausible range, three folds, few candidates. That locates the region in a few seconds, and the refined search can then spend its budget somewhere worth spending it.
Things to try
Read the table. The second editor's top two settings are separated by 0.0001. Decide for yourself whether that is a result.
Watch the optimism. The fifth editor searches random labels and reports 0.642 for something whose truth is 0.5.
Tune a step away. In the fourth editor, note that passthrough won — the PCA step was not helping.
Change the metric. In the last editor, add "recall" and see which setting it prefers.
Where this leaves you
Random search over a sensible range, with a pipeline so the preprocessing is tuned honestly, a scoring chosen for the problem, cv_results_ read rather than skimmed, and the winner's score treated as optimistic until a held-out set says otherwise.
Check yourself
0 of 4
Answer without scrolling back up.
Why is best_score_ optimistic?
Taking a maximum selects for luck as well as quality. On random labels a search reported 0.642 where the truth was 0.5.
When is RandomizedSearchCV usually better than GridSearchCV?
A grid spends its budget evenly across parameters that mostly do not matter. Random sampling varies everything at once.
Two settings differ by 0.0001 with a standard deviation of 0.02. What should you conclude?
A gap far inside one standard deviation is noise. Among ties, the more regularised or simpler setting is the safer bet.
What does scoring= default to for a classifier?
Which means an unset scoring silently optimises accuracy - and on imbalanced data that can pick a different winner than the metric you care about.
Cheat sheet
Hyperparameter Search
Hyperparameters have to be chosen by measurement rather than by taste. The tools that do it are simple; the number they report is the part that needs care.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.