Same budget, two ways of spending it. Grid search tries nine values of everything; random search tries every value of the one that matters.
Overview
The setup
The background shading is the true score surface, and it's on purpose lopsided. The flat axis is a setting that matters a great deal — a learning rate, a smoothing strength. And the vertical axis is one that barely does. The optimum is a narrow vertical band.
This lopsidedness is the normal case, not a contrived one. In most models one or two settings dominate and the rest are close to noise, and you often do not know in advance which is which.
Grid Search against Random Search
This module needs JavaScript: the
numbers are computed in the page rather than recorded.
Worth knowing
Hyperparameters are not equally important. Usually one or two dominate and the rest barely matter.
A grid of n×n points tries only n distinct values of each parameter, however large n×n is.
The same budget spent randomly tries n×n distinct values of each, so it explores the important one far more finely.
The gap widens with dimension. Bergstra and Bengio made this argument in 2012 and it has held up.
Grid Search against Random Search
Why the same number of trials finds a better answer when you stop being systematic.
Count the distinct values
Set the method to Grid with 16 trials. Sixteen points appear in a 4×4 lattice. Read the line under the chart: 16 trials, 4 distinct values of the param that matters.
each point in a column shares an x. The other three trials in that column tell you nothing new about the axis you care about. They only vary the axis you don't.
Now switch to Random, same 16 trials. Sixteen points scattered, and 16 distinct values of the param that matters. Same cost, four times the detail where detail counts.
Push the budget to 64. Grid gives 8 distinct values from 64 trials. Random gives 64. The gap widens as the budget grows, and it widens much faster as dims are added — a grid over five params at four values each is 1,024 trials and still only four values of each.
The argument in one line
A grid of *n* points per axis over *d* axes costs *n^d* trials and buys *n* distinct values of each param. Random search of the same *n^d* trials buys *n^d* distinct values of each param.
If only one axis matters, grid search has wasted all but *n* of its trials on that axis. This is Bergstra and Bengio's 2012 result, and it changed default practice.
Watch the luck
Move the seed slider with random selected. The best point found moves, and the best score wobbles. Random search is random. A single run can be unlucky.
That is a real cost and a smaller one than it looks. Because the surface is broad along the unimportant axis, most draws land a place reasonable, and the distribution of outcomes is much better than grid's ensured coarseness. But it does mean a single random run isn't a strong claim, and checks should allow for the variation.
When grid search is still right
Few params, truly discrete. Three kernels, four values of *k*: a grid is exhaustive and the exhaustiveness is worth having.
Reproducibility matters more than speed. A grid is deterministic and easy to show in a paper.
You're refining. Random search to find the region, a small grid to comb it. This combination is better than either alone.
What replaced both
Bayesian optimisation builds a model of the score surface from the trials so far and proposes the point most likely to improve. Optuna and Hyperopt do this, and on costly goals — where one trial takes an hour — it's clearly better than either method here.
Successive halving and Hyperband attack the cost differently: start many configurations cheaply, kill the worst, give the survivors more budget. When a trial's early speed predicts its final speed, this is sharply more fast.
For any of it cheap, random search remains an great default. It has no settings of its own, it parallelises just right, and it's very hard to use wrongly.
Why random search reaches the good region
Random search's edge over a grid is that it doesn't spend trials re-measuring the axis that doesn't matter, and its chance of landing in the good region follows a simple curve. If the top 5% of the space is what you want, n free random draws miss it only if *all* of them do:
random trials
P(at least one lands in the top 5%)
10
0.40
20
0.64
60
0.95
Sixty trials find the good region 95% of the time, with no assumption about which setting matters. A grid of the same 60 trials over two axes gives about 8 distinct values of each, so if only one axis matters it has spent all but 8 of its trials varying the one that doesn't. And over five axes at four values each, a grid is 1,024 trials and still only four values per param. That is Bergstra and Bengio's result, and it's why random beat grid as the default. The one place it needs care is the sampling scale: a learning rate must be drawn log-uniformly, because a uniform draw between 0.0001 and 0.1 puts 90% of its samples above 0.01 and starves the small values that often matter most.
What replaced both methods for costly goals is Bayesian optimisation — Optuna, Hyperopt — which builds a model of the score surface from the trials so far and proposes the point most likely to improve, clearly better when a single trial takes an hour. Successive halving and Hyperband attack the cost from another angle, starting many configurations cheaply and killing the worst, which wins when early speed predicts final speed. For any of it cheap, random search remains an great default: it has no settings of its own, it parallelises just right, and the one way to misuse it. A linear scale on a param that should be logarithmic. Is the same trap noted above.
Where it goes wrong with Grid Search against Random Search
Sampling scale. A learning rate should be sampled log-uniformly. Uniform between 0.0001 and 0.1 puts 90% of the draws above 0.01.
Tuning on the test set. Same error as all over else. Tune on validation.
Searching a range that excludes the answer. No search finds what is outside its bounds. Check whether the best value sits at the edge of the range. If it does, widen it and search again.
Not searching prep. Put the whole pipeline in the search, so imputation and scaling choices are tuned alongside the model.
Questions people ask about Grid Search against Random Search
Is random search always better than grid search? Not always, but it usually finds a good setting with fewer fits when only a few hyperparameters matter. Grid search is fine for one or two small, discrete parameters.
How many random samples do I need? Around 60 random trials gives a good chance of landing in the top 5 percent of the space. More helps with diminishing returns.
Should I sample on a log scale? Yes for parameters like learning rate, C and alpha that span orders of magnitude. Use a log-uniform distribution so each scale gets equal attention.
Why does grid search waste fits? It repeats the same few values of each parameter across every combination. If one parameter does not matter, you run many near-duplicate experiments.
What comes after random search? Bayesian optimisation and successive halving, which use earlier results to choose where to look next. They are worth it when each fit is expensive.
Can I tune on the test set? No. Use cross-validation on the training data for the search, and touch the test set once at the end.
Why random search finds more with fewer fits
Grid search spends its budget evenly across each param, including the ones that don't matter. Random search doesn't. On the same budget it often wins.
example_01.pyscikit-learn
import numpy as np
import time
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import (GridSearchCV, RandomizedSearchCV,
StratifiedKFold, cross_val_score)
from scipy.stats import randint
X, y = make_classification(n_samples=700, n_features=12, n_informative=5,
flip_y=0.10, random_state=0)
cv = StratifiedKFold(3, shuffle=True, random_state=0)
print("baseline, all defaults: cv %.4f"
% cross_val_score(RandomForestClassifier(random_state=0), X, y, cv=cv).mean())
grid = {"n_estimators": [40, 120], "max_depth": [3, 6, 12, None],
"max_features": [2, 4, 8]}
total = 2 * 4 * 3
print("grid search over %d combinations x 3 folds = %d fits" % (total, total * 3))
t0 = time.time()
gs = GridSearchCV(RandomForestClassifier(random_state=0), grid, cv=cv).fit(X, y)
print(" best %.4f in %.1fs with %s" % (gs.best_score_, time.time()-t0, gs.best_params_))
dist = {"n_estimators": randint(40, 120), "max_depth": randint(2, 20),
"max_features": randint(1, 12)}
t0 = time.time()
rs = RandomizedSearchCV(RandomForestClassifier(random_state=0), dist,
n_iter=total, cv=cv, random_state=0).fit(X, y)
print("random search, the SAME %d fits" % total)
print(" best %.4f in %.1fs with %s" % (rs.best_score_, time.time()-t0, rs.best_params_))
print()
print("how many DISTINCT values does each parameter get tried at?")
print(" grid : n_estimators 2, max_depth 4, max_features 3")
print(" random : n_estimators %d, max_depth %d, max_features %d"
% tuple(len(set(r[k] for r in rs.cv_results_["params"]))
for k in ("n_estimators", "max_depth", "max_features")))
print(" random gets 24 distinct values of every parameter, so it never")
print(" wastes budget on a parameter that turns out not to matter.")
Output
The next block shows why log-uniform distributions matter and demonstrates SVM tuning.
example_02.pyscikit-learn
import numpy as np
from sklearn.datasets import make_classification
from sklearn.svm import SVC
from sklearn.model_selection import RandomizedSearchCV, StratifiedKFold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from scipy.stats import loguniform
X, y = make_classification(n_samples=700, n_features=12, n_informative=5,
flip_y=0.10, random_state=0)
cv = StratifiedKFold(3, shuffle=True, random_state=0)
print("grids explode. adding one parameter with 4 values multiplies:")
for extra in range(5):
print(" %d parameters at 4 values each -> %5d combinations"
% (extra + 1, 4 ** (extra + 1)))
print()
print("scales matter -- C and gamma live on a log scale:")
print(" linear range 0.1 to 100 in 5 steps:", np.round(np.linspace(0.1, 100, 5), 3))
print(" log range, same endpoints :", np.round(np.logspace(-1, 2, 5), 3))
print(" the linear one puts 4 of 5 samples above 25, which is almost")
print(" never where the interesting values are.")
print()
svm_rs = RandomizedSearchCV(make_pipeline(StandardScaler(), SVC()),
{"svc__C": loguniform(1e-2, 1e3),
"svc__gamma": loguniform(1e-4, 1e0)},
n_iter=15, cv=cv, random_state=0).fit(X, y)
print("SVM tuned over log-uniform ranges, 15 draws: cv %.4f" % svm_rs.best_score_)
print(" C %.4f, gamma %.5f"
% (svm_rs.best_params_["svc__C"], svm_rs.best_params_["svc__gamma"]))
print()
print("rule of thumb: grid when you have 2 parameters and know the ranges,")
print("random when you have more than 2 or are still exploring. always")
print("check if the best value landed on the edge -- if it did, the range")
print("was wrong, not the model.")
Output
Check yourself
0 of 3
Answer without scrolling back up.
Why does a 4x4 grid of 16 trials explore an important hyperparameter poorly?
The other trials in a column vary only the axis that does not matter. Random search of the same 16 trials gives 16 distinct values of every parameter.
How should a learning rate be sampled in a random search?
Uniform sampling between 0.0001 and 0.1 puts about 90% of the draws above 0.01, which leaves the small-value region essentially unexplored.
What should you check if the best value found sits at the edge of the search range?
A best value at the boundary is evidence the optimum is beyond it. The search reported the best point it was allowed to consider, not the best point.
Cheat sheet
Grid Search against Random Search
Compare grid and random hyperparameter search side by side. See how random sampling covers more of the space with the same trial budget.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.