Experiment with automated search strategies. Use Grid Search to systematically scan parameters or Random Search to find high-performance zones efficiently.
0.03
4
10
Tuning Dashboard
READY
Trial Status
-- / --
Best Accuracy
0.0%
Search History
#
LR
Nodes
Accuracy
Start a search to see trials.
Drag to Pan | Scroll to Zoom
Overview
Two kinds of count
A param is learned: weights and biases move in training since gradient descent moves them. A hyperparameter is fixed before training and never updated by the optimiser — learning rate, count of layers, neurons per layer, batch size, dropout rate, regularisation strength.
The distinction matters since you cannot use gradient descent to find hyperparameters. There is no derivative of validation accuracy with respect to “count of layers”. The just way to test a hyperparameter setting is to train a net with it and see what happens. It is why tuning is costly. The three main ways — grid search, random search. And Bayesian optimisation — differ in how they explore the space. Grid search is exhaustive and its cost is exponential in the count of hyperparameters. Random search samples uniformly and gets better coverage of each axis for the same budget, since not all hyperparameters matter equally. Bayesian ways (Optuna, Hyperopt) build a surrogate net of the objective and focus trials where the surrogate predicts improvement. It is more fast per trial but adds complexity. The validation score used to check settings must come from held-out data — never from the training set — and cross-validation is the standard choice. Early stopping (halting training when the validation loss stops improving) is both a regulariser and a tuning shortcut: it removes “count of epochs” from the search space fully. This page runs grid and random search on the same net, prints the best settings from each. And covers the Bayesian option.
Parameters are learned from data; hyperparameters are chosen before training starts. Searching that second set well is often worth more than any change to the model itself.
Two kinds of count
A param is learned: weights and biases move in training since gradient descent moves them. A hyperparameter is fixed before training and never updated by the optimiser — learning rate, count of layers, neurons per layer, batch size, dropout rate, regularisation strength.
The distinction matters since you cannot use gradient descent to find hyperparameters. There is no derivative of validation accuracy with respect to “count of layers”. The just way to test a hyperparameter setting is to train a net with it and see what happens. It is why tuning is costly. The three main ways — grid search, random search. And Bayesian optimisation — differ in how they explore the space. Grid search is exhaustive and its cost is exponential in the count of hyperparameters. Random search samples uniformly and gets better coverage of each axis for the same budget, since not all hyperparameters matter equally. Bayesian ways (Optuna, Hyperopt) build a surrogate net of the objective and focus trials where the surrogate predicts improvement. It is more fast per trial but adds complexity. The validation score used to check settings must come from held-out data — never from the training set — and cross-validation is the standard choice. Early stopping (halting training when the validation loss stops improving) is both a regulariser and a tuning shortcut: it removes “count of epochs” from the search space fully. This page runs grid and random search on the same net, prints the best settings from each. And covers the Bayesian option.
Grid search, random search, and why random often wins
Grid search tries each combination on a predefined grid. With 5 learning rates and 5 layer widths that is 25 runs. And adding a third hyperparameter with 5 vals makes it 125. The cost is exponential in the count of hyperparameters.Random search samples combinations at random from ranges you name. Counterintuitively it often finds better settings for the same budget. And the reason is sharp: not all hyperparameters matter equally. If learning rate dominates and layer width barely matters, a 5×5 grid tests just 5 distinct learning rates across 25 runs. Random search with 25 samples tests 25 distinct learning rates. You get five times the resolution on the axis that counts, at no extra cost.Bayesian optimisation goes more, building a net of which regions look promising and sampling there. It is more fast per trial and worth the complexity when each run takes hours.
The settings you choose rather than learn
Params are learned by gradient descent. Hyperparameters are the settings you choose before training starts. And they decide how well that learning goes.
Not all of them matter equally. And the gap is large enough that tuning them in the wrong order wastes most of the effort:
Hyperparameter
Impact
Typical range
Learning rate
Enormous
1e-5 to 1e-1, log scale
Batch size
Moderate
32–256
Number of epochs
Moderate
Decided by early stopping
Weight decay
Moderate
1e-5 to 1e-1, log scale
Architecture depth / width
Moderate
Problem-dependent
Dropout rate
Small to moderate
0.0–0.5
Optimiser choice
Small
AdamW by default
Adam's β values
Negligible
Leave at defaults
The learning rate deserves its position. It affects results more than the count of layers, the width. Or which optimiser you pick — and it is the cheapest to find, via a short exponential sweep.
Random search beats grid search
This is the most useful real result in the topic. And it is counterintuitive.
A grid search over 3 learning rates and 3 dropout rates tries 9 combinations — but just 3 distinct learning rates. If the learning rate is what matters and dropout barely does, you spent 9 runs to sample 3 vals of the key param.
Random search with 9 trials samples 9 distinct learning rates. For the same budget it explores the key dimension three times as thoroughly. And the result generalises: when just a few hyperparameters matter, random search finds better configurations than grid search at equal cost.
Bayesian optimisation goes more by modelling the relationship between settings and score. And choosing each next trial where the expected improvement is highest. Optuna and close libraries build this. And they add another useful feature — pruning. It abandons a trial early once it is clearly losing.
Note log=True on the learning rate and weight decay. Both span orders of magnitude. And sampling them uniformly wastes almost each trial in the top decade.
An order that does not waste effort
Get a working baseline first. One run, sensible defaults, no tuning. All is measured against this.
Find the learning rate with an exponential sweep. Biggest gain, smallest cost.
Fix the schedule — warm-up plus cosine decay.
Set the batch size to what fits, scaling the rate accordingly.
Tune regularisation — weight decay, then dropout — guided by the train/validation gap.
Adjust design if the net is clearly under- or over-capacity.
Random or Bayesian search over whatever stays, if the budget lets.
Two habits that matter more than the search algorithm: change one thing at a time so you can attribute the effect. And fix the random seed so a 1% gap is a real gap rather than run-to-run variation.
Which knobs matter, and how to search them
Hyperparameters are not equal -- a few dominate and the rest are noise. This measures which is which on a real training loop, then checks three ways of searching.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
def make(n, seed):
r = np.random.default_rng(seed)
x = r.uniform(-3, 3, (n, 1))
y = np.sin(1.5 * x) + 0.25 * x + r.normal(0, 0.3, (n, 1))
return x, y
Xtr, ytr = make(120, 0)
Xva, yva = make(600, 1)
def train(hidden, lr, epochs, wd, seed=0):
r = np.random.default_rng(seed)
W1 = r.normal(0, 1.0, (1, hidden)); b1 = r.normal(0, 0.5, (1, hidden))
W2 = r.normal(0, 1.0 / np.sqrt(hidden), (hidden, 1)); b2 = np.zeros((1, 1))
n = len(Xtr)
for _ in range(epochs):
a1 = np.tanh(Xtr @ W1 + b1)
d = 2 * (a1 @ W2 + b2 - ytr) / n
dW2 = a1.T @ d + wd * W2
d1 = (d @ W2.T) * (1 - a1 ** 2)
dW1 = Xtr.T @ d1 + wd * W1
W2 -= lr * dW2; b2 -= lr * d.sum(0, keepdims=True)
W1 -= lr * dW1; b1 -= lr * d1.sum(0, keepdims=True)
if not np.isfinite(W1).all():
return np.inf
val = np.tanh(Xva @ W1 + b1) @ W2 + b2
mse = ((val - yva) ** 2).mean()
return mse if np.isfinite(mse) else np.inf
BASE = dict(hidden=16, lr=0.05, epochs=400, wd=0.0)
print("baseline: %s -> validation MSE %.4f"
% (BASE, train(**BASE)))
print()
Output
Building on the same setup, the next block shows the left cases.
example_02.pyNumPy
import numpy as np
print("ONE AT A TIME. change a single knob, hold the rest:")
sweeps = {
"lr": [0.001, 0.01, 0.05, 0.2],
"hidden": [2, 4, 16, 64],
"epochs": [60, 200, 400, 1000],
"wd": [0.0, 1e-2, 1e-1],
}
spread = {}
for key, values in sweeps.items():
results = []
for v in values:
cfg = dict(BASE); cfg[key] = v
results.append((v, train(**cfg)))
finite = [m for _, m in results if np.isfinite(m)]
spread[key] = (max(finite) - min(finite)) if finite else np.inf
print(" %-8s %s" % (key, " ".join(
"%s=%s" % (v, "diverged" if not np.isfinite(m) else "%.4f" % m)
for v, m in results)))
print()
print("the spread each knob produces -- this is the ranking that matters:")
for key, sp in sorted(spread.items(), key=lambda kv: -kv[1]):
print(" %-8s %.4f %s" % (key, sp, "#" * int(min(sp, 2.0) * 25)))
print()
print("this is the usual shape: one or two knobs dominate and the rest barely")
print("register. tuning in that order is most of the skill -- and it is why")
print("learning rate is almost always the first thing anyone tunes.")
print()
Output
The next block covers the left steps.
example_03.pyNumPy
import numpy as np
print("but one-at-a-time misses INTERACTIONS. lr and epochs are not")
print("independent -- a smaller rate simply needs more steps:")
print("%10s %10s %10s %10s" % ("", "150 ep", "400 ep", "1000 ep"))
for lr in (0.005, 0.05):
row = []
for ep in (150, 400, 1000):
cfg = dict(BASE); cfg["lr"], cfg["epochs"] = lr, ep
m = train(**cfg)
row.append("diverged" if not np.isfinite(m) else "%.4f" % m)
print("%10s %10s %10s %10s" % ("lr=%.3f" % lr, *row))
print(" read down a column and across a row: a rate ten times smaller is")
print(" not simply worse, it is worse AT A GIVEN BUDGET. give it three times")
print(" the epochs and it catches up. the two knobs trade against each")
print(" other, and a one-at-a-time sweep through the baseline cannot see")
print(" that -- it only ever moves along one axis of this table.")
print()
print("GRID vs RANDOM, on the same budget of 8 trials (400 epochs each):")
grid = [(h, lr) for h in (4, 16) for lr in (0.01, 0.05, 0.1, 0.2)]
best_g = min(((train(hidden=h, lr=lr, epochs=1200, wd=0.0), h, lr)
for h, lr in grid), key=lambda t: t[0])
print(" grid (2 hidden x 4 lr): best %.4f at hidden=%d, lr=%.3f"
% best_g)
r = np.random.default_rng(3)
rand = [(int(r.integers(2, 129)), float(10 ** r.uniform(-3, -0.3)))
Output
The next block continues with the left demonstrations.
example_04.pyNumPy
for _ in range(8)]
best_r = min(((train(hidden=h, lr=lr, epochs=1200, wd=0.0), h, lr)
for h, lr in rand), key=lambda t: t[0])
print(" random (8 draws) : best %.4f at hidden=%d, lr=%.4f"
% best_r)
print(" the grid tried 2 distinct values of hidden and 4 of lr.")
print(" random tried %d and %d, for the same number of fits."
% (len({h for h, _ in rand}), len({lr for _, lr in rand})))
print()
print(" on this particular draw the grid happened to win. that is worth")
print(" saying plainly: random search is not reliably better on any ONE")
print(" run. its advantage is that it does not waste budget re-testing a")
print(" parameter that turns out not to matter, and that advantage grows")
print(" with the number of parameters -- which is exactly when a grid")
print(" becomes unaffordable anyway.")
print()
print("note the lr range above is 10**uniform(-3, -0.3), not uniform(0.001,")
print("0.5). learning rate lives on a log scale, and sampling it linearly")
print("puts most of your budget in a region that is almost always too large.")
print()
print("the practical order of operations:")
print(" 1. get the learning rate roughly right first. nothing else matters")
print(" until it is.")
print(" 2. set epochs by early stopping rather than tuning it.")
print(" 3. then capacity (width, depth), then regularisation.")
print(" 4. random search over log-uniform ranges, not a grid.")
print(" 5. check the winner is not on the edge of your range. if it is,")
print(" the range was wrong -- widen it and search again.")
Output
Things to try
Run a small search. Set N Trials to 5 and press Run Trial repeatedly. Results scatter widely — five samples is not enough to distinguish a good region from luck.
Give it a real budget. Press Reset Search, set N Trials to 20. And run again. A pattern emerges: certain learning-rate bands always do better no matter the other settings.
Check the plans. Switch Search Plan between grid and random at the same N Trials. Random covers the learning-rate axis far more finely, since grid keeps re-testing the same few vals.
Test which knob matters. Hold Learning Rate fixed and vary Neurons/Layer across its range, then do the reverse. The learning rate changes the outcome far more — that asymmetry is the whole argument for random search.
Search on a log scale
Learning rate should be sampled logarithmically, not uniformly. Sampling uniformly from 0.0001 to 0.1 puts 90% of the samples above 0.01, leaving the small-rate region — where the answer often is — almost untested.
Sample the exponent instead: draw uniformly from −4 to −1 and use 10x. That gives equal attention to 0.0001–0.001, 0.001–0.01, and 0.01–0.1. The same applies to regularisation strength and any other quantity that spans orders of magnitude.
What trips people up
Tuning on the test set. The single most damaging mistake. Selecting hyperparameters by test performance leaks the test set into the net and the reported score turns optimistic. Tune on a validation split and touch test once, at the end.
Sampling learning rate uniformly. Wastes most of the budget in a range that is almost always too large.
Tuning all at once with no budget. Start with learning rate. It often dominates, then batch size and design. Regularisation is worth tuning just once the net can overfit.
Ignoring seed variance. If two settings differ by less than the run-to-run variation from random initialisation, you have not measured a gap. Repeat the promising ones.
Key takeaway
Hyperparameters sit outside gradient descent. So the just way to test them is to train and measure — which makes the search plan it worth thinking about. Random search beats grid search at equal budget since it spends its samples on more distinct vals of whichever hyperparameter really matters. And anything spanning orders of magnitude should be sampled on a log scale. Whatever the plan, select on validation data and keep the test set untouched.
Not tuning on the test set
Each hyperparameter decision made by looking at a score consumes some of that set's independence. Search fifty configurations against a validation set and keep the best. And its score is optimistic — you selected the luckiest of fifty.
The disciplined arrangement is three splits: train fits the params, validation chooses the hyperparameters. And test is looked at once, at the end, to give the count you report.
For a rigorous estimate, nested cross-validation puts the hyperparameter search inside an outer loop whose test folds never participate in tuning. It costs k times more compute. And it is what you want when the count will inform a real decision.
The pragmatic middle ground — cross-validation for tuning, plus one genuinely untouched hold-out set for the final count — is what most projects should do.
Budget, and knowing when to stop
Hyperparameter search has diminishing returns. And it is worth knowing roughly where they set in.
The first few trials often find most of the on hand gain, since they fix the learning rate. Trials ten to fifty refine it. Beyond a hundred, improvements on a fixed dataset are often fractions of a percent — at which point better data, better features or a other design is a far better use of the compute.
Two shortcuts worth knowing. Tune on a subset — 10–20% of the data — to find a promising region cheaply, then verify the best few configurations on the full dataset. And use early pruning. So a setup that is clearly losing after two epochs does not consume twenty.
Questions people ask
What should I tune first? The learning rate, always. It dominates.
How many trials do I need? Twenty to fifty random or Bayesian trials cover most of the on hand gain for a handful of hyperparameters.
Grid or random search? Random, or Bayesian. Grid search wastes its budget on dimensions that do not matter.
Should I tune the design? Just after the training recipe is right. And prefer a proven design over designing one.
Do I need to tune Adam's betas? Almost never. The defaults are reliable.
Why do my results change between runs with the same settings? Random initialisation, shuffling and non-deterministic GPU kernels. Fix the seeds, and treat gaps less than the run-to-run variation as noise.
Recap in one screen
Hyperparameters are chosen, not learned, and they are not equally key.
The learning rate matters most and is the cheapest to find — sweep it first.
Random and Bayesian search beat grid search at equal budget, since just a few dimensions matter.
Sample learning rate and weight decay on a log scale.
Tune on validation, report on a test set you touched once. And fix the seed before checking anything.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
What does this module say about “Two kinds of count”?
What does this module say about “Grid search, random search, and why random often wins”?
Cheat sheet
Hyperparameter Tuning
Run grid search and random search side by side on the same model and see which finds strong hyperparameters in fewer trials.
DEEP LEARNING · vizlearn.in/deep_learning/hyper-paramter_tuning.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.