Experiment with automated search strategies. Use Grid Search to systematically scan parameters or Random Search to find high-performance zones efficiently.
Overview
Two kinds of number
A parameter is learned: weights and biases move during training because gradient descent moves them. A hyperparameter is fixed before training and never updated by the optimiser — learning rate, number of layers, neurons per layer, batch size, dropout rate, regularisation strength.
The distinction matters because you cannot use gradient descent to find hyperparameters. There is no derivative of validation accuracy with respect to “number of layers”. The only way to evaluate a hyperparameter setting is to train a model with it and see what happens, which is why tuning is expensive.
0.03
4
10
Tuning Dashboard
READY
Trial Status
-- / --
Best Accuracy
0.0%
Search History
#
LR
Nodes
Accuracy
Start a search to see trials.
Drag to Pan | Scroll to Zoom
Hyperparameter Tuning: A Practical Guide
Parameters are learned from data; hyperparameters are chosen before training starts. Searching that second set well is often worth more than any change to the model itself.
Grid search, random search, and why random usually wins
Grid search tries every combination on a predefined grid. With 5 learning rates and 5 layer widths that is 25 runs, and adding a third hyperparameter with 5 values makes it 125. The cost is exponential in the number of hyperparameters.Random search samples combinations at random from ranges you specify. Counterintuitively it usually finds better settings for the same budget, and the reason is sharp: not all hyperparameters matter equally. If learning rate dominates and layer width barely matters, a 5×5 grid tests only 5 distinct learning rates across 25 runs. Random search with 25 samples tests 25 distinct learning rates. You get five times the resolution on the axis that counts, at no extra cost.Bayesian optimisation goes further, building a model of which regions look promising and sampling there. It is more efficient per trial and worth the complexity when each run takes hours.
The settings you choose rather than learn
Parameters are learned by gradient descent. Hyperparameters are the settings you choose before training starts, and they decide how well that learning goes.
Not all of them matter equally, and the difference is large enough that tuning them in the wrong order wastes most of the effort:
Hyperparameter
Impact
Typical range
Learning rate
Enormous
1e-5 to 1e-1, log scale
Batch size
Moderate
32–256
Number of epochs
Moderate
Decided by early stopping
Weight decay
Moderate
1e-5 to 1e-1, log scale
Architecture depth / width
Moderate
Problem-dependent
Dropout rate
Small to moderate
0.0–0.5
Optimiser choice
Small
AdamW by default
Adam's β values
Negligible
Leave at defaults
The learning rate deserves its position. It affects results more than the number of layers, the width, or which optimiser you pick — and it is the cheapest to find, via a short exponential sweep.
Random search beats grid search
This is the most useful practical result in the topic, and it is counterintuitive.
A grid search over 3 learning rates and 3 dropout rates tries 9 combinations — but only 3 distinct learning rates. If the learning rate is what matters and dropout barely does, you spent 9 runs to sample 3 values of the important parameter.
Random search with 9 trials samples 9 distinct learning rates. For the same budget it explores the important dimension three times as thoroughly, and the result generalises: when only a few hyperparameters matter, random search finds better configurations than grid search at equal cost.
Bayesian optimisation goes further by modelling the relationship between settings and score, and choosing each next trial where the expected improvement is highest. Optuna and similar libraries implement this, and they add another useful feature — pruning, which abandons a trial early once it is clearly losing.
Note log=True on the learning rate and weight decay. Both span orders of magnitude, and sampling them uniformly wastes almost every trial in the top decade.
An order that does not waste effort
Get a working baseline first. One run, sensible defaults, no tuning. Everything is measured against this.
Find the learning rate with an exponential sweep. Biggest gain, smallest cost.
Fix the schedule — warm-up plus cosine decay.
Set the batch size to what fits, scaling the rate accordingly.
Tune regularisation — weight decay, then dropout — guided by the train/validation gap.
Adjust architecture if the model is clearly under- or over-capacity.
Random or Bayesian search over whatever remains, if the budget allows.
Two habits that matter more than the search algorithm: change one thing at a time so you can attribute the effect, and fix the random seed so a 1% difference is a real difference rather than run-to-run variation.
Which knobs matter, and how to search them
Hyperparameters are not equal -- a few dominate and the rest are noise. This measures which is which on a real training loop, then compares three ways of searching.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
def make(n, seed):
r = np.random.default_rng(seed)
x = r.uniform(-3, 3, (n, 1))
y = np.sin(1.5 * x) + 0.25 * x + r.normal(0, 0.3, (n, 1))
return x, y
Xtr, ytr = make(120, 0)
Xva, yva = make(600, 1)
def train(hidden, lr, epochs, wd, seed=0):
r = np.random.default_rng(seed)
W1 = r.normal(0, 1.0, (1, hidden)); b1 = r.normal(0, 0.5, (1, hidden))
W2 = r.normal(0, 1.0 / np.sqrt(hidden), (hidden, 1)); b2 = np.zeros((1, 1))
n = len(Xtr)
for _ in range(epochs):
a1 = np.tanh(Xtr @ W1 + b1)
d = 2 * (a1 @ W2 + b2 - ytr) / n
dW2 = a1.T @ d + wd * W2
d1 = (d @ W2.T) * (1 - a1 ** 2)
dW1 = Xtr.T @ d1 + wd * W1
W2 -= lr * dW2; b2 -= lr * d.sum(0, keepdims=True)
W1 -= lr * dW1; b1 -= lr * d1.sum(0, keepdims=True)
if not np.isfinite(W1).all():
return np.inf
val = np.tanh(Xva @ W1 + b1) @ W2 + b2
mse = ((val - yva) ** 2).mean()
return mse if np.isfinite(mse) else np.inf
BASE = dict(hidden=16, lr=0.05, epochs=400, wd=0.0)
print("baseline: %s -> validation MSE %.4f"
% (BASE, train(**BASE)))
print()
print("ONE AT A TIME. change a single knob, hold the rest:")
sweeps = {
"lr": [0.001, 0.01, 0.05, 0.2],
"hidden": [2, 4, 16, 64],
"epochs": [60, 200, 400, 1000],
"wd": [0.0, 1e-2, 1e-1],
}
spread = {}
for key, values in sweeps.items():
results = []
for v in values:
cfg = dict(BASE); cfg[key] = v
results.append((v, train(**cfg)))
finite = [m for _, m in results if np.isfinite(m)]
spread[key] = (max(finite) - min(finite)) if finite else np.inf
print(" %-8s %s" % (key, " ".join(
"%s=%s" % (v, "diverged" if not np.isfinite(m) else "%.4f" % m)
for v, m in results)))
print()
print("the spread each knob produces -- this is the ranking that matters:")
for key, sp in sorted(spread.items(), key=lambda kv: -kv[1]):
print(" %-8s %.4f %s" % (key, sp, "#" * int(min(sp, 2.0) * 25)))
print()
print("this is the usual shape: one or two knobs dominate and the rest barely")
print("register. tuning in that order is most of the skill -- and it is why")
print("learning rate is almost always the first thing anyone tunes.")
print()
print("but one-at-a-time misses INTERACTIONS. lr and epochs are not")
print("independent -- a smaller rate simply needs more steps:")
print("%10s %10s %10s %10s" % ("", "150 ep", "400 ep", "1000 ep"))
for lr in (0.005, 0.05):
row = []
for ep in (150, 400, 1000):
cfg = dict(BASE); cfg["lr"], cfg["epochs"] = lr, ep
m = train(**cfg)
row.append("diverged" if not np.isfinite(m) else "%.4f" % m)
print("%10s %10s %10s %10s" % ("lr=%.3f" % lr, *row))
print(" read down a column and across a row: a rate ten times smaller is")
print(" not simply worse, it is worse AT A GIVEN BUDGET. give it three times")
print(" the epochs and it catches up. the two knobs trade against each")
print(" other, and a one-at-a-time sweep through the baseline cannot see")
print(" that -- it only ever moves along one axis of this table.")
print()
print("GRID vs RANDOM, on the same budget of 8 trials (400 epochs each):")
grid = [(h, lr) for h in (4, 16) for lr in (0.01, 0.05, 0.1, 0.2)]
best_g = min(((train(hidden=h, lr=lr, epochs=1200, wd=0.0), h, lr)
for h, lr in grid), key=lambda t: t[0])
print(" grid (2 hidden x 4 lr): best %.4f at hidden=%d, lr=%.3f"
% best_g)
r = np.random.default_rng(3)
rand = [(int(r.integers(2, 129)), float(10 ** r.uniform(-3, -0.3)))
for _ in range(8)]
best_r = min(((train(hidden=h, lr=lr, epochs=1200, wd=0.0), h, lr)
for h, lr in rand), key=lambda t: t[0])
print(" random (8 draws) : best %.4f at hidden=%d, lr=%.4f"
% best_r)
print(" the grid tried 2 distinct values of hidden and 4 of lr.")
print(" random tried %d and %d, for the same number of fits."
% (len({h for h, _ in rand}), len({lr for _, lr in rand})))
print()
print(" on this particular draw the grid happened to win. that is worth")
print(" saying plainly: random search is not reliably better on any ONE")
print(" run. its advantage is that it does not waste budget re-testing a")
print(" parameter that turns out not to matter, and that advantage grows")
print(" with the number of parameters -- which is exactly when a grid")
print(" becomes unaffordable anyway.")
print()
print("note the lr range above is 10**uniform(-3, -0.3), not uniform(0.001,")
print("0.5). learning rate lives on a log scale, and sampling it linearly")
print("puts most of your budget in a region that is almost always too large.")
print()
print("the practical order of operations:")
print(" 1. get the learning rate roughly right first. nothing else matters")
print(" until it is.")
print(" 2. set epochs by early stopping rather than tuning it.")
print(" 3. then capacity (width, depth), then regularisation.")
print(" 4. random search over log-uniform ranges, not a grid.")
print(" 5. check the winner is not on the edge of your range. if it is,")
print(" the range was wrong -- widen it and search again.")
Output
Things to try
Run a small search. Set N Trials to 5 and press Run Trial repeatedly. Results scatter widely — five samples is not enough to distinguish a good region from luck.
Give it a real budget. Press Reset Search, set N Trials to 20, and run again. A pattern emerges: certain learning-rate bands consistently do better regardless of the other settings.
Compare the strategies. Switch Search Strategy between grid and random at the same N Trials. Random covers the learning-rate axis far more finely, because grid keeps re-testing the same few values.
Test which knob matters. Hold Learning Rate fixed and vary Neurons/Layer across its range, then do the reverse. The learning rate changes the outcome far more — that asymmetry is the whole argument for random search.
Search on a log scale
Learning rate should be sampled logarithmically, not uniformly. Sampling uniformly from 0.0001 to 0.1 puts 90% of the samples above 0.01, leaving the small-rate region — where the answer usually is — almost untested.
Sample the exponent instead: draw uniformly from −4 to −1 and use 10x. That gives equal attention to 0.0001–0.001, 0.001–0.01, and 0.01–0.1. The same applies to regularisation strength and any other quantity that spans orders of magnitude.
What trips people up
Tuning on the test set. The single most damaging mistake. Selecting hyperparameters by test performance leaks the test set into the model and the reported score becomes optimistic. Tune on a validation split and touch test once, at the end.
Sampling learning rate uniformly. Wastes most of the budget in a range that is almost always too large.
Tuning everything at once with no budget. Start with learning rate, which usually dominates, then batch size and architecture. Regularisation is worth tuning only once the model can overfit.
Ignoring seed variance. If two settings differ by less than the run-to-run variation from random initialisation, you have not measured a difference. Repeat the promising ones.
Key takeaway
Hyperparameters sit outside gradient descent, so the only way to evaluate them is to train and measure — which makes the search strategy itself worth thinking about. Random search beats grid search at equal budget because it spends its samples on more distinct values of whichever hyperparameter actually matters, and anything spanning orders of magnitude should be sampled on a log scale. Whatever the strategy, select on validation data and keep the test set untouched.
Not tuning on the test set
Every hyperparameter decision made by looking at a score consumes some of that set's independence. Search fifty configurations against a validation set and keep the best, and its score is optimistic — you selected the luckiest of fifty.
The disciplined arrangement is three splits: train fits the parameters, validation chooses the hyperparameters, and test is looked at once, at the end, to produce the number you report.
For a rigorous estimate, nested cross-validation puts the hyperparameter search inside an outer loop whose test folds never participate in tuning. It costs k times more compute, and it is what you want when the number will inform a real decision.
The pragmatic middle ground — cross-validation for tuning, plus one genuinely untouched hold-out set for the final number — is what most projects should do.
Budget, and knowing when to stop
Hyperparameter search has diminishing returns, and it is worth knowing roughly where they set in.
The first few trials usually find most of the available gain, because they fix the learning rate. Trials ten to fifty refine it. Beyond a hundred, improvements on a fixed dataset are typically fractions of a percent — at which point better data, better features or a different architecture is a far better use of the compute.
Two shortcuts worth knowing. Tune on a subset — 10–20% of the data — to find a promising region cheaply, then verify the best few configurations on the full dataset. And use early pruning, so a configuration that is clearly losing after two epochs does not consume twenty.
Questions people ask
What should I tune first? The learning rate, always. It dominates.
How many trials do I need? Twenty to fifty random or Bayesian trials cover most of the available gain for a handful of hyperparameters.
Grid or random search? Random, or Bayesian. Grid search wastes its budget on dimensions that do not matter.
Should I tune the architecture? Only after the training recipe is right, and prefer a proven architecture over designing one.
Do I need to tune Adam's betas? Almost never. The defaults are robust.
Why do my results change between runs with the same settings? Random initialisation, shuffling and non-deterministic GPU kernels. Fix the seeds, and treat differences smaller than the run-to-run variation as noise.
Recap in one screen
Hyperparameters are chosen, not learned, and they are not equally important.
The learning rate matters most and is the cheapest to find — sweep it first.
Random and Bayesian search beat grid search at equal budget, because only a few dimensions matter.
Sample learning rate and weight decay on a log scale.
Tune on validation, report on a test set you touched once, and fix the seed before comparing anything.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
Hyperparameters sit outside gradient descent, so the only way to evaluate them is to train and measure — which makes the search strategy itself worth thinking about. Random search beats grid search at equal budget because it spends its samples on more distinct values of whichever hyperparameter actually matters, and anything spanning orders of magnitude should be sampled on a log scale.
What does this module say about “Two kinds of number”?
A parameter is learned: weights and biases move during training because gradient descent moves them. A hyperparameter is fixed before training and never updated by the optimiser — learning rate, number of layers, neurons per layer, batch size, dropout rate, regularisation strength.
What does this module say about “Grid search, random search, and why random usually wins”?
Grid search tries every combination on a predefined grid. With 5 learning rates and 5 layer widths that is 25 runs, and adding a third hyperparameter with 5 values makes it 125. The cost is exponential in the number of hyperparameters. Random search samples combinations at random from ranges you specify.
Cheat sheet
Hyperparameter Tuning
Experiment with automated search strategies. Use Grid Search to systematically scan parameters or Random Search to find high-performance zones efficiently.
DEEP LEARNING · vizlearn.in/deep_learning/hyper-paramter_tuning.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.