Quick Context
Hyperparameters are settings defined before training begins — learning rate, batch size, number of layers, dropout rate, etc. Unlike model parameters (weights), they are not learned via gradient descent. Choosing good hyperparameters can be the difference between a model that converges in 10 minutes and one that never converges at all.
1) Parameters vs. Hyperparameters
- Parameters (weights, biases): learned automatically during training via backpropagation.
- Hyperparameters (learning rate, batch size, architecture choices): set by you before training. They control how learning happens.
2) Key Hyperparameters and Their Effects
- Learning Rate — the single most important hyperparameter. Too high → divergence. Too low → slow convergence. Start with 1e-3 for Adam, 1e-2 for SGD.
- Batch Size — larger batches = smoother gradients but less regularization. Smaller batches = noisier but better generalization. Common: 32, 64, 128, 256.
- Number of Layers/Neurons — more capacity = can learn more complex patterns but also overfit more easily.
- Dropout Rate — typically 0.1–0.5. Higher = more regularization. Too high = underfitting.
- Weight Decay (L2) — penalizes large weights. Typical: 1e-4 to 1e-2.
- Number of Epochs — combined with early stopping; set high and let early stopping decide when to stop.
3) Tuning Strategies
- Grid Search — try every combination of predefined values. Exhaustive but exponentially expensive. Only practical with 2–3 hyperparameters.
- Random Search — sample random combinations. Surprisingly effective because most hyperparameters have different sensitivities — random search explores more distinct values per parameter.
- Bayesian Optimization — builds a probabilistic model of the objective function and intelligently picks the next set of hyperparameters. Tools: Optuna, Hyperopt, Weights & Biases Sweeps.
- Manual Tuning — start with known good defaults, change one thing at a time, observe the result. Requires experience but is fast for simple problems.
4) Guided Experiments
- Start with default hyperparameters and observe baseline performance.
- Change only the learning rate — try 0.1, 0.01, 0.001, 0.0001 and compare loss curves.
- With the best LR, now vary batch size — notice how the loss curve smoothness changes.
- Add dropout with different rates — observe the gap between training and validation loss (smaller gap = less overfitting).
5) Common Mistakes
- Tuning multiple things at once — you can't tell what helped. Change one at a time.
- Tuning on the test set — this is data leakage. Use a validation set; touch the test set only once.
- Spending too long on grid search — random search with the same budget usually finds better results.
- Ignoring the learning rate — most training failures are learning rate problems.
6) Key Takeaways
- Hyperparameters control training behavior and must be set before learning begins.
- Learning rate is the most impactful — always tune it first.
- Random search is more efficient than grid search for most problems.
- Always validate on a held-out set, never the test set.