Quick Context
Gradient descent is the optimization engine that drives neural network training. It iteratively adjusts weights in the direction that reduces the loss function, like a ball rolling downhill on a loss landscape.
1) The Core Idea
Given a loss function L(w) that measures how wrong the model's predictions are, gradient descent computes the gradient ∇L(w) — the direction of steepest increase — and takes a step in the opposite direction:
w_new = w_old − α · ∇L(w_old)
where α (alpha) is the learning rate — the step size. Too large: you overshoot the minimum. Too small: training is painfully slow.
2) Three Variants
- Batch Gradient Descent — computes the gradient over the entire dataset each step. Accurate but slow and memory-heavy for large datasets.
- Stochastic Gradient Descent (SGD) — computes the gradient on a single sample. Fast and noisy, which can help escape local minima but causes high variance in updates.
- Mini-Batch Gradient Descent — computes the gradient on a small batch (e.g., 32–256 samples). The standard in practice: balances speed, memory, and stable updates.
3) The Training Loop
for each epoch:
shuffle dataset
for each mini-batch:
predictions = forward_pass(batch, weights)
loss = compute_loss(predictions, targets)
gradients = backpropagation(loss)
weights = weights - learning_rate * gradients
4) Learning Rate: The Most Important Hyperparameter
- Too high (e.g., 1.0) — loss oscillates wildly or diverges to infinity.
- Too low (e.g., 1e-7) — loss decreases but takes forever; may get stuck in a suboptimal minimum.
- Just right (e.g., 1e-3 to 1e-2) — smooth, steady decrease toward the minimum.
- Modern practice: start with a moderate LR and use a scheduler to decrease it over time.
5) Guided Experiments
- Start with default settings and observe the loss curve — it should decrease smoothly.
- Increase the learning rate dramatically and watch the loss diverge or oscillate.
- Use a very small learning rate and see how slowly the loss decreases.
- Try different batch sizes and observe the impact on loss curve smoothness.
6) Common Mistakes
- Not shuffling data between epochs — the model may learn ordering artifacts.
- Ignoring the loss curve — always plot training and validation loss; it tells you everything.
- Fixed learning rate throughout — using a scheduler almost always improves results.
- Confusing epochs with iterations — one epoch = full pass through the dataset; one iteration = one mini-batch update.
7) Key Takeaways
- Gradient descent updates weights by stepping opposite to the gradient of the loss.
- Mini-batch SGD is the de facto standard; pure batch and pure stochastic are extremes.
- The learning rate controls the step size and is the single most impactful hyperparameter.
- Always monitor both training and validation loss to detect divergence or overfitting.