Watch how Gradient Descent iteratively minimizes the Mean Squared Error (MSE) to find the best-fit line. Adjust the learning rate or take manual steps to see the weights update.
Overview
The two gradients
You are fitting y = mx + c by minimising mean squared error. The partial derivatives are:
and each step is m ← m − η · ∂L/∂m, likewise for c. That is the same update rule a network with millions of parameters uses.
Controls
Slope (m)
Intercept (c)
Learning Rate (α)
Epochs (Steps):0
Tip: Drag the line away from the data, then use Gradient Descent to watch it walk its way back to the optimal fit. High learning rates might cause divergence! Drag the background to pan the view.
Visualization
Drag to pan & edit points
Loss & Gradients
Current State
y = 1.00x + 0.00MSE: 0.00
Calculated Gradients
∂L/∂m (Slope)0.00
∂L/∂c (Intercept)0.00
Weight Update Rule
mnew = m - α * (∂L/∂m)
cnew = c - α * (∂L/∂c)
Linear Regression with Gradient Descent: A Practical Guide
Fitting a straight line is the smallest problem that still uses the real training loop. Two parameters, a real loss, real gradients — everything a deep network does, at a scale you can follow by hand.
One step, worked out
Three points: (1, 2), (2, 4), (3, 6). The answer is obviously m = 2, c = 0, but start from m = 0, c = 0 and let the maths find it.
With m = c = 0 every prediction is 0, so the residuals are 2, 4 and 6.
At a learning rate of 0.01: m ← 0 + 0.187 and c ← 0 + 0.08. The line has tilted slightly toward the data. Repeat a few hundred times and it lands on m = 2, c = 0.
The simplest thing gradient descent can do
Fit a line, y = wx + b, by iteratively adjusting w and b to reduce the squared error. It is the smallest complete example of everything training does, with two parameters instead of millions.
Read those two expressions and the algorithm becomes intuitive. Both are driven by the error (ŷ − y). The weight's gradient is additionally multiplied by x, so points far from the origin push the slope harder. The bias's gradient is just the average error, so the bias tracks the overall offset.
Three steps by hand
Data: (1, 2), (2, 4), (3, 6) — the true line is y = 2x. Start at w = 0, b = 0, learning rate 0.1.
Step 2. Predictions are 2.67, 4.53, 6.40; errors +0.67, +0.53, +0.40 — now overshooting slightly, and much smaller.
w ← 1.72, b ← 0.69
Step 3 onwards. The corrections keep shrinking. After a few hundred steps w approaches 2 and b approaches 0.
Two lessons visible in three steps. The first step is enormous because the error is enormous, and the steps shrink automatically as the fit improves — no schedule needed for that. And the loss overshot from negative to positive error: gradient descent oscillates towards a solution rather than approaching monotonically.
Why bother, when a formula exists
Ordinary least squares has an exact closed-form solution — the normal equation — so iterating towards the answer looks like a step backwards. Three reasons it is not:
Scale. The closed form requires inverting a matrix whose size is the number of features, at roughly cubic cost, and it needs the whole dataset in memory. Gradient descent works on mini-batches and scales to any size.
Generality. Change the loss to absolute error, or add a non-linear activation, and the closed form disappears. Gradient descent needs only a derivative.
It is the same algorithm. This is the point of the exercise: what happens here is exactly what happens in a network with a billion parameters. Same loss, same chain rule, same update rule — only the number of parameters differs.
Try this above
Turn on Show Target OLS Fit so you can see where gradient descent should end up.
Set the learning rate very small and step repeatedly — correct direction, painfully slow.
Raise it steadily. There is a point where each step overshoots the minimum and lands further away than it started; the loss rises instead of falling.
Set m and c by hand to something far from the answer and watch how the first few steps are large and later ones shrink — the gradient itself gets smaller as you approach.
What usually goes wrong
Divergence from too high a learning rate is the classic failure, and its signature is unmistakable: the loss increases every step, often to infinity within a dozen iterations. If you see that, the fix is almost always η, not the model.Unscaled x values cause a subtler problem. If x is in the thousands, ∂L/∂m is thousands of times larger than ∂L/∂c, so a learning rate that suits one wrecks the other and the path zig-zags down a narrow valley.
In one line
Two parameters, the same optimiser as a deep network, and slow enough to watch every step.
The learning rate, demonstrated
With two parameters the effect of the learning rate is easy to see directly.
Learning rate
Behaviour on the example above
0.001
Converges, very slowly — thousands of steps
0.1
Converges smoothly in a few hundred steps
0.5
Oscillates around the answer before settling
1.5
Diverges — the loss grows until it overflows
The divergence case is worth understanding: if the step overshoots far enough that the new error is larger than the old one, the next step is larger still, and the process runs away. That is the same mechanism behind NaN losses in deep networks, visible here in two dimensions.
Feature scaling matters for the same reason as in networks. With x values in the thousands, the gradient with respect to w is thousands of times larger than the gradient with respect to b, so the loss surface becomes a narrow ravine and no single learning rate suits both parameters.
import numpy as np
x = np.array([1, 2, 3], dtype=float)
y = np.array([2, 4, 6], dtype=float)
w, b, lr = 0.0, 0.0, 0.1
for step in range(500):
pred = w * x + b
err = pred - y
w -= lr * (2 / len(x)) * (err * x).sum()
b -= lr * (2 / len(x)) * err.sum()
print(w, b) # ~2.0, ~0.0
From this to a neural network
Three additions turn the code above into a network, and nothing else changes conceptually.
More parameters. Replace the scalar w with a matrix, and the multiplication with a matrix product. The gradient formula generalises directly.
An activation function. Wrap the output in a non-linearity and the model can fit curves. The chain rule gains one factor per activation.
Layers. Feed one layer's output into the next. Backpropagation is the bookkeeping that computes all the gradients efficiently rather than recomputing shared factors.
That is genuinely all of it. Everything else — Adam, batch normalisation, dropout, schedules — is a refinement of the same loop: predict, measure error, compute gradients, step.
The smallest possible training loop
One weight, one bias, and a loop -- then the same problem solved exactly, so you can watch gradient descent arrive at an answer you already know.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
n = 200
x = rng.uniform(0, 10, n)
TRUE_W, TRUE_B = 2.4, -3.1
y = TRUE_W * x + TRUE_B + rng.normal(0, 1.2, n)
print("%d points from y = %.1fx %+.1f, plus noise of sd 1.2." % (n, TRUE_W, TRUE_B))
print()
print("the loss, and its two derivatives, written out:")
print(" L = mean((w*x + b - y)^2)")
print(" dL/dw = mean(2 * (w*x + b - y) * x)")
print(" dL/db = mean(2 * (w*x + b - y))")
print()
w, b, lr = 0.0, 0.0, 0.02
print("%8s %10s %10s %14s %14s %12s"
% ("step", "w", "b", "dL/dw", "dL/db", "loss"))
for step in range(301):
err = w * x + b - y
dw = 2 * (err * x).mean()
db = 2 * err.mean()
if step in (0, 1, 2, 10, 50, 150, 300):
print("%8d %10.4f %10.4f %14.4f %14.4f %12.4f"
% (step, w, b, dw, db, (err ** 2).mean()))
w -= lr * dw
b -= lr * db
print()
print("the exact answer, from the normal equation:")
A = np.column_stack([x, np.ones(n)])
exact = np.linalg.solve(A.T @ A, A.T @ y)
print(" closed form w = %.6f, b = %.6f" % (exact[0], exact[1]))
print(" gradient descent w = %.6f, b = %.6f" % (w, b))
print(" true values w = %.6f, b = %.6f" % (TRUE_W, TRUE_B))
print()
print("the closed form is available here and is not for a neural network.")
print("that is the only reason we iterate -- not because iteration is better.")
print()
print("watch the derivatives shrink as it arrives. that is the automatic")
print("braking that makes a fixed learning rate work at all:")
w2, b2 = 0.0, 0.0
for step in range(6):
err = w2 * x + b2 - y
dw, db = 2 * (err * x).mean(), 2 * err.mean()
print(" step %d: |dL/dw| = %9.4f, so the step is %8.4f"
% (step, abs(dw), lr * abs(dw)))
w2 -= lr * dw; b2 -= lr * db
print()
print("the learning rate is the whole game, and it has a hard limit:")
L = 2 * np.linalg.eigvalsh(A.T @ A / n).max()
print(" the largest curvature here is %.4f, so anything above 2/%.4f = %.5f"
% (L, L, 2 / L))
print(" will diverge. testing that:")
for test_lr in (0.001, 0.01, 0.02, 0.028, 0.032):
ww, bb = 0.0, 0.0
for _ in range(300):
e2 = ww * x + bb - y
ww -= test_lr * 2 * (e2 * x).mean()
bb -= test_lr * 2 * e2.mean()
final = (ww * x + bb - y) ** 2
print(" lr=%.3f -> %s"
% (test_lr,
"diverged" if not np.isfinite(final.mean()) or final.mean() > 1e6
else "w=%.4f b=%.4f loss=%.4f" % (ww, bb, final.mean())))
print()
print("and one detail that matters more than it looks: the two parameters")
print("have very different gradient scales, because x runs from 0 to 10:")
err = 0 * x + 0 - y
print(" at the start, |dL/dw| = %.2f and |dL/db| = %.2f -- a factor of %.1f."
% (abs(2 * (err * x).mean()), abs(2 * err.mean()),
abs((err * x).mean() / err.mean())))
print(" so b converges long before w does. centring x fixes it:")
xc = x - x.mean()
Ac = np.column_stack([xc, np.ones(n)])
print(" curvature ratio before centring: %.2f"
% (np.linalg.eigvalsh(A.T @ A).max() / np.linalg.eigvalsh(A.T @ A).min()))
print(" after centring: %.2f"
% (np.linalg.eigvalsh(Ac.T @ Ac).max() / np.linalg.eigvalsh(Ac.T @ Ac).min()))
print(" that ratio is the condition number, and it is exactly the thing")
print(" feature scaling exists to reduce -- in a one-weight model and in a")
print(" billion-parameter one alike.")
Output
Questions people ask
Why the factor of 2 in the gradients? It comes from differentiating the square. Some texts use 1/(2n) in the loss so it cancels; it only rescales the effective learning rate.
Does it always converge? For linear regression the loss is convex, so with a small enough learning rate it converges to the unique global minimum. Networks are not convex, and there is no such guarantee.
How many iterations? Until the loss stops improving. For this problem, hundreds; for networks, thousands to millions of steps.
Do I need to scale the inputs? For gradient descent, yes — unscaled features create a ravine. The closed-form solution is unaffected.
What if I use absolute error instead? The gradient becomes the sign of the error, so every step is the same size regardless of how wrong you are. It is more robust to outliers and slower to settle.
Is this how scikit-learn's LinearRegression works? No — it uses the closed form. SGDRegressor is the gradient-descent version.
Recap in one screen
Fit y = wx + b by stepping both parameters against the gradient of squared error.
Both gradients are driven by the error; the weight's is additionally scaled by the input.
Steps shrink automatically as the fit improves; too large a rate diverges.
The closed-form solution exists but does not scale, does not generalise to other losses, and teaches nothing.
Add a matrix, an activation and a second layer, and this is exactly how a neural network trains.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
Two parameters, the same optimiser as a deep network, and slow enough to watch every step.
What does this module say about “The two gradients”?
You are fitting y = mx + c by minimising mean squared error. The partial derivatives are:
What does this module say about “One step, worked out”?
Three points: (1, 2), (2, 4), (3, 6). The answer is obviously m = 2, c = 0, but start from m = 0, c = 0 and let the maths find it.
Cheat sheet
Linear Regression with Gradient Descent
Watch how Gradient Descent iteratively minimizes the Mean Squared Error (MSE) to find the best-fit line. Adjust the learning rate or take manual steps to see the weights update.
DEEP LEARNING · vizlearn.in/deep_learning/linear_regression_with_gradient_descent.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.