Watch how Gradient Descent iteratively minimizes the Mean Squared Error (MSE) to find the best-fit line. Adjust the learning rate or take manual steps to see the weights update.
Controls
Slope (m)
Intercept (c)
Learning Rate (α)
Epochs (Steps):0
Tip: Drag the line away from the data, then use Gradient Descent to watch it walk its way back to the optimal fit. High learning rates might cause divergence! Drag the background to pan the view.
Visualization
Drag to pan & edit points
Loss & Gradients
Current State
y = 1.00x + 0.00MSE: 0.00
Calculated Gradients
∂L/∂m (Slope)0.00
∂L/∂c (Intercept)0.00
Weight Update Rule
mnew = m - α * (∂L/∂m)
cnew = c - α * (∂L/∂c)
Overview
The two gradients
You are fitting y = mx + c by minimising mean squared error. The partial derivatives are:
and each step is m ← m − η · ∂L/∂m, likewise for c. That is the same update rule a network with millions of params uses. The learning rate η controls the step size: too large and the params overshoot the min and diverge, too small and convergence takes thousands of iterations. For this two-param problem the loss surface is a paraboloid (a bowl). So there is just one min and gradient descent finds it no matter the starting point — a luxury that disappears in deeper networks where the surface has saddle points and local minima. The closed-form solution (the normal equation) computes the best m and c in one matrix step. And for linear regression it is faster and exact. Gradient descent turns needed when the net is nonlinear, the dataset is too large to invert a matrix. Or the loss function has no closed form. Stochastic gradient descent computes the gradient on a random subset (a mini-batch) rather than the full dataset, trading noise for speed. This page runs gradient descent step by step, plots the trajectory on the loss surface, and shows how the learning rate changes the path.
Watch how Gradient Descent iteratively minimizes the Mean Squared Error (MSE) to find the best-fit line.
Linear Regression with Gradient Descent: A Practical Guide
Fitting a straight line is the smallest problem that still uses the real training loop. Two parameters, a real loss, real gradients — everything a deep network does, at a scale you can follow by hand.
The two gradients
You are fitting y = mx + c by minimising mean squared error. The partial derivatives are:
and each step is m ← m − η · ∂L/∂m, likewise for c. That is the same update rule a network with millions of params uses. The learning rate η controls the step size: too large and the params overshoot the min and diverge, too small and convergence takes thousands of iterations. For this two-param problem the loss surface is a paraboloid (a bowl). So there is just one min and gradient descent finds it no matter the starting point — a luxury that disappears in deeper networks where the surface has saddle points and local minima. The closed-form solution (the normal equation) computes the best m and c in one matrix step. And for linear regression it is faster and exact. Gradient descent turns needed when the net is nonlinear, the dataset is too large to invert a matrix. Or the loss function has no closed form. Stochastic gradient descent computes the gradient on a random subset (a mini-batch) rather than the full dataset, trading noise for speed. This page runs gradient descent step by step, plots the trajectory on the loss surface, and shows how the learning rate changes the path.
One step, worked out
Three points: (1, 2), (2, 4), (3, 6). The answer is obviously m = 2, c = 0. But start from m = 0, c = 0 and let the maths find it.
With m = c = 0 each prediction is 0. So the residuals are 2, 4 and 6.
At a learning rate of 0.01: m ← 0 + 0.187 and c ← 0 + 0.08. The line has tilted slightly toward the data. Repeat a few hundred times and it lands on m = 2, c = 0.
The simplest thing gradient descent can do
Fit a line, y = wx + b, by iteratively adjusting w and b to reduce the squared error. It is the smallest complete case of all training does, with two params instead of millions.
Read those two expressions and the algorithm turns intuitive. Both are driven by the error (ŷ − y). The weight's gradient is also multiplied by x. So points far from the origin push the slope harder. The bias's gradient is just the average error. So the bias tracks the total offset.
Three steps by hand
Data: (1, 2), (2, 4), (3, 6) — the true line is y = 2x. Start at w = 0, b = 0, learning rate 0.1.
Step 2. Predictions are 2.67, 4.53, 6.40. Errors +0.67, +0.53, +0.40 — now overshooting slightly. And much less.
w ← 1.72, b ← 0.69
Step 3 onwards. The corrections keep shrinking. After a few hundred steps w ways 2 and b ways 0.
Two lessons visible in three steps. The first step is enormous since the error is enormous. And the steps shrink automatically as the fit improves — no schedule needed for that. And the loss overshot from negative to positive error: gradient descent oscillates towards a solution rather than approaching monotonically.
Why bother, when a formula exists
Ordinary least squares has an exact closed-form solution — the normal equation — so iterating towards the answer looks like a step backwards. Three reasons it is not:
Scale. The closed form needs inverting a matrix whose size is the count of features, at roughly cubic cost. And it needs the whole dataset in memory. Gradient descent works on mini-batches and scales to any size.
Generality. Change the loss to absolute error. Or add a non-linear activation, and the closed form disappears. Gradient descent needs just a derivative.
It is the same algorithm. This is the point of the exercise: what happens here is just what happens in a network with a billion params. Same loss, same chain rule, same update rule — just the count of params differs.
Try this above
Turn on Show Target OLS Fit so you can see where gradient descent should end up.
Set the learning rate very small and step repeatedly — correct way, painfully slow.
Raise it steadily. There is a point where each step overshoots the min and lands more away than it started. The loss rises instead of falling.
Set m and c by hand to something far from the answer and watch how the first few steps are large and later ones shrink — the gradient it gets less as you way.
What often goes wrong
Divergence from too high a learning rate is the classic failure. And its signature is unmistakable: the loss raises each step, often to infinity within a dozen iterations. If you see that, the fix is almost always η, not the net.Unscaled x vals cause a subtler problem. If x is in the thousands, ∂L/∂m is thousands of times more than ∂L/∂c. So a learning rate that suits one wrecks the other and the path zig-zags down a narrow valley.
In one line
Two params, the same optimiser as a deep network. And slow enough to watch each step.
The learning rate, showed
With two params the effect of the learning rate is easy to see right.
Learning rate
Behaviour on the example above
0.001
Converges, very slowly — thousands of steps
0.1
Converges smoothly in a few hundred steps
0.5
Oscillates around the answer before settling
1.5
Diverges — the loss grows until it overflows
The divergence case is worth grasp: if the step overshoots far enough that the new error is more than the old one, the next step is more still. And the process runs away. That is the same mechanism behind NaN losses in deep networks, visible here in two dimensions.
Feature scaling matters for the same reason as in networks. With x vals in the thousands, the gradient with respect to w is thousands of times more than the gradient with respect to b. So the loss surface turns a narrow ravine and no single learning rate suits both params.
import numpy as np
x = np.array([1, 2, 3], dtype=float)
y = np.array([2, 4, 6], dtype=float)
w, b, lr = 0.0, 0.0, 0.1
for step in range(500):
pred = w * x + b
err = pred - y
w -= lr * (2 / len(x)) * (err * x).sum()
b -= lr * (2 / len(x)) * err.sum()
print(w, b) # ~2.0, ~0.0
From this to a neural network
Three additions turn the code above into a network. And nothing else changes conceptually.
More params. Replace the scalar w with a matrix. And the multiplication with a matrix product. The gradient formula generalises right.
An activation function. Wrap the output in a non-linearity and the net can fit curves. The chain rule gains one factor per activation.
Layers. Feed one layer's output into the next. Backpropagation is the bookkeeping that computes all the gradients well rather than recomputing shared factors.
That is genuinely all of it. All else — Adam, batch normalisation, dropout, schedules — is a refinement of the same loop: predict, measure error, compute gradients, step.
The smallest possible training loop
One weight, one bias, and a loop -- then the same problem solved just. So you can watch gradient descent arrive at an answer you now know.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
n = 200
x = rng.uniform(0, 10, n)
TRUE_W, TRUE_B = 2.4, -3.1
y = TRUE_W * x + TRUE_B + rng.normal(0, 1.2, n)
print("%d points from y = %.1fx %+.1f, plus noise of sd 1.2." % (n, TRUE_W, TRUE_B))
print()
print("the loss, and its two derivatives, written out:")
print(" L = mean((w*x + b - y)^2)")
print(" dL/dw = mean(2 * (w*x + b - y) * x)")
print(" dL/db = mean(2 * (w*x + b - y))")
print()
Output
Building on the same setup, the next block shows the left cases.
example_02.pyNumPy
import numpy as np
w, b, lr = 0.0, 0.0, 0.02
print("%8s %10s %10s %14s %14s %12s"
% ("step", "w", "b", "dL/dw", "dL/db", "loss"))
for step in range(301):
err = w * x + b - y
dw = 2 * (err * x).mean()
db = 2 * err.mean()
if step in (0, 1, 2, 10, 50, 150, 300):
print("%8d %10.4f %10.4f %14.4f %14.4f %12.4f"
% (step, w, b, dw, db, (err ** 2).mean()))
w -= lr * dw
b -= lr * db
print()
print("the exact answer, from the normal equation:")
A = np.column_stack([x, np.ones(n)])
exact = np.linalg.solve(A.T @ A, A.T @ y)
print(" closed form w = %.6f, b = %.6f" % (exact[0], exact[1]))
print(" gradient descent w = %.6f, b = %.6f" % (w, b))
print(" true values w = %.6f, b = %.6f" % (TRUE_W, TRUE_B))
print()
print("the closed form is available here and is not for a neural network.")
print("that is the only reason we iterate -- not because iteration is better.")
print()
print("watch the derivatives shrink as it arrives. that is the automatic")
print("braking that makes a fixed learning rate work at all:")
w2, b2 = 0.0, 0.0
for step in range(6):
err = w2 * x + b2 - y
dw, db = 2 * (err * x).mean(), 2 * err.mean()
print(" step %d: |dL/dw| = %9.4f, so the step is %8.4f"
% (step, abs(dw), lr * abs(dw)))
w2 -= lr * dw; b2 -= lr * db
print()
Output
The next block covers the left steps.
example_03.pyNumPy
import numpy as np
print("the learning rate is the whole game, and it has a hard limit:")
L = 2 * np.linalg.eigvalsh(A.T @ A / n).max()
print(" the largest curvature here is %.4f, so anything above 2/%.4f = %.5f"
% (L, L, 2 / L))
print(" will diverge. testing that:")
for test_lr in (0.001, 0.01, 0.02, 0.028, 0.032):
ww, bb = 0.0, 0.0
for _ in range(300):
e2 = ww * x + bb - y
ww -= test_lr * 2 * (e2 * x).mean()
bb -= test_lr * 2 * e2.mean()
final = (ww * x + bb - y) ** 2
print(" lr=%.3f -> %s"
% (test_lr,
"diverged" if not np.isfinite(final.mean()) or final.mean() > 1e6
else "w=%.4f b=%.4f loss=%.4f" % (ww, bb, final.mean())))
print()
print("and one detail that matters more than it looks: the two parameters")
print("have very different gradient scales, because x runs from 0 to 10:")
err = 0 * x + 0 - y
print(" at the start, |dL/dw| = %.2f and |dL/db| = %.2f -- a factor of %.1f."
% (abs(2 * (err * x).mean()), abs(2 * err.mean()),
abs((err * x).mean() / err.mean())))
print(" so b converges long before w does. centring x fixes it:")
xc = x - x.mean()
Ac = np.column_stack([xc, np.ones(n)])
print(" curvature ratio before centring: %.2f"
% (np.linalg.eigvalsh(A.T @ A).max() / np.linalg.eigvalsh(A.T @ A).min()))
print(" after centring: %.2f"
% (np.linalg.eigvalsh(Ac.T @ Ac).max() / np.linalg.eigvalsh(Ac.T @ Ac).min()))
print(" that ratio is the condition number, and it is exactly the thing")
print(" feature scaling exists to reduce -- in a one-weight model and in a")
print(" billion-parameter one alike.")
Output
Questions people ask
Why the factor of 2 in the gradients? It comes from differentiating the square. Some texts use 1/(2n) in the loss so it cancels. It just rescales the effective learning rate.
Does it always converge? For linear regression the loss is convex. So with a small enough learning rate it converges to the unique global min. Networks are not convex, and there is no such guarantee.
How many iterations? Until the loss stops improving. For this problem, hundreds. For networks, thousands to millions of steps.
Do I need to scale the inputs? For gradient descent, yes — unscaled features create a ravine. The closed-form solution is unaffected.
What if I use absolute error instead? The gradient turns the sign of the error. So each step is the same size no matter how wrong you are. It is more reliable to outliers and slower to settle.
Is this how scikit-learn's LinearRegression works? No — it uses the closed form. SGDRegressor is the gradient-descent version.
Recap in one screen
Fit y = wx + b by stepping both params against the gradient of squared error.
Both gradients are driven by the error. The weight's is also scaled by the input.
Steps shrink automatically as the fit improves. Too large a rate diverges.
The closed-form solution exists but does not scale, does not generalise to other losses. And teaches nothing.
Add a matrix, an activation and a second layer. And this is just how a neural network trains.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
What does this module say about “The two gradients”?
What does this module say about “One step, worked out”?
Cheat sheet
Linear Regression with Gradient Descent
Take gradient descent one step at a time and watch the best-fit line and its mean squared error converge together.
DEEP LEARNING · vizlearn.in/deep_learning/linear_regression_with_gradient_descent.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.