The slope of a straight line is easy. The derivative asks a harder question: what is the slope of a curve at a single point? Shrink the gap and watch the answer appear.
Controls
point x2.0
gap h2.000
0.0013
The amber line is the secant through two points. The green line is the true tangent.
Curve, Secant and Tangent
drag the blue point
Live Calculation
secant slope
0
true f′(x)
0
error
0
behaviour
–
Derivatives and Slope
From the slope of a line to the slope at a point — the idea every training algorithm is built on.
The idea in brief
In y = mx + c the slope m is one number for the whole line. A curve has no single slope — it changes everywhere. The derivative gives you the slope at each individual point.
The Trick: Two Points, Then Squeeze
Slope needs two points, so start by cheating — take your point x and another a distance h away, and find the slope of the line through both. That line is the secant:
Now shrink h. The second point slides toward the first, and the secant pivots until it just grazes the curve. That limiting line is the tangent, and its slope is the derivative:
Press Shrink h → 0 and watch the amber secant rotate onto the green tangent while the error collapses toward zero. That animation is the definition of a derivative.
Reading the Sign
f′(x) > 0 — the function is rising here.
f′(x) = 0 — flat: a peak, a valley, or a plateau. Optimisation lives and dies at these points.
f′(x) < 0 — the function is falling.
Tick plot the derivative and slide x across the curve: wherever the original function turns around, its derivative crosses zero.
Why Machine Learning Cares
Training a model means minimising a loss. The derivative answers the only question that matters at each step: which way is downhill, and how steep is it? Gradient descent then takes a step against the slope:
A large derivative means a steep slope and a big correction; a derivative near zero means you have arrived somewhere flat. Every optimiser on this site — SGD, Adam, the lot — is a refinement of that one line.
The slope of a curve at a single point
A straight line has one slope everywhere: rise over run, and any two points give the same answer. A curve does not — it is steep in places and flat in others.
The derivative is the answer to "how steep is this curve right here". You get it by taking two points on the curve, computing the slope between them, and sliding them closer and closer together:
f′(x) = limh→0 [ f(x + h) − f(x) ] / h
Concretely, for f(x) = x² at x = 3, with h shrinking:
h
Slope between x and x+h
1
(16 − 9) / 1 = 7
0.1
(9.61 − 9) / 0.1 = 6.1
0.01
(9.0601 − 9) / 0.01 = 6.01
0.001
… = 6.001
The numbers are converging on 6, and the rule f′(x) = 2x confirms it: 2×3 = 6. The limit is not a trick; it is the value the slope approaches as the two points merge.
Reading the sign and the size
The derivative carries two pieces of information, and both are used constantly in machine learning:
The sign says which way the function is heading. Positive means increasing as x grows; negative means decreasing; zero means momentarily flat.
The magnitude says how fast. A derivative of 100 is a cliff; a derivative of 0.01 is nearly level.
A derivative of zero marks a stationary point — a minimum, a maximum, or a saddle. That is why optimisation is the search for points where the derivative vanishes, and why "the gradient is zero" means training has stopped moving.
The second derivative distinguishes the cases: positive means a valley (minimum), negative means a peak (maximum), zero is inconclusive.
The rules you actually need
Function
Derivative
c (a constant)
0
x
1
xⁿ
n xⁿ⁻¹
eˣ
eˣ
ln x
1/x
sin x
cos x
c f(x)
c f′(x)
f(x) + g(x)
f′(x) + g′(x)
Two more that do the heavy lifting in neural networks:
Product rule: (fg)′ = f′g + fg′
Chain rule: (f(g(x)))′ = f′(g(x)) · g′(x)
The chain rule is the one that matters most: backpropagation is the chain rule applied layer by layer, from the loss backwards to every weight.
Worked example: f(x) = 3x⁴ + 2x − 7. Differentiate term by term: 12x³ + 2 + 0 = 12x³ + 2. At x = 1 the slope is 14.
Why any of this is in a machine learning course
Training a model means minimising a loss function, and the derivative is what tells you which way to step.
new weight = old weight − learning rate × derivative of loss w.r.t. that weight
The minus sign is the whole idea: the derivative points uphill, so you move the opposite way. The learning rate decides how big the step is.
The derivatives of the activation functions explain some famous training problems directly:
Sigmoid has a maximum derivative of 0.25, and it approaches zero at both ends. Multiply several of those together through a deep network and the gradient vanishes.
ReLU has a derivative of exactly 1 for positive inputs and exactly 0 for negative ones. The 1 is why it trains well; the 0 is why neurons can die.
Shrink h and watch
The slope of a chord approaches the derivative as h shrinks — and then stops approaching it, because floating point runs out of digits before the maths runs out of accuracy.
example_01.pyNumPy
import numpy as np
def f(x):
return x ** 2
x0 = 3.0
print("f(x) = x^2, exact derivative at x=3 is 2*3 = 6")
print()
print("%12s %14s %12s" % ("h", "(f(x+h)-f(x))/h", "error"))
for h in (1.0, 0.1, 0.01, 1e-4, 1e-8, 1e-12):
slope = (f(x0 + h) - f(x0)) / h
print("%12.0e %14.8f %12.2e" % (h, slope, abs(slope - 6.0)))
print()
print("smaller h is better until it is not: past about 1e-8 the")
print("subtraction loses precision faster than the approximation gains it.")
Output
Guided tour
Start on f(x) = x² at x = 2 with h large. The secant is visibly wrong; shrink h and the error falls to almost nothing.
Slide to x = 0. The tangent goes flat — the derivative is exactly 0 at the bottom of the parabola, which is what an optimiser is hunting for.
Switch to f(x) = 2x + 1. The secant equals the tangent at every h, because a straight line has the same slope everywhere. There is nothing to shrink.
Try 3 sin(x) and slide across. The derivative swings positive and negative as the wave rises and falls, hitting zero at every crest and trough.
Try e^(x/2) — its derivative grows in proportion to the function itself, which is the defining property of exponentials.
In one line
A derivative is the slope of the tangent, obtained by taking the slope between two points and letting the gap shrink to nothing. Positive means rising, negative means falling, zero means flat — and that sign is the compass every learning algorithm follows.
Where derivatives fail to exist
Not every function has a derivative everywhere, and the exceptions are not academic.
Corners.f(x) = |x| has slope −1 on the left and +1 on the right, and no single answer at zero. ReLU has exactly this corner at the origin. Frameworks resolve it by convention, defining the derivative at zero to be 0 (or sometimes 1), and it causes no practical trouble because landing exactly on zero is vanishingly unlikely.
Jumps. A step function has no derivative at the jump, and a derivative of zero everywhere else. That is precisely why step activations were abandoned: gradient descent has nothing to follow.
Vertical tangents. The slope becomes infinite, as with the cube root at zero.
This is the reason loss functions are chosen to be smooth. Mean squared error is differentiable everywhere; accuracy is a step function of the predictions and cannot be optimised directly, which is why models train on cross-entropy and are evaluated on accuracy.
Numerical derivatives, and automatic differentiation
You can approximate a derivative without any calculus by using a small h:
The central difference (using x−h and x+h) is noticeably more accurate than the one-sided version for the same h. Too large an h and the approximation is poor; too small and floating-point rounding destroys it — around 10⁻⁵ to 10⁻⁷ is the usual sweet spot for doubles.
Deep learning frameworks do not use this. They use automatic differentiation: every operation records how to compute its own derivative, and the chain rule is applied backwards through the recorded graph. The result is exact to floating-point precision and costs about the same as the forward computation — which is what makes training networks with billions of parameters feasible.
Numerical derivatives still have a job: checking that a hand-written gradient is correct. If your analytic gradient and the numerical one disagree, the analytic one is wrong.
Questions people ask
What is the difference between a derivative and a gradient? A derivative is for a function of one variable; a gradient is the vector of partial derivatives for a function of many. Loss functions have millions of inputs, so gradients are what you meet.
Does a zero derivative always mean a minimum? No — it could be a maximum or a saddle point. In high dimensions saddle points are far more common than local minima.
Why does the learning rate matter so much? The derivative gives a direction, not a distance. Too large a step overshoots and diverges; too small and training crawls.
Is the derivative of a constant really zero? Yes — a constant does not change, so its rate of change is zero. This is why the bias term differentiates cleanly.
What does the second derivative tell me? How the slope itself is changing — the curvature. Second-order optimisers use it to choose better step sizes, at a much higher cost per step.
Do I need to compute derivatives by hand? Not in practice; frameworks do it. Knowing the rules is what lets you understand vanishing gradients, dead neurons and why some losses are chosen over others.
Recap in one screen
The derivative is the slope of a curve at a point, defined as a limit of slopes between two points that merge.
The sign says which way the function goes; the magnitude says how fast.
Zero derivative means a stationary point — minimum, maximum or saddle.
Gradient descent steps against the derivative, scaled by the learning rate.
Corners and jumps have no derivative, which is why losses are chosen to be smooth.
Frameworks use automatic differentiation, not finite differences — exact and cheap.
Check yourself
0 of 3
Answer without scrolling back up.
The derivative of a function at a point tells you:
A derivative is an instantaneous rate of change - the slope of the curve at that exact point, which is the tangent line's slope.
At a minimum of a smooth curve, the derivative is:
The curve is momentarily flat at the bottom. This is exactly what gradient descent chases: it keeps stepping until the gradient is near zero and there is no downhill direction left.
Gradient descent subtracts the gradient rather than adding it. Why?
The gradient points in the direction of steepest increase. To reduce the loss you move against it - which is the minus sign in every update rule you will ever see.
Cheat sheet
Derivatives and Slope
The slope of a straight line is easy. The derivative asks a harder question: what is the slope of a curve at a single point? Shrink the gap and watch the answer appear.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.