The first derivative says which way is downhill. The second says how far you can safely go.
Overview
From one derivative to a table of them
For a function of one variable the deriv is one number: the slope. The chain rule and partial derivs modules cover that ground.
Real functions have many inputs and often many outputs, and the derivs have to be organised.
The grad applies when there are many inputs and one output — a loss function. It's a vector holding one partial deriv per input, and it points in the direction of steepest grow.
The Jacobian applies when there are many outputs too. It's a mat with one row per output and one column per input, entry (i, j) being how much output *i* changes when input *j* moves. A grad is a Jacobian with one row.
The Hessian is the mat of second derivs of a single-output function: entry (i, j) is the deriv of the *i*-th partial deriv with respect to input *j*. It shows how the grad changes as you move.
The Jacobian and the Hessian
This module needs JavaScript: the
numbers are computed in the page rather than recorded.
Worth knowing
The gradient collects the first partial derivatives: one number per input, saying which way is uphill.
The Jacobian generalises it to functions with several outputs — one row per output, one column per input.
The Hessian collects the second derivatives. It describes curvature: how the gradient itself changes.
Backpropagation is repeated Jacobian multiplication, done right to left because that keeps the intermediate object a vector rather than a matrix.
The Jacobian and the Hessian
The two mats of derivs each tuner is built on. Why one of them is almost never found.
What curvature buys you
The visualisation is one-dimensional. Here The grad and the Hessian are each a single number, so the tie is visible without any mat form.
The dashed line is the tangent. The first deriv. It says which way is downhill and how steeply, and it says nothing at all about how long that remains true.
The second deriv is the missing piece. Drag the curvature control:
Large plus curvature is a tight valley. The grad changes quickly, so a big step overshoots and lands on the far wall.
Small plus curvature is a shallow bowl. The grad stays roughly constant, so a big step is safe and a small one wastes time.
That is just the info a step size wants, and it's why an tuner that knows the curvature can choose its own step instead of being handed a learning rate.
Minus curvature is a max in that direction. Descent will accelerate away from it, which is often what you want and is why saddle points are less dangerous than they sound.
For a multivariable function the Hessian's eigenvals carry the same info per direction: all plus is a min, all minus a max, mixed signs a saddle.
Newton's method, and why nobody runs it on a network
Knowing the curvature suggests an obvious improvement over gradient descent: step by the grad scaled by the inverse Hessian rather than by a fixed learning rate. That is Newton's method, and near a min it settles as n² — roughly doubling the correct digits each iteration, where gradient descent plods.
It isn't used for neural networks, for a reason that is pure math.
A model with *n* params has a Hessian with *n*² entries. At a million params that is 10¹² numbers to store, and inverting it costs about *n*³. For a model with a billion params the Hessian doesn't fit in the observable universe, let alone in memory.
So the field uses guesses. L-BFGS keeps a low-rank guess built from recent grads. Adam and its relatives keep a main-line guess — one number per param rather than a full mat — which is why they adapt a per-param step size and why they work at scale. Adam is at times given as a main-line guess to second-order optimisation, and that is a fair reading of what it does.
Backpropagation is Jacobians
each layer of a network is a function with many inputs and many outputs, so each has a Jacobian. The chain rule says the deriv of the whole network is the product of them all.
The order of that times is the whole trick. Timesing left to right — forward mode — keeps a full mat at each step. Timesing right to left — reverse mode. This Is backpropagation — starts from the scalar loss, so each mid stays a vector.
That is why training is affordable. The cost of a backward pass is a small multiple of a forward pass, even so of how many params there are, because no Jacobian is ever formed in full.
The curvature test, on two functions
In one dim a plus second deriv means a min and a minus one a max. In many dims the Hessian's *eigenvals* play that role, and two small examples show the whole classification.
f = x² + xy + y² has Hessian [[2, 1], [1, 2]], whose eigenvals are 2 + 1 = 3 and 2 − 1 = 1. Both plus. each direction curves upward, so the stationary point at the origin is a genuine min.
f = x² − y² has Hessian [[2, 0], [0, −2]], eigenvals 2 and −2.
function
Hessian eigenvalues
the origin is
x² + xy + y²
3, 1 (all positive)
a minimum
x² − y²
2, −2 (mixed sign)
a saddle
The mixed sign is the saddle: up along x, down along y, so the grad is zero without it being any kind of extremum. This is just the test Cholesky performs cheaply — a good factorisation means all eigenvals are plus, confirming a min — and it's why, in the millions of dims of a network, stationary points are almost always saddles: needing *each* eigenval plus at once turns vanishingly unlikely.
Where it goes wrong with the Jacobian and the Hessian
Trying to find a Hessian for a large model. It doesn't fit. Use a main-line or low-rank guess.
given a zero grad means a min. It means a stationary point. The Hessian's eigenvals tell you which kind.
Confusing the Jacobian with the grad. The grad is the special case of one output. Frameworks expose jacobian and grad apart for just this reason.
Expecting second-order methods to be a drop-in win. They shine on small, well-conditioned problems and on batch goals, not on stochastic mini-batch training where the curvature estimate is noise.
Questions people ask about the Jacobian and the Hessian
What is the Jacobian? It is the matrix of all first partial derivatives of a vector-valued function. Row i holds the gradient of output i, so it is the best linear approximation near a point.
What is the Hessian? It is the matrix of second partial derivatives of a scalar function. It describes the curvature in every direction.
What does the Hessian tell me at a critical point? If it is positive definite you have a local minimum, negative definite a local maximum, and mixed signs a saddle point.
Why is the Hessian rarely used in deep learning? It has one entry per pair of parameters. A network with a million parameters would need a trillion-entry matrix, so first-order methods and approximations win.
How is the Jacobian used in backpropagation? Each layer's backward pass multiplies the incoming gradient by that layer's Jacobian. Frameworks do it without ever forming the full matrix.
Is the Hessian the Jacobian of something? Yes, it is the Jacobian of the gradient. That is why it is symmetric when the second derivatives are continuous.
First derivatives in a grid, second derivatives in another
The Jacobian collects each first partial deriv of a vector function. The Hessian collects each second partial of a scalar one. Both are built here by number and checked against the closed form.
example_01.pyNumPy
import numpy as np
# f: R^2 -> R^3
def f(v):
x, y = v
return np.array([x ** 2 * y, 5 * x + np.sin(y), x * y ** 3])
def jacobian(fn, v, h=1e-6):
base = fn(v)
cols = []
for i in range(len(v)):
step = np.zeros_like(v); step[i] = h
cols.append((fn(v + step) - base) / h)
return np.column_stack(cols)
p = np.array([2.0, 1.0])
print("Jacobian of f at (2, 1) -- 3 outputs by 2 inputs")
print(np.round(jacobian(f, p), 4))
x, y = p
print("
by hand:")
print(np.round(np.array([[2*x*y, x**2], [5.0, np.cos(y)], [y**3, 3*x*y**2]]), 4))
# g: R^2 -> R
def g(v):
x, y = v
return x ** 3 + 2 * x * y ** 2 + y ** 3
def hessian(fn, v, h=1e-4):
n = len(v)
H = np.zeros((n, n))
for i in range(n):
for j in range(n):
a, b = np.zeros(n), np.zeros(n)
a[i] = h; b[j] = h
H[i, j] = (fn(v + a + b) - fn(v + a) - fn(v + b) + fn(v)) / (h * h)
return H
H = hessian(g, p)
print("Hessian of g at (2, 1)")
print(np.round(H, 3))
print("it is symmetric:", np.allclose(H, H.T, atol=1e-3))
print()
eigs = np.round(np.linalg.eigvalsh(H), 4)
print("eigenvalues %s -> %s" % (eigs,
"bowl (minimum)" if all(eigs > 0) else "saddle or dome"))
Output
Check yourself
0 of 3
Answer without scrolling back up.
What does the Hessian tell you that the gradient does not?
The tangent says which way and how steeply; it says nothing about how long that remains true. Curvature is exactly what a step size wants to know.
Why is Newton's method not used to train large neural networks?
Near a minimum it converges quadratically, which is far better than gradient descent - it is purely the arithmetic that rules it out. Adam keeps a diagonal approximation instead.
Why does backpropagation multiply Jacobians right to left?
Left to right would carry a full matrix at every step. Reverse mode is why a backward pass costs a small multiple of a forward pass at any parameter count.
Cheat sheet
The Jacobian and the Hessian
Compute the Jacobian and Hessian for small functions. See how optimisers use first and second derivatives to choose direction and step size.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.