Optimise something while a constraint holds. The answer is where the level lines just graze the constraint, and lambda is what that costs.
Overview
Why the usual rule stops working
Unconstrained: find where the grad is zero.
Constrained: that rule fails right away, because the unconstrained optimum is often a place the constraint forbids. On the circle above, the goal grows forever in one direction and its grad is never zero anywhere.
Lagrange Multipliers
This module needs JavaScript: the
numbers are computed in the page rather than recorded.
Worth knowing
At an unconstrained optimum the gradient is zero. At a constrained one it need not be — it only has to point along the constraint's gradient.
∇f = λ ∇g. That proportionality is the whole method.
Geometrically: the level line of the objective is tangent to the constraint. If it crossed, you could slide along and do better.
λ is the shadow price — how much the optimum improves per unit of loosened constraint.
Lagrange Multipliers
Optimising under a constraint, and the multiplier that turns out to be worth as much as the answer.
The condition
Look at where the level lines meet the circle.
At most points they cross. Crossing means you can slide along the circle and move to a better level line. So You aren't at the optimum.
At the optimum they're tangent — touching without crossing. Sliding either way makes things worse.
Tangency means the two grads are parallel:
grad f = lambda * grad g
That is the method entire. Solve it together with the constraint itself and you have the pick points.
Drag the goal weights and watch the marked point travel around the circle, staying just where the family of parallel lines grazes it.
The multiplier is the interesting part
lambda looks like bookkeeping. It isn't: it's the shadow price of the constraint — the rate at which the best value raises as the constraint is relaxed.
Drag the radius control. The optimum's value rises, and the readout's lambda says how fast per unit of radius.
In economics that is truly a price: how much more profit one more unit of room is worth, and so what you should be willing to pay for it. In machine learning it's the same quantity under different names — the smoothing strength in ridge regression, and the dual variables in an SVM that spot just which points are support vectors.
A lambda of zero says the constraint isn't binding: you'd have chosen that point anyway, and loosening it buys nothing.
Where it shows up
Ridge regression. "Cut error subject to the coeffs being small" is a constrained problem. The penalised form everyone really writes is its Lagrangian, and the fine weight is lambda.
Support vector machines. The dual formulation is Lagrangian, the multipliers are per-training-point, and the ones that come out non-zero *are* the support vectors.
max entropy. Finding the spread with the most entropy subject to matching known moments makes the exponential family, and the multipliers become its natural params.
PCA. Raising variance subject to unit-length direction gives Cv = lambda v — the multiplier turns out to be the eigenval.
Physics and economics throughout, wherever a thing is optimised under a budget.
Inequalities
Real constraints are often g(x) <= c rather than g(x) = c, and the extension is the KKT rules. The addition worth remembering is *complementary slackness*: for each constraint, either it's tight and its multiplier may be non-zero, or it's slack and its multiplier is zero.
That is what makes SVMs sparse. Points comfortably on the correct side of the margin have slack constraints, so their multipliers are zero, so they contribute nothing to the answer. Only the points pressed against the margin live.
The shadow price, worked
The claim that lambda is a price is easiest to trust on a case you can check by hand. Cut x² + y² — distance-squared from the origin — subject to 3x + 4y = c. The constraint is a line, and the closest point on a line 3x + 4y = c to the origin is at distance c/5 (because √(3²+4²) = 5), so the min value is (c/5)².
Now watch what one unit of c buys:
c
minimum x²+y²
change
24
23.04
25
25.00
+1.96
26
27.04
+2.04
The optimum rises by about 2 per unit of c, and the multiplier the method returns is lambda = 2c/25 = 2 at c = 25. The two agree: lambda *is* the rate at which the best value changes as the constraint moves — the price of one more unit of the resource. The small gap between 1.96 and 2.04 is the curvature of that price across the step. In the limit of an infinitesimal change it's just lambda. This is why, in a real problem, the multiplier is often worth more than the optimum: it tells you what relaxing the constraint is worth before you pay to relax it.
This is why the multiplier reappears under other names across machine learning: it's the smoothing strength in ridge regression (the price of the coeff-size budget), and it's the per-point dual variable in an SVM, where the points whose multiplier comes out non-zero are just the support vectors. In each case lambda answers the question the optimum can't: not "what is the best value" but "what is the constraint costing me". This Is often the number a choice really turns on.
A multiplier of zero carries its own message: the constraint isn't binding, you would have chosen that point anyway, and relaxing it buys nothing — which is as handy to know as a large multiplier that says the constraint is costly and worth paying to loosen.
For inequality constraints the same idea extends to the KKT rules, whose complementary slackness says each constraint is either tight with a possibly non-zero multiplier or slack with a zero one. Which is just what makes an SVM sparse, since points comfortably inside the margin contribute nothing.
Where it goes wrong with Lagrange Multipliers
Losing it finds stationary points, not maxima. The rule holds at constrained minima and saddles too. Check which you've.
given a answer exists. An unbounded goal on an unbounded constraint set has none.
Reading lambda's sign carelessly. It depends on how the Lagrangian was written, and the sign rule differs between texts.
Skipping the constraint qualification. The method assumes the constraint grads are well behaved at the answer. At a cusp or where constraints are collapse it can fail.
Questions people ask about Lagrange Multipliers
What problem do Lagrange multipliers solve? Optimising a function subject to an equality constraint. At the optimum, the gradient of the objective is parallel to the gradient of the constraint.
What does the multiplier itself mean? It is the rate at which the optimal value changes if you relax the constraint slightly. Economists call it the shadow price.
How do I set up the equations? Form L = f - lambda times g, then set all partial derivatives of L, including the one in lambda, to zero and solve.
How do inequality constraints change things? They need the KKT conditions. The multiplier must be non-negative, and either the constraint is active or its multiplier is zero.
Where are they used in machine learning? In the SVM derivation, in maximum entropy models, and in constrained optimisation such as regularisation viewed as a budget.
Does the method find the global optimum? It finds candidate stationary points. You still have to compare them, or know the problem is convex, to be sure.
Optimise with a rule you are not allowed to break
The closest point on a line to the origin, found three ways. By brute force, by substitution. By the multiplier rule. All three agree. The multiplier turns out to mean a thing.
example_01.pyNumPy
import numpy as np
# minimise x^2 + y^2 subject to 3x + 4y = 25
def f(x, y): return x ** 2 + y ** 2
def g(x, y): return 3 * x + 4 * y - 25
print("brute force along the constraint line:")
xs = np.linspace(-5, 12, 400_001)
ys = (25 - 3 * xs) / 4; vals = f(xs, ys); i = vals.argmin()
print(" best point (%.4f, %.4f) f = %.4f" % (xs[i], ys[i], vals[i]))
print("by substitution: df/dx = 0 -> x = 3")
x_star, y_star = 3.0, (25 - 9) / 4
print(" x = %.4f, y = %.4f f = %.4f" % (x_star, y_star, f(x_star, y_star)))
print("by Lagrange: grad f = lam * grad g, plus the constraint.")
# grad f = (2x, 2y); grad g = (3, 4)
# 2x = 3 lam , 2y = 4 lam , 3x + 4y = 25
A = np.array([[2.0, 0.0, -3.0],
[0.0, 2.0, -4.0],
[3.0, 4.0, 0.0]])
b = np.array([0.0, 0.0, 25.0])
x_l, y_l, lam = np.linalg.solve(A, b)
print(" x = %.4f, y = %.4f, lambda = %.4f" % (x_l, y_l, lam))
print(" constraint satisfied: g = %.10f" % g(x_l, y_l))
print()
print("gradients are parallel: grad f = %s, grad g = %s"
% (np.round([2*x_l, 2*y_l], 4).tolist(), [3, 4]))
print("lambda is the price of the constraint:")
mins = []
for c in (24.0, 25.0, 26.0):
xs2 = np.linspace(-5, 12, 200_001)
best = f(xs2, (c - 3 * xs2) / 4).min()
mins.append(best)
print(" 3x + 4y = %.0f -> minimum f = %.4f" % (c, best))
print(" delta: %.4f then %.4f; lambda is %.4f"
% (mins[1] - mins[0], mins[2] - mins[1], lam))
print("lambda tells you what the constraint costs -- often more useful than")
print("the optimum itself.")
Output
Check yourself
0 of 3
Answer without scrolling back up.
At a constrained optimum, what is true of the level line and the constraint?
If they crossed you could slide along the constraint onto a better level line. Tangency means the gradients are parallel, which is the condition.
What does the multiplier lambda measure?
The shadow price. In economics it is literally what an extra unit of capacity is worth; in ridge regression it is the regularisation strength.
Why are SVMs sparse?
Points comfortably on the correct side contribute nothing to the solution. Only the ones pressed against the margin have non-zero multipliers - those are the support vectors.
Cheat sheet
Lagrange Multipliers
Optimise a function under a constraint by introducing a multiplier. See how the multiplier itself tells you the cost of the constraint.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.