The condition
Look at where the level lines meet the circle.
At most points they cross. Crossing means you can slide along the circle and move to a better level line, so you are not at the optimum.
At the optimum they are tangent — touching without crossing. Sliding either way makes things worse.
Tangency means the two gradients are parallel:
grad f = lambda * grad g
That is the method entire. Solve it together with the constraint itself and you have the candidate points.
Drag the objective weights and watch the marked point travel around the circle, staying exactly where the family of parallel lines grazes it.
The multiplier is the interesting part
lambda looks like bookkeeping. It is not: it is the shadow price of the constraint — the rate at which the optimal value improves as the constraint is relaxed.
Drag the radius control. The optimum's value rises, and the readout's lambda says how fast per unit of radius.
In economics that is literally a price: how much more profit one more unit of capacity is worth, and therefore what you should be willing to pay for it. In machine learning it is the same quantity under different names — the regularisation strength in ridge regression, and the dual variables in an SVM that identify exactly which points are support vectors.
A lambda of zero says the constraint is not binding: you would have chosen that point anyway, and loosening it buys nothing.
Where it shows up
Ridge regression. "Minimise error subject to the coefficients being small" is a constrained problem; the penalised form everyone actually writes is its Lagrangian, and the penalty weight is lambda.
Support vector machines. The dual formulation is Lagrangian, the multipliers are per-training-point, and the ones that come out non-zero *are* the support vectors.
Maximum entropy. Finding the distribution with the most entropy subject to matching known moments produces the exponential family, and the multipliers become its natural parameters.
PCA. Maximising variance subject to unit-length direction gives Cv = lambda v — the multiplier turns out to be the [eigenvalue](eigenvalues_and_eigenvectors.html).
Physics and economics throughout, wherever something is optimised under a budget.
Inequalities
Real constraints are often g(x) <= c rather than g(x) = c, and the extension is the KKT conditions. The addition worth remembering is *complementary slackness*: for each constraint, either it is tight and its multiplier may be non-zero, or it is slack and its multiplier is zero.
That is what makes SVMs sparse. Points comfortably on the correct side of the margin have slack constraints, so their multipliers are zero, so they contribute nothing to the solution. Only the points pressed against the margin survive.
Where it goes wrong
Forgetting it finds stationary points, not maxima. The condition holds at constrained minima and saddles too. Check which you have.
Assuming a solution exists. An unbounded objective on an unbounded constraint set has none.
Reading lambda's sign carelessly. It depends on how the Lagrangian was written, and the sign convention differs between texts.
Skipping the constraint qualification. The method assumes the constraint gradients are well behaved at the solution; at a cusp or where constraints are degenerate it can fail.