Repeated multiplication instead of repeated addition. Exponential growth looks harmless for a long time and then becomes overwhelming — watch it outrun a polynomial that starts far ahead of it.
Controls
base b2.00
evaluate at x4.0
Doubling / half-life
y = b^x
growth vs decay
Live Calculation
b^x
0
x³
0
growth per step
0
behaviour
–
Exponentials
Growth proportional to size — slow to start, then impossible to ignore.
Before the details
In y = b^x the variable sits in the exponent. Each step of 1 in x multiplies y by b, rather than adding to it. That single difference is what separates exponential from linear behaviour.
Three Regimes, Set by the Base
b > 1 — growth. Each step multiplies up. Compound interest, viral spread, unchecked model divergence.
b = 1 — flat. 1 to any power is 1. The boundary between the other two.
0 < b < 1 — decay. Each step multiplies down, approaching zero without ever arriving. Learning-rate schedules, radioactive decay, the vanishing gradient.
Slide the base across 1 and watch the curve flip from climbing to falling.
Exponential Always Wins Eventually
Keep the polynomial comparison on and look at the left of the plot: x³ is far ahead. Now slide x rightward. There is a crossover point, and after it 2^x leaves x³ hopelessly behind.
This holds for any polynomial and any base above 1 — exponential growth always overtakes eventually. It is the reason exponential-time algorithms are considered intractable no matter how fast the hardware, and the reason a "small" compounding effect is never actually small.
Doubling Time and Half-Life
A defining feature of exponentials: the time to double is constant. It takes just as long to go from 1 to 2 as from a million to two million. The panel computes this live — at base 2 it is exactly 1 step, at base 1.1 about 7.3 steps.
Why e Is Special
Set the base to e ≈ 2.71828. This is the base at which the curve's slope equals its own height at every point:
No other base has this property, which is why e turns up throughout calculus and why exp is the exponential in almost every ML formula — softmax, sigmoid, Gaussian densities and exponential learning-rate decay all use it.
Growth that feeds on itself
Linear growth adds the same amount each step. Exponential growth multiplies by the same factor each step, so the amount added grows too.
f(x) = a × bˣ
a is the starting value and b is the growth factor. b > 1 grows, 0 < b < 1 decays.
The difference is easy to underestimate. Add 10 a day starting from 100 and after 30 days you have 400. Multiply by 1.1 a day and you have 1,745.
Day
Linear (+10)
Exponential (×1.1)
0
100
100
10
200
259
20
300
673
30
400
1,745
60
700
30,448
The exponential column starts slower and ends in a different universe. That is the shape behind compound interest, viral spread, and the growth of model sizes over the last decade.
The number e, and why it is the default base
e ≈ 2.71828 is the growth factor you get from compounding continuously — the limit of (1 + 1/n)ⁿ as n grows.
Its special property is the reason it is everywhere in calculus:
d/dx eˣ = eˣ
The function is its own derivative. No other base does this (any other base picks up a constant factor of ln(b)), and that is what makes e the natural choice for anything involving rates of change — which is to say, anything involving optimisation.
Its inverse is the natural logarithm: eln(x) = x, and ln(eˣ) = x.
Where exponentials appear in machine learning
The sigmoid.1 / (1 + e^-x) squashes any real number into (0, 1). The exponential is what makes it smooth, and its derivative — s(x)(1 − s(x)) — is what makes it trainable.
Softmax.e^zi / Σ e^zj turns a vector of scores into probabilities that sum to 1. Exponentiating first is what guarantees positivity and exaggerates the differences between scores, so a small lead in logits becomes a large lead in probability.
Learning rate decay.lr = lr0 × e^(-kt) reduces the step size smoothly as training proceeds.
Exponential moving averages. Momentum, Adam and batch normalisation all keep a running average that weights recent values more heavily, with the weights falling off exponentially.
Radial basis kernels.e^(-γ||x-y||²) measures similarity that falls away rapidly with distance, and is what gives an RBF SVM its local behaviour.
Decay, and half-lives
With 0 < b < 1, or a negative exponent, the same shape runs downhill: fast at first, then a long thinning tail that never quite reaches zero.
The half-life is the time taken to halve. It is constant, which is the defining property of exponential decay: whatever the current amount, half of it disappears in the same interval.
That is exactly the behaviour of an exponential moving average with decay 0.9 — a value from ten steps ago carries 0.9¹⁰ ≈ 35% of its original weight, and from a hundred steps ago effectively none. Choosing the decay rate is choosing how long the memory is, and it is why Adam's default of 0.999 for the second moment corresponds to a window of roughly a thousand steps.
Growth that outruns everything
Linear, quadratic and exponential growth on the same table, plus the decay form used for learning-rate schedules.
example_01.pyNumPy
import numpy as np
print("%5s %10s %12s %18s" % ("n", "10n", "n^2", "2^n"))
for n in (1, 5, 10, 20, 30, 40, 60):
print("%5d %10d %12d %18d" % (n, 10 * n, n ** 2, 2 ** n))
print()
print("at n=5 the linear term is still the biggest of the three.")
print("by n=20 the exponential is %d times the quadratic." % (2 ** 20 // 400))
print("by n=60 nothing else is on the chart.")
print()
print("e is the base where the growth rate equals the value:")
h = 1e-7
for x in (0.0, 1.0, 2.0):
slope = (np.exp(x + h) - np.exp(x)) / h
print(" at x=%.0f e^x = %10.6f d/dx e^x = %10.6f" % (x, np.exp(x), slope))
print()
print("decay is the same shape with a negative rate:")
lr0, half_life = 0.1, 10.0
for epoch in (0, 10, 20, 30, 50):
lr = lr0 * np.exp(-np.log(2) * epoch / half_life)
print(" epoch %2d lr = %.6f" % (epoch, lr))
print()
print("compounding: 5%% a year for 40 years multiplies by %.2f" % (1.05 ** 40))
Output
Try it yourself
Base 2, slide x from 0 to 10. The values are 1, 2, 4, 8… 1024. The first few steps look tame; the last is anything but.
Watch the crossover with x³. Note how far right it happens — exponential growth hides for a surprisingly long time before it dominates.
Set the base to 0.5. Now it is halving: 1, 0.5, 0.25… approaching zero but never reaching it. This is exactly a learning-rate decay schedule.
Tick log-scale y. The exponential becomes a perfectly straight line — that is the signature of exponential data, and how you spot it in a real chart.
Try base 1.05 (5% growth). It looks nearly flat, yet the doubling-time panel says about 14 steps — compound interest in one number.
Where that leaves you
Exponentials multiply rather than add, so they have a constant doubling time and eventually beat any polynomial. Bases above 1 explode, bases below 1 decay toward zero, and base e is the one whose slope equals its value — which is why it underlies softmax, sigmoid and every decay schedule you will configure.
Overflow, and the softmax trick
Exponentials grow fast enough to break floating-point arithmetic. e^1000 is infinity as far as a 64-bit float is concerned, and once an infinity enters a calculation the result is NaN.
This is a real problem in softmax, where the inputs are unbounded logits. The standard fix is to subtract the maximum before exponentiating:
import numpy as np
def softmax(z):
z = z - np.max(z) # shift: does not change the result
e = np.exp(z)
return e / e.sum()
softmax(np.array([1000., 1001., 1002.])) # works
np.exp(np.array([1000., 1001., 1002.])) # inf, inf, inf
Subtracting a constant from every logit leaves the softmax unchanged, because the constant cancels between numerator and denominator — but it moves the largest exponent to e^0 = 1, safely inside range. Every framework does this internally, which is one reason you should pass logits to a loss function rather than probabilities you computed yourself.
The mirror problem is underflow: e^-1000 becomes exactly 0, and a subsequent log gives −infinity. Working in log space throughout, with logsumexp, avoids both.
Reading exponential growth honestly
Three habits stop exponential quantities from misleading you.
Use a log scale. On a logarithmic y-axis, exponential growth is a straight line, and a change in the growth rate becomes visible as a change in slope. On a linear axis everything before the last few periods looks flat and identical.
Quote the doubling time. "Growing 7% a month" is abstract; "doubling every ten months" is not. The rule of 72 gives it quickly: divide 72 by the percentage growth rate to get the doubling time.
Remember nothing grows exponentially for long. Real processes hit a limit and flatten into an S-curve — a logistic function, which is the sigmoid again. Extrapolating an exponential past that bend is how forecasts become absurd.
Questions people ask
Why is e used instead of 2 or 10? Because eˣ is its own derivative, which makes every calculus expression cleaner. Other bases work and pick up a constant factor.
What is the difference between exponential and polynomial growth? A polynomial like x² grows by a shrinking proportion; an exponential grows by a constant proportion, and eventually overtakes any polynomial.
Why does softmax use exponentials? They are positive for any input, which is needed for probabilities, and they amplify differences between scores in a smooth, differentiable way.
What causes NaN in my training? Very often an overflow in an exponential, or a log(0). Numerically stable implementations of softmax and cross-entropy exist precisely for this.
What is an exponential moving average? A running average weighting recent values more heavily, with older values fading geometrically. It is what momentum and Adam are built on.
How do I fit an exponential relationship? Take the log of the target and fit a straight line — log(y) = log(a) + x log(b) is linear in x.
Recap in one screen
Exponential growth multiplies by a constant factor each step, so the increments grow too.
e is the base whose function is its own derivative, which is why calculus prefers it.
Sigmoid, softmax, learning-rate decay, momentum and RBF kernels are all built on exponentials.
Exponentials overflow easily — subtract the maximum before exponentiating, or work in log space.
Plot on a log scale and quote a doubling time; nothing grows exponentially for ever.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “Before the details”?
In y = b^x the variable sits in the exponent . Each step of 1 in x multiplies y by b, rather than adding to it. That single difference is what separates exponential from linear behaviour.
What does this module say about “Exponential Always Wins Eventually”?
Keep the polynomial comparison on and look at the left of the plot: x³ is far ahead. Now slide x rightward. There is a crossover point, and after it 2^x leaves x³ hopelessly behind.
What does this module say about “Doubling Time and Half-Life”?
A defining feature of exponentials: the time to double is constant . It takes just as long to go from 1 to 2 as from a million to two million. The panel computes this live — at base 2 it is exactly 1 step, at base 1.1 about 7.3 steps.
Cheat sheet
Exponentials
Repeated multiplication instead of repeated addition. Exponential growth looks harmless for a long time and then becomes overwhelming — watch it outrun a polynomial that starts far ahead of it.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.