Destroy a signal with noise in small steps, then learn to undo one step. Run the undoing from pure noise and you have a sample.
Overview
Forwards is free
Take data and add a little Gaussian noise. Repeat. After enough steps nothing of the first remains and you've pure noise.
This forward process has no params and nothing to learn. Better still, it has a closed form: you can jump straight to any step without modelling the ones before it.
abar_t is the fraction of the first signal still present, and it falls along a fixed schedule. The second panel plots the cosine schedule used here, and the readout gives the signal and noise proportions at what step you select.
That closed form is why training is cheap: pick a random t, jump there right, and train on that one step. No modelling.
Diffusion Models
This module needs JavaScript: the
numbers are computed in the page rather than recorded.
Worth knowing
The forward process is fixed and has no parameters: xt = √ā x0 + √(1−ā) ε.
You can jump to any step in one shot, which is why training samples a random t rather than stepping through.
The model predicts the noise, not the image. Given the noise, the original is recoverable by rearranging one equation.
Sampling is the expensive part: the reverse process is many small steps, where a GAN needs one.
Diffusion Models
Learning to reverse a process that is trivial to run forwards. Why that turned out to work so much better than the options.
Backwards is the model
The generative process runs the other way: start from pure noise and undo one step at a time until an image appears.
Each reverse step needs to know what noise was added, and that is the entire learned component. The network takes x_t and t and predicts epsilon.
Given the noise, getting back the first is rearranging the equation above:
The demo does just this with the *true* noise — the target the model is trained on — and the readout gives the getting back error, which is zero at each step. The forward process is just invertible if you know the noise. Predicting it's the whole job, and it's why the loss is a plain squared error on a noise vector rather than any of it rival.
Why this beats what came before
A stable goal. Predicting noise is plain guided regression. No minimax game, no critic to balance, no mode collapse. Training a diffusion model is boring in a way GAN training never was, and that is the point.
Coverage. GANs can skip parts of the spread. A diffusion model is trained to denoise each example. So It can't quietly drop a mode.
Sharpness. Unlike a VAE, there's no pixel-blending term rewarding a blurred hedge.
The cost is sampling speed. Generation is many sequential passes — at first a thousand, now often ten to fifty with a better sampler — where a GAN needs one. DDIM and distillation methods have shortened this a lot, and it remains the main disadvantage.
The schedule matters
Drag t from 0 to 60 and watch how quickly the signal disappears. The first DDPM used a linear schedule. This Destroys the signal too fast at the end — the last steps are all noise and contribute nothing to learning. The cosine schedule shown here keeps more signal for longer and trains better.
Schedule design is a real part of the field. So is the choice of what the model predicts: the noise, the first, or a mixture called v, all of which are algebraically equal and behave differently .
In practice
Latent diffusion runs the whole process in a VAE's latent space rather than in pixels, cutting the cost enormously. Stable Diffusion is this.
Classifier-free guidance trains the model both with and without a text rule, then extrapolates away from the unconditioned guess at sampling time. It's what makes prompts really steer the output, and turning it up trades range for prompt adherence.
The closed form that makes training cheap
The reason diffusion training is cheap is one equation: the forward process has a closed form, so you can jump to any noise level without modelling the steps before it.
abar_t is the fraction of the first signal still present, falling along a fixed schedule. On the cosine schedule over 60 steps:
step t
signal kept, sqrt(abar)
noise, sqrt(1-abar)
0
1.00
0.00
15
0.92
0.39
30
0.70
0.71
45
0.38
0.93
60
0.00
1.00
Training picks a random t, jumps straight there with one draw of epsilon, and asks the network to predict that epsilon — a plain squared-error regression, no modelling, no adversary. And the forward step is just invertible if you know the noise: rearranging the equation recovers x_0 from x_t and epsilon with zero error. So Predicting the noise is the whole job. The cosine schedule above is chosen over the first linear one just because linear destroys the signal too fast. Its final steps are nearly pure noise and teach almost nothing, while the cosine keeps signal alive longer and trains better.
All that made diffusion win over GANs follows from that squared-error goal. Predicting noise is plain guided regression, so there's no minimax game to balance and no mode collapse. The model is trained to denoise *each* example, so it can't quietly drop part of the spread the way a GAN can. The price is sampling speed: generation is many sequential denoise steps, at first a thousand and now often ten to fifty with a better sampler, where a GAN needs one forward pass. Latent diffusion cuts the cost by running the whole process in a VAE's latent space rather than in pixels. This Is what Stable Diffusion is, and classifier-free guidance is what lets a text prompt really steer the result.
One last point of confusion worth heading off: most diffusion models predict the *noise*, not the image, and some predict the first or a mixture called v. The three are algebraically equal but behave differently in training, and given the network outputs the picture rather than the noise makes the sampling code nonsense.
Where it goes wrong with Diffusion Models
Expecting fast sampling. It's the known weakness.
A schedule that destroys the signal too early. Wasted steps.
Guidance turned up too high. Saturated, oversimplified images and merged range.
given the model predicts the image. Most predict the noise, and confusing the two makes the sampling code nonsense.
Questions people ask about Diffusion Models
How does a diffusion model generate images? It starts from pure noise and removes a little noise at each step with a trained network, until a clean sample remains.
What does the network actually predict? Usually the noise that was added to a noisy input, from which the clean image or the score can be derived.
Why is training cheap even though sampling takes many steps? The forward noising has a closed form, so you can jump to any step directly. Each training step needs only one random timestep.
Why is sampling slow, and how is it sped up? It needs hundreds of sequential network calls. DDIM, better samplers and distillation cut that to a few dozen steps or fewer.
How are diffusion models compared with GANs? They train more stably and cover the data distribution better, at the cost of slower generation. GANs are fast but prone to mode collapse.
What is the noise schedule? The list of how much noise is added at each step. Poor schedules leave signal in the last step or waste steps, so it affects quality.
Destroy the data, then learn to undo it one step at a time
Diffusion adds noise on a schedule until nothing is left, then trains a model to drop a little of it. Both directions are run here on 1-D data, including the closed form that makes training cheap.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
T = 60
betas = np.linspace(1e-4, 0.15, T)
alphas = 1 - betas
abar = np.cumprod(alphas)
print("the FORWARD process adds gaussian noise on a fixed schedule.")
print("it has no parameters and nothing is learned here:")
print("%8s %12s %14s %16s %14s"
% ("step t", "beta_t", "alpha_bar_t", "signal kept", "noise added"))
for t in (0, 10, 25, 40, 59):
print("%8d %12.5f %14.5f %16.4f %14.4f"
% (t, betas[t], abar[t], np.sqrt(abar[t]), np.sqrt(1 - abar[t])))
print()
print("the closed form is the reason this is trainable. instead of applying")
print("%d small noise steps one at a time, any t is one line:" % T)
print(" x_t = sqrt(alpha_bar_t) * x_0 + sqrt(1 - alpha_bar_t) * noise")
Output
The next block shows the data dissolving step by step under that schedule.
example_02.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
T = 60
abar = np.cumprod(1 - np.linspace(1e-4, 0.15, T))
def real_sample(n):
which = rng.random(n) < 0.5
return np.where(which, rng.normal(-2.0, 0.25, n), rng.normal(2.0, 0.25, n))
x0 = real_sample(4000)
print("watch the data dissolve. the last column counts how much mass sits in")
print("the valley between the two bumps -- the real data has almost none,")
print("and a standard normal has about 68 percent:")
print("%8s %14s %10s %10s %16s"
% ("t", "signal kept", "mean", "sd", "mass in |x|<1"))
for t in (0, 10, 25, 40, 59):
xt = np.sqrt(abar[t]) * x0 + np.sqrt(1 - abar[t]) * rng.normal(size=4000)
print("%8d %14.4f %10.3f %10.3f %15.0f%%"
% (t, np.sqrt(abar[t]), xt.mean(), xt.std(),
100 * (np.abs(xt) < 1).mean()))
print("%8s %14.4f %10.3f %10.3f %15.0f%%"
% ("N(0,1)", 0.0, 0.0, 1.0,
100 * (np.abs(rng.normal(size=20000)) < 1).mean()))
print()
print("by t=%d only %.1f%% of the original signal survives, and the last two"
% (T - 1, 100 * np.sqrt(abar[T - 1])))
print("rows are the same distribution. that is the point of the forward")
print("process: it ends somewhere you can SAMPLE FROM without any model.")
Output
The next block trains a small network to predict the noise that was added at each step.
example_03.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
T = 60
abar = np.cumprod(1 - np.linspace(1e-4, 0.15, T))
def real_sample(n):
w = rng.random(n) < 0.5
return np.where(w, rng.normal(-2.0, 0.25, n), rng.normal(2.0, 0.25, n))
print("the model's job is to look at x_t and t, and predict the noise that")
print("was added. train it -- a small network over (x_t, t):")
H = 32; r = np.random.default_rng(1)
W1 = r.normal(0, 0.8, (2, H)); b1 = np.zeros(H)
W2 = r.normal(0, 0.8, (H, H)); b2 = np.zeros(H)
W3 = r.normal(0, 0.3, (H, 1)); b3 = np.zeros(1)
P = [W1, b1, W2, b2, W3, b3]
m = [np.zeros_like(p) for p in P]; v = [np.zeros_like(p) for p in P]
def predict_noise(xt, t):
inp = np.column_stack([xt, t / T])
h1 = np.tanh(inp @ P[0] + P[1])
h2 = np.tanh(h1 @ P[2] + P[3])
return (h2 @ P[4] + P[5]).ravel(), inp, h1, h2
BATCH = 256
for step in range(1, 2501):
x0b = real_sample(BATCH); tb = rng.integers(0, T, BATCH)
eps = rng.normal(size=BATCH)
xt = np.sqrt(abar[tb]) * x0b + np.sqrt(1 - abar[tb]) * eps
pred, inp, h1, h2 = predict_noise(xt, tb)
d = (2 * (pred - eps) / BATCH).reshape(-1, 1)
g4, g5 = h2.T @ d, d.sum(0)
d2 = (d @ P[4].T) * (1 - h2 ** 2); g2, g3 = h1.T @ d2, d2.sum(0)
d1 = (d2 @ P[2].T) * (1 - h1 ** 2); g0, g1 = inp.T @ d1, d1.sum(0)
for i, g in enumerate((g0, g1, g2, g3, g4, g5)):
m[i] = 0.9 * m[i] + 0.1 * g; v[i] = 0.999 * v[i] + 0.001 * g ** 2
P[i] -= 0.01 * (m[i]/(1-0.9**step)) / (np.sqrt(v[i]/(1-0.999**step)) + 1e-8)
tb = rng.integers(0, T, 4000); eps = rng.normal(size=4000)
xt = np.sqrt(abar[tb]) * real_sample(4000) + np.sqrt(1 - abar[tb]) * eps
mse = ((predict_noise(xt, tb)[0] - eps) ** 2).mean()
print(" noise-prediction MSE: %.4f (zero baseline: %.4f)" % (mse, (eps**2).mean()))
Output
The next block uses the trained model to generate samples from pure noise, then compares them against the real data.
example_04.pyNumPy
import numpy as np
rng = np.random.default_rng(0); T = 60
betas = np.linspace(1e-4, 0.15, T); alphas = 1 - betas; abar = np.cumprod(alphas)
def rs(n): return np.where(rng.random(n)<.5, rng.normal(-2,.25,n), rng.normal(2,.25,n))
H = 32; r = np.random.default_rng(1)
P = [r.normal(0,.8,(2,H)), np.zeros(H), r.normal(0,.8,(H,H)), np.zeros(H), r.normal(0,.3,(H,1)), np.zeros(1)]
m = [np.zeros_like(p) for p in P]; v = [np.zeros_like(p) for p in P]
def pn(xt, t):
inp = np.column_stack([xt, t/T]); h1 = np.tanh(inp@P[0]+P[1]); h2 = np.tanh(h1@P[2]+P[3])
return (h2@P[4]+P[5]).ravel(), inp, h1, h2
for s in range(1, 2501):
x0 = rs(256); tb = rng.integers(0,T,256); eps = rng.normal(size=256)
xt = np.sqrt(abar[tb])*x0 + np.sqrt(1-abar[tb])*eps; p,inp,h1,h2 = pn(xt,tb)
d = (2*(p-eps)/256).reshape(-1,1); g4,g5 = h2.T@d, d.sum(0)
d2 = (d@P[4].T)*(1-h2**2); g2,g3 = h1.T@d2, d2.sum(0)
d1 = (d2@P[2].T)*(1-h1**2); g0,g1 = inp.T@d1, d1.sum(0)
for i,g in enumerate((g0,g1,g2,g3,g4,g5)):
m[i] = .9*m[i]+.1*g; v[i] = .999*v[i]+.001*g**2
P[i] -= .01*(m[i]/(1-.9**s))/(np.sqrt(v[i]/(1-.999**s))+1e-8)
print("SAMPLE from pure noise, walking the schedule backwards:")
x = rng.normal(size=3000)
for t in range(T-1, -1, -1):
eh = pn(x, np.full(len(x), t))[0]
mn = (x - betas[t]/np.sqrt(1-abar[t])*eh) / np.sqrt(alphas[t])
x = mn + (np.sqrt(betas[t])*rng.normal(size=len(x)) if t > 0 else 0.)
if t in (59, 40, 25, 10, 0):
print(" t=%2d: mean %+6.3f, sd %5.3f, %.0f%% below zero" % (t, x.mean(), x.std(), 100*(x<0).mean()))
print()
real = rs(4000); edges = np.linspace(-4, 4, 17)
hr_, _ = np.histogram(real, bins=edges); hf_, _ = np.histogram(x, bins=edges)
print("generated vs real, as a histogram:")
for i in range(len(edges)-1):
print(" %5.1f real %-32s fake %s" % (edges[i], "#"*int(30*hr_[i]/hr_.max()), "#"*int(30*hf_[i]/max(hf_.max(),1))))
print()
print(" real: mean %+.3f sd %.3f generated: mean %+.3f sd %.3f" % (real.mean(), real.std(), x.mean(), x.std()))
print()
print("the target is known at every step -- plain supervised regression, no")
print("adversary, no equilibrium. the cost is sampling: a GAN needs one")
print("forward pass; this took %d. that is why DDIM and distillation exist." % T)
Output
Check yourself
0 of 3
Answer without scrolling back up.
What does the network in a diffusion model predict?
Given the noise, recovering the original is just rearranging the forward equation. That makes the loss a plain squared error rather than anything adversarial.
Why can training sample a random step t rather than simulating up to it?
x_t = sqrt(abar)x0 + sqrt(1-abar)eps needs no simulation, which is what makes training cheap.
What is the main disadvantage against a GAN?
The stable objective and full coverage are diffusion's advantages. DDIM and distillation have shortened sampling considerably, but it remains the trade.
Cheat sheet
Diffusion Models
Add noise to an image step by step, train a network to reverse each step, then sample a new image by running the learned reverse chain.
DEEP LEARNING · vizlearn.in/deep_learning/diffusion_models.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.