Two networks with opposite goals. Move the generator's distribution and watch the discriminator's gradient appear and vanish.
Overview
The setup
Two networks, one loss.
The gen maps random noise to samples, trying to make them the same from real data. The critic takes a sample and guesses whether it's real, trying to be right.
They optimise the same goal in opposite directions. That makes it a minimax game, and it's why GAN training is unlike any of it else: there's no loss that falls, and a falling gen loss might mean the gen is raising or the critic is losing.
Generative Adversarial Networks
This module needs JavaScript: the
numbers are computed in the page rather than recorded.
Worth knowing
The generator turns noise into samples. The discriminator tries to tell them from real ones.
They share one loss with opposite signs, so it is a game rather than an optimisation, and it has no loss curve that goes down.
The optimal discriminator is preal / (preal + pfake). Where the distributions do not overlap, it is flat.
Flat means no gradient. That is the vanishing-gradient failure the original GAN loss had, and why Wasserstein GANs exist.
Generative Adversarial Networks
A gen and a critic pushed against each other. The exact way that setup fails.
The second panel is not learned
For fixed real and made spreads, the best critic has a closed form:
D*(x) = p_real(x) / ( p_real(x) + p_fake(x) )
The lower panel plots just that. It isn't a trained network. It's what training drives towards, and its shape is what makes the failure mode legible.
Where the gradient goes
Drag the gen's mean far from the real spread. The share in the readout merges toward zero, and the critic's curve turns flat — near 1 where the real data lives, near 0 where the fakes are, with a steep cliff between.
Flat means no grad. The gen raises by following the slope of the critic's view, and if that view is constant all over the fakes live, there's nothing to follow. The better the critic, the worse this gets, which is a truly awkward property: your gen's learning signal degrades as its opponent raises.
Now bring the mean back. Where the spreads share the curve slopes, and that slope is the gen's grad.
At full share the critic sits at 0.5 all over — it can't tell them apart, which is the balance the whole setup targets.
The failures
Vanishing grads, as above. The non-saturating loss helps. The deeper fix was Wasserstein GAN, which replaces the classifier with a critic guessing earth-mover distance. That distance is helpful even for disjoint spreads, so the grad lives.
Mode collapse. The gen finds a narrow region that fools the critic and stays there. Shrink the gen's spread until it's a spike inside the real spread: it's fooling the critic locally while showing almost none of the data. Nothing in the goal right punishes this, because the critic judges samples one at a time and never sees the lack of range.
Non-settling. The two can cycle indefinitely, each undoing the other. Two-timescale learning rates, spectral norming and grad fines are all responses to it.
No usable progress metric. The loss doesn't say whether samples are raising. FID and Inception Score exist because of this, and both are proxies.
Against the alternatives
Sharp samples. GANs give them where VAEs blur, because nothing rewards blending — a hedged sample is just what the critic catches.
No likelihood. A GAN can't say how probable a given sample is, which rules out a few uses.
Hard to train, as above.
Diffusion models have largely displaced GANs for image generation, trading sampling speed for training steadiness. GANs remain competitive where inference must be a single forward pass, and the rival idea itself lives all over — in domain fitting, in visual losses, in super-detail.
Why a winning discriminator kills training
The vanishing-grad failure isn't vague. It falls straight out of the best critic D*(x) = p_real(x) / (p_real(x) + p_fake(x)). Put the real and fake spreads as unit Gaussians and slide them apart:
separation of the means
D* at the fake's centre
D* at the real's centre
0.5 (heavy overlap)
0.47
0.53
4.0 (barely overlapping)
0.0003
0.9997
When the two share, D* sits near 0.5 all over and its curve *slopes* through the region the fakes occupy. And that slope is the gen's entire learning signal. When they separate, D* saturates to 0.0003 where the fakes live: almost just right correct, and almost just right flat. A flat critic has no slope to follow, so the gen gets no grad just when it's doing worst. This is the awkward heart of the first GAN — the better the critic, the less the gen can learn. And it's why Wasserstein GAN replaced the classifier with a critic guessing earth-mover distance, which stays helpful even for spreads that don't share at all.
This is also why a GAN has no usable progress metric and why mode collapse goes unpunished. There's no single loss that falls — a falling gen loss can mean the gen raised or the critic weakened — so FID and Inception Score exist as external proxies for sample quality. And because the critic judges samples one at a time, a gen that finds one lifelike region and abandons the rest of the spread is never penalised for the lack of range. Nothing in the goal ever sees more than a single sample at once, so the fix has to come from outside it, in minibatch discrimination or in the Wasserstein reformulation.
None of this makes GANs obsolete. They still win where inference must be a single forward pass, and the rival idea itself is all over — in domain fitting, in visual losses, in super-detail — though diffusion models have taken over image generation by trading sampling speed for the training steadiness a minimax game can never quite offer.
The hands-on upshot for anyone training one: watch the samples, not the loss; keep the critic from winning outright, since a perfect critic gives no grad. And check the range of what the gen makes rather than the quality of any single sample, because on their own lifelike outputs are just what mode collapse leaves you with.
Where it goes wrong with Generative Adversarial Networks
Reading the loss curves. They mostly say who is winning, not whether the samples are good. Look at samples.
Letting the critic win. A perfect critic gives no grad.
Skipping mode collapse because the samples look fine on their own. Check the range, not the quality.
Expecting a likelihood. There isn't one.
Questions people ask about Generative Adversarial Networks
How does a GAN work? A generator turns noise into samples while a discriminator tries to tell them from real data. Each improves by competing with the other.
What is mode collapse? The generator produces only a few kinds of output that fool the discriminator, ignoring the rest of the data. Diversity drops even when individual samples look fine.
Why are GANs hard to train? Two networks chase moving targets. If the discriminator wins too early, its gradient to the generator vanishes.
How do I know when a GAN has converged? You cannot rely on the loss, which does not track quality. Use metrics such as FID and inspect samples regularly.
What are common fixes for instability? Use Wasserstein loss with a gradient penalty, spectral normalisation, a lower learning rate for the discriminator, and balanced update schedules.
When should I choose a GAN over a diffusion model? When you need fast single-pass sampling, such as real-time use. For quality and coverage, diffusion models are now more common.
Two networks, one moving target
A GAN trains a gen against a critic that is itself learning. This runs that loop on 1-D data, then shows the two failure modes that make GANs hard.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
def rs(n): return np.where(rng.random(n)<.5, rng.normal(-2,.3,n), rng.normal(2,.3,n))
def sig(z): return 1/(1+np.exp(-np.clip(z,-30,30)))
H = 12
Gw1=rng.normal(0,1,(1,H)); Gb1=np.zeros(H); Gw2=rng.normal(0,.5,(H,1)); Gb2=np.zeros(1)
Dw1=rng.normal(0,1,(1,H)); Db1=np.zeros(H); Dw2=rng.normal(0,.5,(H,1)); Db2=np.zeros(1)
def gen(z):
h=np.tanh(z.reshape(-1,1)@Gw1+Gb1); return h@Gw2+Gb2, h
def disc(x):
h=np.tanh(x.reshape(-1,1)@Dw1+Db1); return (h@Dw2+Db2).ravel(), h
B, LR = 128, 0.02
print("real data: two bumps at -2 and +2. the generator maps noise to numbers")
print("and never sees a single real value -- only the discriminator's score.")
print()
print("%8s %14s %14s %16s %16s" % ("step", "D(real)", "D(fake)", "fake mean", "fake sd"))
for step in range(3001):
xr=rs(B); z=rng.normal(size=B); xf,_=gen(z); lr_,hr=disc(xr); lf_,hf=disc(xf.ravel())
pr,pf=sig(lr_),sig(lf_); dlr=(pr-1)/B; dlf=pf/B
gDw2=hr.T@dlr.reshape(-1,1)+hf.T@dlf.reshape(-1,1); gDb2=dlr.sum()+dlf.sum()
hr_g=(dlr.reshape(-1,1)@Dw2.T)*(1-hr**2); hf_g=(dlf.reshape(-1,1)@Dw2.T)*(1-hf**2)
gDw1=xr.reshape(1,-1)@hr_g+xf.reshape(1,-1)@hf_g; gDb1=hr_g.sum(0)+hf_g.sum(0)
Dw2-=LR*gDw2; Db2-=LR*gDb2; Dw1-=LR*gDw1; Db1-=LR*gDb1
z=rng.normal(size=B); xf,hg=gen(z); lf_,hf=disc(xf.ravel()); pf=sig(lf_)
dl=(pf-1)/B; dx=((dl.reshape(-1,1)@Dw2.T)*(1-hf**2))@Dw1.T
gGw2=hg.T@dx; gGb2=dx.sum(0); hg_g=(dx@Gw2.T)*(1-hg**2)
gGw1=z.reshape(1,-1)@hg_g; gGb1=hg_g.sum(0)
Gw2-=LR*gGw2; Gb2-=LR*gGb2; Gw1-=LR*gGw1; Gb1-=LR*gGb1
if step % 600 == 0:
zs=rng.normal(size=2000); fake=gen(zs)[0].ravel()
print("%8d %14.4f %14.4f %16.4f %16.4f" % (step, sig(disc(rs(2000))[0]).mean(), sig(disc(fake)[0]).mean(), fake.mean(), fake.std()))
Output
The next block generates samples from the trained model and checks for mode collapse.
example_02.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
def rs(n): return np.where(rng.random(n)<.5, rng.normal(-2,.3,n), rng.normal(2,.3,n))
def sig(z): return 1/(1+np.exp(-np.clip(z,-30,30)))
H = 12
Gw1=rng.normal(0,1,(1,H)); Gb1=np.zeros(H); Gw2=rng.normal(0,.5,(H,1)); Gb2=np.zeros(1)
Dw1=rng.normal(0,1,(1,H)); Db1=np.zeros(H); Dw2=rng.normal(0,.5,(H,1)); Db2=np.zeros(1)
def gen(z):
h=np.tanh(z.reshape(-1,1)@Gw1+Gb1); return h@Gw2+Gb2, h
def disc(x):
h=np.tanh(x.reshape(-1,1)@Dw1+Db1); return (h@Dw2+Db2).ravel(), h
B, LR = 128, 0.02
for step in range(3001):
xr=rs(B); z=rng.normal(size=B); xf,_=gen(z); lr_,hr=disc(xr); lf_,hf=disc(xf.ravel())
pr,pf=sig(lr_),sig(lf_); dlr=(pr-1)/B; dlf=pf/B
gDw2=hr.T@dlr.reshape(-1,1)+hf.T@dlf.reshape(-1,1); gDb2=dlr.sum()+dlf.sum()
hr_g=(dlr.reshape(-1,1)@Dw2.T)*(1-hr**2); hf_g=(dlf.reshape(-1,1)@Dw2.T)*(1-hf**2)
gDw1=xr.reshape(1,-1)@hr_g+xf.reshape(1,-1)@hf_g; gDb1=hr_g.sum(0)+hf_g.sum(0)
Dw2-=LR*gDw2; Db2-=LR*gDb2; Dw1-=LR*gDw1; Db1-=LR*gDb1
z=rng.normal(size=B); xf,hg=gen(z); lf_,hf=disc(xf.ravel()); pf=sig(lf_)
dl=(pf-1)/B; dx=((dl.reshape(-1,1)@Dw2.T)*(1-hf**2))@Dw1.T
gGw2=hg.T@dx; gGb2=dx.sum(0); hg_g=(dx@Gw2.T)*(1-hg**2)
gGw1=z.reshape(1,-1)@hg_g; gGb1=hg_g.sum(0)
Gw2-=LR*gGw2; Gb2-=LR*gGb2; Gw1-=LR*gGw1; Gb1-=LR*gGb1
fake = gen(rng.normal(size=4000))[0].ravel(); real = rs(4000)
print("what the generator produced, as a histogram against the real data:")
edges = np.linspace(-4, 4, 17)
hr_, _ = np.histogram(real, bins=edges); hf_, _ = np.histogram(fake, bins=edges)
for i in range(len(edges)-1):
print(" %5.1f to %5.1f real %s" % (edges[i], edges[i+1], "#"*int(40*hr_[i]/hr_.max())))
print(" %19s fake %s" % ("", "#"*int(40*hf_[i]/max(hf_.max(),1))))
print("real: mean %+.3f, sd %.3f, %.0f%% below zero" % (real.mean(), real.std(), 100*(real<0).mean()))
print("fake: mean %+.3f, sd %.3f, %.0f%% below zero" % (fake.mean(), fake.std(), 100*(fake<0).mean()))
left = (fake < 0).mean()
print("MODE COLLAPSE check: %.3f of fakes in the left bump (should be 0.50)." % left)
if min(left, 1-left) < 0.15:
print(" this run collapsed -- it found one bump and stopped.")
else:
print(" this run kept both, but that is not guaranteed -- the loss never")
print(" asks the generator for diversity, only that samples fool D.")
Output
The next block explains why the moving-target structure makes GANs fundamentally hard to train.
example_03.pyNumPy
print("nothing in the loss asks the generator for DIVERSITY. it asks only")
print("that its samples be judged real, and a single perfect sample would")
print("satisfy that completely.")
print()
print("the second failure is the moving target. the discriminator's opinion")
print("changes every step, so the generator is descending a loss surface")
print("that is being rewritten underneath it. that is why:")
print(" - the loss curves tell you almost nothing. a falling generator loss")
print(" can mean the generator improved OR the discriminator got worse.")
print(" - if D wins outright, sigmoid(logit) saturates and the gradient to")
print(" G vanishes. the non-saturating loss used above -- maximise")
print(" log D(fake) rather than minimise log(1 - D(fake)) -- exists")
print(" precisely to keep that gradient alive.")
print(" - balance matters: people alternate 1 D step to 1 G step, clip,")
print(" use spectral normalisation, or switch to a Wasserstein loss to")
print(" keep either side from running away.")
print()
print("this is the honest reason diffusion models displaced GANs for image")
print("generation: they optimise a single stable objective, and this one")
print("does not converge so much as circle.")
Output
Check yourself
0 of 3
Answer without scrolling back up.
Why does a very good discriminator stop the generator learning?
The generator improves by following the slope of the discriminator's opinion. Wasserstein GAN replaces the classifier with a critic whose distance stays informative even for disjoint distributions.
What is mode collapse?
Nothing in the objective punishes it directly, because the discriminator judges samples one at a time and never sees the lack of variety.
Why are GAN samples sharper than a VAE's?
A VAE's pixel-wise reconstruction term rewards hedging: when uncertain, the average of the possibilities scores better than any single one.
Cheat sheet
Generative Adversarial Networks
Train a generator against a discriminator, watch the adversarial loss curves, and see the mode collapse and instability that make GANs hard.
DEEP LEARNING · vizlearn.in/deep_learning/generative_adversarial_networks.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.