The game
Generative adversarial networks learn to produce data without ever being shown what a correct output looks like. Instead, a second network judges whether each sample is real or fake, and the generator learns by trying to fool that judge. The result is a training loop where both networks improve together — or, if things go wrong, neither does. This page works through the math behind that loop, the fixed point it aims for, and the failure modes that make GANs famously hard to train.
A generator maps noise to samples. A critic receives a sample and guesses the chance that it came from the real data rather than the generator. They are trained against each other:
min over G max over D E_data[log D(x)] + E_z[log(1 - D(G(z)))]
The critic wants that formula large. The gen wants it small. There's no target output for the gen anywhere. No one says what a good sample looks like — only a second network's view, which raises as the gen does. That is what makes the approach notable, and it's also the source of each trouble in the rest of this page.
Here is that makes this page exact rather than a demo. Hold the gen fixed. The goal is a sum over x of
p_data(x) log D(x) + p_g(x) log(1 - D(x))
and each x is free of the others. So You can raise pointwise. Differentiating a log d + b log(1 - d) and setting it to zero gives d = a / (a + b), so:
D*(x) = p_data(x) / (p_data(x) + p_g(x))
The blue curve in the explorer is that formula. It needs no training, no seed and no learning rate. Where only real data lives it's 1. Where only made samples live it's 0. Where the two densities are equal it's just 0.5.
Substituting it turns the game into a divergence
Put D* back into the goal and the algebra merges to:
V(D*, G) = -log 4 + 2 * JSD(p_data || p_g)
The Jensen-Shannon divergence is even, non-minus, and zero only when the two spreads are the same. So the gen is cutting a genuine distance between spreads, and the game has a unique global optimum at p_g = p_data, where the value is -log 4 = -1.386 and the critic is reduced to answering 0.5 all over. Not because it's bad, but because there's nothing left to set apart.
Both numbers are on screen. Set the target to one mode and drag the gen's mean and spread until the divergence reaches 0. The value function arrives at −1.386 at the same moment. That is the theorem, and it's being checked by number as you move the slider.
Where the gradient goes
Now set the gen's mean far from the data so the spreads barely share, and read the two grad figures.
With almost no share the best critic is correct about all. Under the gen, D* ~ 0, and log(1 - D*) is flat there — it's already almost log 1 = 0 and moving the gen slightly doesn't change it. The grad vanishes just when the gen is worst.
This is the trap. The better the critic, the more just right it separates the two, and the less the gen can learn. It isn't an optimisation failure. It's a property of the loss.
The first paper's own fix is in the same section that derives the problem. Instead of cutting log(1 - D(G(z))), **raise log D(G(z))** — the non-saturating form. Same fixed point, fully different grad magnitude in just the regime where it matters. Switch the loss control in the explorer with the spreads separated and compare the two numbers. The ratio is the whole reason each setup uses the second form.
Later work went further: Wasserstein GAN replaced the divergence fully, because the JSD between two spreads on disjoint supports is a constant (log 2) and so has no grad at all, while the earth-mover distance still knows which direction is closer.
Mode collapse, exactly
Set the target spread to two modes and leave the gen narrow.
The gen here is a single Gaussian. So It truly can't cover two separated modes. Sitting on one of them scores far better than straddling the gap. The explorer says so, and there's a reason for copying the failure in a model too simple to avoid it: it isolates the part of mode collapse that has nothing to do with room.
A real gen has ample room to cover both modes and merges anyway. The goal is the reason. The critic judges one sample at a time. It can say "this looks real" or "this looks fake". It has no way to say "these are all the same". So a gen that finds one output the critic accepts and makes it forever is scoring just right by the stated goal. Nothing in the equation at the top of this page mentions range.
The fixes all amount to giving the critic batch-level info: minibatch discrimination gives it statistics across the batch, unrolled GANs let the gen see the critic's future response, and WGAN-GP changes the distance so the grad keeps pointing at the uncovered mode.
The architecture
The theory is spread-shaped. The practice is a pair of convolutional networks. The DCGAN table in the explorer builds both at 64×64 and counts their params as you change the base width.
Read the gen top to bottom: a 100-dimensional noise vector is projected and reshaped to 4×4×1024, then four transposed convolutions double the detail each time to 64×64×3. The critic is the same thing reversed, strided convolutions halving the detail down to a single logit.
The near-symmetry is deliberate and it's the hands-on heart of GAN training. If the critic is much stronger it wins right away and the gen's grad disappears. Much weaker and its judgement is noise. DCGAN's other rules are all steadiness patches on the same problem: strided convolutions rather than pooling, batch norm in both networks, no fully-connected hidden layers, ReLU in the gen and LeakyReLU in the critic.
import torch.nn as nn
class Generator(nn.Module):
def __init__(self, z=100, base=64, out_ch=3):
super().__init__()
def up(cin, cout, first=False, last=False):
layers = [nn.ConvTranspose2d(cin, cout, 4,
stride=1 if first else 2,
padding=0 if first else 1, bias=False)]
if last:
return nn.Sequential(*layers, nn.Tanh())
return nn.Sequential(*layers, nn.BatchNorm2d(cout), nn.ReLU(True))
self.net = nn.Sequential(
up(z, base * 8, first=True), # 1x1 -> 4x4
up(base * 8, base * 4), # 4x4 -> 8x8
up(base * 4, base * 2), # 8x8 -> 16x16
up(base * 2, base), # 16x16 -> 32x32
up(base, out_ch, last=True)) # 32x32 -> 64x64
def forward(self, z):
return self.net(z.view(z.size(0), -1, 1, 1))
Tanh on the output isn't decoration: it bounds the gen's range to [−1, 1], and the real images must be normed to the same range or the critic can separate real from fake on scale alone and learns nothing about content. It's the most common first bug.
Reading the training curves, and why they tell you nothing
One consequence of the theory above is worth stating on its own, because it catches everyone who trains a GAN for the first time.
The loss values don't show progress. In plain guided training a falling loss means the model is raising. Here the two losses are measured against each other, and both networks are moving. A critic loss near log 2 means it can't tell real from fake. Which is either the balance you wanted or a critic that has merged. A gen loss that falls steadily often means the critic is losing, not that the samples are good.
The value function in the explorer is the exception, and only because it's found against the *best* critic rather than a network being trained alongside. In a real run you don't have D*, you've what your critic currently is, and the number it makes isn't close between steps.
So GAN progress is measured by looking at samples, or by a sample-based metric. FID — the Frechet Inception Distance — passes made and real images through an Inception network, fits a Gaussian to each set of activations, and reports the distance between them. It isn't a loss, it can't be optimised right, and it's the number papers really report. It also has the property the training loss lacks: it punishes a gen that makes great samples with no range, because a merged set of activations has the wrong covariance.
Where GANs stand now
Diffusion models displaced GANs as the default for image generation, and the reason is on this page. A diffusion model has a stable regression goal — predict the noise that was added. With no second network to balance against, no balance to reach and no mode collapse to detect. Trading rival training for a simple loss and more sampling steps turned out to be the right trade at scale.
GANs remain the right tool where a single forward pass is needed: real-time super-detail, image-to-image translation, on-device generation. And the rival idea itself long outgrew image synthesis — domain-rival training, rival strength and the critic in a visual loss are all this page's structure, applied to a thing else.
Questions people ask
My critic loss went to zero. Is that good? It's the failure mode, not success. With the supports barely shared the best critic is right about all, log(1 - D) is flat there. The gen's grad vanishes just when it's worst. Use the non-saturating form. If it persists, weaken the critic. Fewer updates, a lower learning rate. Or change the goal to WGAN-GP, whose grad doesn't die on disjoint supports.
What should the losses look like when training is healthy? There's no target curve. Both losses are measured against a moving opponent. A critic loss near log 2 = 0.693 means it can't separate real from fake. This Is either the balance you wanted or a critic that has given up. The two look the same on the plot. Judge by samples and by FID.
How do I tell mode collapse from merely bad samples? By range rather than quality. Make a few hundred samples from different noise and look for repeats. FID catches it because a merged set of activations has the wrong covariance. Here Any per-sample quality score won't. The critic only ever judges one sample at a time. Nothing in the goal mentions range at all.
Why Tanh on the gen, and what breaks without it? It bounds the output to [−1, 1], and the real images must be normed to that same range. Or else the critic can separate real from fake on scale alone and never has to learn any of it about content. It's the most common first bug and it presents as "training looks fine, samples are noise".
Should I train the critic more than the gen? The first method allows k critic steps per gen step and used k = 1. A stronger critic gives a better grad *direction* and a vanished one under the saturating loss. The two networks are often kept close in room. With WGAN-GP extra critic steps truly help. This Is a consequence of the different distance rather than a general rule.
Are GANs still worth learning now that diffusion has won? For two reasons. A GAN makes in a single forward pass. This Still decides the matter for real-time super-detail, image-to-image translation and on-device generation. And the rival structure long outgrew image synthesis. Domain-rival training, rival strength and the critic inside a visual loss are all this page applied elsewhere.
Recap in one screen
- A min-max game with no target output for the gen anywhere. Only a second network's view. This Raises as the gen does.
- Hold the gen fixed and the best critic has a closed form,
D*(x) = p_data / (p_data + p_g). This Needs no training at all. - Substituting it turns the game into
2 . JSD - log 4. There's a unique optimum at p_g = p_data where the value is −1.386 and D = 0.5 all over. - Where the spreads barely share the saturating loss has almost no grad. This Is just why each setup raises
log D(G(z)) instead. - Mode collapse is the goal, not the room. A gen that finds one accepted output and repeats it forever is scoring just right by the stated goal.
- The loss values don't measure progress. Samples do. FID does. It punishes a gen with no range.