The Rise and Fall of GANs — An Invention Trained by Rivalry, and Why Diffusion Won

In 2014, GANs upended image generation with one idea: don't write down what 'realistic' means — learn it. This traces the decade that followed, through mode collapse, the Wasserstein rescue, and the reasons diffusion models eventually took the crown.

Generative Adversarial Networks

The metaphor: raise a forger and an inspector in the same room

Say you want to train an art forger, and you have no textbook. Instead, you sit a trainee inspector next to them.

The forger paints; the inspector guesses real or fake. Both start as amateurs, so at first the inspector wins on nothing more than uneven paint. The forger learns to even out the paint. Now the inspector starts noticing brushwork, so the forger imitates brushwork — and after tens of thousands of rounds of this arms race, you are left with a forger whose paintings the inspector can only call at chance.

That is a GAN (Generative Adversarial Network). The forger is the generator, the inspector is the discriminator. When Goodfellow and colleagues published this setup in 2014, it broke the prevailing assumption about how generative models had to work.

The intuition: nobody has to write down what "realistic" means

What was actually new here? The yardstick itself was learned.

Before this, image generation measured "how close is this to the target" with a formula a human chose — typically per-pixel squared error. That has a deep flaw: when several answers are equally valid, the squared-error minimum is their average. If a face could plausibly have hair swept left or swept right, the move that minimizes squared error is a faint blend of both — a blurry face. This is the main reason early autoencoder outputs looked washed out (a problem covered in VAEs from Scratch).

GANs close that escape route. A discriminator learns almost immediately that blurry means fake, so a generator that hedges toward the average gets caught and punished. There is no averaging left to hide in — which is why GAN samples looked so much sharper than anything else at the time.

The mechanism: one minimax objective

The heart of the paper is a single line.

Symbol by symbol: $p_{\text{data}}$ is the distribution real data comes from, $p_z$ is a noise distribution (usually a Gaussian), $G(z)$ is a fake built from that noise, and $D(x)$ returns the probability that $x$ is real.

Put in words, it says: the inspector wants to score real things high and fakes low; the forger wants its fakes scored high. What matters is that both players are pushing on the same single number, one upward and one downward — hence $\min$ next to $\max$. This is not descent on a loss surface. It is a search for the equilibrium of a two-player game, and every difficulty that follows traces back to that distinction.

The paper then works out how the strongest possible inspector behaves when the generator is held fixed.

Here $p_g$ is the distribution the generator produces. Read in words, the optimal discriminator reports the ratio of how much real mass to how much fake mass sits at that point — exactly 0.5 where the two are equally dense. Substitute this back into the objective and the whole thing becomes the Jensen-Shannon divergence between $p_{\text{data}}$ and $p_g$, plus a constant. (Think of it as the symmetric cousin of the quantity in KL Divergence from Scratch.) Optimize it fully and you land on $p_g = p_{\text{data}}$ — the generator matching reality exactly. As theory, it could hardly be cleaner.

The first stumble: the gradient disappears

The clean theory betrays you within the first thousand steps.

Early in training the generator produces garbage, so the discriminator confidently returns $D(G(z)) \approx 0$. Now look at the generator's term, $\log(1 - D(G(z)))$. Near $D = 0$ that function is nearly flat. The generator is told "everything you made is wrong" but not which direction is less wrong. The moment it most needs guidance is the moment it receives the least.

The original paper flags this and prescribes a fix: instead of minimizing $\log(1 - D(G(z)))$, have the generator maximize $\log D(G(z))$. The optimum sits in the same place, but the early-training slope is orders of magnitude steeper. This rewrite — the non-saturating loss — became the default in essentially every GAN implementation since.

The feeling of "you cannot learn on a flat stretch" clicks fastest if you touch a sigmoid yourself.