The Rise and Fall of GANs — An Invention Trained by Rivalry, and Why Diffusion Won

In 2014, GANs upended image generation with one idea: don't write down what 'realistic' means — learn it. This traces the decade that followed, through mode collapse, the Wasserstein rescue, and the reasons diffusion models eventually took the crown.

Generative Adversarial Networks

The metaphor: raise a forger and an inspector in the same room

Say you want to train an art forger, and you have no textbook. Instead, you sit a trainee inspector next to them.

The forger paints; the inspector guesses real or fake. Both start as amateurs, so at first the inspector wins on nothing more than uneven paint. The forger learns to even out the paint. Now the inspector starts noticing brushwork, so the forger imitates brushwork — and after tens of thousands of rounds of this arms race, you are left with a forger whose paintings the inspector can only call at chance.

That is a GAN (Generative Adversarial Network). The forger is the generator, the inspector is the discriminator. When Goodfellow and colleagues published this setup in 2014, it broke the prevailing assumption about how generative models had to work.

The intuition: nobody has to write down what "realistic" means

What was actually new here? The yardstick itself was learned.

Before this, image generation measured "how close is this to the target" with a formula a human chose — typically per-pixel squared error. That has a deep flaw: when several answers are equally valid, the squared-error minimum is their average. If a face could plausibly have hair swept left or swept right, the move that minimizes squared error is a faint blend of both — a blurry face. This is the main reason early autoencoder outputs looked washed out (a problem covered in VAEs from Scratch).

GANs close that escape route. A discriminator learns almost immediately that blurry means fake, so a generator that hedges toward the average gets caught and punished. There is no averaging left to hide in — which is why GAN samples looked so much sharper than anything else at the time.

The mechanism: one minimax objective

The heart of the paper is a single line.

Symbol by symbol: $p_{\text{data}}$ is the distribution real data comes from, $p_z$ is a noise distribution (usually a Gaussian), $G(z)$ is a fake built from that noise, and $D(x)$ returns the probability that $x$ is real.

Put in words, it says: the inspector wants to score real things high and fakes low; the forger wants its fakes scored high. What matters is that both players are pushing on the same single number, one upward and one downward — hence $\min$ next to $\max$. This is not descent on a loss surface. It is a search for the equilibrium of a two-player game, and every difficulty that follows traces back to that distinction.

The paper then works out how the strongest possible inspector behaves when the generator is held fixed.

Here $p_g$ is the distribution the generator produces. Read in words, the optimal discriminator reports the ratio of how much real mass to how much fake mass sits at that point — exactly 0.5 where the two are equally dense. Substitute this back into the objective and the whole thing becomes the Jensen-Shannon divergence between $p_{\text{data}}$ and $p_g$, plus a constant. (Think of it as the symmetric cousin of the quantity in KL Divergence from Scratch.) Optimize it fully and you land on $p_g = p_{\text{data}}$ — the generator matching reality exactly. As theory, it could hardly be cleaner.

The first stumble: the gradient disappears

The clean theory betrays you within the first thousand steps.

Early in training the generator produces garbage, so the discriminator confidently returns $D(G(z)) \approx 0$. Now look at the generator's term, $\log(1 - D(G(z)))$. Near $D = 0$ that function is nearly flat. The generator is told "everything you made is wrong" but not which direction is less wrong. The moment it most needs guidance is the moment it receives the least.

The original paper flags this and prescribes a fix: instead of minimizing $\log(1 - D(G(z)))$, have the generator maximize $\log D(G(z))$. The optimum sits in the same place, but the early-training slope is orders of magnitude steeper. This rewrite — the non-saturating loss — became the default in essentially every GAN implementation since.

The feeling of "you cannot learn on a flat stretch" clicks fastest if you touch a sigmoid yourself.

Mode collapse — perfect forgeries of exactly one thing

Even with the gradient problem patched, GANs carry a disease of their own: mode collapse.

Remember what the generator is actually asked to do. Its goal is to fool the inspector — not to reproduce the variety in the data. If it happens to draw the digit "1" convincingly, its optimal strategy is to emit a "1" no matter what noise comes in. The objective is satisfied. Nothing in it ever asked for diversity.

This falls out of the objective's structure. It is the mirror image of likelihood-based models, which try to cover everything thinly and go blurry: a GAN tends to cover a slice densely and abandon the rest. Sharpness and coverage were two faces of the same coin.

The nasty part is that mode collapse does not show up in the loss numbers. Generator loss and discriminator loss both look plausible and stable. To catch it you have to generate from a fixed set of noise vectors every epoch and look at the pictures with your own eyes.

WGAN — swap out the ruler

Wasserstein GAN, in 2017, located the disease in the choice of distance.

Jensen-Shannon's weakness is that when two distributions do not overlap, it is a constant — and a constant has no slope. High-dimensional image data lies on a thin manifold rather than filling the space, so early in training the generated and real distributions failing to overlap is not an edge case, it is the normal case. The ruler could not measure "getting closer," which is precisely what training needed.

The replacement is the Wasserstein distance (earth mover's distance). Treat one distribution as a pile of dirt and the other as a hole to fill, and define the distance as the minimum "amount × distance" of dirt you must haul. Pile and hole far apart? The number grows accordingly. You always get a reading of how far apart things are, overlap or not.

Computing that directly is intractable, so WGAN uses the dual form (Kantorovich–Rubinstein).

Here $f$ is a function returning a scalar, and $\lVert f \rVert_L \le 1$ means "1-Lipschitz" — nudge the input and the output may not move by more than that nudge, i.e. the function is not allowed to be too steep. Put in words, the formula says: among all not-too-steep functions that score real data high and fake data low, find the one that opens the widest gap; that gap is the distance.

Three things change in the code. First, the discriminator stops being a probability classifier and becomes a "critic" that returns a raw score — the output sigmoid and the $\log$ both disappear, and the loss is a plain difference. Second, you need machinery to keep the critic from getting steep. The original paper clamped weights into a fixed range (weight clipping), which is crude; the following year WGAN-GP replaced it with a gradient penalty that punishes the input-gradient norm for straying from 1, and spectral normalization (dividing weight matrices by their largest singular value) became widespread after that. Third, it became standard to update the critic more often than the generator — five critic steps per generator step in WGAN.

The payoff was large: the critic's loss now falls in step with image quality. In the original formulation, loss was a value relative to how strong your opponent currently was, and reading it told you nothing.

The golden age — until faces became indistinguishable

Once the foundation held, results arrived quickly. DCGAN (2015) handed out a stable convolutional recipe — strided convolutions instead of pooling, batch normalization, Adam with $\beta_1$ lowered to 0.5 — and became the shared vocabulary of later implementations. Then came Progressive Growing, which adds layers from low resolution upward; StyleGAN / StyleGAN2, which injects "style" partway through the generator; and BigGAN, which pushed large-scale class-conditional generation. The Inception Score and FID metrics were established in the same window, moving the field past arguing about samples by eye.

Why diffusion won

Then, in 2021, came a paper with a blunt title: "Diffusion Models Beat GANs on Image Synthesis." The tide turned less because of raw image quality than because of the character of the training.

First, optimization went back to ordinary regression. Training a diffusion model means minimizing the squared error of a noise prediction (see Diffusion Models from Scratch). There is no opponent, so there is no equilibrium to find and no need to keep two networks balanced against each other. The single largest operational cost of GANs — training falling apart — was removed structurally.

Second, coverage. Diffusion models optimize something close to a likelihood, so they try to cover the whole distribution. The GAN's weakness became the newcomer's strength.

Third, conditioning is straightforward. For text-driven generation, classifier-free guidance reduces "how hard should the condition push" to a single knob. Building the equivalent control into a GAN was always fiddly. On top of that, stable training scales: make the model and dataset bigger and quality follows, whereas unstable methods fail more often the larger you go.

GANs still hold one clear advantage: speed. A GAN emits an image from a single forward pass. Diffusion needs iteration. That gap remains real.

GANs did not vanish — they became a component

So GANs are not dead. They stepped back as standalone generators while remaining fully active as a loss term that measures perceptual quality.

• The autoencoder a latent diffusion model uses to compress and restore images is standardly trained with an adversarial loss on its decoder (the VQGAN lineage). Part of the ground diffusion stands on is made of GAN
• The final stage of speech synthesis — neural vocoders turning mel-spectrograms into waveforms — is dominated in practice by adversarial approaches such as HiFi-GAN
• Super-resolution and image-to-image translation (the pix2pix and CycleGAN lineage) produce crisper output with an adversarial term in the mix
• Work on distilling many-step diffusion models into few-step generators also uses adversarial losses for the finishing touch

The 2014 idea — learn the yardstick for realism instead of writing it down — survived. It just survived as a loss function rather than as an architecture.

How this shows up in practice

Who touches it, and when. Far more often than people building generative backbones, it is engineers improving the quality of super-resolution, vocoders, and image translation who add an adversarial term. A common way in is discovering that the VAE decoder you are fine-tuning has a discriminator_loss sitting in its training script — "I use diffusion, so this doesn't concern me" turns out to be wrong.

Parameters you actually tune. The critic-to-generator update ratio (n_critic, conventionally 5 in the WGAN family); the gradient penalty coefficient $\lambda$ (10 in WGAN-GP); Adam's $\beta_1$ (0.5 for GANs — leaving it at the usual 0.9 invites collapse); TTUR, meaning separate learning rates for generator and discriminator; whether spectral normalization is on; and an EMA (exponential moving average) of the generator weights for evaluation. EMA looks like a detail, but without it you often get "training is progressing yet samples wobble every epoch."

Traps that turn into incidents.

• Never read the loss as progress. A vanilla GAN loss is relative to how strong your opponent currently is, so a falling number does not mean rising quality. Watch FID and eyeball a sample grid built from fixed noise. The WGAN critic loss is the exception — it estimates a distance, so it does track quality
• Mode collapse is invisible in the loss. Pin down about ten fixed $z$ vectors and generate from the same ones every epoch. When the outputs start resembling each other, collapse has begun
• FID numbers are implementation-dependent. Sample count, resize interpolation, and feature-extractor weights all shift the value. Do not compare your number against a paper's; use it for relative comparison inside one implementation
• The learning rate is the single largest cause of blowups. Adversarial training hunts an equilibrium, so it diverges more easily than plain minimization — start an order of magnitude lower than instinct suggests

Getting a feel for what a too-large learning rate does is easiest on the simplest possible minimization. In adversarial training, picture two of these happening at once, each reacting to the other's movement.

Questions you will be asked in interviews and design reviews. "Why maximize $\log D$ rather than minimize $\log(1-D)$?" (the early gradient saturates). "Why Wasserstein distance instead of Jensen-Shannon?" (JS is constant when the distributions do not overlap, so it yields no slope). "How would you detect mode collapse?" (not from the loss — from a fixed-noise sample series and diversity-aware metrics). Each one is a slice of GAN history turned into a question.

Summary

• The GAN's invention was learning the yardstick for realism instead of writing it by hand. With the escape into averaging blocked, its samples were dramatically sharper than the alternatives of the time
• The price was unstable adversarial training. Early-training gradient vanishing was patched by the non-saturating loss; the non-overlap problem was attacked with the Wasserstein distance and Lipschitz constraints (weight clipping → gradient penalty → spectral normalization)
• Mode collapse remained structural, because nothing in the objective ever asks for diversity
• Diffusion took the crown not on image quality alone but because training reverted from equilibrium-hunting to plain regression, and therefore scaled reliably
• GANs remain in service as a loss function — running today inside diffusion autoencoders, vocoders, and super-resolution