Paper Explained — GenFirst: Let Generation Shape the Latent Space First, Reconstruction Second

Image generation normally means training a VAE first and bolting a generative model onto its frozen latent space. This paper trains both at once without latent collapse — the key is the entropy term inside the KL objective, plus a simple rule: let generation go first and ramp reconstruction up later.

GenFirst: Generation Before Reconstruction for Stable End-to-End Latent Generative Modeling

What the paper claims

The original title is "GenFirst: Generation Before Reconstruction for Stable End-to-End Latent Generative Modeling" (arXiv:2608.29335, Zheng et al., 29 August 2026).

Here is the abstract in plain terms. Latent generative models normally follow a two-stage pipeline: train a variational autoencoder (VAE) for reconstruction, then train a generative model on top of the frozen latent space. But a latent space optimized for reconstruction is not necessarily a latent space that is friendly to generation, which makes joint training an appealing alternative. Training the two end-to-end directly turns out to be hard: it is prone to latent collapse, and it runs into a generation–reconstruction conflict. The authors analyze how the different objectives shape the latent space and report two findings. First, the entropy term inside the KL divergence objective is essential for preventing collapse — reconstruction and prior fitting both tend to shrink the posterior, while entropy keeps latent uncertainty from degenerating. Second, reconstruction and generation have asymmetric learning dynamics: reconstruction is fast and strongly supervised, generation is slower and harder to optimize. On that basis they achieve the first direct end-to-end training without latent collapse, and propose GenFirst, a simple generation-before-reconstruction strategy.

An analogy: who gets to shape the clay

Picture a sculpture workshop with two craftspeople sharing one lump of clay.

The first is the copyist (reconstruction). Their job is to produce a piece identical to the model in front of them. Because the correct answer is sitting right there, their feedback is fast and unambiguous. The second is the originator (generation). Their job is to learn to make new pieces in the workshop's style, with no model to copy. There is no ground truth, and progress is slow.

Now let both work the same clay at the same time. The copyist is far faster, so the clay sets early into a shape that happens to be convenient for copying. By the time the originator thinks "I wish this had been shaped differently," it is too late. GenFirst's prescription is exactly what you would expect: let the originator settle the overall form first, and let the copyist carve in the details afterwards.

Background: why generate in a latent space at all

Some groundwork first. A 512×512 color image is a few hundred thousand numbers. Running a diffusion model directly on that is painful, so modern image generation splits the work in two:

1. An encoder compresses the image into a small latent vector $z$, and a decoder turns it back into an image (the reconstruction stage).
2. On top of that small $z$-space, a generative model — diffusion or autoregressive — learns the distribution of plausible $z$.

Stage one is typically a VAE. What makes a VAE a VAE is that its encoder outputs a distribution $q(z|x)$ rather than a single point; that machinery is covered in Building a VAE from scratch. Stage two is most often a diffusion model, covered in Introduction to diffusion models.

The problem is that stage one is trained knowing nothing about stage two. It shapes the latent space purely by asking "can I get the original picture back?" The result may be a space stuffed with fine texture detail that is tangled and hard to learn from the generative model's point of view. That is exactly the paper's starting point: latents optimized for reconstruction are not necessarily generation-friendly (Abstract).

Why joint training breaks

So train both from the start — obviously. Except this does not work out of the box, because of latent collapse.

What collapses is the spread of the posterior $q(z|x)$ that the encoder emits. If that distribution loses its width and folds into a point, the latent space stops working as a medium that carries information. In the extreme, every input produces the same $z$. The loss number keeps going down while the inside quietly hollows out.

Intuitively, this is a story about how peaked a distribution is. In the figure below, drag the temperature down and watch a flat distribution spike into a single bar while everything else vanishes. Latent collapse is that same shape of failure happening to the posterior. (This is an analogy about losing spread, not a claim about softmax temperature specifically.)

The paper's point is that two forces push toward that collapse (Abstract): the reconstruction objective, and prior fitting. Both shrink the posterior.