Paper Walkthrough: AURORA-LM — A Diffusion Language Model That Keeps Its Text Latent Full-Width Instead of Compressing It

A walkthrough of AURORA-LM, a continuous-latent diffusion language model that rejects the usual compromise of compressing text latents to make diffusion easier. It keeps a decodable, width-1024 latent and learns its distribution with a low-rank input pathway, width-calibrated noise, and self-trajectory consistency.

AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling

Text is the last holdout of discrete generation

Images, video, and audio are now mostly generated by diffusion models operating in continuous latent spaces. Text is the exception: it is still produced one discrete token at a time, picked from a vocabulary (§1). Even multimodal systems usually go the other way, converting continuous signals into discrete tokens so they fit the language-model toolchain. The paper asks the reverse question: can language itself be represented and generated in a continuous space? (§1)

Think of it as a moving job. You pack a text into a box of continuous vectors (encoding), a moving company — the generative model — learns to produce such boxes from scratch, and the recipient unpacks the box back into text (decoding). The trouble is in the packing. Make the box small and light and the movers have an easy day, but when you open it, the dishes are chipped — the exact original words don't come back.

AURORA-LM (a collaboration between Nanjing University, Nanyang Technological University, and Imperial College London; arXiv:2608.02602) takes a clear position: keep the box big, and train better movers.

The dilemma in prior work: compress it and it ships, but arrives broken

The paper sorts existing continuous language generation into two camps (§1, §2.3).

The first is embedding-space methods. Diffusion-LM and PLAID denoise sequences of word embeddings directly; TEncDM uses contextual features from a pretrained encoder. Borrowing an existing representation is convenient, but its width and organization were never designed to support both generation and faithful decoding at once (§1).

The second is autoencoder-based methods. LD4LG, COSMOS, and Cola-DLM first map text into a purpose-built latent space and then run diffusion there. Compressing and smoothing the latent makes the diffusion model's job easier — but the decoder depends on that same latent, so compression also strips away information needed to recover the exact words and their order. Ease of generation and fidelity of reconstruction end up in a tug-of-war over a single representational bottleneck — that is the paper's diagnosis of the field (§1).

You can feel this "lossy compression eats the details" effect with JPEG. Crank up the compression and the file gets lighter, but the fine structure never comes back. In a text latent, the fine structure that vanishes is the exact wording.

AURORA-LM's answer: fully separate representation from distribution modeling

AURORA-LM splits the work into two stages (§3).

In stage one, a Query-based Encoder-Decoder maps text into a high-capacity latent sequence, trained with a single objective: reconstruct the original tokens. No concessions are made to make generation easier.

In stage two, the autoencoder is frozen, and a Block-causal Diffusion Transformer learns the distribution of its outputs via flow matching — generating blocks left to right while denoising all positions within a block in parallel. Because the representation is fixed before the generator is designed, the usual backwards pressure — "diffusion is struggling, so shrink the latent" — never enters the loop (§2.3, §3).

Component 1: an autoencoder that builds a prefix-readable latent

The encoder turns a token sequence of length $L$ into $N = \operatorname{round}(cL)$ latent vectors of width $D$, where $c$ is a retention ratio and $c=1$ means no sequence compression (§3.2). The trick is in what each query is allowed to see. The $i$-th latent query attends only to the preceding latents and to the first $\lceil iL/N \rceil$ tokens — so a prefix of the latent sequence corresponds exactly to a prefix of the text. The decoder is symmetric: when reconstructing token $j$, it may only look at the matching latent prefix. This alignment is what later makes left-to-right blockwise generation possible (§3.2).