Neural Audio Codecs — EnCodec and the Foundation Under Speech LLMs

MP3 and Opus were hand-designed around what the ear cannot hear. Neural audio codecs learn the coding itself and turn sound into a finite alphabet of integers. From vector quantization to residual VQ, EnCodec's bitrate arithmetic, and why VALL-E and AudioLM are built on top of it — starting from zero.

High Fidelity Neural Audio Compression

Turning sound into a string of numbers

As we saw in audio compression from scratch, MP3 and AAC are machines built by hand: engineers studied the human ear, wrote down masking curves, chose the MDCT, and designed a bit allocation rule that throws away what you cannot hear.

A neural audio codec hands that design over to a network. An encoder folds the waveform into a small representation, a decoder rebuilds it, and the two train together on a large pile of audio. Nobody writes down what to discard; the only target is how the result sounds coming back.

If that were all, this would be a slightly better compressor. The part that actually mattered was a side effect. The intermediate representation is a finite set of integers. One second of audio becomes, say, 600 numbers — the same shape as a string of text, which means a language model can eat it directly.

TTS systems like VALL-E, audio generation like AudioLM, music generation like MusicGen, real-time conversational AI you can talk to like a phone call: almost everything that makes sound today sits on top of this "turn audio into numbers" layer. The codec became important less as a compressor than as a vocabulary factory.

The metaphor: redrawing with a fixed set of colored pencils

Suppose the sender and the receiver own the identical box of 1,024 colored pencils. To send a picture, you never need to send the colors themselves. Say "this dot is number 347" and the receiver pulls out the same pencil. The color will not match exactly, but with 1,024 of them something close always exists. That is vector quantization.

Error remains, of course, and that is where the second trick comes in: take whatever the first pencil failed to cover, and paint that difference again with a second, separate box. Paint what still remains with a third. Each stage brings you closer to the original and costs one more number. That is residual vector quantization (RVQ), and it is the heart of every modern neural audio codec.

The important property is that you can stop partway. Three stages: coarse but cheap. Eight stages: precise but expensive. The same machine gives you a whole ladder of quality and bitrate — and that turns out to matter later.

The intuition: why throw away "continuous"

The encoder's output, left alone, is a vector of real numbers. A 128-dimensional vector in float32 is 4,096 bits per frame — no compression at all. Round it to "which of these 1,024 representative vectors (the codebook) is closest" and one frame costs $\log_2 1024 = 10$ bits. A factor of 400. Discarding continuity in favor of discreteness is what compression actually is here.

Discreteness also buys something unrelated to compression. A language model picks the next item out of a finite set, so it cannot run unless what it consumes is a finite vocabulary. For exactly the reason text needs a tokenizer, audio needs one too — and the neural codec doubles as it.

Mechanism 1: vector quantization

The encoder is a convolutional network that aggressively downsamples in time. Fold 24 kHz audio (24,000 samples per second) by a factor of 320 and you get 75 frames per second, each a D-dimensional vector $z$.

Here $z$ is one frame from the encoder, $c_j$ is the j-th entry of the codebook, $N$ is the codebook size (1,024 is the usual choice), and $\hat{z}$ is the quantized value. In words, equation (1) says: pick the nearest entry you own and transmit only its index $k$. The receiver holds the same codebook, so the index is enough to recover $c_k$.

The catch is that $\arg\min$ has no derivative. If the gradient dies here, the encoder never learns. The VQ-VAE fix is crude and effective: quantize on the forward pass and pretend the quantizer was not there on the backward pass (the straight-through estimator). Whatever gradient arrives at the decoder is copied straight onto the encoder output $z$.

In exchange, you tie the codebook and $z$ together so they cannot drift apart.

$\mathrm{sg}[\cdot]$ marks "no gradient flows past this point" (stop-gradient) and $\beta$ sets how hard the second pull is. In words: the first term drags the codebook entry toward the encoder's output, and the second term penalizes the encoder for wandering away from the codebook. Two ropes pulling both sides toward each other. In practice the first term is often replaced by an exponential moving average over the vectors that got used, rather than a gradient step.

Worth pausing on: the codebook is not a fixed table, it is learned. It is initialized with k-means and then fills up with representatives of whatever shapes the training audio actually contains. Two codecs with 1,024 entries each — one trained on speech, one on music — hold genuinely different sets of colors. That is also why audio outside the training distribution (an instrument that never appeared, extreme noise) degrades badly: unlike a classical codec, a neural one does not promise uniform quality on arbitrary input.

Picking the single closest entry out of a codebook is exactly nearest-neighbor search over embeddings.