Music and Audio Generation from Scratch — Sound as Tokens

A three-minute song is more than 15 million numbers. The history of music generation is largely the history of folding that sequence down to something a model can hold. Codec tokens, the fork between autoregressive and diffusion approaches, how text conditioning actually bites, and how to reconstruct the shape of Suno-style products from published parts — no prior knowledge assumed.

Simple and Controllable Music Generation

A three-minute song is 15 million numbers

Producing three minutes of CD-quality audio (44.1 kHz, stereo) means putting roughly 15.88 million numbers in a convincing order — 44,100 measurements per second, twice over for two channels.

Compare that to text. A 3,000-word article is a few thousand tokens. Audio is four orders of magnitude away. Worse, neighbouring samples are tightly correlated, so a single wrong value doesn't blur into the background — it arrives at your ear as an audible click.

Which is why the history of music and audio generation is, for the most part, the history of folding those 15.88 million numbers down. Get the arithmetic straight before worrying about architectures.

The analogy: write notes, or write catalogue numbers

There were two long-standing ways to do the folding.

The first is the score approach. "Play a C for half a second on piano" — three minutes becomes a few thousand symbols. MIDI-generating models live here, and they are genuinely cheap to run. But turning symbols back into sound is where the singing voice, the grit of an overdriven guitar, and the room's reverb all fall out. A score records what to play, never how it actually sounded.

The second is the waveform approach. Generate the air pressure itself and you can represent anything — but the sequence is hopeless, as above. OpenAI's Jukebox took this route in 2020 and produced songs with vocals, at the cost of hours per track.

What dominates today sits in between: the catalogue approach. Slice the audio into short frames and replace each frame with an integer — "which entry of the codebook does this look most like?" As short as a score, as expressive as a waveform. That sequence of integers is what we call codec tokens, and "sound as tokens" means exactly this substitution.

Counting per second — why tokens make this tractable

The payoff shows up immediately in the arithmetic. The token count is a plain product.

In words: seconds × frames per second × how many integers you stack per frame. Here $T$ is duration, $f$ is the frame rate (how finely a second is chopped), and $n_q$ is the number of stacked codes per frame — the number of residual quantization stages.

MusicGen uses a 32 kHz EnCodec with $f = 50$ and $n_q = 4$. Three minutes is $180 \times 50 \times 4 = 36{,}000$ tokens. The 15.88 million became thirty-six thousand — the same ballpark as a long document.

The same ingredients give you the bitrate.

$$R = f \cdot n_q \cdot \log_2 K$$

That is, frames per second × stages × bits needed to write one index. With a codebook size $K$ of 2048, each index costs 11 bits, so $50 \times 4 \times 11 = 2{,}200$ bps — about 2.2 kbps.

Why stack several codes per frame instead of one — residual quantization — is covered in neural audio codecs. The only premise you need here is one line: audio can be turned into a finite sequence of integers, and turned back.

Fork one: autoregression — keep guessing the next index

Once audio is a sequence of integers, it behaves like text. Predict the next index, append it, repeat. The same machinery that produces sentences produces songs.

Audio adds one wrinkle text doesn't have: each frame carries four indices stacked vertically. A text token is one symbol per position, so there is nothing to decide; audio has to emit four codebook entries for the same instant.

The two obvious answers both hurt. Flatten the four stages into one stream and the sequence gets four times longer, so generation gets four times slower. Emit all four independently and the stages stop agreeing with each other — stage 1 commits to a piano while stage 2 adds the residual of a distorted guitar.