Representing Sound — Mel Spectrograms and Audio Tokens

Why speech models never eat raw waveforms, and what they eat instead: the chain from short-time Fourier transform to the mel scale to the log to discrete tokens. Covers the window-length tradeoff, why MFCCs dropped the DCT, how acoustic and semantic tokens differ, and the config mismatches that silently wreck audio in production.

Robust Speech Recognition via Large-Scale Weak Supervision

What the microphone actually hands you

A microphone does something simple. It measures air pressure at fixed intervals and writes each measurement down as a number. At 16 kHz — the standard rate for speech — that is 16,000 numbers per second, so ten seconds of talking arrives as a strip of 160,000 numbers.

Inside that strip your voice, the noise from the next room, and the reflections off the walls are all dissolved into a single wave. Human ears pull words out of it effortlessly. Handing a machine those 160,000 numbers directly, though, is like asking someone to read a book by studying ink density values on the page. Not impossible. Just a very long way round.

So essentially every audio system converts the waveform into some other representation before the real work begins. This article walks the whole chain as one road: Fourier transform → mel scale → log → discrete tokens. You may already know one link in that chain; what changes the picture is seeing why they come in that order.

The metaphor: sheet music is a compression format

Imagine trying to preserve an orchestral performance on paper. Transcribing the actual air vibrations is hopeless. But we have sheet music, which records only when a pitch sounds, which pitch, and how loudly. It says nothing about the player's timbre, nothing about the hall's reverberation, nothing about the phase of the wave. And yet hand it to a different orchestra and the piece comes back.

A spectrogram is sheet music for machines: sound laid out as a table with pitch running up and time running across, phase thrown away exactly as a score throws it away.

A mel spectrogram is that same score with the staff lines respaced to match human hearing — fine detail down low, broad strokes up high.

And tokenization is what happens when you round each note to one of a fixed set of symbols. Handwriting becomes movable type. Once sound is a string of symbols, the enormous machinery built for language models will happily consume it.

Why not just feed the waveform?

Three reasons.

It is too long. Thirty seconds at 16 kHz is 480,000 samples. Self-attention costs grow with the square of sequence length, so nobody wants to attack that head-on.

Sounds that are identical to the ear are wildly different as numbers. Record the same vowel one millisecond later and every number changes. Shift the phase, scale the amplitude slightly — still the same sound to you, still a completely different array. A network fed raw samples has to learn all of that invariance from scratch before it learns anything about language.

The features that carry meaning are invisible in the time domain. What separates "ah" from "ee" is the position of the resonant peaks the vocal tract produces — the formants. Stare at the waveform as long as you like; the peaks are not there to see. Move to frequency and they appear as visible stripes.

Switching to a frequency representation fixes all three at once: the sequence shrinks to one frame per 10 milliseconds, dropping phase hands you the invariance for free, and formants become picture-like structure.

Mechanism 1: the short-time Fourier transform

The tool that "unmixes a combined wave into its component pitches" is the Fourier transform. (How it works, and why it can be computed so fast, is the subject of FFT from scratch.)

Apply it once to ten seconds of speech, though, and all ten seconds of frequency content collapse into a single graph. When each sound occurred is gone entirely — as if an entire score were summarized as one chord.

The fix is to chop first. Cut a window of roughly 25 milliseconds, transform only what is inside it, slide the window forward by 10 milliseconds, repeat. That is the short-time Fourier transform (STFT).

In words, this builds a table that records how much of frequency k is present in each short slice t of the sound, for every slice and every frequency.

Symbol by symbol: $x$ is the waveform, $N$ the window length in samples, $H$ the hop size, $w$ the window function, $t$ which slice we are on, $k$ which frequency we are asking about. Equation (1) says only this: build a table whose entry $(t, k)$ is "how much of pitch $k$ lives inside slice $t$", for every slice and every pitch.

Each $X$ is a complex number carrying a magnitude (how strong that component is) and an angle (its phase). Discard the angle, keep $|X|^2$, and you have a power spectrogram.

That $e^{-i 2\pi k n / N}$ at the end is just "a sine wave at frequency $k$." Multiplying it into the signal and summing is a dot product. The more the signal points the same way as that sine wave, the larger the sum. Frequency analysis, stripped down, is taking a great many dot products at once.

The window length decides what you are allowed to see

The first thing you choose in an STFT is the window length, and the choice carries a tradeoff you cannot undo later.

A longer window fits more cycles of each wave, so frequency resolution improves and you can separate pitches that sit close together. The price is that everything happening inside that span gets flattened into one frame, so time resolution degrades — the sharp onset of a plosive smears out. Shorten the window and the two effects swap places.

The relationship is roughly $\Delta t \cdot \Delta f \approx \text{constant}$: improve one and the other gets worse. This is not an implementation artifact. It is a structural limit on looking at a signal through time and through frequency at the same time.

Speech settled on 25 ms windows with a 10 ms hop because that matches a useful rule of thumb: over that span, the shape of the mouth and tongue barely changes. Music analysis prefers longer windows (2048 samples is common) — you want to resolve the notes inside a chord, so you spend your budget on the frequency side.

The window function (a Hann window, typically) is worth understanding too. Cut a signal with a rectangular edge and the value drops abruptly to zero at the seam. To a Fourier transform, that cliff looks like "a faint amount of every pitch at once," scattering fake energy across frequencies that were never sounding. This is spectral leakage. Tapering both ends smoothly is how you remove the cliff.

Mechanism 2: the mel scale, or respacing the ruler for ears

The frequency axis the STFT returns is linear: from 0 Hz to 8,000 Hz you might get 513 evenly spaced bins. That even spacing does not match hearing at all.

The gap between 100 Hz and 200 Hz is unmistakable — a full octave. The gap between 8,000 Hz and 8,100 Hz is essentially inaudible. Same 100 Hz of physical difference, wildly different perceptual meaning.

So we respace the ruler.

In words, the mel formula converts a physical frequency in Hz into pitch as the ear experiences it. It stretches the low end and compresses the high end, so each doubling of frequency adds roughly the same number of mels.

Here $f$ is physical frequency in Hz and $m$ is pitch as the ear experiences it, in mels. All equation (2) does is stretch the low end and compress the high end. Because of the logarithm, each doubling of frequency buys roughly the same increase in mels.

On this new ruler we lay out about 80 evenly spaced triangular windows — the mel filterbank — and sum the power sitting under each triangle. That collapses 513 linear bins down to 80 bands. Down low the triangles are narrow and densely packed; up high they are wide and sparse, so low frequencies end up finely recorded and high frequencies coarsely.

In matrix terms it is one multiplication by a sparse $(80 \times 513)$ matrix. That matrix is not learned. It is measured facts about human hearing, baked in as constants.

The same "ruler for ears" shows up in audio compression, with the goals reversed, which is the entertaining part. MP3 uses critical bands to decide what can be thrown away unnoticed; a mel spectrogram uses them to decide what is worth keeping. Discard side versus keep side — the underlying measurements of the ear are the same.

Mechanism 3: the log, or folding the orders of magnitude

Mel filter outputs are still awkward. Power values between a quiet passage and a loud one can differ by nearly ten orders of magnitude. Feed that straight into a loss and the loudest frames dominate training while everything quiet is ignored.

Loudness perception is logarithmic anyway. Ten times the power is nowhere near "ten times as loud." So take the log.

logmel = np.log(mel + 1e-6)   # or np.log(np.maximum(mel, 1e-10))

That unglamorous 1e-6 matters more than it looks. In true silence the power is zero and the log diverges to $-\infty$. The value of that floor decides how low "silence" sits in your features, and changing it changes the dynamic range of the entire representation.

What comes out of this is the log-mel spectrogram, and modern speech recognition and synthesis have converged on it almost universally as the input. Whisper, for instance, takes 16 kHz audio, runs an STFT with 25 ms windows and a 10 ms stride, and feeds 80 log-mel channels (large-v3 raised that to 128). Different numbers, same pipeline this article just walked.

MFCCs: the former standard, and where they still live

Run one more transform — a discrete cosine transform (DCT) — over the log-mel values and keep the first 13 or so, and you have MFCCs, mel-frequency cepstral coefficients. They were the standard speech feature for decades.

Why the DCT? The GMM-HMM systems of that era worked best when feature dimensions were roughly independent. Mel filter outputs are anything but: adjacent triangles overlap, so adjacent bands correlate strongly. The DCT decorrelates them and packs the information into a handful of dimensions.

Deep learning dropped the DCT. Networks handle correlated dimensions on their own, and truncating to 13 numbers is simply lost information. What goes into a modern ASR or TTS model is the log-mel itself.

MFCCs are not dead, though. They are cheap and low-dimensional, so they still power always-on keyword spotting, speaker identification, and voice activity detection. And HuBERT's first training round clustered MFCCs to produce its initial targets — which means the "semantic tokens" we are about to meet started life on top of an old-fashioned feature.

Mechanism 4: from continuous to discrete — tokenizing sound

A log-mel spectrogram is a matrix of continuous values. Tokenization takes one more step, to a sequence of symbols from a finite set.

The motivation is blunt. A language model is a machine for sequences drawn from a finite vocabulary. Turn sound into symbols and all the machinery and infrastructure built for text transfers over unchanged. That is how speech synthesis turns into "predict the next token."

Audio tokens come in two families with different goals.

Acoustic tokens are codes trained so the original waveform can be rebuilt. An encoder compresses the waveform into a short sequence of vectors, each vector is rounded to its nearest entry in a codebook, and a decoder reassembles a waveform from the codes. One rounding stage is far too coarse, so the residual gets quantized again and again — residual vector quantization (RVQ). EnCodec and SoundStream are the canonical examples, covered in depth in neural audio codecs. Timbre, background noise, and room reverb all survive, but each frame needs several codes.

Semantic tokens come from taking the intermediate activations of a self-supervised model such as HuBERT and rounding them, via k-means, to one of a few hundred or few thousand centroids. Nothing here is trained to reconstruct audio, so speaker identity and recording conditions largely fall away and what survives is closer to phonetic content. One code per frame keeps sequences short and long context tractable — but you cannot turn these back into sound on their own.

That asymmetry is exactly why frameworks like AudioLM use both: semantic tokens decide what is said over long spans, acoustic tokens fill in how it sounds. The division of labor maps directly onto the two token types.

In both families the discretization step itself is the same operation: find the closest entry to your vector in a fixed dictionary. Nothing more.

Code: from waveform to tokens

First the waveform-to-log-mel step. In practice it is a one-liner, but every argument maps onto something explained above.

import torch, torchaudio

wav, sr = torchaudio.load("hello.wav")                    # (1, num_samples)
wav = torchaudio.functional.resample(wav, sr, 16_000)     # force 16 kHz

melspec = torchaudio.transforms.MelSpectrogram(
    sample_rate=16_000,
    n_fft=400,          # 25 ms window (16000 x 0.025)
    hop_length=160,     # slide 10 ms at a time
    n_mels=80,          # 80 mel bands
    f_min=0.0, f_max=8000.0,   # ceiling is the Nyquist frequency
    power=2.0,          # power, not magnitude
)
mel = melspec(wav)                    # (1, 80, num_frames)
logmel = torch.log(mel + 1e-6)        # fold the orders of magnitude

Then rounding those frames to codebook entries. "Quantization" sounds forbidding; the body of it is an argmin over distances.

def quantize(frames, codebook):        # frames: (T, D), codebook: (K, D)
    dist = torch.cdist(frames, codebook)   # (T, K) every frame against every code
    idx = dist.argmin(dim=1)               # (T,) the code each frame picked
    return idx, codebook[idx]              # token sequence and its reconstruction

idx is your audio token sequence. From here it behaves like any sequence of text tokens.

The return trip: getting sound back

We threw phase away on the way out, so on the way back it has to be invented. This is the genuinely awkward part of audio representations.

The classical answer is Griffin-Lim: guess a phase, bounce between STFT and inverse STFT, and let the estimate drift toward something self-consistent. It works, and it leaves a characteristic metallic ring.

The modern answer is a neural vocoder. Models like HiFi-GAN generate a waveform directly from a log-mel. Rather than estimating phase, they simply learn to emit audio that sounds right. When you start from tokens instead, the neural codec's own decoder plays the same role.

Which leads to the consequence that causes the most production incidents: the settings used to build the features and the settings used to invert them must not differ by a single value. A vocoder can only interpret the kind of log-mel it was trained on. Number of mel bands, hop length, the log floor — all of it is an unwritten contract.

How this gets used in practice

The people who touch this chain are ML engineers working on ASR and TTS, application developers bolting speech features onto a product, and data engineers building audio preprocessing pipelines. You do not have to write a single line of model code — the moment you own the data preparation, this is your job.

Parameters and tools you will actually type. sample_rate / n_fft / win_length / hop_length / n_mels / f_min and f_max / power / center / norm / mel_scale. The two dominant implementations are librosa.feature.melspectrogram and torchaudio.transforms.MelSpectrogram, with ffmpeg -ar 16000 -ac 1 or sox doing format normalization upstream.

Six ways this bites people.

1. Sample rate mismatch. Hand 44.1 kHz audio to a model expecting 16 kHz and most implementations raise nothing at all. Features with every frequency shifted simply flow through, and accuracy sags a little. Make "resample and downmix to mono as the first step of preprocessing" a mechanical rule, not a judgment call.

2. Different mel filterbank conventions. librosa defaults to the Slaney convention (htk=False, filters area-normalized), which produces different numbers than the HTK convention. If your training code and your vocoder come from different libraries, this diverges silently. The audio changes audibly while no line of code is wrong — a genuinely hard bug to locate.

3. Different log floors. log(x + 1e-6) and log(max(x, 1e-10)) put silence in completely different places. Recompute your feature statistics — including any mean/variance normalization — per dataset and per recipe. Never import someone else's numbers.

4. The implicit padding in center=True. By default the signal is padded on both sides, making the frame count num_samples // hop_length + 1. Anywhere you compute frame counts for CTC alignment or label timing, you will be off by one frame. It is a boundary effect, so the shorter the clip, the larger the relative damage.

5. Power versus magnitude. power=2.0 and power=1.0 differ by a constant factor after the log. It looks like normalization absorbs it, but the relationship to the floor shifts, so everything near silence ends up represented differently.

6. Reusing a tokenizer across domains. k-means centroids and RVQ codebooks are welded to the acoustic conditions of the corpus they were fit on. Push telephone-band audio (8 kHz, band-limited) through a codebook built from 16 kHz studio speech and the codes collapse onto a small subset. Counting codebook utilization is a cheap check that catches exactly this.

How it shows up in interviews and design reviews. "What improves and what degrades when you lengthen the window?" "Why can speech recognition work at all after phase is discarded?" "When would you use MFCCs instead of log-mels?" Every answer is somewhere above: the time-frequency tradeoff; phonetic information sits almost entirely in the shape of the magnitude spectrum; log-mels for deep models, MFCCs for lightweight classical ones.

Summary

• Raw waveforms are avoided because they are too long, offer no invariance, and hide the features that matter
• The STFT is "chop, then Fourier transform." Choosing the window length is choosing whether to see time or frequency in detail
• The mel scale and filterbank are a fixed transform that respaces frequency to match hearing; the log folds the orders of magnitude
• Tokenization is nothing but nearest-neighbor lookup in a codebook. Acoustic tokens aim at reconstruction, semantic tokens at phonetic content
• Feature settings are a contract with whatever converts them back to sound. One mismatch and it breaks quietly

For what gets built on top of this representation, continue to speech recognition from scratch.