Representing Sound — Mel Spectrograms and Audio Tokens

Why speech models never eat raw waveforms, and what they eat instead: the chain from short-time Fourier transform to the mel scale to the log to discrete tokens. Covers the window-length tradeoff, why MFCCs dropped the DCT, how acoustic and semantic tokens differ, and the config mismatches that silently wreck audio in production.

Robust Speech Recognition via Large-Scale Weak Supervision

What the microphone actually hands you

A microphone does something simple. It measures air pressure at fixed intervals and writes each measurement down as a number. At 16 kHz — the standard rate for speech — that is 16,000 numbers per second, so ten seconds of talking arrives as a strip of 160,000 numbers.

Inside that strip your voice, the noise from the next room, and the reflections off the walls are all dissolved into a single wave. Human ears pull words out of it effortlessly. Handing a machine those 160,000 numbers directly, though, is like asking someone to read a book by studying ink density values on the page. Not impossible. Just a very long way round.

So essentially every audio system converts the waveform into some other representation before the real work begins. This article walks the whole chain as one road: Fourier transform → mel scale → log → discrete tokens. You may already know one link in that chain; what changes the picture is seeing why they come in that order.

The metaphor: sheet music is a compression format

Imagine trying to preserve an orchestral performance on paper. Transcribing the actual air vibrations is hopeless. But we have sheet music, which records only when a pitch sounds, which pitch, and how loudly. It says nothing about the player's timbre, nothing about the hall's reverberation, nothing about the phase of the wave. And yet hand it to a different orchestra and the piece comes back.

A spectrogram is sheet music for machines: sound laid out as a table with pitch running up and time running across, phase thrown away exactly as a score throws it away.

A mel spectrogram is that same score with the staff lines respaced to match human hearing — fine detail down low, broad strokes up high.

And tokenization is what happens when you round each note to one of a fixed set of symbols. Handwriting becomes movable type. Once sound is a string of symbols, the enormous machinery built for language models will happily consume it.

Why not just feed the waveform?

Three reasons.

It is too long. Thirty seconds at 16 kHz is 480,000 samples. Self-attention costs grow with the square of sequence length, so nobody wants to attack that head-on.

Sounds that are identical to the ear are wildly different as numbers. Record the same vowel one millisecond later and every number changes. Shift the phase, scale the amplitude slightly — still the same sound to you, still a completely different array. A network fed raw samples has to learn all of that invariance from scratch before it learns anything about language.

The features that carry meaning are invisible in the time domain. What separates "ah" from "ee" is the position of the resonant peaks the vocal tract produces — the formants. Stare at the waveform as long as you like; the peaks are not there to see. Move to frequency and they appear as visible stripes.

Switching to a frequency representation fixes all three at once: the sequence shrinks to one frame per 10 milliseconds, dropping phase hands you the invariance for free, and formants become picture-like structure.

Mechanism 1: the short-time Fourier transform

The tool that "unmixes a combined wave into its component pitches" is the Fourier transform. (How it works, and why it can be computed so fast, is the subject of FFT from scratch.)

Apply it once to ten seconds of speech, though, and all ten seconds of frequency content collapse into a single graph. When each sound occurred is gone entirely — as if an entire score were summarized as one chord.

The fix is to chop first. Cut a window of roughly 25 milliseconds, transform only what is inside it, slide the window forward by 10 milliseconds, repeat. That is the short-time Fourier transform (STFT).

Symbol by symbol: $x$ is the waveform, $N$ the window length in samples, $H$ the hop size, $w$ the window function, $t$ which slice we are on, $k$ which frequency we are asking about. Equation (1) says only this: build a table whose entry $(t, k)$ is "how much of pitch $k$ lives inside slice $t$", for every slice and every pitch.

Each $X$ is a complex number carrying a magnitude (how strong that component is) and an angle (its phase). Discard the angle, keep $|X|^2$, and you have a power spectrogram.

That $e^{-i 2\pi k n / N}$ at the end is just "a sine wave at frequency $k$." Multiplying it into the signal and summing is a dot product. The more the signal points the same way as that sine wave, the larger the sum. Frequency analysis, stripped down, is taking a great many dot products at once.

The window length decides what you are allowed to see

The first thing you choose in an STFT is the window length, and the choice carries a tradeoff you cannot undo later.

A longer window fits more cycles of each wave, so frequency resolution improves and you can separate pitches that sit close together. The price is that everything happening inside that span gets flattened into one frame, so time resolution degrades — the sharp onset of a plosive smears out. Shorten the window and the two effects swap places.

The relationship is roughly $\Delta t \cdot \Delta f \approx \text{constant}$: improve one and the other gets worse. This is not an implementation artifact. It is a structural limit on looking at a signal through time and through frequency at the same time.