Speech Synthesis from Scratch — From Text to a Voice

Speech synthesis invents a waveform tens of thousands of times longer than the handful of characters it starts from. This walks through why naive regression fails (one-to-many and phase), why text → mel spectrogram → waveform became the standard split, and how a few seconds of reference audio is now enough to carry a voice.

WaveNet: A Generative Model for Raw Audio

What exactly are you generating when you generate a voice?

The output of a TTS (Text-To-Speech) system is, stripped of everything else, a list of numbers: how far a speaker cone should push forward or pull back, measured at fixed intervals. That's all it is.

Speech synthesis usually runs at 22,050 Hz or 24,000 Hz. So one second of speech means more than twenty thousand numbers, in the right order, at the right magnitude. Ask a system to say "hello" over 1.5 seconds and you hand it five characters and demand thirty-six thousand numbers back. Conjuring something seven thousand times longer than the input, out of nothing — that is the job.

Speech recognition from scratch covered the opposite direction: collapsing tens of thousands of numbers down to a few characters. Collapsing is a discarding operation, so small errors in the details still land on the same word. Expansion has no such cushion. There is nothing to throw away — instead, every missing detail has to be invented.

Three reasons naive regression falls apart

First, nothing tells you how long anything lasts. Text carries no duration. The vowel in "no" can occupy 50 milliseconds or a full second. That number exists nowhere in the input, and the model has to produce it anyway.

Second, there is no single correct answer. "It'll be sunny tomorrow" can be said in endless ways — faster, slower, with a pause here, with the emphasis there. Your training data contains exactly one of them. If you naively instruct the model to minimize squared error against that one recording, it learns that the safest output is the average of every plausible reading. Averaged speech is blurred at the edges and sounds muffled. In generative modeling this is called over-smoothing.

The fix is to stop aiming at a single target and instead build a distribution and draw from it. How peaked that distribution is turns directly into how the sentence gets delivered.

Third, phase. Human hearing is nearly blind to how a wave is shifted in time. Delay a sound by a thousandth of a second and it sounds identical. As a list of numbers, though, the shifted waveform has essentially nothing in common with the original. Which means per-sample squared error barely correlates with what you hear. Two clips can sound the same and score terribly, or sound completely different and score well. This is the deepest reason you must not point your loss function straight at the waveform.

Divide and conquer: the two-stage split

Fighting all three at once is a losing proposition, and modern TTS answers by cutting the job in half.

text ──[acoustic model]──▶ mel spectrogram ──[vocoder]──▶ waveform

The first half is a language problem: what to say, with what prosody, over what duration. The second half is a signal problem: turn that timbre into actual air pressure. Splitting buys you two things. The second half depends on neither the language nor the speaker, so it can be trained on piles of untranscribed audio and reused under many different front ends. And the first half never has to touch phase at all.

The mel spectrogram: a blueprint for sound

The mel spectrogram that the first half emits is sound redrawn as an image, with frequency running up and time running across.

Building one is mundane. Slice the waveform into windows of roughly 20–50 milliseconds, Fourier-transform each window to measure how much energy sits at each frequency, and slide the window forward about 10 milliseconds at a time (this is the short-time Fourier transform, or STFT). Each slice gives you one vertical column of "how much of each pitch is sounding right now," and the columns line up into a picture.

Then you re-rule the frequency axis to match the ear rather than physics.

Here $f$ is physical frequency in hertz and $m$ is the ear's own scale of pitch. Equation (1) is a compression of the frequency axis which says, in words, just this: you hear fine distinctions down low and only coarse ones up high. The gap between 100 Hz and 200 Hz is obvious; 8,000 Hz and 8,100 Hz are all but indistinguishable. That property of hearing gets baked into the ruler itself.

Bundle the frequency axis into roughly 80 bands on that ruler (the mel filterbank), take the log of the energies, and you have a mel spectrogram. A 24 kHz waveform is 24,000 numbers per second; 80 mel bands every 10 milliseconds is 8,000 per second. Not a dramatic reduction in count — but it is smooth, and it contains no phase. Reason three disappears, and reason two does much less damage.

The flip side is that a mel spectrogram is a blueprint for sound, not sound. Something else has to reconstruct the phase you threw away and turn it back into audio.