Speech Synthesis from Scratch — From Text to a Voice

Speech synthesis invents a waveform tens of thousands of times longer than the handful of characters it starts from. This walks through why naive regression fails (one-to-many and phase), why text → mel spectrogram → waveform became the standard split, and how a few seconds of reference audio is now enough to carry a voice.

WaveNet: A Generative Model for Raw Audio

What exactly are you generating when you generate a voice?

The output of a TTS (Text-To-Speech) system is, stripped of everything else, a list of numbers: how far a speaker cone should push forward or pull back, measured at fixed intervals. That's all it is.

Speech synthesis usually runs at 22,050 Hz or 24,000 Hz. So one second of speech means more than twenty thousand numbers, in the right order, at the right magnitude. Ask a system to say "hello" over 1.5 seconds and you hand it five characters and demand thirty-six thousand numbers back. Conjuring something seven thousand times longer than the input, out of nothing — that is the job.

Speech recognition from scratch covered the opposite direction: collapsing tens of thousands of numbers down to a few characters. Collapsing is a discarding operation, so small errors in the details still land on the same word. Expansion has no such cushion. There is nothing to throw away — instead, every missing detail has to be invented.

Three reasons naive regression falls apart

First, nothing tells you how long anything lasts. Text carries no duration. The vowel in "no" can occupy 50 milliseconds or a full second. That number exists nowhere in the input, and the model has to produce it anyway.

Second, there is no single correct answer. "It'll be sunny tomorrow" can be said in endless ways — faster, slower, with a pause here, with the emphasis there. Your training data contains exactly one of them. If you naively instruct the model to minimize squared error against that one recording, it learns that the safest output is the average of every plausible reading. Averaged speech is blurred at the edges and sounds muffled. In generative modeling this is called over-smoothing.

The fix is to stop aiming at a single target and instead build a distribution and draw from it. How peaked that distribution is turns directly into how the sentence gets delivered.

Third, phase. Human hearing is nearly blind to how a wave is shifted in time. Delay a sound by a thousandth of a second and it sounds identical. As a list of numbers, though, the shifted waveform has essentially nothing in common with the original. Which means per-sample squared error barely correlates with what you hear. Two clips can sound the same and score terribly, or sound completely different and score well. This is the deepest reason you must not point your loss function straight at the waveform.

Divide and conquer: the two-stage split

Fighting all three at once is a losing proposition, and modern TTS answers by cutting the job in half.

text ──[acoustic model]──▶ mel spectrogram ──[vocoder]──▶ waveform

The first half is a language problem: what to say, with what prosody, over what duration. The second half is a signal problem: turn that timbre into actual air pressure. Splitting buys you two things. The second half depends on neither the language nor the speaker, so it can be trained on piles of untranscribed audio and reused under many different front ends. And the first half never has to touch phase at all.

The mel spectrogram: a blueprint for sound

The mel spectrogram that the first half emits is sound redrawn as an image, with frequency running up and time running across.

Building one is mundane. Slice the waveform into windows of roughly 20–50 milliseconds, Fourier-transform each window to measure how much energy sits at each frequency, and slide the window forward about 10 milliseconds at a time (this is the short-time Fourier transform, or STFT). Each slice gives you one vertical column of "how much of each pitch is sounding right now," and the columns line up into a picture.

Then you re-rule the frequency axis to match the ear rather than physics.

Here $f$ is physical frequency in hertz and $m$ is the ear's own scale of pitch. Equation (1) is a compression of the frequency axis which says, in words, just this: you hear fine distinctions down low and only coarse ones up high. The gap between 100 Hz and 200 Hz is obvious; 8,000 Hz and 8,100 Hz are all but indistinguishable. That property of hearing gets baked into the ruler itself.

Bundle the frequency axis into roughly 80 bands on that ruler (the mel filterbank), take the log of the energies, and you have a mel spectrogram. A 24 kHz waveform is 24,000 numbers per second; 80 mel bands every 10 milliseconds is 8,000 per second. Not a dramatic reduction in count — but it is smooth, and it contains no phase. Reason three disappears, and reason two does much less damage.

The flip side is that a mel spectrogram is a blueprint for sound, not sound. Something else has to reconstruct the phase you threw away and turn it back into audio.

The first half: from text to blueprint

Deciding how it reads

The first obstacle arrives before machine learning does. Text normalization expands "2026" into "twenty twenty-six," and "3/4" into either "three quarters" or "March fourth" depending on context. Then grapheme-to-phoneme conversion (G2P) has to settle "read" as present or past tense, "live" as adjective or verb, "bass" as fish or instrument. Tonal and pitch-accent languages need more still: Japanese requires the accent nucleus that separates 雨 (rain) from 飴 (candy). Dictionaries and rules remain strong here, and tools like espeak-ng or pyopenjtalk do the work.

Deciding how long

How you solve reason one — duration — splits the first half into two schools.

Autoregressive models (the Tacotron 2 line) emit mel frames one column at a time, using attention to track where in the text they currently are. Duration is never stated; it falls out of the process. Prosody tends to come out natural, but when attention slips the failures are ugly: skipped words, repeated words, generation that never terminates.

Non-autoregressive models (the FastSpeech line) invert this. A duration predictor first says how many frames each phoneme should occupy, and a length regulator simply copies each phoneme's vector that many times.

import numpy as np

def length_regulate(phoneme_hidden, duration):
    # phoneme_hidden: (n_phonemes, dim)   duration: e.g. [3, 7, 2, 5, ...]
    return np.repeat(phoneme_hidden, duration, axis=0)  # (n_frames, dim)

One line, effectively. If a vowel should last seven frames, stamp its vector down seven times. Reducing the duration problem to one prediction plus a copy is what made FastSpeech fast — every frame can be produced in parallel, and nothing runs away. FastSpeech 2 goes further and predicts pitch and energy alongside duration (the variance adaptor), which is why you can scale speaking rate and expressiveness from outside at inference time.

The second half: the vocoder turns blueprint into air

The vocoder rebuilds the phase you discarded. It comes in clearly marked generations.

Generation 1 — signal processing (Griffin-Lim). Guess a phase, bounce between STFT and inverse STFT dozens of times, and let the phase converge to something self-consistent. No training, very cheap, and a persistent metallic "robot voice" quality.

Generation 2 — autoregressive (WaveNet). Predict waveform samples one at a time, each conditioned on the ones before it. Quality jumped, but producing one second of audio meant more than twenty thousand sequential steps. That is fatal for real-time synthesis, and a wave of follow-up work — distillation into parallel students, flow-based generators — existed mainly to kill that queue.

Generation 3 — adversarial (HiFi-GAN). The generator upsamples the mel with transposed convolutions all the way to waveform length and emits every sample in parallel. No queue at all. The quality comes from the discriminator design: HiFi-GAN pairs a multi-period discriminator, which folds the waveform by candidate periods, with a multi-scale discriminator that inspects it at several resolutions. Voiced speech has strong periodicity from vocal-fold vibration, and giving the critic an explicit eye for periods is what made it work. The generator-versus-critic framework itself is covered in the rise and fall of GANs.

Diffusion vocoders (DiffWave, WaveGrad) arrived later, trading iteration count against quality.

VITS takes a different route and folds the two-stage split back together, combining a VAE, normalizing flows, and adversarial training into one model that runs from text to waveform. With no seam, the whole class of mismatch bugs described below simply cannot occur.

Today: turn audio into tokens and hand it to a language model

A neural audio codec converts sound into a finite alphabet of integer tokens. Once audio is tokens, TTS becomes "translate a sequence of text tokens into a sequence of audio tokens" — squarely a language-modeling problem.

AudioLM and VALL-E follow this route. VALL-E reframes TTS as conditional language modeling over codec tokens. Because residual quantization is hierarchical — coarse structure first, detail after — it splits the work: the first quantizer level is predicted autoregressively, the remaining levels non-autoregressively in bulk.

Two things make this attractive. Nobody has to hand-design the mel pipeline or the phase reconstruction. And the entire language-model toolbox — sampling temperature, top-p, prompting — carries over unchanged. The temperature figure above stops being an illustration and becomes a knob you actually turn: too cold and the delivery is flat and monotone; too hot and you get mispronunciations, repetition, and voices that drift into someone else mid-sentence.

Zero-shot voices: becoming someone in a few seconds

Cloning a voice used to require hours of recordings and a fine-tuning run per speaker. Getting there from a few seconds of reference audio works in two broad ways.

(a) Speaker embeddings. Train an encoder on speaker verification — the task of deciding whether two clips are the same person — then use it to crush the reference clip into a single vector (a d-vector or x-vector) and feed that to the synthesizer as a condition. Verification training pushes same-speaker clips together and different-speaker clips apart, so in the resulting space direction is timbre. Similarity is measured by cosine.

(b) Audio prompting. The VALL-E approach: convert the reference clip into codec tokens and place them at the front of the sequence you are about to generate. The model just continues the sequence — and because it is a continuation, the timbre and speaking style come along. Same logic as few-shot prompting in a text LLM, with no dedicated speaker encoder needed.

The catch with (b) is that it inherits the recording's quirks too — room reverb, microphone coloration, background noise. That is a feature when you want the reference environment reproduced, and a liability when someone hands you a bad clip.

Evaluation: what counts as good

Keep three axes separate.

• Naturalness: MOS, a subjective score collected from many listeners on a five-point scale; CMOS when you are comparing two systems head to head. In the end only ears settle this.
• Intelligibility: run the synthesized audio through a speech recognizer and measure WER/CER. Mispronunciations, skips, and repetitions surface here as a number.
• Speaker similarity: cosine similarity between the reference and synthesized speaker vectors.

Speed is reported as RTF (real-time factor: seconds of compute per second of audio produced). Below 1.0 means you can keep up with playback.

Handle the numbers carefully. MOS shifts with the listener pool, the presentation order, and the playback hardware, so a MOS from one paper means nothing next to a MOS from another. Only relative comparisons inside a single listening test carry weight.

How this shows up on the job

Who touches it, and when. Teams building voice assistants, read-aloud apps, automated narration for games and video, and accessibility features like screen readers. Most of them never train a model: they start from an ESPnet, Coqui TTS, or NVIDIA NeMo recipe, or from a cloud TTS API. You drop down to training your own when the language or dialect isn't covered, when the audio can't leave your infrastructure, or when a specific voice has to be maintained for years.

Parameters you will actually set.

• n_fft / hop_length / n_mels / fmin / fmax and the mel normalization scheme — these must match exactly between the acoustic model and the vocoder
• duration scale (speaking rate) and pitch / energy scales (FastSpeech 2-style prosody control)
• temperature / top_p for token-LM systems
• the vocoder's sampling_rate, plus loudness normalization on the output stage

Traps that turn into incidents.

1. Mel configuration mismatch. The acoustic model was trained with hop=256, the vocoder with hop=300. Nothing errors out, synthesis runs to completion, and the only symptom is audio that sounds tinny or plays too fast or too slow. This is the classic failure when you assemble a two-stage pipeline yourself.
2. The gap between training mels and predicted mels. Vocoders are normally trained on mels computed from real recordings, but in production they receive mels predicted by the acoustic model, which are slightly blurrier. That difference makes the vocoder emit noise. The fix is to fine-tune the vocoder on the acoustic model's predicted mels — and it is one of the reasons end-to-end models like VITS are attractive.
3. Holes in text normalization. Numbers, units, symbols, mixed-script strings, proper nouns. Independent of model quality, this is where "2026" comes out as "two zero two six." Log both the input text and the resulting phoneme string, or you cannot tell a model problem from a G2P problem.
4. Autoregressive models breaking on long input. Tacotron-style attention slips on long text and falls into repetition or non-termination. The standard mitigation is to synthesize sentence by sentence and stitch with short silences.
5. Level and DC offset. Rounding vocoder output straight to 16-bit integers clips and distorts. Always run loudness normalization.
6. Rights to a voice. Zero-shot cloning reproduces someone's voice from seconds of audio. Do not synthesize a real person's voice without their consent and agreement on the use. Disclosure that audio is synthetic, and watermarking, are increasingly expected. Design on the assumption that what is technically possible and what is acceptable are two different sets.

A question that comes up in interviews and design reviews. "Why not just regress the waveform directly?" The answer runs through reason three: a clip shifted in phase sounds identical but is a completely different list of numbers, so a per-sample loss doesn't track perception. That's why the pipeline routes through a mel spectrogram with phase discarded, and delegates phase reconstruction alone to a specialist generative model — the vocoder. Getting that far shows you understand why the split exists rather than just that it does.

Wrapping up

• Speech synthesis is the job of inventing a number sequence tens of thousands of times longer than the text it starts from
• Naive regression fails for two reasons: there is no single right answer (averaging muffles), and phase (irrelevant to the ear, catastrophic to the numbers)
• The standard answer is the two-stage split: text → mel spectrogram (a language problem) → waveform (a signal problem)
• The modern, parallelized form uses a duration predictor for timing and a GAN or diffusion vocoder for the waveform
• Tokenize audio and TTS becomes a language-modeling problem, where seconds of reference audio are enough to carry a voice
• Most incidents live not inside the models but in the configuration mismatch between the parts