Speech Recognition from Scratch — From Waveform to Text

How a stream of numbers from a microphone becomes words, starting from zero: spectrogram features, the alignment problem that CTC solved, autoregressive encoder-decoder models, and Whisper — in the order history solved them.

Turning trembling air into letters

All a microphone does is measure the rise and fall of air pressure at fixed intervals and write down numbers. For speech recognition it measures 16,000 times per second, so three seconds of speech is a ribbon of 48,000 numbers.

Speech recognition is the job of pulling five words out of those 48,000 numbers, and two things make it hard. First, time stretches and shrinks: the same phrase takes one second rushed and three seconds drawled — same letters, three times the numbers. Second, there are no boundaries. Written language puts spaces between words; nowhere in the waveform is there a line.

Don't look at the waveform

Feeding raw samples to a model is a bad idea. When you tell the vowel in "beat" from the vowel in "bat," your ear isn't using instantaneous air pressure — it's using which frequencies are present and in what proportion. A low voice and a high voice both produce a recognizable "ah" because the pattern of the mixture is similar, even though the raw numbers look nothing alike.

So we convert the waveform into how that frequency mixture changes over time. That's a spectrogram, and the recipe is just "take a short window, measure the frequencies inside it, slide the window a little, repeat" — the short-time Fourier transform (STFT).

$$X[t, k] = \sum_{n=0}^{N-1} x[n + tH]\; w[n]\; e^{-2\pi i k n / N}$$

$x$ is the waveform, $N$ the window length, $H$ the hop, $w$ a weighting that tapers the window's edges, $t$ the window index, $k$ the frequency index. All it says is: build a table whose entry $(t,k)$ is "how much of pitch $k$ is present in short segment $t$." In words, the sum slides a short window along the recording and asks one question per pitch — how strongly does this slice rise and fall at that particular rate? — then slides on and asks again. Why decomposing into frequencies needs only additions and multiplications is covered in The FFT from Scratch.

The standard settings: for 16,000 Hz audio, a 25-millisecond window (400 samples) hopped by 10 milliseconds (160 samples). One second becomes 100 frames.

Two more touches. A mel filterbank warps the frequency axis to match hearing — 100 Hz and 200 Hz sound like different notes, while 7,000 Hz and 7,100 Hz sound identical.

$$m = 2595 \log_{10}\!\left(1 + \frac{f}{700}\right)$$

$f$ is ordinary frequency in Hz, $m$ the warped scale: fine distinctions survive at low frequencies and blur at high ones, and along it we bundle roughly 200 frequency bins into 80. In words, the formula is a conversion table from the physics of hertz to the scale a human ear actually reports: down at the bottom a small change in $f$ moves $m$ a lot, while up at the top a large change in $f$ barely moves $m$ at all. The other touch is the logarithm: a whisper and a shout differ by thousands, so we compress the range — which also matches the roughly logarithmic way people perceive loudness.

The result, a table of time × 80 channels of log-mel energy, is what a modern recognizer actually sees. The waveform never appears again.

A historical note: through the early 2010s the dominant feature was MFCC, which applies a discrete cosine transform to the log-mel above and squeezes it to about 13 dimensions — because the acoustic models of the day assumed the feature dimensions were uncorrelated, making a decorrelating step mandatory. Neural networks eat correlated inputs happily, so the step disappeared. What counts as sensible preprocessing is dictated by whatever model sits downstream.

The old way: assemble three components

Before deep learning, the starting point was Bayes' rule.

$$\hat{W} = \arg\max_{W} \; P(X \mid W)\, P(W)$$

$X$ is the acoustic features, $W$ a word sequence. It says: pick the candidate that maximizes "how likely this audio is if those were the words" times "how likely those words are in the first place." In words: instead of guessing words straight out of the audio, you propose word sequences, score each one backwards by how plausibly it would have produced this sound, and let $P(W)$ penalize the sequences nobody would ever say.

Implementing it took three pieces: an acoustic model scoring how well each frame matches each phone state (GMM-HMM for many years), a pronunciation lexicon mapping words to phone sequences, and a language model scoring how natural a word sequence is. A decoder composed all three into one enormous search graph and hunted for the best path. Each part was trained separately, so nothing was globally optimal, and the lexicon was hand-written, so every new language and proper noun meant more manual entries. When the acoustic model became a neural network during the 2010s, accuracy jumped — but the three-component structure survived.

The alignment problem

Say we want a single network that takes audio and emits text. Time-stretching is the wall. You have 300 frames and the ground truth hello — five characters. Training wants to know that frames 1–60 are h, frames 61–95 are e, and so on. Nobody annotates that. All the dataset gives you is the transcript.

CTC: refusing to answer "where"

In 2006, CTC (Connectionist Temporal Classification, Graves et al.) sidestepped this with a direct idea: don't pick one alignment — sum over all of them.

Three ingredients. (1) Add a blank symbol $\varepsilon$ meaning "emit nothing here." (2) The model emits one symbol per frame, producing a length-300 sequence called a path. (3) A collapsing rule turns a path into a string: merge runs of identical symbols, then delete the $\varepsilon$s.

For example h h ε e e l l ε l o o merges to h ε e l ε l o and collapses to hello. The $\varepsilon$ wedged between the two ls is doing real work — without it the merge step would leave a single l. So: to emit a doubled letter, put a blank between the halves.

Countless paths collapse to hello, and CTC defines the probability of the correct answer as the sum over all of them.

$y$ is the target string, $x$ the input audio, $\pi$ a single path, $B$ the collapsing rule, $B^{-1}(y)$ every path that collapses to $y$, $T$ the frame count, $p_t$ the per-frame probability. It says: take every path that collapses to the right answer, multiply out each path's per-frame probabilities, and add them all up. In words, equation (1) is a device for extracting the probability of landing on $y$ without ever answering which frame produced which letter. The disagreement between alignments dissolves into the sum.

Equation (1) has a catch — the number of paths grows exponentially with length, so enumeration is impossible. Dynamic programming rescues it: record the running sum for each prefix and fill a table cell by cell, and the total over exponentially many paths falls out in $O(T \times U)$. This forward algorithm is the heart of every CTC implementation.

CTC carries a weakness that follows from its design. The right-hand side of equation (1) is a product of per-frame probabilities, meaning each frame's output is decided independently of every other one (conditional independence). Dependencies between outputs — spelling, grammar — cannot live inside the model. That's exactly why real CTC systems ship with an external language model and beam search. In return, paths never move backward, so monotonicity is guaranteed and frames can be processed as they arrive — which is why CTC suits low-latency streaming.

Encoder-decoder: "reading the audio aloud"

Around 2016 a different angle appeared, exemplified by Listen, Attend and Spell (LAS). The idea is borrowed from machine translation: an encoder summarizes the audio, and a decoder generates one character at a time, using attention at each step to choose where in the audio to look.

This plugs CTC's hole. The decoder receives its own previous output, so dependencies between outputs are back — the language model moved inside the network instead of hanging off the side. No lexicon, no phones.

The price is clear. Attention may look anywhere, so monotonicity is gone: the model can repeat a phrase, skip ahead and drop words, or emit words that were never spoken (hallucination). And it generally has to hear everything before it starts, so streaming takes extra machinery.

The compromise is the RNN-Transducer (RNN-T): CTC's monotonicity plus a prediction network conditioned on the previous output. It's what powers products where text appears as you speak, like phone dictation.

Transformers, scale, and Whisper

After that, the RNNs became Transformers and the data grew by orders of magnitude. Whisper (2022) is almost aggressively plain in structure: 80-channel log-mel → Transformer encoder → autoregressive decoder, exactly the encoder-decoder above. The novelty isn't architectural — it's 680,000 hours of weakly supervised data plus a multitask format where the task (which language, transcribe or translate, timestamps or not) is specified by tokens fed to the decoder. That paper is read closely in Paper Review: Why Whisper Is Robust.

Look back and watch the parts vanish: the lexicon is gone, phones are gone, the bolt-on language model is gone for most use cases. What remains is feature extraction and one network.

Decoding: picking the likeliest symbol isn't enough

CTC's simplest extraction is greedy decoding — take the per-frame winner, then collapse. It's fast but not optimal under equation (1), because "the path of per-frame winners" and "the string with the highest total probability after collapsing" are different objects. Practical systems run beam search over several live candidates.

Autoregressive decoders have a different failure: repetition loops. Once a phrase has been emitted twice, "the previous output" is that phrase, which makes emitting it again attractive. One remedy is temperature: with softmax as $\exp(z_i / T)$, small $T$ sharpens the distribution onto the single top candidate and larger $T$ flattens it, leaving room for a different choice.

Writing the code

The front end fits in a few lines of numpy.

import numpy as np

N, H = 400, 160                                             # 25ms window / 10ms hop at 16kHz
frames = np.stack([x[i:i+N] * np.hanning(N)
                   for i in range(0, len(x) - N, H)])       # (T, 400)
power  = np.abs(np.fft.rfft(frames, n=N))**2                # (T, 201) per-window spectrum
logmel = np.log(power @ mel_fb.T + 1e-10)                   # mel_fb: (80, 201) -> (T, 80)

Drop the + 1e-10 and silent stretches give you log(0) = -inf, poisoning everything downstream — the kind of line that never appears in the formula and is never optional in the implementation. Greedy CTC decoding, meanwhile, is just the collapsing rule transcribed:

def ctc_greedy(logits, blank=0):
    ids, out, prev = logits.argmax(axis=-1), [], -1
    for k in ids:
        if k != prev and k != blank:    # (a) merge repeats, (b) drop blanks
            out.append(k)
        prev = k
    return out

Testing k != prev before the blank check is what makes it correct: h ε h yields hh while h h yields h. The whole doubled-letter contract lives in those two lines.

Measuring accuracy

The standard metric is WER (word error rate).

$$\mathrm{WER} = \frac{S + D + I}{N}$$

$S$ is substitutions, $D$ deletions, $I$ insertions, $N$ the number of words in the reference. It is the minimum number of edits needed to turn the hypothesis into the reference, divided by the reference length. That minimum is edit distance, computed with the same dynamic-programming table as before. In words: mark up the transcript in red pen, count the corrections, and divide by the length of the reference. Zero percent is word-for-word agreement, and lower is better. Because insertions can push the numerator past $N$, WER can exceed 100%.

One caveat: WER counts formatting differences as errors. "2026" versus "twenty twenty-six," present or absent punctuation, casing — the number moves depending on what normalization ran first.

How this shows up on the job

The people who wire speech recognition into things build meeting minutes, captions, call-center analytics, voice assistants, and video search.

It starts with sample rate. 16 kHz is standard, and it can only capture content up to 8 kHz (the sampling theorem). Upsampling 8 kHz telephone audio to 16 kHz does not bring back the missing highs. Fricatives (s, sh, f) carry their information up there, so extra s-related errors on phone audio are structural, not a bug. Downsampling a 44.1 kHz recording to 16 kHz mono is correct preprocessing — ffmpeg -i in.wav -ar 16000 -ac 1 out.wav is the usual incantation.

Streaming versus batch decides your model family. If text must appear while the person is still talking, you want the monotonic families, CTC or RNN-T. For minutes produced after the meeting ends, encoder-decoder gives better accuracy. Changing this later changes the architecture.

Long audio gets chunked. Whisper-family models are trained on 30-second windows, so long recordings are always split, and a seam landing mid-word damages that word on both sides. Run VAD (voice activity detection) first and cut at silences and most of that class of bug disappears.

Proper nouns aren't in the training data. Company names, product names, internal jargon — the model has never seen them. With CTC you lift their scores using an external language model or hotwords; with encoder-decoder you put the terms in a context prompt (initial_prompt in openai-whisper). Both bend the vocabulary without retraining.

Traps. (1) Hallucination over silence or music — dropping those regions with VAD first is the reliable fix. (2) Comparing two models without matching normalizers — you lose the ability to tell whether the model or the text cleanup moved the number. (3) Resampling and re-encoding repeatedly — degradation accumulates, so keep the original.

The question that comes up in design reviews is "what's the difference between CTC and encoder-decoder?" CTC's outputs are conditionally independent and its alignment is monotonic, so it streams well but needs an external language model for linguistic dependencies. Encoder-decoder is autoregressive, so it carries a language model internally and tends to be more accurate, but it loses monotonicity and gains repetition, deletion, and hallucination as failure modes. RNN-T is the compromise. It isn't about which is better — it's which of latency, accuracy, and failure mode you're willing to trade.

Takeaways

• Never use the raw waveform. STFT → mel filterbank → log gives you a time × 80-channel table
• The central difficulty is alignment: frame count and character count don't match, and nobody labels the correspondence
• CTC dodges it with a blank symbol and a collapsing rule, summing over every possible alignment. Monotonic and fast, but it cannot model dependencies between its own outputs
• Encoder-decoder handles output dependencies with an autoregressive decoder, at the cost of monotonicity — gaining repetition and hallucination as failure modes. Whisper is that approach pushed through with enormous data: no lexicon, no phones, not anymore