Audio & Speech
ASR, TTS, speech LLMs
01
·Audio & Speech·★ MEMBER·PAPER·8 min read
Paper Deep-Dive: Why Whisper Is Robust — Large-Scale Weak Supervision
A reading of the Whisper paper (Radford et al., 2022) built strictly from its own text. Waveform to log-Mel spectrogram, an encoder-decoder that emits text directly, and how 680,000 hours produced robustness without fine-tuning — with the numbers the paper actually measured and the limits it admits to.
02
·Audio & Speech·FREE·11 min read
Speech Recognition from Scratch — From Waveform to Text
How a stream of numbers from a microphone becomes words, starting from zero: spectrogram features, the alignment problem that CTC solved, autoregressive encoder-decoder models, and Whisper — in the order history solved them.
03
·Audio & Speech·★ MEMBER·PAPER·12 min read
Speech Synthesis from Scratch — From Text to a Voice
Speech synthesis invents a waveform tens of thousands of times longer than the handful of characters it starts from. This walks through why naive regression fails (one-to-many and phase), why text → mel spectrogram → waveform became the standard split, and how a few seconds of reference audio is now enough to carry a voice.
04
·Audio & Speech·★ MEMBER·PAPER·13 min read
Representing Sound — Mel Spectrograms and Audio Tokens
Why speech models never eat raw waveforms, and what they eat instead: the chain from short-time Fourier transform to the mel scale to the log to discrete tokens. Covers the window-length tradeoff, why MFCCs dropped the DCT, how acoustic and semantic tokens differ, and the config mismatches that silently wreck audio in production.
05
·Audio & Speech·★ MEMBER·PAPER·10 min read
Paper Walkthrough: SwanTale — Designing Voices from Words Alone, with Speech and Sound in One Waveform
ByteDance's SwanTale handles two jobs in one model: cloning a voice from reference audio (zero-shot) and building voices, ambience, and sound effects purely from a natural-language caption (instruct). We walk through the paper, from the 70M-caption data pipeline to Unified MoE routing and GRPO post-training.
06
·Audio & Speech·★ MEMBER·PAPER·9 min read
Paper Walkthrough: Interpretable MEG Decoding of Perceived Speech — Reading the Decoder's Weights as a Brain Map
From three seconds of magnetoencephalography, this decoder picks the audio a listener was hearing out of 1005 candidates with 39.75% Top-1 accuracy — and, unlike its predecessors, its trained weights translate into cortical locations and rhythms, while paired occlusion experiments reveal which speech features it actually relies on.
07
·Audio & Speech·★ MEMBER·PAPER·14 min read
Paper Walkthrough: VoiceMem — A Left Brain and a Right Brain for Voice Agents, at Zero Added Latency
A from-scratch walkthrough of VoiceMem, a memory system for real-time speech interaction. A factual 'left brain' and an affective 'right brain' run in parallel, and the whole retrieval is hidden inside the silence a VAD already waits out — which is how it wins at a top-5 budget.
08
·Audio & Speech·★ MEMBER·PAPER·12 min read
Paper Walkthrough: Motion-Omni — Speaking and Moving in One Forward Pass
A walkthrough of Motion-Omni, a framework where a spoken dialogue model emits its spoken reply and its full-body gestures in the same pass. We follow the design that conditions motion on hidden states instead of waveforms, what it cost to train, and what the paper admits it has not shown.
09
·Audio & Speech·★ MEMBER·PAPER·9 min read
Paper Explained: Last Translation Benchmark — Measuring Translation with Breaking Examples and Verification Rules
Machine translation benchmarks are saturating, and neither automatic metrics nor human evaluation can be fully trusted. The response: collect human-written examples that break frontier models, and attach handcrafted verification rules to each one. A ground-up reading of Last Translation Benchmark.