Learn › AI

Audio & Speech

ASR, TTS, speech LLMs

01 ·Audio & Speech·FREE·PAPER·8 min read Paper Deep-Dive: Why Whisper Is Robust — Large-Scale Weak Supervision A reading of the Whisper paper (Radford et al., 2022) built strictly from its own text. Waveform to log-Mel spectrogram, an encoder-decoder that emits text directly, and how 680,000 hours produced robustness without fine-tuning — with the numbers the paper actually measured and the limits it admits to. 02 ·Audio & Speech·FREE·11 min read Speech Recognition from Scratch — From Waveform to Text How a stream of numbers from a microphone becomes words, starting from zero: spectrogram features, the alignment problem that CTC solved, autoregressive encoder-decoder models, and Whisper — in the order history solved them. 03 ·Audio & Speech·FREE·PAPER·12 min read Speech Synthesis from Scratch — From Text to a Voice Speech synthesis invents a waveform tens of thousands of times longer than the handful of characters it starts from. This walks through why naive regression fails (one-to-many and phase), why text → mel spectrogram → waveform became the standard split, and how a few seconds of reference audio is now enough to carry a voice. 04 ·Audio & Speech·FREE·PAPER·14 min read Representing Sound — Mel Spectrograms and Audio Tokens Why speech models never eat raw waveforms, and what they eat instead: the chain from short-time Fourier transform to the mel scale to the log to discrete tokens. Covers the window-length tradeoff, why MFCCs dropped the DCT, how acoustic and semantic tokens differ, and the config mismatches that silently wreck audio in production. 05 ·Audio & Speech·★ MEMBER·PAPER·10 min read Paper Walkthrough: SwanTale — Designing Voices from Words Alone, with Speech and Sound in One Waveform ByteDance's SwanTale handles two jobs in one model: cloning a voice from reference audio (zero-shot) and building voices, ambience, and sound effects purely from a natural-language caption (instruct). We walk through the paper, from the 70M-caption data pipeline to Unified MoE routing and GRPO post-training. 06 ·Audio & Speech·★ MEMBER·PAPER·9 min read Paper Walkthrough: Interpretable MEG Decoding of Perceived Speech — Reading the Decoder's Weights as a Brain Map From three seconds of magnetoencephalography, this decoder picks the audio a listener was hearing out of 1005 candidates with 39.75% Top-1 accuracy — and, unlike its predecessors, its trained weights translate into cortical locations and rhythms, while paired occlusion experiments reveal which speech features it actually relies on. 07 ·Audio & Speech·★ MEMBER·PAPER·14 min read Paper Walkthrough: VoiceMem — A Left Brain and a Right Brain for Voice Agents, at Zero Added Latency A from-scratch walkthrough of VoiceMem, a memory system for real-time speech interaction. A factual 'left brain' and an affective 'right brain' run in parallel, and the whole retrieval is hidden inside the silence a VAD already waits out — which is how it wins at a top-5 budget. 08 ·Audio & Speech·★ MEMBER·PAPER·12 min read Paper Walkthrough: Motion-Omni — Speaking and Moving in One Forward Pass A walkthrough of Motion-Omni, a framework where a spoken dialogue model emits its spoken reply and its full-body gestures in the same pass. We follow the design that conditions motion on hidden states instead of waveforms, what it cost to train, and what the paper admits it has not shown. 09 ·Audio & Speech·★ MEMBER·PAPER·9 min read Paper Walkthrough: AuK — Folding Speech Generation and Editing Into One Model AuK is an open-source foundation model that puts speech generation, editing, enhancement and separation behind one door: a natural-language instruction plus optional audio. A ground-up walkthrough of its data design, MMDiT backbone, rectified flow, human-preference and GRPO post-training, and four-step distillation. 10 ·Audio & Speech·★ MEMBER·PAPER·10 min read Paper Walkthrough: X-AuT — Pull Audio-Encoder Layers Out Two at a Time, Then Backfill from a Bigger Teacher Delete whole layers from a speech LLM's audio encoder and the decoder starts guessing 'end of sentence' too early, dropping characters. X-AuT picks which layers to remove with short trial runs, then backfills in three stages: distillation from a larger teacher, supervision on the student's own decoding context, and LoRA finetuning. Qwen3-ASR-0.6B goes from 18 to 16 layers with macro-average error falling 5.61% → 5.27%, and 14 layers land at 5.75% — single-run numbers, read here through the equations and the recipe. 11 ·Audio & Speech·★ MEMBER·PAPER·9 min read Paper Walkthrough: Dynamic Band Splitting — Letting Each Mixture Decide Where the Frequency Bands Go Band-split source separators carve the spectrum into hand-designed frequency bands and use the same carving for every input. DBS keeps the number of bands fixed but predicts where the boundaries go, mixture by mixture. We walk through boundary scoring, hard Top-K, weighted pooling and full-resolution reconstruction, then read the results on three datasets and the limits.