#audio
18 articles
01
·Audio & Speech·★ MEMBER·PAPER·12 min read
Paper Walkthrough: Motion-Omni — Speaking and Moving in One Forward Pass
A walkthrough of Motion-Omni, a framework where a spoken dialogue model emits its spoken reply and its full-body gestures in the same pass. We follow the design that conditions motion on hidden states instead of waveforms, what it cost to train, and what the paper admits it has not shown.
02
·Audio & Speech·★ MEMBER·PAPER·9 min read
Paper Explained: Last Translation Benchmark — Measuring Translation with Breaking Examples and Verification Rules
Machine translation benchmarks are saturating, and neither automatic metrics nor human evaluation can be fully trusted. The response: collect human-written examples that break frontier models, and attach handcrafted verification rules to each one. A ground-up reading of Last Translation Benchmark.
03
·★ MEMBER·PAPER·15 min read
Paper Walkthrough — The Missing Temporal Link: Putting Script Timing on the Video–Audio Axis with Temporal Context Routing
Video and audio stay in sync with each other while both drift off the script's timeline. This walkthrough follows the paper's own equations and numbers to show how TCR closes that third, missing temporal link with a single additive term in the cross-attention logits.
04
·Agents·★ MEMBER·PAPER·10 min read
Paper Walkthrough: SecOPD — Grading One Token at a Time to Cut Adaptive Prompt Injection by an Order of Magnitude
Defensively fine-tuned LLMs still fall to adaptive prompt injections at close to 100%. The culprit is training that hands the whole response a single score. SecOPD grades every token using a teacher that never saw the injection — explained from first principles.
05
·★ MEMBER·PAPER·14 min read
Paper Walkthrough: DreamX-Creator — Making Sound and Picture Together in 7B, Then Finishing at 2K in One Step
A ground-up walkthrough of a 7B model that denoises audio and video inside one generative process: the gated cross-modal attention, the modality-aware reinforcement learning, the one-step 2K refiner — and the unusually heavy caveats the authors put on their own results.
06
·Audio & Speech·★ MEMBER·PAPER·14 min read
Paper Walkthrough: VoiceMem — A Left Brain and a Right Brain for Voice Agents, at Zero Added Latency
A from-scratch walkthrough of VoiceMem, a memory system for real-time speech interaction. A factual 'left brain' and an affective 'right brain' run in parallel, and the whole retrieval is hidden inside the silence a VAD already waits out — which is how it wins at a top-5 budget.
07
·★ MEMBER·PAPER·11 min read
Paper Walkthrough: TLive-Omni — An Omni-Modal Model That Watches, Listens, and Tells You the Timestamp
A from-scratch walkthrough of TLive-Omni, an omni-modal understanding model built for e-commerce live streaming: Per-vGrid timestamped audio-video interleaving, a three-stage SFT recipe, and Faithful-RFT, which rewards answer faithfulness instead of visible reasoning — grounded strictly in the paper's own numbers.
08
·★ MEMBER·PAPER·14 min read
Paper Walkthrough: EchoWM — A World You Can Walk Into, and Hear, Driven by Camera Intent
Hand it a reference view and a stream of 'where I want to go', and it keeps generating 720p video together with environmental sound, music and speech. A walkthrough of EchoWM from the paper itself: one camera-intent interface for both first and third person, a single translation scale shared across the whole dataset, and the sink-plus-FIFO cache that makes long rollouts possible.
09
·Audio & Speech·★ MEMBER·PAPER·13 min read
Representing Sound — Mel Spectrograms and Audio Tokens
Why speech models never eat raw waveforms, and what they eat instead: the chain from short-time Fourier transform to the mel scale to the log to discrete tokens. Covers the window-length tradeoff, why MFCCs dropped the DCT, how acoustic and semantic tokens differ, and the config mismatches that silently wreck audio in production.
10
·Audio Codecs·★ MEMBER·PAPER·11 min read
Why Opus Won — The Design of a Modern Audio Codec
Audio codecs used to split cleanly into speech codecs and music codecs. Opus ended the split by stacking two engines, SILK and CELT, inside a single bitstream. This piece works up from zero: how linear prediction and the MDCT divide the spectrum between them, why CELT refuses to let band energy collapse, what actually sets a codec's latency, and the technical and institutional reasons Opus became WebRTC's mandatory codec.
11
·Audio Codecs·★ MEMBER·10 min read
MP3 and Psychoacoustics — The Science of Sounds You Cannot Hear
MP3 does not throw away sounds that aren't there — it throws away sounds that are there and inaudible. This piece works through the three holes in human hearing (absolute threshold, simultaneous masking, temporal masking) down to the equations, then follows SMR into bit allocation, the encoder's two nested loops, and the bit reservoir.
12
·Agents·★ MEMBER·PAPER·11 min read
Paper Walkthrough: OmniScientist — An AI Scientist That Actually Looks at the Raw Data
Automating every step of the research workflow doesn't help much if the only thing the agent ever sees is somebody else's summary table. OmniScientist puts raw waveforms, images and 3-D point clouds in front of the agent for the whole research lifecycle and enforces the gates in Python — walked through from the paper itself, up to the 21.7% finding buried in a seismic benchmark.
13
·Audio & Speech·★ MEMBER·PAPER·10 min read
Paper Walkthrough: SwanTale — Designing Voices from Words Alone, with Speech and Sound in One Waveform
ByteDance's SwanTale handles two jobs in one model: cloning a voice from reference audio (zero-shot) and building voices, ambience, and sound effects purely from a natural-language caption (instruct). We walk through the paper, from the 70M-caption data pipeline to Unified MoE routing and GRPO post-training.
14
·Audio & Speech·★ MEMBER·PAPER·9 min read
Paper Walkthrough: Interpretable MEG Decoding of Perceived Speech — Reading the Decoder's Weights as a Brain Map
From three seconds of magnetoencephalography, this decoder picks the audio a listener was hearing out of 1005 candidates with 39.75% Top-1 accuracy — and, unlike its predecessors, its trained weights translate into cortical locations and rhythms, while paired occlusion experiments reveal which speech features it actually relies on.
15
·★ MEMBER·PAPER·9 min read
Paper Walkthrough: AURORA-LM — A Diffusion Language Model That Keeps Its Text Latent Full-Width Instead of Compressing It
A walkthrough of AURORA-LM, a continuous-latent diffusion language model that rejects the usual compromise of compressing text latents to make diffusion easier. It keeps a decodable, width-1024 latent and learns its distribution with a low-rank input pathway, width-calibrated noise, and self-trajectory consistency.
16
·Audio Codecs·★ MEMBER·8 min read
Audio Compression from Scratch — The Art of Discarding What You Cannot Hear
How does MP3 sound 'the same' at less than a tenth of the size? We build up audio codecs from zero — auditory masking, the MDCT, and bit allocation — and finish with when to reach for MP3, AAC, or Opus.
17
·Audio Codecs·★ MEMBER·8 min read
Audio Compression from Scratch — The Art of Discarding What You Cannot Hear
How does MP3 sound 'the same' at less than a tenth of the size? We build up audio codecs from zero — auditory masking, the MDCT, and bit allocation — and finish with when to reach for MP3, AAC, or Opus.
18
·Audio & Speech·★ MEMBER·PAPER·8 min read
Paper Deep-Dive: Why Whisper Is Robust — Large-Scale Weak Supervision
A reading of the Whisper paper (Radford et al., 2022) built strictly from its own text. Waveform to log-Mel spectrogram, an encoder-decoder that emits text directly, and how 680,000 hours produced robustness without fine-tuning — with the numbers the paper actually measured and the limits it admits to.