Paper Walkthrough: AuK — Folding Speech Generation and Editing Into One Model

AuK is an open-source foundation model that puts speech generation, editing, enhancement and separation behind one door: a natural-language instruction plus optional audio. A ground-up walkthrough of its data design, MMDiT backbone, rectified flow, human-preference and GRPO post-training, and four-step distillation.

AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing

Making speech and fixing speech used to be different software

The paper's title is "AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing" (arXiv:2609.08936, published 2026-09-08).

Here is the abstract in plain terms. AuK unifies speech generation and editing behind a common interface — a natural-language instruction plus optional audio context. To support that breadth, the authors build roughly 3.03 billion instruction–audio instances and 1.95 million hours of effective supervision across five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. The model has three parts: a multimodal large language model (MLLM) for semantic conditioning, a VAE trained jointly on speech, general audio and music for acoustic conditioning, and a rectified-flow Transformer that runs dual-stream MMDiT blocks followed by unified single-stream DiT blocks. Training moves from a generation-only warm-up to joint generation–editing pre-training, then splits: human-feedback preference optimization for open-ended editing, reward-based reinforcement learning for generation. Distilling the result with consistency initialization and task-routed Decoupled DMD yields AuK-Flash, which runs four steps without classifier-free guidance (CFG) and is 4.5× faster in wall-clock time than the full model under matched conditions.

Why unification is hard

Users ask for these things in one breath: read this in that voice, swap out this one word, make it angrier, pull out just this speaker. Splitting them across task-specific models fragments the experience and duplicates modeling effort — so put one door in front of all of them (§1).

The paper is candid about the difficulty. The output constraints differ fundamentally: generation creates new speech, content editing changes only selected regions, paralinguistic and acoustic editing must preserve linguistic content, and enhancement or separation must keep only the components the instruction names. And evaluation is heterogeneous: generation has automatic yardsticks in recognition accuracy and speaker similarity, while open-ended editing depends on subjective judgment.

Data: five families, one shape

Every example in all five families has the same shape — instruction, optional input audio, target waveform (§2). Acoustic editing is built deterministically with signal processing across five speaking-rate multipliers, six loudness offsets and six pitch shifts; paralinguistic editing covers eight emotions, timbre, de-accenting, 39 nonverbal event types and whisper-style conversion.

The zero-shot TTS construction is the clever part (§2.1). The usual recipe splits one recording into prompt and target. AuK instead pairs distinct utterances from the same speaker, yielding n×(n−1) instances from n utterances. Because the two recordings differ in content and conditions, the model is pushed to separate speaker-invariant traits from utterance-specific variation — and the prompt transcript is never needed, so deployment no longer depends on a separate ASR system.

Putting a number on "the same voice"

Generation is judged on whether the words are right and whether the voice matches. That second one, SIM, runs the reference and the generated audio through a speaker encoder $\psi$ and takes the cosine between the resulting vectors. The paper reuses the same form as a reinforcement-learning reward (§4.3.2).

Here $y_i$ is a generated candidate, $y_{\text{ref}}$ the reference audio, and $\psi(\cdot)$ a speaker encoder that summarizes a recording as one vector. Equation (1) in words: turn each recording into an arrow and check how closely the two arrows point the same way. The value lands between −1 and 1, and closer to 1 means "judged to be the same person." Differences in recording level show up in vector length, so looking only at direction is exactly what you want.