Paper Walkthrough: SwanTale — Designing Voices from Words Alone, with Speech and Sound in One Waveform

ByteDance's SwanTale handles two jobs in one model: cloning a voice from reference audio (zero-shot) and building voices, ambience, and sound effects purely from a natural-language caption (instruct). We walk through the paper, from the 70M-caption data pipeline to Unified MoE routing and GRPO post-training.

SwanTale: Unified Multi-Speaker Speech and Audio Generation for Instruct and Zero-Shot Tasks

The recording session with no voice actor

Animation dubbing, audio dramas, advertising, games, podcasts. In real media production, there often is no reference recording of "the voice we want." As the paper puts it, creation frequently starts from the opposite end (§1): you design a voice from scratch to fit a character, describe in words how a line should be performed, and place that line inside a soundscape — rain on the eaves, a murmuring crowd. And once a voice is designed, you want to reuse it later.

Classical zero-shot TTS — give the model a few seconds of reference audio and it speaks new content in that voice — only covers the case where the voice already exists. Instruct TTS, which builds voices from natural-language descriptions, does exist, but the paper points out that most such systems generate speech only. If ambience and sound effects come from a separate downstream pipeline, each piece can sound plausible on its own while its timing, loudness, and reverberation drift away from the speech (§1).

SwanTale (a ByteDance technical report) is a multi-speaker speech-and-audio generation model that handles both tasks in one network: zero-shot cloning from reference audio, and instruct generation where a caption alone produces the voices and the acoustic scene together.

Two tasks, three walls

The paper frames the inputs like this (§1):

• Instruct task: a caption describing the environment (location, reverberation, persistent background sound), speaker styles (not just gender and age but persona, timbre, habitual delivery), and fine-grained content (emotion shifts, emphasis, local sound effects)
• Zero-shot task: reference audio plus the same fine-grained content caption

Three challenges stand in the way (§1). (1) Data scarcity — caption data at this granularity barely exists. (2) Task compatibility — the two tasks feed speaker identity through different doors (text vs. reference audio), and joint training must keep them from weakening each other. (3) Multi-audio-modality complexity — a single waveform mixes speech, ambience, transient effects, sometimes singing and music, each with a completely different temporal structure (speech must align with the words, ambience should stay stable, effects are momentary, singing must stay in tune).

One concept worth loading before the deep dive: the lifeline of zero-shot generation is "does the voice match?", and voice similarity is conventionally measured as the cosine similarity — the closeness in direction — between speaker embedding vectors. It appears both in the paper's evaluation metrics (§4.2) and in its reinforcement-learning reward (§3.5). The figure below gives you the geometric intuition.

The data-side answer: SwanData-Caption (§2)

Before the model comes the data. SwanData-Caption is a four-stage pipeline — coverage design, speech preprocessing, caption annotation, refinement — that turns raw media audio into roughly 70 million multi-level caption records (§2).

The raw material is real, speech-centered media audio: short dramas, advertisements, animation. Areas the real corpora leave thin are filled with three synthetic subsets generated by a pronunciation-aware TTS teacher — elderly speech, short Chinese/English utterances averaging 1.5 seconds, and pronunciation-challenging text such as polyphonic characters and brand names — 100k utterances each (§2.1). Humans are acutely sensitive to mispronounced names, yet models rarely see such examples; this is a deliberate correction of that asymmetry.

Preprocessing runs vocal separation → speaker diarization → ASR → alignment (§2.2), with one practical twist: punctuation from ASR is not trusted, because it follows semantics rather than actual pauses. Instead, the downstream caption annotator re-punctuates by meaning (including exclamation and question marks), and the aligner's pause evidence corrects it.

Captions use a three-field schema (§2.3): Environment (the scene and its persistent background bed), Speakers (stable traits of everyone who actually speaks — gender and age, but also persona, timbre, habitual style), and Content (a chronological transcript with <S1> speaker tags, local emotion changes, and <Audio>-tagged effects). To stop the annotator from producing impoverished, boilerplate speaker descriptions, the pipeline adds a "style-persona library" — condensed style matrices for animation, drama, and advertising content. The governing principle: never write an attribute you cannot hear, which keeps unsupported labels out of the supervision.

Refinement (§2.4) then applies waveform-quality filters (drop PESQ below 2.0, STOI below 0.85, and so on), checks gender/age labels against the waveform with SwanVerifier (Appendix §7), and runs human audits. Expressiveness is audited not with absolute MOS scores but with best–worst comparison — pick the most and least expressive of four candidates — which reduces calibration noise across annotators.