audio-video

Definition

Covers the joint treatment of the audio and visual streams of the same recording — lip motion and speech, an impact on screen and the sound it makes. What sets it apart from multimodality in general is time: the streams are naturally aligned frame by frame, which supplies free supervision during pre-training and also becomes the hardest thing to hold onto in generation, where drifting sync is immediately obvious to viewers.

Explainers using this term

• Paper Walkthrough — The Missing Temporal Link: Putting Script Timing on the Video–Audio Axis with Temporal Context RoutingThe Missing Temporal Link: Temporal Context Routing for Script-Driven Audio-Video Generation
• Paper Walkthrough: DreamX-Creator — Making Sound and Picture Together in 7B, Then Finishing at 2K in One StepDreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution