Paper Walkthrough — The Missing Temporal Link: Putting Script Timing on the Video–Audio Axis with Temporal Context Routing

Video and audio stay in sync with each other while both drift off the script's timeline. This walkthrough follows the paper's own equations and numbers to show how TCR closes that third, missing temporal link with a single additive term in the cross-attention logits.

The Missing Temporal Link: Temporal Context Routing for Script-Driven Audio-Video Generation

In sync with each other, out of sync with the script

The paper we are reading is The Missing Temporal Link: Temporal Context Routing for Script-Driven Audio-Video Generation (arXiv:2609.02367, published 2026-09-02). Following the paper, we will call the method TCR.

Its argument, in short. Joint audio-video generators have made real progress on visual quality and on audio-visual synchronization — the lips and the voice line up. What they still do poorly is control over when a shot transition happens and when a line is spoken. In script-driven production, that kind of timing error breaks narrative coherence. The diagnosis: current joint generators do align video and audio on a shared temporal axis, but the shot and dialogue timing written into a structured prompt is encoded only in the prompt's text representation, unaligned with the temporal coordinates of either modality. So video and audio remain synchronized with each other while both fail to follow the script timeline. The paper's answer is to extend temporal alignment beyond video and audio to include the structured script: Temporal Context Routing maps script timing onto the shared temporal axis and routes each prompt's guidance to the corresponding positions in both modalities. Across 200 test scripts it cuts Shot Boundary MAE by 96%, from 1.11 s to 0.042 s, and lifts Dialogue Acc@0.5 s from 28.3% to 84.1%, while keeping visual quality and audio-visual sync comparable to the baselines. A user study finds participants prefer TCR on all five evaluated dimensions (Abstract).

What makes this worth reading is that none of it comes from a new network. TCR has zero learnable parameters. The whole mechanism is one extra term added to a score table.

An analogy: a crew that never got the schedule

Picture a shoot. The camera operator and the sound recordist work off the same monitor, so their timing with each other is flawless. When the camera cuts, the audio changes with it. No problem there.

But the only thing the director handed them was a recording of someone reading the schedule aloud: "cut at 2.3 seconds, line from 1.7 to 3.1." Both of them heard the numbers. Neither transferred them onto their own stopwatch. So the two stay beautifully in step with each other while the cut slides to 3.4 seconds.

The "missing temporal link" of the title is exactly this: the break between the numbers on the schedule and the ticks on the stopwatch. TCR writes the director's instruction into the stopwatch instead of leaving it in the audio of someone's voice.

The intuition: there were only two rulers

Stated more technically, script-driven generation involves three temporal rulers:

1. the axis along which video latents are laid out,
2. the axis along which audio latents are laid out,
3. the axis on which the script declares "this shot runs 0.0–2.3 s."

Existing joint generators put their effort into aligning 1 with 2 (§1, §2). For 3, a numeric time_range is simply serialized into the prompt string and pushed through the text encoder. But a text encoder produces meaning vectors, not coordinates. From the model's point of view, the third ruler is a number printed on paper with no tick marks on it.

Video-focused work has explored when a local text prompt should influence generation, but it does not address how a structured script should jointly control video and audio (§1). That leaves the failure mode intact: fixing one modality alone just drags the other along, because the two are busy staying synchronized with each other.

The paper's central insight (§1) fits in a sentence: extend temporal alignment from the video–audio pair to a triple that includes the structured script, representing each prompt's specified timing as an explicit control signal aligned with the temporal coordinates of both modalities.

Three difficulties follow (§1). First, shot prompts and dialogue prompts can occupy distinct, partially overlapping spans — a line can carry across a cut — so the two prompt types cannot be forced into one temporal segmentation. Second, learning this control needs fine-grained annotation, while the annotations you start with are coarse. Third, none of it may come at the cost of visual quality or audio-visual synchronization.

Groundwork: a cross-attention score is a dot product

Before the equations, it helps to be clear about where the extra term lands.

The generator pairs each video latent at a given time (a query $\mathbf{q}^v_i$) with each text token of the script (a key $\mathbf{k}^v_j$) to build a table of "how much should this moment attend to this word." The entries are dot products. The audio tower keeps its own separate table of the same shape. This is the machinery covered in Attention from Scratch; the only difference is that the asking side (video, audio) and the answering side (script text) are different objects.

A large dot product means two vectors point in a similar direction. Turning that by hand first makes it obvious what the added term is bending.

The mechanism: turn an interval into a parabola, add it to the logits

TCR starts by rewriting the interval $I_j=[s_j,e_j]$ that token $j$ inherits from its parent prompt as a center and a radius (§3.3).

$c_j$ is the midpoint of the interval and $r_j$ is half its length; $\epsilon=10^{-4}$ seconds keeps a zero-length interval from breaking the division. Equation (1) simply restates "0.0–2.3 s" as "center 1.15 s, radius 1.15 s."

Then the core. For modality $m\in\{v,a\}$ ($v$ for video, $a$ for audio) and a query sitting at temporal coordinate $t^m_i$, the routing score is

Read aloud: take how far the current moment is from the center of that token's assigned interval, divide by the interval's own length to normalize it, square it, and flip the sign. It is 0 at the center, $-\beta/2$ at either endpoint, and falls away quickly outside. Tokens with no associated interval get $B^m_{ij}=0$.

Dividing by $r_j$ is doing real work. A 0.5-second line and a 5-second shot differ by 10× in absolute time, but after the division both become the same shaped hill that reaches $-\beta/2$ at its edges. Long intervals fall off gently, short ones sharply, with no extra bookkeeping.