Paper Explained: Reason Through the Latent! — Visual information being in a latent state doesn't mean the answer uses it

Several latent visual reasoners keep their accuracy even when you corrupt the latent state — the answer is not actually flowing through it. CVRR (arXiv:2609.06746) starts recurrence from the question state after the image has been absorbed, then closes every other image-conditioned route before decoding, making latent computation necessary while keeping performance. A walkthrough grounded only in the paper's text, including its causal interventions.

Reason Through the Latent! Making Latent Visual Reasoning Necessary

In an open-book exam, your reasoning is optional

Picture an open-book exam. A student may well reason through the problem in their head, but if they can open the textbook at the moment of writing the answer, nothing guarantees that the reasoning was ever used. Swap out the thoughts and the same answer still gets written, straight from the book.

The paper covered here, "Reason Through the Latent! Making Latent Visual Reasoning Necessary" (arXiv:2609.06746; Park, Jung and Kang, Korea University / AIGEN Sciences), asks exactly this question of AI models that reason over images (vision-language models, VLMs).

In summary, the paper's abstract says the following. Latent visual reasoning aims to perform multimodal reasoning through hidden-state computation rather than explicit textual chains of thought. But visual information being present in a latent state does not imply that the model relies on that state when producing its answer, especially when alternative image-conditioned paths remain available. The authors introduce Causal Visual Recurrent Reasoning (CVRR), which initializes recurrence from the question hidden state after the pretrained VLM has incorporated the image, repeatedly updates that state while re-reading the same fixed visual evidence, and, before decoding, removes the visual states and the original multimodal KV cache so that only the final recurrent state carries image-conditioned information to the answer. Across $V^*$, MMVP, BLINK and MME-RealWorld-Lite, CVRR retains strong performance under this strict interface, while compatible latent reasoners fail to recover comparable visual competence even when retrained under the same constraint. Causal interventions further show that predictions remain sensitive to recurrent content when the question is held fixed, and that persistent visual evidence causally revises the recurrent trajectory.

Intuition: "informative" and "used" are two different things

The paper opens with a rather blunt experiment (§2.1). For each existing latent visual reasoner, the proposed latent state is replaced with one of three things: a blank-image latent, a matched latent from an example with a different answer, or row-norm-matched noise. Crucially, the original multimodal answer context (image tokens, KV caches) is left untouched.

Several methods barely move. The paper singles out UniVLR, which changes by at most 0.5 percentage points across all three replacements. The central point is that a latent state can be informative without being causally necessary for the prediction.

Note what the paper is not claiming. As §1 states explicitly, the authors do not argue that every useful latent reasoner must eliminate all redundant visual paths. The claim is narrower: if the scientific claim is that prediction depends on a proposed latent computation, latent informativeness alone is insufficient evidence.

Preliminary experiment: the question tokens have already absorbed the image

If you do close the bypass, which representation should carry the pretrained visual competence? §2.2 builds four states at an intermediate layer of the frozen Qwen2.5-VL-7B-Instruct and feeds each alone into the same frozen upper decoder.

• Full MM: the full multimodal hidden state with visual and question tokens → 75.9%
• Multimodal Q: only the question rows from that same multimodal pass → 75.4%
• Text-only Q: question rows from a pass without the image → 35.6%
• Text+Raw Visual: text-only question rows concatenated with raw visual features → 35.6%

The gap between Full MM and Multimodal Q is 0.5 points. By this layer, nearly all of the visual competence has migrated into the question rows; dropping the image tokens costs almost nothing. Conversely, bolting raw visual features onto a text-only state recovers nothing. Two design requirements fall out: (1) the latent path should start from the question state that has already read the image, and (2) alternative image-to-answer routes should be removed.

Mechanism: three parts

Structurally CVRR is "find the boundary → loop one layer → cut every bypass before decoding" (§3; the full procedure is in Appendix C).

Part 1: locate the causal visual-read boundary, layer by layer (§3.1)

The starting layer for recurrence is not chosen arbitrarily. The paper looks for the layer where the image has already entered the question state substantially, yet the visual tokens still exert downstream influence that recurrence can re-read. It uses activation patching: build pairs $x=(I,q,y)$ and $x'=(I',q,y')$ sharing a question but with different images and answers, then at each layer $\ell$ replace one example's visual-token activations with the other's and measure how much the preference for the original answer (a log-probability margin) drops, in both directions.

In words, this measures how much confidence in the original answer is lost when the image contents are swapped at layer $\ell$, normalized so that swapping at layer 0 counts as 1. $M_x$ is how strongly the model prefers $x$'s answer over $x'$'s, and $P_\ell(x\leftarrow x')$ overwrites $x$'s visual activations with $x'$'s at layer $\ell$.

In plain terms: "if I swap the image contents at layer $\ell$, how much confidence in the answer is lost", normalized so that swapping at layer 0 counts as 1. $M_x$ is how strongly the model prefers $x$'s answer over $x'$'s, and $P_\ell(x\leftarrow x')$ is the operation of overwriting $x$'s visual activations with $x'$'s at layer $\ell$. The layer just before the first sustained decline becomes the boundary. For Qwen2.5-VL-7B this gives $\ell^*=20$, and the next layer $L_{21}$ initializes recurrence.

Part 2: reuse one layer to re-read fixed visual evidence (§3.2)

Take the frozen backbone's output at the boundary, apply the next layer $f_0=L_{21}$ once in its pretrained form, and split the result: the question rows become $h_1$ (the initial recurrent state), the visual rows become $v$ (persistent visual evidence). From then on, at each step only the question rows are replaced with the previous state, the visual rows stay fixed at $v$, and the same layer (now with LoRA) is applied again.

In words, each step puts the previous thoughts $h_{t-1}$ back into the original scaffold, reads them together with the fixed visual evidence through the same layer once more, and then blends the new thoughts with the old ones in proportion $\beta$.

Read it as: "put last step's thoughts $h_{t-1}$ back into the original scaffold, read them together with the fixed image evidence through the same layer once more, and blend the new thoughts with the old ones in proportion $\beta$." $P_Q$ selects the question rows, $f_\phi$ is the shared layer with LoRA parameters $\phi$, and $\beta$ is the mixing coefficient (0.50 in Appendix B.1). No separate cross-attention module is added; the decoder layer's own causal self-attention does the re-reading, so token ordering, mask and positions are all preserved.

The only trainable parameters are the LoRA weights on this shared layer. Appendix B.1 reports rank 32, $\alpha=16$, 2,883,584 trainable parameters out of roughly 8.295B — 0.0348%. The figure below lets you feel what a low-rank delta can and cannot capture.