Paper Walkthrough — Beyond Retrieval: LatentStream Turns Retrieved Video Into Latent Memory

For never-ending video streams, LatentStream stops appending retrieved evidence as extra context and instead internalizes it into fixed-length latent memory tokens. A ground-up walkthrough of its hierarchical memory, latent evolution, and confidence-driven test-time optimization.

Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding

Where this paper sits

The original title is "Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding" (arXiv:2609.04131v1 [cs.CV], published 3 September 2026). It comes from Hongyu Qu and colleagues at Nanjing University of Science and Technology, Ant Group, the National University of Singapore, and the Chinese University of Hong Kong.

Here is the abstract in plain terms. Streaming video understanding asks a multimodal large language model (MLLM) to process visual input that never stops arriving and to answer questions that can be posed at any moment — all under strict causality (no peeking at frames it hasn't seen) and a bounded memory budget. Existing systems usually compress past observations into an external memory bank and, once a query arrives, retrieve the relevant evidence and hand it to the model as extra visual context. The paper's objection is that this "store-and-retrieve" paradigm keeps that evidence as external visual context, so it never becomes a compact, evolving latent memory capable of continuously guiding streaming reasoning. Their answer is LatentStream, a shift from store-and-retrieve to retrieve-and-internalize, built from three coordinated pieces — hierarchical streaming memory, hierarchical latent memory evolution, and confidence-guided test-time optimization — which they report achieves new state-of-the-art results on both online and offline video benchmarks.

An analogy: the rookie in the CCTV room

Picture someone new to a surveillance desk. Asked a question, they rewind the recording, hunt for the moment, and point at the screen: "there it is." That is store-and-retrieve. It produces an answer, but the footage they found was a one-off reference — nothing stays in their head.

An experienced operator behaves differently. After enough hours, summaries settle into their working memory: the loading bay gets busy at this hour, that person always takes the same route. And those summaries determine where to look next. What they carry is not the footage but a state extracted from it. Making that transition is exactly what LatentStream is after (§1).

Why retrieval alone isn't enough

The paper's central claim is that prior work concentrates on which history to access, while leaving how the accessed evidence gets internalized into a latent state largely unexplored (§1). The consequence, in their words, is that query-agnostic streaming memory and query-conditioned reasoning stay only loosely coupled.

Put differently: in the conventional setup there is no bridge between memory and reasoning. Each query pulls evidence in from outside, pastes it into the prompt, answers, and throws it away. The next query starts over from nothing. When the paper says the model's latent space is "a natural substrate for bridging this gap," breaking that cycle is what it means.

The retrieval half itself is familiar: score candidates by embedding similarity and keep the top few, the same machinery covered in RAG fundamentals. Getting a feel for it first makes the rest of this easier to follow.

Piece 1: hierarchical memory, built before any query (HSM)

The first component is Query-agnostic Hierarchical Streaming Memory (§3.2). Memory is split into three levels, $\mathcal{M}=\{\mathcal{M}^{s},\mathcal{M}^{m},\mathcal{M}^{l}\}$ — short, mid, and long term. Freshly arrived visual tokens go densely into short-term memory so recent detail survives intact.

The interesting part is how overflow gets pushed downward. The paper reaches for Jenks Natural Breaks (Jenks and Caspall, 1971), the classical technique used to bin values on choropleth maps. The point is that the cut points are not fixed thresholds — they are derived from the score distribution actually being observed right now. For the short-to-mid transition, three-class Jenks partitioning over temporal importance scores yields two breakpoints that route representations into Drop, Compress, or Preserve (Eq. 1). For mid-to-long, two-class Jenks partitioning is applied to the distances $d_{ij}=1-\cos(\mathbf{v}_i,\mathbf{v}_j)$ between neighbouring preserved tokens: the low-distance group is spatially redundant and gets merged. Crucially, this whole structure is built online, before any query arrives. That is what makes it query-agnostic.

Piece 2: latent memory tokens with expanding receptive fields (HME)

Hierarchical Latent Memory Evolution (§3.3) is the heart of the method. Latent memory tokens (LMTs) are split into three groups, so that at evolution iteration $r$: