Paper Explained: Video-DeepResearch — Agents That Watch a Video, Then Chase Down Every Lead

A walkthrough of Video-DeepResearch, which pulls visual clues out of a video and corroborates them with web search. We follow the primary source to see how two failure modes — a modality bias that makes models dodge visual tools, and knowledge leakage that lets them answer from memory alone — are beaten with stage-wise tool unlocking and a two-phase SFT + GRPO recipe.

Video-DeepResearch: Towards the Next-Generation Multimodal Deepresearch Agent

What makes "researching" a video hard for an AI

"In which city was the award ceremony held for the prize later won by the person in this video?" Answering that takes two entirely different skills. First, the visual job: watch the footage carefully enough to establish who and what is on screen. Then the investigative job: search the web about that person and stitch several sources together. Think of a detective story — one part is the forensics team lifting a suspect's face from the security footage; the other is the officers taking that photo door to door.

Text-only Deep Research agents came first, then Vision-DeepResearch for still images. What this paper takes on is the continuous video stream — the setting the authors call Video-DeepResearch (Video-DR). Unlike a still image, video carries dense information that changes over time, so the agent has to decide for itself which moment and which region to look at (§1).

Two pathologies in today's agents

The authors begin with a preliminary study, dropping existing models straight into Video-DR tasks, and isolate two failure modes (§2, Empirical Study).

Pathology 1: modality bias, or dodging the visual tools. By the paper's measurements, even Qwen3.5-397B-A17B — among the strongest open-source models available — averages just 0.10 visual tool calls per task. Text search, meanwhile, runs at 1.27. The numbers show the tendency plainly: rather than look at the video, the model tries to get by on text search.

Pathology 2: parametric knowledge leakage. GPT-5 made 0.00 visual and 0.12 text tool calls — essentially reaching for nothing — and still scored 57% on an existing benchmark. It wasn't answering by investigating the video; it was answering from knowledge memorized during training. That says less about the model than about the benchmark: something meant to measure tool-use ability was measuring recall instead.

These two findings are the starting point for everything that follows — the data design, the training recipe, and the benchmark.

Putting the problem in an equation

The paper formalizes Video-DR as sequential decision-making (§2, Problem Formulation). Given a question $Q$ and a sequence of video frames $V$, the agent picks an action $a_i$ at each step $i$.

Read aloud: the next move $a_i$ is drawn stochastically by the policy $\pi_\theta$ — the model that serves as the agent's brain — from everything it has seen so far, the history $\mathcal{H}_i$ (the question, the video, and every past action together with what that action returned).

Spelled out in words: at every step the agent is not following a fixed script, it is drawing its next move, and the only thing it draws on is the record of what has happened up to that point. Change what the earlier steps returned and the next step changes with it — which is exactly why the order in which tools are made available, the subject of the rest of this article, can steer the agent's behavior at all.

The menu of actions includes Select_Keyframe, which pulls out the information-dense moments; Crop_Search, which cuts a rectangle around an object in a frame and runs image search on it; and ordinary text search.

Keyframe selection rests on similarity between embedding vectors. In the paper's data construction, consecutive frames whose CLIP similarity exceeds 0.8 are thrown out as redundant, and each video is narrowed to at most 20 frames (Appendix C).

A 30K data engine — throwing away anything memory can solve

No training data existed for this, so the authors built a pipeline that synthesizes VQA (video question-answering) pairs from raw footage (§3.1), in three stages. First, videos are gathered from existing datasets and YouTube, cut down by length rules, and then screened by a model (Qwen3.5-35B-A3B) that judges how complex the content is, discarding anything too simple. Next, keyframe candidates proposed by CLIP similarity are confirmed by a larger model, objects inside those frames are cropped with bounding boxes, and image search is run on the crops. A separate model checks whether the search results actually match the meaning of the crop, producing metadata of the form ⟨frame, rectangle, object name, search summary⟩. Finally, that metadata is turned into questions: factual ones about a single object, and compositional ones spanning several.