Paper Walkthrough: GST-Bench — Can VLMs Build a Global Map of a Scene from Video?

Show a VLM a walkthrough video of a house, then ask 'from where you're standing now, which way is the sofa?' — even the strongest model scores barely half of what humans do. A walkthrough of GST-Bench from ByteDance Seed: the shortcut-proof benchmark design, results across 22 models, and the training data that closed 27 points of the gap.

GST-Bench: Can VLMs Develop Global Spatial Awareness from Video?

In One Sentence

GST-Bench (arXiv:2608.05747, ByteDance Seed / Zhejiang University / National University of Singapore) is a video VQA benchmark that tests whether vision-language models (VLMs) can assemble a global spatial map of a scene from a long egocentric video. Across 2,762 human-verified questions built from 6,790 minutes of synthetic video, the best of 22 state-of-the-art VLMs scores 42.68 — far below the human score of 79.08 (Abstract).

The Metaphor: Apartment-Hunting by Video Only

Imagine choosing an apartment based solely on a walkthrough video the realtor filmed — entering at the front door, wandering through every room. After you finish watching, someone asks: "You are now standing in the corner of the kitchen (a spot that never appeared in the video). Which direction is the bedroom closet from here, and how many meters away?"

Remembering individual frames isn't enough to answer. You have to fuse the scattered views into one floor plan, place yourself on it, and reason about the position of an object you cannot currently see. Humans do this effortlessly every day. The paper argues this ability — global spatial awareness — is precisely the hole in existing benchmarks: prior spatial benchmarks mostly test local perception from one or a few images ("which of A and B is closer?"), and even video-based ones mix in questions solvable from a single frame, while grading directions with coarse labels like front/back/left/right (§1, §2).

Three Questions, Twelve Tasks

GST-Bench organizes its tasks around three fundamental questions an embodied agent must answer about a scene (§3.2):

• Self localization (Where am I?): given a novel view that never appeared in the video, infer your position and orientation on a top-down map (2 tasks)
• Object localization (Where is the target?): for a target that is not visible in the current view, infer its egocentric direction, its metric distance, and its position on the map. Targets are specified either by category name (semantic) or by a red bounding box drawn in the video (visual) (6 tasks)
• Scene structure understanding (What does the scene look like?): pick the top-down image matching the explored scene from four candidates (at three abstraction levels: photo-realistic, occupancy map, bare floor plan), or match a short motion clip to one of four trajectories drawn on the map (4 tasks)

Directions must be answered as numeric angles, not "to the left," and distances in meters. A predicted direction and the ground-truth direction are compared like two arrows: the smaller the angle between them, the better the score. You can get a feel for "measuring direction as the angle between arrows" by rotating the two vectors in the figure below.

Four Design Choices That Block Shortcuts

The most instructive part of this benchmark is how it prevents models from being right for the wrong reasons (§1, §3.3):

1. The target object is guaranteed to be invisible from the query view. Single-frame perception cannot solve the task even in principle; integrating memories from the video is mandatory
2. Scoring uses precise numbers, not coarse categories. Instead of "which is closer, A or B?", models must output the actual distance or angle, removing ambiguity
3. Top-down images directly probe global–local correspondence — can the model project egocentric observations onto a bird's-eye representation?
4. Query viewpoints are sampled off the video's camera trajectory, so simply matching the query image against video frames doesn't work

The data is generated automatically from 50 indoor scenes in the OmniGibson simulator with BEHAVIOR-1K. Because everything lives in simulation, the 3D coordinates of every camera and object are known, so ground-truth distances, angles, and top-down projections can be computed exactly. Every evaluation sample is then manually verified — annotators confirm the target is identifiable in the video and that the off-trajectory query view can still be localized by a human — leaving 2,762 questions (§3.3). The idea of reasoning over a top-down scene representation closely parallels BEV representations in autonomous driving — see our BEV representation explainer.

How It Scores

Questions come in four formats — distance, angle, point, and multiple choice — each with a tailored metric (§4.1). Distances use Mean Relative Accuracy (following VSI-Bench: averaging, over confidence thresholds 0.50–0.95, whether the relative error stays under each threshold). Angles use circular error:

Put in words, the equation says: measure the gap between the direction the model answered and the direction that is actually correct, going both ways around the circle, and keep whichever of the two gaps is smaller. Here $\hat{\alpha}$ is the predicted angle and $\alpha$ the ground truth. Equation (1) just encodes the common-sense fact that 350° and 10° are 20° apart, not 340° — you take the shorter way around the clock face. The score is the average of whether this error falls within tolerances of 15°, 30°, and 45°. Point predictions are scored by pixel distance against five thresholds, multiple choice by plain accuracy, and the overall score is the simple mean of the 12 subtasks.

Results: The Best Model Reaches 54% of Human Performance

Twenty-two models — 5 proprietary, 12 open-source, 5 embodied-tuned — were evaluated zero-shot (§4.2):

• Humans score 79.08; the best model, Gemini-3-Pro, scores 42.68. The 36.4-point gap is widest on orientation estimation (21.52 vs. 85.00) and object position on the map (42.23 vs. 93.00)
• Open-source models hover near random. Random guessing scores 20.01; the best open model, InternVL3.5-38B, reaches only 30.71 — and that lead is mostly carried by the easier top-down multiple-choice tasks, with the other ten tasks within a few points of random. Four models actually score below random
• The one outlier is distance estimation, where the human baseline itself is low (41.50) and Gemini-2.5-Pro essentially matches it (42.00). The paper reads this not as model strength but as evidence that recovering absolute metric distance is intrinsically hard for humans too
• Embodied tuning brings no advantage. RoboBrain2.5-8B (24.61) and Cosmos-Reason2-8B (21.64), both tuned for grounded spatial reasoning, fall below the plain Qwen3-VL-8B (25.89). The authors conjecture that current embodied post-training emphasizes local affordances and spatial relations rather than long-horizon spatial memory and cross-viewpoint alignment (§4.2)

Pinpointing Where Models Fail

Is the failure one of basic spatial perception, or of integration across frames? To disentangle the two, the paper builds controlled variants of the same questions (§4.3): Local-Video swaps the query view for one where the target is visible (the video remains but becomes logically unnecessary), and Local-Image further removes the video, reducing the task to single-image spatial perception.

Two qualitatively different bottlenecks emerge. Proprietary models see locally but fail to integrate globally: Gemini-3-Pro's direction score nearly triples from 22.11 (Global) to 61.20 (Local-Image). Single-image spatial perception is fine — the failure lives in cross-frame integration. Open-source models fail at both stages: some even get worse when the task is made easier (InternVL3.5-8B drops 8.03 points on direction), and even when they improve, absolute scores stay low (InternVL3.5-2B reaches 25.40 on Local-Image direction versus Gemini-2.5-Pro's 66.49).

Targeted Training Closes 27 Points

Fine-tuning Qwen3-VL-8B on GST-Train — a training set built by the same pipeline, with scenes held out from the benchmark — lifts its score from 25.89 to 53.52, a 27.63-point jump, surpassing every zero-shot model in the study, proprietary ones included (§4.4). The encouraging implication: the global-reasoning gap is largely a hole in training data, not an architectural destiny. That said, 53.52 is still well short of the human 79.08, and the paper itself states that targeted supervision improves but does not fully solve the challenge.

Limitations and How to Read the Numbers

• Everything is simulation footage. That's what makes exact ground truth computable, but whether the same conclusions hold on real-world video is outside this paper's scope (transfer to real footage is untested — this caveat is mine, not a claim the paper makes)
• The human baseline uses 20 samples per task (Table 1) — plenty to establish the trend, but small enough that fine-grained point differences should be read with error bars in mind
• Scenes are indoor only, and the fine-tuned model was trained on data from the same generation pipeline as the benchmark. The paper explicitly separates the scenes (§4.4), but distributional closeness between train and test likely flatters the fine-tuned score somewhat

How This Gets Used in Practice

This benchmark matters most to people putting VLMs on things that move.

• Robotics, drone, and AR engineers choosing a model: a model advertised as "strong at spatial reasoning" backed only by single-image benchmarks tells you nothing about navigation or exploration performance. In practice, add video-based, global-integration benchmarks like VSI-Bench and GST-Bench to your selection criteria. The paper's negative result — embodied tuning failing to improve global reasoning — is a concrete reason not to take "embodied-ready" marketing at face value
• Evaluation engineers: the scoring design transfers directly — MRA over thresholds 0.50–0.95 for distances, angular tolerances of 15°/30°/45° for directions. Grading directions as a four-way front/back/left/right choice cannot distinguish genuine understanding from lucky guessing
• Training data builders: the pattern of machine-generating QA from a simulator with full 3D ground truth (OmniGibson + BEHAVIOR-1K), then filtering with human verification generalizes to any task where the answer can be computed from geometry
• A pitfall to know: when feeding long videos to a VLM, your frame-sampling strategy can silently delete the spatial cues the model needs. If you can't reproduce benchmark numbers, suspect your own preprocessing (sampling rate, resolution) before suspecting the model — a standard debugging move in practice

For the bigger picture of VLMs acting as agents, see LLM Agents Basics; for the visual backbone fundamentals, see our Vision Transformer explainer.

Takeaway

What GST-Bench pins down is a concrete deficit in today's VLMs: they can see the frames, but no map forms in their head. The paper quantifies that deficit with shortcut-proof task design, localizes the cause to the integration stage with controlled ablations, and shows 27 points of it can be recovered with targeted data — a textbook example of how to build a diagnostic benchmark. Before mounting a VLM on a robot, it's worth spending ten minutes with these twelve tasks.