Paper Walkthrough: GST-Bench — Can VLMs Build a Global Map of a Scene from Video?

Show a VLM a walkthrough video of a house, then ask 'from where you're standing now, which way is the sofa?' — even the strongest model scores barely half of what humans do. A walkthrough of GST-Bench from ByteDance Seed: the shortcut-proof benchmark design, results across 22 models, and the training data that closed 27 points of the gap.

GST-Bench: Can VLMs Develop Global Spatial Awareness from Video?

In One Sentence

GST-Bench (arXiv:2608.05747, ByteDance Seed / Zhejiang University / National University of Singapore) is a video VQA benchmark that tests whether vision-language models (VLMs) can assemble a global spatial map of a scene from a long egocentric video. Across 2,762 human-verified questions built from 6,790 minutes of synthetic video, the best of 22 state-of-the-art VLMs scores 42.68 — far below the human score of 79.08 (Abstract).

The Metaphor: Apartment-Hunting by Video Only

Imagine choosing an apartment based solely on a walkthrough video the realtor filmed — entering at the front door, wandering through every room. After you finish watching, someone asks: "You are now standing in the corner of the kitchen (a spot that never appeared in the video). Which direction is the bedroom closet from here, and how many meters away?"

Remembering individual frames isn't enough to answer. You have to fuse the scattered views into one floor plan, place yourself on it, and reason about the position of an object you cannot currently see. Humans do this effortlessly every day. The paper argues this ability — global spatial awareness — is precisely the hole in existing benchmarks: prior spatial benchmarks mostly test local perception from one or a few images ("which of A and B is closer?"), and even video-based ones mix in questions solvable from a single frame, while grading directions with coarse labels like front/back/left/right (§1, §2).

Three Questions, Twelve Tasks

GST-Bench organizes its tasks around three fundamental questions an embodied agent must answer about a scene (§3.2):

• Self localization (Where am I?): given a novel view that never appeared in the video, infer your position and orientation on a top-down map (2 tasks)
• Object localization (Where is the target?): for a target that is not visible in the current view, infer its egocentric direction, its metric distance, and its position on the map. Targets are specified either by category name (semantic) or by a red bounding box drawn in the video (visual) (6 tasks)
• Scene structure understanding (What does the scene look like?): pick the top-down image matching the explored scene from four candidates (at three abstraction levels: photo-realistic, occupancy map, bare floor plan), or match a short motion clip to one of four trajectories drawn on the map (4 tasks)

Directions must be answered as numeric angles, not "to the left," and distances in meters. A predicted direction and the ground-truth direction are compared like two arrows: the smaller the angle between them, the better the score. You can get a feel for "measuring direction as the angle between arrows" by rotating the two vectors in the figure below.

Four Design Choices That Block Shortcuts

The most instructive part of this benchmark is how it prevents models from being right for the wrong reasons (§1, §3.3):

1. The target object is guaranteed to be invisible from the query view. Single-frame perception cannot solve the task even in principle; integrating memories from the video is mandatory
2. Scoring uses precise numbers, not coarse categories. Instead of "which is closer, A or B?", models must output the actual distance or angle, removing ambiguity
3. Top-down images directly probe global–local correspondence — can the model project egocentric observations onto a bird's-eye representation?
4. Query viewpoints are sampled off the video's camera trajectory, so simply matching the query image against video frames doesn't work