Paper Walkthrough — UrbanGround: Where MLLM Agents Break Down on a Real Street

Drop an MLLM agent into a real-scale replica of Hong Kong built from territory-wide 3D geospatial data. Visual recognition clears 90%, orientation sits near 40%, long-range navigation is close to 0%. A walkthrough of the benchmark that measures the gap between seeing and moving.

UrbanGround: From Local Perception to Spatial Agency in a Real-Scale City

Seeing and moving are not the same ability

Hand a current multimodal large language model (MLLM) one street photo and it will read the signs and tell you what kind of shop you are looking at. Now put that same model on the actual street and tell it to walk to a destination. Is seeing the same ability as moving? That is the question this paper is built around.

The original title is "UrbanGround: From Local Perception to Spatial Agency in a Real-Scale City" (arXiv:2608.27456, published 2026-08-27, Shanghai Jiao Tong University and collaborators).

The abstract, in summary: MLLMs can interpret a street view, but urban agency depends on whether that local evidence stays useful after the agent starts to move. The authors propose UrbanGround, a physically constrained replica of Hong Kong offering closed-loop first-person interaction plus an interactive map, then work through three questions in order — (1) can an agent ground a local scene well enough to answer spatial questions after active observation, (2) does that grounding support navigation as destinations get farther away and less explicit, and (3) does the behavior survive changes in route availability and pedestrian motion. Contemporary agents show useful atomic abilities in visual recognition and short-range spatial reasoning, while orientation and pedestrian-aware movement remain unreliable. The central failure emerges over extended exploration: local abilities do not compose into sustained goal-directed behavior, and errors accumulate without effective correction.

An analogy: "northeast" after one corner

You arrive in an unfamiliar city, check a map, register that the station is northeast, then walk into an arcade and turn one corner. Can you still point northeast with confidence? Reading a sign converts evidence in front of you into meaning. Holding the station's bearing means continuously updating evidence that has left your view, relative to your own heading. The first works from a single photo; the second requires remembering who you were one step ago.

The paper formalizes this as three cumulative capabilities (§3.1). Grounding forms task-relevant local relations from what you observe. Persistence keeps those relations usable after the earlier evidence leaves the first-person view. Adaptation rebuilds the state when conditions invalidate an earlier assumption. RQ1 probes grounding, RQ2 persistence, RQ3 adaptation.

Why bearing is hard becomes tangible as two arrows. If the angle between your heading and the goal direction is small, moving forward closes the gap; past 90 degrees, it widens. The agent recovers that angle from images alone, every step.

All of Hong Kong, streamed into Unity

The environment is built from the 3D Digital Map released by the Hong Kong Lands Department (§3.2). A territory-wide textured mesh reconstructed from oblique aerial imagery (the 3D Visualisation Map, distributed as Cesium 3D Tiles under WGS84) supplies the visible city, while georeferenced 3D line features derived from pedestrian road records (the 3D Pedestrian Network) describe how walkable space connects.

UrbanGround loads that tile hierarchy into Unity at runtime and uses the active geometry not just for rendering but as collision geometry. Buildings, walls, and elevation changes genuinely block movement; the agent cannot pass through visible structures by updating coordinates. The pedestrian network only defines intended walkable space — it does not pin the agent to a graph edge. On top of this, a continuous clock drives sky, sun position, shadows, and after-dark lighting, weather adds rain and fog, and the sidewalks carry animated pedestrians from the open-source Microsoft Rocketbox avatar library (§3.3).

The five-level ladder and the setup

Tasks form a five-level ladder that increases the spatial state a run must keep alive while holding the interface fixed (§3.5). Level 1 is local environment understanding (visual recognition, orientation, active exploration QA) and supports RQ1. Level 2 is navigation under explicit instructions (short-range, long-range, instructional, constrained), Level 3 is exploration under implicit instructions (place-type search, intent inference), and Level 4 is multi-task planning (time-window scheduling, multi-stop routing) — these support RQ2. Level 5 is dynamic environment interaction (a mid-route road closure, navigation among moving pedestrians) and supports RQ3.

The study runs on 810 manually verified base instances across Hong Kong, and every instance was completed by human testers under the same 100-step limit given to the agents (§4.1). Since one movement action lasts at most two seconds, 100 steps allows up to 200 seconds of commanded motion. Single-endpoint navigation succeeds within 15 meters of the destination with any enforced constraint satisfied. Evaluated models: GPT-5.5/5.4/5.2, Claude-Opus-5/4.6, Gemini-3.6-Flash/3.1-Pro, Doubao-Seed-2.0-Pro, GLM-5V-Turbo, and Kimi-K3.