UrbanGround
From Local Perception to Spatial Agency in a Real-Scale City
Tianjie Ju, Zheng Wu, Yueqing Sun, Yuhan Cui, Bobo Li, Shengqiong Wu, Pengzhou Cheng, Haodong Zhao, Zongru Wu, Xinbei Ma, Doris Zhang, Kunling Li, Mong-Li Lee, Wynne Hsu, Hao Fei, Qi Gu, Gongshen Liu, Zhuosheng Zhang
Read the Paper on arXivPaper Overview
UrbanGround (Ju, Wu, Sun et al. — Shanghai Jiao Tong University, National University of Singapore, Meituan, and collaborators, August 2026) takes a capability everyone already agrees MLLMs have — reading a street view — and asks the question that actually matters for a city: does that reading survive the next step the agent takes?
A multimodal model can name the storefront in front of it. But an urban agent does not receive its observations as a photo album; it receives them through a continuing physical process where every movement produces the next first-person view. Turn a corner and the landmark that justified your last decision vanishes, even though its location still matters. Strong recognition at one viewpoint therefore tells you almost nothing about whether spatial understanding stays coherent through continued action.
The paper makes this testable by building the first sandbox for it: a physically constrained, georegistered replica of Hong Kong, streamed from the Lands Department's territory-wide 3D geospatial data into Unity, with real collision geometry, a controllable clock and weather, and animated pedestrian crowds. Agents enter from a first-person view, act through a structured interface, and their trajectories are recorded in geographic coordinates.
The evaluation follows the spatial problem as it grows — from grounding a local scene (RQ1), to holding a goal across a route (RQ2), to adapting when the city itself changes (RQ3). The finding is consistent and sobering: contemporary agents (GPT-5.5, Claude-Opus-5, Gemini-3.6-Flash, Kimi-K3 and others) have useful atomic skills, yet navigation collapses once a route spans a few city blocks — best short-range success reaches 75.0% while long-range success sits near 0% — because local competence never composes into sustained, self-correcting behavior.
Chapter Roadmap
Click any topic to jump in
The Spatial Agency Problem
An MLLM can read a street view — but does that local evidence stay useful once the agent starts to move and the landmark leaves the frame?
Closed-Loop Urban Interaction
Grounding, persistence, adaptation — three cumulative capabilities, formalized as a hidden-state loop the model never observes directly.
A Real-Scale Hong Kong
Territory-wide 3D tiles and a georeferenced pedestrian graph streamed into Unity, with physical collision, time-of-day, weather, and animated crowds.
Perception & Action Interface
First-person RGB plus an interactive map in; one structured action per turn out. No shortest path, no remaining distance, no route API.
The Spatial Agency Ladder
810 verified tasks across five levels that raise the spatial state required for success while holding the interface fixed.
Findings and the central gap
What Agents Can and Cannot Do
Strong local recognition, weak orientation, and navigation that collapses past a few blocks — the same separation recurs at every scale.
The Compositionality Gap
Local abilities do not compose into sustained goal-directed behavior; errors accumulate without effective correction.
The paper opens with a distinction that is easy to miss and organizes everything after it: the difference between perceiving a place and acting within it. Recent MLLM agents recognize urban objects from individual observations extremely well. But an agent does not stand still and describe photographs — it is embedded in a physical loop where each action it takes changes the very evidence it will see next.
That loop is what breaks the naive assumption. Once the agent turns a corner, a landmark that supported its last decision may disappear even though its location still matters. The earlier evidence then has to be reconciled with the new view so the agent keeps a usable estimate of where it is. A model can be excellent at the single-frame task and still lose its spatial account the moment the frame moves — so strong recognition at one viewpoint does not demonstrate that spatial understanding will remain coherent through continued action.
UrbanGround decomposes spatial agency into three cumulative capabilities, and the ordering is the whole thesis. Grounding forms task-relevant local relations from locally acquired observations — the atomic skill. Persistence keeps those relations usable after the earlier evidence has left the first-person view, and for global tasks additionally requires aligning map context with the geometry met on the ground. Adaptation concerns whether the state stays reliable when external conditions alter the visible evidence — when new evidence invalidates an earlier assumption, adaptation demands a corresponding revision of the working state.
Each capability presupposes the one before it, which is why the paper can locate exactly where an agent fails rather than only that it failed. RQ1 primarily probes grounding; RQ2 retains grounding while stressing persistence over a route; RQ3 presupposes both and examines robustness when the city changes. Prior settings could not expose this staircase: game sandboxes tie success to game-specific mechanics, bounded indoor benchmarks rarely require an account to stay useful beyond a local area, and street-view or aerial urban studies move the agent between sampled viewpoints without continuous physical contact with the same city.
Key Points
The core reframing: an urban agent receives observations through a continuing physical process where each movement produces the next first-person view — not as independent images
Strong single-viewpoint recognition does not imply coherent spatial understanding through action — once the agent moves, the supporting landmark can leave the frame while its location still matters
Spatial agency is defined as three cumulative capabilities: Grounding (local relations from local observation), Persistence (relations stay usable after evidence leaves view), Adaptation (state stays reliable when the scene changes)
The capabilities are strictly ordered — persistence presupposes grounding, adaptation presupposes both — which is what lets the benchmark pinpoint where an agent breaks
RQ1 → grounding, RQ2 → persistence, RQ3 → adaptation — the three research questions map one-to-one onto the capability ladder
Existing environments each miss part of the transition: game worlds impose game-specific mechanics, indoor benchmarks are spatially bounded, and street-view/aerial urban work lacks continuous physical contact with one real city