UrbanGround

To make the three capabilities measurable, the paper writes the interaction down as a control loop — and the important design choice is what the agent is not allowed to see.

Let $s_t$ be the hidden interaction state at turn $t$. It holds the georegistered agent pose, the current interface state, task progress, and city conditions. The evaluated model never reads $s_t$ directly. What it receives is an observation bundle $o_t = O(s_t)$: the current first-person or map view plus any task-specific notice or status exposed that turn. From this the model maintains a working spatial state $m_t$, chooses one structured action $a_t = \pi(m_t, g)$ toward objective $g$, and the simulator advances the world by $s_{t+1} = T(s_t, a_t, \xi_t)$.

The variable $m_t$ is deliberately not a memory module. It is an analytical description of the task-relevant spatial information induced in the model's interaction context — the paper does not assume any explicit memory mechanism and does not observe $m_t$ either. This matters because it keeps the benchmark honest about what it is testing: whatever spatial account the model carries has to live in its ordinary context, not in privileged scaffolding.

The exogenous term $\xi_t$ is where adaptation gets its teeth. For matched time-and-weather episodes the scene condition varies in the initial state $s_0$ while $\xi_t$ stays fixed; but $\xi_t$ can also change during execution — a road-closure event or pedestrian motion — so an earlier plan can be invalidated mid-episode. Each non-terminal action updates the physical or interface state, which determines the next evidence $o_{t+1}$, closing the loop.

The framework realizes this through three layers with a clean separation of concerns. The geospatial layer anchors the visible city and pedestrian connectivity in one geographic frame and records each trajectory in the same coordinates. The simulation layer implements the transition $T$ through continuous physical motion with collision, and controls the changes represented by $\xi_t$. The agent layer exposes the observation function $O$ plus the structured action interface. Keeping the geographic world and the embodiment fixed across every evaluated model is what makes cross-model comparison mean something.

The environment is what separates UrbanGround from generated or assembled urban simulators: it does not model a plausible city, it streams an existing one.

The geospatial layer is built on the 3D Digital Map released by the Hong Kong Lands Department, using two datasets. The 3D Visualisation Map is a tile-based textured mesh reconstructed from oblique aerial imagery covering the full territory, distributed through an open API as Cesium 3D Tiles under WGS84. The 3D Pedestrian Network is a set of georeferenced 3D line features derived from pedestrian road records. The former captures the visible form of the city; the latter describes how its pedestrian spaces connect — and Hong Kong is an unusually demanding choice, with dense development on steep terrain and a pedestrian network that repeatedly shifts between street level and elevated passages.

Unity loads the 3D Tiles hierarchy at runtime and transforms each tile's geographic coordinates into a shared simulation frame. Tiles stream by viewing scale, and the active geometry supplies both the rendered surface and the collision geometry used by the character controller — so the agent moves on the surface it actually sees and cannot cross a visible structure by editing its coordinates. The pedestrian network is imported into the same coordinate system as a connected graph, each vertex keeping its geographic position, each edge a walkable segment. Crucially it does not constrain the agent to graph edges: the agent moves freely in continuous space, and its trajectory is compared to the network after the fact — which is what makes off-network movement observable rather than impossible.

The simulation layer then makes the registered city a continuously evolving world. A continuous time-of-day clock drives the sky, sun, ambient light and shadows, with artificial lighting after dark; it can be pinned to an hour for controlled comparison or allowed to advance. The weather system adds rain and fog that alter visibility and rendering, and rain produces particles that interact with the scene — yielding an exposure signal that distinguishes covered paths from open streets, so route quality can depend on whether an agent picks shelter under rain. And the city is not empty: sidewalks are populated with animated pedestrians from the open-source Microsoft Rocketbox avatar library, spawned on the registered network and assigned routes over its edges, with positions and collisions recorded every step so navigation success can be weighed against trajectory safety.

The agent layer connects an external MLLM to the running Unity world as a separate process over a client-server interface, so any model or agent framework can drive the same embodiment without touching the Unity implementation. That isolation is what keeps the embodiment identical across every evaluated system.

The primary observation is an RGB image rendered from the agent's first-person camera, presenting the real city from the pose the previous physical action produced. A bounded frame buffer may be carried in the observation context to provide visual history across consecutive decisions. For tasks needing global reasoning, an interactive map tool offers a georeferenced overhead view marking the agent's current location, which it can pan and zoom to inspect areas outside its first-person view — but the map does not expose a computed or highlighted route. The design principle throughout is that the interface reveals no privileged information: no remaining distance, no shortest path, no route API.

The action space is a strict one-per-turn selection from $A = A_{fp} \cup A_{map} \cup \{\text{terminate}\}$. The first-person actions are $A_{fp} = \{\text{move}, \text{sprint}, \text{look}, \text{jump}, \text{open map}\}$; the map actions are $A_{map} = \{\text{map select}, \text{map pan}, \text{map zoom}, \text{map orbit}, \text{close map}\}$. The move and sprint actions specify one of four movement directions and a duration, and may additionally carry yaw rate, pitch rate, jump, and jump-at fields — so the agent can adjust its view or jump within the same movement action. Because the direction field already covers lateral movement, strafe is not a separate action. The look action changes yaw and pitch without translating; jump is a standalone hop; the map actions open, close, select a point, and control the map view. The terminate action ends the episode.

An episode ends when the agent submits an answer, signals completion, or reaches 100 interaction steps. Since each movement action lasts at most two seconds, that budget permits up to 200 seconds of commanded motion — enough that model failures cannot be blamed on tasks being infeasible within the horizon. It is a small, honest interface: rich enough to move and look through a real city, deliberately stripped of anything that would do the spatial reasoning for the agent.

The tasks are organized as a five-level ladder that increases the spatial state required for success while keeping the interaction interface fixed. Holding the interface constant is what isolates the variable of interest: what changes from level to level is not the controls but how much spatial account must survive across action.

Level 1 — Local Environment Understanding (220 instances) studies what can be grounded around the current pose. Visual recognition and orientation test recognition and heading-relative localization; active exploration adds a brief search for evidence outside the initial view. This is the main source of RQ1 tasks. Level 2 — Navigation under Explicit Instructions (240) introduces goals over routes: short- and long-range tasks vary how long grounded evidence must remain useful, while instructional and constrained variants test whether route language or path restrictions stay active during execution. Level 3 — Exploration under Implicit Instructions (120) moves to place-type search and implicit intent inference, where the agent must infer where to go before it can navigate. Level 4 — Multi-Task Planning (120) extends persistence to time-window scheduling and multi-stop routing, maintaining and executing a plan across several goals under task-provided progress updates. Levels 2–4 together provide the RQ2 tasks and stress persistence while retaining local grounding.

Level 5 — Dynamic Environment Interaction (110) introduces change during execution. A one-time system notice discloses a road closure that invalidates an earlier route, with the unavailable segment marked on the map; moving pedestrians require the agent to adjust its motion around visible obstacles. Matched time-and-weather conditions additionally apply to Level 1 and the short-navigation task in Level 2, changing the visual evidence at the same task locations. These provide the RQ3 tasks, examining adaptation while presupposing the earlier capabilities.

The whole study contains 810 manually verified base instances distributed across urban regions of Hong Kong chosen to differ in street pattern, terrain, and vertical pedestrian connection. Every instance was completed by human testers under the same 100-step limit used for the agents. The scoring is deliberately physical: question-answering episodes are scored against the annotated answer, but single-endpoint navigation succeeds only when the final position is within 15 meters of the destination and any enforced constraint is satisfied. Pedestrian-network adherence (PNA) — the fraction of action time whose post-action state lies on the registered network — is tracked throughout, and dynamic tasks add closure-violation, pedestrian-contact, and rain-exposure records.

The results read as a descent down the ladder, and the pattern is remarkably consistent across twelve contemporary models — GPT-5.6-Luna/5.5/5.4/5.2, Claude-Opus-5/4.6, Gemini-3.8-Flash/3.6-Flash/3.1-Pro, Doubao-Seed-2.0-Pro, GLM-5V-Turbo, and Kimi-K3.

RQ1 — grounding mostly works, orientation does not. On local question answering, most models are strong on visual recognition where the instruction and evidence are explicit, and answer accuracy declines only moderately on active exploration. Overall Level-1 accuracy tops out at 79.1 (Claude-Opus-5), with Gemini-3.6-Flash at 77.7 and Kimi-K3 at 76.4. But orientation is a shared atomic weakness: performance falls markedly, with several systems approaching the random-guessing baseline for a four-option question — Gemini-3.1-Pro reaches just 23.3 on orientation, below chance, and even GPT-5.2 sits at 33.3. Identifying a visible landmark plainly does not guarantee a stable representation of its direction relative to the current heading. Pedestrian-network adherence exposes a second gap: agents often favor a shorter path to the evidence and sacrifice road compliance to answer faster.

RQ2 — navigation collapses past a few blocks. With an explicit, visible destination on short routes, some models do meaningfully well — GPT-5.6-Luna hits 81.3 on ShortNav, and GPT-5.5 and Claude-Opus-4.6 both reach 75.0. But when the same point-to-point task is evaluated at a longer scale, nearly every model fails: long-range success is 0.0 for half the models and never above 5.0. This is not merely a longer distance — a route that decomposes into short-range goals should be solvable one hop at a time. The models cannot compose those atomic capabilities because errors accumulate during execution without effective correction. Overall navigation stays low across the eight subtasks: 22.9 (Claude-Opus-4.6), 22.5 (Kimi-K3), 20.8 (GPT-5.5) at the top; several models under 10. A distance-stratified analysis confirms success concentrates in the shorter bins and declines with horizon even within nominally local tasks. Trajectory analysis softens the picture only slightly: more than 50% of long-range episodes still end closer to the goal than they began (up to 81.3%), and multi-stop agents reach 6.8–20.7% of required destinations on average — useful local direction, without preserved global route state.

RQ3 — plausible movement outlives effective adaptation. Dusk and night reduce local QA accuracy by lowering sign and storefront visibility, and robustness is uneven even within a family — GPT-5.5 and Gemini-3.6-Flash stay comparatively stable while GPT-5.4 and Gemini-3.1-Pro swing more. Curiously, agents often do worse at dusk than at night, possibly from dusk lighting interference. Weather and time affect navigation less systematically than QA. The sharpest result is dynamic: under a road closure, goal-reaching rate stays near zero while PNA stays high (~93–96%) and safe-progress rate remains low (10–46.7%); under moving pedestrians, the pedestrian-collision rate is high for every model (76.3–90.0%). Staying on the pedestrian network plainly does not imply collision-aware or goal-directed control.

The paper's conclusion is a single, well-supported diagnosis: urban agency cannot be inferred from isolated perception or from success over a short route. Current MLLMs possess useful local spatial skills — visual recognition and short-range reasoning — but these skills lose reliability as interaction extends through the city, and the central failure emerges over extended exploration where local abilities do not compose into sustained goal-directed behavior.

What makes the finding precise rather than merely pessimistic is the shape of the failure, which recurs at every spatial scale. Models recognize local evidence more reliably than they orient it; they execute useful local motion more reliably than they complete a route; and they preserve compliant movement more reliably than they update a plan after the city changes. Read together, these say the same thing at three resolutions: the grounded spatial state ceases to guide action toward the objective long before the agent stops producing plausible local movements. An agent can keep making locally sensible steps — staying on the pedestrian network, moving toward the estimated goal direction — for the entire episode while the goal-directed thread has already been lost.

The dynamic tasks make the gap most visible. When a route becomes invalid mid-execution, agents often continue producing locally compliant movements even after their spatial plan has become obsolete — a road-closure notice is acknowledged on the map yet the safe-progress rate stays low. When blocked by a central obstacle, an agent frequently repeats the same forward action without revising the route, unable to distinguish geometric proximity from a traversable path. This is the difference between plausible local behavior and effective adaptation, and it is exactly what a single-viewpoint benchmark can never surface.

The constructive reading is a research agenda. City-scale agency depends on maintaining a spatial estimate after the current view disappears, and revising it whenever earlier plans cease to be feasible — persistence and adaptation, the two capabilities that grounding alone does not buy. UrbanGround's contribution is to make that gap observable in a real geographic setting, where every action changes the evidence available to the agent, so future work on agents that move reliably through complex, open-ended cities has a place to measure itself. The environment ships on the web and as native macOS, Windows, and Linux builds, with the tasks released alongside.