A research team from the National University of Singapore, the University of Oxford, Zhejiang University, and Beijing University of Posts and Telecommunications has introduced GeoWorld and LiveGeo, a platform and benchmark designed to evaluate autonomous large language model agents in continuously changing geographic environments. Submitted to arXiv on November 12, 2026 (arXiv:2611.17267), the benchmark moves beyond traditional static map evaluations by testing whether AI agents can actively collect spatial evidence, verify whether prior assumptions remain accurate over time, and adapt itineraries when urban conditions fluctuate.
Moving Beyond Static Spatial Reasoning in AI
Geographic spatial reasoning has emerged as a major benchmark frontier for autonomous AI agents, yet existing evaluation suites such as MapEval, CityBench, and TravelPlanner test models against static snapshots of maps, routes, and points of interest. In practical real-world operations—such as autonomous delivery, fieldwork logistics, emergency evacuation planning, and campus transport—geographic environments are never frozen. Traffic jams develop, venues close unexpectedly, transit services experience sudden delays, and temporary safety curfews alter route viability. When AI agents plan exclusively against static spatial graphs, their calculated routes quickly become invalid in production settings.
To systematically address this evaluation gap within AI and ML research outputs, the researchers developed GeoWorld, an extensible simulation platform grounded in real-world geographic structures. GeoWorld simulates dynamic spatial systems across six global metropolitan centers: Shanghai, Tokyo, Singapore, London, New York, and Cape Town. Across these cities, the platform maps 88 municipal regions, 1,376 verified points of interest (POIs), and 107,372 cached route trajectories. Rather than forcing environmental changes strictly in response to user actions, GeoWorld introduces independent temporal processes: world states shift along their own timelines while maintaining comprehensive state histories for reproducible testing.
The LiveGeo Benchmark: Structure and Evaluation Categories
Built on top of GeoWorld, the LiveGeo benchmark introduces over 1,200 verifiable task instances spanning atomic operations, compositional tasks, and dynamic real-time evaluation. The benchmark is structured into three distinct tiers to isolate specific agent failure modes:
- Atomic Geographic Tasks: Evaluate core spatial primitives. These comprise Grounding (resolving multimodal or textual clues to exact spatial entities, with 132 validated instances), Discovery (identifying destinations that satisfy complex intent constraints, with 171 instances), and Mobility Planning (routing feasible origin-destination trips across multimodal transit networks, with 175 instances). Across all atomic tasks, instances achieved human verification pass rates ranging between 89.5% and 96.4%.
- Compositional Geographic Tasks: Test multi-stage spatial problem solving. These include Multi-Entity coordination (handling joint spatial constraints across multiple venues, with 176 instances), Long-Horizon planning (scheduling multi-stop itineraries under strict time and opening-hour constraints, with 150 instances), Safety-Aware traversal (avoiding active incident perimeters and hazard zones, with 121 instances), and Region-Aware transit (managing inter-district crossings and administrative boundaries, with 163 instances).
- Dynamic Evaluation: Tests online adaptation across two operational settings: Shared dynamics, where global disruptions (such as weather events or transit strikes) occur on an external schedule independent of agent progress, and Task-Only dynamics, where environmental disruptions directly intersect the agent’s planned trajectory.
This rigorous separation allows researchers to determine whether an agent fails due to basic spatial ignorance or due to brittle plan revision mechanisms, establishing a new paradigm for how a synthetic benchmark can simulate complex physical environments.
Experimental Findings: Frontier Models Falter in Dynamic Settings
The research team conducted extensive evaluations on nine leading frontier language models: GPT-5.6 Sol, Claude Opus 5, Claude Sonnet 5, Gemini 3.1 Pro Preview, Gemini 3.7 Flash, Grok 4.5, Qwen3.8-Max, Kimi K3, and DeepSeek-V4-Flash-Vision-Exp. The experimental results reveal substantial shortcomings across the board, demonstrating that high scores on conventional static benchmarks fail to predict operational reliability in dynamic settings.
On static atomic tasks, top-performing models achieved modest success rates. In the Grounding category, Gemini 3.7 Flash achieved 37.1% accuracy, followed by GPT-5.6 Sol at 31.1%, Qwen3.8-Max at 28.0%, and Claude Opus 5 and Sonnet 5 at 26.5% each. In Discovery tasks, Gemini 3.7 Flash scored 38.0%, Qwen3.8-Max reached 32.7%, and Gemini 3.1 Pro scored 30.4%. However, performance deteriorated dramatically in complex planning: on Long-Horizon compositional tasks, Gemini 3.7 Flash and Gemini 3.1 Pro achieved only 8.7% success, Grok 4.5 scored 6.0%, GPT-5.6 Sol reached 2.0%, Claude Opus 5 and Qwen3.8-Max scored 1.3%, and Claude Sonnet 5 recorded 0.0%.
The dynamic evaluation exposed an even more severe vulnerability: the near-complete collapse of agent reliability when the geographic environment changes during task execution. In baseline static mobility tasks, top models scored between 18.1% and 26.2%. When dynamic perturbations were introduced:
- GPT-5.6 Sol: Success plunged from 18.1% in static settings to 1.3% under shared dynamics and 2.7% under task-only dynamics.
- Claude Opus 5: Dropped from 21.5% static to 1.3% under shared dynamics and 4.7% under task-only dynamics.
- Claude Sonnet 5: Declined from 22.8% static to 2.0% in shared dynamics and 2.7% in task-only dynamics.
- Gemini 3.7 Flash: Demonstrated the strongest resilience among tested models, maintaining 13.4% in shared dynamic settings and 14.1% in task-only dynamics, though still dropping significantly from its 26.2% static baseline.
- Qwen3.8-Max: Fell from 22.8% static to 4.0% in shared dynamics and 4.7% in task-only dynamics.
Core Failure Modes: Evidence Decay and Plan Invalidation
Through rigorous trajectory logging and error analysis, the study identified three primary architectural bottlenecks preventing LLM agents from achieving reliable autonomous spatial behavior:
- Evidence Decay Neglect: Agents operate under a persistent assumption that once an observation is made (for instance, confirming that a road is clear or a museum is open), that fact remains permanently valid. When external conditions change mid-execution, agents rarely initiate active verification checks to confirm whether prior premises hold.
- Inability to Translate New Evidence into Valid Actions: When confronted with explicit alerts about blocked roads or closed venues, models frequently acknowledge the disruption in their internal reasoning chains yet fail to reflect it in their subsequent API tool calls, often submitting the same invalid path or making micro-adjustments that violate broader spatial constraints.
- Cascading Long-Horizon Fragility: Multi-stop itinerary planning requires continuous constraint satisfaction across time windows, operating budgets, and geographic adjacency. A single disruption at step two causes cascading failures across remaining milestones, with models frequently abandoning unvisited goals rather than dynamically re-optimizing their schedules.
Implications for University Research Offices and AI Governance
The release of GeoWorld and LiveGeo carries direct implications for academic research institutions, computer science departments, and campus research administration offices managing autonomous system deployments:
First, the benchmark highlights why research offices and procurement committees cannot rely solely on vendor-reported static benchmark scores when assessing autonomous agents for physical world deployment. Whether evaluating campus autonomous shuttles, delivery robotics, environmental sampling drones, or automated disaster response coordination, institutions must demand verification under dynamic perturbation testing.
Second, the methodology underscores the growing necessity of standardized evaluation transparency. As research institutions adopt AI governance frameworks, documenting model failure rates across changing physical parameters should become standard practice within an institution’s AI evaluation card or model reporting registry.
Third, the findings highlight the critical role of data provenance in autonomous decision systems. Because agents fail primarily when temporal validity expires, scientific workflows that rely on AI agents to gather geospatial data or monitor physical assets must integrate structured provenance ontologies, such as W3C PROV-O, ensuring that every spatial assertion records its exact timestamp, verification sensor, and valid time horizon.
By publishing the complete GeoWorld platform and LiveGeo benchmark codebases openly, the authoring team provides the academic community with an extensible testbed to build the next generation of resilient, verification-aware autonomous agents.
Sources
- Zhou, P., Liu, X., Yin, Y., Ju, T., Li, B., Wu, S., Fei, H., Lee, M.-L., and Hsu, W. (2026). LiveGeo: Benchmarking Geographic Agents in Continuously Evolving Worlds. arXiv preprint arXiv:2611.17267. Checked 2026-11-14. https://arxiv.org/abs/2611.17267
- Zhou, P., et al. (2026). LiveGeo Full HTML Text and Experimental Data (v1). arXiv. Checked 2026-11-14. https://arxiv.org/html/2611.17267v1
