K-文化的一切——从回归到 K-美妆，发送到您的邮箱订阅邮件

METAL MEDIA

360CityArena: A Realistic Virtual Urban Navigation Benchmark for Embodied Agents

arXiv:2608.088142026-08-08

用360度视频复原东京秋叶原打造的城市导航测试场里,表现最好的AI只拿到人类五分之一左右的分数

360CityArena是一个基于602段360度视频拼接而成的东京秋叶原逼真虚拟街区,包含175个人工设计的任务,用来考察AI智能体在真实城市环境中的理解、导航和空间推理能力。人类参与者平均正确率达77.3%,而测试中表现最好的AI模型Gemini 2.5 Flash只有17.1%。AI在地图导航、物体计数和地标之间的空间关系推理上尤其吃力。

METAL MEDIA 解读图

360CityArena的构建与测试流程

证据状态已报告实测结果

  1. 搭建虚拟城市将602段360度视频拼接成由193个节点、305条连接组成的可通行路网,复原秋叶原85条街道
  2. 设计175个任务环境理解、路径推理、空间推理三大类共七个子类型,并标注简单、中等、困难三档难度
  3. 运行AI智能体GPT-5、Claude Sonnet 4.5、Gemini 2.5 Flash、Qwen2.5-VL、InternVL3.5在Unity环境中结合视觉输入、地图和记忆逐步行动
  4. 测量人类基线五名曾到访秋叶原的参与者完成相同任务,平均正确率达77.3%
  5. 对比并诊断失败原因最佳AI(17.1%)远落后于人类(77.3%),失败被归类为动作、定位、感知、探索、规划五种类型
这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

  1. 现有的室外测试环境要么不够逼真(传统3D模拟器),要么虽然逼真但无法连续移动、缺少动态元素(谷歌街景),要么视频片段太短无法覆盖大范围街道网络,因此作者把602段360度视频拼接成一个由193个节点、305条连接构成的可通行路网,复原了秋叶原85条街道(南北约750米、东西约650米)。
  2. 在这个环境之上,团队手工设计了175个任务,分为环境理解、路径推理、空间推理三大类共七个子类型,包括自我定位、用语言或图片寻找地标、按地图导航、按语言指令导航、推理地标之间的相对位置、数物体数量,并按简单、中等、困难三档标注难度。
  3. 他们让GPT-5、Claude Sonnet 4.5、Gemini 2.5 Flash等商用AI模型以及Qwen2.5-VL、InternVL3.5等开源模型充当智能体在这个环境中执行任务,并与五名曾实地到访过秋叶原的人类参与者的表现进行对比。
  4. 人类在地图导航、图片寻找地标、语言指令导航、空间关系推理这几项任务上正确率约达90%,而所有测试的AI模型在各项任务上都远远落后于人类,且随着任务难度上升,AI的表现明显下滑。
  5. 给AI智能体额外提供当前位置信息(地图上标出的位置标记)并没有稳定提升表现,有些任务反而变差,这说明AI模型难以把地图信息和眼前看到的实际画面对应起来。
Figure 1: 360CityArena. We introduce a benchmark for evaluating embodied agents in a photorealistic reconstruction of Akihabara, Tokyo, Japan, built from interconnected 360° video trajectories. The benchmark covers realistic urban streets and evaluates agents on diverse tasks requiring environment understanding, path reasoning, and spatial reasoning.
Figure 1: 360CityArena. We introduce a benchmark for evaluating embodied agents in a photorealistic reconstruction of Akihabara, Tokyo, Japan, built from interconnected 360° video trajectories. The benchmark covers realistic urban streets and evaluates agents on diverse tasks requiring environment understanding, path reasoning, and spatial reasoning.
Figure 2: Examples in each task type in 360CityArena. (i) Environment Understanding tasks include Localization, Landmark Search with Language, and Landmark Search with Image, where the agent infers its location or navigates to a specified landmark. (ii) Path Reasoning tasks evaluate the agent’s ability to plan and execute routes, such as following map-based paths or vision-language navigation. (iii) Spatial Reasoning tasks assess relational understanding and quantitative perception, including identifying spatial relations between landmarks and counting objects in the environment.
Figure 2: Examples in each task type in 360CityArena. (i) Environment Understanding tasks include Localization, Landmark Search with Language, and Landmark Search with Image, where the agent infers its location or navigates to a specified landmark. (ii) Path Reasoning tasks evaluate the agent’s ability to plan and execute routes, such as following map-based paths or vision-language navigation. (iii) Spatial Reasoning tasks assess relational understanding and quantitative perception, including identifying spatial relations between landmarks and counting objects in the environment.
Table 1: Comparison of urban navigation environments – realism, structural complexity, dynamics, interactivity, and exploration capability.
EnvironmentCategoryPhoto- realismStructural complexityDynamicsInteractionDistrict-scale explorationMotion
EmbodiedCity [12]3D simulatorLowLowMedium✓✓Continuous
MetaUrban [44]3D simulatorLowMediumMedium✓✓Continuous
CARLA [9]3D simulatorLowLowLow✓✓Continuous (clip)
Vid2Sim [45]Video-to-simHighHighMedium✓✗Continuous
StreetLearn [30]GSV-basedHighHighLow✗✓Discrete
360CityArena (Ours)360° videoHighHighHigh✗✓Continuous (trajectory)
Figure 3: Example of the visual observations in 360CityArena. The agent moves through the city while scanning its surroundings. At branching points, it must choose a route.
Figure 3: Example of the visual observations in 360CityArena. The agent moves through the city while scanning its surroundings. At branching points, it must choose a route.
Figure 4: Success rate by difficulty level across tasks and models (%). This graph shows model and human performance for each task, divided into three difficulty levels: Easy (E), Medium (M), and Hard (H).
Figure 4: Success rate by difficulty level across tasks and models (%). This graph shows model and human performance for each task, divided into three difficulty levels: Easy (E), Medium (M), and Hard (H).
Table 2: Overall Results (%). Comparison of model and human performance across seven spatial and reasoning tasks, grouped into three major categories. Gemini 2.5 Flash achieves the highest overall performance, while the strongest model varies across individual tasks. All LMMs still fall far short of human performance.
Environment UnderstandingPath ReasoningSpatial Reasoning
LocLandmark (Lang)Landmark (Img)Map NavVLNObj CountRel Reason
GPT-58.016.048.00.08.02.432.0
Claude Sonnet 4.54.04.016.04.04.010.88.0
Gemini 2.5 Flash12.028.036.00.08.024.012.0
Qwen2.5-VL-32B-Instruct4.016.020.00.00.018.84.0
InternVL3.5-8B4.020.020.00.012.02.80.0
InternVL3.5-38B0.016.00.00.012.07.24.0
Human68.064.092.092.088.045.292.0
Figure 5: Failure cause breakdown by task category across models (%). This figure shows the distribution of failure causes for each model across three task categories. Failures are categorized into five types: Action, Grounding, Perception, Explore, and Planning, and each stacked bar reports the percentage breakdown within the corresponding (model, task category) setting.
Figure 5: Failure cause breakdown by task category across models (%). This figure shows the distribution of failure causes for each model across three task categories. Failures are categorized into five types: Action, Grounding, Perception, Explore, and Planning, and each stacked bar reports the percentage breakdown within the corresponding (model, task category) setting.
Figure 6: Example of the agent’s views for Landmark Search with Language, resulting in failure. The agent is instructed to search for “Jonathan”. From t = 8 to t = 38, the gaze repeatedly moved up and down. At t = 39, the gaze briefly shifted to the right, but from t = 41 onward, it returned to the same up–down movement.
Figure 6: Example of the agent’s views for Landmark Search with Language, resulting in failure. The agent is instructed to search for “Jonathan”. From t = 8 to t = 38, the gaze repeatedly moved up and down. At t = 39, the gaze briefly shifted to the right, but from t = 41 onward, it returned to the same up–down movement.
Table 3: Comparison with and without location information. We observe that the performance did not improve consistently across tasks; in some cases, accuracy decreased instead.
Environment UnderstandingPath ReasoningSpatial Reasoning
LocLandmark (Lang)Landmark (Img)Map NavVLNObj CountRel Reason
GPT-58.016.048.00.08.02.432.0
GPT-5 (w/o location)-24.044.04.08.00.048.0
t = 8. Look above.
t = 8. Look above.
t = 40. Look to the right.
t = 40. Look to the right.

研究结果

  • 人类参与者平均正确率为77.3%,而测试中表现最好的AI模型(Gemini 2.5 Flash)只有17.1%,差距很大。
  • 用图片寻找地标的任务总体上比用语言描述寻找更容易(例如GPT-5为48.0%对16.0%,Claude为16.0%对4.0%,Qwen为20.0%对16.0%),但InternVL系列模型在用图片时没有明显提升(8B为20.0%对20.0%,38B为0.0%对16.0%)。
  • 随着任务难度从简单升到困难,AI的表现下降,例如GPT-5在环境理解任务上的正确率从28.0%降到25.0%再降到18.5%,在路径推理任务上从11.1%降到0.0%再到0.0%。
  • 给AI额外提供位置信息后表现并未一致提升:用语言寻找地标的正确率从24.0%降到16.0%,关系性空间推理从48.0%降到32.0%,而物体计数只有小幅改善。
  • 失败原因分析显示,GPT-5的低层动作错误较少(12%)但探索策略失败占主导(55%);Gemini 2.5 Flash和InternVL3.5-38B主要在低层动作控制上出问题(约40%);所有模型在空间推理任务中忽略细微视觉信息导致的感知失败都明显增多(Gemini达38%)。
t = 41. Look below.
t = 41. Look below.
Figure 7: Example of the agent’s views for Landmark Search with Image, resulting in success. The agent is instructed to search for “Jonathan” given in image form.
Figure 7: Example of the agent’s views for Landmark Search with Image, resulting in success. The agent is instructed to search for “Jonathan” given in image form.

可应用场景

  • 在真实部署之前,用贴近实景的条件预先测试城市导览机器人或为视障人士引路的AI助手的表现。
  • 研究AI智能体如何将地图信息与眼前实际场景对应起来,以及比较其对文字指令和图片指令的理解差异。
  • 作为构建逼真虚拟城市训练数据、用于训练具身AI模型的方法参考案例。
t = 5. Go past.
t = 5. Go past.
t = 16. Spot blue and white vertical stripes.
t = 16. Spot blue and white vertical stripes.

局限与待验证事项

  • 由于环境由预先录制的360度视频路径构成,智能体只能沿拍摄过的路线移动,无法自由到达任意位置,也无法与物体进行物理交互。
  • 目前只覆盖了东京秋叶原这一个街区,尚未验证结论是否能推广到其他城市或地区。
  • 跨越视频片段边界时画面可能出现不连贯,但作者在失败原因分析中报告这并非主要失败原因。
  • 人类基线仅来自五名熟悉秋叶原的参与者,因此该数据代表的是熟悉当地情况的人的表现上限,而非一般人群的平均水平。
  • 将地图位置信息与真实视觉线索对齐的能力尚未被单独评估或训练,留待未来研究。

为什么重要

要打造能在真实城市里为人提供帮助的导览机器人或为视障人士引路的AI助手,就需要贴近真实街道的测试环境,而360CityArena在接近实景的条件下具体暴露出当前AI的短板。它尤其说明,把地图信息和第一人称视角画面对齐、以及有策略地探索而不是原地打转,是当前AI模型明显欠缺的能力。

本文术语

  • LMM(大型多模态模型) · 能够同时理解图像和文本并作出回应的大型AI模型
  • 位姿图(pose graph) · 把360度视频拍摄点按位置和朝向连接起来形成的可通行路径网络
  • 视觉-语言导航(VLN) · 要求智能体阅读并执行多步骤自然语言指令来移动的任务
  • 平均相对准确率(MRA) · 针对数值估计任务、综合考虑误差大小并在多个容差标准下取平均的评分方式
  • 具身智能体 · 通过摄像头等传感器感知环境并在环境中采取行动的AI系统

论文原文摘要（英文）

We present 360CityArena, a benchmark for evaluating the urban exploration capabilities of embodied agents within a photorealistic environment constructed from 360-degree videos. Existing outdoor benchmarks either lack sufficient photorealism or complexity, resulting in a considerable gap from real-world urban environments. 360CityArena is built on a realistic reconstruction of the Akihabara district in Tokyo, Japan, using 602 360-degree video segments covering 85 streets, and consists of 175 meticulously human-crafted tasks. It encompasses three task categories: Environment Understanding, Path Reasoning, and Spatial Reasoning, covering fundamental abilities required for urban exploration, such as localization, landmark search, path planning, and relational spatial reasoning, thereby enabling comprehensive evaluation in realistic urban scenes. Our evaluation using state-of-the-art LMM-based agents shows that even the strongest model, Gemini 2.5 Flash, performs far below human level (human: 77.3% vs. Gemini 2.5 Flash: 17.1%), revealing substantial challenges that remain in city-scale embodied navigation and reasoning. 360CityArena provides a necessary and challenging testbed for photorealistic urban-district navigation and spatial reasoning.

作者 · Kenta Watanabe

在 arXiv 阅读

最新论文

全部论文 →

METAL MEDIA 最新报道

图片来源： Kenta Watanabe et al., arXiv:2608.08814, arxiv-nonexclusive