K-文化的一切——从回归到 K-美妆，发送到您的邮箱订阅邮件

METAL MEDIA

Solving Is Not Drawing: A Benchmark for Diagrammatic Reasoning in Olympiad Geometry

arXiv:2608.181112026-08-20

能解出奥数几何题的AI,却画不出题目本身依赖的图形

研究者构建了一个包含954道奥数几何题的基准数据集,每道题都配有人工绘制的Asymptote代码版标准图形,用来检验AI模型除了能解出答案外,是否也能准确画出题目对应的几何图形。测试Claude Sonnet 4.6、Gemini 3.1 Flash-lite、GPT-5.4-mini、Kimi K2.5和GLM 5.1后发现,尽管这些模型在AIME 2025等解题基准上几乎接近满分,但它们生成的图形代码平均只有36.14%能成功编译,视觉语言模型评委给出的几何保真度平均分也仅为46.58分(满分100)。即使提供完整的参考解答,也只能让生成代码在文本层面更接近标准答案,并不能让实际渲染出的图形更准确。

METAL MEDIA 解读图

能解出奥数几何题的AI,却画不出题目本身依赖的图形

  1. 01构建了一个开源数据集,包含954道自成体系的奥数几何题(其中297道为更难的子集),每题都配有完整解答和人工制作的标准图形,图形以专为精确几何作图设计的代码语言Asymptote呈现。
  2. 02在两种设置下测试了五个基础模型:只给题目本身,或给题目加完整参考解答,然后要求模型输出绘制图形的Asymptote代码。
  3. 03使用文本/代码重叠度指标(BLEU、CodeBLEU、ROUGE-L)、图像相似度指标LPIPS、以视觉语言模型充当评委的打分方式,以及代码是否能成功编译的检验方式进行评估。
  4. 04发现代码平均编译成功率仅为36.14%,即便是评分最高的模型,视觉语言模型评委给出的保真度分数也从未超过约53分(满分100),而这些模型在AIME 2025解题任务上的准确率却高达87%到100%。
  5. 05发现编译成功率最高的模型并非几何图形最准确的模型,提供参考解答虽然提升了代码与标准文本的相似度,却没有真正提升图形本身的正确性。
这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

  1. 构建了一个开源数据集,包含954道自成体系的奥数几何题(其中297道为更难的子集),每题都配有完整解答和人工制作的标准图形,图形以专为精确几何作图设计的代码语言Asymptote呈现。
  2. 在两种设置下测试了五个基础模型:只给题目本身,或给题目加完整参考解答,然后要求模型输出绘制图形的Asymptote代码。
  3. 使用文本/代码重叠度指标(BLEU、CodeBLEU、ROUGE-L)、图像相似度指标LPIPS、以视觉语言模型充当评委的打分方式,以及代码是否能成功编译的检验方式进行评估。
  4. 发现代码平均编译成功率仅为36.14%,即便是评分最高的模型,视觉语言模型评委给出的保真度分数也从未超过约53分(满分100),而这些模型在AIME 2025解题任务上的准确率却高达87%到100%。
  5. 发现编译成功率最高的模型并非几何图形最准确的模型,提供参考解答虽然提升了代码与标准文本的相似度,却没有真正提升图形本身的正确性。
Figure 1: Overview of the Data Curation and Evaluation Processes. We web-scrape high-quality mathematical olympiad problems, namely from national and international competitions with well-documented solutions. We then extract human-made Asymptote code to generate the diagrams corresponding to each solution. To create our dataset, we filter our data, eliminating problems that are not self-contained or not geometric. We then prompt GPT-5.4-mini to generate a difficulty score and a summary of each solution’s approach for further analysis. Using a difficulty cutoff, we curate a subset of harder geometry problems. We evaluate foundation models on a range of metrics with two different experimental settings: 1) a baseline with solely the problem statement and 2) including the full reference solution and approach.
Figure 1: Overview of the Data Curation and Evaluation Processes. We web-scrape high-quality mathematical olympiad problems, namely from national and international competitions with well-documented solutions. We then extract human-made Asymptote code to generate the diagrams corresponding to each solution. To create our dataset, we filter our data, eliminating problems that are not self-contained or not geometric. We then prompt GPT-5.4-mini to generate a difficulty score and a summary of each solution’s approach for further analysis. Using a difficulty cutoff, we curate a subset of harder geometry problems. We evaluate foundation models on a range of metrics with two different experimental settings: 1) a baseline with solely the problem statement and 2) including the full reference solution and approach.
Table 1: Experimental Results (Mean Score Scaled to 100). LPIPS is lower-is-better. VLM Critic raw scores were on a 0–10 scale and have been linearly rescaled to 0–100 for consistency. “With” denotes that the model was given the full reference solution and approach as context; “Without” denotes that the model was given only the problem statement.
MethodGPT5.4-miniClaude Sonnet 4.6Gemini 3.1 Flash-liteKimi K 2.5Glm 5.1
WithWithoutWithWithoutWithWithoutWithWithoutWithWithout
Text/Code-Based Metrics
↑ Keyword Match F155.451.455.952.847.442.149.447.954.651.5
↑ BLEU18.516.815.715.316.613.919.116.417.816.3
↑ chrF++35.732.934.732.430.527.233.830.135.131.9
↑ CodeBLEU17.115.115.114.314.812.417.715.416.415.0
↑ ROUGE-L22.421.620.220.224.823.424.924.022.621.9
Image-Based Metrics
↓ LPIPS54.053.750.651.352.652.747.949.550.853.3
VLM-as-a-Judge
↑ VLM Critic37.946.449.551.042.645.250.853.244.245.0
Compilation Metrics
↑ Compile Success Rate21.720.827.641.359.259.330.922.040.038.6
Figure 2: Mean VLM judge score against VLM-evaluable rate for each model and prompt setting (circle: with solution; diamond: no solution). Dashed lines mark the across-run means. Higher evaluable rates do not correspond to higher judge scores: the most reliably renderable models score below the mean on geometric fidelity, while the best-judged runs render less often.
Figure 2: Mean VLM judge score against VLM-evaluable rate for each model and prompt setting (circle: with solution; diamond: no solution). Dashed lines mark the across-run means. Higher evaluable rates do not correspond to higher judge scores: the most reliably renderable models score below the mean on geometric fidelity, while the best-judged runs render less often.
Table 2: Final Dataset Breakdown based on Difficulty
Dataset12345678910Total
AMC01429919584511500379
AIME001363831161055280440
USAJMO000256350021
USAMO00133514156148
IMO00001131208
Others005121216940058
Total0144817119922518892161954
Figure 3: Change in each metric when the full reference solution is provided, relative to the no-solution setting (with minus without), per model. Left: text- and code-similarity metrics improve uniformly. Right: rendered-image metrics change little and often negatively, indicating that solution context raises code overlap without improving diagram fidelity.
Figure 3: Change in each metric when the full reference solution is provided, relative to the no-solution setting (with minus without), per model. Left: text- and code-similarity metrics improve uniformly. Right: rendered-image metrics change little and often negatively, indicating that solution context raises code overlap without improving diagram fidelity.

为什么重要

许多几何题的解法依赖于图形是否画得准确,一个推理正确但画出的图形前后矛盾的模型,无法被信任用于验证作图或向其他工具提供可靠图形。这说明数学解题基准分数高,并不代表模型真正理解或能够构建出题目背后的几何关系。

Figure 5: Breakdown of Problems by Difficulty Score
Figure 5: Breakdown of Problems by Difficulty Score

本文术语

  • Asymptote · 一种专为精确绘制几何图形设计的编程语言和矢量图形工具
  • 视觉语言模型(VLM)评委 · 利用能同时理解图像和文本的模型充当评委,为生成结果的质量打分的评估方式
  • LPIPS · 利用深度学习特征衡量两张图片在人眼看来相似程度的指标,数值越低表示越相似
  • BLEU/CodeBLEU/ROUGE-L · 通过比较词语或字符片段的重叠程度来衡量生成文本或代码与标准答案相似度的一组指标
  • 编译成功率 · 生成的代码能够无错误地渲染成图像的比例
Figure 6: Breakdown of All Problems by Difficulty Score
Figure 6: Breakdown of All Problems by Difficulty Score

无法转载的图表

  • Figure 4: Overview of the Problems Filtered during the Dataset Curation Process. We web-scraped a total of 1441 raw datapoints, before filtering out the problems that were not geometric and not self-contained.
在原文中查看图表 →

论文原文摘要（英文）

Foundation models such as GPT and Claude now solve olympiad-level mathematics with remarkable proficiency, so much so that geometry problem solving has become a standard proxy for their mathematical reasoning. Yet solving a geometry problem and drawing the figure it depends on are not the same skill: progress often hinges on a faithful diagram with the right auxiliary constructions and incidences, and it is unclear that a model which reasons its way to the answer can also produce one. A growing collection of benchmarks, including MathVista, and MathVerse, measures whether models reach the correct answer, but to our knowledge, none isolate the distinct ability to construct the diagram itself, leaving this capability unmeasured. We introduce an open-source benchmark that targets this gap: 954 self-contained olympiad geometry problems, with a 297-problem hard subset, each paired with its solution and a human-authored, high-fidelity diagram in renderable Asymptote code, together with a suite of text-, code-, image-, VLM-, and constraint-based metrics for what we term diagrammatic reasoning. Evaluating current foundation models reveals a pronounced gap between solving and drawing: their diagrams are markedly less faithful, with an average compile success rate of only 36.14\%. Strong mathematical reasoning, we find, does not imply the ability to construct accurate geometric diagrams. Our benchmark and dataset can be accessed at https://huggingface.co/datasets/max98765/hard_geometry_problems_with_diagrams.

作者 · Hsien Xin Peng, Anthony Kim, Alvin Li, Calvin Supasanya, Shivank Garg, Kevin Zhu

在 arXiv 阅读

最新论文

全部论文 →

METAL MEDIA 最新报道

图片来源： Hsien Xin Peng et al., arXiv:2608.18111, CC BY-SA 4.0