Solving Is Not Drawing: A Benchmark for Diagrammatic Reasoning in Olympiad Geometry

arXiv:2608.181112026-08-20

AI models that ace olympiad geometry problems still can't draw the diagrams those problems depend on

Researchers built a benchmark of 954 olympiad geometry problems, each paired with a human-made reference diagram written in Asymptote code, to test whether AI models can draw geometrically faithful figures, not just solve for the answer. Testing Claude Sonnet 4.6, Gemini 3.1 Flash-lite, GPT-5.4-mini, Kimi K2.5, and GLM 5.1 showed that even though these models nearly saturate solving benchmarks like AIME 2025, their generated diagrams only compiled successfully 36.14% of the time on average and scored just 46.58 out of 100 on geometric fidelity as judged by a vision-language model. Giving models the full reference solution improved how closely their code resembled the reference text, but did not make the actual rendered diagrams more accurate.

METAL MEDIA explanatory visual

AI models that ace olympiad geometry problems still can't draw the diagrams those problems depend on

01Built an open dataset of 954 self-contained olympiad geometry problems (297 harder ones in a separate subset), each with a full solution and a human-authored ground-truth diagram in Asymptote, a code language built for precise geometric drawing.
02Tested five foundation models in two settings: given only the problem statement, or given the problem plus the full reference solution, then asked each to output Asymptote code for the diagram.
03Measured results with text/code overlap scores (BLEU, CodeBLEU, ROUGE-L), an image similarity score called LPIPS, a vision-language model acting as a judge, and a simple check of whether the code even compiles.
04Found compile success rates averaged only 36.14%, and even the best VLM-judged fidelity score never exceeded about 53 out of 100, despite these same models scoring 87-100% on solving AIME 2025 problems.
05Found that models with the highest compile rates were not the ones producing the most geometrically accurate diagrams, and that providing the reference solution boosted code-text similarity but not actual diagram correctness.

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

Built an open dataset of 954 self-contained olympiad geometry problems (297 harder ones in a separate subset), each with a full solution and a human-authored ground-truth diagram in Asymptote, a code language built for precise geometric drawing.
Tested five foundation models in two settings: given only the problem statement, or given the problem plus the full reference solution, then asked each to output Asymptote code for the diagram.
Measured results with text/code overlap scores (BLEU, CodeBLEU, ROUGE-L), an image similarity score called LPIPS, a vision-language model acting as a judge, and a simple check of whether the code even compiles.
Found compile success rates averaged only 36.14%, and even the best VLM-judged fidelity score never exceeded about 53 out of 100, despite these same models scoring 87-100% on solving AIME 2025 problems.
Found that models with the highest compile rates were not the ones producing the most geometrically accurate diagrams, and that providing the reference solution boosted code-text similarity but not actual diagram correctness.

Figure 1: Overview of the Data Curation and Evaluation Processes. We web-scrape high-quality mathematical olympiad problems, namely from national and international competitions with well-documented solutions. We then extract human-made Asymptote code to generate the diagrams corresponding to each solution. To create our dataset, we filter our data, eliminating problems that are not self-contained or not geometric. We then prompt GPT-5.4-mini to generate a difficulty score and a summary of each solution’s approach for further analysis. Using a difficulty cutoff, we curate a subset of harder geometry problems. We evaluate foundation models on a range of metrics with two different experimental settings: 1) a baseline with solely the problem statement and 2) including the full reference solution and approach.

Table 1: Experimental Results (Mean Score Scaled to 100). LPIPS is lower-is-better. VLM Critic raw scores were on a 0–10 scale and have been linearly rescaled to 0–100 for consistency. “With” denotes that the model was given the full reference solution and approach as context; “Without” denotes that the model was given only the problem statement.
Method	GPT5.4-mini	Claude Sonnet 4.6	Gemini 3.1 Flash-lite	Kimi K 2.5	Glm 5.1
	With	Without	With	Without	With	Without	With	Without	With	Without
Text/Code-Based Metrics
↑ Keyword Match F1	55.4	51.4	55.9	52.8	47.4	42.1	49.4	47.9	54.6	51.5
↑ BLEU	18.5	16.8	15.7	15.3	16.6	13.9	19.1	16.4	17.8	16.3
↑ chrF++	35.7	32.9	34.7	32.4	30.5	27.2	33.8	30.1	35.1	31.9
↑ CodeBLEU	17.1	15.1	15.1	14.3	14.8	12.4	17.7	15.4	16.4	15.0
↑ ROUGE-L	22.4	21.6	20.2	20.2	24.8	23.4	24.9	24.0	22.6	21.9
Image-Based Metrics
↓ LPIPS	54.0	53.7	50.6	51.3	52.6	52.7	47.9	49.5	50.8	53.3
VLM-as-a-Judge
↑ VLM Critic	37.9	46.4	49.5	51.0	42.6	45.2	50.8	53.2	44.2	45.0
Compilation Metrics
↑ Compile Success Rate	21.7	20.8	27.6	41.3	59.2	59.3	30.9	22.0	40.0	38.6

Figure 2: Mean VLM judge score against VLM-evaluable rate for each model and prompt setting (circle: with solution; diamond: no solution). Dashed lines mark the across-run means. Higher evaluable rates do not correspond to higher judge scores: the most reliably renderable models score below the mean on geometric fidelity, while the best-judged runs render less often.

Table 2: Final Dataset Breakdown based on Difficulty
Dataset	2	3	4	5	6	7	8	9	10	Total
AMC	14	29	91	95	84	51	15	0	0	379
AIME	0	13	63	83	116	105	52	8	0	440
USAJMO	0	0	2	5	6	3	5	0	0	21
USAMO	0	1	3	3	5	14	15	6	1	48
IMO	0	0	0	1	1	3	1	2	0	8
Others	0	5	12	12	16	9	4	0	0	58
Total	14	48	171	199	225	188	92	16	1	954

Figure 3: Change in each metric when the full reference solution is provided, relative to the no-solution setting (with minus without), per model. Left: text- and code-similarity metrics improve uniformly. Right: rendered-image metrics change little and often negatively, indicating that solution context raises code overlap without improving diagram fidelity.

Why it matters

Many geometry problems depend on getting the diagram right, so a model that reasons correctly but draws an inconsistent figure can't be trusted for tasks like verifying constructions or feeding diagrams to other tools. This shows that scoring well on math benchmarks doesn't mean a model actually understands or can construct the geometric relationships behind a problem.

Figure 5: Breakdown of Problems by Difficulty Score

Terms in this paper

Asymptote · 기하학적 도형을 정밀하게 그리기 위해 설계된 프로그래밍 언어이자 벡터 그래픽 도구
VLM judge · 이미지와 텍스트를 함께 이해하는 비전-언어 모델을 심사위원처럼 활용해 결과물의 품질을 점수로 매기는 평가 방식
LPIPS · 두 이미지가 사람 눈에 얼마나 비슷해 보이는지를 딥러닝 특징을 이용해 측정하는 지표, 값이 낮을수록 유사함
BLEU/CodeBLEU/ROUGE-L · 생성된 텍스트나 코드가 정답과 얼마나 겹치는지 단어·문자 조각 단위로 비교하는 점수들
Compile Success Rate · 생성된 코드가 오류 없이 실제 이미지로 변환되는 비율

Figure 6: Breakdown of All Problems by Difficulty Score

Figures we cannot republish

Figure 4: Overview of the Problems Filtered during the Dataset Curation Process. We web-scraped a total of 1441 raw datapoints, before filtering out the problems that were not geometric and not self-contained.

See the figures in the original paper →

Original abstract (English)

Foundation models such as GPT and Claude now solve olympiad-level mathematics with remarkable proficiency, so much so that geometry problem solving has become a standard proxy for their mathematical reasoning. Yet solving a geometry problem and drawing the figure it depends on are not the same skill: progress often hinges on a faithful diagram with the right auxiliary constructions and incidences, and it is unclear that a model which reasons its way to the answer can also produce one. A growing collection of benchmarks, including MathVista, and MathVerse, measures whether models reach the correct answer, but to our knowledge, none isolate the distinct ability to construct the diagram itself, leaving this capability unmeasured. We introduce an open-source benchmark that targets this gap: 954 self-contained olympiad geometry problems, with a 297-problem hard subset, each paired with its solution and a human-authored, high-fidelity diagram in renderable Asymptote code, together with a suite of text-, code-, image-, VLM-, and constraint-based metrics for what we term diagrammatic reasoning. Evaluating current foundation models reveals a pronounced gap between solving and drawing: their diagrams are markedly less faithful, with an average compile success rate of only 36.14\%. Strong mathematical reasoning, we find, does not imply the ability to construct accurate geometric diagrams. Our benchmark and dataset can be accessed at https://huggingface.co/datasets/max98765/hard_geometry_problems_with_diagrams.

Authors · Hsien Xin Peng, Anthony Kim, Alvin Li, Calvin Supasanya, Shivank Garg, Kevin Zhu

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

NCT 127 drop BLINGY, their first full album in two years

Figures: Hsien Xin Peng et al., arXiv:2608.18111, CC BY-SA 4.0