MathForm: Scaling Mathematical Autoformalization with Knowledge Retrieval and Verification-Guided Refinement

arXiv:2608.142212026-08-14

Retrieving the right math-library knowledge and letting a compiler point out mistakes lets a small 8B model beat 32B specialized systems at turning math text into verified code

Autoformalization, converting natural-language math statements into machine-checkable Lean 4 code, is not just translation but requires correctly mapping concepts onto Mathlib's huge library of definitions and types. The authors built MathForm, a pipeline that retrieves relevant Mathlib knowledge before generation and then iteratively fixes candidates using compiler errors and semantic-consistency feedback, producing a verified dataset of about 367K examples called FormalVerse. A model trained on this data, MathForm-8B, reaches average pass rates of 88.06% (syntax) and 72.37% (semantic consistency) across six benchmarks, beating several 32B specialized autoformalizers.

METAL MEDIA explanatory visual

Retrieving the right math-library knowledge and letting a compiler point out mistakes lets a small 8B model beat 32B specialized systems at turning math text into verified code

01Prior methods leaned on a model's memorized knowledge, causing it to cite nonexistent lemmas or violate library conventions, while typical data pipelines just filtered large batches of single-pass outputs without explaining failures
02MathForm adds a retrieval planner that pulls relevant Mathlib definitions before generation, then runs up to three rounds of refinement guided by compiler diagnostics and a semantic-consistency judge (QwQ-32B during construction, gpt-oss-120b for final evaluation)
03The resulting FormalVerse dataset (about 367K verified examples) was used to train Qwen3-8B via supervised fine-tuning followed by reinforcement learning (DAPO) to produce MathForm-8B
04On the hardest FATE-H and FATE-X benchmarks, MathForm-8B reached semantic-consistency pass rates of 63% and 37%, beating the strongest specialized baselines by 10 and 12 percentage points, with gains growing on more abstract algebra topics
05Under a controlled 100K-example comparison, models trained on FormalVerse beat those trained on other public Lean 4 datasets by up to 18.83 percentage points in semantic-consistency pass rate

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

Prior methods leaned on a model's memorized knowledge, causing it to cite nonexistent lemmas or violate library conventions, while typical data pipelines just filtered large batches of single-pass outputs without explaining failures
MathForm adds a retrieval planner that pulls relevant Mathlib definitions before generation, then runs up to three rounds of refinement guided by compiler diagnostics and a semantic-consistency judge (QwQ-32B during construction, gpt-oss-120b for final evaluation)
The resulting FormalVerse dataset (about 367K verified examples) was used to train Qwen3-8B via supervised fine-tuning followed by reinforcement learning (DAPO) to produce MathForm-8B
On the hardest FATE-H and FATE-X benchmarks, MathForm-8B reached semantic-consistency pass rates of 63% and 37%, beating the strongest specialized baselines by 10 and 12 percentage points, with gains growing on more abstract algebra topics
Under a controlled 100K-example comparison, models trained on FormalVerse beat those trained on other public Lean 4 datasets by up to 18.83 percentage points in semantic-consistency pass rate

Figure 1: Macro-average Pass@8 (%) across six benchmarks among specialized autoformalizers. MathForm-8B achieves the strongest overall performance within this category despite its smaller model size.

Table 1: Pass@8 pass rates (%) under Syntax Check (SC) and Consistency Check (CC) for specialized autoformalizers on six benchmarks. AVG is the equally weighted macro-average across all six benchmarks. For each column, the best result is shown in bold and the second best is underlined.
	AVG	FormalMATH	ProverBench	CombiBench	FATE-M	FATE-H	FATE-X
Model	SC	CC	SC	CC	SC	CC	SC	CC	SC	CC	SC	CC	SC	CC
Specialized Autoformalizers
Herald Translator-7B	64.12	27.63	95.29	47.76	78.74	37.36	77.00	5.00	70.67	54.67	42.00	15.00	21.00	6.00
Kimina-Autoformalizer-7B	73.20	34.37	99.29	76.24	96.55	56.32	95.00	16.00	77.33	44.67	43.00	8.00	28.00	5.00
Mathesis-HPO-7B	76.20	34.96	99.06	79.29	97.13	59.77	96.00	15.00	84.00	48.67	50.00	4.00	31.00	3.00
StepFun-Formalizer-7B	58.12	39.55	97.41	81.41	89.66	59.20	79.00	28.00	60.67	52.67	17.00	12.00	5.00	4.00
StepFun-Formalizer-32B	63.65	44.47	99.06	85.88	92.53	64.94	86.00	32.00	71.33	60.00	23.00	17.00	10.00	7.00
Goedel-Formalizer-V2-8B	78.24	60.08	98.82	94.12	98.28	89.66	89.00	42.00	87.33	82.67	62.00	44.00	34.00	8.00
Goedel-Formalizer-V2-32B	78.28	63.74	99.06	94.59	98.28	92.53	91.00	49.00	89.33	85.33	63.00	48.00	29.00	13.00
ReForm-8B	81.76	66.21	99.06	94.12	98.85	90.80	86.00	47.00	94.67	91.33	67.00	53.00	45.00	21.00
ReForm-32B	81.61	68.41	99.06	95.53	98.28	94.25	93.00	55.00	91.33	88.67	69.00	52.00	39.00	25.00
Ours
MathForm-8B-SFT	84.38	66.53	99.29	91.06	100.00	90.80	83.00	43.00	98.00	91.33	80.00	58.00	46.00	25.00
MathForm-8B	88.06	72.37	100.00	95.06	100.00	94.83	93.00	47.00	99.33	97.33	82.00	63.00	54.00	37.00

Figure 2: Overview of the MathForm data construction and training pipeline. The system combines Mathlib knowledge retrieval, compilation and semantic verification, and iterative refinement to generate reliable formal data, followed by trajectory reconstruction and training of MathForm-8B.

Table 2: Ablation results for SC and CC pass rates (%) of the refinement pipeline with two generators on the FATE series. Single denotes single-pass generation; BoN denotes Best-of-N; Feedback denotes feedback-only iteration; and Retrieval denotes retrieval-only single-pass generation. AVG is the equally weighted average over FATE-M, FATE-H, and FATE-X.
	AVG	FATE-M	FATE-H	FATE-X
Method	SC	CC	SC	CC	SC	CC	SC	CC
gpt-oss-120b
Single	27.33	26.43	52.00	51.30	21.00	19.00	9.00	9.00
BoN	42.67	41.10	70.00	69.30	40.00	37.00	18.00	17.00
Feedback	42.57	40.77	68.70	67.30	41.00	38.00	18.00	17.00
Retrieval	32.23	29.00	56.70	52.00	27.00	26.00	13.00	9.00
MathForm	49.67	48.00	76.00	74.00	46.00	44.00	27.00	26.00
Qwen3-235B-A22B-Thinking-2507
Single	7.43	7.43	13.30	13.30	5.00	5.00	4.00	4.00
BoN	18.77	18.77	33.30	33.30	14.00	14.00	9.00	9.00
Feedback	28.77	28.77	51.30	51.30	22.00	22.00	13.00	13.00
Retrieval	9.90	8.57	16.70	16.70	8.00	6.00	5.00	3.00
MathForm	37.57	36.23	60.70	58.70	32.00	31.00	20.00	19.00

Figure 3: Distribution of natural-language problem sources in FormalVerse.

Table 3: Pass@8 SC and CC pass rates (%) of models trained on different datasets across six benchmarks. All models are initialized from Qwen3-8B and trained on 100K examples under the same trajectory reconstruction and training configuration. AVG is the equally weighted macro-average across all six benchmarks. The best result in each column is shown in bold and the second best is underlined.
	AVG	FormalMATH	ProverBench	CombiBench	FATE-M	FATE-H	FATE-X
Training Dataset	SC	CC	SC	CC	SC	CC	SC	CC	SC	CC	SC	CC	SC	CC
NuminaMath-LEAN	66.24	41.49	99.53	85.18	96.55	72.41	87.00	25.00	69.33	49.33	35.00	16.00	10.00	1.00
FineLeanCorpus	78.25	46.53	100.00	84.47	98.85	74.71	96.00	29.00	90.67	68.00	53.00	17.00	31.00	6.00
FormalVerse	77.17	60.32	98.82	90.59	98.85	89.66	78.00	36.00	95.33	84.67	62.00	46.00	30.00	15.00

Figure 4: Numbers of natural-language-to-formal-language pairs accepted in successive refinement rounds. Later rounds contribute an additional 31.0% of all retained pairs.

Table 4: Judge-model reliability on the human-annotated semantic-consistency test set (Mean@3).
Judge Model	Accuracy	Precision	Recall	F1
gpt-oss-120b	0.8917	0.8755	0.9133	0.8940
QwQ-32B	0.8567	0.8367	0.8867	0.8609
gpt-oss-20b	0.8500	0.8142	0.9067	0.8579

Figure 5: Training dynamics during reinforcement learning. The left panel shows the training reward and its exponential moving average (EMA), while the right panel reports the Mean@3 pass rate on FATE-H over the course of training.

Table 5: Supervised fine-tuning hyperparameters.
Hyperparameter	Value
Maximum sequence length	16,384
Global batch size	128
Learning rate	2.0×10−5
Epochs	3
LR scheduler	Cosine
Warmup ratio	0.1
Precision	bf16

Figure 6: Category distribution of the natural-language mathematical problems in FormalVerse.

Table 7: Pass@8 SC and CC pass rates (%) of general-purpose LLMs and MathForm-8B across six benchmarks. AVG is the equally weighted macro-average across all six benchmarks. The best result in each column is shown in bold and the second best is underlined.
	AVG	FormalMATH	ProverBench	CombiBench	FATE-M	FATE-H	FATE-X
Model	SC	CC	SC	CC	SC	CC	SC	CC	SC	CC	SC	CC	SC	CC
General-Purpose LLMs
DeepSeek-V4-Pro	78.34	76.54	97.88	96.24	94.83	93.68	85.00	81.00	87.33	87.33	70.00	68.00	35.00	33.00
Qwen3.7-Plus	86.33	83.38	99.29	98.59	100.00	97.70	92.00	87.00	94.67	92.00	78.00	77.00	54.00	48.00
Qwen3-235B-A22B-Thinking-2507	58.94	55.37	91.29	88.24	81.03	75.29	59.00	50.00	71.33	70.67	32.00	30.00	19.00	18.00
Qwen3-32B	43.07	36.55	80.94	72.94	61.49	50.00	43.00	27.00	46.00	43.33	20.00	19.00	7.00	7.00
DeepSeek-R1-0528-Qwen3-8B	47.28	37.65	72.24	61.18	52.30	40.23	36.00	15.00	36.00	30.00	4.00	4.00	4.00	1.00
Qwen3-8B	27.25	17.06	61.88	42.12	37.93	27.59	29.00	7.00	26.67	22.67	4.00	3.00	4.00	0.00
Ours
MathForm-8B	88.06	72.37	100.00	95.06	100.00	94.83	93.00	47.00	99.33	97.33	82.00	63.00	54.00	37.00

Table 8: Human-evaluation results (%) on FATE-M and FATE-H, based on one randomly sampled candidate per problem. The best result in each column is shown in bold.
	FATE-M	FATE-H
Model	SC	CC	SC	CC
StepFun-Formalizer-32B	42.67	36.00	16.00	12.00
Goedel-Formalizer-V2-32B	75.33	68.00	38.00	27.00
ReForm-32B	78.67	74.00	50.00	41.00
MathForm-8B	86.67	76.67	54.00	42.00

Why it matters

Training AI to prove math theorems requires huge amounts of verified formal-language data that humans cannot easily produce by hand, and this work shows a more accurate automated way to generate it. It also demonstrates that a small model paired with good data and a verification pipeline can outperform much larger specialized systems, which matters for cost-efficient AI development.

Terms in this paper

Autoformalization · Automatically translating natural-language math statements into a machine-checkable formal language like Lean 4
Lean 4 / Mathlib · A proof assistant language for machine-verifiable math (Lean) and its large library of formal definitions and theorems (Mathlib)
Syntax Check (SC) / Consistency Check (CC) · SC checks whether generated code compiles correctly; CC checks whether it truly preserves the meaning of the original statement
Pass@8 · A metric measuring how often a model succeeds within 8 attempts per problem
DAPO / Reinforcement Learning (RL) · A training method that adjusts a model using reward signals comparing multiple generated candidates to favor better ones

Original abstract (English)

Autoformalization is commonly framed as translating natural-language mathematical statements into machine-verifiable formal languages such as Lean 4. However, faithful formalization requires more than translation. Models must map mathematical concepts to the complex hierarchy of types and definitions in formal libraries such as Mathlib, while ensuring that generated statements preserve the meaning of the source propositions. Existing approaches struggle because they rely heavily on the model's parametric memory for library-specific knowledge, while common data construction pipelines often resort to filtering single-pass outputs and lack mechanisms for feedback-driven revision. To address these challenges, we introduce MathForm, an autoformalization framework for constructing verified training data through Mathlib knowledge retrieval and verification-guided iterative refinement. Before generation, a retrieval planner gathers relevant definitions and existing formalizations from Mathlib to guide the formalization generator. Generated statements are then revised using compiler diagnostics and semantic-consistency feedback. Using this framework, we construct FormalVerse, a Lean 4 dataset containing approximately 367K verified examples across diverse mathematical domains and sources. We then train MathForm-8B through supervised fine-tuning followed by reinforcement learning. Across six benchmarks, MathForm-8B achieves average Pass@8 rates of 88.06% under Syntax Check (SC) and 72.37% under Consistency Check (CC), outperforming multiple specialized 32B autoformalizers. On the challenging FATE-H and FATE-X subsets, it attains CC pass rates of 63% and 37%, exceeding the strongest specialized baselines in both cases.

Authors · Lushi Pu

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Lushi Pu et al., arXiv:2608.14221, CC BY 4.0