Reference-Free Post-Training of Open Large Language Models for Multilingual Machine Translation

What they did

1. Standard translation model improvement relies on reference (correct) translations, which are scarce for low-resource languages; this work instead trains using only source sentences, without references.
2. Starting from the supervised-finetuned MiLMMT-46-v0.1 models, the team applied GRPO reinforcement learning with a reward averaging two reference-free quality-estimation models (XCOMET, COMETKiwi), gated by a language-identification check that penalizes outputs in the wrong language.
3. To counteract quality drift or reward-hacking side effects from pure RL, they linearly blended (interpolated) the SFT checkpoint and the RL checkpoint at a 50/50 ratio (alpha=0.5), producing the final MiLMMT-46-v1.0 models.
4. Across three model sizes (1B, 4B, 12B), the resulting models improved reference-free XCOMET and COMETKiwi scores on WMT24++ by an average of 2.75 and 2.44 points respectively over their SFT counterparts, and MiLMMT-46-v1.0 achieved leading reference-free scores against Seed-X, HY-MT2, TranslateGemma, and proprietary systems including Google Translate, Gemini 3 Pro, and GPT-5.
5. A separate on-policy distillation experiment transferred knowledge from the 12B teacher to 1B and 4B student models; the students approached but did not exceed the quality frontier set by RL plus checkpoint interpolation.

Findings

• Averaged across the three model scales, reference-free XCOMET and COMETKiwi on WMT24++ improved by 2.75 and 2.44 points respectively over the SFT baseline.
• On FLORES+, reference-based XCOMET improved by 1.17 points, reference-free XCOMET by 1.41 points, and COMETKiwi by 1.17 points, while spBLEU decreased by 1.21 points, relative to the SFT baseline.
• MiLMMT-46-12B-v1.0 achieved the highest reference-free XCOMET and COMETKiwi scores on WMT24++ and across all four FLORES+ direction groups, outperforming evaluated proprietary systems including Google Translate, Gemini 2.5/3 Pro, and GPT-5.
• MiLMMT-46-1B-v1.0 outperformed TranslateGemma-4B on every reported WMT24++ and FLORES+ metric despite using only a quarter of the parameters.
• On-policy distillation (RL+OPD, lambda=1) into 1B and 4B students approached but did not consistently match the corresponding MiLMMT-46-v1.0 baselines (e.g., 4B model: 34.17 vs. 33.96 spBLEU, 90.82 vs. 90.91 reference-based XCOMET on FLORES+).

Where it can be used

• Post-training translation models for low-resource language pairs using only source-side text, when reference translations are hard to obtain.
• Designing language-identification-gated reward functions in RL to reduce fluent-but-wrong-language outputs in generative translation systems.
• Using on-policy distillation as an alternative to running full reinforcement learning separately at every model scale when transferring quality gains to smaller models.
• Applying SFT-RL checkpoint interpolation to control the trade-off between lexical overlap (spBLEU) and learned quality-estimation scores in deployed systems.

Limits and open work

• Evaluation is limited to 46 languages, the WMT24++ and FLORES+ benchmarks, and the specific baseline models compared, so performance on other languages or domains is unverified.
• RL tends to lower spBLEU (word-overlap score), and checkpoint interpolation only partially recovers this rather than eliminating the trade-off.
• On-policy distillation approaches but does not surpass the quality frontier set by the 12B RL-plus-interpolation teacher, and the reasons for this gap are not further analyzed.
• The authors plan future work on more robust, human-aligned reward models, extending reference-free post-training to multimodal multilingual translation, and testing at larger model scales.
• The reference-free quality estimators (XCOMET, COMETKiwi) used for both reward and evaluation may carry their own biases, which were not independently validated in this study.

Why it matters

This shows translation quality can be improved for low-resource languages using only abundant source-side text, reducing the dependency on scarce reference-translation data. The released open models and code let other researchers and developers apply the same reference-free post-training recipe to their own languages or domains.

Terms in this paper

• GRPO (Group Relative Policy Optimization) · A reinforcement learning method that generates several candidate translations and learns from the relative differences in their scores
• reference-free quality estimation (QE) · Automatic scoring models like XCOMET and COMETKiwi that judge translation quality from source and output text alone, without a correct reference
• checkpoint interpolation · Combining two saved model weight snapshots by linearly mixing them in a given ratio to form a new model
• on-policy distillation (OPD) · A method where a larger teacher model scores the smaller student's own generated outputs to train the student
• spBLEU · A traditional translation metric measuring word/phrase overlap with a correct reference translation

Latest papers

• SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the timearXiv:2608.197992026-08-21
• FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papersarXiv:2608.197582026-08-21
• PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad actionarXiv:2608.198612026-08-21
• EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise itarXiv:2608.198912026-08-21
• EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknessesarXiv:2608.198802026-08-21
• Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversightarXiv:2608.192162026-08-21
• PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappearsarXiv:2608.197462026-08-21
• Automated Summarization of Financial News Using Large Language Models and Retrieval-Augmented Generation: An Early Empirical Study (Fall 2023)Testing AI summaries of stock news, the simple approach beat the trendy retrieval-based onearXiv:2608.195262026-08-21

Latest from METAL MEDIA

• NCT 127 drop BLINGY, their first full album in two years