Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL

What they did

1. Problem: self-rewarding RL methods like TTRL let a model grade its own answers via majority vote, but if the model is systematically wrong, it just reinforces its own mistakes and training can collapse.
2. Solution: Co-RL trains multiple separate models (no shared parameters or gradients) that each generate several answers to the same unlabeled question; one model's majority-vote answer becomes the reward target for another model's rollouts, and vice versa.
3. Diversity matters: using different model families, sizes, or slightly rephrased versions of the same problem for each model reduces overlapping errors, which the paper shows produces more accurate pseudo-labels and better final performance than pairing identical models.
4. Results: Co-RL beat prior label-free methods (TTRL, Intuitor, RENT, Co-rewarding) across seven text benchmarks and four multimodal math benchmarks, matched or beat supervised training with ground-truth rewards in several cases, and outperformed a prior multi-agent method (CoMAS) by 4.0% on average while using only two agents instead of more.
5. The paper also proves mathematically that self-grading tends to amplify whichever answer a model already favors (right or wrong), while cross-agent grading can correct errors when the two models have complementary strengths.

Why it matters

As AI reasoning surpasses what humans can reliably verify, methods that don't need labeled data become essential for continued improvement. This shows that having multiple different models supervise each other -- rather than relying on human labels or a single model's self-judgment -- can be a practical, scalable way to keep improving reasoning models.

Terms in this paper

• Reinforcement learning (RL) · A training method where a model improves by receiving reward signals for its outputs rather than direct correct-answer labels.
• Ground-truth supervision / verifiable reward · A known correct answer used to check and reward a model's output.
• Self-rewarding RL · A model grades its own outputs (e.g., via majority vote) instead of using external correct answers.
• GRPO (Group Relative Policy Optimization) · An RL algorithm that scores a group of sampled responses relative to each other, without needing a separate critic model.
• Majority vote / pseudo-label · Treating the most common answer among several sampled responses as if it were the correct answer.

Figures we cannot republish

• Figure 1: Comparison of Co-RL with prior label-free RL methods. Both TTRL and Co-rewarding derive rewards from self-generated agreement, and CoMAS scores multi-turn interactions with one of its own agents acting as judge. Co-RL instead derives rewards directly from peer votes. Beyond two agents, the votes pass along a directed ring (N=3 shown).
• Figure 2: (a) Agreement between the two models, (b) pseudo-label accuracy, and (c) evaluation performance; (d) Error overlap before RL for two pairs each from a different family, the same family, and the same model under a different seed.
• Figure 3: Overview of Co-RL with two agents. Each agent samples K responses to the same unlabeled question and generates a pseudo label with majority vote. Each rollout is then rewarded by agreement with the cohort’s pseudo label, and updates its own policy, with no sharing parameters and no gradient exchange except the cross-reward process.
• Figure 4: Training dynamics at four scales, one column per backbone (Qwen2.5-3B, Llama-3.2-3B, Qwen2.5-7B, Llama-3.1-8B). (a) MATH-500 validation accuracy, (b) standard deviation of the reward within a rollout group, normalized to its value at the first step, and (c) mean completion length. Runs marked diverged leave the plotted range.
• Figure 5: Training dynamics for Qwen2.5-VL-7B trained with InternVL3.5-8B on open-r1. (a) Evaluation accuracy, (b) mean completion length, and (c) the accuracy of the exchanged pseudo-labels together with the agreement between the two agents. Co-RL keeps improving and holds its completion length, while TTRL peaks and then degrades in both.

Latest papers

• SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the timearXiv:2608.197992026-08-21
• FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papersarXiv:2608.197582026-08-21
• PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad actionarXiv:2608.198612026-08-21
• EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise itarXiv:2608.198912026-08-21
• EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknessesarXiv:2608.198802026-08-21
• Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversightarXiv:2608.192162026-08-21
• PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappearsarXiv:2608.197462026-08-21
• Automated Summarization of Financial News Using Large Language Models and Retrieval-Augmented Generation: An Early Empirical Study (Fall 2023)Testing AI summaries of stock news, the simple approach beat the trendy retrieval-based onearXiv:2608.195262026-08-21

Latest from METAL MEDIA

• NCT 127 drop BLINGY, their first full album in two years