Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL

arXiv:2608.172532026-08-18

没有标准答案标签,让两三个不同的AI模型互相打分,也能提升推理能力

用强化学习训练语言模型或视觉语言模型通常需要标准答案来判断对错,但这类标注成本高且日益稀缺。这篇论文提出的Co-RL让两个或多个独立训练的模型对同一个无标签问题各自生成多个答案,通过多数投票互相打分作为奖励,而不是自己给自己打分。只要配对的模型足够不同,犯的错误就不会重叠,这样互相纠错比自我打分更有效,在文本基准上平均提升3.0%到8.6%,在多模态基准上提升2.3%到7.2%。

METAL MEDIA 解读图

没有标准答案标签,让两三个不同的AI模型互相打分,也能提升推理能力

01问题:像TTRL这样的自我奖励方法让模型用多数投票给自己的答案打分,但如果模型本身系统性地犯错,这种方式只会不断强化错误,导致训练崩溃。
02解决方案:Co-RL同时训练多个不共享参数、不交换梯度的独立模型。每个模型对同一无标签问题生成多个答案,一个模型的多数投票答案作为另一个模型的奖励标准,反之亦然。
03多样性是关键:让配对模型来自不同模型系列、不同规模,或者用改写过的同一问题分别训练,可以减少两个模型犯相同错误的比例。论文показывает这样比配对相同模型能得到更准确的伪标签和更好的最终效果。
04实验结果:Co-RL在七个文本基准和四个多模态数学基准上超过了之前的无标签方法(TTRL、Intuitor、RENT、Co-rewarding),在部分设置下达到甚至超过了使用真实标签的监督训练效果,并且比之前的多智能体方法CoMAS平均高出4.0%,而且只用了一半数量的智能体。
05论文还从数学上证明:自我打分倾向于不断强化模型当前更偏好的答案,无论对错;而跨模型互相打分时,只要两个模型各有所长,就能真正纠正彼此的错误。

这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

问题:像TTRL这样的自我奖励方法让模型用多数投票给自己的答案打分,但如果模型本身系统性地犯错,这种方式只会不断强化错误,导致训练崩溃。
解决方案:Co-RL同时训练多个不共享参数、不交换梯度的独立模型。每个模型对同一无标签问题生成多个答案,一个模型的多数投票答案作为另一个模型的奖励标准,反之亦然。
多样性是关键:让配对模型来自不同模型系列、不同规模,或者用改写过的同一问题分别训练,可以减少两个模型犯相同错误的比例。论文показывает这样比配对相同模型能得到更准确的伪标签和更好的最终效果。
实验结果:Co-RL在七个文本基准和四个多模态数学基准上超过了之前的无标签方法(TTRL、Intuitor、RENT、Co-rewarding),在部分设置下达到甚至超过了使用真实标签的监督训练效果,并且比之前的多智能体方法CoMAS平均高出4.0%,而且只用了一半数量的智能体。
论文还从数学上证明:自我打分倾向于不断强化模型当前更偏好的答案,无论对错;而跨模型互相打分时,只要两个模型各有所长,就能真正纠正彼此的错误。

Table 1: Full performance across seven benchmarks for 3B models (%). For each benchmark, the best label-free result is shown in bold and the second best is underlined, with ties sharing the marking. Base and GT-Reward serve as references and are excluded from the ranking. Co-RL (Same family) trains two agents initialized from the same base model. Co-RL (Different family) pairs one agent from each of the two families. Co-RL (Different family+) further decouples the training data. Appendix D.1 extends the comparison to 7B and 8B models.
Method	GSM8K	MATH500	AMC	HEval	GPQA	MBPP	LCB	Avg
Qwen2.5-3B
Base	73.4	56.6	28.9	39.0	21.2	52.2	13.7	40.7
GT-Reward	76.2	64.6	36.1	65.2	20.7	54.4	14.5	47.4
TTRL	80.4	66.4	31.3	63.4	22.2	51.8	15.9	47.3
RENT	75.6	62.8	31.3	59.2	18.2	52.4	14.5	44.9
Intuitor	74.9	64.2	26.5	59.8	27.3	50.4	16.4	45.6
Co-rewarding-II	75.5	63.4	30.1	61.0	24.8	53.2	11.0	45.6
Co-RL (Same family)	78.5	66.0	37.4	65.8	22.2	56.0	15.2	48.7
Co-RL (Different family)	80.1	66.8	33.7	64.0	22.7	56.8	15.2	48.5
Co-RL (Different family+)	81.0	66.6	36.1	62.8	25.8	55.6	17.2	49.3
Llama-3.2-3B-Instruct
Base	73.6	43.8	18.1	51.2	21.2	50.8	12.0	38.7
GT-Reward	78.8	53.8	25.3	60.4	20.7	50.2	12.1	43.0
TTRL	77.9	50.2	26.5	59.2	24.8	51.2	12.0	43.1
RENT	75.4	45.2	12.0	59.2	17.7	49.4	11.5	38.6
Intuitor	75.8	40.8	21.7	54.3	21.7	51.4	12.0	39.7
Co-rewarding-II	75.4	53.4	24.1	54.9	23.7	49.2	12.1	41.8
Co-RL (Same family)	78.4	52.4	26.5	57.9	21.7	49.6	12.4	42.7
Co-RL (Different family)	80.5	56.2	27.7	59.2	21.2	50.4	11.0	43.7
Co-RL (Different family+)	78.4	55.2	30.1	59.2	22.2	50.4	12.0	43.9

Table 2: Comparison under the CoMAS multi-agent RL setting (%). All methods train Qwen2.5-3B-Instruct on the same prompt mixture and are evaluated following the CoMAS protocol. Results for prior methods are reported from Xue et al. 2026.
Method	GSM8K	MATH-500	HumanEval	MBPP	MMLU	GPQA	SciBench	Avg
Base	85.40	55.00	73.78	55.80	63.20	28.79	36.47	56.92
MAPoRL	85.80	55.40	75.61	57.00	63.20	31.47	39.08	58.22
TTRL	88.20	56.80	73.78	59.00	63.80	27.23	38.48	58.18
CoMAS	87.20	55.80	77.44	59.20	65.60	29.69	37.68	58.94
Co-RL (Different family)	89.5	68.6	82.32	68.00	65.80	29.69	36.87	62.97

Table 3: Three-agent Co-RL with heterogeneous model families (%). Qwen2.5-3B, Llama-3.2-3B-Instruct, and Qwen3-1.7B are jointly trained in a single Co-RL run. For each model, we compare against the base model, training with ground-truth rewards (GT-Reward), and self-rewarding with majority-vote pseudo-labels (TTRL).
Model	Method	GSM8K	MATH500	AMC	HEval	GPQA	MBPP	LCB	Avg
Qwen2.5-3B	Base	73.4	56.6	28.9	39.0	21.2	52.2	13.7	40.7
GT-Reward	76.2	64.6	36.1	65.2	20.7	54.4	14.5	47.4
TTRL	80.4	66.4	31.3	63.4	22.2	51.8	15.9	47.3
Co-RL (Different family)	79.8	66.3	33.6	64.6	23.2	56.0	15.8	48.5
Llama-3.2-3B-Instruct	Base	73.6	43.8	18.1	51.2	21.2	50.8	12.0	38.7
GT-Reward	78.8	53.8	25.3	60.4	20.7	50.2	12.1	43.0
TTRL	77.9	50.2	26.5	59.2	24.8	51.2	12.0	43.1
Co-RL (Different family)	77.8	54.2	28.8	64.4	25.1	50.9	11.7	44.7
Qwen3-1.7B	Base	67.0	60.9	27.5	40.0	15.3	50.6	12.4	39.1
GT-Reward	67.1	67.0	34.3	70.1	25.2	51.2	15.2	47.2
TTRL	70.3	67.6	32.1	69.5	24.8	52.0	15.1	47.3
Co-RL (Different family)	69.3	67.6	32.7	64.2	27.1	54.6	15.3	47.3

Table 4: Vision-language results for the small pair, Qwen2.5-VL-3B with InternVL3.5-2B, trained separately on open-r1 and MMR1 (%). Base is graded once with the corrected multiple-choice grader and is therefore identical across the two training sets. Base and GT-Reward serve as references and are excluded from the ranking.
Backbone	Data	Method	MathVision	MathVerse	MathVista	We-Math	Avg
InternVL-3.5-2B	open-r1	GT-Reward	26.55	35.33	59.60	59.31	45.20
Base	24.77	34.21	55.60	57.87	43.11
TTRL	25.86	34.24	57.60	62.47	45.04
Co-RL (Different family)	26.25	34.92	58.90	61.55	45.40
MMR1	GT-Reward	25.99	34.37	59.00	59.25	44.65
Base	24.77	34.21	55.60	57.87	43.11
TTRL	26.38	35.36	57.70	61.78	45.30
Co-RL (Different family)	26.05	34.80	58.60	61.15	45.15
Qwen2.5-VL-3B	open-r1	GT-Reward	21.71	31.29	60.90	57.99	42.97
Base	18.55	26.04	52.70	51.67	37.24
TTRL	21.15	30.05	57.40	61.55	42.54
Co-RL (Different family)	21.94	30.48	60.20	62.93	43.89
MMR1	GT-Reward	19.57	27.34	59.40	57.82	41.03
Base	18.55	26.04	52.70	51.67	37.24
TTRL	17.99	24.72	56.30	52.87	37.97
Co-RL (Different family)	21.05	28.91	57.20	57.30	41.12

Table 5: The four outcomes for a pair of models A and B. Every problem falls into exactly one cell, and all four diversity measures are counts over these cells.
	B correct	B wrong
A correct	both correct	only A correct
A wrong	only B correct	both wrong

Table 6: Error decoupling before RL, by what the two models differ in, sorted by κ within each block.
Decoupling	Pair	κ↓	c↑ (%)	w↓ (%)	u↑ (%)
3B tier
different family	Llama-3.2-3B × Phi-3.5-mini	0.31	32.8	3.0	53.0
different family	Qwen2.5-3B × Llama-3.2-3B	0.38	31.2	2.4	63.0
different family	Qwen2.5-3B × Phi-3.5-mini	0.38	31.2	4.0	55.4
different family	Qwen2.5-3B × MiniCPM3-4B	0.41	29.4	4.4	60.4
same family	Qwen2.5-3B × Qwen3-1.7B-Base	0.52	24.2	4.2	63.2
seed only	Qwen3-1.7B-Base × itself	0.52	24.0	5.0	66.4
seed only	Qwen2.5-3B × itself	0.56	22.0	4.4	62.6
7B tier
different family	Qwen2.5-7B × Llama-3.1-8B	0.42	29.4	1.8	71.4
same family	Qwen2.5-7B × Qwen2.5-3B	0.51	24.2	3.8	69.8
same family	Qwen2.5-7B × Qwen3-1.7B-Base	0.51	24.4	4.0	70.4
seed only	Llama-3.1-8B × itself	0.51	24.6	3.0	62.0
seed only	Qwen2.5-7B × itself	0.58	19.0	5.2	74.6

Table 7: One model held fixed, partner varied. Capability is identical to the seed-only row along each ladder, so the source of the partner is the only variable.
Partner	Decoupling	κ↓	c↑ (%)	w↓ (%)
Anchor: Qwen2.5-3B
itself, new seed	seed only	0.56	22.0	4.4
Qwen3-1.7B-Base	same family	0.52	24.2	4.2
MiniCPM3-4B	different family	0.41	29.4	4.4
Phi-3.5-mini	different family	0.38	31.2	4.0
Llama-3.2-3B	different family	0.38	31.2	2.4
Anchor: Qwen2.5-7B
itself, new seed	seed only	0.58	19.0	5.2
Qwen2.5-3B	same family	0.51	24.2	3.8
Qwen3-1.7B-Base	same family	0.51	24.4	4.0
Llama-3.1-8B	different family	0.42	29.4	1.8

Table 8: Results at 7B and 8B on the seven-benchmark suite (%). Base and GT-Reward serve as references and are excluded from the ranking.
Method	GSM8K	MATH500	AMC	HEval	GPQA	MBPP	LCB	Avg
Qwen2.5-7B
Base	82.9	70.0	39.8	47.6	18.7	62.8	21.1	49.0
GT-Reward	84.8	77.6	49.4	56.1	23.7	64.4	25.5	54.5
TTRL	80.6	74.8	39.8	51.8	25.8	65.4	23.9	51.7
RENT	78.8	75.4	47.0	50.6	29.8	61.6	26.2	52.8
Intuitor	82.9	75.4	41.0	51.8	28.3	64.0	24.8	52.6
Co-rewarding-II	81.9	72.6	43.4	52.4	26.8	64.0	25.9	52.4
Co-RL (Same family)	78.9	74.6	41.0	52.4	25.8	61.8	25.0	51.4
Co-RL (Different family)	81.3	75.2	44.6	52.4	26.3	65.6	26.5	53.1
Co-RL (Different family+)	80.2	74.4	38.6	54.3	37.9	63.2	26.6	53.6
Llama-3.1-8B-Instruct
Base	82.9	49.6	18.1	65.2	22.2	58.4	16.8	44.7
GT-Reward	82.7	53.2	25.3	64.0	30.3	59.2	15.2	47.1
TTRL	83.9	51.0	27.7	64.6	21.2	58.2	16.3	46.1
RENT	79.5	48.2	21.7	67.7	19.7	60.0	16.0	44.7
Intuitor	79.7	45.8	21.7	65.8	26.8	58.0	16.1	44.8
Co-rewarding-II	84.7	52.0	24.1	67.1	22.2	59.8	16.5	46.6
Co-RL (Same family)	85.4	51.4	22.9	68.3	23.2	60.4	15.8	46.8
Co-RL (Different family)	83.6	54.8	27.7	67.7	18.2	57.6	17.7	46.8
Co-RL (Different family+)	85.4	55.6	26.5	64.6	27.3	57.2	17.1	47.7

Table 9: Vision-language results at 7B to 12B on open-r1, with InternVL3.5-8B as the shared partner.
Backbone	Method	MathVision	MathVerse	MathVista	We-Math	Avg
Qwen2.5-VL-7B	GT-Reward	26.74	41.07	71.90	67.01	51.68
Base	23.36	33.32	56.60	62.47	43.94
TTRL	23.62	37.26	69.40	65.23	48.88
Co-RL (Different family)	26.87	38.43	71.00	68.22	51.13
InternVL-3.5-8B	GT-Reward	37.24	43.35	69.30	73.51	55.85
Base	29.21	36.65	65.70	60.69	48.06
TTRL	35.07	41.24	68.60	71.72	54.16
Co-RL (Different family)	35.30	40.74	70.60	70.98	54.40
Gemma-3-12B	GT-Reward	30.89	33.63	56.90	59.25	45.17
Base	27.20	32.70	46.70	60.50	41.78
TTRL	27.93	36.37	54.70	58.79	44.45
Co-RL (Different family)	32.01	35.91	55.60	66.72	47.56

Table 10: Matched-budget comparison between TTRL and Co-RL on text reasoning benchmarks. Both settings train the same two base models, Qwen2.5-3B and Llama-3.2-3B-Instruct. The ensemble rows pool four rollouts from each of the two models for majority voting (maj@8, T=0.6). Avg is the macro-average over the three benchmarks. For each benchmark, the best result is in bold and the second best is underlined, with ties sharing the marking.
Setting	GSM8K	MATH-500	AMC	Avg
TTRL (Qwen2.5-3B)	88.2	68.8	39.8	65.6
TTRL (Llama-3.2-3B)	65.7	56.0	27.7	49.8
TTRL (ensemble)	88.2	68.0	38.6	64.9
Co-RL (Qwen2.5-3B)	87.4	72.8	37.4	65.9
Co-RL (Llama-3.2-3B)	87.3	58.8	33.7	59.9
Co-RL (ensemble)	90.1	70.8	39.8	66.9

Table 11: Matched-budget comparison between TTRL and Co-RL on multimodal reasoning benchmarks. Both settings train the same two base models, Qwen2.5-VL-3B and InternVL3.5-2B. The ensemble rows pool four rollouts from each of the two models for majority voting (maj@8, T=0.6, top-p 0.95). Rows are grouped by training set. MMR1 runs use the corrected multiple-choice grader and open-r1 runs the legacy grader, so the two blocks are not compared against each other.
Setting	MathVision	MathVerse	MathVista	We-Math	Avg
open-r1
TTRL (Qwen2.5-VL)	22.96	31.45	61.10	63.39	44.73
TTRL (InternVL3.5)	29.67	38.91	62.30	67.24	49.53
TTRL (ensemble)	27.24	35.13	65.40	67.41	48.80
Co-RL (Qwen2.5-VL)	25.43	35.66	64.80	65.80	47.92
Co-RL (InternVL3.5)	30.46	38.60	63.30	67.87	50.06
Co-RL (ensemble)	28.95	38.48	67.00	69.08	50.88
MMR1
TTRL (Qwen2.5-VL)	17.27	30.71	63.40	60.57	42.99
TTRL (InternVL3.5)	28.78	39.47	63.70	66.90	49.71
TTRL (ensemble)	25.53	37.77	67.00	66.44	49.19
Co-RL (Qwen2.5-VL)	25.86	34.59	66.00	64.94	47.85
Co-RL (InternVL3.5)	30.79	40.94	65.30	67.53	51.14
Co-RL (ensemble)	30.49	39.75	69.40	69.54	52.30

为什么重要

随着AI的推理能力逐渐超出人类可靠评判的范围,不依赖人工标注也能持续提升模型的方法变得至关重要。这项研究表明,让多个不同的模型互相监督,而不是依赖人工标签或模型自我判断,是一条实用且可扩展的路径。

本文术语

强化学习(RL) · 模型通过获得奖励信号而不是直接的正确答案标签来逐步改进的训练方法。
标准答案监督/可验证奖励 · 用已知的正确答案来检查并奖励模型输出。
自我奖励强化学习 · 模型不依赖外部正确答案,而是通过多数投票等方式给自己的输出打分。
GRPO(群体相对策略优化) · 一种强化学习算法,通过比较一组采样答案之间的相对好坏来打分,不需要单独的评估模型。
多数投票/伪标签 · 把多次生成的答案中出现次数最多的那个当作正确答案来使用。

无法转载的图表

Figure 1: Comparison of Co-RL with prior label-free RL methods. Both TTRL and Co-rewarding derive rewards from self-generated agreement, and CoMAS scores multi-turn interactions with one of its own agents acting as judge. Co-RL instead derives rewards directly from peer votes. Beyond two agents, the votes pass along a directed ring (N=3 shown).
Figure 2: (a) Agreement between the two models, (b) pseudo-label accuracy, and (c) evaluation performance; (d) Error overlap before RL for two pairs each from a different family, the same family, and the same model under a different seed.
Figure 3: Overview of Co-RL with two agents. Each agent samples K responses to the same unlabeled question and generates a pseudo label with majority vote. Each rollout is then rewarded by agreement with the cohort’s pseudo label, and updates its own policy, with no sharing parameters and no gradient exchange except the cross-reward process.
Figure 4: Training dynamics at four scales, one column per backbone (Qwen2.5-3B, Llama-3.2-3B, Qwen2.5-7B, Llama-3.1-8B). (a) MATH-500 validation accuracy, (b) standard deviation of the reward within a rollout group, normalized to its value at the first step, and (c) mean completion length. Runs marked diverged leave the plotted range.
Figure 5: Training dynamics for Qwen2.5-VL-7B trained with InternVL3.5-8B on open-r1. (a) Evaluation accuracy, (b) mean completion length, and (c) the accuracy of the exchanged pseudo-labels together with the agreement between the two agents. Co-RL keeps improving and holds its completion length, while TTRL peaks and then degrades in both.

在原文中查看图表 →

论文原文摘要（英文）

Reinforcement learning (RL) has emerged as a powerful approach for improving reasoning in language and vision-language models, yet its strongest successes still depend heavily on ground-truth supervision (e.g., verifiable reward). Such annotations are costly to obtain and become increasingly scarce as reasoning capabilities advance beyond what humans can reliably evaluate. Self-rewarding RL reduces this dependence by enabling models to derive reward signals from their own completions. However, training solely on self-generated feedback can reinforce existing biases and suboptimal behaviors, reduce response diversity, and ultimately lead to homogenized responses and training collapse. In this work, we show that unsupervised reasoning can emerge through cooperative multi-agent training. We introduce Co-RL, a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers. We further show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops. This diversity consistently improves reasoning performance, maintains behavioral diversity, and mitigates training collapse. Across text-only and multimodal domains, Co-RL consistently outperforms the base models and prior label-free approaches, while matching or surpassing supervised methods, without access to any ground-truth labels. Concretely, Co-RL yields average gains of 3.0-8.6% across seven text-only benchmarks for LLMs and 2.3-7.2% across four multimodal benchmarks for VLMs. Code is available at https://github.com/DrStranded/Co-RL.

作者 · Yunhao Yang

在 arXiv 阅读

Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL

没有标准答案标签,让两三个不同的AI模型互相打分,也能提升推理能力

他们做了什么

为什么重要

本文术语

无法转载的图表

最新论文

METAL MEDIA 最新报道