SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning

arXiv:2608.198422026-08-21

训练AI智能体时,只跑一次任务、不用额外评判模型也能学得更好

SAPO用同一个语言模型,在一次生成过程中同时输出决定行动的策略和评估行动好坏的价值判断。以往的方法要么需要对同一个任务反复尝试多次再比较结果,要么需要单独训练一个评判模型,而SAPO利用文本从左到右生成的自然顺序,在一次尝试里就同时得到这两者。在ALFWorld和WebShop这两个多步骤任务环境中,SAPO的成功率平均比PPO高15.1个百分点,比GRPO高12.1个百分点,同时每轮训练耗时比PPO减少33.2%。

METAL MEDIA 解读图

训练AI智能体时,只跑一次任务、不用额外评判模型也能学得更好

01现有的分组比较方法(如GRPO)需要对同一任务反复采样多次结果再比较,当各次结果奖励接近时学习信号会消失,而且采样次数和性能之间存在昂贵的权衡
02SAPO在同一个语言模型生成文本的不同位置读取信息:动作生成之前总结当前状态,动作生成之后评估这个动作,全部在一次因果生成序列中完成
03通过从轨迹末尾向前逐步回溯计算每一步的贡献(结合λ回报和批量归一化),无需多次同步采样也能稳定估计每一轮的价值
04在Qwen2.5-1.5B和7B模型上,ALFWorld和WebShop测试显示,SAPO成功率平均比PPO高15.1个百分点,比GRPO高12.1个百分点,同时省去了单独评判模型的内存开销,每轮训练时间比PPO减少33.2%

这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

现有的分组比较方法(如GRPO)需要对同一任务反复采样多次结果再比较,当各次结果奖励接近时学习信号会消失,而且采样次数和性能之间存在昂贵的权衡
SAPO在同一个语言模型生成文本的不同位置读取信息:动作生成之前总结当前状态,动作生成之后评估这个动作,全部在一次因果生成序列中完成
通过从轨迹末尾向前逐步回溯计算每一步的贡献(结合λ回报和批量归一化),无需多次同步采样也能稳定估计每一轮的价值
在Qwen2.5-1.5B和7B模型上,ALFWorld和WebShop测试显示,SAPO成功率平均比PPO高15.1个百分点,比GRPO高12.1个百分点,同时省去了单独评判模型的内存开销,每轮训练时间比PPO减少33.2%

Table 1: Evaluation results on ALFWorld and WebShop. For each RL training method, we report the mean and standard deviation over three random seeds. The ALFWorld contains six categories: Pick & Place (Pick), Examine in Light (Look), Clean & Place (Clean), Heat & Place (Heat), Cool & Place (Cool), and Pick Two & Place (Pick2). Most entries in this table are reported by Feng et al. [4]. Notably, the baseline GiGPOw/o std replaces the group-relative normalization std with one.
Type	Method	ALFWorld	WebShop
Pick	Look	Clean	Heat	Cool	Pick2	All	Score	Succ.
Closed-Source Model
Prompting	GPT-4o	75.3	60.8	31.2	56.7	21.6	49.8	48.0	31.8	23.7
Prompting	Gemini-2.5-Pro	92.8	63.3	62.1	69.0	26.6	58.7	60.3	42.5	35.9
Qwen2.5-1.5B-Instruct
Prompting	ReAct	17.4	20.5	15.7	6.2	7.7	2.0	12.8	40.1	11.3
Prompting	Reflexion	35.3	22.2	21.7	13.6	19.4	3.7	21.8	55.8	21.9
RL Training	RLOO	88.3±3.0	52.8±8.6	71.0±5.9	62.8±8.7	66.4±5.5	56.9±4.7	69.7±2.5	73.9±5.6	52.1±6.7
RL Training	EMPG	85.5	33.5	78.9	76.2	74.7	89.1	73.7	80.4	60.8
RL Training	GiGPOw/ std	94.4±5.9	67.5±4.6	94.8±3.8	94.4±7.8	79.8±4.7	76.4±5.4	86.7±1.7	83.1±1.6	65.0±3.2
RL Training	GiGPOw/o std	96.0±1.4	76.5±3.9	91.8±5.5	91.3±6.3	71.7±8.4	79.5±7.7	86.1±4.7	83.5±1.8	67.4±4.5
RL Training	PPO (with critic)	64.8±3.5	40.5±6.9	57.1±4.9	60.6±6.6	46.4±4.0	47.4±1.9	54.4±3.1	73.8±3.0	51.5±2.9
RL Training	GRPO	85.3±1.5	53.7±8.0	84.5±6.8	78.2±7.9	59.7±5.0	53.5±5.6	72.8±3.6	75.8±3.5	56.8±3.8
RL Training	SAPO	92.0±2.9	76.9±6.3	100.0±0.0	100.0±0.0	82.8±4.7	82.4±5.0	90.1±2.3	82.21.4	63.71.6
Qwen2.5-7B-Instruct
Prompting	ReAct	48.5	35.4	34.3	13.2	18.2	17.6	31.2	46.2	19.5
Prompting	Reflexion	62.0	41.6	44.9	30.9	36.3	23.8	42.7	58.1	28.8
RL Training	RLOO	87.6±4.3	78.2±8.3	87.3±5.8	81.3±7.6	71.9±5.2	48.9±8.4	75.5±4.6	80.3±3.2	65.7±4.0
RL Training	EMPG	92.9	75.2	74.8	86.3	73.7	65.3	78.5	81.0	69.3
RL Training	GiGPOw/ std	97.7±1.6	82.7±7.9	98.8±1.6	83.7±7.2	89.3±8.2	79.2±6.6	90.8±1.3	84.4±2.9	72.8±3.2
RL Training	GiGPOw/o std	91.8±5.4	88.6±6.3	95.9±3.2	90.2±2.6	86.5±5.5	85.2±7.5	90.2±2.3	86.2±2.6	75.2±3.8
RL Training	PPO (with critic)	92.3±4.0	64.0±8.4	92.5±2.4	89.5±7.0	80.3±2.0	68.8±8.3	80.4±2.7	81.4±3.1	68.7±5.1
RL Training	GRPO	90.8±5.1	66.1±6.7	89.3±5.4	74.7±6.9	72.5±5.4	64.7±7.3	77.6±5.2	79.3±2.8	66.1±3.7
RL Training	SAPO	99.0±1.4	82.3±2.1	100.0±0.0	97.9±4.7	79.7±3.9	91.7±1.6	94.0±1.7	88.6±1.8	82.4±2.0

Figure 2: Per-iteration runtime breakdown of PPO and SAPO on ALFWorld using Qwen2.5-1.5B. The vertical axis is shown on a logarithmic scale to accommodate the large variation in module runtimes. N/A indicates that SAPO does not require the corresponding value-model or critic-update module. Overall, SAPO reduces the measured per-iteration runtime from 451.2 s to 301.4 s, corresponding to a 33.2% reduction compared with PPO.

为什么重要

这意味着训练需要多步骤与环境交互的AI智能体时,可以在降低内存和计算成本的同时反而提升性能。对计算资源有限、又想训练长周期交互式智能体的研究者和企业来说,这能带来实际的成本节省。

本文术语

策略(policy) · 决定在特定情境下采取什么行动的AI规则
价值函数(value function) · 预测某个状态或行动未来会带来多好结果的函数
评判模型(critic) · 与策略模型分开设置、专门用来评估行动好坏的辅助模型
PPO · 一种限制策略每次更新幅度以保持训练稳定的经典强化学习方法
GRPO · 通过比较同一任务多次尝试的结果来训练、无需单独评判模型的方法

论文原文摘要（英文）

Agentic reinforcement learning (RL) has become a critical stage in the post-training of large language models. Existing critic-free, group-relative methods estimate policy advantages from multiple rollouts, avoiding the substantial memory overhead of conventional proximal policy optimization (PPO) and achieving strong performance on long-horizon interactive tasks. Despite their success, recent studies revealed three limitations: (1) Lack explicit value generalization and effective temporal credit assignment; (2) Suffer from potential advantage collapse in long-horizon complex tasks; (3) Require a costly trade-off between sampling budget and policy performance. In this work, we propose Single-rollout Autoregressive Policy Optimization (SAPO), a low-memory and compute-efficient framework in which the policy and value functions share a single autoregressive backbone. SAPO exploits the autoregressive structure of LLMs to produce policy and value predictions at distinct causal boundaries with shared parameters, while independently optimizing the PPO objectives and auxiliary on-policy SARSA objectives. To robustly estimate the contribution of each turn, we further introduce a trajectory-level generalized advantage estimator that combines lambda-returns with batch normalization. Experiments across ALFWorld and WebShop with Qwen2.5-1.5B/7B show that SAPO trains stably and outperforms PPO and GRPO by mean +15.1 and +12.1 percentage points, respectively, while eliminating the memory cost of a separate critic model and reducing per-iteration runtime by 33.2% over PPO.

作者 · Dayang Liang, Lang Feng, Bo An, Yunlong Liu

在 arXiv 阅读

METAL MEDIA 最新报道

NCT 127发行时隔两年全新正规专辑《BLINGY》

图片来源： Dayang Liang et al., arXiv:2608.19842, CC BY 4.0

SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning

训练AI智能体时,只跑一次任务、不用额外评判模型也能学得更好

他们做了什么

为什么重要

本文术语

最新论文

METAL MEDIA 最新报道