K-文化的一切——从回归到 K-美妆，发送到您的邮箱订阅邮件

METAL MEDIA

Agent Lightning v1.0: Towards Harnessed Agentic RL

arXiv:2608.175282026-08-17

让AI智能体在真实使用的工具外壳里直接做强化学习训练,顺带揪出了一堆隐藏问题

现在的AI智能体运行在一个负责管理工具调用、对话上下文和执行流程的外壳(harness)里,这篇论文研究了如果把强化学习(RL)训练直接接入这个外壳会遇到什么新问题。作者发现文本被重新切分成不同token、一次智能体运行被拆成多个训练样本却不知如何分配奖励等问题,并在轻量框架Agent Lightning v1.0(约3500行代码)中给出了解决方案。用这个框架训练编程智能体Qwen3.5-9B,仅用6000条训练数据就把SWE-bench Verified得分从41.8%提升到56.4%。

METAL MEDIA 解读图

让AI智能体在真实使用的工具外壳里直接做强化学习训练,顺带揪出了一堆隐藏问题

  1. 01提出harnessed agentic RL概念,即直接通过智能体实际部署时使用的外壳(负责工具调用、上下文构建、执行流程的软件层)来做强化学习训练,而不是把智能体逻辑重新写进训练框架里
  2. 02发现一次智能体完整运行(rollout)可能因为文本重新转换成token时的偏差而被拆分成多个训练样本,不同框架在奖励和优势值分配上做法不一致,处理不当会导致训练不稳定
  3. 03主张奖励和损失应该按整次rollout为单位统一计算,而不是按拆分后的样本单位计算,并在约3500行代码的Agent Lightning v1.0中实现了这一设计
  4. 04提出collocated async RL方法,让生成数据和更新模型权重共享同一批GPU、错峰使用,相比传统同步训练实现约2倍端到端加速,同时用更少GPU
  5. 05公开完整的数据清洗流程和训练脚本,仅用约6000条训练样本就把编程智能体在SWE-bench Verified上的得分从41.8%提升到56.4%,提升14.6个百分点
这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

  1. 提出harnessed agentic RL概念,即直接通过智能体实际部署时使用的外壳(负责工具调用、上下文构建、执行流程的软件层)来做强化学习训练,而不是把智能体逻辑重新写进训练框架里
  2. 发现一次智能体完整运行(rollout)可能因为文本重新转换成token时的偏差而被拆分成多个训练样本,不同框架在奖励和优势值分配上做法不一致,处理不当会导致训练不稳定
  3. 主张奖励和损失应该按整次rollout为单位统一计算,而不是按拆分后的样本单位计算,并在约3500行代码的Agent Lightning v1.0中实现了这一设计
  4. 提出collocated async RL方法,让生成数据和更新模型权重共享同一批GPU、错峰使用,相比传统同步训练实现约2倍端到端加速,同时用更少GPU
  5. 公开完整的数据清洗流程和训练脚本,仅用约6000条训练样本就把编程智能体在SWE-bench Verified上的得分从41.8%提升到56.4%,提升14.6个百分点
Table 1: API Gateway endpoints.
MethodEndpointComment
POST/api/rolloutsCreate a batch of rollouts.
GET/api/rolloutsList rollouts, optionally filtered by state.
GET/api/rollouts/{rollout_id}Get one rollout.
PATCH/api/rollouts/{rollout_id}Update rollout status.
POST/api/rollouts/{rollout_id}/attempt/{attempt_id}/eventsAppend an event to a rollout attempt.
GET/api/rollouts/{rollout_id}/eventsRead rollout events.
POST/api/modelsRegister model endpoints.
DELETE/api/modelsRemove all registered model endpoints.
POST/proxy/rollout/{rollout_id}/attempt/{attempt_id}/mode/{mode}/openai/v1/chat/completionsForward an OpenAI-compatible model call.

为什么重要

让智能体按照它实际部署时使用的工具外壳来训练,能缩小训练和实际使用之间的差距,而这篇论文首次系统性地梳理了这样做会遇到的技术陷阱并给出可验证的解决办法和开源代码。对于像编程智能体这类此前缺乏数据和训练脚本的领域,这项工作提供了低资源、可复现的训练路径,对研究者和开发者都有实际参考价值。

本文术语

  • 智能体外壳(harness) · 负责管理AI智能体工具调用、对话上下文和执行流程的软件层
  • 强化学习(RL) · 让模型通过行动并获得奖励反馈来不断改进的训练方法
  • rollout · 智能体从头到尾完成一次任务的完整执行记录
  • 重新分词(retokenization) · 生成的文本被重新转换成token(模型处理的最小单位)时,切分方式和原来不同的现象
  • SWE-bench Verified · 用于评估AI编程智能体修复真实软件缺陷能力的基准测试

无法转载的图表

  • Figure 4: Traditional agentic RL, where each rollout is one training sample (left), versus harnessed agentic RL, where a rollout can expand into a dynamic number of samples that inherit its reward (right).
  • Figure 5: An example batch with three rollouts of different sample counts and response lengths.
  • Figure 6: Sync RL, async RL, and our collocated async RL. Collocated async RL shares the same GPUs between rollout and update while still avoiding the need to wait for the slowest rollout.
  • Figure 7: Search-agent training dynamics. From left to right: mean training reward and mean validation reward.
  • Figure 8: General instruction-following agent training dynamics. From left to right: mean training reward and mean validation reward.
  • Figure 9: Coding-agent training dynamics for Sample-level Advantage, Rollout-level Advantage, and Rollout-level Advantage + Rollout-level Norm. Left: validation reward. Right: policy entropy.
  • Figure 10: Rollout-merging behavior for the Rollout-level Advantage + Rollout-level Norm run. Left: fraction of rollouts that yield exactly one training sample. Right: average number of training samples produced per rollout. Dashed lines mark the mean over training.
  • Figure 11: The objects stored by the API Gateway.
  • Figure 12: The Rollout Controller reconciles rollout status in the API Gateway with agent executions running as Kubernetes Jobs or local processes.
在原文中查看图表 →

论文原文摘要（英文）

Modern agents operate inside agent harnesses that manage tools, context, and control flow, making the harness a critical part of the agent system. Our original Agent Lightning introduced a disaggregated architecture that connects arbitrary agents to RL training through an LLM endpoint proxy, an approach later adopted by frameworks such as verl Uni-Agent, AReaL 2.0, slime, and Polar. We refer to this paradigm as harnessed agentic RL, where the deploy-time harness directly participates in model post-training. Harnessed agentic RL differs fundamentally from traditional agentic RL: the harness, rather than the training engine, owns the environment interaction loop, while the trainer observes only sequences of LLM request-response pairs. This introduces challenges in retokenization, sample merging, advantage calculation, loss normalization, and backend scheduling, which can substantially affect training stability and effectiveness. We present Agent Lightning v1.0, a lightweight framework for harnessed agentic RL implemented in approximately 3,500 lines of code. It supports arbitrary agent harnesses and serves as a practical testbed for studying these challenges. We evaluate it on instruction-following, search, and coding agents, and provide a complete reproducible pipeline for coding-agent RL. Using only 6K training examples and modest compute, RL improves Qwen3.5-9B on SWE-bench Verified from 41.8% to 56.4%, a 14.6-point absolute gain. We release the complete workflow and training scripts to facilitate reproducible research on harnessed agentic RL.

作者 · Zhiyuan He

在 arXiv 阅读

最新论文

全部论文 →

METAL MEDIA 最新报道