Agent Lightning v1.0: Towards Harnessed Agentic RL
Training AI agents through the exact same tool-and-workflow wrapper they run in production, and the hidden bugs that surfaces
Modern AI agents run inside a 'harness' that manages tool calls, conversation context, and control flow, and this paper studies what happens when reinforcement learning (RL) training is done through that same harness instead of a simplified training loop. The authors identify new problems this creates -- like text being re-split into different tokens, and one agent run turning into several training pieces with unclear reward assignment -- and fix them in a lightweight framework called Agent Lightning v1.0, built in about 3,500 lines of code. Using it, they trained a coding agent (Qwen3.5-9B) with only 6,000 training examples and raised its SWE-bench Verified score from 41.8% to 56.4%.
METAL MEDIA explanatory visual
Training AI agents through the exact same tool-and-workflow wrapper they run in production, and the hidden bugs that surfaces
- 01Defines 'harnessed agentic RL': training an AI agent through the same production harness (the layer managing tools, context, and control flow) that it will actually run in, instead of reimplementing the agent loop inside the training framework
- 02Shows that one agent run (rollout) can silently split into multiple training samples due to text-to-token conversion mismatches, and different existing frameworks handle reward/advantage assignment for this inconsistently, sometimes causing unstable training
- 03Argues rewards and loss should be normalized per whole rollout rather than per split sample, and implements this in Agent Lightning v1.0, a framework of about 3,500 lines of code
- 04Introduces 'collocated async RL,' where rollout generation and model weight updates share the same GPU pool but take turns, achieving roughly 2x speedup over standard synchronous training while using fewer GPUs
- 05Releases a full data-cleaning pipeline and training scripts that improved a coding agent's SWE-bench Verified score from 41.8% to 56.4% (a 14.6 percentage point gain) using only about 6,000 training examples
What they did
- Defines 'harnessed agentic RL': training an AI agent through the same production harness (the layer managing tools, context, and control flow) that it will actually run in, instead of reimplementing the agent loop inside the training framework
- Shows that one agent run (rollout) can silently split into multiple training samples due to text-to-token conversion mismatches, and different existing frameworks handle reward/advantage assignment for this inconsistently, sometimes causing unstable training
- Argues rewards and loss should be normalized per whole rollout rather than per split sample, and implements this in Agent Lightning v1.0, a framework of about 3,500 lines of code
- Introduces 'collocated async RL,' where rollout generation and model weight updates share the same GPU pool but take turns, achieving roughly 2x speedup over standard synchronous training while using fewer GPUs
- Releases a full data-cleaning pipeline and training scripts that improved a coding agent's SWE-bench Verified score from 41.8% to 56.4% (a 14.6 percentage point gain) using only about 6,000 training examples
| Method | Endpoint | Comment |
|---|---|---|
| POST | /api/rollouts | Create a batch of rollouts. |
| GET | /api/rollouts | List rollouts, optionally filtered by state. |
| GET | /api/rollouts/{rollout_id} | Get one rollout. |
| PATCH | /api/rollouts/{rollout_id} | Update rollout status. |
| POST | /api/rollouts/{rollout_id}/attempt/{attempt_id}/events | Append an event to a rollout attempt. |
| GET | /api/rollouts/{rollout_id}/events | Read rollout events. |
| POST | /api/models | Register model endpoints. |
| DELETE | /api/models | Remove all registered model endpoints. |
| POST | /proxy/rollout/{rollout_id}/attempt/{attempt_id}/mode/{mode}/openai/v1/chat/completions | Forward an OpenAI-compatible model call. |
Why it matters
Training agents through their real deployment harness closes the gap between how they're trained and how they're actually used, but doing so correctly requires solving subtle technical problems this paper is first to systematically document. The released framework and reproducible scripts give researchers and engineers a practical, low-resource path to apply RL training to real-world agent harnesses like coding assistants.
Terms in this paper
- agent harness · the surrounding software that manages an AI agent's tool use, conversation context, and control flow
- reinforcement learning (RL) · a training method where a model improves by taking actions and receiving reward signals
- rollout · one complete execution trace of an agent performing a task
- retokenization · when generated text is converted back into tokens (model input units) and ends up split differently than originally
- SWE-bench Verified · a benchmark that evaluates an AI coding agent's ability to fix real software bugs
Figures we cannot republish
- Figure 4: Traditional agentic RL, where each rollout is one training sample (left), versus harnessed agentic RL, where a rollout can expand into a dynamic number of samples that inherit its reward (right).
- Figure 5: An example batch with three rollouts of different sample counts and response lengths.
- Figure 6: Sync RL, async RL, and our collocated async RL. Collocated async RL shares the same GPUs between rollout and update while still avoiding the need to wait for the slowest rollout.
- Figure 7: Search-agent training dynamics. From left to right: mean training reward and mean validation reward.
- Figure 8: General instruction-following agent training dynamics. From left to right: mean training reward and mean validation reward.
- Figure 9: Coding-agent training dynamics for Sample-level Advantage, Rollout-level Advantage, and Rollout-level Advantage + Rollout-level Norm. Left: validation reward. Right: policy entropy.
- Figure 10: Rollout-merging behavior for the Rollout-level Advantage + Rollout-level Norm run. Left: fraction of rollouts that yield exactly one training sample. Right: average number of training samples produced per rollout. Dashed lines mark the mean over training.
- Figure 11: The objects stored by the API Gateway.
- Figure 12: The Rollout Controller reconciles rollout status in the API Gateway with agent executions running as Kubernetes Jobs or local processes.
Original abstract (English)
Modern agents operate inside agent harnesses that manage tools, context, and control flow, making the harness a critical part of the agent system. Our original Agent Lightning introduced a disaggregated architecture that connects arbitrary agents to RL training through an LLM endpoint proxy, an approach later adopted by frameworks such as verl Uni-Agent, AReaL 2.0, slime, and Polar. We refer to this paradigm as harnessed agentic RL, where the deploy-time harness directly participates in model post-training. Harnessed agentic RL differs fundamentally from traditional agentic RL: the harness, rather than the training engine, owns the environment interaction loop, while the trainer observes only sequences of LLM request-response pairs. This introduces challenges in retokenization, sample merging, advantage calculation, loss normalization, and backend scheduling, which can substantially affect training stability and effectiveness. We present Agent Lightning v1.0, a lightweight framework for harnessed agentic RL implemented in approximately 3,500 lines of code. It supports arbitrary agent harnesses and serves as a practical testbed for studying these challenges. We evaluate it on instruction-following, search, and coding agents, and provide a complete reproducible pipeline for coding-agent RL. Using only 6K training examples and modest compute, RL improves Qwen3.5-9B on SWE-bench Verified from 41.8% to 56.4%, a 14.6-point absolute gain. We release the complete workflow and training scripts to facilitate reproducible research on harnessed agentic RL.
Read on arXivLatest papers
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the time
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papers
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad action
- EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise it
- EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknesses
- Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversight
- PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappears
- Automated Summarization of Financial News Using Large Language Models and Retrieval-Augmented Generation: An Early Empirical Study (Fall 2023)Testing AI summaries of stock news, the simple approach beat the trendy retrieval-based one