Figure 1: Overview of the Lego-RL training infrastructure.
Table 1: Comparison of representative agentic RL frameworks. ✓: supported; △: partial/conditional support; –: not reported. R3: rollout routing replay. Observability denotes monitoring training runs and diagnosing execution- or trajectory-level failures.
Harness-native fidelity
Execution & reward
Observability
Framework
Black-box harness
Token-in/ Token-out
History alignment
R3
Fully async
Sandbox execution
Reward-hack defense
Training observability
verl (Sheng et al. 2025)
–
✓
–
✓
✓
△
–
△
slime (THUDM 2025)
✓
✓
✓
✓
✓
△
△
–
MOLT (NVIDIA NeMo 2026)
△
✓
–
✓
✓
–
–
–
SkyRL-Agent (Cao et al. 2025)
△
△
△
✓
✓
✓
–
–
AReaL (Fu et al. 2025)
✓
△
–
–
✓
–
–
–
Agent Lightning (Luo et al. 2025)
△
✓
–
–
△
–
–
△
Polar (Xu et al. 2026)
✓
✓
✓
–
✓
✓
–
–
rLLM (Berkeley Sky Computing Lab 2026)
✓
✓
–
✓
✓
✓
–
✓
OpenForgeRL (Yu et al. 2026)
✓
△
–
–
✓
✓
–
–
ALE (ROLL/ROCK) (Wang et al. 2025a)
–
–
–
–
✓
✓
△
–
Lego-RL
✓
✓
✓
✓
✓
✓
✓
✓
Figure 2: Closed-loop operational workflow of Lego-RL. The five stages cover data preparation, run validation, training, live observability, and human review. Stage (3) corresponds to the training infrastructure shown in Figure 1, while the agent plugin acts as the control plane.
Table 2: SWE-bench Verified performance across the three coding agents. All numbers are measured by us under the same harness version and evaluation protocol (temperature 0.7, 200 turns, 200k context budget).
Coding agent
Model
SWE-bench Verified (%)
OpenHands SDK
Qwen3.5-35B-A3B (Qwen Team 2026a)
64.0
Qwen3.6-35B-A3B (Qwen Team 2026b)
67.4
KAT-Coder-V2.5-Dev (KwaiKAT Team 2026)
67.0
Lego-RL-Qwen3.5-35B-A3B
70.4(+6.4)
Claude Code
Qwen3.5-35B-A3B (Qwen Team 2026a)
62.4
Qwen3.6-35B-A3B (Qwen Team 2026b)
63.4
KAT-Coder-V2.5-Dev (KwaiKAT Team 2026)
66.8
Lego-RL-Qwen3.5-35B-A3B
68.2(+5.8)
OpenCode
Qwen3.5-35B-A3B (Qwen Team 2026a)
57.2
Qwen3.6-35B-A3B (Qwen Team 2026b)
60.6
KAT-Coder-V2.5-Dev (KwaiKAT Team 2026)
64.8
Lego-RL-Qwen3.5-35B-A3B
66.6(+9.4)
Figure 3: Training behavior of OpenHands SDK, Claude Code, and OpenCode over three epochs (126 training steps), showing training reward, validation reward, policy entropy, and mean response length.
Table 3: Rollout-to-training alignment over the three matched production runs: probabilities captured at the serving boundary against trainer-side recomputation over the corresponding assistant tokens. All statistics are medians over training steps.
|Δlogp¯| per trajectory (×10−3)
Coding agent
Pearson r
KL (×10−3)
p50
p90
p99
OpenHands SDK
0.9993
0.75
0.7
1.2
2.1
Claude Code
0.9980
1.35
0.7
1.3
2.7
OpenCode
0.9993
0.60
0.6
1.1
2.0
Figure 4: Trajectory termination profiles across agent scaffolds. Bars show the proportion of trajectories by termination reason; timeout and environment-setup failures are excluded from optimization.
Table 4: Stage-wise wall-clock statistics across 3,699 OpenHands SDK training trials. Mean-time fraction is computed relative to the mean total trial duration; Other includes scheduling, trajectory handling, and sandbox cleanup.
Stage
Mean (s)
p50
p90
p99
Mean-time fraction
Sandbox setup
21.6
7.7
41.8
275.2
2.3%
Agent setup
4.2
4.0
4.8
8.4
0.5%
Agent execution
840.5
708.4
1604.3
2801.1
91.3%
Verification
35.9
4.1
29.6
928.5
3.9%
Other
20.3
8.1
12.3
965.5
2.2%
Trial total
920.4
770.0
1742.7
3189.1
100%
Figure 5: In-batch reward distributions across training. (a–c) Distribution of tasks by the number of successful rollouts out of eight in the first and last epoch for each scaffold. The 0/8 and 8/8 groups provide no group-relative advantage. (d) Combined proportion of these zero-variation groups across epochs.
Table 5: Ablation of sandbox optimizations.
Median latency
Paired ratio
Optimization
Stage
With
Without
Median
p10–p90
n
Lazy image pull
sandbox setup
1.57 s
2.66 s
1.7×
—
100
Prebuilt task image
sandbox setup
1.04 s
36.2 s
33.2×
17.9–67.5×
50
Mounted agent runtime
agent setup
0.51 s
7.82 s
15.4×
14.5–16.6×
50
Packaged grading toolchain
verification
3.81 s
2.72 s
0.71×
0.67–0.73×
50
Figure 6: Task-selection ablation across four 951-task pools. (a) Held-out validation solve rate, measured over the validation tasks that executed. (b) Training verifier reward; levels are pool-specific, so only the slopes are comparable.
Table 6: Reward-integrity failure modes and their mitigations. Incidence rates are measured prior to deploying the defenses; “—” indicates cases not separately quantified.
Failure Mode
Incidence
Defense
Agent-side: shortcut exploitation
Reads git history
4.6–20.5%
Rebase history to a single commit during agent phase; restore before grading
Downloads reference fix
1.9%
Per-phase egress firewall in privilege-separated sidecar
Edits test files
2.4–19.4%
Withhold tests until grading; revert test-path edits
Environment-side: reward detached from agent
Grader applies reference patch
2.5%
Audit affected instances out of pool; flag live if degenerate reward propagates
Grader requires network access
—
Package all grade-time dependencies; ensure deterministic invocation
Incomplete repository build
—
Hermetic fail-fast build; report setup failure explicitly, not as zero reward
Figure 7: Failure diagnosis with the Live UI. (a) Per-step termination reasons for an environment-failure run. (b) Assisted analysis of a collapsed run and the corresponding early-stop condition. (a) and (b) are two different diagnostic runs, neither is the Claude Code production run reported elsewhere in this section.
Table 7: Routing-replay configurations compared. Expert overlap and top-1 agreement are undefined when replay is disabled.
Routing replay configuration
Pearson r
Mean |Δp|
Expert overlap
Top-1 agreement
Disabled
0.9946
0.0062
—
—
Enabled, misaligned
0.7503
0.0954
0.083
0.026
Enabled, aligned
0.9993
0.0025
0.996
0.985
Figure 8: Behavioral analysis with the Live UI, Claude Code run. (a) Tool-use trajectories for eight rollouts of one task. (b) Task-level solve rates in the first and last sampled epochs.
Table 8: Evidence supplied for the collapsed run. t is the least-squares slope divided by its standard error over all logged steps; the gradient norm does not clear |t|=2 and is therefore read as noise. Tokens per turn is a ratio of two rows above it and carries no separate trend statistic.
Series
First five steps
Last five steps
t
training reward
0.351
0.050
−7.5
turns per trajectory
18.9
0.96
−14.8
response length (tokens)
12,170
2,276
−7.7
tokens per turn
650
2,397
—
policy entropy
0.134
0.230
+7.2
KL term of the loss
0.0046
0.130
+3.4
rollout–training agreement
0.995
0.970
−3.3
gradient norm
0.31
0.034
+0.4
Figure 9: Trainer schedule under synchronous and asynchronous execution.
Table 9: Agent behaviors before and after training, over 420 trajectories at each end of the production OpenHands SDK run; the pass@k rows are over prompt groups in the first and last third of the run.
Behavior
First
Last
Δ
Reads back a file it edited
73.6%
98.1%
+24.5
Runs the test suite
85.0%
93.6%
+8.6
Files explored before 1st edit
3.45
6.92
+3.47
Ends with an explicit finish
88.3%
91.9%
+3.6
Reproduces failure before editing
6.7%
11.2%
+4.5
Solves despite a failed command
63.9%
66.8%
+2.9
Malformed tool calls (of all calls)
1.07%
0.15%
−0.92
Coverage, pass@8
83.2%
87.9%
+4.7
Reliability, pass8
28.3%
39.4%
+11.1
Figure 10: Lazy versus full image delivery over the same 100 task images. Nydus streams image chunks on demand, whereas OCI denotes the conventional pull, which materializes the entire image before the container starts. The panels report startup latency, cumulative network and disk traffic, and in-container read throughput.
Table 10: Resolved hyperparameters of the three production runs (Qwen3.5-35B-A3B through the OpenHands SDK, Claude Code, and OpenCode).
Policy loss
GSPO
Clip range (sequence-level)
(3×10−4, 4×10−4)
Advantage estimator
GRPO
Loss aggregation
seq-mean-token-mean
KL reward penalty
none
KL loss coefficient
10−3
Learning rate
1×10−6
Learning-rate schedule
constant
Gradient clip
1.0
Prompts per batch
64
Rollouts per prompt
8
Micro-batch per GPU
1
Rollout temperature
1.0
Rollout top-p
1.0
Validation temperature
0.7
Validation samples per instance
1
Prompt budget
30k tokens
Response budget
170k tokens
Staleness threshold
1
Partial-rollout recovery
on
Importance-sampling correction
off
Training pool
2,699 tasks
Epochs
3
Figure 11: Reasoning share of the response over training on a fixed 120-task validation cohort. (a) Mean over tasks (solid) and character-weighted mean (dashed); (b) Median share at each task’s first (open) and last (filled) sampled epoch, grouped by number of rollouts solved.
Figure 12: Agent behaviors before and after training, computed over 420 trajectories at each end of the production OpenHands SDK run. The pass@k and passk rows are computed over prompt groups in the first and last third of the run.
本文术语
强化学习(RL) · 通过对结果给予奖励来不断改进系统行为方式的训练方法
harness(运行框架) · 管理AI智能体工具调用、上下文和执行反馈的软件程序
沙盒(sandbox) · 用于安全运行和测试代码的隔离环境
混合专家模型(MoE) · 一种大模型结构,每次只激活其中部分子网络(专家)
SWE-bench Verified · 用于检验智能体解决真实软件仓库问题能力的基准测试
论文原文摘要（英文）
Reinforcement learning for coding agents increasingly relies on long-running agent harnesses to manage tool integration, repository contexts, and execution feedback. However, the native execution environments of these harnesses are inherently misaligned with policy-gradient training: environmental crashes and reward hacking corrupt outcome signals, while train-inference discrepancies decouple rollout behavior from policy updates. To address this, we present LEGO-RL, a framework that bridges native coding-agent harnesses with scalable policy-gradient optimization without modifying their internal control flow. LEGO-RL is built upon three pillars: (1) faithful optimization via in-process LLM proxying that captures raw generation streams for token-level alignment and robust trainer-side log-probability recomputation, even under harness-side compaction or re-serialization; (2) reliable execution via scalable sandbox orchestration featuring image caching and stage-wise defenses to mitigate reward hacking; and (3) observable training through an integrated plugin that automates validation and monitoring, paired with a Live UI for granular trajectory diagnostics. We evaluate LEGO-RL by training the sparse MoE model Qwen3.5-35B-A3B with GSPO across three native coding-agent harnesses. LEGO-RL improves Qwen3.5-35B-A3B across OpenHands SDK (64.0% to 70.4%), Claude Code (62.4% to 68.2%), and OpenCode (57.2% to 66.6%) on SWE-bench Verified, while maintaining a rollout-training probability correlation above 0.99.
