Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox›
LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents
arXiv:2608.173932026-08-17
A framework called LEGO-RL fixes broken reward signals and mismatches when training coding AI agents with reinforcement learning
LEGO-RL connects existing coding-agent programs like Claude Code, OpenHands SDK, and OpenCode to reinforcement learning training without changing how those agents work internally. It fixes two core problems: corrupted reward signals from crashes or cheating, and mismatches between what the agent actually did and what the training process calculates. Training the Qwen3.5-35B-A3B model with LEGO-RL across all three agent setups produced large gains on the SWE-bench Verified coding benchmark.
METAL MEDIA explanatory visual
A framework called LEGO-RL fixes broken reward signals and mismatches when training coding AI agents with reinforcement learning
01Training coding agents with reinforcement learning requires exactly reproducing, at training time, the tokens the agent generated live -- but existing agent programs (harnesses) often rewrite or compress their own history, breaking that reproduction
02LEGO-RL inserts an in-process proxy at the point where the agent calls the language model, capturing the exact tokens and probabilities generated, and for mixture-of-experts models it also replays which internal experts were used so training matches what actually happened
03Inside isolated sandboxes, the framework caches container images for speed and adds stage-by-stage defenses to stop agents from gaming the reward (like peeking at grading answers), while filtering out failed runs so they don't corrupt training
04A live monitoring dashboard lets researchers trace training curve problems down to specific failed trials, termination reasons, and task-level behavior
05Solve rates on SWE-bench Verified rose from 64.0% to 70.4% on OpenHands SDK, 62.4% to 68.2% on Claude Code, and 57.2% to 66.6% on OpenCode, while the correlation between rollout-time and training-time probabilities stayed above 0.99
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.
What they did
Training coding agents with reinforcement learning requires exactly reproducing, at training time, the tokens the agent generated live -- but existing agent programs (harnesses) often rewrite or compress their own history, breaking that reproduction
LEGO-RL inserts an in-process proxy at the point where the agent calls the language model, capturing the exact tokens and probabilities generated, and for mixture-of-experts models it also replays which internal experts were used so training matches what actually happened
Inside isolated sandboxes, the framework caches container images for speed and adds stage-by-stage defenses to stop agents from gaming the reward (like peeking at grading answers), while filtering out failed runs so they don't corrupt training
A live monitoring dashboard lets researchers trace training curve problems down to specific failed trials, termination reasons, and task-level behavior
Solve rates on SWE-bench Verified rose from 64.0% to 70.4% on OpenHands SDK, 62.4% to 68.2% on Claude Code, and 57.2% to 66.6% on OpenCode, while the correlation between rollout-time and training-time probabilities stayed above 0.99
Figure 1: Overview of the Lego-RL training infrastructure.
Table 1: Comparison of representative agentic RL frameworks. ✓: supported; △: partial/conditional support; –: not reported. R3: rollout routing replay. Observability denotes monitoring training runs and diagnosing execution- or trajectory-level failures.
Harness-native fidelity
Execution & reward
Observability
Framework
Black-box harness
Token-in/ Token-out
History alignment
R3
Fully async
Sandbox execution
Reward-hack defense
Training observability
verl (Sheng et al. 2025)
–
✓
–
✓
✓
△
–
△
slime (THUDM 2025)
✓
✓
✓
✓
✓
△
△
–
MOLT (NVIDIA NeMo 2026)
△
✓
–
✓
✓
–
–
–
SkyRL-Agent (Cao et al. 2025)
△
△
△
✓
✓
✓
–
–
AReaL (Fu et al. 2025)
✓
△
–
–
✓
–
–
–
Agent Lightning (Luo et al. 2025)
△
✓
–
–
△
–
–
△
Polar (Xu et al. 2026)
✓
✓
✓
–
✓
✓
–
–
rLLM (Berkeley Sky Computing Lab 2026)
✓
✓
–
✓
✓
✓
–
✓
OpenForgeRL (Yu et al. 2026)
✓
△
–
–
✓
✓
–
–
ALE (ROLL/ROCK) (Wang et al. 2025a)
–
–
–
–
✓
✓
△
–
Lego-RL
✓
✓
✓
✓
✓
✓
✓
✓
Figure 2: Closed-loop operational workflow of Lego-RL. The five stages cover data preparation, run validation, training, live observability, and human review. Stage (3) corresponds to the training infrastructure shown in Figure 1, while the agent plugin acts as the control plane.
Table 2: SWE-bench Verified performance across the three coding agents. All numbers are measured by us under the same harness version and evaluation protocol (temperature 0.7, 200 turns, 200k context budget).
Coding agent
Model
SWE-bench Verified (%)
OpenHands SDK
Qwen3.5-35B-A3B (Qwen Team 2026a)
64.0
Qwen3.6-35B-A3B (Qwen Team 2026b)
67.4
KAT-Coder-V2.5-Dev (KwaiKAT Team 2026)
67.0
Lego-RL-Qwen3.5-35B-A3B
70.4(+6.4)
Claude Code
Qwen3.5-35B-A3B (Qwen Team 2026a)
62.4
Qwen3.6-35B-A3B (Qwen Team 2026b)
63.4
KAT-Coder-V2.5-Dev (KwaiKAT Team 2026)
66.8
Lego-RL-Qwen3.5-35B-A3B
68.2(+5.8)
OpenCode
Qwen3.5-35B-A3B (Qwen Team 2026a)
57.2
Qwen3.6-35B-A3B (Qwen Team 2026b)
60.6
KAT-Coder-V2.5-Dev (KwaiKAT Team 2026)
64.8
Lego-RL-Qwen3.5-35B-A3B
66.6(+9.4)
Figure 3: Training behavior of OpenHands SDK, Claude Code, and OpenCode over three epochs (126 training steps), showing training reward, validation reward, policy entropy, and mean response length.
Table 3: Rollout-to-training alignment over the three matched production runs: probabilities captured at the serving boundary against trainer-side recomputation over the corresponding assistant tokens. All statistics are medians over training steps.
|Δlogp¯| per trajectory (×10−3)
Coding agent
Pearson r
KL (×10−3)
p50
p90
p99
OpenHands SDK
0.9993
0.75
0.7
1.2
2.1
Claude Code
0.9980
1.35
0.7
1.3
2.7
OpenCode
0.9993
0.60
0.6
1.1
2.0
Figure 4: Trajectory termination profiles across agent scaffolds. Bars show the proportion of trajectories by termination reason; timeout and environment-setup failures are excluded from optimization.
Table 4: Stage-wise wall-clock statistics across 3,699 OpenHands SDK training trials. Mean-time fraction is computed relative to the mean total trial duration; Other includes scheduling, trajectory handling, and sandbox cleanup.
Stage
Mean (s)
p50
p90
p99
Mean-time fraction
Sandbox setup
21.6
7.7
41.8
275.2
2.3%
Agent setup
4.2
4.0
4.8
8.4
0.5%
Agent execution
840.5
708.4
1604.3
2801.1
91.3%
Verification
35.9
4.1
29.6
928.5
3.9%
Other
20.3
8.1
12.3
965.5
2.2%
Trial total
920.4
770.0
1742.7
3189.1
100%
Figure 5: In-batch reward distributions across training. (a–c) Distribution of tasks by the number of successful rollouts out of eight in the first and last epoch for each scaffold. The 0/8 and 8/8 groups provide no group-relative advantage. (d) Combined proportion of these zero-variation groups across epochs.
Table 5: Ablation of sandbox optimizations.
Median latency
Paired ratio
Optimization
Stage
With
Without
Median
p10–p90
n
Lazy image pull
sandbox setup
1.57 s
2.66 s
1.7×
—
100
Prebuilt task image
sandbox setup
1.04 s
36.2 s
33.2×
17.9–67.5×
50
Mounted agent runtime
agent setup
0.51 s
7.82 s
15.4×
14.5–16.6×
50
Packaged grading toolchain
verification
3.81 s
2.72 s
0.71×
0.67–0.73×
50
Figure 6: Task-selection ablation across four 951-task pools. (a) Held-out validation solve rate, measured over the validation tasks that executed. (b) Training verifier reward; levels are pool-specific, so only the slopes are comparable.
Table 6: Reward-integrity failure modes and their mitigations. Incidence rates are measured prior to deploying the defenses; “—” indicates cases not separately quantified.
Failure Mode
Incidence
Defense
Agent-side: shortcut exploitation
Reads git history
4.6–20.5%
Rebase history to a single commit during agent phase; restore before grading
Downloads reference fix
1.9%
Per-phase egress firewall in privilege-separated sidecar
Edits test files
2.4–19.4%
Withhold tests until grading; revert test-path edits
Environment-side: reward detached from agent
Grader applies reference patch
2.5%
Audit affected instances out of pool; flag live if degenerate reward propagates
Grader requires network access
—
Package all grade-time dependencies; ensure deterministic invocation
Incomplete repository build
—
Hermetic fail-fast build; report setup failure explicitly, not as zero reward
Figure 7: Failure diagnosis with the Live UI. (a) Per-step termination reasons for an environment-failure run. (b) Assisted analysis of a collapsed run and the corresponding early-stop condition. (a) and (b) are two different diagnostic runs, neither is the Claude Code production run reported elsewhere in this section.
Table 7: Routing-replay configurations compared. Expert overlap and top-1 agreement are undefined when replay is disabled.
Routing replay configuration
Pearson r
Mean |Δp|
Expert overlap
Top-1 agreement
Disabled
0.9946
0.0062
—
—
Enabled, misaligned
0.7503
0.0954
0.083
0.026
Enabled, aligned
0.9993
0.0025
0.996
0.985
Figure 8: Behavioral analysis with the Live UI, Claude Code run. (a) Tool-use trajectories for eight rollouts of one task. (b) Task-level solve rates in the first and last sampled epochs.
Table 8: Evidence supplied for the collapsed run. t is the least-squares slope divided by its standard error over all logged steps; the gradient norm does not clear |t|=2 and is therefore read as noise. Tokens per turn is a ratio of two rows above it and carries no separate trend statistic.
Series
First five steps
Last five steps
t
training reward
0.351
0.050
−7.5
turns per trajectory
18.9
0.96
−14.8
response length (tokens)
12,170
2,276
−7.7
tokens per turn
650
2,397
—
policy entropy
0.134
0.230
+7.2
KL term of the loss
0.0046
0.130
+3.4
rollout–training agreement
0.995
0.970
−3.3
gradient norm
0.31
0.034
+0.4
Figure 9: Trainer schedule under synchronous and asynchronous execution.
Table 9: Agent behaviors before and after training, over 420 trajectories at each end of the production OpenHands SDK run; the pass@k rows are over prompt groups in the first and last third of the run.
Behavior
First
Last
Δ
Reads back a file it edited
73.6%
98.1%
+24.5
Runs the test suite
85.0%
93.6%
+8.6
Files explored before 1st edit
3.45
6.92
+3.47
Ends with an explicit finish
88.3%
91.9%
+3.6
Reproduces failure before editing
6.7%
11.2%
+4.5
Solves despite a failed command
63.9%
66.8%
+2.9
Malformed tool calls (of all calls)
1.07%
0.15%
−0.92
Coverage, pass@8
83.2%
87.9%
+4.7
Reliability, pass8
28.3%
39.4%
+11.1
Figure 10: Lazy versus full image delivery over the same 100 task images. Nydus streams image chunks on demand, whereas OCI denotes the conventional pull, which materializes the entire image before the container starts. The panels report startup latency, cumulative network and disk traffic, and in-container read throughput.
Table 10: Resolved hyperparameters of the three production runs (Qwen3.5-35B-A3B through the OpenHands SDK, Claude Code, and OpenCode).
Policy loss
GSPO
Clip range (sequence-level)
(3×10−4, 4×10−4)
Advantage estimator
GRPO
Loss aggregation
seq-mean-token-mean
KL reward penalty
none
KL loss coefficient
10−3
Learning rate
1×10−6
Learning-rate schedule
constant
Gradient clip
1.0
Prompts per batch
64
Rollouts per prompt
8
Micro-batch per GPU
1
Rollout temperature
1.0
Rollout top-p
1.0
Validation temperature
0.7
Validation samples per instance
1
Prompt budget
30k tokens
Response budget
170k tokens
Staleness threshold
1
Partial-rollout recovery
on
Importance-sampling correction
off
Training pool
2,699 tasks
Epochs
3
Figure 11: Reasoning share of the response over training on a fixed 120-task validation cohort. (a) Mean over tasks (solid) and character-weighted mean (dashed); (b) Median share at each task’s first (open) and last (filled) sampled epoch, grouped by number of rollouts solved.
Why it matters
Teams building coding agents can now apply reinforcement learning improvements to their existing agent software without rebuilding it, while catching hidden failures that would otherwise silently corrupt training. This matters as agent-based coding tools move from research demos to production-scale training pipelines.
Figure 12: Agent behaviors before and after training, computed over 420 trajectories at each end of the production OpenHands SDK run. The pass@k and passk rows are computed over prompt groups in the first and last third of the run.
Terms in this paper
reinforcement learning (RL) · a training method where a system improves its behavior based on rewards for outcomes
harness · the software that manages an AI agent's tool use, context, and execution feedback
sandbox · an isolated environment where code is run and tested safely
mixture-of-experts (MoE) · a large model architecture that only activates a subset of its internal sub-networks per input
SWE-bench Verified · a benchmark that checks how well an agent resolves real software repository issues
Original abstract (English)
Reinforcement learning for coding agents increasingly relies on long-running agent harnesses to manage tool integration, repository contexts, and execution feedback. However, the native execution environments of these harnesses are inherently misaligned with policy-gradient training: environmental crashes and reward hacking corrupt outcome signals, while train-inference discrepancies decouple rollout behavior from policy updates. To address this, we present LEGO-RL, a framework that bridges native coding-agent harnesses with scalable policy-gradient optimization without modifying their internal control flow. LEGO-RL is built upon three pillars: (1) faithful optimization via in-process LLM proxying that captures raw generation streams for token-level alignment and robust trainer-side log-probability recomputation, even under harness-side compaction or re-serialization; (2) reliable execution via scalable sandbox orchestration featuring image caching and stage-wise defenses to mitigate reward hacking; and (3) observable training through an integrated plugin that automates validation and monitoring, paired with a Live UI for granular trajectory diagnostics. We evaluate LEGO-RL by training the sparse MoE model Qwen3.5-35B-A3B with GSPO across three native coding-agent harnesses. LEGO-RL improves Qwen3.5-35B-A3B across OpenHands SDK (64.0% to 70.4%), Claude Code (62.4% to 68.2%), and OpenCode (57.2% to 66.6%) on SWE-bench Verified, while maintaining a rollout-training probability correlation above 0.99.
