코딩 AI 에이전트를 강화학습으로 훈련시킬 때 생기는 거짓 신호와 속도 문제를 잡아주는 틀, LEGO-RL
arXiv:2608.173932026-08-17
LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents
코딩 AI 에이전트를 강화학습으로 훈련시킬 때 생기는 거짓 신호와 속도 문제를 잡아주는 틀, LEGO-RL
LEGO-RL은 Claude Code, OpenHands SDK, OpenCode 같은 기존 코딩 에이전트 프로그램을 건드리지 않은 채로 강화학습 훈련에 연결해주는 프레임워크다. 실행 환경 오류나 보상 조작으로 학습 신호가 오염되는 문제, 그리고 실제 행동과 학습 계산이 어긋나는 문제를 동시에 해결한다. Qwen3.5-35B-A3B 모델을 세 가지 에이전트 환경에서 훈련시킨 결과 SWE-bench Verified 문제 해결률이 모두 크게 올랐다.
METAL MEDIA 해설 도표
코딩 AI 에이전트를 강화학습으로 훈련시킬 때 생기는 거짓 신호와 속도 문제를 잡아주는 틀, LEGO-RL
01강화학습으로 코딩 에이전트를 훈련시키려면 에이전트가 실제로 사용하는 프로그램(하니스)의 내부 동작을 손대지 않으면서도, 그 안에서 실제로 어떤 토큰이 생성됐는지 정확히 기록해야 하는데 기존 방식은 이를 놓치기 쉬웠다
02LEGO-RL은 모델 호출이 오가는 지점에 프록시를 심어 실제 생성된 토큰과 확률값을 그대로 포착하고, 여러 전문가 모델 중 일부만 쓰는 구조(MoE)에서도 훈련 시 그 선택을 그대로 재현하도록 만들었다
03샌드박스(격리된 실행 공간)에서 이미지 캐싱과 단계별 방어 장치를 두어 코드가 몰래 채점 정보를 훔쳐보는 식의 보상 조작을 막고, 훈련 도중 실패한 시도가 전체 학습을 방해하지 않도록 걸러낸다
04실시간 모니터링 화면(Live UI)을 통해 어떤 단계에서 실패가 나는지, 어떤 작업이 풀리고 안 풀리는지를 사람이 직접 들여다보고 진단할 수 있게 했다
05OpenHands SDK에서 64.0%→70.4%, Claude Code에서 62.4%→68.2%, OpenCode에서 57.2%→66.6%로 SWE-bench Verified 해결률이 올랐고, 롤아웃 시점과 학습 시점의 확률 값 일치도가 0.99 이상으로 유지됐다
METAL MEDIA이 원문을 바탕으로 재구성한 해설 도표이며, 논문 저자의 원문 figure가 아닙니다.
무엇을 했나
강화학습으로 코딩 에이전트를 훈련시키려면 에이전트가 실제로 사용하는 프로그램(하니스)의 내부 동작을 손대지 않으면서도, 그 안에서 실제로 어떤 토큰이 생성됐는지 정확히 기록해야 하는데 기존 방식은 이를 놓치기 쉬웠다
LEGO-RL은 모델 호출이 오가는 지점에 프록시를 심어 실제 생성된 토큰과 확률값을 그대로 포착하고, 여러 전문가 모델 중 일부만 쓰는 구조(MoE)에서도 훈련 시 그 선택을 그대로 재현하도록 만들었다
샌드박스(격리된 실행 공간)에서 이미지 캐싱과 단계별 방어 장치를 두어 코드가 몰래 채점 정보를 훔쳐보는 식의 보상 조작을 막고, 훈련 도중 실패한 시도가 전체 학습을 방해하지 않도록 걸러낸다
실시간 모니터링 화면(Live UI)을 통해 어떤 단계에서 실패가 나는지, 어떤 작업이 풀리고 안 풀리는지를 사람이 직접 들여다보고 진단할 수 있게 했다
OpenHands SDK에서 64.0%→70.4%, Claude Code에서 62.4%→68.2%, OpenCode에서 57.2%→66.6%로 SWE-bench Verified 해결률이 올랐고, 롤아웃 시점과 학습 시점의 확률 값 일치도가 0.99 이상으로 유지됐다
Figure 1: Overview of the Lego-RL training infrastructure.
Table 1: Comparison of representative agentic RL frameworks. ✓: supported; △: partial/conditional support; –: not reported. R3: rollout routing replay. Observability denotes monitoring training runs and diagnosing execution- or trajectory-level failures.
Harness-native fidelity
Execution & reward
Observability
Framework
Black-box harness
Token-in/ Token-out
History alignment
R3
Fully async
Sandbox execution
Reward-hack defense
Training observability
verl (Sheng et al. 2025)
–
✓
–
✓
✓
△
–
△
slime (THUDM 2025)
✓
✓
✓
✓
✓
△
△
–
MOLT (NVIDIA NeMo 2026)
△
✓
–
✓
✓
–
–
–
SkyRL-Agent (Cao et al. 2025)
△
△
△
✓
✓
✓
–
–
AReaL (Fu et al. 2025)
✓
△
–
–
✓
–
–
–
Agent Lightning (Luo et al. 2025)
△
✓
–
–
△
–
–
△
Polar (Xu et al. 2026)
✓
✓
✓
–
✓
✓
–
–
rLLM (Berkeley Sky Computing Lab 2026)
✓
✓
–
✓
✓
✓
–
✓
OpenForgeRL (Yu et al. 2026)
✓
△
–
–
✓
✓
–
–
ALE (ROLL/ROCK) (Wang et al. 2025a)
–
–
–
–
✓
✓
△
–
Lego-RL
✓
✓
✓
✓
✓
✓
✓
✓
Figure 2: Closed-loop operational workflow of Lego-RL. The five stages cover data preparation, run validation, training, live observability, and human review. Stage (3) corresponds to the training infrastructure shown in Figure 1, while the agent plugin acts as the control plane.
Table 2: SWE-bench Verified performance across the three coding agents. All numbers are measured by us under the same harness version and evaluation protocol (temperature 0.7, 200 turns, 200k context budget).
Coding agent
Model
SWE-bench Verified (%)
OpenHands SDK
Qwen3.5-35B-A3B (Qwen Team 2026a)
64.0
Qwen3.6-35B-A3B (Qwen Team 2026b)
67.4
KAT-Coder-V2.5-Dev (KwaiKAT Team 2026)
67.0
Lego-RL-Qwen3.5-35B-A3B
70.4(+6.4)
Claude Code
Qwen3.5-35B-A3B (Qwen Team 2026a)
62.4
Qwen3.6-35B-A3B (Qwen Team 2026b)
63.4
KAT-Coder-V2.5-Dev (KwaiKAT Team 2026)
66.8
Lego-RL-Qwen3.5-35B-A3B
68.2(+5.8)
OpenCode
Qwen3.5-35B-A3B (Qwen Team 2026a)
57.2
Qwen3.6-35B-A3B (Qwen Team 2026b)
60.6
KAT-Coder-V2.5-Dev (KwaiKAT Team 2026)
64.8
Lego-RL-Qwen3.5-35B-A3B
66.6(+9.4)
Figure 3: Training behavior of OpenHands SDK, Claude Code, and OpenCode over three epochs (126 training steps), showing training reward, validation reward, policy entropy, and mean response length.
Table 3: Rollout-to-training alignment over the three matched production runs: probabilities captured at the serving boundary against trainer-side recomputation over the corresponding assistant tokens. All statistics are medians over training steps.
|Δlogp¯| per trajectory (×10−3)
Coding agent
Pearson r
KL (×10−3)
p50
p90
p99
OpenHands SDK
0.9993
0.75
0.7
1.2
2.1
Claude Code
0.9980
1.35
0.7
1.3
2.7
OpenCode
0.9993
0.60
0.6
1.1
2.0
Figure 4: Trajectory termination profiles across agent scaffolds. Bars show the proportion of trajectories by termination reason; timeout and environment-setup failures are excluded from optimization.
Table 4: Stage-wise wall-clock statistics across 3,699 OpenHands SDK training trials. Mean-time fraction is computed relative to the mean total trial duration; Other includes scheduling, trajectory handling, and sandbox cleanup.
Stage
Mean (s)
p50
p90
p99
Mean-time fraction
Sandbox setup
21.6
7.7
41.8
275.2
2.3%
Agent setup
4.2
4.0
4.8
8.4
0.5%
Agent execution
840.5
708.4
1604.3
2801.1
91.3%
Verification
35.9
4.1
29.6
928.5
3.9%
Other
20.3
8.1
12.3
965.5
2.2%
Trial total
920.4
770.0
1742.7
3189.1
100%
Figure 5: In-batch reward distributions across training. (a–c) Distribution of tasks by the number of successful rollouts out of eight in the first and last epoch for each scaffold. The 0/8 and 8/8 groups provide no group-relative advantage. (d) Combined proportion of these zero-variation groups across epochs.
Table 5: Ablation of sandbox optimizations.
Median latency
Paired ratio
Optimization
Stage
With
Without
Median
p10–p90
n
Lazy image pull
sandbox setup
1.57 s
2.66 s
1.7×
—
100
Prebuilt task image
sandbox setup
1.04 s
36.2 s
33.2×
17.9–67.5×
50
Mounted agent runtime
agent setup
0.51 s
7.82 s
15.4×
14.5–16.6×
50
Packaged grading toolchain
verification
3.81 s
2.72 s
0.71×
0.67–0.73×
50
Figure 6: Task-selection ablation across four 951-task pools. (a) Held-out validation solve rate, measured over the validation tasks that executed. (b) Training verifier reward; levels are pool-specific, so only the slopes are comparable.
Table 6: Reward-integrity failure modes and their mitigations. Incidence rates are measured prior to deploying the defenses; “—” indicates cases not separately quantified.
Failure Mode
Incidence
Defense
Agent-side: shortcut exploitation
Reads git history
4.6–20.5%
Rebase history to a single commit during agent phase; restore before grading
Downloads reference fix
1.9%
Per-phase egress firewall in privilege-separated sidecar
Edits test files
2.4–19.4%
Withhold tests until grading; revert test-path edits
Environment-side: reward detached from agent
Grader applies reference patch
2.5%
Audit affected instances out of pool; flag live if degenerate reward propagates
Grader requires network access
—
Package all grade-time dependencies; ensure deterministic invocation
Incomplete repository build
—
Hermetic fail-fast build; report setup failure explicitly, not as zero reward
Figure 7: Failure diagnosis with the Live UI. (a) Per-step termination reasons for an environment-failure run. (b) Assisted analysis of a collapsed run and the corresponding early-stop condition. (a) and (b) are two different diagnostic runs, neither is the Claude Code production run reported elsewhere in this section.
Table 7: Routing-replay configurations compared. Expert overlap and top-1 agreement are undefined when replay is disabled.
Routing replay configuration
Pearson r
Mean |Δp|
Expert overlap
Top-1 agreement
Disabled
0.9946
0.0062
—
—
Enabled, misaligned
0.7503
0.0954
0.083
0.026
Enabled, aligned
0.9993
0.0025
0.996
0.985
Figure 8: Behavioral analysis with the Live UI, Claude Code run. (a) Tool-use trajectories for eight rollouts of one task. (b) Task-level solve rates in the first and last sampled epochs.
Table 8: Evidence supplied for the collapsed run. t is the least-squares slope divided by its standard error over all logged steps; the gradient norm does not clear |t|=2 and is therefore read as noise. Tokens per turn is a ratio of two rows above it and carries no separate trend statistic.
Series
First five steps
Last five steps
t
training reward
0.351
0.050
−7.5
turns per trajectory
18.9
0.96
−14.8
response length (tokens)
12,170
2,276
−7.7
tokens per turn
650
2,397
—
policy entropy
0.134
0.230
+7.2
KL term of the loss
0.0046
0.130
+3.4
rollout–training agreement
0.995
0.970
−3.3
gradient norm
0.31
0.034
+0.4
Figure 9: Trainer schedule under synchronous and asynchronous execution.
Table 9: Agent behaviors before and after training, over 420 trajectories at each end of the production OpenHands SDK run; the pass@k rows are over prompt groups in the first and last third of the run.
Behavior
First
Last
Δ
Reads back a file it edited
73.6%
98.1%
+24.5
Runs the test suite
85.0%
93.6%
+8.6
Files explored before 1st edit
3.45
6.92
+3.47
Ends with an explicit finish
88.3%
91.9%
+3.6
Reproduces failure before editing
6.7%
11.2%
+4.5
Solves despite a failed command
63.9%
66.8%
+2.9
Malformed tool calls (of all calls)
1.07%
0.15%
−0.92
Coverage, pass@8
83.2%
87.9%
+4.7
Reliability, pass8
28.3%
39.4%
+11.1
Figure 10: Lazy versus full image delivery over the same 100 task images. Nydus streams image chunks on demand, whereas OCI denotes the conventional pull, which materializes the entire image before the container starts. The panels report startup latency, cumulative network and disk traffic, and in-container read throughput.
Table 10: Resolved hyperparameters of the three production runs (Qwen3.5-35B-A3B through the OpenHands SDK, Claude Code, and OpenCode).
Policy loss
GSPO
Clip range (sequence-level)
(3×10−4, 4×10−4)
Advantage estimator
GRPO
Loss aggregation
seq-mean-token-mean
KL reward penalty
none
KL loss coefficient
10−3
Learning rate
1×10−6
Learning-rate schedule
constant
Gradient clip
1.0
Prompts per batch
64
Rollouts per prompt
8
Micro-batch per GPU
1
Rollout temperature
1.0
Rollout top-p
1.0
Validation temperature
0.7
Validation samples per instance
1
Prompt budget
30k tokens
Response budget
170k tokens
Staleness threshold
1
Partial-rollout recovery
on
Importance-sampling correction
off
Training pool
2,699 tasks
Epochs
3
Figure 11: Reasoning share of the response over training on a fixed 120-task validation cohort. (a) Mean over tasks (solid) and character-weighted mean (dashed); (b) Median share at each task’s first (open) and last (filled) sampled epoch, grouped by number of rollouts solved.
왜 중요한가
코딩 에이전트를 강화학습으로 개선하려는 연구자와 기업이 기존에 이미 잘 만들어둔 에이전트 프로그램을 새로 뜯어고치지 않고도 안전하게 성능을 끌어올릴 수 있는 실용적 도구를 준다. 훈련 도중 어디서 문제가 생기는지 추적 가능하게 만든 점은 실제 서비스 규모의 훈련에서 신뢰도를 높이는 데 중요하다.
Figure 12: Agent behaviors before and after training, computed over 420 trajectories at each end of the production OpenHands SDK run. The pass@k and passk rows are computed over prompt groups in the first and last third of the run.
이 논문의 용어
강화학습(RL) · 결과에 따라 보상을 주고받으며 정책(행동 방식)을 개선해나가는 학습 방법
하니스(harness) · 에이전트가 도구 사용, 대화 맥락, 실행 결과를 관리하도록 만든 실행 프로그램
