Table 1: Training the Reasoning Agent on diverse synthetic games improves held-out reasoning and code benchmarks at every backbone scale. The eight held-out benchmarks probe four capability families: competition math (AIME 2025/2026, Avg@32), science reasoning (GPQA-Diamond, accuracy), code generation (LiveCodeBench-v6, Pass@1), and procedural reasoning across four cognitive skills (Reasoning-Gym, win rate at hard). Both fixed-environment baselines are retrained per backbone from the same base model for 400 training iterations; Fixed-env RLVE follows the official RLVE sampling and curriculum settings. Avg is the unweighted mean over the eight benchmarks; green subscripts denote absolute pp gain over the same-model base. Best per column within each backbone block in bold, second best underlined.
Model
AIME’25
AIME’26
GPQA-D
LCB-v6
Reasoning-Gym (Stojanovski et al. 2026)
Δ vs Base
(MAA)
(MAA)
(Rein et al. 2023)
(Jain et al. 2024)
RG-Math
RG-Algo.
RG-Cog.
RG-Logic
Avg
Qwen3 backbones: fixed-environment baselines and SPADE (game environment design)
Qwen3-4B-Instruct-2507
47.4
58.3
55.9
35.1
31.6
11.8
16.4
54.7
38.9
–
Fixed-env GRPO
47.1
58.6
56.2
35.4
34.0
13.5
18.1
56.3
39.9
+1.0
Fixed-env RLVE
49.6
62.1
57.3
35.9
38.6
16.3
21.0
59.1
42.5
+3.6
+ SPADE (Games)
48.9+1.5
60.2+1.9
58.1+2.2
37.2+2.1
44.6+13.0
19.8+8.0
23.1+6.7
60.8+6.1
44.1
+5.2
Qwen3-8B
67.1
71.2
59.4
46.3
47.2
19.6
24.8
63.1
49.8
–
Fixed-env GRPO
67.4
71.0
59.9
46.8
49.8
21.9
26.5
64.6
51.0
+1.2
Fixed-env RLVE
69.6
75.0
61.6
47.4
52.7
25.0
31.0
67.9
53.8
+3.9
+ SPADE (Games)
68.8+1.7
73.1+1.9
62.9+3.5
49.4+3.1
57.3+10.1
29.2+9.6
33.8+9.0
69.7+6.6
55.5
+5.7
Qwen3-30B-A3B-Instruct-2507
61.5
73.5
70.4
43.2
45.0
18.0
23.0
67.0
50.2
–
Fixed-env GRPO
61.2
73.8
70.9
43.7
48.1
20.3
24.6
68.4
51.4
+1.2
Fixed-env RLVE
56.9
69.8
69.8
42.5
55.8
24.7
30.9
73.7
53.0
+2.8
+ SPADE (Games)
62.8+1.3
74.4+0.9
75.8+5.4
47.3+4.1
63.3+18.3
32.1+14.1
37.7+14.7
72.8+5.8
58.3
+8.1
Table 2: Synthetic tool-use environments match dedicated data-synthesis systems and surpass them where multi-step interaction matters most. Per-domain results on BFCL v4 multi-turn (Patil et al. 2025), τ2-bench (Barres et al. 2025), and ACEBench-Agent (Chen et al. 2025a). Reference rows are transcribed from the cited papers. For our rows, Avg is the unweighted mean of the shown subcolumns and the final Avg averages the three benchmarks, computed before rounding. For reference rows, Avg is likewise the unweighted mean of the shown subcolumns, or the cited paper’s own aggregate where the subcolumns are not reported; Agent-World and AWM print τ2 aggregates of 61.8/65.4 and (task-weighted) 33.5/39.0, and we print the means of the shown domains for cross-row consistency. The final Avg is omitted because each reference system skips at least one benchmark. ‘–’ = not reported by the cited paper. SPADE rows in bold.
BFCL v4 (multi-turn)
τ2-bench
ACEBench-Agent
Model
Base
Miss Func
Miss Param
Long Ctx
Avg
Retail
Airline
Telecom
Avg
Multi Step
Multi Turn
Avg
Avg
Δ
Synthetic-environment agents (per-split numbers as reported by the cited papers)
AgentScaler-30B-A3B (Fang et al. 2025a)
–
–
–
–
–
70.2
60.0
55.3
61.8
–
–
60.0
Agent-World-8B (Dong et al. 2026)
–
–
–
–
44.5
72.8
40.0
50.9
54.6
–
–
–
Agent-World-14B (Dong et al. 2026)
–
–
–
–
53.9
74.5
52.0
56.1
60.9
–
–
–
AWM-8B (Wang et al. 2026b)
–
–
–
–
45.0
41.2
38.5
23.5
34.4
–
–
–
AWM-14B (Wang et al. 2026b)
–
–
–
–
51.9
63.6
31.5
17.8
37.6
–
–
–
EnvScaler-4B (Song et al. 2026)
51.0
34.0
28.0
39.0
38.0
–
–
–
–
80.0
61.1
70.6
EnvScaler-8B (Song et al. 2026)
55.5
36.0
35.0
41.0
41.9
–
–
–
–
85.0
60.0
72.5
Ours: SPADE post-training on Qwen3 backbones (tool-use environment design)
Qwen3-4B-Instruct-2507
34.0
16.0
12.5
25.5
22.0
43.0
32.0
18.0
31.0
55.0
41.7
48.4
33.8
+ SPADE ♠ (Tool Use)
46.0
26.5
22.0
34.7
32.3+10.3
47.2
35.6
21.5
34.8+3.8
65.0
49.5
57.3+8.9
41.4
+7.7
Qwen3-8B
52.0
30.0
24.0
35.6
35.4
34.0
26.5
18.0
26.2
63.3
56.7
60.0
40.5
+ SPADE ♠ (Tool Use)
58.0
36.0
30.0
43.2
41.8+6.4
37.8
29.5
21.2
29.5+3.3
73.0
65.0
69.0+9.0
46.8
+6.2
Qwen3-30B-A3B-Instruct-2507
66.0
44.0
38.0
48.0
49.0
62.0
50.0
35.0
49.0
70.0
54.0
62.0
53.3
+ SPADE ♠ (Tool Use)
72.0
50.0
44.0
52.9
54.7+5.7
65.5
53.5
38.8
52.6+3.6
82.0
69.8
75.9+13.9
61.1
+7.7
Table 3: The full adaptive configuration outperforms every partial and frozen-designer control. Games setting, Qwen3-30B-A3B-Instruct-2507. Best checkpoint per variant on suite average; Avg is the unweighted mean over the same eight benchmarks as Table 1; full trajectories in Figure 11. Best in bold, second best underlined.
Components
Benchmarks
Setting
ED design
ED trained
Corpus grounding
Env. memory
AIME’25
AIME’26
GPQA-D
LCB-v6
RG-Math
RG-Algo.
RG-Cog.
RG-Logic
Avg
Qwen3-30B-A3B-Instruct-2507
–
–
–
–
61.5
73.5
70.4
43.2
45.0
18.0
23.0
67.0
50.2
SPADE ♠
Self
✓
✓
✓
62.8
74.4
75.8
47.3
63.3
32.1
37.7
72.8
58.3
w/o memory
Self
✓
✓
✗
59.3
75.0
72.3
45.7
49.1
22.9
30.7
70.9
53.2
w/o corpus grounding
Self
✓
✗
✓
61.1
74.1
71.8
46.3
51.6
22.3
32.4
68.7
53.5
w/o ED training and memory
Self
✗
✓
✗
59.4
73.5
65.8
39.1
22.5
10.0
7.6
46.0
40.5
Fixed Environment Designer (GPT-5.5)
GPT-5.5
✗
✓
✓
59.9
72.8
74.2
42.6
51.2
24.3
30.7
68.0
53.0
Table 4: Symbols used throughout the paper.
Symbol
Description
MDP and environment
𝒮
State space
𝒜
Action space
T(s′∣s,a)
Transition function
R(s,a)
Reward function
ρ0
Initial state distribution
s,s′,a
State, next state, action
ℰ
Space of valid (executable Python) environments
e∈ℰ
A single environment instance
eb
The b-th environment in a generation batch
Policy and roles
πθ
Shared LLM policy with parameters θ
πD
Policy in Environment Designer role, πθ(⋅∣role=D)
πA
Policy in Reasoning Agent role, πθ(⋅∣role=A)
role=D
System-prompt switch selecting Environment Designer role
role=A
System-prompt switch selecting Reasoning Agent role
Hints and rewards
h
Privileged hint (strategy / partial solution / key observation)
hb
Hint for the b-th environment
y,yi
Reasoning Agent response (rollout)
yi′
Reasoning Agent rollout sampled with the privileged hint in context
rA(y∣e)
Per-rollout correctness reward (without hint)
rA(y∣e,h)
Per-rollout correctness reward conditioned on hint
plateau 0.6, band [0.4,0.6] ([0.2,0.4] later for 4B)
+ floored regret 0.4
Regret scale (normalizer)
0.15
Plateau ramp width
0.25
Delayed Environment Designer update
4 rollouts
Corpus grounding
15k docs (10k math, 5k science)
Environment memory
on
Table 6: Full ablation breakdown (games setting, Qwen3-30B-A3B-Instruct-2507). Best checkpoint per variant on the suite average. AIME reports Avg@32; GPQA-D accuracy; LCB-v6 Pass@1; Reasoning-Gym (RG) win rate at hard; GEM the overall win rate across the GEM game suite (Liu et al. 2025c). Best in bold.
Setting
Ckpt
AIME’25
AIME’26
GPQA-D
LCB-v6
RG-Math
RG-Algo.
RG-Cog.
RG-Logic
GEM
Avg
Qwen3-30B-A3B-Instruct-2507
–
61.5
73.5
70.4
43.2
45.0
18.0
23.0
67.0
41.0
50.2
SPADE ♠
303
62.8
74.4
75.8
47.3
63.3
32.1
37.7
72.8
50.2
58.3
Environment Designer w/ learning potential
–
62.4
74.1
74.2
46.1
57.8
27.9
33.3
71.1
47.4
55.9
2-skill curriculum
399
60.6
71.3
71.7
44.7
53.6
26.7
30.5
70.7
46.0
53.7
w/o corpus grounding
111
61.1
74.1
71.8
46.3
51.6
22.3
32.4
68.7
45.1
53.5
w/o memory
111
59.3
75.0
72.3
45.7
49.1
22.9
30.7
70.9
42.4
53.2
w/o Environment Designer training and memory
271
59.4
73.5
65.8
39.1
22.5
10.0
7.6
46.0
26.4
40.5
Fixed Environment Designer (GPT-5.5)
175
59.9
72.8
74.2
42.6
51.2
24.3
30.7
68.0
45.6
53.0
Table 7: Environment quality over training. Learnability nearly doubles while well-posedness, verifiability, and structural richness hold constant: the quality gains come from sharper difficulty targeting rather than simpler environments. Win-rate rows use the released per-step evaluation logs.
Quality signal
Early (0–40)
Mid (150–250)
Late (340–396)
Learnable-band fraction (win-rate ∈[0.2,0.8])
0.16
0.16
0.31
Reasoning Agent win-rate
0.30
0.46
0.62
Well-posed (LLM rubric)
0.98
0.97
0.97
Verifiable terminal answer (LLM rubric)
0.90
0.91
0.93
Interaction depth (turns / episode)
8.8
8.2
9.8
Program length (lines of code)
316
333
321
Hidden state variables
13.0
13.3
13.4
Table 8: Corpus ablations collapse environment diversity. Vendi Score per 100 environments (SBERT embeddings; mean over 20 balanced draws) on the verified-matched 30B-A3B runs: identical backbone, skill set, and Environment Designer system prompt, differing in the listed ablations. The identical analysis under TF-IDF/LSA embeddings reproduces the corpus/no-corpus separation (0.53/0.58/0.48/0.05). A fifth run with a static pre-generated environment pool (Reasoning Agent-only training, no live Environment Designer) is excluded as a different generation protocol; its fixed pool measures Vendi/n=0.12.
Figure 2: SPADE generates an adaptive, multi-turn curriculum. Four environments the Environment Designer produces over one 30B-A3B run, from step 0 (early) to step 384 (late); each card shows the agent’s first observation, the generated Python environment, and the designer-written hint. Every environment is a complete MDP with a reset()/step() interface, and the tasks shift toward state-gated, multi-turn interaction as the Reasoning Agent improves. Unlike a fixed human-curated pool or a frozen synthetic generator, this curriculum keeps moving with the learner.
Figure 4: The SPADE framework. Top: the Environment Designer conditions on the environment memory M and pretraining corpus C to emit an executable environment e and a privileged hint h. Bottom: the Reasoning Agent plays e with and without h; the return gap is the Environment Designer’s hint-based regret rD(e) (Eq. 3) and task correctness is the Reasoning Agent reward. Both rewards update the shared policy πθ via GRPO.
Figure 5: How a privileged hint changes Reasoning Agent play. Two positive-regret examples from the canonical 30B games run. Left: each environment’s task prompt and privileged hint, quoted verbatim (ellipses mark elided text; the standardized answer-format sentence is omitted from the hint). Right: one logged Reasoning Agent rollout per arm, condensed while preserving action order, feedback, and values; elided turns are marked and named. The two arms are independent plays with independently seeded resets, so board layouts and probe outcomes differ across arms. The dashed box on each environment card reports the two displayed rollout returns and their single-pair gap; the Environment Designer reward in Equation 3 is instead the difference of the arm means over all logged rollouts (0.00→1.00 for the fiber task, 0.30→0.65 for the audio task). An expanded task–hint set appears in Appendix J.1.1.
Figure 6: Training on diverse synthetic games improves science reasoning, code generation, and procedural reasoning while competition math is preserved (games setting, Qwen3-30B-A3B-Instruct-2507). Top row: competition math (AIME 2025/2026 Avg@32), science reasoning (GPQA-Diamond accuracy), and code generation (LiveCodeBench-v6 Pass@1). Bottom row: procedural reasoning across four cognitive skills (Reasoning-Gym win rate at hard). Markers denote logged evaluation checkpoints; the dashed line marks the untrained base model.
Figure 7: Full SPADE raises the learnable share of its environment budget to roughly a third by the end of training; component ablations decline or collapse. Share of each rollout’s 24 environments that is learnable, defined as Reasoning Agent win rate in [0.2,0.8] and weighted by the number of valid environments generated in each 16-step window. Matched 30B-A3B settings over a common 400-step budget; unfilled rollout capacity contributes zero by construction.
Figure 9: Trained Environment Designer environments stop revealing the solution method in the prompt. Physics environments from one 30B-A3B run at steps 20, 192, and 384; the step-20 environment retains the scaffold’s default class name. The formula-reveal rate (percentage on each panel) falls from 25% to 5% over 473 environments. Rightmost (red): over steps 290–312 the no-corpus ablation emits the same RotatingMazeEnv task 41 consecutive times. The complete source of the step-384 environment, together with one further exemplar, appears in Appendix J.3.2.
Figure 10: From front-loaded derivation to evidence-first interaction. Reasoning Agent episodes from one 30B-A3B run at steps 0, 200, and 300. At step 0 the agent derives in advance and cannot recover from format errors; by step 200 it tests short hypotheses and revises on evidence; by step 300 it probes first and derives once. Benchmark gains of late checkpoints (Table 1) confirm the model keeps its long-form derivation ability. Transcripts verbatim (math glyphs transliterated to ASCII; environment feedback abridged); token counts use the backbone’s tokenizer.
Figure 11: Removing Environment Designer training and memory together drops self-play below base; removing either one alone has an above-base selected checkpoint but peaks early and can fall below base late. One curve per variant of Table 3 (Qwen3-30B-A3B-Instruct-2507, games setting). Top row: AIME 2025/2026 Avg@32, GPQA-Diamond accuracy, and LiveCodeBench-v6 Pass@1; bottom row: the four Reasoning-Gym categories; the dashed line marks the untrained base model.
Figure 12: Rewarding the Environment Designer matters more than which reward is used, but regret still leads. Averaged trajectories over GPQA-Diamond, LiveCodeBench-v6, and the four Reasoning-Gym categories (Qwen3-30B-A3B-Instruct-2507, games setting); the dashed line marks the corresponding untrained base. Eight-benchmark checkpoint results are reported in Table 6.
Figure 13: SPADE’s average gain over base grows with model size, from +5.2 at 4B to +8.1 at 30B-A3B, while matched-budget Fixed-env GRPO stays near +1.2. Left: average gain over each backbone’s own base across the eight benchmarks of Table 1 (large markers), the eight per-benchmark gains beside each mean, Fixed-env GRPO in gray. Right: Environment Designer hint-based regret over training (dark: EMA-smoothed; light: per-step). Only the 30B-A3B estimate stays positive; at 4B and 8B it dips below zero for long stretches, where the finite-sample estimate turns negative even though regret is non-negative at the optimum (Section 4). Both smaller backbones still gain over base (+5.2, +5.7), so the environments help even where the signal is noisy.
Figure 14: Curriculum breadth accounts for most of the gain. Suite average (eight benchmarks of Table 1), six-skill vs. two-skill curriculum. Per-benchmark panels: Figure 15.
Figure 15: The full 6-skill curriculum lifts held-out benchmarks more than the restricted 2-skill variant; curriculum breadth drives the gains. Qwen3-30B-A3B-Instruct-2507, games setting. Top row: AIME 2025/2026 Avg@32, GPQA-Diamond accuracy, and LiveCodeBench-v6 Pass@1. Bottom row: the four Reasoning-Gym categories; the dashed line marks the untrained base model. Discussed in Section 8.
Figure 16: SPADE improves a second backbone family: all four Reasoning-Gym categories end above the untrained Nemotron-30B-A3B-BF16 base (RG-Cognition +9.6, RG-Algorithmic +9.3, RG-Math +2.6, RG-Logic +3.2). Reasoning-Gym win rate across checkpoints; the dashed line marks the untrained base model. Gains arrive after an initial dip early in training, consistent with the Environment Designer initially generating environments too hard for the Reasoning Agent, as observed on the Qwen backbones.
Figure 17: Reward granularity increases over training. Left: mean distinct reward levels per environment, including strictly partial levels. Right: early-to-late change in distinct levels overall and by domain (canonical 30B games run).
Figure 18: Per-skill learning (canonical SPADE-30B run). (a) Per-skill Reasoning Agent win rate over training (EMA over ∼26 logged points/skill). (b) Win-rate gain (last 25% minus first 25%): Logical Deduction and Spatial Reasoning improve most; Causal Inference, which starts high, declines slightly.
Figure 19: Environment-diversity dynamics across the verified-matched runs. Windowed Vendi score (40-step windows, subsampled to exactly 24 environments; mean ± s.d. over 12 draws). The corpus-grounded variants hold the mixed-population reference level for their full run lengths; the no-corpus run starts near-collapsed, peaks at Vendi ∼5.2 near step 100, where its best evaluation checkpoint (111) also falls, and re-collapses as optimization continues. Environment counts differ across runs with run length and acceptance rate; the windowed score uses fixed 24-environment subsamples, so counts do not bias the curves.
Figure 20: Task context makes hint utility legible. Four same-record, positive-regret task–hint pairs from the canonical 30B games run. The task summaries retain the goal, hidden information, and usable interaction while removing generic runtime scaffolding. Hint excerpts omit only the standardized answer-format sentence and runtime wrapper. In each header, the two values report the mean return without hint / with hint.
Continuous self-improvement requires an ever-expanding pool of self-generated, diverse, adaptive goals. For language agents, existing training environment pools (hand-curated, statically synthesized, or frozen-verifier) keep the goal distribution fixed as the learner scales. We introduce SPADE (Self-Play in Adaptive Synthetic Executable Environments), a self-play RL framework in which a single LLM plays two roles: an Environment Designer that writes complete, long-horizon training environments as executable code with an OpenAI Gym-style reset()/step() interface, and a Reasoning Agent that learns to act in them. Each is a stateful, multi-turn environment (state transitions, reward functions, and verification code), so one interface spans reasoning problems and multi-step agentic tool use. The Reasoning Agent's regret is estimated using the gap between its reward with and without privileged hints; in optimizing this regret signal the Environment Designer learns to target environments at the edge of the agent's capabilities while keeping them feasible. Through extensive experimentation, we find several components critical to success: grounding the Environment Designer on documents sampled from a large pretraining corpus, and giving it an accumulated environment memory. Scaling to 30B-parameter models, SPADE improves over the strongest fixed-environment baseline by +5.3 on average across eight held-out math, science, code, and reasoning benchmarks, and lifts the tool-use setting by +5.7 on BFCL-v4 multi-turn and +13.9 on ACEBench-Agent; on the games setting, the margin over the strongest baseline grows with model scale. By making environment design itself a learnable component, SPADE takes a concrete step toward open-ended self-improvement.
