SPADE: Self-Play in Adaptive Synthetic Executable Environments

What they did

1. Existing training setups rely on hand-built or fixed auto-generated problem pools, so once a model masters them, there's nothing left to keep it improving
2. In SPADE, the same model acts as an Environment Designer that writes full game environments (with state, rewards, and verification code) as Python programs, and as a Reasoning Agent that plays those games to learn
3. The designer is rewarded by the gap between the agent's performance with versus without a privileged hint ('hint-based regret'), which pushes it toward problems that are solvable but genuinely challenging, not too easy or impossible
4. The designer is grounded on documents sampled from a large pretraining text corpus and keeps a memory of past environments, which keeps the generated tasks diverse instead of collapsing into repetitive patterns
5. At 30B scale, SPADE improved an average of +5.3 points across eight math/science/code/reasoning benchmarks over the strongest fixed-environment baseline, gained up to +13.9 on a tool-use benchmark, and the average gain grew from +5.2 at 4B to +8.1 at 30B as model size increased

Why it matters

Hand-curated or frozen synthetic problem sets eventually run dry once a model has learned everything in them, but SPADE shows a model can keep generating appropriately-difficult new challenges for itself as it improves. This is a concrete step toward AI systems that keep training themselves without needing constant human-supplied data.

Terms in this paper

• Self-play · a training method where a single model plays two interacting roles to improve itself through the interaction
• GRPO · a reinforcement learning method that compares a group of sampled responses against each other to update the policy
• MDP (Markov Decision Process) · a mathematical framework describing sequential decision problems with states, actions, rewards, and transitions
• Hint-based regret · a measure of how much a hint helps the agent, used to gauge whether a task sits at the right difficulty level
• Gym-style interface · a standard reinforcement learning environment format with reset() to start and step() to take an action and get the next state/reward

Figures we cannot republish

• Figure 2: SPADE generates an adaptive, multi-turn curriculum. Four environments the Environment Designer produces over one 30B-A3B run, from step 0 (early) to step 384 (late); each card shows the agent’s first observation, the generated Python environment, and the designer-written hint. Every environment is a complete MDP with a reset()/step() interface, and the tasks shift toward state-gated, multi-turn interaction as the Reasoning Agent improves. Unlike a fixed human-curated pool or a frozen synthetic generator, this curriculum keeps moving with the learner.
• Figure 4: The SPADE framework. Top: the Environment Designer conditions on the environment memory M and pretraining corpus C to emit an executable environment e and a privileged hint h. Bottom: the Reasoning Agent plays e with and without h; the return gap is the Environment Designer’s hint-based regret rD​(e) (Eq. 3) and task correctness is the Reasoning Agent reward. Both rewards update the shared policy πθ via GRPO.
• Figure 5: How a privileged hint changes Reasoning Agent play. Two positive-regret examples from the canonical 30B games run. Left: each environment’s task prompt and privileged hint, quoted verbatim (ellipses mark elided text; the standardized answer-format sentence is omitted from the hint). Right: one logged Reasoning Agent rollout per arm, condensed while preserving action order, feedback, and values; elided turns are marked and named. The two arms are independent plays with independently seeded resets, so board layouts and probe outcomes differ across arms. The dashed box on each environment card reports the two displayed rollout returns and their single-pair gap; the Environment Designer reward in Equation 3 is instead the difference of the arm means over all logged rollouts (0.00→1.00 for the fiber task, 0.30→0.65 for the audio task). An expanded task–hint set appears in Appendix J.1.1.
• Figure 6: Training on diverse synthetic games improves science reasoning, code generation, and procedural reasoning while competition math is preserved (games setting, Qwen3-30B-A3B-Instruct-2507). Top row: competition math (AIME 2025/2026 Avg@32), science reasoning (GPQA-Diamond accuracy), and code generation (LiveCodeBench-v6 Pass@1). Bottom row: procedural reasoning across four cognitive skills (Reasoning-Gym win rate at hard). Markers denote logged evaluation checkpoints; the dashed line marks the untrained base model.
• Figure 7: Full SPADE raises the learnable share of its environment budget to roughly a third by the end of training; component ablations decline or collapse. Share of each rollout’s 24 environments that is learnable, defined as Reasoning Agent win rate in [0.2,0.8] and weighted by the number of valid environments generated in each 16-step window. Matched 30B-A3B settings over a common 400-step budget; unfilled rollout capacity contributes zero by construction.
• Figure 9: Trained Environment Designer environments stop revealing the solution method in the prompt. Physics environments from one 30B-A3B run at steps 20, 192, and 384; the step-20 environment retains the scaffold’s default class name. The formula-reveal rate (percentage on each panel) falls from 25% to 5% over 473 environments. Rightmost (red): over steps 290–312 the no-corpus ablation emits the same RotatingMazeEnv task 41 consecutive times. The complete source of the step-384 environment, together with one further exemplar, appears in Appendix J.3.2.
• Figure 10: From front-loaded derivation to evidence-first interaction. Reasoning Agent episodes from one 30B-A3B run at steps 0, 200, and 300. At step 0 the agent derives in advance and cannot recover from format errors; by step 200 it tests short hypotheses and revises on evidence; by step 300 it probes first and derives once. Benchmark gains of late checkpoints (Table 1) confirm the model keeps its long-form derivation ability. Transcripts verbatim (math glyphs transliterated to ASCII; environment feedback abridged); token counts use the backbone’s tokenizer.
• Figure 11: Removing Environment Designer training and memory together drops self-play below base; removing either one alone has an above-base selected checkpoint but peaks early and can fall below base late. One curve per variant of Table 3 (Qwen3-30B-A3B-Instruct-2507, games setting). Top row: AIME 2025/2026 Avg@32, GPQA-Diamond accuracy, and LiveCodeBench-v6 Pass@1; bottom row: the four Reasoning-Gym categories; the dashed line marks the untrained base model.
• Figure 12: Rewarding the Environment Designer matters more than which reward is used, but regret still leads. Averaged trajectories over GPQA-Diamond, LiveCodeBench-v6, and the four Reasoning-Gym categories (Qwen3-30B-A3B-Instruct-2507, games setting); the dashed line marks the corresponding untrained base. Eight-benchmark checkpoint results are reported in Table 6.
• Figure 13: SPADE’s average gain over base grows with model size, from +5.2 at 4B to +8.1 at 30B-A3B, while matched-budget Fixed-env GRPO stays near +1.2. Left: average gain over each backbone’s own base across the eight benchmarks of Table 1 (large markers), the eight per-benchmark gains beside each mean, Fixed-env GRPO in gray. Right: Environment Designer hint-based regret over training (dark: EMA-smoothed; light: per-step). Only the 30B-A3B estimate stays positive; at 4B and 8B it dips below zero for long stretches, where the finite-sample estimate turns negative even though regret is non-negative at the optimum (Section 4). Both smaller backbones still gain over base (+5.2, +5.7), so the environments help even where the signal is noisy.
• Figure 14: Curriculum breadth accounts for most of the gain. Suite average (eight benchmarks of Table 1), six-skill vs. two-skill curriculum. Per-benchmark panels: Figure 15.
• Figure 15: The full 6-skill curriculum lifts held-out benchmarks more than the restricted 2-skill variant; curriculum breadth drives the gains. Qwen3-30B-A3B-Instruct-2507, games setting. Top row: AIME 2025/2026 Avg@32, GPQA-Diamond accuracy, and LiveCodeBench-v6 Pass@1. Bottom row: the four Reasoning-Gym categories; the dashed line marks the untrained base model. Discussed in Section 8.
• Figure 16: SPADE improves a second backbone family: all four Reasoning-Gym categories end above the untrained Nemotron-30B-A3B-BF16 base (RG-Cognition +9.6, RG-Algorithmic +9.3, RG-Math +2.6, RG-Logic +3.2). Reasoning-Gym win rate across checkpoints; the dashed line marks the untrained base model. Gains arrive after an initial dip early in training, consistent with the Environment Designer initially generating environments too hard for the Reasoning Agent, as observed on the Qwen backbones.
• Figure 17: Reward granularity increases over training. Left: mean distinct reward levels per environment, including strictly partial levels. Right: early-to-late change in distinct levels overall and by domain (canonical 30B games run).
• Figure 18: Per-skill learning (canonical SPADE-30B run). (a) Per-skill Reasoning Agent win rate over training (EMA over ∼26 logged points/skill). (b) Win-rate gain (last 25% minus first 25%): Logical Deduction and Spatial Reasoning improve most; Causal Inference, which starts high, declines slightly.
• Figure 19: Environment-diversity dynamics across the verified-matched runs. Windowed Vendi score (40-step windows, subsampled to exactly 24 environments; mean ± s.d. over 12 draws). The corpus-grounded variants hold the mixed-population reference level for their full run lengths; the no-corpus run starts near-collapsed, peaks at Vendi ∼5.2 near step 100, where its best evaluation checkpoint (111) also falls, and re-collapses as optimization continues. Environment counts differ across runs with run length and acceptance rate; the windowed score uses fixed 24-environment subsamples, so counts do not bias the curves.
• Figure 20: Task context makes hint utility legible. Four same-record, positive-regret task–hint pairs from the canonical 30B games run. The task summaries retain the goal, hidden information, and usable interaction while removing generic runtime scaffolding. Hint excerpts omit only the standardized answer-format sentence and runtime wrapper. In each header, the two values report the mean return without hint / with hint.

Latest papers

• SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the timearXiv:2608.197992026-08-21
• FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papersarXiv:2608.197582026-08-21
• PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad actionarXiv:2608.198612026-08-21
• EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise itarXiv:2608.198912026-08-21
• EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknessesarXiv:2608.198802026-08-21
• Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversightarXiv:2608.192162026-08-21
• Beyond Imitation: Filtering On-Policy Distillation by Reasoning ProgressA fix for AI models that get penalized by their teacher even when they're reasoning correctlyarXiv:2608.194082026-08-21
• PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappearsarXiv:2608.197462026-08-21

Latest from METAL MEDIA

• TUIDE debut with 'SUN KISS,' an Iceland-shot K-pop road movie
• NCT 127 drop BLINGY, their first full album in two years