AI가 스스로 문제를 내고 스스로 풀면서 실력을 키우는 훈련법

무엇을 했나

1. 기존에는 사람이 미리 만든 문제 세트나 고정된 자동 생성기를 썼는데, 모델이 그 문제들을 다 풀고 나면 더 이상 성장할 방법이 없었다
2. SPADE는 같은 모델이 '환경 설계자' 역할일 때는 상태, 보상, 검증 코드를 갖춘 완전한 게임(파이썬 프로그램)을 만들고, '추론 에이전트' 역할일 때는 그 게임을 플레이한다
3. 설계자에게는 정답 힌트를 알려줬을 때와 안 알려줬을 때 에이전트 성적 차이(힌트 기반 후회 점수)를 보상으로 주어, 너무 쉽지도 너무 어렵지도 않은 딱 배울 만한 난이도의 문제를 만들도록 유도한다
4. 사전학습에 쓰인 방대한 문서를 참고 자료로 주고, 과거에 만든 환경들을 기억하게 해서 문제가 계속 새롭고 다양하게 나오도록 했다
5. 30B 모델 기준 수학, 과학, 코드, 추론 8개 벤치마크 평균 5.3점 상승, 도구 사용 벤치마크에서도 최대 13.9점 상승을 기록했고, 모델 크기가 커질수록(4B→30B) 성능 향상 폭도 커졌다(5.2→8.1)

왜 중요한가

사람이 일일이 문제를 만들거나 고정된 자동 생성기를 쓰는 방식은 언젠가 한계에 부딪히는데, SPADE는 모델이 스스로 문제 난이도를 조절하며 계속 새로운 훈련거리를 만들어내는 구조를 보여준다. 이는 AI가 인간 개입 없이 지속적으로 스스로 발전하는 방법에 한 걸음 다가간 사례다.

이 논문의 용어

• 셀프플레이(Self-Play) · 하나의 모델이 서로 다른 두 역할을 맡아 상호작용하며 스스로를 훈련시키는 방식
• GRPO · 여러 응답을 그룹으로 묶어 상대적으로 비교해 정책을 업데이트하는 강화학습 기법
• MDP(마르코프 결정 과정) · 상태, 행동, 보상, 전이 확률로 이루어진 순차적 의사결정 문제의 수학적 표현
• 힌트 기반 후회(hint-based regret) · 힌트를 줬을 때와 안 줬을 때 성적 차이로 문제의 학습 난이도를 측정하는 지표
• Gym 스타일 인터페이스 · reset()으로 초기 상태를 받고 step()으로 행동 후 다음 상태와 보상을 받는 표준 강화학습 환경 형식

본문에 싣지 못한 그림

• Figure 2: SPADE generates an adaptive, multi-turn curriculum. Four environments the Environment Designer produces over one 30B-A3B run, from step 0 (early) to step 384 (late); each card shows the agent’s first observation, the generated Python environment, and the designer-written hint. Every environment is a complete MDP with a reset()/step() interface, and the tasks shift toward state-gated, multi-turn interaction as the Reasoning Agent improves. Unlike a fixed human-curated pool or a frozen synthetic generator, this curriculum keeps moving with the learner.
• Figure 4: The SPADE framework. Top: the Environment Designer conditions on the environment memory M and pretraining corpus C to emit an executable environment e and a privileged hint h. Bottom: the Reasoning Agent plays e with and without h; the return gap is the Environment Designer’s hint-based regret rD​(e) (Eq. 3) and task correctness is the Reasoning Agent reward. Both rewards update the shared policy πθ via GRPO.
• Figure 5: How a privileged hint changes Reasoning Agent play. Two positive-regret examples from the canonical 30B games run. Left: each environment’s task prompt and privileged hint, quoted verbatim (ellipses mark elided text; the standardized answer-format sentence is omitted from the hint). Right: one logged Reasoning Agent rollout per arm, condensed while preserving action order, feedback, and values; elided turns are marked and named. The two arms are independent plays with independently seeded resets, so board layouts and probe outcomes differ across arms. The dashed box on each environment card reports the two displayed rollout returns and their single-pair gap; the Environment Designer reward in Equation 3 is instead the difference of the arm means over all logged rollouts (0.00→1.00 for the fiber task, 0.30→0.65 for the audio task). An expanded task–hint set appears in Appendix J.1.1.
• Figure 6: Training on diverse synthetic games improves science reasoning, code generation, and procedural reasoning while competition math is preserved (games setting, Qwen3-30B-A3B-Instruct-2507). Top row: competition math (AIME 2025/2026 Avg@32), science reasoning (GPQA-Diamond accuracy), and code generation (LiveCodeBench-v6 Pass@1). Bottom row: procedural reasoning across four cognitive skills (Reasoning-Gym win rate at hard). Markers denote logged evaluation checkpoints; the dashed line marks the untrained base model.
• Figure 7: Full SPADE raises the learnable share of its environment budget to roughly a third by the end of training; component ablations decline or collapse. Share of each rollout’s 24 environments that is learnable, defined as Reasoning Agent win rate in [0.2,0.8] and weighted by the number of valid environments generated in each 16-step window. Matched 30B-A3B settings over a common 400-step budget; unfilled rollout capacity contributes zero by construction.
• Figure 9: Trained Environment Designer environments stop revealing the solution method in the prompt. Physics environments from one 30B-A3B run at steps 20, 192, and 384; the step-20 environment retains the scaffold’s default class name. The formula-reveal rate (percentage on each panel) falls from 25% to 5% over 473 environments. Rightmost (red): over steps 290–312 the no-corpus ablation emits the same RotatingMazeEnv task 41 consecutive times. The complete source of the step-384 environment, together with one further exemplar, appears in Appendix J.3.2.
• Figure 10: From front-loaded derivation to evidence-first interaction. Reasoning Agent episodes from one 30B-A3B run at steps 0, 200, and 300. At step 0 the agent derives in advance and cannot recover from format errors; by step 200 it tests short hypotheses and revises on evidence; by step 300 it probes first and derives once. Benchmark gains of late checkpoints (Table 1) confirm the model keeps its long-form derivation ability. Transcripts verbatim (math glyphs transliterated to ASCII; environment feedback abridged); token counts use the backbone’s tokenizer.
• Figure 11: Removing Environment Designer training and memory together drops self-play below base; removing either one alone has an above-base selected checkpoint but peaks early and can fall below base late. One curve per variant of Table 3 (Qwen3-30B-A3B-Instruct-2507, games setting). Top row: AIME 2025/2026 Avg@32, GPQA-Diamond accuracy, and LiveCodeBench-v6 Pass@1; bottom row: the four Reasoning-Gym categories; the dashed line marks the untrained base model.
• Figure 12: Rewarding the Environment Designer matters more than which reward is used, but regret still leads. Averaged trajectories over GPQA-Diamond, LiveCodeBench-v6, and the four Reasoning-Gym categories (Qwen3-30B-A3B-Instruct-2507, games setting); the dashed line marks the corresponding untrained base. Eight-benchmark checkpoint results are reported in Table 6.
• Figure 13: SPADE’s average gain over base grows with model size, from +5.2 at 4B to +8.1 at 30B-A3B, while matched-budget Fixed-env GRPO stays near +1.2. Left: average gain over each backbone’s own base across the eight benchmarks of Table 1 (large markers), the eight per-benchmark gains beside each mean, Fixed-env GRPO in gray. Right: Environment Designer hint-based regret over training (dark: EMA-smoothed; light: per-step). Only the 30B-A3B estimate stays positive; at 4B and 8B it dips below zero for long stretches, where the finite-sample estimate turns negative even though regret is non-negative at the optimum (Section 4). Both smaller backbones still gain over base (+5.2, +5.7), so the environments help even where the signal is noisy.
• Figure 14: Curriculum breadth accounts for most of the gain. Suite average (eight benchmarks of Table 1), six-skill vs. two-skill curriculum. Per-benchmark panels: Figure 15.
• Figure 15: The full 6-skill curriculum lifts held-out benchmarks more than the restricted 2-skill variant; curriculum breadth drives the gains. Qwen3-30B-A3B-Instruct-2507, games setting. Top row: AIME 2025/2026 Avg@32, GPQA-Diamond accuracy, and LiveCodeBench-v6 Pass@1. Bottom row: the four Reasoning-Gym categories; the dashed line marks the untrained base model. Discussed in Section 8.
• Figure 16: SPADE improves a second backbone family: all four Reasoning-Gym categories end above the untrained Nemotron-30B-A3B-BF16 base (RG-Cognition +9.6, RG-Algorithmic +9.3, RG-Math +2.6, RG-Logic +3.2). Reasoning-Gym win rate across checkpoints; the dashed line marks the untrained base model. Gains arrive after an initial dip early in training, consistent with the Environment Designer initially generating environments too hard for the Reasoning Agent, as observed on the Qwen backbones.
• Figure 17: Reward granularity increases over training. Left: mean distinct reward levels per environment, including strictly partial levels. Right: early-to-late change in distinct levels overall and by domain (canonical 30B games run).
• Figure 18: Per-skill learning (canonical SPADE-30B run). (a) Per-skill Reasoning Agent win rate over training (EMA over ∼26 logged points/skill). (b) Win-rate gain (last 25% minus first 25%): Logical Deduction and Spatial Reasoning improve most; Causal Inference, which starts high, declines slightly.
• Figure 19: Environment-diversity dynamics across the verified-matched runs. Windowed Vendi score (40-step windows, subsampled to exactly 24 environments; mean ± s.d. over 12 draws). The corpus-grounded variants hold the mixed-population reference level for their full run lengths; the no-corpus run starts near-collapsed, peaks at Vendi ∼5.2 near step 100, where its best evaluation checkpoint (111) also falls, and re-collapses as optimization continues. Environment counts differ across runs with run length and acceptance rate; the windowed score uses fixed 24-environment subsamples, so counts do not bias the curves.
• Figure 20: Task context makes hint utility legible. Four same-record, positive-regret task–hint pairs from the canonical 30B games run. The task summaries retain the goal, hidden information, and usable interaction while removing generic runtime scaffolding. Hint excerpts omit only the standardized answer-format sentence and runtime wrapper. In each header, the two values report the mean return without hint / with hint.

최신 논문

• AI 코딩 에이전트에게 과학 소프트웨어 수리를 시켜보니, 절반도 제대로 못 고쳤다AI 코딩 에이전트에게 과학 소프트웨어 수리를 시켜보니, 절반도 제대로 못 고쳤다arXiv:2608.197992026-08-21
• 논문 속 시연이 아니라 실제 서비스에 넣을 수 있는 희소 어텐션 만들기논문 속 시연이 아니라 실제 서비스에 넣을 수 있는 희소 어텐션 만들기arXiv:2608.197582026-08-21
• 고객상담 AI 상담원이 규정을 '한 번의 행동'이 아니라 '전체 절차'로 지키게 만드는 방법고객상담 AI 상담원이 규정을 '한 번의 행동'이 아니라 '전체 절차'로 지키게 만드는 방법arXiv:2608.198612026-08-21
• 로봇 팔에게 사람의 시연 없이 새 일 시키기, 말 잘하는 AI가 대신 가르친다로봇 팔에게 사람의 시연 없이 새 일 시키기, 말 잘하는 AI가 대신 가르친다arXiv:2608.198912026-08-21
• 에이전트 학습용 환경을 새로 만드는 대신, 기존 환경에 '패치 부품'을 씌워 그 에이전트의 약점에 맞게 바꾸는 방법에이전트 학습용 환경을 새로 만드는 대신, 기존 환경에 '패치 부품'을 씌워 그 에이전트의 약점에 맞게 바꾸는 방법arXiv:2608.198802026-08-21
• AI 모델을 '소유'하지 못한 조직은 안전 통제도 절반밖에 못 한다AI 모델을 '소유'하지 못한 조직은 안전 통제도 절반밖에 못 한다arXiv:2608.192162026-08-21
• AI가 선생님 모델을 따라 배우다가, 정답에 다가가는 '좋은 생각'까지 억누르는 문제를 잡아낸다AI가 선생님 모델을 따라 배우다가, 정답에 다가가는 '좋은 생각'까지 억누르는 문제를 잡아낸다arXiv:2608.194082026-08-21
• AI가 특정 사람 말투를 흉내내도록 시켜봤더니, 결국 AI 자신의 말투에서 못 벗어난다AI가 특정 사람 말투를 흉내내도록 시켜봤더니, 결국 AI 자신의 말투에서 못 벗어난다arXiv:2608.197462026-08-21

METAL MEDIA 최신 기사

• TUIDE, 아이슬란드에서 촬영한 K팝 로드무비 'SUN KISS'로 데뷔
• NCT 127, 2년 만의 정규앨범 'BLINGY' 발매