K-文化的一切——从回归到 K-美妆,发送到您的邮箱订阅邮件

METAL MEDIA

Governance Records as Supervision: Verifier-Selected Self-Training for Structured Workflow Repair

arXiv:2608.183242026-08-20

让外部验证程序来挑选AI自己生成的计划,就能把便宜模型训练得更可靠

研究者测试了一种方法:让AI模型自己多次尝试生成方案,再由一个独立的外部验证程序挑出真正正确的那些,用这些结果来训练同一模型更便宜的执行模式,过程中不提供标准答案,也不使用更强的老师模型。在名为PlanBench的积木重新规划测试中,仅用验证器认可的24个方案训练后,80个全新测试案例的通过数从1个升到57个。

METAL MEDIA 解读图

让外部验证程序来挑选AI自己生成的计划,就能把便宜模型训练得更可靠

  1. 01在PlanBench这个可由机器自动判断对错的积木堆叠重新规划任务中,Qwen3-14B用较慢的'思考'模式多次尝试生成方案,只保留独立验证工具VAL判定通过的方案作为训练数据
  2. 02用这24个验证通过的方案训练同一模型更便宜、更快的'非思考'执行模式,过程中从未直接给出正确答案,也没有使用更强的教师模型
  3. 03在80个全新测试案例中,通过数从训练前的1个提升到训练后的57个,其中56个案例得到改善、0个案例变差,响应速度仅为思考模式的约1/56
  4. 04一项对照实验固定了其他所有条件,只改变'谁来挑选训练目标':验证器挑选的目标比模型自己挑选的目标在160个新案例中多通过33个,说明独立验证是关键因素,而不仅仅是数据量的问题
  5. 05该方法在使用更强的推理模型Phi-4-reasoning作为教师时同样有效,使基础版Phi-4的通过数从2个提升到51个(满分80)
这是 METAL MEDIA 制作的解读图,并非论文作者提供的原图。

他们做了什么

  1. 在PlanBench这个可由机器自动判断对错的积木堆叠重新规划任务中,Qwen3-14B用较慢的'思考'模式多次尝试生成方案,只保留独立验证工具VAL判定通过的方案作为训练数据
  2. 用这24个验证通过的方案训练同一模型更便宜、更快的'非思考'执行模式,过程中从未直接给出正确答案,也没有使用更强的教师模型
  3. 在80个全新测试案例中,通过数从训练前的1个提升到训练后的57个,其中56个案例得到改善、0个案例变差,响应速度仅为思考模式的约1/56
  4. 一项对照实验固定了其他所有条件,只改变'谁来挑选训练目标':验证器挑选的目标比模型自己挑选的目标在160个新案例中多通过33个,说明独立验证是关键因素,而不仅仅是数据量的问题
  5. 该方法在使用更强的推理模型Phi-4-reasoning作为教师时同样有效,使基础版Phi-4的通过数从2个提升到51个(满分80)
Figure 1: Fresh external outcomes on the shared 80-case PlanBench evaluation. The Qwen path is teacher-free same-checkpoint self-distillation. The Phi path uses a separate stronger reasoning teacher. Exact counts are printed above the bars.
Figure 1: Fresh external outcomes on the shared 80-case PlanBench evaluation. The Qwen path is teacher-free same-checkpoint self-distillation. The Phi path uses a separate stronger reasoning teacher. Exact counts are printed above the bars.
Table 1: The two external paths share the governance-selected pipeline but not the same causal classification.
PathTarget producerTraining recipientTarget provenanceFrozen reading
QwenQwen3-14B thinkingSame checkpoint, non-thinkingNo oracle; no stronger teacherSemantic only
PhiPhi-4-reasoningSeparate base Phi-4No oracle; stronger teacherFull-band distillation
Figure 2: Matched target-selection ablation on 160 fresh PlanBench cases. The left panel shows downstream VAL acceptance for base and three 24-target training arms. The right panel shows that the VAL advantage over blinded model self-selection is positive in both block-count strata.
Figure 2: Matched target-selection ablation on 160 fresh PlanBench cases. The left panel shows downstream VAL acceptance for base and three 24-target training arms. The right panel shows that the VAL advantage over blinded model self-selection is positive in both block-count strata.
Table 2: Fresh synthetic one-response repair after training only on model- generated, verifier-selected targets. The two Generator A rows use different case-disjoint evaluation partitions. The later Generator A step does not clear its frozen materiality gate.
Construction and partitionBaseFirst adapterSecond adapterLater adapter
Generator A, initial10/4821/48n/an/a
Generator A, later9/4821/4833/4835/48
Generator B, Qwen12/8044/8062/80n/a
Generator C, Qwen57/128128/128126/128n/a
Generator B, Phi self-training10/8077/80n/an/a
Figure 3: Verified performance versus mean request latency on the external evaluation. Arrows connect the expensive target-producing regime to the cheap trained specialist. The source-versus-specialist semantic comparisons are descriptive compression diagnostics, not preregistered superiority tests.
Figure 3: Verified performance versus mean request latency on the external evaluation. Arrows connect the expensive target-producing regime to the cheap trained specialist. The source-versus-specialist semantic comparisons are descriptive compression diagnostics, not preregistered superiority tests.
Table 3: Formal Qwen target acquisition on the fresh harvest partition.
Harvest measureObservedFrozen gate
Distinct VAL-admitted targets24/32at least 16
First-attempt successes13/32reported
Successes within eight attempts24/32reported
Beyond-first-attempt headroom11at least 8
Completed HTTP requests118/118at least 98%
Completion tokens419,201at most 2,500,000
A100 time0.649 hat most 6 h
Figure 4: Qwen self-sourced one-response learning across three synthetic constructions. Generator C’s high base rate and first-round saturation limit its evidence about iteration.
Figure 4: Qwen self-sourced one-response learning across three synthetic constructions. Generator C’s high base rate and first-round saturation limit its evidence about iteration.
Table 4: Fresh Qwen evaluation on the external 80-case partition. Token counts are totals across each arm.
Qwen armVAL acceptedSchema-validMean latencyCompletion tokens
Base, non-thinking1/8071/800.624 s1,729
Base, thinking30/8077/8065.964 s254,960
Self-24, non-thinking57/8080/801.186 s2,965
Figure 5: First-attempt and 32-attempt success on matched pools. Panels use different case sets and are not pooled. Training expands fixed-budget reachability through Default30 on Generator A and Round 1 on Generator B; Generator C saturates the frozen frontier after Round 1.
Figure 5: First-attempt and 32-attempt success on matched pools. Panels use different case sets and are not pooled. Training expands fixed-budget reachability through Default30 on Generator A and Round 1 on Generator B; Generator C saturates the frozen frontier after Round 1.
Table 5: Matched target sets and training outcomes. Only target selection changes across arms.
SelectorVAL-valid targetsTrain timeFinal loss
VAL24/2464.1 s0.3545
Qwen self-selection20/2464.2 s0.3905
Schema-only frozen order15/2464.0 s0.5077
Figure 6: Zero-shot transfer from Generator A adapters to Generator B. Default30 passes the aggregate gate but crosses the frozen family-regression guard. Round3 is a predeclared secondary trajectory arm and cannot replace the primary treatment after results are opened.
Figure 6: Zero-shot transfer from Generator A adapters to Generator B. Default30 passes the aggregate gate but crosses the frozen family-regression guard. Round3 is a predeclared secondary trajectory arm and cannot replace the primary treatment after results are opened.
Table 6: Fresh outcomes for the matched target-selection ablation.
Evaluation armVAL acceptedSchema-validMean latency
Base, non-thinking1/160129/1601.395 s
Schema-selected adapter55/160152/1601.410 s
Model-self-selected adapter69/160159/1601.385 s
VAL-selected adapter102/160160/1601.202 s
Table 7: Fresh Phi evaluation on the same external 80-case partition. Token counts are totals across each arm.
Phi armVAL acceptedSchema-validMean latencyCompletion tokens
Base Phi-42/8035/801.027 s3,362
Phi-4-reasoning30/8041/80150.283 s544,408
Distilled Phi-451/8080/800.981 s2,531
Table 8: Synthetic evidence map. All evaluation partitions are case-disjoint from their training targets. Generator separation is by implementation, not by independent authorship.
StageModelTraining-target sourceFresh evaluationEvidentiary role
Oracle precursorPhi-4, Qwen3-14BProcedural schedule, verifier-confirmed144 rows per modelTeachability and signal attribution
Generator AQwen3-14BSame-model accepted samples48-row one-response and 32-row search poolsBounded loop, cumulative learning, stopping
Generator BQwen3-14BSame-model accepted samples80-row one-response and 32-row search poolsNative construction replication and transfer boundary
Generator CQwen3-14BSame-model accepted samples128-row one-response and 32-row search poolsConstruction-separated first-round replication
Generator BPhi-4Same-model accepted samples80-row one-response and 32-row search poolsCross-model method replication
Table 9: Qwen training and inference feedback cross. The primary causal comparison fixes localized inference and changes only the training trace.
Training stateInference feedbackPassTransportCompound
BaseLocalized31/144128/1445/48
Localized-trainedLocalized139/144144/14445/48
BaseVerdict-only35/144135/1449/48
Localized-trainedVerdict-only115/144144/14433/48
Verdict-only-trainedLocalized133/144144/14441/48
Verdict-only-trainedVerdict-only136/144144/14444/48
Table 10: Target-hidden search attribution. “Later” counts targets first accepted after at least one verifier rejection.
Search conditionAcceptedPass@1LaterCallsTransport
Fixed generic12/328/324678678/678
Fixed localized16/3210/326591590/591
Adaptive localized16/3211/325539539/539
Table 11: Fresh one-response transitions from verifier-selected self-training. The Generator A rows use separate case-disjoint partitions. Generator C Round 1 reaches 128/128; its final Round 2 arm reaches 126/128 and does not improve on the saturated first-round measurement.
Construction and modelEarlier armLater armPaired reading
A, Qwen initialBase 10/48Self-16 21/4814 gains, 3 losses, p=0.012726
A, Qwen Round 2Self-16 25/48Default30 35/48Net +10, p=0.021271
A, Qwen Round 3Default30 33/48Round3 35/483 gains, 1 loss, p=0.625; stop
B, Qwen Round 1Base 12/80B-Round1 44/80Net +32, p=4.07×10−9
B, Qwen Round 2B-Round1 44/80B-Round2 62/80Net +18, p=4.01×10−5
C, Qwen finalBase 57/128C-Round2 126/12869 gains, 0 losses, p=3.388×10−21
B, Phi roundBase 10/80Self-sourced LoRA 77/8067 gains, 0 losses, p=1.3553×10−20

为什么重要

这为企业提供了一条实用路径:只需运行一次昂贵但偶尔才成功的AI推理来收集经验证的样本,之后就能部署一个便宜、可靠的专用模型反复使用,适用于计划编排、合规检查等可自动验证的任务。但该结果仅限于对错可由机器判定的任务、单一测试领域和特定难度范围,不能证明模型具备通用规划能力或适用于所有企业场景。

本文术语

  • 验证器(Verifier) · 用机械规则检查模型输出是否符合要求的程序
  • VAL · 一个独立于本研究开发、专门检查PlanBench计划是否有效的外部工具
  • LoRA(低秩适配) · 一种不重新训练整个模型、只训练少量附加模块即可改变模型表现的高效训练方法
  • 自蒸馏(self-distillation) · 用同一模型某种模式产生的结果,去训练该模型另一种更便宜模式的方法
  • PlanBench · 用于测试AI模型规划能力的一个外部标准基准测试

论文原文摘要(英文)

Machine-verifiable workflows produce governance records linking a task contract, model attempt, verifier decision, accepted output, and target origin. We test whether these records can supervise bounded models, consolidating occasional or expensive capability into reliable one-shot execution. On fresh, structure-disjoint PlanBench replanning cases, Qwen3-14B thinking generated 24 plans admitted by the independently authored VAL verifier. Those plans trained the same checkpoint for non-thinking execution, without oracle targets or a stronger teacher. On 80 unopened cases, VAL-accepted plans increased from 1 to 57, with 56 paired gains and zero regressions; thinking reached 30. The adapter was schema-valid on all cases and used approximately 1/56 of thinking's mean latency. The separate paired interface-cure gate did not pass. A matched ablation fixed the source cases, 52-candidate pool, 24-target count, model, recipe, and seed while changing target selection. On 160 new cases, base, schema-selected, model-self-selected, and VAL-selected execution reached 1, 55, 69, and 102 accepted plans. VAL exceeded self-selection by paired net +33 (p=0.0000019647), with gains in both difficulty strata. Independent semantic selection is therefore load-bearing relative to matched alternatives within this band. A complementary Phi stronger-teacher arm raised base Phi-4 from 2 to 51 accepted plans and from 35 to 80 schema-valid outputs. Earlier synthetic experiments establish teachability, cumulative learning, construction robustness, and stopping boundaries. The results support verifier-selected supervision for bounded, machine-checkable capabilities, not arbitrary planning, enterprise validity, or unrestricted self-improvement.

作者 · Jesus Salas

在 arXiv 阅读

最新论文

全部论文 →

METAL MEDIA 最新报道

图片来源: Jesus Salas et al., arXiv:2608.18324, CC BY 4.0