Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox›
On-Policy Self-Distillation without Any Supervision
arXiv:2608.062962026-08-08
A language model can grade and re-teach its own math answers with no external answer key
Post-training large language models usually needs ground-truth solutions or a bigger teacher model to guide them, but this paper proposes u-OPSD, a method that lets the model generate multiple attempts at the same problem and use the majority-vote answer as a stand-in for a ground-truth solution. Only the attempts that disagree with the majority get corrected by distilling the model's own consensus-conditioned prediction into them, and this label-free approach matches or beats supervised methods that use real ground truth on math benchmarks. Code is publicly released.
METAL MEDIA explanatory visual
u-OPSD: How the model becomes its own teacher without an answer key
Evidence statusMeasured results reported
1. Sample multiple rolloutsThe model independently generates G=8 attempts at the same problem
2. Majority voteThe most common final answer becomes the pseudo ground truth; rollouts are split into ones that agree and ones that disagree with it, and prompts below the confidence threshold (tau=0.5) are skipped
3. Condition the teacherThe longest rollout that agrees with the majority vote is used in place of a real solution to form the teacher's next-token distribution
4. Distill into disagreeing rolloutsThe teacher's predicted distribution is transferred (via forward KL) into the disagreeing student rollouts, correcting the model exactly where it was confidently wrong
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.
What they did
Existing on-policy self-distillation (OPSD) methods let a model train on its own generations, but still need external help such as ground-truth solutions, environment feedback, or a larger guiding model.
u-OPSD samples G=8 rollouts per problem and, if the fraction of rollouts agreeing on an answer reaches a self-consistency threshold (tau=0.5), treats the majority-vote answer as a pseudo ground truth.
The longest rollout that agrees with the majority vote is used as a teacher reference; its next-token distribution is distilled into the disagreeing rollouts so the model corrects itself exactly where it was confidently wrong.
On five math benchmarks (AIME24, AIME25, HMMT25, MATH500, AMC23), u-OPSD improved over base Qwen3 models in non-thinking mode by 8.5 and 10.7 points at 4B and 8B, and beat supervised OPSD (which uses real ground truth) by 3.2 and 2.3 points on average.
In thinking mode, u-OPSD stayed on par with OPSD (0.9 points ahead at 4B, tied at 8B) and outperformed GRPO by 0.7 and 1.1 points.
Figure 1: Comparison between OPSD / SDFT with ground-truth solution or ICLs (left), SDPO with rich feedback from the environment (middle) and our u-OPSD without any supervision.
Table 1: Performance comparison on math reasoning benchmarks for Qwen3 models with non-thinking mode.
Method
AIME24
AIME25
HMMT25
MATH500
AMC23
Avg.
Qwen3-4B
Base
25.83
17.78
10.83
84.10
66.25
40.96
w/ GT.
+ SFT
26.67
19.72
13.06
84.85
69.38
42.73
+ GRPO
25.00
22.50
15.00
86.20
80.62
45.86
+ OPSD
32.22
20.83
16.39
85.75
76.25
46.29
[1.2pt/1.4pt] w/o GT.
+ TTRL
25.00
20.83
11.94
83.60
67.50
41.77
+ RENT
22.22
20.28
11.39
84.00
74.38
42.45
+ Intuitor
23.89
20.00
11.67
83.70
70.62
41.98
+ u-OPSD
37.50
27.78
14.44
86.50
81.25
49.49
Qwen3-8B
Base
27.50
23.33
13.61
84.05
69.38
43.57
w/ GT.
+ SFT
26.94
21.67
11.94
84.10
72.50
43.43
+ GRPO
30.56
21.94
13.06
87.85
73.75
45.43
+ OPSD
41.67
28.06
18.33
87.15
85.00
52.04
[1.2pt/1.4pt] w/o GT.
+ TTRL
27.22
21.11
13.06
84.45
71.88
43.54
+ RENT
28.33
21.67
10.83
84.00
70.00
42.97
+ Intuitor
26.11
22.50
11.67
84.20
73.12
43.52
+ u-OPSD
45.56
34.72
18.61
89.55
83.12
54.31
Figure 2: Overview of Unsupervised On-policy Self-Distillation (u-OPSD), which replaces ground-truth supervision in On-Policy Self-distillation with pseudo-labels generated from the model’s own majority-vote consensus.
Table 2: Thinking mode, per benchmark at step 150, under the protocol of Table 1. Shading and bold as in that table.
Method
AIME24
AIME25
HMMT25
MATH500
AMC23
Avg.
Qwen3-4B
Base
74.17
64.72
45.56
94.80
95.00
74.85
w/ GT.
+ SFT
73.06
69.17
41.67
95.40
94.38
74.73
+ GRPO
73.89
69.44
43.61
95.45
99.38
76.35
+ OPSD
75.28
68.06
43.06
95.20
99.38
76.20
[1.2pt/1.4pt] w/o GT.
+ TTRL
72.78
68.33
45.56
95.90
96.25
75.76
+ RENT
74.72
65.83
43.06
95.25
99.38
75.65
+ Intuitor
76.39
68.33
42.78
95.35
97.50
76.07
+ u-OPSD
76.39
68.06
46.94
95.75
98.12
77.05
Qwen3-8B
Base
75.56
66.67
45.00
96.35
96.88
76.09
w/ GT.
+ SFT
76.39
69.72
43.89
95.80
95.00
76.16
+ GRPO
76.94
69.17
47.78
95.70
95.00
76.92
+ OPSD
80.83
69.72
46.67
95.75
96.88
77.97
[1.2pt/1.4pt] w/o GT.
+ TTRL
77.22
68.61
46.94
95.75
96.25
76.95
+ RENT
77.50
70.28
45.83
95.95
96.25
77.16
+ Intuitor
76.94
70.28
44.17
96.20
95.62
76.64
+ u-OPSD
76.94
71.39
47.50
96.00
98.12
77.99
Figure 3: Training curves of the Qwen3-4B runs quoted in Tables 1 and 2, on AIME24, AIME25 and MATH500 over the first four checkpoints, in thinking (left three columns) and non-thinking mode (right three); the dashed line is the base model. Top: u-OPSD against the supervised arms. Bottom: u-OPSD against the label-free ones. The axis counts checkpoints: u-OPSD saves every 25 steps and GRPO every 50, so the GRPO points span its steps 50–200. The two modes separate at a glance—in non-thinking mode u-OPSD breaks away from every other arm, while in thinking mode all methods stay within about two points of base.
Table 3: Qwen3-30B-A3B-Instruct-2507, non-thinking, scored as pass@1 rather than average@n: generation i of every problem forms one single-sample run, and i=1,2,3 give three estimates, reported as mean ± population standard deviation. Each arm is shown at the checkpoint with the best five-benchmark mean under this metric. Because the metric differs from Tables 1 and 2, the two are not comparable and the numbers are kept apart. Bold marks the best value in each column.
Model
Method
AIME24
AIME25
HMMT25
MATH500
AMC23
Avg.
Qwen3-30B-A3B -Instruct-2507
Base
80.00±0.00
63.33±2.72
43.33±2.72
96.33±0.25
95.83±1.18
75.77±0.85
OPSD
78.89±3.14
61.11±3.14
47.78±1.57
97.20±0.43
96.67±2.36
76.33±1.58
u-OPSD
75.56±5.67
65.56±4.16
50.00±0.00
97.00±0.43
99.17±1.18
77.46±2.09
Qwen3-4B -Instruct-2507
Base
66.67±7.20
53.33±5.44
27.78±4.16
93.87±0.34
93.33±1.18
67.00±2.25
OPSD
62.22±5.09
52.22±5.09
31.11±5.09
94.93±0.64
95.00±0.00
67.10±1.13
u-OPSD
68.89±5.67
57.78±1.57
28.89±3.14
94.20±0.86
94.17±1.18
68.78±1.70
Table 4: Performance under combinations of the teacher-reference selection (rows) and the distillation-target selection (columns), on Qwen3-8B non-thinking, G=8, τ=0.5, k=1. Each cell is the five-benchmark average at the best checkpoint in 25–150, with the step given in parentheses. “label-only” strips the teacher’s reference down to the boxed pseudo-label.
Teacher ref. \ distill
longest
random
shortest
label-only
43.40 (125)
43.00 (100)
41.55 (25)
shortest (default)
57.10 (125)
54.37 (75)
55.23 (150)
random
57.96 (75)
56.93 (75)
55.77 (75)
longest
59.00 (75)
57.90 (100)
55.01 (125)
Table 5: Ablation of the disagreeing-rollout selection policy (matched decay schedule). Each cell shows step 150 / best checkpoint in 25–150. All variants use G=8, τ=0.5.
Variant
AIME24
AIME25
HMMT25
MATH500
AMC23
OPSD (supervised)
27.50 / 27.50
20.00 / 23.61
12.50 / 13.33
83.80 / 84.65
71.88 / 72.50
disagree-1 (default)
33.89 / 35.28
27.78 / 27.78
14.72 / 16.11
87.60 / 87.60
79.38 / 79.38
disagree-2
29.72 / 31.11
23.33 / 27.50
15.28 / 16.67
86.20 / 86.30
75.62 / 80.00
disagree-3
27.78 / 34.72
25.83 / 31.11
13.33 / 18.33
85.45 / 87.10
72.50 / 76.25
disagree-all (no cap)
29.44 / 30.83
22.50 / 25.83
10.00 / 13.33
85.25 / 86.00
73.75 / 74.38
longest-1
33.06 / 34.72
23.33 / 26.67
14.72 / 18.33
85.50 / 86.80
73.75 / 78.12
Table 6: Comparison of divergence computation strategy: Full vocabulary is logit distillation over every token (2); sampled token evaluates the two policies only at the token the student drew (26); top-k rows truncate the teacher to its k largest entries. We report on Qwen3-8B non-thinking at the best checkpoint.
Variant
AIME24
AIME25
HMMT25
MATH500
AMC23
Avg.
Student token
29.44
19.72
12.78
84.70
70.62
43.45
top-k
20
50.00
37.78
19.44
90.20
88.12
56.94
50
48.06
37.50
24.17
89.45
85.62
55.85
100
53.89
43.33
20.56
91.65
85.62
59.01
200
51.67
40.00
22.50
90.15
88.75
58.11
Full-vocabulary
53.89
37.50
20.28
89.90
85.62
57.10
Table 7: Divergence family Dβ under u-OPSD, on Qwen3-8B non-thinking, longest-1, G=8, τ=0.5. Each arm is shown at its best checkpoint in 25–150. Objective names follow 59, who report the same comparison under gold supervision.
Objective
AIME24
AIME25
HMMT25
MATH500
AMC23
Avg.
Base
27.50
23.33
13.61
84.05
69.38
43.57
Forward KL KL(πT∥πS), β=0 (default)
53.89
37.50
20.00
89.75
84.38
57.10
Reverse KL KL(πS∥πT), β=1
training diverges (not scored)
JSD (β=0.5)
29.44
21.94
12.22
83.70
69.38
43.34
Table 8: Reproduction of published OPSD results with the released code and hyperparameters (avg@12, temperature 1.0). “pub.” denotes the numbers published in the official OPSD repository; “ours” our rerun. Checkpoint columns are steps 50/75/100/150, matching the checkpoints published for this configuration.
Config
Bench
base
checkpoints
Qwen3-4B (non-thinking)
AIME24 pub.
23.1
20.3
27.5
31.1
32.8
AIME24 ours
22.2
23.1
27.2
32.2
31.9
AIME25 pub.
21.4
21.4
20.8
21.1
21.9
AIME25 ours
17.8
20.8
23.1
20.8
21.1
HMMT25 pub.
10.8
11.1
13.1
16.4
14.4
HMMT25 ours
12.2
10.6
12.8
16.4
11.7
Table 9: GRPO under matched and mismatched reasoning modes. Each row is the best of ten checkpoints (steps 50–500) by five-benchmark mean, scored at temperature 1.0 against the base model in the evaluation mode of that row. The two “thinking → non-thinking” rows reuse the checkpoints of the rows above them; only the evaluation prompt differs.
Train
Eval
Step
AIME24
AIME25
HMMT25
MATH500
AMC23
Avg.
Δ base
Qwen3-4B
thinking
thinking
300
73.89
69.44
43.61
95.45
99.38
76.35
+1.79
thinking
non-thinking
500
25.00
21.39
13.89
83.75
76.25
44.06
+2.62
non-thinking
non-thinking
200
25.00
22.50
15.00
86.20
80.62
45.86
+4.43
Qwen3-8B
thinking
thinking
250
76.94
69.17
47.78
95.70
95.00
76.92
+0.66
thinking
non-thinking
100
27.22
22.50
13.33
85.20
67.50
43.15
+1.20
non-thinking
non-thinking
500
30.56
21.94
13.06
87.85
73.75
45.43
+3.48
Table 10: Self-consistency threshold τ on Qwen3-8B non-thinking, longest-1, G=8. Each cell shows step 150 / best checkpoint in 25–150. τ is the fraction of valid rollouts that must agree for a prompt to receive a pseudo-label; τ=0 accepts every prompt.
τ
AIME24
AIME25
HMMT25
MATH500
AMC23
Avg.
0
53.33 / 57.50
41.39 / 46.67
22.50 / 27.22
90.70 / 92.15
88.12 / 89.38
59.21 / 61.83
0.3
50.83 / 53.33
34.44 / 41.67
20.28 / 23.06
89.85 / 90.65
85.00 / 88.12
56.08 / 58.59
0.5
43.06 / 53.89
35.83 / 37.50
20.28 / 20.28
88.00 / 89.90
82.50 / 85.62
53.93 / 57.10
0.7
32.78 / 37.50
23.61 / 26.67
13.06 / 14.44
85.95 / 87.70
74.38 / 76.25
45.96 / 48.18
0.9
32.22 / 32.22
20.83 / 22.50
13.61 / 13.61
84.70 / 84.85
70.62 / 72.50
44.40 / 44.40
Table 11: Number of rollouts per prompt G on Qwen3-8B non-thinking, longest-1, τ=0.5. Each cell shows step 150 / best checkpoint in 25–150.
G
AIME24
AIME25
HMMT25
MATH500
AMC23
Avg.
4
47.22 / 50.28
35.83 / 37.78
18.06 / 21.11
88.15 / 90.20
84.38 / 87.50
54.73 / 56.99
8
43.06 / 53.89
35.83 / 37.50
20.28 / 20.28
88.00 / 89.90
82.50 / 85.62
53.93 / 57.10
12
53.61 / 57.22
40.83 / 43.33
20.56 / 26.39
91.00 / 92.00
86.25 / 90.00
58.45 / 61.79
16
51.67 / 55.28
41.67 / 44.17
22.22 / 22.22
89.90 / 89.90
87.50 / 87.50
58.59 / 59.37
Table 12: How the teacher is updated, on Qwen3-8B non-thinking, longest-1, G=8, τ=0.5. Each cell shows step 150 / best checkpoint in 25–150. “fixed” freezes the teacher at the initial policy (the base model with the adapter disabled), which is OPSD’s own setting and ours everywhere else; the EMA rows let the teacher track the student at the given decay.
Teacher
AIME24
AIME25
HMMT25
MATH500
AMC23
Avg.
fixed
43.06 / 53.89
35.83 / 37.50
20.28 / 20.28
88.00 / 89.90
82.50 / 85.62
53.93 / 57.10
EMA 0.999
43.61 / 52.78
36.67 / 41.39
18.33 / 23.33
88.40 / 90.50
87.50 / 87.50
54.90 / 58.82
EMA 0.99
52.78 / 54.17
39.72 / 43.89
23.33 / 23.33
89.85 / 91.00
83.75 / 86.88
57.89 / 58.88
EMA 0.995
54.17 / 55.00
40.28 / 43.33
21.94 / 22.50
90.10 / 90.75
83.75 / 89.38
58.05 / 59.47
Table 13: Learning-rate schedule ablation: best AIME24 checkpoint (avg@12) per method under an effectively-constant LR (30-epoch horizon, ≈5×10−6 throughout) vs. the matched 150-step linear decay used throughout Tables 1 and 2. The constant-LR column is OPSD’s native configuration, so the released figure is directly comparable there; no released run exists under the decayed schedule.
Method
Constant LR
Linear decay
OPSD (supervised)
32.22
27.50
as released
32.8
–
disagree-1
37.50
35.28
disagree-2
32.78
31.11
disagree-3
33.61
34.72
disagree-all
33.61
30.83
longest-1
31.94
34.72
Findings
Averaged over five math benchmarks, u-OPSD improved over base Qwen3-4B/8B non-thinking models by 8.5 and 10.7 points respectively, and beat ground-truth-supervised OPSD by 3.2 and 2.3 points.
In thinking mode, u-OPSD was 0.9 points ahead of OPSD at 4B and tied at 8B, while beating GRPO by 0.7 and 1.1 points respectively.
On instruction-tuned models, u-OPSD raised Qwen3-30B-A3B-Instruct-2507 from 75.77 to 77.46 and Qwen3-4B-Instruct-2507 from 67.00 to 68.78, exceeding OPSD by 1.1 and 1.7 points.
Among 64 training prompts analyzed, 96.3% of rollouts yielded a parsable answer, 94.0% of prompts cleared the self-consistency threshold, and 86.7% of the resulting pseudo-labels matched the actual gold answer.
Stripping the teacher's reference down to just the boxed pseudo-label (rather than the full reasoning trace) cost 11.4 to 15.6 points; using symmetric Jensen-Shannon divergence instead of forward KL cost 13.8 points, dropping performance to roughly the untrained baseline.
Where it can be used
Building training pipelines for domains lacking ground-truth labels, where a model's own majority-vote consensus substitutes for gold answers.
Improving model accuracy on competition-style math or other domains where final answers can be automatically extracted and checked for exact match.
Using this label-free approach as a comparison baseline against ground-truth-based methods like OPSD or GRPO in future post-training research.
Limits and open work
Experiments cover only the Qwen3 model family (4B and 8B) and one domain, competition mathematics, where final answers can be cleanly extracted and canonicalized.
Gains are much larger in non-thinking mode than in thinking mode, since stronger thinking-mode base models already leave less room for improvement.
The pseudo-labels are bounded by whatever answer the base model most often produces; the paper measured 13.3% of pseudo-labels being wrong in-domain and did not test training under deliberately corrupted votes.
Variance was only partly addressed with 12 samples per checkpoint across five benchmarks; seed-replicated error bars for every training setup are still pending and planned for a revision.
GRPO's mode-mismatch experiment tested transferring thinking-trained checkpoints to non-thinking evaluation, but the reverse direction was not tested.
Why it matters
If self-generated consensus can substitute for gold answers, models could keep improving on tasks where labeled data is expensive, unreliable, or simply doesn't exist, such as open-ended or hard-to-verify domains. This points toward a path for continual self-improvement that doesn't depend on ever-larger supervised datasets or stronger external teacher models.
Terms in this paper
on-policy self-distillation (OPSD) · A training method where a model teaches itself using its own generated outputs rather than a separate teacher model
majority vote · Sampling many answers to the same problem and treating the most common answer as likely correct
self-consistency threshold (tau) · The minimum fraction of sampled answers that must agree before that consensus answer is trusted for training
GRPO · A reinforcement learning method that updates the policy using rewards computed relative to a group of sampled rollouts, checked against a ground-truth answer
forward KL divergence · A way of measuring the difference between a teacher's and student's predicted probability distributions, used to pull the student toward the teacher
Original abstract (English)
On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine "self"-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency. We propose unsupervised on-policy self-distillation (U-OPSD). U-OPSD first samples multiple rollouts and constructs a pseudo solution by majority vote under a self-consistency threshold. It then conditions the model's distribution on the pseudo-solution and distills itself on the disagreeing completions, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, U-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT) such as OPSD and GRPO. On five mathematical reasoning benchmarks, i.e., AIME24, AIME25, HMMT25, MATH500, and AMC23, U-OPSD improves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at 4B and 8B scales, and outperforms OPSD by 3.2% and 2.3% on average, respectively. In thinking mode, U-OPSD stays on par with OPSD, ahead by 0.9% at 4B and level at 8B and surpassing GRPO by 0.7% and 1.1%, respectively. Code is available at [https://github.com/williamium3000/u-opsd](https://github.com/williamium3000/u-opsd).
