그리스어로 생각하는 AI 모델을 만들어보니, 정확도 점수는 거의 의미가 없었다

arXiv:2608.177442026-08-18

Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See

연구자는 Alibaba, OpenAI, NVIDIA가 만든 세 개의 대형 AI 모델을 그리스어로 추론하도록 미세조정했다. 그런데 정확도 점수만 보면 별 차이가 없었고, 심지어 무작위 시드값만 바꿔도 점수가 7.7점이나 흔들려서 그동안 측정한 어떤 데이터·훈련법 효과보다 컸다. 진짜 변화는 정확도로는 안 보이는 곳, 즉 모델이 어떤 언어로 생각하는지, 얼마나 효율적으로 생각하는지에 있었다.

METAL MEDIA 해설 도표

그리스어로 생각하는 AI 모델을 만들어보니, 정확도 점수는 거의 의미가 없었다

01원래 모델들은 그리스어 질문을 받아도 속으로는 전혀 그리스어로 생각하지 않았다(1000개 중 0개). 답은 맞히지만 사용자가 읽거나 검증할 수 없는 형태로 사고 과정이 진행된 셈이다.
02지도미세조정(SFT, 정답이 붙은 예시로 모델을 추가 학습시키는 방법)을 거치자 네 개 모델 모두 약 98%의 문제에서 질문 언어(그리스어)로 사고하게 되었고, 문법 품질도 좋아졌으며 다른 능력은 거의 잃지 않았다.
03SFT가 못 고치는 결함도 있었다: 답의 4분의 1은 요청된 형식을 건너뛰었고, 답이 사고 과정 칸으로 새어나갔으며, '영어로 생각해'라는 명시적 지시도 절반 이하로만 지켜졌다.
04검증 가능한 보상을 이용한 강화학습(RLVR)을 훈련 전에 미리 설계해 적용하자 형식 무시 문제(24%→2.5%)와 답 누출 문제(3.5%→0%)는 확실히 고쳐졌고, 영어 지시 순응도도 개선됐다. 다만 그리스어로 생각하는 습관은 정확도만 노리는 학습으로는 바뀌지 않았다.
05연구팀은 다섯 개의 모델 체크포인트를 공개했고, 이 측정·검증 방법은 그리스어뿐 아니라 다른 자원 부족 언어에도 적용 가능하다.

METAL MEDIA이 원문을 바탕으로 재구성한 해설 도표이며, 논문 저자의 원문 figure가 아닙니다.

무엇을 했나

원래 모델들은 그리스어 질문을 받아도 속으로는 전혀 그리스어로 생각하지 않았다(1000개 중 0개). 답은 맞히지만 사용자가 읽거나 검증할 수 없는 형태로 사고 과정이 진행된 셈이다.
지도미세조정(SFT, 정답이 붙은 예시로 모델을 추가 학습시키는 방법)을 거치자 네 개 모델 모두 약 98%의 문제에서 질문 언어(그리스어)로 사고하게 되었고, 문법 품질도 좋아졌으며 다른 능력은 거의 잃지 않았다.
SFT가 못 고치는 결함도 있었다: 답의 4분의 1은 요청된 형식을 건너뛰었고, 답이 사고 과정 칸으로 새어나갔으며, '영어로 생각해'라는 명시적 지시도 절반 이하로만 지켜졌다.
검증 가능한 보상을 이용한 강화학습(RLVR)을 훈련 전에 미리 설계해 적용하자 형식 무시 문제(24%→2.5%)와 답 누출 문제(3.5%→0%)는 확실히 고쳐졌고, 영어 지시 순응도도 개선됐다. 다만 그리스어로 생각하는 습관은 정확도만 노리는 학습으로는 바뀌지 않았다.
연구팀은 다섯 개의 모델 체크포인트를 공개했고, 이 측정·검증 방법은 그리스어뿐 아니라 다른 자원 부족 언어에도 적용 가능하다.

Figure 1: fallback% (the rate at which a model never emits the requested answer line) by recipe and by language-matched checkpoint. It separates “cannot reason” from “will not answer in the requested form”: every recipe failure in the paper is visible here before it is visible anywhere else. The Qwen recipes are the side experiment of §7; the Gpt-OSS language-matched arm’s pre-repair Greek-lane rate (70%) is the family defect at its largest (its traces are present and on-language; the model simply does not close with the line the scorer is told to read), and the released checkpoint’s format-repair dose cuts it to 26% (§11), still the highest of the releases. The dashed reference is a base model on the English control (3.4%), showing the failure is not inherent to the benchmark.

Table 1: Total and routed-expert parameters counted from the released tensors; active-per-token is non-routed + k/E of routed. Gpt-OSS ships MXFP4-packed experts, so its figures are vendor-reported. NemotronH (Nano) and Nemotron-3.5-Lightning are Mamba/MoE hybrids (Mamba, MoE and attention layers of 52; the state-space line of Gu and Dao 2023, Dao and Gu 2024 and the vendor’s Nemotron-H report [21]); the other two are MoE transformers. The right-most column names the fine-tuned Sophea reasoning release each base produces; all four SFT releases are language-matched, and a fifth release, Sophea-Qwen3.6-v1.1, is the RLVR refinement of the Qwen row (§15.1). Throughout, base refers to the vendor checkpoint and fine-tuned to its Sophea reasoning model. Lightning is the next-generation release in the Nemotron line; Nano and Lightning share the 31.6B / 6-of-128 routing profile and are reported together as the Nemotron family (Nano is the same-generation member of that family).
lab	base model	total	active	routing	fine-tuned release
Alibaba	Qwen3.6-35B-A3B	36.0B	3.97B	8 of 256	Sophea-Qwen3.6-v1
OpenAI	Gpt-OSS-20B	20.9B	3.60B	4 of 32	Sophea-OSS-v1
NVIDIA	NemotronH-30B-A3B	31.6B	3.58B	6 of 128	Sophea-Nemo-3-Nano-v1
NVIDIA	Nemotron-3.5-Lightning-30B-A3B	31.6B	3.58B	6 of 128	Sophea-Nemo-3.5-Lightning-v1

Figure 2: Left: one configuration, three seeds, nothing else changed. Right: every accuracy effect we measured over the project, against that range. All of them fit inside it.

Table 2: The reasoning half is math-heavy and thin on the axes we evaluate hardest: commonsense is 6.6% and logic 8.7% of it.
domain (reasoning half)	rows	share
mathematics	22,608	38.2%
science	11,064	18.7%
world knowledge	5,884	10.0%
deductive logic	5,151	8.7%
medical	4,898	8.3%
commonsense	3,904	6.6%
reading comprehension	3,828	6.5%
physics	1,770	3.0%

Figure 3: Every accuracy effect we measured, against the 7.7-point seed-to-seed band (shaded). Attribute-level effects (data selection and corpus versions) sit inside the band we would have been tempted to narrate. The only effect that survives is the replicated phase-2 comparison of §7 (+6.9 pp, 15 arms, permutation p=0.0008): it is inside the band geometrically, but it is estimated across 15 independently trained arms rather than read off a single run, which is precisely the difference the floor teaches. The language-matched accuracy cost (−6.9 pp) is drawn as an open (unfilled) marker on purpose: it is a single-run measurement inside the floor, disclosed rather than interpreted, and not a recipe property. The equal magnitudes of the +6.9 and the −6.9 are a coincidence: different lanes, different arms.

Table 3: Two-Phase continues the same adapter from Reasoning onto the hybrid mix; One-Phase never separates the two halves. The three recipes fail in different ways (§7).
name	trained on	rows
Base	nothing (the released model)	—
Reasoning	reasoning rows only	59,107
Subset	a subset of the same rows	15,607
Two-Phase	reasoning rows, then + the direct half	59,107+
One-Phase	one pass over both halves at once	59,107+

Figure 4: The same two checkpoints on four dimensions, the base and the structure-selected reasoning-only arm (Subset, the “Reason-16k” of Figure 11), on the 1,000-item probe lane. Only the first is accuracy.

Table 4: sd=4.4 pp, range =7.7 pp. Note the last column.
seed	mean	logic	fallback	trace Greek
42	76.2	56.2	10%	1.00
43	68.7	35.6	41%	1.00
44	76.4	54.0	3%	1.00

Figure 5: Median trace length in tokens (each family’s own tokenizer, 5,156 matched items), base against the fine-tuned release. The word-level shortening must repay Greek’s 2.3–2.5× token fertility before it becomes a serving saving: Qwen’s does, NemotronH’s breaks even, Gpt-OSS’s does not.

Table 5: Selection buys −0.5 pp (0.31σ); a 4× smaller pool costs nothing. The 62,562 is the reasoning count of the pre-gate pool the experiment was run against (62,562 reasoning, 60,214 direct); the 59,107 quoted everywhere else is the post-gate train half (59,107 reasoning, 58,985 direct), so the table’s all row and the paper’s reasoning-half count differ by the 3,455 rows the decontamination/dedup gate and the val reserve removed.
phase-1 rows	selection	mean
62,562	all	69.5
15,607	top-by-structure	69.5
15,607	uniform random	69.0

Figure 6: Overrun = trace ≥3× the median trace for that item across arms. Easy = at least 80% of arms answer correctly. Cells pool the arms of each column; hard-item think-mode rates are compressed by generation-cap truncation (§5), so the easy row carries the section’s claims.

Table 6: Per-domain results on the Greek think lane; here and in every table, ↑ marks metrics where higher is better and ↓ where lower is better (math = the two math slices, commonsense = HellaSwag+Winogrande, logic = ProofWriter macro-recall). Rows are grouped by family, each release indented under its own base, so every legitimate accuracy delta is a within-block comparison and the cross-base reading (the one an earlier draft of this paper fell for) is structurally discouraged. Every release now carries its own base row: the NemotronH base dump is the same vintage as the Qwen base dump, and the Nemotron-3.5 base lane, absent until this revision, was generated with the identical lane settings and scored with the same scorer as every other row. The same base is also measured on the NLU retention suite (Table 14, Greek macro 57.5), a direct-mode instrument whose numbers are not comparable to this think-lane table. The Nano release also appears in the retention suite (Table 13), and its fallback shape is the mirror of Gpt-OSS’s: negligible on math and commonsense (1.7/1.8%) but 39.9% on logic, so its 9.8 logic macro-recall is substantially a format floor on the one domain where its answer form breaks. Accuracy and fallback move together: every release keeps the base’s domain ordering, and the Sophea-OSS format-repair dose pulls the commonsense fallback back toward math’s level (83.5→25.7) while cutting logic’s by a third (77.4→50.9). The pre-repair row is reference only; the release is the repaired arm. Conditioning on rows that emit the requested answer line (excluding the fallback-scored path) shrinks the apparent deficits sharply: the Qwen release reads 95.8/78.8/39.5 against its base’s 96.0/82.8/42.7, and the same condition makes Gpt-OSS pre-repair 74.6→92.2 / 53.7→60.2 / 25.2→36.5 and Sophea-OSS-v1 78.6→94.4 / 53.3→58.2 / 26.5→30.0 (math/commonsense/logic), so a substantial share of every raw gap is format compliance, not reasoning.
	anchored accuracy	fallback %
arm	math	cs	logic	math	cs	logic
	↑ higher is better	↓ lower is better
Qwen base (Greek think)	92.9	82.4	42.6	3.3	0.5	0.2
Sophea-Qwen3.6-v1	83.3	75.8	37.3	18.2	22.9	46.3
Gpt-OSS base (Greek think)	90.7	63.2	32.2	2.4	5.2	0.7
before repair (reference)	74.6	53.7	25.2	34.9	83.5	77.4
Sophea-OSS-v1 (repaired)	78.6	53.3	26.5	17.2	25.7	50.9
NemotronH base (Greek think)	86.6	49.2	26.8	8.3	6.2	6.6
Sophea-Nemo-3-Nano-v1	71.5	39.7	9.8	1.7	1.8	39.9
Nemotron-3.5 base (Greek think)	76.9	46.3	25.0	1.4	1.9	4.4
Sophea-Nemo-3.5-Lightning-v1	75.6	40.9	23.8	16.1	14.7	5.3

Figure 7: Each point is an independently trained arm. Corpus version varies within both groups.

Table 7: Language-matching across the three families: all four released checkpoints, the two measurable bases, and the one-directional reference arm. Greek fid.: Greek-trace fidelity on the Greek benchmark; both bases sit at exactly 0.0 (0/5,156 traces reach ratio 0.9), and all four language-matched checkpoints land at the same level (97.4–98.1%). EN ctrl: English-control compliance, 100% for every arm. Switches: in-question language switches per 100 sentences on the Greek lane; the bases switch 16.4 (Qwen) and 9.7 (Gpt-OSS) times, every fine-tuned arm (the one-directional reference included) zero. The base fidelity and switch cells are one instrument run: both base dumps rescored with the current scorer on the same day (an earlier draft printed 19.0 for the Qwen base from an earlier dump vintage). Leak: answer-channel leak, the one place families differ (10.3% Sophea-OSS-v1, 5.2% Lightning, 3.5% Qwen, 0.0% Nano, the only arm at zero; base leak is measured on the English control, fine-tuned leak on the Greek think lane). †Arms of the earlier one-directional recipe, shown for reference (a different recipe, not releases); the Qwen one-directional row is the recipe-evolution comparison: it locks against explicit language instructions (Table 10), not against English questions. §Measured on the 1,900-item paired probe (99.3% English, 0.0% Greek, 0.7% too short to score); this arm was never run on the 1,100-item control the other EN-ctrl cells use. ‡Measured on the pre-repair language-matched arm: the repair dose adds 2% trace-less answer-format rows and does not touch trace language, but the released arm’s own English control has not been generatively re-run. The recipe generalises to every family tried; the leak’s size does not.
	Greek fid.	EN ctrl	switches	leak
arm	(%)↑	(%)↑	/100 sent.↓	(%)↓
Qwen base	0.0	100	16.4	15.6
Qwen lang-matched	98.0	100	0.0	3.5
Qwen one-dir.†	98.7	99.3§	0.0	1.5
Gpt-OSS base	0.0	100	9.7	0.5
Gpt-OSS lang-matched	98.1	100‡	0.0	10.3
Nemotron-3.5 lang-matched	98.1	100	0.0	5.2
NemotronH lang-matched	97.4	100	0.0	0.0
NemotronH one-dir.†	98.4	100	0.0	11.3

Figure 8: The four withdrawn findings of this section, drawn: the effect as first believed (open circle) against the same quantity after its control (filled). Three flip sign; the un-anchored scorer artifact (+29.8 pp, 18σ, one probe) collapses entirely: its endpoint is drawn at zero because the prose reports the artifact removed by anchored scoring, not as a measured residual. Rows are different instruments and lanes, so magnitudes compare within a row, not across rows; the other pairs’ numbers appear in this section’s prose. The section’s fifth failure (§8.5) is a scorer defect rather than an effect-size withdrawal, so it has no before/after pair to draw.

Table 8: single-phase = one pass over both halves; two-phase = reasoning-only then hybrid; reasoning-only = the reasoning half alone. Empty-trace rate is measured in think mode, fallback% in direct mode; means are over all arms of each recipe. Fallback ranges are over the individual arms of each recipe (2–12% across the three reasoning-only arms; 33–40% across the two-phase arms); the three seed replicates of §4 widen it further (3–41% across seeds 42–44, all three of which are among the six phase-1-only arms above), so the fuller picture is 2–41% when seed noise is admitted.
recipe	mean	empty	fallback
One-Phase	66.1	23.6%	12%
Two-Phase	64.4–69.9	0.0–1.3%	33–40%
reasoning-only	73.6	0.0%	2–12%

Figure 9: Trace-language composition per lane, Qwen line: the share of traces that are Greek (character ratio ≥0.9), mixed, or English (≤0.1), on the same instrument as Table 7’s fidelity column. Greek-question rows are the full 5,156-item benchmark; English-question rows are the 1,100-item control (base) and the 1,900-item probe (matched). Two honesty notes drawn rather than hidden: the base’s Greek-question traces are not mostly pure English: only 4.2% are; the mass is mixed script with English scaffolding (median ratio 0.33, §5); and the one-directional arm’s Greek-question lane decomposes as 98.7 Greek / 1.3 mixed / 0.0 English on the same instrument (n=5,156, uninstructed); its English-question lane, generated later on the same 1,900-item probe, measures 99.3% English (Table 7). Uninstructed defaults only; the instructed override is Table 10.

Table 9: The base is ∼5 pp worse on our translations; every fine-tune is slightly better.
arm	human	machine	Δ
Base	95.6	90.7	−4.9 (−3.14σ)
Subset	94.0	95.1	+1.1
Reasoning	93.6	95.3	+1.7
Two-Phase	87.6	90.4	+2.8

Figure 10: Every per-benchmark retention delta behind Table 14: fine-tuned release minus its own base (Lightning against the Lightning base; Sophea-OSS-v1 is the format-repaired release), Titan-1 suite, non-reasoning mode. Per-benchmark binomial sigma applies and a delta is a direction, not a test (Table 13); the value is the pattern: the Qwen column is near-white in both languages, the Sophea-OSS-v1 residual concentrates on the knowledge-heavy benchmarks (MMLU-el −10.1) while HellaSwag gains in both languages on every family, and MMLU is the largest English loss for all three non-Qwen releases.

Table 10: The one-directional fine-tune’s traces are identical under both instructions, and it does not matter how the instruction is phrased. Rows classified as neither/mixed scripts (28% of the base under the Greek instruction) are omitted, so rows need not sum to 100. The matched ft rows, measured after the language-matched re-training of §10, show the override partially returning; its mixed share under the English instruction is 30.4%.
model, instruction	English traces	Greek traces
base, “reason in English”	100.0%	0.0%
base, “reason in Greek”	72.0%	0.0%
fine-tune, “reason in English”	0.0%	98.7%
fine-tune, “reason in Greek”	0.0%	98.7%
matched ft, “reason in English”	44.8%	24.8%
matched ft, “reason in Greek”	0.0%	98.5%

Figure 11: Left: accuracy per axis on the 5,156-item benchmark for the Qwen recipe arms (the cross-family releases are Table 6), with the seed-to-seed noise floor shaded around the Qwen base. Right: the same arms’ cost in words per correct answer (token-denominated cost is Figure 5). Math uses the 250 human-translated items only (§8); logic is macro-recall (§3).

Table 11: Instructed-override compliance on the five released checkpoints: the share of traces in the instructed language when the instruction contradicts the question (n=1,000 Greek-question and n=1,100 English-question items; same scorer as Table 10). Every release obeys “switch to Greek”; only the Qwen line and Gpt-OSS obey “switch to English”, and the RLVR refinement widens Qwen’s opening (44.8→53.9%, §15.1). Two instrument notes, both checked: the Nemotron dumps do not separate trace from answer, so the mandatory Greek answer line falls inside the scored text: re-scoring with that line stripped leaves both Nemotron arms at 0.0% (the Qwen control moves 44.8→49.5%, the expected direction); and the Nemotron arms return 5–9% of rows too short to score against Qwen’s 0%, consistent with the low reasoning ceiling reported in §13.
released checkpoint	“in English”↑	“in Greek”↑
	(Greek Q)	(English Q)
Sophea-Qwen3.6-v1	44.8%	83.7%
Sophea-Qwen3.6-v1.1 (RLVR)	53.9%	85.7%
Sophea-OSS-v1	62.5%	93.3%
Sophea-Nemo-3.5-Lightning-v1	0.0%	92.7%
Sophea-Nemo-3-Nano-v1	0.0%	87.5%
one-directional (reference)	0.0%	91.4%

Table 12: English questions get English traces from every arm measured. The fine-tunes in the first three rows are the dual-mode / reasoning-recipe checkpoints (§7), kept deliberately: they are the recipe-class evidence that one-directional training does not lock English questions. The last two rows are the language-matched checkpoints: the released Qwen checkpoint, re-run on this exact control (94.4, +0.4 over its base, 100% English traces, consistent with its 1,900-item probe above), and the Gpt-OSS language-matched arm evaluated after merging (the pre-repair arm; the released, format-repaired checkpoint’s control has not been generatively re-run, Table 7). Together the rows confirm the language-matching result is not Qwen-specific; none of this retests the hard one-directional lock of Table 10, which remains a property of training in a single output direction.
arm (n=1,100)	accuracy	reasons in EN
Qwen base / ft	94.0 / 96.2	100% / 100%
Gpt-OSS base / ft	95.5 / 96.1	100% / 100%
NemotronH base / ft	92.7 / 93.7	100% / 100%
Qwen language-matched	94.4	100%
Gpt-OSS language-matched	91.9	100%

Table 13: Forgetting on the Titan-1 suite. Top: Gpt-OSS on the nine Greek non-reasoning benchmarks, base against the language-matched fine-tune, before and after the format-repair dose: the language-matched fine-tune initially costs −7.3 points, and the repair dose recovers 4.1 of them, so the residual is −3.2 not zero. Below: the macro deltas this paper reports per family, fine-tuned arm vs its own base; the format-repaired Gpt-OSS is the fine-tuned arm we stand behind. Qwen is flat in both languages; Nano gains Greek (+3.8) against the family’s low 0.49 Nano base (mostly catch-up, not forgetting); Lightning is read against its own base, measured separately (Greek macro 57.5, well above Nano’s 48.6), and gains +1.7; an earlier +10.6 reading against the shared Nano base is superseded by that measurement. Per benchmark against binomial standard error; a macro delta is a direction, not a test. †On English the pre-repair language-matched Gpt-OSS lost −7.7 points (75.7→68.0); re-measured on the repaired release, the English macro is 76.7, +1.0 above base: the repair dose recovered both lanes, so the pre-repair loss was format behaviour throughout. Macro deltas are computed on unrounded macros; Table 14 prints one decimal, so its Qwen columns round to 75.7→75.6 Greek / 85.1→85.2 English.
Greek macro	base	language-matched	+ format repair
Gpt-OSS	61.2	53.9	58.0
macro deltas vs base (points):
Sophea-Qwen3.6-v1	−0.01 Greek / +0.08 English
Sophea-OSS-v1	−3.2 Greek / +1.0 English†
Sophea-Nemo-3-Nano-v1	+3.8 Greek / −2.0 English
Sophea-Nemo-3.5-Lightning-v1	+1.7 Greek / −1.1 English

Table 14: Every benchmark behind the macros of Table 13 (Titan-1 suite, non-reasoning mode; accuracy %). Qwen ft is Sophea-Qwen3.6-v1 (language-matched); Gpt-OSS ft is Sophea-OSS-v1, the format-repaired release (its pre-repair intermediate is in Table 13); Nano ft is Sophea-Nemo-3-Nano-v1, the language-matched Nano release; Lightning ft is Sophea-Nemo-3.5-Lightning-v1 (language-matched); the two Nemotron generations share the same 31.6B / 6-of-128 routing profile (Table 1) and each is read against its own base column. (The earlier one-directional recipe arm is reported only in §13’s recipe history, not as a release.) The per-benchmark view localises the Gpt-OSS story: before the repair the loss concentrated on the instruction-format-heavy reading benchmarks (Belebele-el −17.7, ARC-Challenge −13.1 el / −19.1 en, MMLU-en −16.0) while HellaSwag and Winogrande gained in both languages, and the repair dose recovers most where the loss was format (Belebele-el regains +13.0 of the 17.7). English retention on the released Gpt-OSS checkpoint recovers to +1.0 above its base; Lightning retains English within 1.1 points of its own base.
	Qwen	Gpt-OSS	Nemotron
benchmark	base	ft	base	ft	Nano base	Nano ft	Ltng. base	Ltng. ft
Greek (nine benchmarks, the Greek macro of Table 13):
ARC-Challenge-el	91.8	92.0	77.1	68.6	54.2	59.0	68.7	72.8
ARC-Easy-el	96.9	97.1	85.4	79.9	62.3	70.1	75.8	81.4
Belebele-el	92.7	93.7	86.0	81.3	67.0	78.0	81.3	83.3
GreekMMLU	83.8	83.0	66.3	61.3	58.8	53.3	64.2	62.2
HellaSwag-el	57.6	59.4	42.9	52.1	39.2	47.6	44.5	51.2
Medical-MCQA-el	81.2	80.3	42.1	38.7	27.8	31.2	41.7	38.2
TruthfulQA-el	39.1	36.5	39.7	35.5	31.7	33.2	34.2	33.7
Winogrande-el	61.0	62.5	55.4	59.6	53.3	58.3	55.8	60.1
MMLU-el	76.9	76.2	55.5	45.4	43.2	40.8	51.3	49.6
Greek macro	75.7	75.6	61.2	58.0	48.6	52.4	57.5	59.2
English retention (five benchmarks):
ARC-Challenge	96.1	95.7	90.0	86.8	88.3	82.2	92.0	89.1
ARC-Easy	99.1	99.1	96.0	95.7	96.2	93.4	97.8	96.4
HellaSwag	73.2	74.2	57.0	68.4	67.9	73.1	72.7	75.9
Winogrande	73.2	74.2	64.6	66.7	69.9	72.9	72.0	73.3
MMLU	83.9	82.6	70.8	65.8	69.9	60.5	75.8	69.7
English macro	85.1	85.2	75.7	76.7	78.4	76.4	82.0	80.9

Table 15: Supported and unsupported configuration choices, each with its evidential basis.
Supported by our evidence	basis
Drop the hybrid phase 2; train reasoning-only	+6.9 pp over 15 runs, p=0.0008 (§7); and the direct mode it sacrifices costs nothing measurable on direct-mode NLU (E3, §11)
Generate exploratory traces, not explanations	structure 0.27→0.68 on the same questions
Forbid formulaic openings in the generator	24/24 identical openings without it
Report fallback% beside accuracy	separates reasoning from format compliance
Run the seed control before any ablation	7.7 pp (§4)
Probe trace-language steerability per checkpoint	same recipe, four checkpoints, 62.5/44.8/0/0 (§10.1)
Keep a human-translated benchmark subset	caught a 3.1σ artifact (§8)
Test numeric normalisation in the target language’s own convention	5 of 6 Greek-thousands answers scored wrong while right (§8.5)
Gate metrics on length correlation	removed 3 of our 7
Fix format and leak defects with verifiable-reward RL, not more SFT data	fallback →2.5%, leak →0%, control flat, ∼78 GPU-h/arm (§15.1)
Run a random-reward control arm with any RLVR claim	it reproduced our baseline on every axis; without it, every gain here would be unattributable (§15.1)
NOT supported	why
A specific corpus version	five expansions, all inside the noise floor
A data-selection method	selection ≈ random, −0.5 pp (0.31σ)
A LoRA rank, stride, or epoch count	never varied; no evidence either way
A per-domain model choice	two of three per-domain score spreads exceed the seed range itself (§12)

Table 16: The pre-registered RLVR round, scored on the held-out instruments (Greek think lane n=5,156; override lanes n=1,000/1,100), same scorer, same day. The before row is the released checkpoint’s frozen dump rescored with the current scorer; its full same-day regeneration reproduced all 5,156 responses bit-identically (greedy decoding is deterministic here), and the plain-mode lanes to the decimal (44.8/83.7): the before column is not a provenance caveat but a verified constant. The control row is the entire admissibility argument: on every axis the random-reward arm reproduces the baseline, so nothing below can be elicitation. Accuracy is shown for completeness only: +3.3–3.5 is inside the ±7.7 seed floor of §4 and is claimed by nobody.
	Greek		fall-		override	reverse
arm	fid. %↑	acc↑	back %↓	leak %↓	el→EN %↑	en→EL %↑
SFT checkpoint (before)	97.98	73.7	24.1	3.53	44.8	83.7
random reward (control)	98.06	74.0	22.1	3.61	44.1	83.8
correct+format+term.	98.22	77.2	2.5	0.00	48.8	84.7
+ language	98.02	77.2	5.6	0.04	45.9	84.5
+ override	98.27	77.0	2.8	0.02	53.9	85.7

왜 중요한가

저자원 언어로 AI를 서비스하려는 회사나 커뮤니티에게, '정확도 점수가 좋아졌다'는 흔한 주장이 실제로는 거의 근거가 없을 수 있다는 경고다. 대신 모델이 어떤 언어로 사고하는지, 토큰을 얼마나 쓰는지, 형식을 잘 지키는지 같은 구체적 행동 지표를 봐야 진짜 개선을 확인할 수 있다.

이 논문의 용어

SFT (지도미세조정) · 정답이 달린 예시 데이터로 이미 학습된 모델을 추가로 훈련시켜 특정 행동을 하도록 만드는 방법
RLVR (검증 가능한 보상 기반 강화학습) · 정답 여부를 자동으로 확인할 수 있는 보상 신호를 이용해 모델을 추가로 훈련시키는 강화학습 방식
MoE (전문가 혼합 모델) · 전체 파라미터 중 일부 '전문가' 부분만 골라 쓰는 구조로, 큰 모델을 싸게 서비스할 수 있게 해준다
시드(seed) · 훈련 시작 시 난수를 결정하는 값으로, 같은 설정이라도 시드가 다르면 결과가 달라질 수 있다
언어 충실도(language fidelity) · 모델이 사고 과정을 실제로 요청된 언어로 작성하는 정도

저자 · Ayoub Kirouane

arXiv에서 원문 보기

METAL MEDIA 최신 기사

그림 출처: Ayoub Kirouane et al., arXiv:2608.17744, arxiv-nonexclusive

그리스어로 생각하는 AI 모델을 만들어보니, 정확도 점수는 거의 의미가 없었다

그리스어로 생각하는 AI 모델을 만들어보니, 정확도 점수는 거의 의미가 없었다

무엇을 했나

왜 중요한가

이 논문의 용어

최신 논문

METAL MEDIA 최신 기사