Motif 3: Technical Report

arXiv:2608.091192026-08-10

A 314-billion-parameter open-weight Mixture-of-Experts model called Motif 3 has been released, activating only 13.2 billion parameters per token

Motif 3 is a Mixture-of-Experts (MoE) language model with 314 billion total parameters but only 13.2 billion activated per token. It was pretrained on about 12.5 trillion tokens using a new attention design called GDLA plus several stability techniques, then fine-tuned through six specialist teacher models and a distillation process into one unified model. Reported results show GDLA reaching lower training loss with fewer tokens than prior attention methods, and the final model performing competitively against leading open-weight models across a broad benchmark suite.

METAL MEDIA explanatory visual

Motif 3 pipeline: architecture design, large-scale pretraining, multi-teacher distillation

Evidence statusMeasured results and planned work

Sparse MoE design314B total parameters, but only 8 of 384 routed experts per layer are used, activating 13.2B parameters per token
GDLA attentionCombines differential (noise-suppressing) attention with compressed latent key-value representations to lower loss and cache cost
Low-precision & long-context training systemMXFP8 computation and window-aware context parallelism enable pretraining on 12.5T tokens with context lengths up to 256K
Seven specialist teachersSix RL-trained specialists (agentic, professional-work, long-context/abstention, math, code-and-science, chat) plus one SFT-trained software-engineering teacher
Multi-teacher On-Policy DistillationMerges the seven teachers' capabilities into a single unified student model

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

Motif 3 uses fine-grained sparse MoE: out of 384 routed experts per layer, only 8 are selected per token, activating 13.2B of the 314B total parameters.
It introduces Grouped Differential Latent Attention (GDLA), combining a noise-suppressing 'differential attention' mechanism with Multi-head Latent Attention's compressed key-value cache to save memory during inference.
It also adds Expert-Specific PolyNorm (per-expert learned activation coefficients), a modified manifold-constrained hyper-connection (mHC) for multi-stream residual connections, and multi-token prediction (MTP) for stability and inference efficiency.
Pretraining used roughly 12.5 trillion tokens of web, STEM, code, math, multilingual, and domain-specific data, with MXFP8 low-precision computation and window-aware context parallelism enabling training at context lengths up to 256K tokens.
Post-training combined general supervised fine-tuning, six RL-trained specialist teachers, one SFT-trained software-engineering teacher, and Multi-teacher On-Policy Distillation (MOPD) to merge all capabilities into a single model.

Figure 2: Attention training-loss comparison. GDLA achieves lower loss than GDA and MLA and reaches a loss of 3.2 with 9.2% fewer training tokens than MLA.

Table 1: Summary of the principal architectural configuration and model dimensions used in Motif 3, including its hybrid attention pattern, fine-grained expert structure, and long-context support.
Property	Configuration
Total parameters	Approximately 314B
Activated parameters	Approximately 13.2B per token
Transformer layers	53 (2 dense and 51 MoE)
Hidden dimension	4,096
Attention mechanism	Grouped Differential Latent Attention with output gating
Attention pattern	Hybrid GDLA (1 full / 3 sliding-window)
Query / KV heads	80 / 16
Signal / noise query heads	64 / 16
Query-key / Value head dim	192 / 128
FFN dimension (dense / expert)	12,288 / 1,280
Routed experts	384, with top-8 routing
Shared experts	1
Expert activation	Expert-Specific PolyNorm
Residual architecture	Modified manifold-constrained hyper-connections
MTP head	1 layer for self-speculative decoding
Maximum context length	262,144 tokens (256K)

Figure 3: Overall low-precision training recipe. Color encodes numerical precision: teal for MXFP8, blue for BF16, and peach for FP32. Dashed borders mark collectives that cross ranks. Only the row-wise MXFP8 weight is All-Gathered; the column-wise copy required by Dgrad is produced locally by a fused row-to-column transcode kernel. Expert activations are quantized once before EP dispatch, allowing the dispatch itself to use MXFP8. Gradient synchronization exchanges BF16 shards while performing each reduction locally in FP32.

Table 3: Principal expert-health signals monitored during Motif 3 pretraining.
Metric	Failure mode indicated
Dispatch min/median	Starvation or a dead expert caused by routing imbalance
Maximum expert-token count	Concentrated traffic and possible expert overload
Output-weight min/median	Hidden collapse in which an expert receives tokens but its output projection contributes increasingly little
Routed/shared RMS, mean, and maximum	Dominance of shared experts, routed-expert collapse, or a change in the routed-output distribution
Maximum expert-output cosine similarity	Functional collapse in which nominally distinct experts learn similar representations
Routed and shared abs-max/RMS	Activation outliers and early warning signs of numerical instability

Figure 6: MoE component training comparisons from controlled experiments using models with approximately 10 billion parameters. (a) Expert-Specific PolyNorm maintains a higher mean effective rank in the expert gate weights than SwiGLU across layers, indicating a more evenly distributed singular-value spectrum. (b) Decaying router noise reduces the maximum number of tokens assigned to an expert more rapidly and guides the routing distribution toward the median-load regime early in training.

Table 4: Evaluation results for the Motif 3 pretrained base model. CoT denotes chain-of-thought prompting.
MMLU 5-shot	MMLU-Pro 5-shot CoT	ARC-C 25-shot	WinoGrande 5-shot	HellaSwag 10-shot	PIQA 0-shot	GSM8K 8-shot CoT	MATH 4-shot CoT	HumanEval 0-shot	MBPP 3-shot
86.20	68.56	94.71	80.90	88.30	85.14	93.93	70.58	73.70	84.60

Table 5: The seven specialist teachers and their capability coverage.
Teacher	Coverage
Agentic tool use	Interactive shell and tool environments; multi-step task execution
Professional work	Open-ended occupational deliverables graded by comparison
Software engineering	Repository-level modifications verified through test execution
Long-context reasoning & abstention	Retrieval and synthesis over very long inputs; calibrated abstention
Mathematics	Competition and proof-style problems; symbolic and judged answer checking
Code and science	Program synthesis, scientific computing, and physical reasoning
Chat	Dialogue quality, instruction following, and safety

Figure 7: Mean reward over cumulative RL compute for the six GRPO-trained specialist teachers. The lighter curves show the per-update reward measurements, and the darker curves show their smoothed trends.

Table 6: Evaluation results for Motif 3. An asterisk (*) indicates that the corresponding result for Motif 3 is evaluated on the public subset only.
Benchmark	Motif 3 314B-A13B	MiniMax-3 428B-A23B	GLM-5.1 744B-A40B	Kimi-K2.6 1T-A32B	Qwen-3.7 Max	DS-v4-Pro 1.6T-A49B
[0pt][0pt] Agentic
GDPval-AA v2	38.7	44.4	37.8	34.4	39.0	40.2
τ2-Bench Telecom	94.7	88.9	97.7	95.9	94.7	96.2
τ3-Banking	35.3	15.3	13.6	23.3	12.0	30.1
ITBench-AA	51.5∗	-	40.3	31.2	42.5	38.3
[0pt][0pt] Coding
SWE-bench Verified	76.2	75.0	76.4	76.2	80.4	77.4
Terminal-Bench 2.1	74.9	65.2	61.8	65.9	75.0	64.0
SciCode	40.6	45.4	43.8	53.5	53.5	50.0
[0pt][0pt] Reasoning and Knowledge
IMO-AnswerBench	83.2	-	83.8	81.8	90.0	89.8
Apex Shortlist	75.5	-	71.1	77.4	44.5	85.8
GPQA Diamond	83.4	92.9	86.8	91.1	92.4	88.8
HLE	37.0	39.0	30.1	37.5	41.4	37.5
CritPt	6.6	3.7	4.6	8.0	11.4	12.9
AA-Omniscience Accuracy	30.1	16.7	23.7	32.6	31.0	42.9
AA-Omniscience Non-Hallucination	71.6	81.6	70.1	59.5	74.0	5.9
[0pt][0pt] Long Context and Instruction Following
AA-LCR	72.3	80.3	68.0	76.7	75.0	70.0
IFBench	78.2	82.9	76.3	76.0	79.1	76.5

Findings

In controlled experiments with roughly 10-billion-parameter models, GDLA achieved lower training loss than both GDA and MLA, reaching a loss of 3.2 with 9.2% fewer training tokens than MLA.
In the same scale of controlled experiments, Expert-Specific PolyNorm maintained a higher effective rank (a measure of how evenly singular values are distributed) in expert gate weights than SwiGLU.
Decaying router noise reduced the maximum number of tokens assigned to a single expert more quickly and pushed the routing load distribution toward the median early in training.
Across the full evaluation suite, Motif 3 was reported to perform competitively against leading open-weight models, with particularly strong results on long-horizon agentic tasks, mathematical reasoning, scientific knowledge, and hallucination-sensitive evaluation.

Where it can be used

The GDLA and stability techniques (mHC scale annealing, expert-balancing methods) can serve as reference designs for teams building or scaling their own MoE architectures.
The unified model's coverage of reasoning, coding, tool use, professional document work, and long-context understanding makes it a candidate for teams looking to consolidate multiple capabilities into one deployable model.
The low-precision (MXFP8) training recipe and context-parallel strategy for handling sequences up to 256K tokens can inform infrastructure design for training very large or long-context models.

Limits and open work

Performance comparisons against other open-weight models rely on leaderboard-reported scores rather than a fully controlled, identical re-evaluation of all models.
Key architectural components like GDLA and PolyNorm were validated only in controlled experiments at roughly 10-billion-parameter scale, not through direct ablations on the full 314-billion-parameter model.
Some benchmark results for Motif 3 are marked as evaluated only on a public subset, making direct comparison with full benchmark scores of other models imprecise for those entries.
Certain post-training components, such as the Korean-focused chat teacher rubric, were designed and validated primarily for Korean-language behavior and may need further generalization checks for broader multilingual use.

Why it matters

This report gives a detailed, large-scale engineering account of how a compact team built and stabilized a 300B+-parameter MoE model, which is valuable as a reference point for teams working on open-weight foundation models. It also documents concrete low-precision and long-context training techniques that other groups can study when scaling their own systems.

Terms in this paper

Mixture-of-Experts (MoE) · An architecture where only a subset of many expert sub-networks is used per input, allowing large total capacity with lower per-token compute
GDLA · Grouped Differential Latent Attention, this paper's new attention design combining differential (noise-canceling) attention with compressed latent key-value representations
MXFP8 · An 8-bit low-precision numerical format used to reduce memory and computation costs
MOPD (Multi-teacher On-Policy Distillation) · A distillation method that transfers knowledge from multiple specialist teacher models into one student model
GRPO · A reinforcement learning method used to train the specialist teacher models

Original abstract (English)

We introduce Motif 3, a decoder-only Mixture-of-Experts language model with 314 billion total parameters and 13.2 billion activated per token. Each sparse MoE layer contains 384 routed experts, with eight selected per token. This fine-grained sparsity provides substantial expert capacity while limiting computation. Motif 3 is built around Grouped Differential Latent Attention (GDLA), which integrates grouped differential attention with the compressed key-value representation of Multi-head Latent Attention. The architecture further incorporates modified manifold-constrained hyper-connections, Expert Specific PolyNorm activations, and multi-token prediction to improve optimization stability, expert specialization, and inference efficiency. We pretrain Motif 3 on approximately 12.5 trillion tokens spanning web documents, STEM, code, mathematics, multilingual content, and domain-specialized corpora. Expert-balancing and numerical-stabilization techniques support stable training at scale, while selective MXFP8 computation and communication, memory-efficient fused kernels, and window-aware context parallelism enable training with context lengths up to 256K tokens. Our post-training pipeline combines general supervised fine-tuning, six specialist teachers trained with reinforcement learning, a software-engineering teacher trained with supervised fine-tuning, and Multi-teacher On-Policy Distillation. The resulting unified model consolidates complementary capabilities in reasoning, coding, tool use, professional work, long-context understanding, calibrated abstention, and instruction following. Across a broad evaluation suite, Motif 3 demonstrates competitive performance against leading open weight models, including strong results on long-horizon agentic tasks, mathematical reasoning, scientific knowledge, and hallucination-sensitive evaluation.

Authors · Junghwan Lim, Joon Son Chung, Sungmin Lee, Wai Ting Cheung, Gihun Cho, Minsu Ha, Sangho Kang, Beomgyu Kim, Dongseok Kim, Jangwoong Kim, Taehyun Kim, Taewhan Kim, Jeesoo Lee, Jeongdoo Lee, Junhyeok Lee, Dongpin Oh, Hyeyeon Cho, Dahye Choi, Jaeheui Her, Hanbin Jung, Changjin Kang, Minjae Kim, Youngrok Kim, Hyukjin Kweon, Hongjoo Lee, Yeongjae Park, Bokki Ryu

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Junghwan Lim et al., arXiv:2608.09119, arxiv-nonexclusive