Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox›
Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Study
arXiv:2608.182612026-08-20
Training MoE routers to be cache-friendly cuts memory misses a lot, but not without hurting accuracy, according to a pre-registered negative-result study
The author shows that serving a 235-billion-parameter Mixture-of-Experts model (Qwen3-235B) on an 8GB consumer GPU is limited by memory bandwidth, not compute, confirmed by measurements matching a simple bytes-per-token-over-bandwidth model. They then pre-registered an experiment to see if a router can be trained to keep reusing the same experts (making it more 'cacheable') without hurting language-model quality. The mechanism worked at cutting cache misses by up to 60%, but every tested configuration broke the pre-registered rule that perplexity (a quality score) worsen by no more than 1%.
METAL MEDIA explanatory visual
Training MoE routers to be cache-friendly cuts memory misses a lot, but not without hurting accuracy, according to a pre-registered negative-result study
01Real measurements on Qwen3-235B (134GB, 4-bit quantized) running from an SSD showed decode speed of 0.44 tokens/second, matching a first-principles model of bandwidth divided by bytes needed per token; trying to batch 32 concurrent requests to amortize disk reads instead collapsed due to OS paging thrash
02A custom tracing tool (llama-moe-trace), built as a small non-invasive addition to llama.cpp, revealed that Qwen3-30B's router reuses the same experts between adjacent tokens twice as often as chance, that 95% of traffic concentrates on 52.5% of all experts, and that code-domain routing barely overlaps with prose/math domains
03Training small MoE models (137M to 340M parameters) with an extra loss term rewarding expert reuse was tested against pre-registered pass/fail thresholds (at least 30% fewer cache misses, no more than 1% perplexity increase), but miss reduction and quality cost turned out to be tightly linked, so no setting satisfied both conditions at once
04Combining trained locality with a separate, training-free trick that reroutes to a similar cached expert at inference time produced a strong synergy, reaching about 80% fewer cache misses at roughly 3.4% perplexity cost—far cheaper than either method alone—at both the 137M and 340M model sizes
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.
What they did
Real measurements on Qwen3-235B (134GB, 4-bit quantized) running from an SSD showed decode speed of 0.44 tokens/second, matching a first-principles model of bandwidth divided by bytes needed per token; trying to batch 32 concurrent requests to amortize disk reads instead collapsed due to OS paging thrash
A custom tracing tool (llama-moe-trace), built as a small non-invasive addition to llama.cpp, revealed that Qwen3-30B's router reuses the same experts between adjacent tokens twice as often as chance, that 95% of traffic concentrates on 52.5% of all experts, and that code-domain routing barely overlaps with prose/math domains
Training small MoE models (137M to 340M parameters) with an extra loss term rewarding expert reuse was tested against pre-registered pass/fail thresholds (at least 30% fewer cache misses, no more than 1% perplexity increase), but miss reduction and quality cost turned out to be tightly linked, so no setting satisfied both conditions at once
Combining trained locality with a separate, training-free trick that reroutes to a similar cached expert at inference time produced a strong synergy, reaching about 80% fewer cache misses at roughly 3.4% perplexity cost—far cheaper than either method alone—at both the 137M and 340M model sizes
Table 1: Memory hierarchy on the test machine (i9-12900, RTX 3070 8 GB, 32 GB DDR4, WD SN530 NVMe). The 4-bit 235B model is 134 GB; only ∼40 GB fits the fast tiers.
Tier
Bandwidth
Capacity
Share of 134 GB model
VRAM (GDDR6)
∼448 GB/s
8 GB
6%
System RAM (DDR4)
∼50 GB/s
32 GB
24%
NVMe SSD (PCIe 3, DRAM-less)
∼2.4 GB/s
≫
∼70%
Table 2: Serving measurements, Qwen3-235B-A22B Q4_K_M. Aggregate throughput at batch B across concurrent streams. The union-of-experts model predicts aggregate rate should rise with B; it does to B=8 then collapses.
Condition
Measured
Model
Decode, single stream (warm)
0.441 tok/s
—
Decode, single stream (cold)
0.128 tok/s
0.20 tok/s
Prefill
0.25 tok/s
—
Aggregate, B=1
0.128 tok/s
0.112
Aggregate, B=8
0.189 tok/s
0.20
Aggregate, B=32
0.087 tok/s
0.35
Table 3: Routing profile of Qwen3-30B-A3B (llama-moe-trace, 8k tokens × 4 domains). Cache hit rates at the 235B’s 13.4% fast-memory budget (layer 24, code domain).
Metric
Value
P(expert reused at next token)
0.444 (chance 0.223, 2.0×)
Working set (95% of traffic)
52.5% of experts
LRU hit rate @ 13.4% budget
65.9%
LFU hit rate @ 13.4% budget
60.1%
Static-pin hit rate @ 13.4% budget
59.2%
Belady oracle @ 13.4% budget
79.1%
Table 4: Cross-domain expert-usage similarity (Qwen3-30B-A3B). Code’s expert set is nearly disjoint from the others.
code
general
math
medical
code
1.00
0.16
0.14
0.11
general
0.16
1.00
0.33
0.42
math
0.14
0.33
1.00
0.35
medical
0.11
0.42
0.35
1.00
Table 5: Main results (200M tokens/arm). Metrics averaged over prose/code/math. “hit” is the fraction of expert-loads already resident.
run
arm
λ
PPL
reuse
LRU hit@25%
static hit@50%
a-main (s1)
A
—
32.0
0.336
0.494
0.747
a-main (s2)
A
—
31.9
0.312
0.480
0.734
b-l02
B
0.02
32.0
0.407
0.564
0.794
b-l03
B
0.03
32.5
0.451
0.608
0.817
b-main (s1)
B
0.05
32.6
0.634
0.788
0.930
b-main (s2)
B
0.05
32.9
0.635
0.796
0.933
c-main
C
—
32.9
0.470
0.733
0.991
Table 6: Scale rung. Locality tax (arm A baseline vs arm B λ=0.05 at τ=0) and the stacked training-free rerouting point (τ=0.5 on arm B), at 137M and 340M; same 200M-token budget, cap=25% experts, LRU, seed 1. The tax does not shrink with scale; the stacking advantage persists.
locality tax (τ=0)
stacked (τ=0.5)
size
ΔPPL
miss red.
ΔPPL
miss red.
137M
+2.0%
59%
+2.4%
80%
340M
+2.5%
57%
+3.4%
82%
Table 7: Per-domain metrics, baseline (A) vs. locality (λ=0.05), each averaged over 2 seeds. “hit@25%” is LRU; “static@50%” is static-pin.
arm
domain
PPL
ΔPPL
reuse
hit@25%
static@50%
A (λ=0)
prose
63.9
—
0.285
0.432
0.765
A (λ=0)
code
4.7
—
0.384
0.544
0.722
A (λ=0)
math
27.3
—
0.303
0.486
0.738
B (λ=0.05)
prose
65.8
+3.0%
0.619
0.777
0.930
B (λ=0.05)
code
4.7
+0.0%
0.685
0.829
0.954
B (λ=0.05)
math
27.8
+1.8%
0.599
0.771
0.912
Table 8: Complete run log. Metrics averaged over prose/code/math.
run
arm
λ
tokens
PPL
reuse
LRU hit@25%
static@50%
a-sanity
A
—
50M
69.3
0.307
0.469
0.731
b-l05
B
0.05
50M
69.5
0.644
0.809
0.957
b-l01
B
0.01
50M
67.7
0.354
0.518
0.779
b-l20
B
0.20
50M
70.1
0.861
0.855
0.977
a-main s1
A
—
200M
32.0
0.336
0.494
0.747
a-main s2
A
—
200M
31.9
0.312
0.480
0.734
b-l02
B
0.02
200M
32.0
0.407
0.564
0.794
b-l03
B
0.03
200M
32.5
0.451
0.608
0.817
b-main s1
B
0.05
200M
32.6
0.634
0.788
0.930
b-main s2
B
0.05
200M
32.9
0.635
0.796
0.933
c-main
C
—
200M
32.9
0.470
0.733
0.991
Table 9: Hyperparameters (identical across arms; only the router-loss weights λ,μ differ).
Model
Optimization
dmodel
384
optimizer
AdamW (β=0.9,0.95)
layers
8
weight decay
0.1
attention heads
6
peak LR
6×10−4
experts E
16
schedule
one-cycle, 2% warmup
top-k
2
grad clip
1.0
expert dff
768
precision
bf16 autocast
vocab (GPT-2 BPE)
50257
batch × seq
8×1024
total / active params
137M / 38M
balance weight α
0.01
context length
1024
domain weight μ
0.1
Why it matters
The finding is an honest, rigorously pre-registered demonstration that the appealing idea of 'just train the router to be cache-friendly for free' does not hold up under a strict multi-domain test at small model scale, which matters for anyone trying to run huge MoE models on consumer hardware. It also points to a more promising practical direction—combining training-time and inference-time tricks—and releases open tooling and traces that others can build on.
Terms in this paper
Mixture-of-Experts (MoE) · a model architecture that activates only a small subset of many expert sub-networks per token, keeping total parameters huge but per-token computation small
router · the part of an MoE model that decides which expert sub-networks handle each token
memory-bandwidth bound · a situation where speed is limited by how fast data can be read from memory, not by how fast the processor can compute
perplexity · a score measuring how well a language model predicts text; lower is better
LRU cache · a caching policy that discards the least recently used item first when space runs out
Original abstract (English)
Serving a 235B-parameter Mixture-of-Experts (MoE) model on a single 8 GB GPU is bottlenecked not by compute but by memory bandwidth: decode must stream each token's active experts from whichever tier holds them, and on consumer hardware most experts sit on an SSD far slower than RAM. We quantify this bandwidth wall on Qwen3-235B (Q4_K_M, 134 GB): measured decode is 0.44 tok/s warm, matching a bytes-per-token / bandwidth model, while a batching scheme that should amortize one disk sweep instead collapses at batch 32 from paging thrash. We build llama-moe-trace, a zero-surgery router-telemetry tool, and measure routing on Qwen3-30B: adjacent-token expert reuse is 2.0x chance, 95% of traffic uses 52.5% of experts, and an LRU cache of 13.4% of experts serves 66% of requests. We then ask whether cacheability is trainable: we pre-register training of 137M MoE language models with auxiliary locality and domain router losses, under joint criteria on cache-miss reduction and perplexity. The mechanism works (misses down up to 60%; a 99% static-pin hit rate) but every configuration fails the pre-registered <=1% perplexity gate -- miss reduction and quality are tightly coupled. Concurrent StickyMoE reports the same loss as near-free on single-domain sub-25M models; on multi-domain 137M we find the tax real. Our contribution is this pre-registered, stricter-criterion, multi-domain evaluation plus edge-serving measurements. A 340M rung shows the tax does not shrink with scale (it rises slightly). We further show training-free cache-aware rerouting stacks with trained locality -- together ~80% miss reduction at <=3.4% perplexity at both sizes, far cheaper than either alone -- while domain-primed prefetching does not help. All code, traces, and the pre-registration are released.
