Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Study
Table 1: Memory hierarchy on the test machine (i9-12900, RTX 3070 8 GB, 32 GB DDR4, WD SN530 NVMe). The 4-bit 235B model is 134 GB; only ∼40 GB fits the fast tiers.
Tier
Bandwidth
Capacity
Share of 134 GB model
VRAM (GDDR6)
∼448 GB/s
8 GB
6%
System RAM (DDR4)
∼50 GB/s
32 GB
24%
NVMe SSD (PCIe 3, DRAM-less)
∼2.4 GB/s
≫
∼70%
Table 2: Serving measurements, Qwen3-235B-A22B Q4_K_M. Aggregate throughput at batch B across concurrent streams. The union-of-experts model predicts aggregate rate should rise with B; it does to B=8 then collapses.
Condition
Measured
Model
Decode, single stream (warm)
0.441 tok/s
—
Decode, single stream (cold)
0.128 tok/s
0.20 tok/s
Prefill
0.25 tok/s
—
Aggregate, B=1
0.128 tok/s
0.112
Aggregate, B=8
0.189 tok/s
0.20
Aggregate, B=32
0.087 tok/s
0.35
Table 3: Routing profile of Qwen3-30B-A3B (llama-moe-trace, 8k tokens × 4 domains). Cache hit rates at the 235B’s 13.4% fast-memory budget (layer 24, code domain).
Metric
Value
P(expert reused at next token)
0.444 (chance 0.223, 2.0×)
Working set (95% of traffic)
52.5% of experts
LRU hit rate @ 13.4% budget
65.9%
LFU hit rate @ 13.4% budget
60.1%
Static-pin hit rate @ 13.4% budget
59.2%
Belady oracle @ 13.4% budget
79.1%
Table 4: Cross-domain expert-usage similarity (Qwen3-30B-A3B). Code’s expert set is nearly disjoint from the others.
code
general
math
medical
code
1.00
0.16
0.14
0.11
general
0.16
1.00
0.33
0.42
math
0.14
0.33
1.00
0.35
medical
0.11
0.42
0.35
1.00
Table 5: Main results (200M tokens/arm). Metrics averaged over prose/code/math. “hit” is the fraction of expert-loads already resident.
run
arm
λ
PPL
reuse
LRU hit@25%
static hit@50%
a-main (s1)
A
—
32.0
0.336
0.494
0.747
a-main (s2)
A
—
31.9
0.312
0.480
0.734
b-l02
B
0.02
32.0
0.407
0.564
0.794
b-l03
B
0.03
32.5
0.451
0.608
0.817
b-main (s1)
B
0.05
32.6
0.634
0.788
0.930
b-main (s2)
B
0.05
32.9
0.635
0.796
0.933
c-main
C
—
32.9
0.470
0.733
0.991
Table 6: Scale rung. Locality tax (arm A baseline vs arm B λ=0.05 at τ=0) and the stacked training-free rerouting point (τ=0.5 on arm B), at 137M and 340M; same 200M-token budget, cap=25% experts, LRU, seed 1. The tax does not shrink with scale; the stacking advantage persists.
locality tax (τ=0)
stacked (τ=0.5)
size
ΔPPL
miss red.
ΔPPL
miss red.
137M
+2.0%
59%
+2.4%
80%
340M
+2.5%
57%
+3.4%
82%
Table 7: Per-domain metrics, baseline (A) vs. locality (λ=0.05), each averaged over 2 seeds. “hit@25%” is LRU; “static@50%” is static-pin.
arm
domain
PPL
ΔPPL
reuse
hit@25%
static@50%
A (λ=0)
prose
63.9
—
0.285
0.432
0.765
A (λ=0)
code
4.7
—
0.384
0.544
0.722
A (λ=0)
math
27.3
—
0.303
0.486
0.738
B (λ=0.05)
prose
65.8
+3.0%
0.619
0.777
0.930
B (λ=0.05)
code
4.7
+0.0%
0.685
0.829
0.954
B (λ=0.05)
math
27.8
+1.8%
0.599
0.771
0.912
Table 8: Complete run log. Metrics averaged over prose/code/math.
run
arm
λ
tokens
PPL
reuse
LRU hit@25%
static@50%
a-sanity
A
—
50M
69.3
0.307
0.469
0.731
b-l05
B
0.05
50M
69.5
0.644
0.809
0.957
b-l01
B
0.01
50M
67.7
0.354
0.518
0.779
b-l20
B
0.20
50M
70.1
0.861
0.855
0.977
a-main s1
A
—
200M
32.0
0.336
0.494
0.747
a-main s2
A
—
200M
31.9
0.312
0.480
0.734
b-l02
B
0.02
200M
32.0
0.407
0.564
0.794
b-l03
B
0.03
200M
32.5
0.451
0.608
0.817
b-main s1
B
0.05
200M
32.6
0.634
0.788
0.930
b-main s2
B
0.05
200M
32.9
0.635
0.796
0.933
c-main
C
—
200M
32.9
0.470
0.733
0.991
Table 9: Hyperparameters (identical across arms; only the router-loss weights λ,μ differ).
Serving a 235B-parameter Mixture-of-Experts (MoE) model on a single 8 GB GPU is bottlenecked not by compute but by memory bandwidth: decode must stream each token's active experts from whichever tier holds them, and on consumer hardware most experts sit on an SSD far slower than RAM. We quantify this bandwidth wall on Qwen3-235B (Q4_K_M, 134 GB): measured decode is 0.44 tok/s warm, matching a bytes-per-token / bandwidth model, while a batching scheme that should amortize one disk sweep instead collapses at batch 32 from paging thrash. We build llama-moe-trace, a zero-surgery router-telemetry tool, and measure routing on Qwen3-30B: adjacent-token expert reuse is 2.0x chance, 95% of traffic uses 52.5% of experts, and an LRU cache of 13.4% of experts serves 66% of requests. We then ask whether cacheability is trainable: we pre-register training of 137M MoE language models with auxiliary locality and domain router losses, under joint criteria on cache-miss reduction and perplexity. The mechanism works (misses down up to 60%; a 99% static-pin hit rate) but every configuration fails the pre-registered <=1% perplexity gate -- miss reduction and quality are tightly coupled. Concurrent StickyMoE reports the same loss as near-free on single-domain sub-25M models; on multi-domain 137M we find the tax real. Our contribution is this pre-registered, stricter-criterion, multi-domain evaluation plus edge-serving measurements. A 340M rung shows the tax does not shrink with scale (it rises slightly). We further show training-free cache-aware rerouting stacks with trained locality -- together ~80% miss reduction at <=3.4% perplexity at both sizes, far cheaper than either alone -- while domain-primed prefetching does not help. All code, traces, and the pre-registration are released.
