Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox

METAL MEDIA

FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution

arXiv:2608.161572026-08-16

A serving system that lets one personal computer run massive open-weight AI models

FreeToken is a serving system that runs huge MoE (mixture-of-experts) models, too large to fit in a single consumer GPU's memory, at interactive speed on hardware ranging from laptops to workstations. It measures the GPU-CPU PCIe bandwidth and CPU processing bandwidth on the actual machine in real time, then decides how to split expert-module work between the GPU cache and direct CPU execution. As a result, it delivers 1.5-2.3x higher decode throughput than the best existing edge serving systems on an RTX 5090, and serves a 35B-parameter model on an 8GB laptop GPU faster than the median decode speed of the Codex coding agent.

METAL MEDIA explanatory visual

A serving system that lets one personal computer run massive open-weight AI models

  1. 01MoE models route each token through only a few experts, but the full pool of experts is far larger than GPU memory, so most experts must sit in host (CPU) memory and be fetched on demand — this fetching delay was the main bottleneck for edge serving
  2. 02During prefill (processing the prompt), FreeToken double-buffers computation and loading: while the GPU computes the current layer, the next layer's experts stream over PCIe in the background, and when an agent edits conversation history, checkpoints saved at special-token boundaries let the system re-process only the newly changed part instead of the whole context
  3. 03During decode (generating tokens), frequently used experts are kept in a GPU LRU cache, while cache misses are split between PCIe transfer and direct CPU execution using a ratio (q*) computed from bandwidths measured on that specific machine
  4. 04Because a personal GPU is often shared with other apps, FreeToken can resize its GPU expert cache on the fly without restarting the whole serving engine
  5. 05On an RTX 5090, it served Qwen3.6-35B-A3B at 77-83 tokens/sec and DeepSeek-V4-Flash at 22-25 tokens/sec, and on a single workstation GPU it served the 753B-parameter GLM-5.2 model at twice the speed of llama.cpp
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

  1. MoE models route each token through only a few experts, but the full pool of experts is far larger than GPU memory, so most experts must sit in host (CPU) memory and be fetched on demand — this fetching delay was the main bottleneck for edge serving
  2. During prefill (processing the prompt), FreeToken double-buffers computation and loading: while the GPU computes the current layer, the next layer's experts stream over PCIe in the background, and when an agent edits conversation history, checkpoints saved at special-token boundaries let the system re-process only the newly changed part instead of the whole context
  3. During decode (generating tokens), frequently used experts are kept in a GPU LRU cache, while cache misses are split between PCIe transfer and direct CPU execution using a ratio (q*) computed from bandwidths measured on that specific machine
  4. Because a personal GPU is often shared with other apps, FreeToken can resize its GPU expert cache on the fly without restarting the whole serving engine
  5. On an RTX 5090, it served Qwen3.6-35B-A3B at 77-83 tokens/sec and DeepSeek-V4-Flash at 22-25 tokens/sec, and on a single workstation GPU it served the 753B-parameter GLM-5.2 model at twice the speed of llama.cpp
Table 1: Test systems. BP is the measured host-to-device expert-transfer bandwidth over PCIe; BH is the measured effective bandwidth of the CPU-side MoE expert kernel. On the three rented servers the CPU-thread and DRAM columns give container quotas.
SystemGPU (VRAM)PCIeBPCPU (threads)DRAMBH
(GB/s)(GiB)(GB/s)
5090RTX 5090 (32 GB)5.0 ×1652.72× Xeon Gold 6459C (32)DDR5 18077.3
4090RTX 4090 (24 GB)4.0 ×1625.12× Xeon Platinum 8358P (32)DDR4 24063.2
3090RTX 3090 (24 GB)4.0 ×1625.32× Xeon Gold 6330 (28)DDR4 18056.7
5090 desktopRTX 5090 (32 GB)5.0 ×1649.0Ryzen 9 9950X3D (32)DDR5 19253.8
4060 laptopRTX 4060 Laptop (8 GB)4.0 ×811.8Core i9-13900H (20)LPDDR5 3247.5
PRO 6000RTX PRO 6000 (96 GB)5.0 ×1651.5Xeon Platinum 8559C (48)DDR5 512178

Why it matters

Even though top-tier AI models are released as open weights, running them has effectively required datacenter-class hardware; this work shows that laptops and gaming desktops people already own can serve them at practical speeds. It suggests that AI accessibility depends less on whether weights are open and more on the quality of the serving software.

Terms in this paper

  • MoE (Mixture of Experts) · A model architecture where only a subset of expert modules is activated per input
  • Prefill · The stage where the model processes the entire input prompt at once
  • Decode · The stage where the model generates output tokens one at a time
  • LRU cache · A cache that evicts the least-recently-used items first
  • TTFT (Time To First Token) · The time from a request being sent to the first response token appearing

Figures we cannot republish

  • Figure 1: FreeToken serves the models on the cost–capability Pareto frontier, at interactive speed on consumer hardware. (a) Blended API list price (9:1 input:output mix, following the token economics measured on real coding-agent traces (Zhu et al. 2026)) versus Code Arena Elo (LMArena 2026) for representative hosted models. Blue squares mark models FreeToken serves, tagged with the consumer GPU class that serves them; the frontier segment from DeepSeek-V4-Flash to GLM-5.2 is exactly this set. Kimi-K3 releases open weights but exceeds consumer memory (594 GB); Qwen3.5-35B stands in for its successor Qwen3.6-35B, which has no arena rating yet. (b) Mean decode throughput on real agentic workloads for the strongest model each hardware tier holds (coding agents on the first two tiers, a math agent on the third), against actively maintained edge engines. The dashed line marks the median decode speed of Codex in production traces (33 tok/s (Zhu et al. 2026)); × marks configurations an engine cannot serve.
  • Figure 2: FreeToken overview. (1) Prefill: expert loading is double-buffered at full-layer granularity, streaming layer l+1 over PCIe while the GPU computes layer l; recurrent-state checkpoints are anchored at special-token boundaries, so a context edit resumes from the nearest surviving anchor and re-prefills only the new suffix. (2) Decode: most routed experts hit the shared LRU expert cache (here 8 of 12, following temporal locality). The m=4 misses are divided by q⋆=m​BP/BH between cache fills over PCIe (one expert) and in-place CPU execution (three), using bandwidths profiled on the deployed machine; the GPU and CPU partial outputs merge exactly. The host-resident expert pool remains the source of truth throughout.
  • Figure 3: End-to-end serving on the RTX 5090 across four workloads (1. AIME, 2. OpenCode+SWE, 3. Claude Code+SWE, 4.OpenClaw+Email/Cal) and two models (Qwen3.6-35B-A3B BF16 and DeepSeek-V4-Flash MXFP4). Top: decode TPS; bottom: mean TTFT (log scale). × marks configurations an engine cannot serve (Ollama and MoE-Infinity lack DSV4 support; MoE-Infinity provides no usable server for multi-turn agents).
  • Figure 4: (a) Prefill TPS versus prompt length (RTX 5090, Qwen3.6-35B BF16), with and without FreeToken’s pipelined full-layer loading. (b) Decode-time expert miss rate versus cache size (as a percentage of the expert pool) under the three engines’ placement policies, replayed on identical routing traces; lines are means over W1–W4, bands the min–max range.
  • Figure 5: Coding-agent decode TPS across consumer GPUs (SWE issues via the OpenCode harness), Qwen3.6-35B-A3B. 4060 laptop using NVFP4, the other Qwen3.6 columns BF16. The RTX PRO 6000 column is a separate demonstration: GLM-5.2 (753B-A40B, NVFP4) on the math workload; Ollama is not run there. × marks configurations an engine cannot serve.
See the figures in the original paper →

Original abstract (English)

Frontier open-weight models are increasingly available, but serving them still largely assumes datacenter infrastructure. We present FreeToken, an edge-native MoE serving system that treats a personal machine not as a small GPU, but as a unified, elastic inference platform. FreeToken co-designs the full serving stack, including model layout and loading, expert residency, CPU--GPU execution, agentic state reuse, and runtime memory management, around two realities of local AI: agent workloads continuously change their execution pattern, and edge hardware exposes heterogeneous resources whose balance differs from machine to machine. Rather than committing to a fixed offloading strategy, FreeToken continuously maps computation and model state onto the resources actually available. FreeToken supports more than 20 MoE models and real coding and tool-using agents across hardware ranging from an 8GB laptop GPU to a single workstation GPU. More importantly, it changes what these machines can practically serve, from a 35B model on a laptop to a 284B model on a gaming desktop and the 753B GLM-5.2 on a single workstation GPU. FreeToken turns open weights into deployable local software, making the machines users already own a practical platform for frontier-scale intelligence. We release the system at flashml.ai.

Authors · Shuo Yang

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA