FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution
让一台个人电脑也能运行超大规模开源AI模型的服务系统
FreeToken是一个服务系统,能让原本装不进单张消费级GPU显存的超大型MoE(混合专家)模型,在从笔记本到工作站的各类消费级硬件上以交互速度运行。它会实时测量该机器上GPU与CPU之间的PCIe带宽以及CPU处理带宽,据此动态决定把专家模块的计算分配给GPU缓存还是直接在CPU上执行。结果是在RTX 5090上,其解码吞吐量比现有最好的边缘服务系统高1.5到2.3倍,而在仅8GB显存的笔记本GPU上运行350亿参数模型的速度,也超过了编程助手Codex的中位数解码速度。
METAL MEDIA 解读图
让一台个人电脑也能运行超大规模开源AI模型的服务系统
- 01MoE模型每个token只会用到少数几个专家模块,但完整的专家池远大于GPU显存,因此大部分专家只能放在CPU内存中按需调取,而这个调取延迟正是以往边缘部署的主要瓶颈
- 02在prefill(处理输入提示)阶段,FreeToken采用双缓冲机制:GPU计算当前层的同时,下一层的专家权重通过PCIe在后台预先加载;当智能体编辑对话历史时,系统会在特殊标记边界处保存检查点,使得只需重新处理新增的那一小段内容
- 03在decode(逐个生成token)阶段,常用的专家模块被保留在GPU的LRU缓存中,未命中缓存的专家则按照该机器实测的PCIe带宽与CPU处理带宽计算出的比例(q*),分别交给PCIe传输和CPU直接执行来处理
- 04由于个人GPU常常要和其他程序共享显存,FreeToken可以在不重启整个服务引擎的情况下,动态调整GPU专家缓存的大小
- 05在RTX 5090上,Qwen3.6-35B-A3B模型达到每秒77至83个token,DeepSeek-V4-Flash达到每秒22至25个token;在单张工作站级GPU上,7530亿参数的GLM-5.2模型的运行速度是llama.cpp的两倍
他们做了什么
- MoE模型每个token只会用到少数几个专家模块,但完整的专家池远大于GPU显存,因此大部分专家只能放在CPU内存中按需调取,而这个调取延迟正是以往边缘部署的主要瓶颈
- 在prefill(处理输入提示)阶段,FreeToken采用双缓冲机制:GPU计算当前层的同时,下一层的专家权重通过PCIe在后台预先加载;当智能体编辑对话历史时,系统会在特殊标记边界处保存检查点,使得只需重新处理新增的那一小段内容
- 在decode(逐个生成token)阶段,常用的专家模块被保留在GPU的LRU缓存中,未命中缓存的专家则按照该机器实测的PCIe带宽与CPU处理带宽计算出的比例(q*),分别交给PCIe传输和CPU直接执行来处理
- 由于个人GPU常常要和其他程序共享显存,FreeToken可以在不重启整个服务引擎的情况下,动态调整GPU专家缓存的大小
- 在RTX 5090上,Qwen3.6-35B-A3B模型达到每秒77至83个token,DeepSeek-V4-Flash达到每秒22至25个token;在单张工作站级GPU上,7530亿参数的GLM-5.2模型的运行速度是llama.cpp的两倍
| System | GPU (VRAM) | PCIe | BP | CPU (threads) | DRAM | BH |
|---|---|---|---|---|---|---|
| (GB/s) | (GiB) | (GB/s) | ||||
| 5090 | RTX 5090 (32 GB) | 5.0 ×16 | 52.7 | 2× Xeon Gold 6459C (32) | DDR5 180 | 77.3 |
| 4090 | RTX 4090 (24 GB) | 4.0 ×16 | 25.1 | 2× Xeon Platinum 8358P (32) | DDR4 240 | 63.2 |
| 3090 | RTX 3090 (24 GB) | 4.0 ×16 | 25.3 | 2× Xeon Gold 6330 (28) | DDR4 180 | 56.7 |
| 5090 desktop | RTX 5090 (32 GB) | 5.0 ×16 | 49.0 | Ryzen 9 9950X3D (32) | DDR5 192 | 53.8 |
| 4060 laptop | RTX 4060 Laptop (8 GB) | 4.0 ×8 | 11.8 | Core i9-13900H (20) | LPDDR5 32 | 47.5 |
| PRO 6000 | RTX PRO 6000 (96 GB) | 5.0 ×16 | 51.5 | Xeon Platinum 8559C (48) | DDR5 512 | 178 |
为什么重要
尽管顶尖AI模型已经以开源权重的形式发布,但实际运行它们此前几乎离不开数据中心级设备,这项工作证明人们已经拥有的笔记本或游戏台式机也能以实用速度运行这些模型。这说明AI的可及性越来越取决于服务软件的水平,而不仅仅是模型权重是否公开。
本文术语
- MoE(混合专家) · 模型内置多个专家模块,每次输入只激活其中一小部分的架构
- Prefill(预填充) · 模型一次性处理完整输入提示的阶段
- Decode(解码) · 模型逐个生成输出token的阶段
- LRU缓存 · 优先淘汰最久未被使用内容的缓存策略
- TTFT(首个token耗时) · 从发出请求到收到第一个回复token所需的时间
无法转载的图表
- Figure 1: FreeToken serves the models on the cost–capability Pareto frontier, at interactive speed on consumer hardware. (a) Blended API list price (9:1 input:output mix, following the token economics measured on real coding-agent traces (Zhu et al. 2026)) versus Code Arena Elo (LMArena 2026) for representative hosted models. Blue squares mark models FreeToken serves, tagged with the consumer GPU class that serves them; the frontier segment from DeepSeek-V4-Flash to GLM-5.2 is exactly this set. Kimi-K3 releases open weights but exceeds consumer memory (594 GB); Qwen3.5-35B stands in for its successor Qwen3.6-35B, which has no arena rating yet. (b) Mean decode throughput on real agentic workloads for the strongest model each hardware tier holds (coding agents on the first two tiers, a math agent on the third), against actively maintained edge engines. The dashed line marks the median decode speed of Codex in production traces (33 tok/s (Zhu et al. 2026)); × marks configurations an engine cannot serve.
- Figure 2: FreeToken overview. (1) Prefill: expert loading is double-buffered at full-layer granularity, streaming layer l+1 over PCIe while the GPU computes layer l; recurrent-state checkpoints are anchored at special-token boundaries, so a context edit resumes from the nearest surviving anchor and re-prefills only the new suffix. (2) Decode: most routed experts hit the shared LRU expert cache (here 8 of 12, following temporal locality). The m=4 misses are divided by q⋆=mBP/BH between cache fills over PCIe (one expert) and in-place CPU execution (three), using bandwidths profiled on the deployed machine; the GPU and CPU partial outputs merge exactly. The host-resident expert pool remains the source of truth throughout.
- Figure 3: End-to-end serving on the RTX 5090 across four workloads (1. AIME, 2. OpenCode+SWE, 3. Claude Code+SWE, 4.OpenClaw+Email/Cal) and two models (Qwen3.6-35B-A3B BF16 and DeepSeek-V4-Flash MXFP4). Top: decode TPS; bottom: mean TTFT (log scale). × marks configurations an engine cannot serve (Ollama and MoE-Infinity lack DSV4 support; MoE-Infinity provides no usable server for multi-turn agents).
- Figure 4: (a) Prefill TPS versus prompt length (RTX 5090, Qwen3.6-35B BF16), with and without FreeToken’s pipelined full-layer loading. (b) Decode-time expert miss rate versus cache size (as a percentage of the expert pool) under the three engines’ placement policies, replayed on identical routing traces; lines are means over W1–W4, bands the min–max range.
- Figure 5: Coding-agent decode TPS across consumer GPUs (SWE issues via the OpenCode harness), Qwen3.6-35B-A3B. 4060 laptop using NVFP4, the other Qwen3.6 columns BF16. The RTX PRO 6000 column is a separate demonstration: GLM-5.2 (753B-A40B, NVFP4) on the math workload; Ollama is not run there. × marks configurations an engine cannot serve.
论文原文摘要（英文）
Frontier open-weight models are increasingly available, but serving them still largely assumes datacenter infrastructure. We present FreeToken, an edge-native MoE serving system that treats a personal machine not as a small GPU, but as a unified, elastic inference platform. FreeToken co-designs the full serving stack, including model layout and loading, expert residency, CPU--GPU execution, agentic state reuse, and runtime memory management, around two realities of local AI: agent workloads continuously change their execution pattern, and edge hardware exposes heterogeneous resources whose balance differs from machine to machine. Rather than committing to a fixed offloading strategy, FreeToken continuously maps computation and model state onto the resources actually available. FreeToken supports more than 20 MoE models and real coding and tool-using agents across hardware ranging from an 8GB laptop GPU to a single workstation GPU. More importantly, it changes what these machines can practically serve, from a 35B model on a laptop to a 284B model on a gaming desktop and the 753B GLM-5.2 on a single workstation GPU. FreeToken turns open weights into deployable local software, making the machines users already own a practical platform for frontier-scale intelligence. We release the system at flashml.ai.
在 arXiv 阅读最新论文
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?让AI编程助手去修复真实科学软件,连最强的那个也有一半以上任务没做对
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving把稀疏注意力从论文原型变成能真正上线服务的加速方案
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM Agents让客服AI坐席不只是拦住一个危险动作,而是把整个流程走对
- EXIMO: VLM Guided Exploration of VLA Policies不用人工遥控演示,让会说话的AI来教机械臂做新家务
- EnvHarness: Awakening Static Worlds for Agent Learning不重新搭建训练环境,而是给现有环境套一层可插拔组件,针对每个智能体的具体弱点重新塑形
- Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the Model租用AI而非拥有AI的机构,安全监管能力只剩一半
- Beyond Imitation: Filtering On-Policy Distillation by Reasoning ProgressAI模仿老师模型学习时,会误伤本来推理正确的步骤,新方法专门过滤掉这种误伤
- PersonalBench: Measuring the Authorship Gap in LLM Personalization让AI模仿某人的文风,结果发现它始终摆脱不了自己的腔调