LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

arXiv:2608.068672026-08-06

一套让研究者能公平比较各种大模型调度系统的统一平台

根据不同问题把请求分配给最合适、最省钱的大语言模型的系统各自为政,难以公平比较。这篇论文把这类调度统一成一个由五个组件构成的通用框架,并据此建立了开放基准xRouteBench和开源基础设施LLMRouter,在同一标准下评测了16种以上的调度器。结果显示,学习式调度器相比最强的单一固定模型平均相对提升14.6%,在成本预算收紧时轻量调度器更具竞争力。

METAL MEDIA 解读图

LLMRouter从问题到答案的统一流程

证据状态已报告实测结果

输入调度状态问题本身、可选的用户上下文,以及目前为止的对话历史共同构成调度器看到的状态。
五个组件定义任意调度器上下文编码器、模型编码器、打分函数、决策规则和学习信号,这五个可替换的部件描述了每一种调度器类型。
自动化数据流水线收集问题后发送给全部18个候选模型,同时记录回答质量和token成本,生成用于训练与测试的问题-模型矩阵。
xRouteBench五赛道评测在通用大模型任务、长期记忆、视觉、时间序列、个性化五个赛道上,用同一候选池和指标比较16种以上调度器的性能与成本表现。
真实场景部署验证通过Slack收集真实用户偏好数据,并在五种多智能体协作结构中测试,检验基准评测结果是否能在真实环境中站得住脚。

这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

现有的大模型调度系统各自采用不同的设计和代码实现,难以公平比较;本文把调度统一为一个由五个组件构成的序列决策过程:上下文编码器、模型编码器、打分函数、决策规则和学习信号。
基于这一统一框架,作者搭建了自动化数据流水线,让每个候选模型都在每个基准问题上运行,同时记录回答质量和推理成本,由此构建了xRouteBench基准(共4767条测试样本,覆盖通用任务、长期记忆、视觉、时间序列和个性化五个赛道)。
他们发布了开源基础设施LLMRouter,在统一接口下实现了16种以上代表性调度器,支持部署为兼容OpenAI接口的服务接入Slack等即时通讯平台,也提供无需写代码的ComfyUI可视化界面。
在18个候选大模型(参数量从7B到671B、来自两家API提供商)和xRouteBench五个赛道上的实验显示,学习式调度器相比最强的固定模型基线平均相对提升14.6%,成本预算越紧,轻量级调度器越有竞争力,基于用户信息的个性化调度也持续带来更好的个性化匹配效果。
研究团队还在Slack上收集了15名用户共234条真实成对偏好数据,并在五种多智能体协作结构中测试了调度器,发现离线基准的排名并不总能直接转移到真实用户场景。

Figure 1: Overview of LLM routing. Routing is driven by three needs (left), namely cost efficiency, capability matching, and user preference. Our unified formulation (right) casts all of them as one decision process: a context encoder Eq represents the routing state of query, persona, and interaction history, a model encoder Em represents each candidate, and the router dispatches the query or its sub-queries to selected models and aggregates their responses into the answer. The single-turn, multi-turn, and personalized families differ only in which part of the state they observe.

Table 1: Instantiation of the unified routing formulation for the three router families. For each family, the table specifies the routing state s, the context and model encoders Eq and Em, the routing action defined by the scoring function g and decision rule d, and the learning signal ℒ used to optimize response quality and inference cost.
Family	State s	Encoders Eq,Em	Routing action (scoring g, decision d)	Learning signal ℒ (surrogate of Eq. 1)
Single-turn	(q)	Eq(q),Em(m)	a=arg⁡maxm∈ℳ⁡g⁡(Eq(q),Em(m))	fit g to per-candidate reward perf⁡(ym∣q)−λcm
Multi-turn	(q,ht)	Eq(q,ht),Em(m)	at∼d⁡({g⁡(Eq(q,ht),Em(m))}m)	maximize episode return 𝔼τ[perf⁡(y∣q)−λc(τ)]
Personalized	(q,u,ht)	Eq(q,u,ht),Em(m)	a=arg⁡maxm∈ℳ⁡g⁡(Eq(q,u,ht),Em(m))	fit g to comparisons m+≻um− observing perfu

Figure 2: Task composition of xRouteBench. The benchmark covers generic LLM tasks, memory, vision, time-series, and personalized routing, with percentages indicating the proportion of test queries contributed by each dataset.

Table 2: Results on xRouteBench under the performance-first setting (α,β)=(1.0,0.0). Scores are reported across the Generic LLM Tasks, memory, vision, and time-series tracks, together with their average. Following the original implementations where applicable, all multi-turn routers use Qwen2.5-3B-Instruct as the base model. Top two results are highlighted in bold and underline.
Router	Generic LLM Tasks	Memory	Vision	TimeSeries	Avg
LoCoMo	LongMemEval	Geometry3K	MathVista	Video
Rule-based baselines
Smallest-LLM	57.55	25.44	36.77	27.87	35.00	33.33	49.61	37.94
Largest-LLM	70.29	26.59	35.57	37.70	33.00	22.22	45.67	38.72
Single-turn routers
kNNRouter	71.37	25.24	38.74	31.15	41.00	29.63	51.97	41.30
SVMRouter	74.21	27.64	38.68	42.62	47.00	29.63	55.91	45.10
MLPRouter	68.12	26.78	32.27	27.87	34.00	29.63	56.69	39.34
MFRouter	67.23	24.49	34.91	40.98	29.00	22.22	51.97	38.69
EloRouter	64.15	25.70	37.27	45.90	50.00	25.93	63.78	44.68
Hybrid LLM	64.68	25.89	36.56	32.79	37.00	33.33	51.18	40.20
RouterDC	80.56	24.93	36.77	16.39	24.00	25.93	45.67	36.32
GraphRouter	80.54	25.94	33.93	42.62	50.00	22.22	62.99	45.46
CausalLM	66.90	25.40	37.60	24.60	34.00	33.33	45.70	38.22
Multi-turn routers
Router-R1	35.64	24.60	17.28	14.75	18.00	22.22	23.62	22.30
kNN-MultiRound	13.99	24.70	18.32	16.39	30.00	25.93	33.07	23.20
LLM-MultiRound	12.98	24.60	17.44	14.29	31.03	25.93	30.33	22.37

Figure 3: Architecture of LLMRouter. The system consists of six modules that support routing data construction, router implementation and training, inference, evaluation, and deployment.

Table 3: Performance comparison on the personalized track. Top two results are highlighted in bold and underline.
Router	Acc.	Router	Acc.
GMTRouter	68.78	RouterDC	56.44
PersonalizedRouter	67.86	MFRouter	54.39
EloRouter	66.40	MLPRouter	52.93
GraphRouter	65.23	kNNRouter	51.76
SVMRouter	65.08	CausalLM	46.78
Largest-LLM	58.05	Router-R1	45.46
Hybrid LLM	57.91	Smallest-LLM	42.53

Figure 5: Router rankings across the Generic LLM Tasks, memory, vision, and time-series tracks as the cost weight β increases. Each cell gives a router’s rank under the weighted performance–cost objective, with smaller rank values indicating better performance.

Table 4: Router performance on held-out real-user sessions collected through the Slack deployment. Accuracy measures how often each router’s model selection agrees with the users’ pairwise preferences.
Router	Acc.	Router	Acc.
PersonalizedRouter	83.05	RouterDC	65.25
EloRouter	82.20	kNNRouter	60.17
MLPRouter	78.81	kNN-MultiRound	60.17
SVMRouter	77.12	Smallest-LLM	55.08
Hybrid LLM	73.73	MFRouter	51.69
GMTRouter	70.70	Largest-LLM	41.53
GraphRouter	67.17	CausalLM	27.97

Figure 6: Performance–cost trade-offs of routers averaged across the xRouteBench tracks. Each point represents an operating setting with a different cost weight β, where higher performance and lower per-query inference cost are preferred.

Table 5: Router performance on the Generic LLM Tasks test split when each node in a multi-agent system is routed independently. Results are reported across five coordination topologies, with the final column showing the average performance.
Router	Star	Tree	Graph	Chain	Plan-Exec-Sum	Avg
Largest-LLM	69.00	67.00	77.20	69.00	75.20	71.48
kNNRouter	74.80	78.60	78.60	76.60	71.80	76.08
SVMRouter	76.20	75.60	80.00	74.40	75.20	76.28
MLPRouter	75.40	76.60	76.80	78.00	71.40	75.64
MFRouter	75.40	74.20	81.00	78.60	73.20	76.48
EloRouter	73.80	72.40	78.60	76.60	75.20	75.32
GraphRouter	68.20	70.80	66.20	72.00	69.00	69.24
RouterDC	77.60	79.60	74.20	72.00	76.20	75.92

Figure 7: Representative multi-agent system architectures and coordination topologies: (a) star-based centralized coordination, (b) hierarchical tree-based delegation, (c) graph-based peer interaction, (d) sequential chain collaboration, and (e) planner–executor–summarizer workflow.

Table 6: The eight test sets of xRouteBench. Sizes are the number of test queries; metrics are exact match (EM), multiple-choice accuracy (MC), token-level F1, execution-based code pass rate, math answer matching, and a persona-conditioned LLM judge.
Category	Test set	Content	#Test	Metric
Generic LLM Tasks	Generic mix	13 subtasks	3,729	EM/MC/F1/GSM8K/MATH/code
Memory	LoCoMo	long-conversation QA	314	F1
LongMemEval	long-term memory QA	101	F1
TimeSeries	TimeSeries	7 reasoning skills	127	MC
Vision	Geometry3K	geometry math (image)	61	EM
MathVista	visual math reasoning	100	EM/MC
Charades-Ego	egocentric video	27	EM
Personalized	Chatbot Arena / MT-Bench	preference prompts	308	LLM judge
Total			4,767

Figure 8: One example from each dataset in the Visual Reasoning track, shown with the description that Gemma-3-27B-IT produces for its image. Geometry3K (a) provides a geometry diagram, and MathVista (b) provides a scientific figure. Each description is appended to the problem text to form the query the router sees.

Table 7: Composition of the Generic LLM Tasks track. The table lists the 13 subtasks, their target skills, and the number of test queries, totaling 3,729 examples.
Subtask	Skill	#Test
MBPP	code generation	500
MATH	mathematical reasoning	500
GSM8K	mathematical reasoning	500
MMLU-Pro	knowledge QA	500
OpenBookQA	knowledge QA	500
ARC-Challenge	knowledge QA	500
MMLU	knowledge QA	500
CommonsenseQA	commonsense QA	50
BoolQ	commonsense QA	50
SQuAD	reading comprehension	50
HellaSwag	commonsense QA	50
HumanEval	code generation	16
AIME (2020–2024)	competition math	13

Figure 11: Slack interface for collecting pairwise user preferences. Two anonymized model responses are shown as Answer A and Answer B in randomized order, and users select A, B, or a tie for each interaction turn.

Table 8: Built-in routers in LLMRouter, grouped by routing family. For each method, the table summarizes the information available to the routing decision (State) and the rule used to select or aggregate candidate models (Selection).
Router	State	Selection
Rule-based baselines
Smallest-LLM	candidate parameter counts	always selects the smallest candidate
Largest-LLM	candidate parameter counts	always selects the largest candidate
Single-turn routers
kNNRouter	query embedding and nearby logged queries	votes over the models preferred by nearest neighbors
SVMRouter	query embedding	kernel classifier predicts a candidate
MLPRouter	query embedding	MLP classifier predicts a candidate
MFRouter	query and model latent factors	ranks candidates by their interaction score
EloRouter	logged pairwise model outcomes	always selects the highest-rated candidate
RouterDC	query and candidate representations	contrastive query–model matching score
Hybrid LLM	query embedding and a small/large model pair	predicts whether the small model is sufficient
AutoMix	small-model draft and verification signal	accepts the draft or escalates to the large model
GraphRouter	query–model interaction graph	predicts performance on query–model edges
CausalLM Router	textual query and candidate list	generates the selected model name
Multi-turn routers
Router-R1	query and accumulated search results	iteratively searches specialists or terminates and aggregates
kNN-MultiRound	sub-queries and their embeddings	routes each sub-query with kNN and aggregates the answers
LLM-MultiRound	textual query, decomposition, and candidate list	an LLM chooses routes for sub-queries and aggregates
Personalized routers
GMTRouter	user, session, query, model, and response interactions	predicts user-conditioned model preference
PersonalizedRouter	user features, task description, query, and model	predicts preference for a user–query pair

Figure 12: The ComfyUI interface of LLMRouter. The source benchmarks and the candidate pool enter at the left, the data-engine node produces the query–model matrix, and each router node consumes the matrix and reports its evaluation, so the graph traces the routing pipeline from data construction to evaluation. Each router node exposes the hyperparameters of its library configuration as typed widgets.

Table 9: The 18 candidate LLMs, sorted by blended average price. Prices in USD per 1M tokens.
#	Model	Params	Input	Output	Service
1	gemma-2-9b-it	9B	0.10	0.10	NVIDIA
2	llama-3-8b-instruct-lite	8B	0.10	0.10	Together
3	gpt-oss-20b	20B	0.05	0.20	Together
4	rnj-1-instruct	15B	0.15	0.15	Together
5	mistral-7b-instruct-v0.3	7B	0.20	0.20	NVIDIA
6	mistral-small-3-24b-instruct	24B	0.10	0.30	Together
7	qwen2.5-7b-instruct	7B	0.20	0.20	NVIDIA
8	qwen2.5-7b-instruct-turbo	7B	0.30	0.30	Together
9	gpt-oss-120b	120B	0.15	0.60	Together
10	llama-4-maverick	402B	0.27	0.85	Together
11	mixtral-8x7b-instruct-v0.1	46.7B	0.60	0.60	NVIDIA
12	qwen3-next-80b-a3b-instruct	80B	0.15	1.50	Together
13	qwen3-coder-next	200B	0.50	1.20	Together
14	llama-3.3-70b-instruct-turbo	70B	0.88	0.88	Together
15	llama3-70b-instruct	70B	0.90	0.90	NVIDIA
16	deepseek-v3.1	671B	0.60	1.70	Together
17	mixtral-8x22b-instruct-v0.1	140.6B	1.20	1.20	NVIDIA
18	cogito-v2-1-671b	671B	1.25	1.25	Together

Table 10: The five multi-agent topologies, their structures, and the number of LLM calls per query.
Topology	Structure	LLM calls
Star	planner decomposes → 3 actors in parallel → planner consolidates	6
Tree	root planner → 2 sub-planners refine → 2 actors → root consolidates	7
Graph	3 actors answer independently → one full-communication revision round	7
Chain	3 agents relay sequentially, each verifying and improving the previous answer	4
Plan-Exec-Sum	planner emits 3 atomic sub-queries → 3 executors → summarizer merges	6

研究结果

在18个候选大模型和xRouteBench五个赛道上,学习式调度器相比最强的固定模型基线,性能平均相对提升了14.6%。
没有任何一种调度器在所有任务上都占优:RouterDC在通用大模型任务上表现最好,SVMRouter在LoCoMo任务上最好,而GraphRouter平均表现最佳但并非每个具体任务的第一名。
多轮调度并没有持续优于单轮调度,其表现对用于分解与汇总的基础模型(Qwen2.5-3B-Instruct)的能力较为敏感。
在个性化赛道上,GMTRouter的人设评审准确率达到68.78,领先于PersonalizedRouter(67.86)以及不使用用户信息的最佳调度器EloRouter(66.40)。
随着成本权重增大(图5、图6),调度器排名发生显著变化;始终选用最大模型的方案成本最高但性能仅属中等,不敌学习式调度器。在234条真实Slack偏好数据上,PersonalizedRouter以83.05的准确率领先,而在模拟评测中排名第一的GMTRouter降至第六;在五种多智能体协作结构实验中,MFRouter取得76.48的平均成绩,优于始终使用最大模型的71.48。

可应用场景

为同时运营多个大语言模型、需要在成本与质量间权衡的服务设计和调优调度系统
在开发新的调度算法时,通过修改配置文件复用LLMRouter现成的数据流水线和评测工具,而无需重建整套基础设施
在长对话历史、图像/视频、时间序列等多种输入类型上比较不同调度方案的表现
构建能够根据用户个人偏好调整模型选择的个性化聊天助手
在多智能体系统中为不同角色节点(规划者、执行者、总结者等)分别选择最合适的模型

局限与待验证事项

所有多轮调度器都使用同一个基础模型(Qwen2.5-3B-Instruct)进行问题分解与结果汇总,尚未验证结论是否适用于其他基础模型。
基于人设评审的模拟评测排名与真实用户偏好数据的排名并不一致(GMTRouter从模拟第一降至真实场景第六),说明离线基准结果未必能直接迁移到实际部署环境。
候选模型池仅限于18个开源权重模型(7B到671B,来自两家特定API提供商),尚未在闭源商用模型或其他定价结构下验证结果。
真实用户实验规模较小,仅有15名用户、40个会话、234条成对偏好记录,统计代表性有限。
Plan-Exec-Sum多智能体拓扑结构沿用了GraphPlanner的设计但未重新训练其规划器,是实验设计上的一处简化。

为什么重要

随着AI产品越来越多地混合使用多个大语言模型来平衡成本与质量,决定每个问题该交给哪个模型处理成为核心工程问题,这项工作为从业者提供了一套统一的评测标准和工具,便于比较和构建适合自身场景的调度方案,而不必每次都重新搭建评测体系。

本文术语

大模型调度(LLM routing) · 针对每个输入问题,决定由候选的多个大语言模型中的哪一个来回答
上下文编码器 / 模型编码器 · 分别用来表示问题及对话状态(上下文编码器)和表示每个候选模型特点(模型编码器)的组件
决策规则 · 把打分结果转化为实际选择哪个模型、或何时停止调度的规则
xRouteBench · 本文构建的调度评测基准,涵盖通用问答、长期记忆、视觉、时间序列和个性化任务
性能-成本权衡 · 回答质量越高往往消耗的token或费用越多,需要在两者之间找到平衡点

论文原文摘要（英文）

No single large language model (LLM) is optimal across all queries and budget constraints, making model routing essential for cost-effective deployment. Existing routers adopt diverse formulations and implementations, making fair comparison and extension difficult. We present a unified formulation of LLM routing as a sequential decision process characterized by five components: context encoders, model encoders, scoring functions, decision rules, and learning signals, covering single-turn, multi-turn, and personalized routing. Based on this formulation, we develop an automated pipeline for constructing routing supervision and evaluating routers jointly on response quality and inference cost. The resulting benchmark, xRouteBench, spans generic LLM, memory-augmented, vision, time-series, and personalized routing tasks. We further introduce LLMRouter, an open-source modular infrastructure with more than 16 representative routers. Our empirical study shows that learned routers outperform the strongest fixed-model baseline by 14.6% relatively, lightweight routers become more competitive under tight cost constraints, and user-conditioned routing consistently improves personalization.

作者 · Tao Feng

在 arXiv 阅读

METAL MEDIA 最新报道

图片来源： Tao Feng et al., arXiv:2608.06867, CC BY 4.0