Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox›
LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers
arXiv:2608.068672026-08-06
A shared toolkit that lets researchers fairly compare AI systems that decide which language model should answer each question
Systems that route each query to the cheapest suitable large language model (LLM) have been built in incompatible ways, making them hard to compare. This paper unifies LLM routing into one framework with five components and builds an open benchmark (xRouteBench) and open-source infrastructure (LLMRouter) with more than 16 routers evaluated under the same protocol. Learned routers beat the strongest single fixed model by 14.6% on average, and lightweight routers become more competitive when cost budgets tighten.
METAL MEDIA explanatory visual
LLMRouter's unified pipeline from query to answer
Evidence statusMeasured results reported
Routing state inputA query, optional user context, and the conversation history so far are combined into the state the router observes.
Five components define any routerContext encoder, model encoder, scoring function, decision rule, and learning signal are the five interchangeable parts used to describe every router type.
Automated data pipelineQueries are curated, sent to all 18 candidate models, and scored for both quality and token cost, producing a query-model matrix used for both training supervision and testing.
xRouteBench evaluation across five tracksGeneric LLM tasks, memory, vision, time-series, and personalized tracks compare more than 16 routers on the same candidate pool and metrics, balancing performance against cost.
Real-world deployment checksLive Slack user preference collection and five multi-agent collaboration topologies test whether benchmark results hold up outside simulation.
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.
What they did
Existing LLM routers were built with incompatible formulations and codebases, making fair comparison hard; this work unifies routing as a sequential decision process defined by five components: context encoder, model encoder, scoring function, decision rule, and learning signal.
Using this unified formulation, the authors built an automated pipeline that runs every candidate model on every benchmark query to construct routing supervision jointly scoring answer quality and inference cost, producing the xRouteBench benchmark (4,767 test instances across five tracks: generic LLM tasks, memory, vision, time-series, and personalized routing).
They released LLMRouter, an open-source infrastructure implementing more than 16 representative routers behind one shared interface, supporting deployment as an OpenAI-compatible server on messaging platforms and a code-free ComfyUI visual interface.
Across 18 candidate LLMs (7B to 671B parameters, from two API providers) and the five xRouteBench tracks, learned routers outperformed the strongest fixed-model baseline by 14.6% relatively, lightweight routers became more competitive under tight cost constraints, and user-conditioned (personalized) routing consistently improved matching to individual preferences.
The team also tested routers on 234 real pairwise preference records from 15 Slack users and on five multi-agent collaboration topologies, finding that offline benchmark rankings did not always transfer to real users.
Figure 1: Overview of LLM routing. Routing is driven by three needs (left), namely cost efficiency, capability matching, and user preference. Our unified formulation (right) casts all of them as one decision process: a context encoder Eq represents the routing state of query, persona, and interaction history, a model encoder Em represents each candidate, and the router dispatches the query or its sub-queries to selected models and aggregates their responses into the answer. The single-turn, multi-turn, and personalized families differ only in which part of the state they observe.
Table 1: Instantiation of the unified routing formulation for the three router families. For each family, the table specifies the routing state s, the context and model encoders Eq and Em, the routing action defined by the scoring function g and decision rule d, and the learning signal ℒ used to optimize response quality and inference cost.
Family
State s
Encoders Eq,Em
Routing action (scoring g, decision d)
Learning signal ℒ (surrogate of Eq. 1)
Single-turn
(q)
Eq(q),Em(m)
a=argmaxm∈ℳg(Eq(q),Em(m))
fit g to per-candidate reward perf(ym∣q)−λcm
Multi-turn
(q,ht)
Eq(q,ht),Em(m)
at∼d({g(Eq(q,ht),Em(m))}m)
maximize episode return 𝔼τ[perf(y∣q)−λc(τ)]
Personalized
(q,u,ht)
Eq(q,u,ht),Em(m)
a=argmaxm∈ℳg(Eq(q,u,ht),Em(m))
fit g to comparisons m+≻um− observing perfu
Figure 2: Task composition of xRouteBench. The benchmark covers generic LLM tasks, memory, vision, time-series, and personalized routing, with percentages indicating the proportion of test queries contributed by each dataset.
Table 2: Results on xRouteBench under the performance-first setting (α,β)=(1.0,0.0). Scores are reported across the Generic LLM Tasks, memory, vision, and time-series tracks, together with their average. Following the original implementations where applicable, all multi-turn routers use Qwen2.5-3B-Instruct as the base model. Top two results are highlighted in bold and underline.
Router
Generic LLM Tasks
Memory
Vision
TimeSeries
Avg
LoCoMo
LongMemEval
Geometry3K
MathVista
Video
Rule-based baselines
Smallest-LLM
57.55
25.44
36.77
27.87
35.00
33.33
49.61
37.94
Largest-LLM
70.29
26.59
35.57
37.70
33.00
22.22
45.67
38.72
Single-turn routers
kNNRouter
71.37
25.24
38.74
31.15
41.00
29.63
51.97
41.30
SVMRouter
74.21
27.64
38.68
42.62
47.00
29.63
55.91
45.10
MLPRouter
68.12
26.78
32.27
27.87
34.00
29.63
56.69
39.34
MFRouter
67.23
24.49
34.91
40.98
29.00
22.22
51.97
38.69
EloRouter
64.15
25.70
37.27
45.90
50.00
25.93
63.78
44.68
Hybrid LLM
64.68
25.89
36.56
32.79
37.00
33.33
51.18
40.20
RouterDC
80.56
24.93
36.77
16.39
24.00
25.93
45.67
36.32
GraphRouter
80.54
25.94
33.93
42.62
50.00
22.22
62.99
45.46
CausalLM
66.90
25.40
37.60
24.60
34.00
33.33
45.70
38.22
Multi-turn routers
Router-R1
35.64
24.60
17.28
14.75
18.00
22.22
23.62
22.30
kNN-MultiRound
13.99
24.70
18.32
16.39
30.00
25.93
33.07
23.20
LLM-MultiRound
12.98
24.60
17.44
14.29
31.03
25.93
30.33
22.37
Figure 3: Architecture of LLMRouter. The system consists of six modules that support routing data construction, router implementation and training, inference, evaluation, and deployment.
Table 3: Performance comparison on the personalized track. Top two results are highlighted in bold and underline.
Router
Acc.
Router
Acc.
GMTRouter
68.78
RouterDC
56.44
PersonalizedRouter
67.86
MFRouter
54.39
EloRouter
66.40
MLPRouter
52.93
GraphRouter
65.23
kNNRouter
51.76
SVMRouter
65.08
CausalLM
46.78
Largest-LLM
58.05
Router-R1
45.46
Hybrid LLM
57.91
Smallest-LLM
42.53
Figure 5: Router rankings across the Generic LLM Tasks, memory, vision, and time-series tracks as the cost weight β increases. Each cell gives a router’s rank under the weighted performance–cost objective, with smaller rank values indicating better performance.
Table 4: Router performance on held-out real-user sessions collected through the Slack deployment. Accuracy measures how often each router’s model selection agrees with the users’ pairwise preferences.
Router
Acc.
Router
Acc.
PersonalizedRouter
83.05
RouterDC
65.25
EloRouter
82.20
kNNRouter
60.17
MLPRouter
78.81
kNN-MultiRound
60.17
SVMRouter
77.12
Smallest-LLM
55.08
Hybrid LLM
73.73
MFRouter
51.69
GMTRouter
70.70
Largest-LLM
41.53
GraphRouter
67.17
CausalLM
27.97
Figure 6: Performance–cost trade-offs of routers averaged across the xRouteBench tracks. Each point represents an operating setting with a different cost weight β, where higher performance and lower per-query inference cost are preferred.
Table 5: Router performance on the Generic LLM Tasks test split when each node in a multi-agent system is routed independently. Results are reported across five coordination topologies, with the final column showing the average performance.
Router
Star
Tree
Graph
Chain
Plan-Exec-Sum
Avg
Largest-LLM
69.00
67.00
77.20
69.00
75.20
71.48
kNNRouter
74.80
78.60
78.60
76.60
71.80
76.08
SVMRouter
76.20
75.60
80.00
74.40
75.20
76.28
MLPRouter
75.40
76.60
76.80
78.00
71.40
75.64
MFRouter
75.40
74.20
81.00
78.60
73.20
76.48
EloRouter
73.80
72.40
78.60
76.60
75.20
75.32
GraphRouter
68.20
70.80
66.20
72.00
69.00
69.24
RouterDC
77.60
79.60
74.20
72.00
76.20
75.92
Figure 7: Representative multi-agent system architectures and coordination topologies: (a) star-based centralized coordination, (b) hierarchical tree-based delegation, (c) graph-based peer interaction, (d) sequential chain collaboration, and (e) planner–executor–summarizer workflow.
Table 6: The eight test sets of xRouteBench. Sizes are the number of test queries; metrics are exact match (EM), multiple-choice accuracy (MC), token-level F1, execution-based code pass rate, math answer matching, and a persona-conditioned LLM judge.
Category
Test set
Content
#Test
Metric
Generic LLM Tasks
Generic mix
13 subtasks
3,729
EM/MC/F1/GSM8K/MATH/code
Memory
LoCoMo
long-conversation QA
314
F1
LongMemEval
long-term memory QA
101
F1
TimeSeries
TimeSeries
7 reasoning skills
127
MC
Vision
Geometry3K
geometry math (image)
61
EM
MathVista
visual math reasoning
100
EM/MC
Charades-Ego
egocentric video
27
EM
Personalized
Chatbot Arena / MT-Bench
preference prompts
308
LLM judge
Total
4,767
Figure 8: One example from each dataset in the Visual Reasoning track, shown with the description that Gemma-3-27B-IT produces for its image. Geometry3K (a) provides a geometry diagram, and MathVista (b) provides a scientific figure. Each description is appended to the problem text to form the query the router sees.
Table 7: Composition of the Generic LLM Tasks track. The table lists the 13 subtasks, their target skills, and the number of test queries, totaling 3,729 examples.
Subtask
Skill
#Test
MBPP
code generation
500
MATH
mathematical reasoning
500
GSM8K
mathematical reasoning
500
MMLU-Pro
knowledge QA
500
OpenBookQA
knowledge QA
500
ARC-Challenge
knowledge QA
500
MMLU
knowledge QA
500
CommonsenseQA
commonsense QA
50
BoolQ
commonsense QA
50
SQuAD
reading comprehension
50
HellaSwag
commonsense QA
50
HumanEval
code generation
16
AIME (2020–2024)
competition math
13
Figure 11: Slack interface for collecting pairwise user preferences. Two anonymized model responses are shown as Answer A and Answer B in randomized order, and users select A, B, or a tie for each interaction turn.
Table 8: Built-in routers in LLMRouter, grouped by routing family. For each method, the table summarizes the information available to the routing decision (State) and the rule used to select or aggregate candidate models (Selection).
Router
State
Selection
Rule-based baselines
Smallest-LLM
candidate parameter counts
always selects the smallest candidate
Largest-LLM
candidate parameter counts
always selects the largest candidate
Single-turn routers
kNNRouter
query embedding and nearby logged queries
votes over the models preferred by nearest neighbors
SVMRouter
query embedding
kernel classifier predicts a candidate
MLPRouter
query embedding
MLP classifier predicts a candidate
MFRouter
query and model latent factors
ranks candidates by their interaction score
EloRouter
logged pairwise model outcomes
always selects the highest-rated candidate
RouterDC
query and candidate representations
contrastive query–model matching score
Hybrid LLM
query embedding and a small/large model pair
predicts whether the small model is sufficient
AutoMix
small-model draft and verification signal
accepts the draft or escalates to the large model
GraphRouter
query–model interaction graph
predicts performance on query–model edges
CausalLM Router
textual query and candidate list
generates the selected model name
Multi-turn routers
Router-R1
query and accumulated search results
iteratively searches specialists or terminates and aggregates
kNN-MultiRound
sub-queries and their embeddings
routes each sub-query with kNN and aggregates the answers
LLM-MultiRound
textual query, decomposition, and candidate list
an LLM chooses routes for sub-queries and aggregates
Personalized routers
GMTRouter
user, session, query, model, and response interactions
predicts user-conditioned model preference
PersonalizedRouter
user features, task description, query, and model
predicts preference for a user–query pair
Figure 12: The ComfyUI interface of LLMRouter. The source benchmarks and the candidate pool enter at the left, the data-engine node produces the query–model matrix, and each router node consumes the matrix and reports its evaluation, so the graph traces the routing pipeline from data construction to evaluation. Each router node exposes the hyperparameters of its library configuration as typed widgets.
Table 9: The 18 candidate LLMs, sorted by blended average price. Prices in USD per 1M tokens.
#
Model
Params
Input
Output
Service
1
gemma-2-9b-it
9B
0.10
0.10
NVIDIA
2
llama-3-8b-instruct-lite
8B
0.10
0.10
Together
3
gpt-oss-20b
20B
0.05
0.20
Together
4
rnj-1-instruct
15B
0.15
0.15
Together
5
mistral-7b-instruct-v0.3
7B
0.20
0.20
NVIDIA
6
mistral-small-3-24b-instruct
24B
0.10
0.30
Together
7
qwen2.5-7b-instruct
7B
0.20
0.20
NVIDIA
8
qwen2.5-7b-instruct-turbo
7B
0.30
0.30
Together
9
gpt-oss-120b
120B
0.15
0.60
Together
10
llama-4-maverick
402B
0.27
0.85
Together
11
mixtral-8x7b-instruct-v0.1
46.7B
0.60
0.60
NVIDIA
12
qwen3-next-80b-a3b-instruct
80B
0.15
1.50
Together
13
qwen3-coder-next
200B
0.50
1.20
Together
14
llama-3.3-70b-instruct-turbo
70B
0.88
0.88
Together
15
llama3-70b-instruct
70B
0.90
0.90
NVIDIA
16
deepseek-v3.1
671B
0.60
1.70
Together
17
mixtral-8x22b-instruct-v0.1
140.6B
1.20
1.20
NVIDIA
18
cogito-v2-1-671b
671B
1.25
1.25
Together
Table 10: The five multi-agent topologies, their structures, and the number of LLM calls per query.
Topology
Structure
LLM calls
Star
planner decomposes → 3 actors in parallel → planner consolidates
Across 18 candidate LLMs and the five xRouteBench tracks, learned routers achieved a 14.6% relative improvement in performance over the strongest fixed-model baseline.
No single router dominated every task: RouterDC performed best on Generic LLM Tasks and SVMRouter on LoCoMo, while GraphRouter had the best average but was not the top performer in every individual task.
Multi-turn routing did not consistently outperform single-turn routing, with performance sensitive to the capability of the base model (Qwen2.5-3B-Instruct) used for decomposition and aggregation.
On the personalized track, GMTRouter reached 68.78 persona-judge accuracy, ahead of PersonalizedRouter (67.86) and the best user-agnostic router, EloRouter (66.40).
Router rankings shifted substantially as the cost weight increased (Figure 5/6), and always calling the largest model incurred the highest cost yet only mediocre performance compared to learned routers. On 234 real Slack preference records, PersonalizedRouter led with 83.05 accuracy while GMTRouter (top in simulation) dropped to sixth; across five multi-agent topologies, MFRouter achieved the best average score of 76.48 versus 71.48 for always using the largest model.
Where it can be used
Designing and tuning routing systems for services that operate multiple LLMs under cost and quality constraints
Developing new routing algorithms by reusing LLMRouter's shared data pipeline and evaluation tools via configuration changes instead of rebuilding infrastructure
Comparing routing performance across diverse input types such as long conversational history, images/video, and time-series data
Building personalized chat assistants that adapt model selection to individual user preferences
Assigning different models to different roles (planner, executor, summarizer) within multi-agent systems
Limits and open work
All multi-turn routers relied on the same base model (Qwen2.5-3B-Instruct) for decomposition and aggregation, so results may not generalize to other base models.
Rankings from the simulated persona-judge evaluation did not match rankings from real human preference data (GMTRouter dropped from first to sixth), indicating offline benchmark results may not directly transfer to deployment.
The candidate pool was limited to 18 open-weight models (7B to 671B) served through two specific API providers, so results on closed commercial models or different pricing structures were not tested.
The real-user study involved only 15 users, 40 sessions, and 234 pairwise records, a small sample for statistical generalization.
The Plan-Exec-Sum multi-agent topology reused the GraphPlanner design without retraining its planner, a simplification noted by the authors.
Why it matters
As AI products increasingly mix multiple LLMs to balance cost and quality, deciding which model handles which query becomes a core engineering problem, and this work gives practitioners a common yardstick and toolkit to build and compare routing strategies instead of reinventing evaluation pipelines each time.
Terms in this paper
LLM routing · Deciding, for each incoming query, which of several candidate language models should answer it
context encoder / model encoder · The part of a router that represents the query/conversation state (context encoder) and the part that represents each candidate model's characteristics (model encoder)
decision rule · The rule that turns computed scores into an actual choice of model or a decision to stop routing
xRouteBench · The benchmark built in this paper spanning generic QA, memory, vision, time-series, and personalized routing tasks
performance-cost trade-off · The balance between answer quality and the token/monetary cost spent to obtain it, tunable via a weighting parameter
Original abstract (English)
No single large language model (LLM) is optimal across all queries and budget constraints, making model routing essential for cost-effective deployment. Existing routers adopt diverse formulations and implementations, making fair comparison and extension difficult. We present a unified formulation of LLM routing as a sequential decision process characterized by five components: context encoders, model encoders, scoring functions, decision rules, and learning signals, covering single-turn, multi-turn, and personalized routing. Based on this formulation, we develop an automated pipeline for constructing routing supervision and evaluating routers jointly on response quality and inference cost. The resulting benchmark, xRouteBench, spans generic LLM, memory-augmented, vision, time-series, and personalized routing tasks. We further introduce LLMRouter, an open-source modular infrastructure with more than 16 representative routers. Our empirical study shows that learned routers outperform the strongest fixed-model baseline by 14.6% relatively, lightweight routers become more competitive under tight cost constraints, and user-conditioned routing consistently improves personalization.
