Routers Notation
Similarity-weighted ranking RSW
Matrix factorization RMF
BERT classifier RCLS
LLM scoring RLLM
LLM pair Strong (Ms) Weak (Mw)
1 Llama-3.1-8B 4-bit Mixtral 8x7B
2 Llama-3.1-8B Mistral-7B-Instruct-v0.3
3 Llama-3.1-8B Llama-2-7B-chat-hf
4 GPT-4-1106-preview 4-bit Mixtral 8x7B
Benchmark Description
MT-Bench [71] 160 open-ended questions
MMLU [35] 14,042 multi-choice questions
GSM8K [24] 1,319 grade-school math problems
Figure 3: Summary of our setup for routers, underlying LLMs, and benchmark datasets used in the experiments.
In all experiments, we assume that the adversary’s goal is to reroute queries to the strong model. In Appendix E, we
evaluate efficacy of the attack when the goal is to reroute to the weak model.
Target routers. We focus our evaluation on the four prescriptive routing algorithms proposed by Ong et al. [47],
which provides open-source code and trained parameters, and does so for a representative variety of routing ap-
proaches: similarity-based classification [41, 59], an MLP constructed via matrix factorization [59], BERT-based clas-
sification [27, 53, 59], and a fine-tuned LLM.
The routers we evaluate were trained in a supervised fashion using a set of reference (training) queries whose performance
score on each of the considered models is known. The scores were computed from a collection of human pairwise rankings
of model answers for each of the queries. We note that while the routers we consider are all learned using this training
set, there is no reason to believe a non-learning-based approach (e.g., rule based) to routing would be more adversarially
robust.
We now outline the routing methods considered in this work. See Ong et al. [47] for their full implementation details.
Similarity-weighted ranking: The first method is based on the Bradley-Terry (BT) model [17]. For a given user query,
this model derives a function to compute the probability of the weak model being preferred over the strong model. The
probability-function expressions all share parameters, which are optimized to minimize the sum of cross-entropy losses
over the training-set queries, where each element in the sum is weighted by the respective query’s similarity with the
user’s query (computed as embeddings cosine similarity, with the embedding derived using OpenAI’s text-embedding-3-
small [6]). We denote this method as RSW .
Matrix factorization: The second method is based on matrix factorization. The training queries are used to train a bilinear
function mapping a model’s embedding and a query’s embedding to a score corresponding to how well the model performs
on the query. Routing is done by computing the score of the input query for each model, and choosing the highest-scoring
model. We denote this method as RMF .
BERT classifier: The third method involves fine-tuning a classifier, based on the BERT-base architecture [26], to predict
which of the two models produces a better response for the given query or whether they do equally well (a tie). The
routing decision is based on the probability of the weak model providing a better response versus the strong model or the
tie. We denote this method as RCLS .
LLM classifier: The last method is based on asking an LLM to provide a score in the range 1–5 of how an AI expert
would struggle to respond to a given query based on the query’s complexity. For this, Ong et al. fine-tuned a Llama-3-8B
model [4] using their reference set of queries and corresponding scores. We denote this method as RLLM .
Underlying LLMs. In [47], Ong et al. trained the routers with GPT-4-1106-preview [14] as the strong model and Mixtral
8x7B [39] as the weak model. They report successful generalization between the underlying LLMs, stating that their
routers trained for a particular strong-weak LLM pair can be used with other strong-weak LLM pairs.
To allow our evaluation to scale, we use as the strong model Ms the open-sourced Llama-3.1-8B [3] and as Mw the