Surrogate ˆRSW ˆRMF ˆRCLS ˆRLLM
Target RMF RCLS RLLM RSW RCLS RLLM RSW SFM RLLM RSW RMF RCLS
MT-Bench 0.4 0 .8 0 .6 1.4 0 .7 0 .3 1.7 0 .3 0 .7 0.8 −0.6 0 .0
MMLU 0.1 0 .8 1 .1 0.2 0 .2 1 .1 0.3 0 .8 0 .9 1.3 1 .2 0 .9
GSM8K 1.9 1 .7 0 .6 1.6 1 .7 0 .2 1.7 1 .0 0 .4 1.3 1 .3 1 .7
Table 6: Differences between average perplexity of responses to the original and confounded queries, in the black-box
setting, when the confounder gadget was generated for a different surrogate router than the target, for LLM pair 1. Positive
values indicate a lower average perplexity (more natural) of responses to the confounded queries; higher values are better
for the attacker. Standard errors were omitted for readability but are0.2 on average. As in the white-box setting, the attack
does not increase the average response perplexity.
Surrogate ˆRSW ˆRMF ˆRCLS ˆRLLM
Target RMF RCLS RLLM RSW RCLS RLLM RSW SFM RLLM RSW RMF RCLS
LLM pair 1
MT-Bench −0.1 −0.1 0 .0 −0.1 −0.1 0 .0 −0.1 0 .0 0 .1 −0.2 −0.1 −0.2
MMLU −0.1 0 .3 −0.2 4.8 1 .0 0 .5 2.5 −1.3 −0.8 2.6 −0.9 0 .3
GSM8K 14.9 9 .6 15 .2 18.6 13 .8 14 .7 13.4 6 .8 12 .6 13.6 11 .3 10 .4
LLM pair 2
MT-Bench −0.1 −0.1 −0.1 −0.2 −0.2 −0.2 −0.1 −0.1 0 .0 −0.2 −0.2 −0.2
MMLU 1.6 4 .0 4 .2 7.9 5 .0 4 .4 5.0 −2.9 3 .2 5.2 −0.9 3 .8
GSM8K 13.6 8 .7 18 .5 18.9 14 .4 18 .3 13.1 4 .0 15 .5 11.3 8 .4 10 .8
LLM pair 3
MT-Bench 0.2 0 .0 0 .1 −0.1 −0.1 0 .0 0.0 0 .2 0 .2 −0.1 0 .1 −0.1
MMLU 5.0 6 .8 5 .8 11.3 9 .1 4 .7 8.1 −3.7 4 .8 7.8 0 .1 7 .2
GSM8K 20.5 13 .4 20 .9 24.3 18 .6 21 .6 17.9 11 .2 18 .9 16.7 15 .2 14 .2
Table 7: Differences between average benchmark specific scores of responses to the original and confounded queries,
when the confounder gadget was generated for a different surrogate router than the target (black-box setting) for three
LLM pairs. Positive values indicate a higher average score for responses to the confounded queries; higher values are
better for the attacker. Results are averaged across gadgets. Standard errors were omitted for readability and are on
average 0.1, 0.8, and 1.8 for MT-bench, MMLU and GSM8K, respectively. Aligned with the white-box setting, results
show almost no decrease in performance, and improvement when there is a performance gap for the LLM pair.
Results for LLM pair 4. As discussed in Section 5, we replace the strong model that was used by Ong et al. [47], GPT-4-
1106-preview (rank 28 in the Chatbot Arena leaderboard [1, 21]), with the open-sourced Llama-3.1-8B (rank 58) to reduce
the costs of our extensive set of evaluations. In this section we perform a smaller-scale evaluation of the quality-enhancing
attack performance when using GPT as the strong model, i.e., LLM pair 4. We evaluate this setting using three of the
n = 10 confounder gadgets for each router.
Table 10 shows the results across benchmarks in the white-box setting. Compared to the pair 1 setting (Table 3), the attack
results in a higher increase in benchmark performance. This further demonstrates higher attack effect on response quality
when the performance gap between the weak and strong models is higher.
7 Rerouting Commercial Routers
We evaluate our rerouting attack on several commercial routers: Unify [12], NotDiamond [7], OpenRouter [11], and
Martian [5]. These routers are available through black-box APIs. Therefore, we use our black-box attack with the
40 gadgets optimized for the open-sourced routers RSW , RMF , RCLS , and RLLM (10 per router). We perform this
evaluation using the MT-bench benchmark.
Unify. This router lets users specify a list of models from different providers and a metric configuration for routing
decisions. The available metrics are quality, time to first token, inter-token latency, and cost. The user can specify the
weight for each metric. Time, latency, and cost metrics are static and precomputed. The quality metric is computed for
12