0 20 40 60
Iterations
0.220
0.225
0.230
0.235
0.240
0.245Routing score
Attack #0
Attack #1
Attack #2
Attack #3
Attack #4
Attack #5
Attack #6
Attack #7
Attack #8
Attack #9
(a) RSW
0 20 40 60
Iterations
0.2
0.4
0.6
0.8Routing score
Attack #0
Attack #1
Attack #2
Attack #3
Attack #4
Attack #5
Attack #6
Attack #7
Attack #8
Attack #9 (b) RMF
0 20 40 60
Iterations
0.5
0.6
0.7
0.8
0.9Routing score
Attack #0
Attack #1
Attack #2
Attack #3
Attack #4
Attack #5
Attack #6
Attack #7
Attack #8
Attack #9 (c) RCLS
0 20 40 60
Iterations
0.4
0.5
0.6
0.7
0.8Routing score
Attack #0
Attack #1
Attack #2
Attack #3
Attack #4
Attack #5
Attack #6
Attack #7
Attack #8
Attack #9 (d) RLLM
Figure 4: Convergence of gadget generation against different routing algorithms.
RSW RMF RCLS RLLM
Upgrade Strong Upgrade Strong Upgrade Strong Upgrade Strong
MT-Bench 100 ± 0 81 → 100 ± 0 100 ± 0 58 → 100 ± 0 100 ± 0 67 → 100 ± 0 73 ± 5 57 → 88 ± 2
MMLU 90 ± 1 43 → 94 ± 1 78 ± 4 53 → 90 ± 2 100 ± 0 47 → 100 ± 0 95 ± 1 53 → 98 ± 1
GSM8K 98 ± 0 52 → 99 ± 0 100 ± 0 54 → 100 ± 0 100 ± 0 56 → 100 ± 0 94 ± 3 53 → 97 ± 1
Table 1: The white-box attack’s rerouting success rate. “Upgrade” is the percentage of “Weak” queries successfully
rerouted to the strong model by adding a confounder gadget. “Strong” shows the change in the percentage of queries that
are routed to the strong LLM Ms before and after our rerouting attack.
fastest (50 iterations on average), RMF the slowest (66 iterations on average). Interestingly, the score of RSW does not
increase much during optimization but is still sufficient for a successful attack.
Runtime varies significantly when generating gadgets against different routing methods. On a machine with one A40 GPU,
4 CPUs, and 180G RAM, a single iteration takes 36.9 s, 8.4 s, 0.8 s, and 6.9 s for the RSW , RMF , RCLS , and RLLM
routers, respectively. On average, it takes around 31 minutes to generate a gadget for theRSW router, 9 minutes forRMF ,
50s for RCLS , and 6 minutes for RLLM .
Rerouting success rates. Recall that our attack adds the same confounder gadget to all queries. We start by evaluating the
reroute success rates: What fraction of confounded queries does the target router send to the strong model Ms? We show
the results for our attack in Table 1. Focusing first on the columns labeled ‘Strong’, the notation “ X → Y ± Z” relays
that for unmodified queries, X% are routed to the strong model Ms, and that for the n = 10 confounders we achieved an
average of Y % of queries sent to Ms. The standard error is Z. Note that while calibration targets ϵ = 0.5, we see some
natural variance in X for the test sets.
To breakdown success further, we additionally report the upgrade rate, which focuses on the percentage of queries that
were (a) originally routed to the weak model, and (b) routed to the strong model after they were modified with the
confounder gadget. Because in our attacks few queries get “downgraded” (confounders cause them to be rerouted to the
weak model instead of strong), the upgrade rate dictates the success rate.
As can be seen, the gadgets reroute almost all weak queries to the strong model. In most cases we see 100% success, or
close to it. The worst case still achieves 88% rerouting success, boosting the fraction of queries sent to the strong LLM by
1.5x. Rerouting fails only for some queries that even after confounding are sent to the weak model: the fixed gadget did
not sufficiently increase the router’s estimate of those queries’ complexity. This is the only source of error for the attack:
no queries in these experiments got “downgraded”, i.e., a query that would otherwise be sent to Ms ends up rerouted to
Mw. This also means that adding the confounder to every single query does not have negative impact on rerouting efficacy.
We report standard error values for both the upgrade rates and the total percentage of queries routed to the strong model.