RSW RMF RCLS RLLM
Original Confounded Original Confounded Original Confounded Original Confounded
MT-Bench 9.2 9 .2 ± 0.0 9.1 9 .3 ± 0.0 9.2 9 .1 ± 0.0 8.9 9 .1 ± 0.1
MMLU 76 84 ± 1 76 81 ± 0 76 84 ± 0 78 84 ± 1
GSM8K 62 86 ± 0 65 88 ± 1 68 90 ± 2 66 85 ± 2
Table 10: Benchmark-specific average scores of responses to the original and confounded queries with GPT-4-1106-
preview as the strong model (LLM pair 4), in the white-box setting. Results demonstrate a higher increase in performance
with respect to the LLM pair 1 setting, due to the larger performance gap between the models.
confounder gadgets, queries originally routed to GPT are still routed to GPT and no queries are ever routed to Claude. For
queries originally routed to Llama, some gadgets result in all of them being rerouted to GPT, and some have no impact.
Specifically, 4 out of the 10 gadgets we optimized using RSW caused all queries to be rerouted to GPT,2/10 using RMF ,
and 3/10 using RLLM . None of the gadgets optimized using RCLS had any impact on routing. In terms of costs, having
all queries being rerouted to GPT results with an average cost of $0.25, a greater than 8× increase over the cost of the
original queries. Given the lack of documentation of the routing algorithm being used, we are unsure what explains the
variability across gadgets.
Martian. This router is supposed to let the user provide a list of models and to specify the maximum amount the user is
willing to pay for a query or for 1M tokens. Unfortunately, as of November 14, 2024, the router appears to ignore the list
models provided by the user, and forwards the input to the same LLM regardless of it. We tested this in settings including
one, two, or multiple models. While responses do not specify which LLM was used, they were identical across settings,
so we excluded Martian from our evaluation. We notified Martian about the seemingly buggy behavior.
8 Defenses
Defenses against rerouting should be cheap. If the per-query cost of the defense is comparable to the per-query cost of a
strong LLM, deploying the defense will defeat the main purpose of LLM routing, which is to reduce the cost of responding
to queries.
Perplexity-based filtering. As explained in Section 6, perplexity is a measure of how “natural” the text looks. Perplexity-
based filtering has been suggested in many contexts as a defense against adversarial text inputs [16, 36]. This defense
computes the perplexity of multiple “trusted” texts, then compares it with the perplexity of the suspicious text. If the latter
is significantly higher, or above some predefined threshold, the text is considered adversarial. Specifically, we assume the
defender has access to a set of unmodified queries. The defender computes their perplexity values and uses these values
to establish a threshold. Given a new query, the defender checks if its perplexity exceeds the threshold. If so, the query
is flagged as adversarial. The defender can then decide how to handle such queries. Options include rejecting them or
routing them all to the weak model. Computing the perplexity of a query can be cheap to do, e.g., using GPT-2 as we do
in this work; this makes it viable for use as a defense that doesn’t undermine the benefits of routing.
To evaluate the effectiveness of such a defense against our attack, we compare the perplexity values of original and
confounded queries. Figure 5 presents histograms of perplexity values for both the original evaluated GSM8K queries and
their corresponding confounded versions, generated using one of the confounder gadgets, sampled uniformly at random.
Additionally, the figure displays the ROC curve for the defense that detects confounded queries by checking if their
perplexity exceeds a threshold. As can be seen, the confounded queries exhibit significantly higher perplexity values,
making them readily distinguishable from the original queries. For instance, in the case of the RSW router, setting the