RSW RMF RCLS RLLM
MT-Bench 100 100 100 100
MMLU 100 96 100 100
GSM8K 100 100 100 100
Table 8: Upgrade rates for query-specific gadgets, in the white-box setting. Results are nearly perfect, i.e. nearly all
confounded queries are routed to the strong model.
Surrogate ˆRSW ˆRMF ˆRCLS ˆRLLM
Target RMF RCLS RLLM RSW RCLS RLLM RSW SFM RLLM RSW RMF RCLS
MT-Bench 100 83 71 100 83 48 100 73 52 100 67 83
MMLU 96 57 89 95 43 83 74 13 83 77 11 30
GSM8K 100 68 74 100 73 68 81 65 70 88 54 64
Table 9: Upgrade rates for query-specific gadgets, in the black-box setting. In most cases results are better than in the
query-independent setting, at the cost of a more resource intensive process.
each query using a neural scoring function that was trained on prompts from several open datasets (e.g., Open Hermes [62])
and labeled using an LLM-as-a-judge [71].
For our evaluation, we configure the router to choose between GPT-4o [2] as the strong model and Mixtral 8x7B [39] as
the weak model. We focus on the cost and quality metrics, and set the weight of time and latency to 0 so that they are
not factored into routing decisions. We manually calibrate the weights to 1 for the quality metric and 0.02 for the cost
metric. These weights result in 49% of the original, unmodified queries being routed to the strong model and 51% to the
weak model, resulting in a total cost of $0.13 for the 72 MT-bench queries. Adding confounder gadgets generated for the
four open-sourced evaluated routers results in upgrade rates of 79%, 88%, 91%, and 89%, respectively, averaged across
10 gadgets. The downgrade rate is zero in all cases. In terms of costs, the addition of the confounder gadget increased
the cost to $0.22, $0.23, $0.22, and $0.21, respectively, averaged across 10 gadgets. In other words, the rerouting attack
increased the cost of processing the queries, on average, by a factor of 1.7×.
NotDiamond. This router lets users route their queries to a list of predefined models. Available objectives are to maximize
quality, or balance quality and cost, or balance quality and latency. The exact details of the routing logic are not specified.
We focus on cost-aware routing, for which the API docs state that “NotDiamond will automatically determine when a
query is simple enough to use a cheaper model without degrading the quality of the response.” NotDiamond provides a
router selection tool which gives the routing decision for a particular query without forwarding the query to the chosen
model (thereby incurring no costs). We use this for our evaluation—of course a real attack would target the NotDiamond
API when used for actual routing.
Similar to the Unify experiments, we set GPT-4o as the strong model and Mixtral-8x7b as the weak model. Cost-aware
routing routes 82% of the original queries to the strong model,18% to the weak model. Confounded queries generated for
RSW , RMF , RCLS , and RLLM achieve upgrade rates of 21%, 18%, 21%, and 15%, respectively. The downgrade rates
are 1–3%.
As opposed to our calibrated routers, NotDiamond aggressively routes to the stronger model even for unmodified queries
in most settings. We tried several strong/weak model pairs including GPT-4o/Mistral-7B-Instruct-v0.2, GPT-4o/GPT-4o-
mini, and Claude-3-Opus/Claude-3-Sonnet, and observed a similar 20%–80% split between strong and weak.
When we changed the strong model to OpenAI’s o1-mini and kept Mixtral-8x7b as the weak model, 54% of the original
queries were routed to the strong model, 46% to the weak model. In this setting, confounder gadgets yield 13–16%
upgrade rates and, on average, 3–6% downgrade rates. We conclude that while the attack is still effective, NotDiamond is
more robust than Unify.
OpenRouter. This framework offers a unified interface for LLMs, and additionally offers a system that routes users’
queries between three specific models: Llama-3-70b, Claude-3.5-Sonnet, and GPT-4o. Queries are routed “depending on