Figure 1: LLM routers classify queries and route complex ones to an expensive/strong model, others to a cheaper/weak
model. To control costs, LLM routers can be calibrated to maintain (for an expected workload) a specific ratio between
queries sent to the strong and weak models.
To initiate the study of this problem, we show that existing LLM routing algorithms are not adversarially robust. We
design, implement, and evaluate a method that generates query-independent adversarial token sequences we call “con-
founder gadgets.” If a gadget is added to any query, this query is routed to the strong model with high probability. Next,
we show that this attack is effective even in the transfer setting where the adversary does not have full knowledge of the
target LLM router (it is black-box), but has access to another router (e.g., an internally trained surrogate). We also evaluate
the integrity of commercial LLM routers, showing that they can be confounded as well.
Third, we investigate defenses. Our basic method generates gadgets that have anomalously high perplexity. Confounded
queries are thus easily distinguished from normal queries and can be filtered out by the routing system. Unfortunately, this
defense can be evaded by an adversary who incorporates a low-perplexity objective into the gadget generation algorithm,
producing gadgets that have low perplexity—and yet are effective at re-routing queries to the strong model. We also
discuss higher-level defenses, such as identifying users whose queries are routed to the strong model with abnormal
frequency.
Routing attacks can be deployed for various adversarial objectives, e.g., to ensure that the adversary always obtains the
highest-quality answer regardless of the target applications’s internal routing policies and cost constraints, or to mali-
ciously inflate the target’s LLM costs. As LLM control planes grow in importance and sophistication, we hope that this
work will motivate further research on their adversarial robustness.
2 LLM Control Planes and Routing
Inference using large language models (LLMs) is traditionally monolithic: a single model is applied to an input or se-
quence of inputs. This methodology can be sub-optimal for various reasons. State-of-the-art models are often expensive,
with API access to LLMs costing as much as several dollars for each query. Elsewhere, distinct LLMs may excel at dif-
ferent tasks, and selectively using them may improve overall quality on a diverse workload. Finally, combining multiple
LLMs, even all trained for similar tasks, may become increasingly prevalent as performance improvements of individual
LLMs plateaus [8–10].
Researchers and practitioners are therefore now developing inference architectures that use multiple LLMs to answer
queries. These LLMs are orchestrated by what we call an LLM control plane (borrowing the terminology from network-
ing [13]). The control plane may route queries or parts of queries to different LLMs, derive new strings to query to
underlying LLMs, combine answers from underlying LLMs, and more.
LLM routers. A prominent example of this emerging class of LLM control planes are LLM routers [27, 41, 47, 53, 59].
LLM routers decide which of the two (or, sometimes, more) LLMs to use to answer a query. In prescriptive routing,
the router applies some lightweight classifier to the input query that determines which underlying LLM to utilize for a
response. The classifier is itself a learned function that scores the complexity of the query. Deployments can then configure
a score threshold for when to route a query to the more expensive LLM. This threshold can be tuned using representative
workloads to achieve a desired cost-performance trade-off. Figure 1 shows the basic workflow of binary LLM routers.
Non-prescriptive routing [15, 20, 68] uses the responses from one or more underlying LLMs to determine which response