REROUTING LLM R OUTERS
A PREPRINT
Avital Shafran
The Hebrew University
of Jerusalem
Roei Schuster
Wild Moose
Thomas Ristenpart
Cornell Tech
Vitaly Shmatikov
Cornell Tech
ABSTRACT
LLM routers aim to balance quality and cost of generation by classifying queries and routing them to
a cheaper or more expensive LLM depending on their complexity. Routers represent one type of what
we call LLM control planes: systems that orchestrate use of one or more LLMs. In this paper, we
investigate routers’ adversarial robustness.
We first define LLM control plane integrity, i.e., robustness of LLM orchestration to adversarial in-
puts, as a distinct problem in AI safety. Next, we demonstrate that an adversary can generate query-
independent token sequences we call “confounder gadgets” that, when added to any query, cause LLM
routers to send the query to a strong LLM.
Our quantitative evaluation shows that this attack is successful both in white-box and black-box settings
against a variety of open-source and commercial routers, and that confounding queries do not affect
the quality of LLM responses. Finally, we demonstrate that gadgets can be effective while maintaining
low perplexity, thus perplexity-based filtering is not an effective defense. We finish by investigating
alternative defenses.
1 Introduction
Large language models (LLMs) exhibit remarkable capabilities on many tasks. Today, hundreds of open-source and
proprietary LLMs are available at different prices, ranging from expensive, state-of-the-art models to cheaper, smaller,
less capable ones. LLM operators typically provide API access to their models (especially higher-quality models) on a
pay-per-query basis. This imposes non-trivial costs on LLM-based applications and systems.
Developers who want to integrate LLMs into their applications must therefore consider both utility and cost. They want
to maximize the quality of responses to their queries while minimizing the cost. The two objectives conflict with each
other: larger models tend to generate higher-quality answers but charge more per query. For example, at the time of
this writing, GPT-3.5-turbo costs $0.5/$1.5 per 1M input/output tokens, GPT-4o-mini $0.15/$0.6, GPT-4o $2.5/$10,
o1-preview $15/$60. The difference in quality between models is not uniform across queries. For some queries, even a
cheap model can generate an acceptable response. More complex queries require an expensive model to obtain a quality
answer.
A natural solution to balancing performance and economic considerations is to take advantage of the availability of mul-
tiple LLMs at different price-performance points. Recently proposed LLM routingsystems [5, 12, 27, 47, 53] orchestrate
two or more LLMs and adaptively route each query to the cheapest LLM they deem likely to generate a response of
sufficient quality. In the two-LLM case, let Ms be an expensive, high-quality model and Mw a weaker, lower-grade one.
Given query q, the routing algorithm R(·) applies a classifier to q that outputs 0 if Mw is sufficient for answering q, or 1
if Ms is required. The system then routes q accordingly.
LLM routing is an example of a general class of systems we call LLM control planes, which orchestrate the use of multiple
LLMs to process inputs, as further described in Section 2.
Our contributions. First, we introduce LLM control plane integrityas a novel problem in AI safety. Recently proposed
LLM control-plane algorithms are learned, calibrated classifiers (see Section 2). Their inputs are queries from potentially
adversarial users. Robustness of control-plane algorithms to adversarial queries is a new problem, distinct from adversarial
robustness of the underlying LLMs.
arXiv:2501.01818v1  [cs.CR]  3 Jan 2025