In contrast to routers motivated by controlling costs, several LLM router designs focus solely on improving quality of
responses [31, 45, 57, 58].
The LLM routers described thus far do not modify the queries or individual LLM responses. Other types of control planes
do. Ensemble approaches such as mixture-of-expert (MoE) [29, 30, 52, 56] architectures select a subset of underlying
models to apply to each token of a query and merge their responses. LLM synthesis [40] architectures operate similarly,
but route the entire query to a subset of underlying LLMs and merge their responses. These approaches reduce inference
costs by using fewer and/or less complex underlying models.
Applications of LLM routers. A key use case for LLM routers is to help LLM-based application reduce cost. Several
commercial routers, including Unify [12], Martian [5], NotDiamond [7], and others, offer this as a service. By replacing a
few lines of code, the application can send user queries to a router service, rather than directly to some LLM provider. The
service selects the optimal LLM and forwards the queries. Commercial router services claim that this results in significant
cost savings: up to 98% in the case of Martian [5], and 10× in the case of NotDiamond [7].
3 LLM Control Plane Integrity
In this section, we define LLM control plane integrity. Informally, it means that decisions made about underlying LLM
queries made by the control plane algorithms cannot be subverted by adversarial queries. Looking ahead, we will focus
on one class of control plane: predictive LLM routing as used to manage cost.
Formalizing control planes. An LLM control plane Rω is a potentially randomized algorithm. It is parameterized by
a string ω, called the parameters. It utilizes some number n of LLMs denoted by M. We will mostly focus on the
case of n = 2, and, for reasons that will be clear in a moment, use Ms (“strong”) and Mw (“weak”) to denote the two
underlying LLMs. Then inference on an input x ∈ Xfor some set X of allowed queries is performed by computing
a response via y ←$ RM
ω (x). Here we use ←$ to denote running R with fresh random coins; we use ← when R is
deterministic. We focus on inference for a single query, but it is straightforward to extend our abstraction for control
planes to include sessions: the controller would maintain state across invocations, potentially adapting its behavior as a
function of a sequence of queries and responses.
LLM control planes should, in general, be relatively computationally lightweight, at least compared to the underlying
LLMs. This is particularly so in the cost-motivated usage of control planes, as a computationally or financially expensive
control plane would eat into cost savings incurred by utilizing cheaper underlying LLMs for some queries. For example,
predictive binary routers use relatively simple classifiers to determine which of Ms or Mw should be used to respond to a
query.
Inference flow. Given a set of LLMs M, a control plane Rω, and an input x, an LLM inference flow is the sequence of
LLM invocations Mij (zj) for 1 ≤ j ≤ m and ij ∈ {w, s} made when executing RM
ω (x). Here m is the total number of
LLM invocations, and z1, . . . , zm are the queries made to the underlying LLMs. Should R be randomized, the sequence
and its length are random variables. An inference flow can be written as a transcript
T = (i1, z1), (i2, z2), . . . ,(im, zm)
of pairs of model indexes ij ∈ {w, s} and model inputs zj. Note that for simplicity we ignore the potential for paral-
lelization, assuming execution proceeds serially. For binary routers, we have m = 1 and T ∈ {(w, x), (s, x)}. We write
submitting a sequence of inferences ⃗ x= ⃗ x1, . . . , ⃗ xq to a control plane as
RM
ω (⃗ x) = (RM
ω (⃗ x1), . . . , RM
ω (⃗ xq))
where note that each invocation could result in multiple underlying LLM invocations. In the binary router case, however,
each invocation results in a single LLM invocation.