ω (⃗ xq))
where note that each invocation could result in multiple underlying LLM invocations. In the binary router case, however,
each invocation results in a single LLM invocation.
An inference flow policy dictates the control plane designer’s intention regarding use of the underlying models. For
example, an application may want to ensure that only a small fraction of queries go to the expensive model Ms. We can
define this as a predicate over a sequence of transcripts. In our binary router example, the policy can be more simply
defined as a predicate P over (input, model) pairs (⃗ x1, i1), . . . ,(⃗ xq, iq) since this fully defines the sequence of transcripts.
For example, a policy might specify that the strong model is used in at most an ϵ fraction of inferences:
P((⃗ x1, i1), . . . ,(⃗ xq, iq)) =


qX
j=1
I(ij)
q ≤ ϵ


3