AI orchestration engine: routing every request to the right model

An enterprise AI platform never runs on a single model for very long: it lines up an inexpensive one that answers quickly, a slower one that reasons, a third specialised in code, and sometimes an external API for whatever is not confidential. What decides, request by request, is called the orchestration engine : and that is where cost, latency and confidentiality are settled.

By QDNA · Published August 28, 2026 · 10 min read

Why a single model is a poor default

The first version of an enterprise AI application always calls one model, through a client library, with a key in an environment variable. This approach works well enough at first, and it quietly contracts three debts the organisation pays later.

A cost debt settles in first, because the distribution of incoming requests is heavily skewed. Rewriting a sentence, classifying a document, extracting a field from a form or summarising a paragraph make up most of the volume without calling for any long-reasoning capability, so that sending them to the most expensive model amounts to paying reasoning prices for operations that do not reason.

A latency debt comes next, since reasoning models emit thinking tokens before producing their answer. On a classification task that preamble is wasted time, and it shows: an interface that takes eight seconds to tick a box reads as broken, whatever the accuracy of the result.

A confidentiality debt comes last, and of the three it is by far the most expensive to settle. When the call is hard-wired, there is nowhere in the code to write "this document does not leave". The decision was taken once, at authoring time, for every request to come : including the ones nobody imagined.

The gain is not merely theoretical, and it has been measured since 2024 on public benchmarks. The authors of RouteLLM, published at ICLR 2025 by a team from Berkeley and Anyscale, train a router on preference data to arbitrate between a strong and a weak model, and report a cost reduction of more than a factor of two in certain cases without compromising response quality. RouterBench, which gathers over four hundred thousand inference outcomes, has since provided a common evaluation framework for this line of work.

Sources: Ong et al., RouteLLM: Learning to Route LLMs with Preference Data, and RouterBench. These results cover specific model pairs and evaluation sets: they establish that the technique works, not the gain you will obtain on your own traffic, whose distribution is yours alone.

The three parts of an orchestration engine

The architecture that answers those three debts has three successive layers, each fulfilling one function and one only, which keeps the system legible for whoever has to audit it.

1. The classifier

The classifier answers a single question, that of the nature of the request, and two families of answer coexist to do so, which a mature platform uses together rather than choosing between them.

Rule-based routing decides on explicit criteria: caller identity, originating application, context length, a pattern in the text, a classification label carried by a document. It is deterministic, readable and auditable : three properties that matter when a data protection officer asks why a given request went where it went.

Semantic routing computes a vector of the request with an embedding model and compares it to reference examples. It recognises an intent no rule anticipated: "summarise this" and "give me a three-point synthesis" land in the same place without anyone listing the phrasings. The price is that it is sometimes wrong, and you have to decide what happens when confidence is low.

The combination that survives production: rules first, as a circuit breaker, for everything that must remain deterministic, then semantics second to refine the choice among the destinations the rules allowed, and never in the reverse order.

2. The routing policy

The routing policy turns the qualification into a destination, and four criteria cover most situations provided one sees that they are not of the same kind: the first is a constraint, the other three are optimisations.

Order matters. Sensitivity is evaluated first and eliminates destinations; the other three choose among what is left. A policy that optimises cost before evaluating sensitivity will, one day, send a contract to an API because it was cheaper.

3. The gateway

The third layer decides nothing at all, since it merely executes and records, which is the role of an LLM gateway, exposing a single interface to applications while speaking each runtime's own protocol.

• Virtual keys and quotas: each team, each application gets its key with its budget. An agent loop that spirals burns its own quota, not production's.
• Failover: if a runtime does not answer, the request goes to a backup destination : provided the sensitivity policy allows it. This is the most common breaking point; more on it below.
• Logging: who asked what, to which model, at what cost. Without that trace, none of the earlier decisions is demonstrable.
• Stable aliases: applications call a role name, not a model name. Swapping the model behind an alias then requires no application deployment.

Runtimes, at the end of the chain, serve the model: vLLM for high-concurrency serving, llama.cpp for a workstation, an external API for whatever is not sensitive. The orchestration engine does not replace any of those runtimes, it simply chooses between them.

The three failures you actually see

An orchestration engine rarely fails loudly: it fails by continuing to answer, which makes the failure considerably harder to detect than a clean outage would be.

Failover that betrays the policy is at once the most serious of these failures and the most ordinary. The local runtime goes down, the gateway falls back to the backup API exactly as configured, and confidential documents go to a third party for the whole duration of the incident. Nobody notices anything at all, for the simple reason that the service itself was never interrupted. The rule that prevents it is easy to state and easy to forget: on a flow marked sensitive, an outage must produce an error, never a fallback outside the perimeter.

The cache that crosses partitions, a gateway that caches responses is much faster and cheaper. If the cache key does not include caller identity and clearance scope, a response computed for an authorised user will be served again to one who was not. The cache must therefore be partitioned along the same boundaries as the data it holds.

The classifier that drifts, the reference examples of a semantic router were chosen at one point in time, on the usage of that moment. Six months later users ask different questions, confidence scores fall, and the router sends more and more requests to its default destination : often the most expensive one. You have to measure the distribution of decisions over time, not only their accuracy on day one.

Routing is where sovereignty is enforced

This argument justifies the investment on its own, and it carries further than any reasoning based on cost, whose gain remains circumstantial.

A confidentiality policy, a contractual commitment, an impact assessment: all of these describe what should happen to data. The router is the only place in the system where that description becomes an executed decision, for every request, and logged. Until a rule is implemented there, it remains an intention, and nobody can demonstrate it was honoured.

That has a practical architectural consequence: the sensitivity classifier must be local, including when the request will end up at an external provider. Asking an API to evaluate "is this data allowed to leave for an API?" means you have already sent it. It is the same reasoning as in generative AI and GDPR: control happens on the data path, not in the terms of service.

A sovereign AI agent is therefore not an agent that never uses an external service. It is an agent whose every outbound call was authorised by an explicit rule, and leaves a trace.

What to measure

An orchestration engine that is not instrumented cannot be steered, and yet four indicators are enough to know whether it is doing the work expected of it.

• Distribution of decisions: what share of volume goes to each destination. It is the first number to look at, and the one that reveals classifier drift.
• Cost per request and per team, not total cost. The total always falls at first; what matters is knowing which application consumes what.
• Time to first token, per destination. That is what the user perceives, and it differs hugely between a reasoning model and a small one : see prefill, decode and KV cache.
• Fallback rate, and its reason. Frequent fallback is either a masked outage or a policy that is too strict; both are information.

Where to start

The implementation sequence that works does not start with the classifier, contrary to what intuition suggests.

Put the gateway in place first, with a single model sitting behind it, because although it brings no routing capability yet, it does bring virtual keys, quotas and logging, and above all it reveals the real distribution of requests, which nobody knows before having measured it. Two weeks of recorded traces are worth considerably more than six meetings spent estimating the same distribution.

Then add a second destination and a single rule, the sensitivity one, articulated in two branches: it is the rule that brings the most value and demands the least arbitration. Semantic routing and fine cost arbitration come only third, once you finally know what you are routing.

We detail the building blocks of this chain in the multi-agent orchestration section: gateway, semantic router, agent harness, private MCP servers and memory. To put yours in place, a no-commitment conversation is enough to frame the first milestone.