AI for business applications: CRM, support, ERP, documents

When each business application connects to its own AI provider, access keys, budgets and consumption tracking scatter across the organisation, with no overview of who consumes what. A single internal AI gateway centralises this access. It assigns a key and a budget per application, keeps sensitive data on your infrastructure, and makes spend traceable application by application.

By QDNA · Published 16 July 2026 · 9 min read

The problem: integrating AI into every application is expensive

When each business application embeds its own AI provider, the same work repeats in every team: contract negotiation, managing a separate key, cost monitoring, classifying data sensitivity, updating models. Keys multiply, budgets become opaque, and compliance requires effort specific to each integration. Switching provider means rebuilding connectors where the organisation already has a support team in place.

Moving to a single internal API reverses the burden. Applications call a stable entry point, and the platform handles model selection, keys, budgets and logging. Business teams consume a capability; they no longer integrate a different provider for every new application.

A single internal API behind the application estate

The CRM, customer support, ERP and document management address the same API, exposed by the LiteLLM gateway on the internal network. Each application keeps its own code and user interface; the AI function changes provider without recompilation. The platform applies the security policy, routes the request and allocates the cost.

Migrating to the internal API takes nothing away from applications: it removes the complexity of choosing and integrating a provider. The same interface serves the local sovereign model and the external model, based on the semantic router's decision.

The LiteLLM gateway: virtual keys, budgets and quotas

LiteLLM acts as an OpenAI-compatible gateway placed in front of the models. Each user, team or application receives a virtual key, distinct from the vendor's own key. The virtual key carries three safeguards: a request cap, a token cap and a budget expressed over a cumulative time window. Alerts warn before limits are reached, and spend is broken down by key, by user, by team and by model.

Local models, whose marginal cost is zero, remain accessible once an external model's budget is exhausted. This sovereign fallback guarantees service continuity when capped spend is reached, and it preserves latency for sensitive workloads. The choice of model is decided at the gateway level, never in application code.

The semantic router: the right model in the right place

The semantic router classifies each request before the model call. Using embedding similarity, it distinguishes confidential content from public content, then directs the request to a local sovereign model or to an external model. Filtering prevents data leakage, and the decision layer logs the choice. The application no longer decides on its own whether content leaves the premises or stays; the policy is applied once, at a centralised point.

The eBPF micro-segmentation provided by Cilium restricts outbound destinations per service. Hubble makes every flow visible from layer 3 to layer 7. Tetragon monitors executions, privilege escalations and unauthorised connections, and logs every attempt. No data leaves the platform through an unplanned channel, and the entire chain remains demonstrable at audit.

The architecture behind the internal API

The internal API placed in front of the application estate rests on a set of building blocks documented in the architecture dossier. The LiteLLM page describes the gateway, virtual keys and quotas. The semantic router page details classification and routing policy. The private MCP servers page covers the self-hosted connectors that equip agents without exposing an external API. The API and exposure layer brings together the platform's interfaces and protocols. The models layer lists the sovereign and remote models available. The inference runtime layer hosts vLLM, Triton, Dynamo and llama.cpp, and adapts the engine to workload and hardware.

This organisation makes the internal API a platform component rather than an isolated development effort. Applications integrate once, and the platform evolves upstream, without touching the application estate.

Measuring cost and value per application

VictoriaMetrics and VictoriaLogs collect metrics and logs across the whole stack: vLLM inference, the LiteLLM gateway, agents, Cilium and Hubble network policies, hardware telemetry. A single, queryable stream shows who consumes what, where latency sits, and which application delivers value. Spend can be read at the virtual-key level, hence at the level of the application calling the API.

Reconciling AI cost with the business process becomes possible. A sales leadership team compares AI cost per opportunity handled in the CRM. A support team measures AI cost per ticket resolved. A finance team tracks the local share versus the external share, and arbitrates the switchover point between the two regimes. Investment decisions on sovereign accelerators feed on this data, and next year's budget is built from this year's actual usage.

References

• Implementation timeline, EU AI Act service desk (European Commission)
• EU AI Act implementation timeline