On-premises or API: how many users before local LLM pays off?

Pay-per-token compares poorly with an amortised installation. Here is where the break-even point lies and why the KV cache changes everything.

By QDNA · Published 5 July 2026 · 10 min read

Two opposing cost structures

A pay-as-you-go interface has almost no fixed cost, but a cost that grows with every token. An on-premises installation carries a high upfront investment, then a marginal cost close to zero. The two curves cross at a break-even point that depends on usage intensity.

The break-even point by number of users

These thresholds are indicative. The presence of sensitive data mandates on-premises regardless of headcount, for compliance and sovereignty reasons.

The KV cache, on-premises' trump card

Agentic workloads are dominated by input tokens, in a ratio that can reach one hundred to one. A ten-thousand-token system prompt sent a thousand times a day costs dearly remotely and almost nothing locally.

The vLLM inference engine reuses the prefix cache for free. Time to first token drops sharply when the start of a conversation repeats, and repeated loops become nearly free. A remote interface, by contrast, bills any token not held in cache at a markedly higher rate. The combination of document cache and prompt cache sharply reduces the cost of agentic scenarios.

Sovereign hosting in France

Two paths exist. On-premises combines hardware investment with French electricity, largely nuclear in origin, hence stable and low-carbon. Managed colocation in a sovereign French data center avoids buying premises and preserves legal control, unlike operators subject to extraterritorial law.

Rationalising without exposing data

A shared hardware base serves all agents, with per-team quotas. Innovation on public data can overflow to a remote service, while sensitive data stays local. This setup rationalises cost without exposing information. The general framework is presented in our definition of the sovereign AI platform.

Fake local and its hidden costs

A team sometimes believes it runs locally while paying a remote interface bill every month. Field reports give telling orders of magnitude, converted here into euros. A team that thought it had gone local was paying close to 2,400 euros a month in remote API calls, of which 800 euros for a simple sentiment analysis buried inside a conversational agent. A young healthcare start-up carried around 300 euros a month in cloud-related compliance costs. Replacing the remote provider with a single server, on the order of 2,000 euros, brought these items down to zero. The annual saving reaches 28,800 euros in the first case and 3,600 euros in the second. The rule fits in one sentence: a partial local setup remains a billed dependency.

Levers that reduce cost per token

On-premises cost drops further once the inference engine is tuned. Several levers add up, with no extra hardware.

• Continuous batching increases throughput by thirty to one hundred percent at equal latency.
• Prefix and KV caching turn repeated input tokens into nearly free tokens.
• Quantisation fits more model onto the same graphics card, raising tokens per second and per euro.
• Speculative decoding reduces latency by a factor of two to three on conversational workloads.
• A compact exchange format between agents cuts token consumption by roughly half. For five hundred agents, the saving reaches tens of thousands of tokens per request.

The gateway unifies more than a thousand models behind a single interface. Switching from one provider to another takes one line of configuration, and the per-call overhead is measured in microseconds. Virtual keys set a budget and limits per team, while spend stays stored in your own database, on your own infrastructure in Europe. The bill stops being an unknown.

References

• Artificial Analysis, performance and cost comparisons
• vLLM, official documentation
• On-Premise vs Cloud: Generative AI Total Cost of Ownership